[Home]

Summary:ASTERISK-05887: [patch] ATF - Asterisk Testing Framework
Reporter:Olle Johansson (oej)Labels:
Date Opened:2005-12-22 07:39:37.000-0600Date Closed:2011-06-07 14:10:49
Priority:MinorRegression?No
Status:Closed/CompleteComponents:Core/General
Versions:Frequency of
Occurrence
Related
Issues:
Environment:Attachments:( 0) codetest_1-1-05.diff
Description:I have the framework for building tests within Asterisk. The test code will only be compiled and installed if it is enabled in the Makefile. This can be used to build test functions of internal functions in Asterisk, like SIP parsers, Realtime drivers, manager interfaces and other things that need to conform to a set of criteria that can be tested.

Please test and give me input. The basic framework is very basic and ready for trunk if accepted. After that, we can start developing test functions in different modules.

****** ADDITIONAL INFORMATION ******

The test framework is available in svn

http://svn.digium.com/svn/asterisk/team/oej/moduletest/

Comments:By: Russell Bryant (russell) 2005-12-22 13:47:05.000-0600

First of all, I think this is a *great* idea.

Currently, you have the ability to execute a single function, 'codetest', in a module.  What do you think about allowing modules to register callbacks for as many different tests as it wants?

We could even write some macros for the register/unregister functions, as well as for declaring the test functions themselves so that they will only be built with ENABLE_TESTS (or whatever it is) defined.  That way, you would not have to manually put an #ifdef everywhere you were writing something related to test code.

If you're interested, I'd be happy to help contribute to your branch.  :)
By: Olle Johansson (oej) 2005-12-22 15:15:29.000-0600

I think each module will call as many tests as it wants from the single codetest function. I am working on a template for chan_sip, see my next commit.

I really appreciate your offer to help in this, partner. Let's discuss over IRC or Jabber.
By: Russell Bryant (russell) 2005-12-24 20:48:42.000-0600

I have created a branch with an implementation of some of my ideas ...

svn co http://svn.digium.com/svn/asterisk/team/russell/codetest russell_codetest

I have added codetest.c and include/asterisk/codetest.h.  These files contain all of the necessary definitions to let any file (including builtin files such as channel.c or pbx.c) register code tests.  I have also added one of your tests from chan_sip.c.  Of course, no extra code gets built if ENABLE_CODE_TESTS is not defined.

The following CLI commands have been implemented ...

This command will show all files that have registered code tests:
    *CLI> codetest show files
    Files with Registered Code Tests
    --------------------------------
    Filename: 'chan_sip.c'  (1 test)


This command will show the names of all tests registered by a specific file:
    *CLI> codetest show file chan_sip.c
    Registered Code Tests for File 'chan_sip.c'
    ---------------------------------------------
    Test Name: 'tags'

    (Total Tests Registered: 1)


This command will run all tests registered by a specific file:
    *CLI> codetest run chan_sip.c
    Executing test 'tags' for file 'chan_sip.c' ...
    Dec 24 22:44:33 ERROR[17637]: chan_sip.c:13096 test_tags_func: Wrong From tag in packet one, parse error. Tag = <none>
    Test 'tags' *** FAILED *** for File 'chan_sip.c' !!!


This command will run a specific test for a specified file:
    *CLI> codetest run chan_sip.c tags
    Executing test 'tags' for file 'chan_sip.c' ...
    Dec 24 22:44:43 ERROR[17637]: chan_sip.c:13096 test_tags_func: Wrong From tag in packet one, parse error. Tag = <none>
    Test 'tags' *** FAILED *** for File 'chan_sip.c' !!!


Take a look and let me know what you think!
By: Russell Bryant (russell) 2005-12-25 14:04:32.000-0600

Maybe we can merge these two methods.

We could allow files to register individual tests.  Also, when a module is loaded, if the codetest function is present, we can automatically register that as a test for that module.
By: Russell Bryant (russell) 2006-01-01 19:34:14.000-0600

I'd also like to note that I added the (untested) ability to pass arguments to tests from the CLI.  However, this only works when running an individual test.

*CLI> codetest run my_file.c my_test arg1 arg2 arg3 ... argN

will execute the appropriate test with the argc argument set to 'N' and the values of argv[] set to { "arg1", "arg2", "arg3", ..., "argN" }
By: Russell Bryant (russell) 2006-01-02 00:38:08.000-0600

oej has indicated on IRC that he is interested in pursuing work with the code I have developed.

I have attached a diff that shows the current changes that are present in my codetest branch.  Please provide feedback based on all of the changes *except* the tests themselves.  They are a proof of concept at this point, and not necessarily an indication of the final versions of tests we are proposing to merge into the trunk.
By: Olle Johansson (oej) 2006-01-02 01:39:33.000-0600

I think both methods can work together. However, we need to implement several CLI commands. From our discussion, I can remember:

* test module <modulename> <args>
* test run <testname> <args>
* test list
* test all [tests | modules]

I also prefer having all the test code at the end of the module if possible, like I did in chan_sip.c
By: Russell Bryant (russell) 2006-01-02 02:19:24.000-0600

- Personally, I think I'd rather reserve the CLI command 'test' in case we were to implement some kind of testing that would be done by users.  I was using 'codetest' to make it very clear that this was actually testing code.  I guess that could just be decided by whoever reviews and (maybe) commits it.

- Do you not think it makes sense to have the file that registered the test in the CLI command?  I felt like it helped make it more clear.  For example, 'codetest run chan_sip.c tags' as opposed to 'codetest run tags'.
By: Olle Johansson (oej) 2006-01-02 02:55:24.000-0600

I think we should try to reach agreement and not leave it up to the committer. That is a solution we can use when we can not agree at all, even after wrestling :-). I can accept your argument, since it's the reason I did not use "test" from start.

I see a lot of use of named tests within core, and there's no module name there. We should define a naming that starts with a well-known module tag, like all SIp tests starting with "sip-", to avoid collissions. Some tests could in theory span several modules, and that part of the command does not make sense.
By: Olle Johansson (oej) 2006-01-26 02:28:36.000-0600

Shall we ask Mr. Fleming to open up a group branch for this?
By: Russell Bryant (russell) 2006-01-26 11:01:33.000-0600

Sure.  :)
By: Olle Johansson (oej) 2006-03-10 07:08:35.000-0600

Asked KPF for a group branch some time ago. Still waiting...
By: Olle Johansson (oej) 2006-03-29 18:47:49.000-0600

Closing this until I and Russell have time to restart.
By: Olle Johansson (oej) 2006-04-15 18:27:31

We moved this to a new group branch
By: Russell Bryant (russell) 2006-05-03 20:59:14

I'm closing this out for now since we're not doing anything with this code right now.  The branch will stay in svn for when we decide to use it for something.
By: Digium Subversion (svnbot) 2008-01-15 16:08:49.000-0600

Repository: asterisk
Revision: 7621

A   team/russell/codetest/

------------------------------------------------------------------------
r7621 | russell | 2008-01-15 16:08:49 -0600 (Tue, 15 Jan 2008) | 3 lines

create branch for playing with ideas for oej's code testing framework
discussed in issue ASTERISK-5887 in mantis

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=7621
By: Digium Subversion (svnbot) 2008-01-15 16:08:50.000-0600

Repository: asterisk
Revision: 7622

U   team/russell/codetest/Makefile
U   team/russell/codetest/asterisk.c
U   team/russell/codetest/channels/chan_sip.c
A   team/russell/codetest/codetest.c
A   team/russell/codetest/include/asterisk/codetest.h
U   team/russell/codetest/include/asterisk.h

------------------------------------------------------------------------
r7622 | russell | 2008-01-15 16:08:50 -0600 (Tue, 15 Jan 2008) | 3 lines

add an implementation of some of my ideas
I will post details to issue ASTERISK-5887 in a few minutes

------------------------------------------------------------------------

http://svn.digium.com/view/asterisk?view=rev&revision=7622