30. Pike Test Suite

FIXME: The goals of the test suite and an overview of it

30.1. Running Tests

The most common way of running tests from the test suite is to use the top level make target verify which installs a test Pike in the build directory and use it while running the entire test suite. The following test-related make targets are defined in the top level make file.

tinstall
Makes and installs a test Pike binary in the build directory. If a test Pike binary was already installed, it will first be removed.
just_verify
Runs the test suite with the flags "-F -a -v", without installing a new test Pike binary.
testsuites
Creates testsuite files in the build tree from the testsuite.in-files in the src/ and lib/ trees.
verify
Runs the testsuites, tinstall and just_verify targets.
verify_installed
Runs the test suit with the flags "-F -a -v", with the Pike binary installed on the system.
check
Alias for the verify make target.
sure
Alias for the verify make target.
verbose_verify
Runs the tinstall make target and then runs the test suite with the flags "-F -v -v -a".
gdb_verify
Runs the test suite inside of gdb. The test suite is started with the flags "-F -v -v -a".
valgrind_verify
Runs the test suite inside of valgrind. The test suite is started with the flags "-F -v -a".
valgrind_just_verify
Runs the test suite inside of valgrind, without installing a test pike. The test suite is started with the flags "-F -v -a".

It is possible to alter the flags given to the test_install program by using the TESTARGS make variable.

make verify TESTARGS="-a -v4 -l2 -t1 -c1 -m"

30.1.1. The Test Program

The actual testing is done by the program bin/test_pike.pike, which can be run as a stand alone application to test any Pike binary with any test suite or test suites. The Pike binary that executes the test program will be tested, and it will be tested with the test suites provided as arguments to the test program.

/home/visbur/Pike/7.2/bin/pike bin/test_pike.pike testsuite1 testsuite2

The individual testsuite files are generated from testsuite.in files scattered about the lib/ and src/ trees. When you run the make targets described above, those are made for you automagically, but to do it by hand (i e if you added a test to one of them), cd to the top directory and run

make testsuites

The testsuite files have now appeared in build/arch in locations corresponding to where they lived in the pike tree, except those from the lib/ hierarchy; those end up in build/arch/tlib.

The test_pike.pike program takes the following attributes.

-h, --help
Displays a help message listing all possible arguments.
-a, --auto
Let the test program find the testsuits self. It will search for files named testsuite or module_testsuite in the current working directory and all subdirectories.
--no-watchdog
Normally the the test program has a watchdog activated that aborts testing if a test takes more than 20 minutes to complete (or 80 minutes if Pike is compiled with dmalloc). With this argument the watchdog will not be used.
--watchdog=pid
Run only the watchdog and monitor the process with the given pid.
-v[level], --verbose[=level]
Select the level of verbosity. Every verbose level includes the printouts from the levels below.
0No extra printouts.
1Some additional information printed out after every finished block of tests.
2Some extra information about test that will or won't be run.
3Every test is printed out.
4Time spent in individual tests are printed out.
10The actual pike code compiled, including wrappers, is printed. Note that the code will be quoted.
$ pike bin/test_pike.pike -v1 testsuite 
Doing tests in testsuite (1 tests)
Total tests: 1  (0 tests skipped)       
$ pike bin/test_pike.pike -v2 testsuite
Doing tests in testsuite (1 tests)
Doing test 1 (1 total) at /home/nilsson/Pike/7.3/lib/modules/ADT.pmod/testsuite.in:9
Failed tests: 0.                        
Total tests: 1  (0 tests skipped)
$ pike bin/test_pike.pike -v4 testsuite 
Doing tests in testsuite (1 tests)
Doing test 1 (1 total) at /home/nilsson/Pike/7.3/lib/modules/ADT.pmod/testsuite.in:9
  0: mixed a() { 
  1:   object s = ADT.Stack();
  2:   s->push(1);
  3:   return s->pop();
  4: ; }
  5: mixed b() { return 1; }

Time in a(): 0.000, Time in b(): 0.000000
Failed tests: 0.                        
Total tests: 1  (0 tests skipped)
$ pike bin/test_pike.pike -v10 testsuite 
Doing tests in testsuite (1 tests)
Doing test 1 (1 total) at /home/nilsson/Pike/7.3/lib/modules/ADT.pmod/testsuite.in:9
  0: mixed a() { 
  1:   object s = ADT.Stack();
  2:   s->push(1);
  3:   return s->pop();
  4: ; }
  5: mixed b() { return 1; }

  0: mixed a() { 
  1:   object s = ADT.Stack();
  2:   s->push(1);
  3:   return s->pop();
  4: ; }
  5: mixed b() { return 1; }
  6: int __cpp_line=__LINE__; int __rtl_line=[int]backtrace()[-1][1];
  7: 
  8: int \30306\30271\30310=0;
  9: 

Time in a(): 0.000, Time in b(): 0.000000
Failed tests: 0.                        
Total tests: 1  (0 tests skipped)
-p, --prompt
The user will be asked before every test is run.
-sX, --start-test=X
Where in the testsuite testing should start, e.g. ignores X tests in every testsuite.
-eX, --end-after=X
How many tests should be run.
-f, --fail
If set, the test program exits on first failure.
-F, --fork
If set, each testsuite will run in a separate process.
-lX, --loop=X
The number of times the testsuite should be run. Default is 1.
-tX, --trace=X
Run tests with trace level X.
-c[X], --check[=X]
The level of extra pike consistency checks performed.
1_verify_internals is run before every test.
2_verify_internals is run after every compilation.
3_verify_internals is run after every test.
4An extra gc and _verify_internals is run before every test.
X<0For values below zero, _verify_internals will be run before every n:th test, where n=abs(X).
-m, --mem, --memory
Prints out memory allocations after the tests.
-T, --notty
Format output for non-tty.
-d, --debug
Opens a debug port.

30.2. Writing New Tests

Whenever you write a new function in a module or in Pike itself it is good to add a few test cases in the test suite to ensure that regressions are spotted as soon as they appear or to aid in finding problems when porting Pike to another platform. Since you have written the code, you are the one best suited to come up with tricky tests cases. A good test suite for a function includes both some trivial tests to ensure that the basic functionality works and some nasty tests to test the borderlands of what the function is capable of, e.g. empty in parameters.

Also, when a bug in Pike has been found, a minimized test case the triggers the bug should also be added to the test suite. After all, this test case has proven to be a useful once.

30.2.1. test_any

The test_any macro tests if the result of two pike expressions are similar, e.g. if a==b. Technically the actual test preformed is !(a!=b). The first expression should be a complete block, that returns a value, while the other expression should be a simple pike statement.

test_any([[
  int f (int i) {i = 0; return i;};
  return f (1);
]],0)

30.2.2. test_any_equal

The test_any_equal macro tests if the result of two pike expressions are identical, e.g. if equal(a,b). The first expression should be a complete block, that returns a value, while the other expression should be a simple pike statement.

test_any_equal([[
  mixed a=({1,2,3});
  a[*] += 1;
  return a;
]], [[ ({2,3,4}) ]])

30.2.3. test_eq

The test_eq macro tests if the result of two pike statements are similar, e.g. if a==b. Technicaly the actual test performed is !(a!=b).

test_eq(1e1,10.0);

30.2.4. test_equal

The test_equal macro tests if the result of two pike statements are identical, e.g. if equal(a,b).

test_equal([[ ({10,20})[*] + 30  ]], [[ ({40, 50}) ]])

30.2.5. test_do

test_do simply executes its code. This test fails if there is any compilation error or if an error is thrown during execution.

test_do([[
  int x;
  if (time())
    x = 1;
  else
    foo: break foo;
]])

30.2.6. test_true

This test succeeds if the pike expression is evaluated into a non-zero value.

test_true([[1.0e-40]]);

30.2.7. test_false

This test succeeds if the pike expression is evaluated into a zero value.

test_false(glob("*f","foo"))

30.2.8. test_compile

The test_compile macro only tries to compile an expression. It fails upon compilarion warnings or errors.

test_compile([[Stdio.File foo=Stdio.File();]])

30.2.9. test_compile_any

Tests if the code compiles, just as test_compile, but is a complete block of code and not just an expression.

test_compile_any([[
  void foo() 
  {
    Stdio.File bar(int x, int y)
    { 
      return 0;
    }; 
  } 
]])

30.2.10. test_compile_error

Does the inverse of test_compile; verifies that the code does not compile.

test_compile_error([[ int a="a"; ]])

30.2.11. test_compile_error_any

Does the inverse of test_compile_any; verifies that the code does not compile.

test_compile_error_any([[
  int a=5;
  string b="a";
  a=b;
]])

30.2.12. test_compile_warning

30.2.13. test_eval_error

30.2.14. test_define_program

30.2.15. test_program

30.2.16. cond

30.2.17. ifefun

30.2.18. nonregression