Changes between Version 1 and Version 2 of Building/RunningTests

Oct 24, 2006 10:45:45 AM (9 years ago)

cut-n-paste more from the README


  • Building/RunningTests

    v1 v2  
    12= GHC Test framework = 
    4041For more details, see below. 
     43= Running the testsuite with a compiler other than GHC = 
     45(to be written.  The plan is something like:  
     47        cvs checkout fpconfig 
     48        cd fptools 
     49        cvs checkout testsuite 
     50        autoconf 
     51        ./configure 
     52        cd testsuite 
     53        make TEST_HC=nhc98 COMPILER=nhc98 
     57= Running individual tests or subdirectories of the testsuite = 
     59Most of the subdirectories in the testsuite have a Makefile.  In these 
     60subdirectories you can use 'make' to run the test driver in two 
     63        make            -- run all the tests in the current directory 
     64        make accept     -- run the tests, accepting the current output 
     66The following variables may be set on the make command line: 
     68        TESTS                   -- specific tests to run 
     69        TEST_HC                 -- compiler to use 
     70        EXTRA_HC_OPTS           -- extra flags to send to the Haskell compiler 
     71        EXTRA_RUNTEST_OPTS      -- extra flags to give the test driver 
     72        CONFIG                  -- use a different configuration file 
     73        COMPILER                -- stem of a different configuration file 
     74                                -- from the config directory [default: ghc] 
     75        WAY                     -- just this way 
     77The following ways are defined (for GHC, also see the file config/ghc): 
     79        normal                  -- no special options 
     80        opt                     -- -O 
     81        optasm                  -- -O -fasm 
     82        prof                    -- -O -prof -auto-all 
     83        profasm                 -- -O -prof -auto-all -fasm 
     84        unreg                   -- -unreg 
     85        ghci                    -- (run only, not compile) run test under GHCi 
     86        extcore                 -- -fext-core 
     87        optextcore              -- -O -fext-core 
     88        threaded                -- -threaded 
     90certain ways are enabled automatically if the GHC build in the local 
     91tree supports them.  Ways that are enabled this way are optasm, prof, 
     92profasm, unreg, threaded, and ghci. 
     94= Updating tests when the output changes = 
     96If the output of a test has changed, but the new output is still 
     97correct, you can automatically update the sample output to match the 
     98new output like so: 
     100        make accept TESTS=<test-name> 
     102where <test-name> is the name of the test.  In a directory which 
     103contains a single test, or if you want to update *all* the tests in 
     104the current directory, just omit the 'TESTS=<test-name>' part. 
     106= Adding a new test = 
     108For a test which can be encapsulated in a single source file, follow 
     109these steps: 
     111 1. Find the appropriate place for the test.  The GHC regression suite 
     112    is generally organised in a "white-box" manner: a regression which 
     113    originally illustrated a bug in a particular part of the compiler 
     114    is placed in the directory for that part.  For example, typechecker 
     115    regression tests go in the typechecker/ directory, parser tests 
     116    go in parser/, and so on.   
     118    It's not always possible to find a single best place for a test; 
     119    in those cases just pick one which seems reasonable. 
     121    Under each main directory may be up to three subdirectories: 
     123       should_compile: 
     124           tests which need to compile only 
     126       should_fail:     
     127           tests which should fail to compile and generate a particular error message 
     129       should_run: 
     130           tests which should compile, run with some specific input, and generate a particular output. 
     132    We don't always divide the tests up like this, and it's not 
     133    essential to do so (the directory names have no meaning as 
     134    far as the test driver is concerned).         
     137 2. Having found a suitable place for the test, give the test a name. 
     138    Follow the convention for the directory in which you place the 
     139    test: for example, in typecheck/should_compile, tests are named 
     140    tc001, tc002, and so on.  Suppose you name your test T, then 
     141    you'll have the following files: 
     143      T.hs 
     144        The source file containing the test 
     146      T.stdin   (for tests that run, and optional) 
     147        A file to feed the test as standard input when it 
     148        runs. 
     150      T.stdout  (for tests that run, and optional) 
     151        For tests that run, this file is compared against 
     152        the standard output generated by the program.  If  
     153        T.stdout does not exist, then the program must not 
     154        generate anything on stdout. 
     156      T.stderr  (optional) 
     157        For tests that run, this file is compared 
     158        against the standard error generated by the program. 
     160        For tests that compile only, this file is compared 
     161        against the standard error output of the compiler, 
     162        which is normalised to eliminate bogus differences 
     163        (eg. absolute pathnames are removed, whitespace 
     164        differences are ignored, etc.) 
     167 1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form 
     169      test(<name>, <opt-fn>, <test-fn>, <args>) 
     171    where 
     173    ''<name>'' is the name of the test, in quotes (' or "). 
     175    ''<opt-fn>''  is a function (i.e. any callable object in Python) 
     176    which allows the options for this test to be changed. 
     177    There are several pre-defined functions which can be 
     178    used in this field: 
     180        '''normal'''                don't change any options from the defaults 
     182        '''skip'''                  skip this test 
     184        '''omit_ways(ways)'''       skip this test for certain ways 
     186        '''only_ways(ways)'''       do this test certain ways only 
     188        '''omit_compiler_types(compilers)'''                           skip this test for certain compilers 
     190        '''only_compiler_types(compilers)'''       do this test for certain compilers only 
     192        '''expect_fail'''           this test is an expected failure 
     194        '''expect_fail_for(ways)''' expect failure for certain ways  
     196        '''expect_fail_if_platform(plat)'''      expect failure on a certain platform 
     198        '''expect_fail_if_compiler_type(compiler)'''   expect failure from a certain compiler 
     200        '''set_stdin(file)'''       use a different file for stdin 
     202        '''exit_code(n)'''          expect an exit code of 'n' from the prog 
     204        '''extra_run_opts(opts)'''  pass some extra opts to the prog 
     206        '''no_clean'''              don't clean up after this test 
     208      You can compose two of these functions together by 
     209      saying compose(f,g).  For example, to expect an exit 
     210      code of 3 and omit way 'opt', we could use 
     212      compose(omit_ways(['opt']), exit_code(3)) 
     214      as the <opt-fn> argument.  Calls to compose() can of 
     215      course be nested. 
     217    ''<test-fn>'' 
     218    is a function which describes how the test should be 
     219    run, and determines the form of <args>.  The possible 
     220    values are: 
     222      compile 
     223            Just compile the program, the compilation should succeed. 
     225      compile_fail 
     226            Just compile the program, the 
     227            compilation should fail (error 
     228            messages will be in T.stderr). 
     230      compile_and_run  
     231            Compile the program and run it, 
     232            comparing the output against the  
     233            relevant files. 
     235      multimod_compile 
     236            Compile a multi-module program 
     237            (more about multi-module programs 
     238            below). 
     240      multimod_compile_fail 
     241            Compile a multi-module program, 
     242            and expect the compilation to fail 
     243            with error messages in T.stderr 
     245      multimod_compile_and_run 
     246            Compile and run a multi-module 
     247            program. 
     249      run_command 
     250            Just run an arbitrary command.  The 
     251            output is checked against T.stdout and 
     252            T.stderr, and the stdin and expected 
     253            exit code can be changed in the same 
     254            way as for compile_and_run. 
     256      run_command_ignore_output 
     257            Same as run_command, except the output 
     258            (both stdout and stderr) from the 
     259            command is ignored. 
     261      ghci_script 
     262            Runs the current compiler, passing 
     263            --interactive and using the specified 
     264            script as standard input. 
     266    ''<args>'' 
     267    is a list of arguments to be passed to <test-fn>. 
     269    For compile, compile_fail and compile_and_run, <args> 
     270    is a list with a single string which contains extra 
     271    compiler options with which to run the test.  eg. 
     273    test(tc001, normal, compile, ['-fglasgow-exts']) 
     275    would pass the flag -fglasgow-exts to the compiler 
     276    when compiling tc001. 
     278    The multimod_ versions of compile and compile_and_run 
     279    expect an extra argument on the front of the list: the 
     280    name of the top module in the program to be compiled 
     281    (usually this will be 'Main'). 
     284A multi-module test is straightforward.  It usually goes in a 
     285directory of its own (although this isn't essential), and the source 
     286files can be named anything you like.  The test must have a name, in 
     287the same way as a single-module test; and the stdin/stdout/stderr 
     288files follow the name of the test as before.  In the same directory, 
     289place a file 'test.T' containing a line like 
     291   test(multimod001, normal, multimod_compile_and_run, \ 
     292                 [ 'Main', '-fglasgow-exts', '', 0 ]) 
     294as described above. 
     296For some examples, take a look in tests/ghc-regress/programs. 
     298= The details = 
     300The test suite driver is just a set of Python scripts, as are all of 
     301the .T files in the test suite.  The driver (driver/ first 
     302searches for all the .T files it can find, and then proceeds to 
     303execute each one, keeping a track of the number of tests run, and 
     304which ones succeeded and failed. 
     306The script takes several options: 
     308  --config <file> 
     310       <file> is just a file containing Python code which is  
     311       executed.   The purpose of this option is so that a file 
     312       containing settings for the configuration options can 
     313       be specified on the command line.  Multiple --config options 
     314       may be given. 
     316  --rootdir <dir> 
     318       <dir> is the directory below which to search for .T files 
     319       to run. 
     321  --output-summary <file> 
     323       In addition to dumping the test summary to stdout, also 
     324       put it in <file>.  (stdout also gets a lot of other output 
     325       when running a series of tests, so redirecting it isn't   
     326       always the right thing). 
     328  --only <test> 
     330       Only run tests named <test> (multiple --only options can 
     331       be given).  Useful for running a single test from a .T file 
     332       containing multiple tests. 
     334  -e <stmt> 
     336       executes the Python statement <stmt> before running any tests. 
     337       The main purpose of this option is to allow certain 
     338       configuration options to be tweaked from the command line; for 
     339       example, the build system adds '-e config.accept=1' to the 
     340       command line when 'make accept' is invoked. 
     342Most of the code for running tests is located in driver/ 
     343Take a look. 
     345There is a single Python class (TestConfig) containing the global 
     346configuration for the test suite.  It contains information such as the 
     347kind of compiler being used, which flags to give it, which platform 
     348we're running on, and so on.  The idea is that each platform and 
     349compiler would have its own file containing assignments for elements 
     350of the configuration, which are sourced by passing the appropriate 
     351--config options to the test driver.  For example, the GHC 
     352configuration is contained in the file config/ghc. 
     354A .T file can obviously contain arbitrary Python code, but the general 
     355idea is that it contains a sequence of calls to the function test(), 
     356which resides in  As described above, test() takes four 
     359      test(<name>, <opt-fn>, <test-fn>, <args>) 
     361The function <opt-fn> is allowed to be any Python callable object, 
     362which takes a single argument of type TestOptions.  TestOptions is a 
     363class containing options which affect the way that the current test is 
     364run: whether to skip it, whether to expect failure, extra options to 
     365pass to the compiler, etc. (see for the definition of the 
     366TestOptions class).  The idea is that the <opt-fn> function modifies 
     367the TestOptions object that it is passed.  For example, to expect 
     368failure for a test, we might do this in the .T file: 
     370   def fn(opts): 
     371      opts.expect = 'fail' 
     373   test(test001, fn, compile, ['']) 
     375so when fn is called, it sets the instance variable "expect" in the 
     376instance of TestOptions passed as an argument, to the value 'fail'. 
     377This indicates to the test driver that the current test is expected to 
     380Some of these functions, such as the one above, are common, so rather 
     381than forcing every .T file to redefine them, we provide canned 
     382versions.  For example, the provided function expect_fail does the 
     383same as fn in the example above.  See for all the canned 
     384functions we provide for <opt-fn>. 
     386The argument <test-fn> is a function which performs the test.  It 
     387takes three or more arguments: 
     389      <test-fn>( <name>, <way>, ... ) 
     391where <name> is the name of the test, <way> is the way in which it is 
     392to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments 
     393are constructed from the list <args> in the original call to test(). 
     394The following <test-fn>s are provided at the moment: 
     396           compile 
     397           compile_fail 
     398           compile_and_run 
     399           multimod_compile 
     400           multimod_compile_fail 
     401           multimod_compile_and_run 
     402           run_command 
     403           run_command_ignore_output 
     404           ghci_script 
     406and obviously others can be defined.  The function should return 
     407either 'pass' or 'fail' indicating that the test passed or failed