Changes between Version 1 and Version 2 of Building/RunningTests


Ignore:
Timestamp:
Oct 24, 2006 10:45:45 AM (7 years ago)
Author:
malcolm.wallace@…
Comment:

cut-n-paste more from the README

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests

    v1 v2  
     1[[PageOutline]] 
    12= GHC Test framework = 
    23 
     
    4041For more details, see below. 
    4142 
    42  
    43  
     43= Running the testsuite with a compiler other than GHC = 
     44 
     45(to be written.  The plan is something like:  
     46{{{ 
     47        cvs checkout fpconfig 
     48        cd fptools 
     49        cvs checkout testsuite 
     50        autoconf 
     51        ./configure 
     52        cd testsuite 
     53        make TEST_HC=nhc98 COMPILER=nhc98 
     54}}} 
     55) 
     56 
     57= Running individual tests or subdirectories of the testsuite = 
     58 
     59Most of the subdirectories in the testsuite have a Makefile.  In these 
     60subdirectories you can use 'make' to run the test driver in two 
     61ways: 
     62{{{ 
     63        make            -- run all the tests in the current directory 
     64        make accept     -- run the tests, accepting the current output 
     65}}} 
     66The following variables may be set on the make command line: 
     67{{{ 
     68        TESTS                   -- specific tests to run 
     69        TEST_HC                 -- compiler to use 
     70        EXTRA_HC_OPTS           -- extra flags to send to the Haskell compiler 
     71        EXTRA_RUNTEST_OPTS      -- extra flags to give the test driver 
     72        CONFIG                  -- use a different configuration file 
     73        COMPILER                -- stem of a different configuration file 
     74                                -- from the config directory [default: ghc] 
     75        WAY                     -- just this way 
     76}}} 
     77The following ways are defined (for GHC, also see the file config/ghc): 
     78{{{ 
     79        normal                  -- no special options 
     80        opt                     -- -O 
     81        optasm                  -- -O -fasm 
     82        prof                    -- -O -prof -auto-all 
     83        profasm                 -- -O -prof -auto-all -fasm 
     84        unreg                   -- -unreg 
     85        ghci                    -- (run only, not compile) run test under GHCi 
     86        extcore                 -- -fext-core 
     87        optextcore              -- -O -fext-core 
     88        threaded                -- -threaded 
     89}}} 
     90certain ways are enabled automatically if the GHC build in the local 
     91tree supports them.  Ways that are enabled this way are optasm, prof, 
     92profasm, unreg, threaded, and ghci. 
     93 
     94= Updating tests when the output changes = 
     95 
     96If the output of a test has changed, but the new output is still 
     97correct, you can automatically update the sample output to match the 
     98new output like so: 
     99{{{ 
     100        make accept TESTS=<test-name> 
     101}}} 
     102where <test-name> is the name of the test.  In a directory which 
     103contains a single test, or if you want to update *all* the tests in 
     104the current directory, just omit the 'TESTS=<test-name>' part. 
     105 
     106= Adding a new test = 
     107 
     108For a test which can be encapsulated in a single source file, follow 
     109these steps: 
     110 
     111 1. Find the appropriate place for the test.  The GHC regression suite 
     112    is generally organised in a "white-box" manner: a regression which 
     113    originally illustrated a bug in a particular part of the compiler 
     114    is placed in the directory for that part.  For example, typechecker 
     115    regression tests go in the typechecker/ directory, parser tests 
     116    go in parser/, and so on.   
     117 
     118    It's not always possible to find a single best place for a test; 
     119    in those cases just pick one which seems reasonable. 
     120 
     121    Under each main directory may be up to three subdirectories: 
     122 
     123       should_compile: 
     124           tests which need to compile only 
     125 
     126       should_fail:     
     127           tests which should fail to compile and generate a particular error message 
     128 
     129       should_run: 
     130           tests which should compile, run with some specific input, and generate a particular output. 
     131     
     132    We don't always divide the tests up like this, and it's not 
     133    essential to do so (the directory names have no meaning as 
     134    far as the test driver is concerned).         
     135 
     136 
     137 2. Having found a suitable place for the test, give the test a name. 
     138    Follow the convention for the directory in which you place the 
     139    test: for example, in typecheck/should_compile, tests are named 
     140    tc001, tc002, and so on.  Suppose you name your test T, then 
     141    you'll have the following files: 
     142 
     143      T.hs 
     144        The source file containing the test 
     145 
     146      T.stdin   (for tests that run, and optional) 
     147        A file to feed the test as standard input when it 
     148        runs. 
     149 
     150      T.stdout  (for tests that run, and optional) 
     151        For tests that run, this file is compared against 
     152        the standard output generated by the program.  If  
     153        T.stdout does not exist, then the program must not 
     154        generate anything on stdout. 
     155 
     156      T.stderr  (optional) 
     157        For tests that run, this file is compared 
     158        against the standard error generated by the program. 
     159 
     160        For tests that compile only, this file is compared 
     161        against the standard error output of the compiler, 
     162        which is normalised to eliminate bogus differences 
     163        (eg. absolute pathnames are removed, whitespace 
     164        differences are ignored, etc.) 
     165 
     166 
     167 1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form 
     168{{{ 
     169      test(<name>, <opt-fn>, <test-fn>, <args>) 
     170}}} 
     171    where 
     172 
     173    ''<name>'' is the name of the test, in quotes (' or "). 
     174 
     175    ''<opt-fn>''  is a function (i.e. any callable object in Python) 
     176    which allows the options for this test to be changed. 
     177    There are several pre-defined functions which can be 
     178    used in this field: 
     179 
     180        '''normal'''                don't change any options from the defaults 
     181 
     182        '''skip'''                  skip this test 
     183 
     184        '''omit_ways(ways)'''       skip this test for certain ways 
     185 
     186        '''only_ways(ways)'''       do this test certain ways only 
     187 
     188        '''omit_compiler_types(compilers)'''                           skip this test for certain compilers 
     189 
     190        '''only_compiler_types(compilers)'''       do this test for certain compilers only 
     191 
     192        '''expect_fail'''           this test is an expected failure 
     193 
     194        '''expect_fail_for(ways)''' expect failure for certain ways  
     195 
     196        '''expect_fail_if_platform(plat)'''      expect failure on a certain platform 
     197 
     198        '''expect_fail_if_compiler_type(compiler)'''   expect failure from a certain compiler 
     199 
     200        '''set_stdin(file)'''       use a different file for stdin 
     201 
     202        '''exit_code(n)'''          expect an exit code of 'n' from the prog 
     203 
     204        '''extra_run_opts(opts)'''  pass some extra opts to the prog 
     205 
     206        '''no_clean'''              don't clean up after this test 
     207 
     208      You can compose two of these functions together by 
     209      saying compose(f,g).  For example, to expect an exit 
     210      code of 3 and omit way 'opt', we could use 
     211{{{ 
     212      compose(omit_ways(['opt']), exit_code(3)) 
     213}}} 
     214      as the <opt-fn> argument.  Calls to compose() can of 
     215      course be nested. 
     216 
     217    ''<test-fn>'' 
     218    is a function which describes how the test should be 
     219    run, and determines the form of <args>.  The possible 
     220    values are: 
     221 
     222      compile 
     223            Just compile the program, the compilation should succeed. 
     224 
     225      compile_fail 
     226            Just compile the program, the 
     227            compilation should fail (error 
     228            messages will be in T.stderr). 
     229                                 
     230      compile_and_run  
     231            Compile the program and run it, 
     232            comparing the output against the  
     233            relevant files. 
     234 
     235      multimod_compile 
     236            Compile a multi-module program 
     237            (more about multi-module programs 
     238            below). 
     239 
     240      multimod_compile_fail 
     241            Compile a multi-module program, 
     242            and expect the compilation to fail 
     243            with error messages in T.stderr 
     244 
     245      multimod_compile_and_run 
     246            Compile and run a multi-module 
     247            program. 
     248 
     249      run_command 
     250            Just run an arbitrary command.  The 
     251            output is checked against T.stdout and 
     252            T.stderr, and the stdin and expected 
     253            exit code can be changed in the same 
     254            way as for compile_and_run. 
     255 
     256      run_command_ignore_output 
     257            Same as run_command, except the output 
     258            (both stdout and stderr) from the 
     259            command is ignored. 
     260 
     261      ghci_script 
     262            Runs the current compiler, passing 
     263            --interactive and using the specified 
     264            script as standard input. 
     265 
     266    ''<args>'' 
     267    is a list of arguments to be passed to <test-fn>. 
     268 
     269    For compile, compile_fail and compile_and_run, <args> 
     270    is a list with a single string which contains extra 
     271    compiler options with which to run the test.  eg. 
     272{{{                 
     273    test(tc001, normal, compile, ['-fglasgow-exts']) 
     274}}} 
     275    would pass the flag -fglasgow-exts to the compiler 
     276    when compiling tc001. 
     277 
     278    The multimod_ versions of compile and compile_and_run 
     279    expect an extra argument on the front of the list: the 
     280    name of the top module in the program to be compiled 
     281    (usually this will be 'Main'). 
     282 
     283 
     284A multi-module test is straightforward.  It usually goes in a 
     285directory of its own (although this isn't essential), and the source 
     286files can be named anything you like.  The test must have a name, in 
     287the same way as a single-module test; and the stdin/stdout/stderr 
     288files follow the name of the test as before.  In the same directory, 
     289place a file 'test.T' containing a line like 
     290{{{ 
     291   test(multimod001, normal, multimod_compile_and_run, \ 
     292                 [ 'Main', '-fglasgow-exts', '', 0 ]) 
     293}}} 
     294as described above. 
     295 
     296For some examples, take a look in tests/ghc-regress/programs. 
     297 
     298= The details = 
     299 
     300The test suite driver is just a set of Python scripts, as are all of 
     301the .T files in the test suite.  The driver (driver/runtests.py) first 
     302searches for all the .T files it can find, and then proceeds to 
     303execute each one, keeping a track of the number of tests run, and 
     304which ones succeeded and failed. 
     305 
     306The script runtests.py takes several options: 
     307 
     308  --config <file> 
     309   
     310       <file> is just a file containing Python code which is  
     311       executed.   The purpose of this option is so that a file 
     312       containing settings for the configuration options can 
     313       be specified on the command line.  Multiple --config options 
     314       may be given. 
     315 
     316  --rootdir <dir> 
     317 
     318       <dir> is the directory below which to search for .T files 
     319       to run. 
     320 
     321  --output-summary <file> 
     322 
     323       In addition to dumping the test summary to stdout, also 
     324       put it in <file>.  (stdout also gets a lot of other output 
     325       when running a series of tests, so redirecting it isn't   
     326       always the right thing). 
     327 
     328  --only <test> 
     329 
     330       Only run tests named <test> (multiple --only options can 
     331       be given).  Useful for running a single test from a .T file 
     332       containing multiple tests. 
     333 
     334  -e <stmt> 
     335 
     336       executes the Python statement <stmt> before running any tests. 
     337       The main purpose of this option is to allow certain 
     338       configuration options to be tweaked from the command line; for 
     339       example, the build system adds '-e config.accept=1' to the 
     340       command line when 'make accept' is invoked. 
     341 
     342Most of the code for running tests is located in driver/testlib.py. 
     343Take a look. 
     344 
     345There is a single Python class (TestConfig) containing the global 
     346configuration for the test suite.  It contains information such as the 
     347kind of compiler being used, which flags to give it, which platform 
     348we're running on, and so on.  The idea is that each platform and 
     349compiler would have its own file containing assignments for elements 
     350of the configuration, which are sourced by passing the appropriate 
     351--config options to the test driver.  For example, the GHC 
     352configuration is contained in the file config/ghc. 
     353 
     354A .T file can obviously contain arbitrary Python code, but the general 
     355idea is that it contains a sequence of calls to the function test(), 
     356which resides in testlib.py.  As described above, test() takes four 
     357arguments: 
     358 
     359      test(<name>, <opt-fn>, <test-fn>, <args>) 
     360 
     361The function <opt-fn> is allowed to be any Python callable object, 
     362which takes a single argument of type TestOptions.  TestOptions is a 
     363class containing options which affect the way that the current test is 
     364run: whether to skip it, whether to expect failure, extra options to 
     365pass to the compiler, etc. (see testlib.py for the definition of the 
     366TestOptions class).  The idea is that the <opt-fn> function modifies 
     367the TestOptions object that it is passed.  For example, to expect 
     368failure for a test, we might do this in the .T file: 
     369{{{ 
     370   def fn(opts): 
     371      opts.expect = 'fail' 
     372 
     373   test(test001, fn, compile, ['']) 
     374}}} 
     375so when fn is called, it sets the instance variable "expect" in the 
     376instance of TestOptions passed as an argument, to the value 'fail'. 
     377This indicates to the test driver that the current test is expected to 
     378fail. 
     379 
     380Some of these functions, such as the one above, are common, so rather 
     381than forcing every .T file to redefine them, we provide canned 
     382versions.  For example, the provided function expect_fail does the 
     383same as fn in the example above.  See testlib.py for all the canned 
     384functions we provide for <opt-fn>. 
     385 
     386The argument <test-fn> is a function which performs the test.  It 
     387takes three or more arguments: 
     388 
     389      <test-fn>( <name>, <way>, ... ) 
     390 
     391where <name> is the name of the test, <way> is the way in which it is 
     392to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments 
     393are constructed from the list <args> in the original call to test(). 
     394The following <test-fn>s are provided at the moment: 
     395 
     396           compile 
     397           compile_fail 
     398           compile_and_run 
     399           multimod_compile 
     400           multimod_compile_fail 
     401           multimod_compile_and_run 
     402           run_command 
     403           run_command_ignore_output 
     404           ghci_script 
     405 
     406and obviously others can be defined.  The function should return 
     407either 'pass' or 'fail' indicating that the test passed or failed 
     408respectively.