Changes between Version 1 and Version 2 of Building/RunningTests


Ignore:
Timestamp:
Oct 24, 2006 10:45:45 AM (9 years ago)
Author:
malcolm.wallace@…
Comment:

cut-n-paste more from the README

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests

    v1 v2  
     1[[PageOutline]]
    12= GHC Test framework =
    23
     
    4041For more details, see below.
    4142
    42 
    43 
     43= Running the testsuite with a compiler other than GHC =
     44
     45(to be written.  The plan is something like:
     46{{{
     47        cvs checkout fpconfig
     48        cd fptools
     49        cvs checkout testsuite
     50        autoconf
     51        ./configure
     52        cd testsuite
     53        make TEST_HC=nhc98 COMPILER=nhc98
     54}}}
     55)
     56
     57= Running individual tests or subdirectories of the testsuite =
     58
     59Most of the subdirectories in the testsuite have a Makefile.  In these
     60subdirectories you can use 'make' to run the test driver in two
     61ways:
     62{{{
     63        make            -- run all the tests in the current directory
     64        make accept     -- run the tests, accepting the current output
     65}}}
     66The following variables may be set on the make command line:
     67{{{
     68        TESTS                   -- specific tests to run
     69        TEST_HC                 -- compiler to use
     70        EXTRA_HC_OPTS           -- extra flags to send to the Haskell compiler
     71        EXTRA_RUNTEST_OPTS      -- extra flags to give the test driver
     72        CONFIG                  -- use a different configuration file
     73        COMPILER                -- stem of a different configuration file
     74                                -- from the config directory [default: ghc]
     75        WAY                     -- just this way
     76}}}
     77The following ways are defined (for GHC, also see the file config/ghc):
     78{{{
     79        normal                  -- no special options
     80        opt                     -- -O
     81        optasm                  -- -O -fasm
     82        prof                    -- -O -prof -auto-all
     83        profasm                 -- -O -prof -auto-all -fasm
     84        unreg                   -- -unreg
     85        ghci                    -- (run only, not compile) run test under GHCi
     86        extcore                 -- -fext-core
     87        optextcore              -- -O -fext-core
     88        threaded                -- -threaded
     89}}}
     90certain ways are enabled automatically if the GHC build in the local
     91tree supports them.  Ways that are enabled this way are optasm, prof,
     92profasm, unreg, threaded, and ghci.
     93
     94= Updating tests when the output changes =
     95
     96If the output of a test has changed, but the new output is still
     97correct, you can automatically update the sample output to match the
     98new output like so:
     99{{{
     100        make accept TESTS=<test-name>
     101}}}
     102where <test-name> is the name of the test.  In a directory which
     103contains a single test, or if you want to update *all* the tests in
     104the current directory, just omit the 'TESTS=<test-name>' part.
     105
     106= Adding a new test =
     107
     108For a test which can be encapsulated in a single source file, follow
     109these steps:
     110
     111 1. Find the appropriate place for the test.  The GHC regression suite
     112    is generally organised in a "white-box" manner: a regression which
     113    originally illustrated a bug in a particular part of the compiler
     114    is placed in the directory for that part.  For example, typechecker
     115    regression tests go in the typechecker/ directory, parser tests
     116    go in parser/, and so on. 
     117
     118    It's not always possible to find a single best place for a test;
     119    in those cases just pick one which seems reasonable.
     120
     121    Under each main directory may be up to three subdirectories:
     122
     123       should_compile:
     124           tests which need to compile only
     125
     126       should_fail:   
     127           tests which should fail to compile and generate a particular error message
     128
     129       should_run:
     130           tests which should compile, run with some specific input, and generate a particular output.
     131   
     132    We don't always divide the tests up like this, and it's not
     133    essential to do so (the directory names have no meaning as
     134    far as the test driver is concerned).       
     135
     136
     137 2. Having found a suitable place for the test, give the test a name.
     138    Follow the convention for the directory in which you place the
     139    test: for example, in typecheck/should_compile, tests are named
     140    tc001, tc002, and so on.  Suppose you name your test T, then
     141    you'll have the following files:
     142
     143      T.hs
     144        The source file containing the test
     145
     146      T.stdin   (for tests that run, and optional)
     147        A file to feed the test as standard input when it
     148        runs.
     149
     150      T.stdout  (for tests that run, and optional)
     151        For tests that run, this file is compared against
     152        the standard output generated by the program.  If
     153        T.stdout does not exist, then the program must not
     154        generate anything on stdout.
     155
     156      T.stderr  (optional)
     157        For tests that run, this file is compared
     158        against the standard error generated by the program.
     159
     160        For tests that compile only, this file is compared
     161        against the standard error output of the compiler,
     162        which is normalised to eliminate bogus differences
     163        (eg. absolute pathnames are removed, whitespace
     164        differences are ignored, etc.)
     165
     166
     167 1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
     168{{{
     169      test(<name>, <opt-fn>, <test-fn>, <args>)
     170}}}
     171    where
     172
     173    ''<name>'' is the name of the test, in quotes (' or ").
     174
     175    ''<opt-fn>''  is a function (i.e. any callable object in Python)
     176    which allows the options for this test to be changed.
     177    There are several pre-defined functions which can be
     178    used in this field:
     179
     180        '''normal'''                don't change any options from the defaults
     181
     182        '''skip'''                  skip this test
     183
     184        '''omit_ways(ways)'''       skip this test for certain ways
     185
     186        '''only_ways(ways)'''       do this test certain ways only
     187
     188        '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
     189
     190        '''only_compiler_types(compilers)'''       do this test for certain compilers only
     191
     192        '''expect_fail'''           this test is an expected failure
     193
     194        '''expect_fail_for(ways)''' expect failure for certain ways
     195
     196        '''expect_fail_if_platform(plat)'''      expect failure on a certain platform
     197
     198        '''expect_fail_if_compiler_type(compiler)'''   expect failure from a certain compiler
     199
     200        '''set_stdin(file)'''       use a different file for stdin
     201
     202        '''exit_code(n)'''          expect an exit code of 'n' from the prog
     203
     204        '''extra_run_opts(opts)'''  pass some extra opts to the prog
     205
     206        '''no_clean'''              don't clean up after this test
     207
     208      You can compose two of these functions together by
     209      saying compose(f,g).  For example, to expect an exit
     210      code of 3 and omit way 'opt', we could use
     211{{{
     212      compose(omit_ways(['opt']), exit_code(3))
     213}}}
     214      as the <opt-fn> argument.  Calls to compose() can of
     215      course be nested.
     216
     217    ''<test-fn>''
     218    is a function which describes how the test should be
     219    run, and determines the form of <args>.  The possible
     220    values are:
     221
     222      compile
     223            Just compile the program, the compilation should succeed.
     224
     225      compile_fail
     226            Just compile the program, the
     227            compilation should fail (error
     228            messages will be in T.stderr).
     229                               
     230      compile_and_run
     231            Compile the program and run it,
     232            comparing the output against the
     233            relevant files.
     234
     235      multimod_compile
     236            Compile a multi-module program
     237            (more about multi-module programs
     238            below).
     239
     240      multimod_compile_fail
     241            Compile a multi-module program,
     242            and expect the compilation to fail
     243            with error messages in T.stderr
     244
     245      multimod_compile_and_run
     246            Compile and run a multi-module
     247            program.
     248
     249      run_command
     250            Just run an arbitrary command.  The
     251            output is checked against T.stdout and
     252            T.stderr, and the stdin and expected
     253            exit code can be changed in the same
     254            way as for compile_and_run.
     255
     256      run_command_ignore_output
     257            Same as run_command, except the output
     258            (both stdout and stderr) from the
     259            command is ignored.
     260
     261      ghci_script
     262            Runs the current compiler, passing
     263            --interactive and using the specified
     264            script as standard input.
     265
     266    ''<args>''
     267    is a list of arguments to be passed to <test-fn>.
     268
     269    For compile, compile_fail and compile_and_run, <args>
     270    is a list with a single string which contains extra
     271    compiler options with which to run the test.  eg.
     272{{{               
     273    test(tc001, normal, compile, ['-fglasgow-exts'])
     274}}}
     275    would pass the flag -fglasgow-exts to the compiler
     276    when compiling tc001.
     277
     278    The multimod_ versions of compile and compile_and_run
     279    expect an extra argument on the front of the list: the
     280    name of the top module in the program to be compiled
     281    (usually this will be 'Main').
     282
     283
     284A multi-module test is straightforward.  It usually goes in a
     285directory of its own (although this isn't essential), and the source
     286files can be named anything you like.  The test must have a name, in
     287the same way as a single-module test; and the stdin/stdout/stderr
     288files follow the name of the test as before.  In the same directory,
     289place a file 'test.T' containing a line like
     290{{{
     291   test(multimod001, normal, multimod_compile_and_run, \
     292                 [ 'Main', '-fglasgow-exts', '', 0 ])
     293}}}
     294as described above.
     295
     296For some examples, take a look in tests/ghc-regress/programs.
     297
     298= The details =
     299
     300The test suite driver is just a set of Python scripts, as are all of
     301the .T files in the test suite.  The driver (driver/runtests.py) first
     302searches for all the .T files it can find, and then proceeds to
     303execute each one, keeping a track of the number of tests run, and
     304which ones succeeded and failed.
     305
     306The script runtests.py takes several options:
     307
     308  --config <file>
     309 
     310       <file> is just a file containing Python code which is
     311       executed.   The purpose of this option is so that a file
     312       containing settings for the configuration options can
     313       be specified on the command line.  Multiple --config options
     314       may be given.
     315
     316  --rootdir <dir>
     317
     318       <dir> is the directory below which to search for .T files
     319       to run.
     320
     321  --output-summary <file>
     322
     323       In addition to dumping the test summary to stdout, also
     324       put it in <file>.  (stdout also gets a lot of other output
     325       when running a series of tests, so redirecting it isn't 
     326       always the right thing).
     327
     328  --only <test>
     329
     330       Only run tests named <test> (multiple --only options can
     331       be given).  Useful for running a single test from a .T file
     332       containing multiple tests.
     333
     334  -e <stmt>
     335
     336       executes the Python statement <stmt> before running any tests.
     337       The main purpose of this option is to allow certain
     338       configuration options to be tweaked from the command line; for
     339       example, the build system adds '-e config.accept=1' to the
     340       command line when 'make accept' is invoked.
     341
     342Most of the code for running tests is located in driver/testlib.py.
     343Take a look.
     344
     345There is a single Python class (TestConfig) containing the global
     346configuration for the test suite.  It contains information such as the
     347kind of compiler being used, which flags to give it, which platform
     348we're running on, and so on.  The idea is that each platform and
     349compiler would have its own file containing assignments for elements
     350of the configuration, which are sourced by passing the appropriate
     351--config options to the test driver.  For example, the GHC
     352configuration is contained in the file config/ghc.
     353
     354A .T file can obviously contain arbitrary Python code, but the general
     355idea is that it contains a sequence of calls to the function test(),
     356which resides in testlib.py.  As described above, test() takes four
     357arguments:
     358
     359      test(<name>, <opt-fn>, <test-fn>, <args>)
     360
     361The function <opt-fn> is allowed to be any Python callable object,
     362which takes a single argument of type TestOptions.  TestOptions is a
     363class containing options which affect the way that the current test is
     364run: whether to skip it, whether to expect failure, extra options to
     365pass to the compiler, etc. (see testlib.py for the definition of the
     366TestOptions class).  The idea is that the <opt-fn> function modifies
     367the TestOptions object that it is passed.  For example, to expect
     368failure for a test, we might do this in the .T file:
     369{{{
     370   def fn(opts):
     371      opts.expect = 'fail'
     372
     373   test(test001, fn, compile, [''])
     374}}}
     375so when fn is called, it sets the instance variable "expect" in the
     376instance of TestOptions passed as an argument, to the value 'fail'.
     377This indicates to the test driver that the current test is expected to
     378fail.
     379
     380Some of these functions, such as the one above, are common, so rather
     381than forcing every .T file to redefine them, we provide canned
     382versions.  For example, the provided function expect_fail does the
     383same as fn in the example above.  See testlib.py for all the canned
     384functions we provide for <opt-fn>.
     385
     386The argument <test-fn> is a function which performs the test.  It
     387takes three or more arguments:
     388
     389      <test-fn>( <name>, <way>, ... )
     390
     391where <name> is the name of the test, <way> is the way in which it is
     392to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments
     393are constructed from the list <args> in the original call to test().
     394The following <test-fn>s are provided at the moment:
     395
     396           compile
     397           compile_fail
     398           compile_and_run
     399           multimod_compile
     400           multimod_compile_fail
     401           multimod_compile_and_run
     402           run_command
     403           run_command_ignore_output
     404           ghci_script
     405
     406and obviously others can be defined.  The function should return
     407either 'pass' or 'fail' indicating that the test passed or failed
     408respectively.