Changes between Version 1 and Version 2 of Building/RunningTests

Oct 24, 2006 10:45:45 AM (9 years ago)

cut-n-paste more from the README


  • Building/RunningTests

    v1 v2  
    12= GHC Test framework =
    4041For more details, see below.
     43= Running the testsuite with a compiler other than GHC =
     45(to be written.  The plan is something like:
     47        cvs checkout fpconfig
     48        cd fptools
     49        cvs checkout testsuite
     50        autoconf
     51        ./configure
     52        cd testsuite
     53        make TEST_HC=nhc98 COMPILER=nhc98
     57= Running individual tests or subdirectories of the testsuite =
     59Most of the subdirectories in the testsuite have a Makefile.  In these
     60subdirectories you can use 'make' to run the test driver in two
     63        make            -- run all the tests in the current directory
     64        make accept     -- run the tests, accepting the current output
     66The following variables may be set on the make command line:
     68        TESTS                   -- specific tests to run
     69        TEST_HC                 -- compiler to use
     70        EXTRA_HC_OPTS           -- extra flags to send to the Haskell compiler
     71        EXTRA_RUNTEST_OPTS      -- extra flags to give the test driver
     72        CONFIG                  -- use a different configuration file
     73        COMPILER                -- stem of a different configuration file
     74                                -- from the config directory [default: ghc]
     75        WAY                     -- just this way
     77The following ways are defined (for GHC, also see the file config/ghc):
     79        normal                  -- no special options
     80        opt                     -- -O
     81        optasm                  -- -O -fasm
     82        prof                    -- -O -prof -auto-all
     83        profasm                 -- -O -prof -auto-all -fasm
     84        unreg                   -- -unreg
     85        ghci                    -- (run only, not compile) run test under GHCi
     86        extcore                 -- -fext-core
     87        optextcore              -- -O -fext-core
     88        threaded                -- -threaded
     90certain ways are enabled automatically if the GHC build in the local
     91tree supports them.  Ways that are enabled this way are optasm, prof,
     92profasm, unreg, threaded, and ghci.
     94= Updating tests when the output changes =
     96If the output of a test has changed, but the new output is still
     97correct, you can automatically update the sample output to match the
     98new output like so:
     100        make accept TESTS=<test-name>
     102where <test-name> is the name of the test.  In a directory which
     103contains a single test, or if you want to update *all* the tests in
     104the current directory, just omit the 'TESTS=<test-name>' part.
     106= Adding a new test =
     108For a test which can be encapsulated in a single source file, follow
     109these steps:
     111 1. Find the appropriate place for the test.  The GHC regression suite
     112    is generally organised in a "white-box" manner: a regression which
     113    originally illustrated a bug in a particular part of the compiler
     114    is placed in the directory for that part.  For example, typechecker
     115    regression tests go in the typechecker/ directory, parser tests
     116    go in parser/, and so on. 
     118    It's not always possible to find a single best place for a test;
     119    in those cases just pick one which seems reasonable.
     121    Under each main directory may be up to three subdirectories:
     123       should_compile:
     124           tests which need to compile only
     126       should_fail:   
     127           tests which should fail to compile and generate a particular error message
     129       should_run:
     130           tests which should compile, run with some specific input, and generate a particular output.
     132    We don't always divide the tests up like this, and it's not
     133    essential to do so (the directory names have no meaning as
     134    far as the test driver is concerned).       
     137 2. Having found a suitable place for the test, give the test a name.
     138    Follow the convention for the directory in which you place the
     139    test: for example, in typecheck/should_compile, tests are named
     140    tc001, tc002, and so on.  Suppose you name your test T, then
     141    you'll have the following files:
     143      T.hs
     144        The source file containing the test
     146      T.stdin   (for tests that run, and optional)
     147        A file to feed the test as standard input when it
     148        runs.
     150      T.stdout  (for tests that run, and optional)
     151        For tests that run, this file is compared against
     152        the standard output generated by the program.  If
     153        T.stdout does not exist, then the program must not
     154        generate anything on stdout.
     156      T.stderr  (optional)
     157        For tests that run, this file is compared
     158        against the standard error generated by the program.
     160        For tests that compile only, this file is compared
     161        against the standard error output of the compiler,
     162        which is normalised to eliminate bogus differences
     163        (eg. absolute pathnames are removed, whitespace
     164        differences are ignored, etc.)
     167 1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
     169      test(<name>, <opt-fn>, <test-fn>, <args>)
     171    where
     173    ''<name>'' is the name of the test, in quotes (' or ").
     175    ''<opt-fn>''  is a function (i.e. any callable object in Python)
     176    which allows the options for this test to be changed.
     177    There are several pre-defined functions which can be
     178    used in this field:
     180        '''normal'''                don't change any options from the defaults
     182        '''skip'''                  skip this test
     184        '''omit_ways(ways)'''       skip this test for certain ways
     186        '''only_ways(ways)'''       do this test certain ways only
     188        '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
     190        '''only_compiler_types(compilers)'''       do this test for certain compilers only
     192        '''expect_fail'''           this test is an expected failure
     194        '''expect_fail_for(ways)''' expect failure for certain ways
     196        '''expect_fail_if_platform(plat)'''      expect failure on a certain platform
     198        '''expect_fail_if_compiler_type(compiler)'''   expect failure from a certain compiler
     200        '''set_stdin(file)'''       use a different file for stdin
     202        '''exit_code(n)'''          expect an exit code of 'n' from the prog
     204        '''extra_run_opts(opts)'''  pass some extra opts to the prog
     206        '''no_clean'''              don't clean up after this test
     208      You can compose two of these functions together by
     209      saying compose(f,g).  For example, to expect an exit
     210      code of 3 and omit way 'opt', we could use
     212      compose(omit_ways(['opt']), exit_code(3))
     214      as the <opt-fn> argument.  Calls to compose() can of
     215      course be nested.
     217    ''<test-fn>''
     218    is a function which describes how the test should be
     219    run, and determines the form of <args>.  The possible
     220    values are:
     222      compile
     223            Just compile the program, the compilation should succeed.
     225      compile_fail
     226            Just compile the program, the
     227            compilation should fail (error
     228            messages will be in T.stderr).
     230      compile_and_run
     231            Compile the program and run it,
     232            comparing the output against the
     233            relevant files.
     235      multimod_compile
     236            Compile a multi-module program
     237            (more about multi-module programs
     238            below).
     240      multimod_compile_fail
     241            Compile a multi-module program,
     242            and expect the compilation to fail
     243            with error messages in T.stderr
     245      multimod_compile_and_run
     246            Compile and run a multi-module
     247            program.
     249      run_command
     250            Just run an arbitrary command.  The
     251            output is checked against T.stdout and
     252            T.stderr, and the stdin and expected
     253            exit code can be changed in the same
     254            way as for compile_and_run.
     256      run_command_ignore_output
     257            Same as run_command, except the output
     258            (both stdout and stderr) from the
     259            command is ignored.
     261      ghci_script
     262            Runs the current compiler, passing
     263            --interactive and using the specified
     264            script as standard input.
     266    ''<args>''
     267    is a list of arguments to be passed to <test-fn>.
     269    For compile, compile_fail and compile_and_run, <args>
     270    is a list with a single string which contains extra
     271    compiler options with which to run the test.  eg.
     273    test(tc001, normal, compile, ['-fglasgow-exts'])
     275    would pass the flag -fglasgow-exts to the compiler
     276    when compiling tc001.
     278    The multimod_ versions of compile and compile_and_run
     279    expect an extra argument on the front of the list: the
     280    name of the top module in the program to be compiled
     281    (usually this will be 'Main').
     284A multi-module test is straightforward.  It usually goes in a
     285directory of its own (although this isn't essential), and the source
     286files can be named anything you like.  The test must have a name, in
     287the same way as a single-module test; and the stdin/stdout/stderr
     288files follow the name of the test as before.  In the same directory,
     289place a file 'test.T' containing a line like
     291   test(multimod001, normal, multimod_compile_and_run, \
     292                 [ 'Main', '-fglasgow-exts', '', 0 ])
     294as described above.
     296For some examples, take a look in tests/ghc-regress/programs.
     298= The details =
     300The test suite driver is just a set of Python scripts, as are all of
     301the .T files in the test suite.  The driver (driver/ first
     302searches for all the .T files it can find, and then proceeds to
     303execute each one, keeping a track of the number of tests run, and
     304which ones succeeded and failed.
     306The script takes several options:
     308  --config <file>
     310       <file> is just a file containing Python code which is
     311       executed.   The purpose of this option is so that a file
     312       containing settings for the configuration options can
     313       be specified on the command line.  Multiple --config options
     314       may be given.
     316  --rootdir <dir>
     318       <dir> is the directory below which to search for .T files
     319       to run.
     321  --output-summary <file>
     323       In addition to dumping the test summary to stdout, also
     324       put it in <file>.  (stdout also gets a lot of other output
     325       when running a series of tests, so redirecting it isn't 
     326       always the right thing).
     328  --only <test>
     330       Only run tests named <test> (multiple --only options can
     331       be given).  Useful for running a single test from a .T file
     332       containing multiple tests.
     334  -e <stmt>
     336       executes the Python statement <stmt> before running any tests.
     337       The main purpose of this option is to allow certain
     338       configuration options to be tweaked from the command line; for
     339       example, the build system adds '-e config.accept=1' to the
     340       command line when 'make accept' is invoked.
     342Most of the code for running tests is located in driver/
     343Take a look.
     345There is a single Python class (TestConfig) containing the global
     346configuration for the test suite.  It contains information such as the
     347kind of compiler being used, which flags to give it, which platform
     348we're running on, and so on.  The idea is that each platform and
     349compiler would have its own file containing assignments for elements
     350of the configuration, which are sourced by passing the appropriate
     351--config options to the test driver.  For example, the GHC
     352configuration is contained in the file config/ghc.
     354A .T file can obviously contain arbitrary Python code, but the general
     355idea is that it contains a sequence of calls to the function test(),
     356which resides in  As described above, test() takes four
     359      test(<name>, <opt-fn>, <test-fn>, <args>)
     361The function <opt-fn> is allowed to be any Python callable object,
     362which takes a single argument of type TestOptions.  TestOptions is a
     363class containing options which affect the way that the current test is
     364run: whether to skip it, whether to expect failure, extra options to
     365pass to the compiler, etc. (see for the definition of the
     366TestOptions class).  The idea is that the <opt-fn> function modifies
     367the TestOptions object that it is passed.  For example, to expect
     368failure for a test, we might do this in the .T file:
     370   def fn(opts):
     371      opts.expect = 'fail'
     373   test(test001, fn, compile, [''])
     375so when fn is called, it sets the instance variable "expect" in the
     376instance of TestOptions passed as an argument, to the value 'fail'.
     377This indicates to the test driver that the current test is expected to
     380Some of these functions, such as the one above, are common, so rather
     381than forcing every .T file to redefine them, we provide canned
     382versions.  For example, the provided function expect_fail does the
     383same as fn in the example above.  See for all the canned
     384functions we provide for <opt-fn>.
     386The argument <test-fn> is a function which performs the test.  It
     387takes three or more arguments:
     389      <test-fn>( <name>, <way>, ... )
     391where <name> is the name of the test, <way> is the way in which it is
     392to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments
     393are constructed from the list <args> in the original call to test().
     394The following <test-fn>s are provided at the moment:
     396           compile
     397           compile_fail
     398           compile_and_run
     399           multimod_compile
     400           multimod_compile_fail
     401           multimod_compile_and_run
     402           run_command
     403           run_command_ignore_output
     404           ghci_script
     406and obviously others can be defined.  The function should return
     407either 'pass' or 'fail' indicating that the test passed or failed