Changes between Version 1 and Version 2 of Building/RunningTests/Adding


Ignore:
Timestamp:
Jul 8, 2011 10:49:38 PM (4 years ago)
Author:
dterei
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests/Adding

    v1 v2  
    2828
    2929 Under each main directory there are usually up to three subdirectories:
    30        '''should_compile''':   
    31            test cases which need to compile only
    32        '''should_fail''':   
    33            test cases which should fail to compile and generate a particular error message
    34        '''should_run''':
    35            test cases which should compile, run with some specific input, and generate a particular output.
     30       '''should_compile''': test cases which need to compile only [[BR]]
     31       '''should_fail''': test cases which should fail to compile and generate a particular error message [[BR]]
     32       '''should_run''': test cases which should compile, run with some specific input, and generate a particular output.
    3633   
    3734 We don't always divide the test cases up like this, and it's not
     
    361358compiler version.  For example, the sample `stderr` for GHC 6.8.x would go in the file
    362359`T.stderr-ghc-6.8`.
    363 
    364 == The details ==
    365 
    366 The testsuite driver is just a set of Python scripts, as are all of
    367 the .T files in the testsuite.  The driver (driver/runtests.py) first
    368 searches for all the .T files it can find, and then proceeds to
    369 execute each one, keeping a track of the number of tests run, and
    370 which ones succeeded and failed.
    371 
    372 The script runtests.py takes several options:
    373 
    374   --config <file>
    375  
    376        <file> is just a file containing Python code which is
    377        executed.   The purpose of this option is so that a file
    378        containing settings for the configuration options can
    379        be specified on the command line.  Multiple --config options
    380        may be given.
    381 
    382   --rootdir <dir>
    383 
    384        <dir> is the directory below which to search for .T files
    385        to run.
    386 
    387   --output-summary <file>
    388 
    389        In addition to dumping the test summary to stdout, also
    390        put it in <file>.  (stdout also gets a lot of other output
    391        when running a series of tests, so redirecting it isn't 
    392        always the right thing).
    393 
    394   --only <test>
    395 
    396        Only run tests named <test> (multiple --only options can
    397        be given).  Useful for running a single test from a .T file
    398        containing multiple tests.
    399 
    400   -e <stmt>
    401 
    402        executes the Python statement <stmt> before running any tests.
    403        The main purpose of this option is to allow certain
    404        configuration options to be tweaked from the command line; for
    405        example, the build system adds '-e config.accept=1' to the
    406        command line when 'make accept' is invoked.
    407 
    408 Most of the code for running tests is located in driver/testlib.py.
    409 Take a look.
    410 
    411 There is a single Python class (TestConfig) containing the global
    412 configuration for the testsuite.  It contains information such as the
    413 kind of compiler being used, which flags to give it, which platform
    414 we're running on, and so on.  The idea is that each platform and
    415 compiler would have its own file containing assignments for elements
    416 of the configuration, which are sourced by passing the appropriate
    417 --config options to the test driver.  For example, the GHC
    418 configuration is contained in the file config/ghc.
    419 
    420 A .T file can obviously contain arbitrary Python code, but the general
    421 idea is that it contains a sequence of calls to the function test(),
    422 which resides in testlib.py.  As described above, test() takes four
    423 arguments:
    424 
    425       test(<name>, <opt-fn>, <test-fn>, <args>)
    426 
    427 The function <opt-fn> is allowed to be any Python callable object,
    428 which takes a single argument of type TestOptions.  TestOptions is a
    429 class containing options which affect the way that the current test is
    430 run: whether to skip it, whether to expect failure, extra options to
    431 pass to the compiler, etc. (see testlib.py for the definition of the
    432 TestOptions class).  The idea is that the <opt-fn> function modifies
    433 the TestOptions object that it is passed.  For example, to expect
    434 failure for a test, we might do this in the .T file:
    435 {{{
    436    def fn(opts):
    437       opts.expect = 'fail'
    438 
    439    test(test001, fn, compile, [''])
    440 }}}
    441 so when fn is called, it sets the instance variable "expect" in the
    442 instance of TestOptions passed as an argument, to the value 'fail'.
    443 This indicates to the test driver that the current test is expected to
    444 fail.
    445 
    446 Some of these functions, such as the one above, are common, so rather
    447 than forcing every .T file to redefine them, we provide canned
    448 versions.  For example, the provided function expect_fail does the
    449 same as fn in the example above.  See testlib.py for all the canned
    450 functions we provide for <opt-fn>.
    451 
    452 The argument <test-fn> is a function which performs the test.  It
    453 takes three or more arguments:
    454 
    455       <test-fn>( <name>, <way>, ... )
    456 
    457 where <name> is the name of the test, <way> is the way in which it is
    458 to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments
    459 are constructed from the list <args> in the original call to test().
    460 The following <test-fn>s are provided at the moment:
    461 
    462            compile
    463            compile_fail
    464            compile_and_run
    465            multimod_compile
    466            multimod_compile_fail
    467            multimod_compile_and_run
    468            run_command
    469            run_command_ignore_output
    470            ghci_script
    471 
    472 and obviously others can be defined.  The function should return
    473 either 'pass' or 'fail' indicating that the test passed or failed
    474 respectively.