Changes between Version 1 and Version 2 of Building/RunningTests/Adding

Jul 8, 2011 10:49:38 PM (6 years ago)



  • Building/RunningTests/Adding

    v1 v2  
    2929 Under each main directory there are usually up to three subdirectories:
    30        '''should_compile''':   
    31            test cases which need to compile only
    32        '''should_fail''':   
    33            test cases which should fail to compile and generate a particular error message
    34        '''should_run''':
    35            test cases which should compile, run with some specific input, and generate a particular output.
     30       '''should_compile''': test cases which need to compile only [[BR]]
     31       '''should_fail''': test cases which should fail to compile and generate a particular error message [[BR]]
     32       '''should_run''': test cases which should compile, run with some specific input, and generate a particular output.
    3734 We don't always divide the test cases up like this, and it's not
    361358compiler version.  For example, the sample `stderr` for GHC 6.8.x would go in the file
    364 == The details ==
    366 The testsuite driver is just a set of Python scripts, as are all of
    367 the .T files in the testsuite.  The driver (driver/ first
    368 searches for all the .T files it can find, and then proceeds to
    369 execute each one, keeping a track of the number of tests run, and
    370 which ones succeeded and failed.
    372 The script takes several options:
    374   --config <file>
    376        <file> is just a file containing Python code which is
    377        executed.   The purpose of this option is so that a file
    378        containing settings for the configuration options can
    379        be specified on the command line.  Multiple --config options
    380        may be given.
    382   --rootdir <dir>
    384        <dir> is the directory below which to search for .T files
    385        to run.
    387   --output-summary <file>
    389        In addition to dumping the test summary to stdout, also
    390        put it in <file>.  (stdout also gets a lot of other output
    391        when running a series of tests, so redirecting it isn't 
    392        always the right thing).
    394   --only <test>
    396        Only run tests named <test> (multiple --only options can
    397        be given).  Useful for running a single test from a .T file
    398        containing multiple tests.
    400   -e <stmt>
    402        executes the Python statement <stmt> before running any tests.
    403        The main purpose of this option is to allow certain
    404        configuration options to be tweaked from the command line; for
    405        example, the build system adds '-e config.accept=1' to the
    406        command line when 'make accept' is invoked.
    408 Most of the code for running tests is located in driver/
    409 Take a look.
    411 There is a single Python class (TestConfig) containing the global
    412 configuration for the testsuite.  It contains information such as the
    413 kind of compiler being used, which flags to give it, which platform
    414 we're running on, and so on.  The idea is that each platform and
    415 compiler would have its own file containing assignments for elements
    416 of the configuration, which are sourced by passing the appropriate
    417 --config options to the test driver.  For example, the GHC
    418 configuration is contained in the file config/ghc.
    420 A .T file can obviously contain arbitrary Python code, but the general
    421 idea is that it contains a sequence of calls to the function test(),
    422 which resides in  As described above, test() takes four
    423 arguments:
    425       test(<name>, <opt-fn>, <test-fn>, <args>)
    427 The function <opt-fn> is allowed to be any Python callable object,
    428 which takes a single argument of type TestOptions.  TestOptions is a
    429 class containing options which affect the way that the current test is
    430 run: whether to skip it, whether to expect failure, extra options to
    431 pass to the compiler, etc. (see for the definition of the
    432 TestOptions class).  The idea is that the <opt-fn> function modifies
    433 the TestOptions object that it is passed.  For example, to expect
    434 failure for a test, we might do this in the .T file:
    435 {{{
    436    def fn(opts):
    437       opts.expect = 'fail'
    439    test(test001, fn, compile, [''])
    440 }}}
    441 so when fn is called, it sets the instance variable "expect" in the
    442 instance of TestOptions passed as an argument, to the value 'fail'.
    443 This indicates to the test driver that the current test is expected to
    444 fail.
    446 Some of these functions, such as the one above, are common, so rather
    447 than forcing every .T file to redefine them, we provide canned
    448 versions.  For example, the provided function expect_fail does the
    449 same as fn in the example above.  See for all the canned
    450 functions we provide for <opt-fn>.
    452 The argument <test-fn> is a function which performs the test.  It
    453 takes three or more arguments:
    455       <test-fn>( <name>, <way>, ... )
    457 where <name> is the name of the test, <way> is the way in which it is
    458 to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments
    459 are constructed from the list <args> in the original call to test().
    460 The following <test-fn>s are provided at the moment:
    462            compile
    463            compile_fail
    464            compile_and_run
    465            multimod_compile
    466            multimod_compile_fail
    467            multimod_compile_and_run
    468            run_command
    469            run_command_ignore_output
    470            ghci_script
    472 and obviously others can be defined.  The function should return
    473 either 'pass' or 'fail' indicating that the test passed or failed
    474 respectively.