Changes between Version 1 and Version 2 of Building/RunningTests/Adding


Ignore:
Timestamp:
Jul 8, 2011 10:49:38 PM (3 years ago)
Author:
dterei
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests/Adding

    v1 v2  
    2828 
    2929 Under each main directory there are usually up to three subdirectories: 
    30        '''should_compile''':     
    31            test cases which need to compile only 
    32        '''should_fail''':     
    33            test cases which should fail to compile and generate a particular error message 
    34        '''should_run''': 
    35            test cases which should compile, run with some specific input, and generate a particular output. 
     30       '''should_compile''': test cases which need to compile only [[BR]] 
     31       '''should_fail''': test cases which should fail to compile and generate a particular error message [[BR]] 
     32       '''should_run''': test cases which should compile, run with some specific input, and generate a particular output. 
    3633     
    3734 We don't always divide the test cases up like this, and it's not 
     
    361358compiler version.  For example, the sample `stderr` for GHC 6.8.x would go in the file 
    362359`T.stderr-ghc-6.8`. 
    363  
    364 == The details == 
    365  
    366 The testsuite driver is just a set of Python scripts, as are all of 
    367 the .T files in the testsuite.  The driver (driver/runtests.py) first 
    368 searches for all the .T files it can find, and then proceeds to 
    369 execute each one, keeping a track of the number of tests run, and 
    370 which ones succeeded and failed. 
    371  
    372 The script runtests.py takes several options: 
    373  
    374   --config <file> 
    375    
    376        <file> is just a file containing Python code which is  
    377        executed.   The purpose of this option is so that a file 
    378        containing settings for the configuration options can 
    379        be specified on the command line.  Multiple --config options 
    380        may be given. 
    381  
    382   --rootdir <dir> 
    383  
    384        <dir> is the directory below which to search for .T files 
    385        to run. 
    386  
    387   --output-summary <file> 
    388  
    389        In addition to dumping the test summary to stdout, also 
    390        put it in <file>.  (stdout also gets a lot of other output 
    391        when running a series of tests, so redirecting it isn't   
    392        always the right thing). 
    393  
    394   --only <test> 
    395  
    396        Only run tests named <test> (multiple --only options can 
    397        be given).  Useful for running a single test from a .T file 
    398        containing multiple tests. 
    399  
    400   -e <stmt> 
    401  
    402        executes the Python statement <stmt> before running any tests. 
    403        The main purpose of this option is to allow certain 
    404        configuration options to be tweaked from the command line; for 
    405        example, the build system adds '-e config.accept=1' to the 
    406        command line when 'make accept' is invoked. 
    407  
    408 Most of the code for running tests is located in driver/testlib.py. 
    409 Take a look. 
    410  
    411 There is a single Python class (TestConfig) containing the global 
    412 configuration for the testsuite.  It contains information such as the 
    413 kind of compiler being used, which flags to give it, which platform 
    414 we're running on, and so on.  The idea is that each platform and 
    415 compiler would have its own file containing assignments for elements 
    416 of the configuration, which are sourced by passing the appropriate 
    417 --config options to the test driver.  For example, the GHC 
    418 configuration is contained in the file config/ghc. 
    419  
    420 A .T file can obviously contain arbitrary Python code, but the general 
    421 idea is that it contains a sequence of calls to the function test(), 
    422 which resides in testlib.py.  As described above, test() takes four 
    423 arguments: 
    424  
    425       test(<name>, <opt-fn>, <test-fn>, <args>) 
    426  
    427 The function <opt-fn> is allowed to be any Python callable object, 
    428 which takes a single argument of type TestOptions.  TestOptions is a 
    429 class containing options which affect the way that the current test is 
    430 run: whether to skip it, whether to expect failure, extra options to 
    431 pass to the compiler, etc. (see testlib.py for the definition of the 
    432 TestOptions class).  The idea is that the <opt-fn> function modifies 
    433 the TestOptions object that it is passed.  For example, to expect 
    434 failure for a test, we might do this in the .T file: 
    435 {{{ 
    436    def fn(opts): 
    437       opts.expect = 'fail' 
    438  
    439    test(test001, fn, compile, ['']) 
    440 }}} 
    441 so when fn is called, it sets the instance variable "expect" in the 
    442 instance of TestOptions passed as an argument, to the value 'fail'. 
    443 This indicates to the test driver that the current test is expected to 
    444 fail. 
    445  
    446 Some of these functions, such as the one above, are common, so rather 
    447 than forcing every .T file to redefine them, we provide canned 
    448 versions.  For example, the provided function expect_fail does the 
    449 same as fn in the example above.  See testlib.py for all the canned 
    450 functions we provide for <opt-fn>. 
    451  
    452 The argument <test-fn> is a function which performs the test.  It 
    453 takes three or more arguments: 
    454  
    455       <test-fn>( <name>, <way>, ... ) 
    456  
    457 where <name> is the name of the test, <way> is the way in which it is 
    458 to be run (eg. opt, optasm, prof, etc.), and the rest of the arguments 
    459 are constructed from the list <args> in the original call to test(). 
    460 The following <test-fn>s are provided at the moment: 
    461  
    462            compile 
    463            compile_fail 
    464            compile_and_run 
    465            multimod_compile 
    466            multimod_compile_fail 
    467            multimod_compile_and_run 
    468            run_command 
    469            run_command_ignore_output 
    470            ghci_script 
    471  
    472 and obviously others can be defined.  The function should return 
    473 either 'pass' or 'fail' indicating that the test passed or failed 
    474 respectively.