Changes between Version 22 and Version 23 of Building/RunningTests


Ignore:
Timestamp:
Aug 21, 2009 9:46:55 AM (5 years ago)
Author:
simonpj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests

    v22 v23  
    132132    go in parser/, and so on.   
    133133 
    134     It's not always possible to find a single best place for a test; 
    135     in those cases just pick one which seems reasonable. 
    136  
    137     Under each main directory may be up to three subdirectories: 
    138  
    139        should_compile: 
     134 It's not always possible to find a single best place for a test; 
     135 in those cases just pick one which seems reasonable. 
     136 
     137 Under each main directory may be up to three subdirectories: 
     138       '''should_compile''':     
    140139           tests which need to compile only 
    141  
    142        should_fail:     
     140       '''should_fail''':     
    143141           tests which should fail to compile and generate a particular error message 
    144  
    145        should_run: 
     142       '''should_run''': 
    146143           tests which should compile, run with some specific input, and generate a particular output. 
    147144     
    148     We don't always divide the tests up like this, and it's not 
    149     essential to do so (the directory names have no meaning as 
    150     far as the test driver is concerned).         
     145 We don't always divide the tests up like this, and it's not 
     146 essential to do so (the directory names have no meaning as 
     147 far as the test driver is concerned).         
    151148 
    152149 
    153150 2. Having found a suitable place for the test, give the test a name. 
    154     For regression tests, we often just name the test after the bug number (e.g. 2047). 
     151    For regression tests, we often just name the test after the bug number (e.g. T2047). 
    155152    Alternatively, follow the convention for the directory in which you place the 
    156153    test: for example, in typecheck/should_compile, tests are named 
     
    182179 
    183180 
    184  1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form 
     181 3. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form 
    185182{{{ 
    186183      test(<name>, <setup>, <test-fn>, <args>) 
    187184}}} 
    188     where 
    189  
    190     ''<name>'' is the name of the test, in quotes (' or "). 
    191  
    192     ''<setup>''  is a function (i.e. any callable object in Python) 
    193     which allows the options for this test to be changed. 
    194     There are many pre-defined functions which can be 
    195     used in this field: 
    196  
    197         '''normal'''                don't change any options from the defaults 
    198  
    199         '''skip'''                  skip this test 
    200  
    201         '''skip_if_no_ghci'''       skip unless GHCi is available 
    202  
    203         '''skip_if_fast'''          skip if "fast" is enabled 
    204  
    205         '''omit_ways(ways)'''       skip this test for certain ways 
    206  
    207         '''only_ways(ways)'''       do this test certain ways only 
    208  
    209         '''extra_ways(ways)'''      add some ways which would normally be disabled 
    210  
    211         '''omit_compiler_types(compilers)'''                           skip this test for certain compilers 
    212  
    213         '''only_compiler_types(compilers)'''       do this test for certain compilers only 
    214  
    215         '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number 
    216  
    217         '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways 
    218  
    219         '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type 
    220  
    221         '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given 
    222  
    223         '''if_tag(tag, f)'''        do `f` if the compiler has a given tag 
    224  
    225         '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag 
    226  
    227         '''set_stdin(file)'''       use a different file for stdin 
    228  
    229         '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`) 
    230  
    231         '''exit_code(n)'''          expect an exit code of 'n' from the prog 
    232  
    233         '''extra_run_opts(opts)'''  pass some extra opts to the prog 
    234  
    235         '''no_clean'''              don't clean up after this test 
    236  
    237         '''extra_clean(files)'''    extra files to clean after the test has completed 
    238  
    239         '''reqlib(P)'''             requires package P 
    240  
    241         '''req_profiling'''         requires profiling 
    242  
    243         '''ignore_output'''         don't try to compare output 
    244  
    245         '''alone'''                 don't run this test in parallel with anything else 
    246  
    247         '''literate'''              look for a `.lhs` file instead of a `.hs` file 
    248  
    249         '''c_src'''                 look for a `.c` file 
    250  
    251         '''cmd_prefix(string)'''    prefix this string to the command when run 
    252  
    253         '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output 
    254  
    255     To use more than one modifier on a test, just put them in a list. 
    256     For example, to expect an exit code of 3 and omit way 'opt', we could use 
    257 {{{ 
    258       [ omit_ways(['opt']), exit_code(3) ] 
    259 }}} 
    260       as the `<setup>` argument. 
    261  
    262     The following should normally not be used; instead, use the `expect_broken*` 
    263     functions above so that the problem doesn't get forgotten about, and when we 
    264     come back to look at the test later we know whether current behaviour is why 
    265     we marked it as expected to fail: 
    266  
    267         '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it. 
    268  
    269         '''expect_fail_for(ways)''' expect failure for certain ways  
    270  
    271     ''<test-fn>'' 
    272     is a function which describes how the test should be 
    273     run, and determines the form of <args>.  The possible 
    274     values are: 
    275  
    276       compile 
    277             Just compile the program, the compilation should succeed. 
    278  
    279       compile_fail 
    280             Just compile the program, the 
    281             compilation should fail (error 
    282             messages will be in T.stderr). 
    283             This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler. 
    284  
    285       compile_and_run  
    286             Compile the program and run it, 
    287             comparing the output against the  
    288             relevant files. 
    289  
    290       multimod_compile 
    291             Compile a multi-module program 
    292             (more about multi-module programs 
    293             below). 
    294  
    295       multimod_compile_fail 
    296             Compile a multi-module program, 
    297             and expect the compilation to fail 
    298             with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler. 
    299  
    300       multimod_compile_and_run 
    301             Compile and run a multi-module 
    302             program. 
    303  
    304       compile_and_run_with_prefix 
    305             Same as compile_and_run, but with command to use to run the execution of the result binary. 
    306  
    307       multimod_compile_and_run_with_prefix 
    308             Same as multimod_compile_and_run, but with command to use to run the execution of the result binary. 
    309  
    310       run_command 
    311             Just run an arbitrary command.  The output is checked 
    312             against `T.stdout` and `T.stderr` (unless `ignore_output` 
    313             is used), and the stdin and expected exit code can be 
    314             changed in the same way as for compile_and_run. 
    315  
    316       ghci_script 
    317             Runs the current compiler, passing 
    318             --interactive and using the specified 
    319             script as standard input. 
    320  
    321     ''<args>'' 
    322     is a list of arguments to be passed to <test-fn>. 
    323  
    324     For compile, compile_fail and compile_and_run, <args> 
    325     is a list with a single string which contains extra 
    326     compiler options with which to run the test.  eg. 
    327 {{{                 
    328     test('tc001', normal, compile, ['-fglasgow-exts']) 
    329 }}} 
    330     would pass the flag -fglasgow-exts to the compiler 
    331     when compiling tc001. 
    332  
    333     The multimod_ versions of compile and compile_and_run 
    334     expect an extra argument on the front of the list: the 
    335     name of the top module in the program to be compiled 
    336     (usually this will be 'Main'). 
    337  
     185The format of these fields is described in the [wiki:Building/RunningTests#Formatofthetestentries next section]. 
     186 
     187  
    338188 
    339189A multi-module test is straightforward.  It usually goes in a 
     
    350200 
    351201For some examples, take a look in tests/ghc-regress/programs. 
     202 
     203= Format of the test entries = 
     204 
     205Each test in a `test.T` file is specified by a line the form 
     206{{{ 
     207      test(<name>, <setup>, <test-fn>, <args>) 
     208}}} 
     209 
     210== The <name> field == 
     211 
     212''<name>'' is the name of the test, in quotes (' or "). 
     213 
     214== The <setup> field == 
     215 
     216''<setup>''  is a function (i.e. any callable object in Python) 
     217which allows the options for this test to be changed. 
     218There are many pre-defined functions which can be 
     219used in this field: 
     220 
     221 * '''normal'''                don't change any options from the defaults 
     222 * '''skip'''                  skip this test 
     223 * '''skip_if_no_ghci'''       skip unless GHCi is available 
     224 
     225 * '''skip_if_fast'''          skip if "fast" is enabled 
     226 
     227 * '''omit_ways(ways)'''       skip this test for certain ways 
     228 
     229 * '''only_ways(ways)'''       do this test certain ways only 
     230 
     231 * '''extra_ways(ways)'''      add some ways which would normally be disabled 
     232 
     233 * '''omit_compiler_types(compilers)'''                           skip this test for certain compilers 
     234 
     235 * '''only_compiler_types(compilers)'''       do this test for certain compilers only 
     236 
     237 * '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number 
     238 
     239 * '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways 
     240 
     241 * '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type 
     242 
     243 * '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given 
     244 
     245 * '''if_tag(tag, f)'''        do `f` if the compiler has a given tag 
     246 
     247 * '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag 
     248 
     249 * '''set_stdin(file)'''       use a different file for stdin 
     250 
     251 * '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`) 
     252 
     253 * '''exit_code(n)'''          expect an exit code of 'n' from the prog 
     254 
     255 * '''extra_run_opts(opts)'''  pass some extra opts to the prog 
     256 
     257 * '''no_clean'''              don't clean up after this test 
     258 
     259 * '''extra_clean(files)'''    extra files to clean after the test has completed 
     260 
     261 * '''reqlib(P)'''             requires package P 
     262 
     263 * '''req_profiling'''         requires profiling 
     264 
     265 * '''ignore_output'''         don't try to compare output 
     266 
     267 * '''alone'''                 don't run this test in parallel with anything else 
     268 
     269 * '''literate'''              look for a `.lhs` file instead of a `.hs` file 
     270 
     271 * '''c_src'''                 look for a `.c` file 
     272 
     273 * '''cmd_prefix(string)'''    prefix this string to the command when run 
     274 
     275 * '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output 
     276 
     277The following should normally not be used; instead, use the `expect_broken*` 
     278functions above so that the problem doesn't get forgotten about, and when we 
     279come back to look at the test later we know whether current behaviour is why 
     280we marked it as expected to fail: 
     281 
     282 * '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it. 
     283 
     284 * '''expect_fail_for(ways)''' expect failure for certain ways  
     285 
     286To use more than one modifier on a test, just put them in a list. 
     287For example, to expect an exit code of 3 and omit way 'opt', we could use 
     288{{{ 
     289      [ omit_ways(['opt']), exit_code(3) ] 
     290}}} 
     291as the `<setup>` argument. 
     292 
     293== The <test-fn> field == 
     294 
     295''<test-fn>'' 
     296is a function which describes how the test should be 
     297run, and determines the form of <args>.  The possible 
     298values are: 
     299 
     300 * '''compile'''  Just compile the program, the compilation should succeed. 
     301 
     302 * '''compile_fail''' 
     303   Just compile the program, the 
     304   compilation should fail (error 
     305   messages will be in T.stderr). 
     306   This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler. 
     307 
     308 * '''compile_and_run''' 
     309   Compile the program and run it, 
     310   comparing the output against the  
     311   relevant files. 
     312 
     313 * '''multimod_compile''' 
     314   Compile a multi-module program 
     315   (more about multi-module programs 
     316   below). 
     317 
     318 * '''multimod_compile_fail''' 
     319   Compile a multi-module program, 
     320   and expect the compilation to fail 
     321   with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler. 
     322 
     323 * '''multimod_compile_and_run''' 
     324   Compile and run a multi-module 
     325   program. 
     326 
     327 * '''compile_and_run_with_prefix''' 
     328   Same as compile_and_run, but with command to use to run the execution of the result binary. 
     329 
     330 * '''multimod_compile_and_run_with_prefix''' 
     331   Same as multimod_compile_and_run, but with command to use to run the execution of the result binary. 
     332 
     333 * '''run_command''' 
     334   Just run an arbitrary command.  The output is checked 
     335   against `T.stdout` and `T.stderr` (unless `ignore_output` 
     336   is used), and the stdin and expected exit code can be 
     337   changed in the same way as for compile_and_run. 
     338 
     339 * '''ghci_script''' 
     340   Runs the current compiler, passing 
     341   --interactive and using the specified 
     342   script as standard input. 
     343 
     344== The <args> field == 
     345 
     346''<args>'' is a list of arguments to be passed to <test-fn>. 
     347 
     348For compile, compile_fail and compile_and_run, <args> 
     349is a list with a single string which contains extra 
     350compiler options with which to run the test.  eg. 
     351{{{                 
     352    test('tc001', normal, compile, ['-fglasgow-exts']) 
     353}}} 
     354would pass the flag -fglasgow-exts to the compiler 
     355when compiling tc001. 
     356 
     357The multimod_ versions of compile and compile_and_run 
     358expect an extra argument on the front of the list: the 
     359name of the top module in the program to be compiled 
     360(usually this will be 'Main'). 
     361 
     362 
    352363 
    353364= Sample output files =