Changes between Version 22 and Version 23 of Building/RunningTests


Ignore:
Timestamp:
Aug 21, 2009 9:46:55 AM (6 years ago)
Author:
simonpj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests

    v22 v23  
    132132    go in parser/, and so on. 
    133133
    134     It's not always possible to find a single best place for a test;
    135     in those cases just pick one which seems reasonable.
    136 
    137     Under each main directory may be up to three subdirectories:
    138 
    139        should_compile:
     134 It's not always possible to find a single best place for a test;
     135 in those cases just pick one which seems reasonable.
     136
     137 Under each main directory may be up to three subdirectories:
     138       '''should_compile''':   
    140139           tests which need to compile only
    141 
    142        should_fail:   
     140       '''should_fail''':   
    143141           tests which should fail to compile and generate a particular error message
    144 
    145        should_run:
     142       '''should_run''':
    146143           tests which should compile, run with some specific input, and generate a particular output.
    147144   
    148     We don't always divide the tests up like this, and it's not
    149     essential to do so (the directory names have no meaning as
    150     far as the test driver is concerned).       
     145 We don't always divide the tests up like this, and it's not
     146 essential to do so (the directory names have no meaning as
     147 far as the test driver is concerned).       
    151148
    152149
    153150 2. Having found a suitable place for the test, give the test a name.
    154     For regression tests, we often just name the test after the bug number (e.g. 2047).
     151    For regression tests, we often just name the test after the bug number (e.g. T2047).
    155152    Alternatively, follow the convention for the directory in which you place the
    156153    test: for example, in typecheck/should_compile, tests are named
     
    182179
    183180
    184  1. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
     181 3. Edit all.T in the relevant directory and add a line for the test.  The line is always of the form
    185182{{{
    186183      test(<name>, <setup>, <test-fn>, <args>)
    187184}}}
    188     where
    189 
    190     ''<name>'' is the name of the test, in quotes (' or ").
    191 
    192     ''<setup>''  is a function (i.e. any callable object in Python)
    193     which allows the options for this test to be changed.
    194     There are many pre-defined functions which can be
    195     used in this field:
    196 
    197         '''normal'''                don't change any options from the defaults
    198 
    199         '''skip'''                  skip this test
    200 
    201         '''skip_if_no_ghci'''       skip unless GHCi is available
    202 
    203         '''skip_if_fast'''          skip if "fast" is enabled
    204 
    205         '''omit_ways(ways)'''       skip this test for certain ways
    206 
    207         '''only_ways(ways)'''       do this test certain ways only
    208 
    209         '''extra_ways(ways)'''      add some ways which would normally be disabled
    210 
    211         '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
    212 
    213         '''only_compiler_types(compilers)'''       do this test for certain compilers only
    214 
    215         '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number
    216 
    217         '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways
    218 
    219         '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type
    220 
    221         '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given
    222 
    223         '''if_tag(tag, f)'''        do `f` if the compiler has a given tag
    224 
    225         '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag
    226 
    227         '''set_stdin(file)'''       use a different file for stdin
    228 
    229         '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`)
    230 
    231         '''exit_code(n)'''          expect an exit code of 'n' from the prog
    232 
    233         '''extra_run_opts(opts)'''  pass some extra opts to the prog
    234 
    235         '''no_clean'''              don't clean up after this test
    236 
    237         '''extra_clean(files)'''    extra files to clean after the test has completed
    238 
    239         '''reqlib(P)'''             requires package P
    240 
    241         '''req_profiling'''         requires profiling
    242 
    243         '''ignore_output'''         don't try to compare output
    244 
    245         '''alone'''                 don't run this test in parallel with anything else
    246 
    247         '''literate'''              look for a `.lhs` file instead of a `.hs` file
    248 
    249         '''c_src'''                 look for a `.c` file
    250 
    251         '''cmd_prefix(string)'''    prefix this string to the command when run
    252 
    253         '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output
    254 
    255     To use more than one modifier on a test, just put them in a list.
    256     For example, to expect an exit code of 3 and omit way 'opt', we could use
    257 {{{
    258       [ omit_ways(['opt']), exit_code(3) ]
    259 }}}
    260       as the `<setup>` argument.
    261 
    262     The following should normally not be used; instead, use the `expect_broken*`
    263     functions above so that the problem doesn't get forgotten about, and when we
    264     come back to look at the test later we know whether current behaviour is why
    265     we marked it as expected to fail:
    266 
    267         '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
    268 
    269         '''expect_fail_for(ways)''' expect failure for certain ways
    270 
    271     ''<test-fn>''
    272     is a function which describes how the test should be
    273     run, and determines the form of <args>.  The possible
    274     values are:
    275 
    276       compile
    277             Just compile the program, the compilation should succeed.
    278 
    279       compile_fail
    280             Just compile the program, the
    281             compilation should fail (error
    282             messages will be in T.stderr).
    283             This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler.
    284 
    285       compile_and_run
    286             Compile the program and run it,
    287             comparing the output against the
    288             relevant files.
    289 
    290       multimod_compile
    291             Compile a multi-module program
    292             (more about multi-module programs
    293             below).
    294 
    295       multimod_compile_fail
    296             Compile a multi-module program,
    297             and expect the compilation to fail
    298             with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler.
    299 
    300       multimod_compile_and_run
    301             Compile and run a multi-module
    302             program.
    303 
    304       compile_and_run_with_prefix
    305             Same as compile_and_run, but with command to use to run the execution of the result binary.
    306 
    307       multimod_compile_and_run_with_prefix
    308             Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
    309 
    310       run_command
    311             Just run an arbitrary command.  The output is checked
    312             against `T.stdout` and `T.stderr` (unless `ignore_output`
    313             is used), and the stdin and expected exit code can be
    314             changed in the same way as for compile_and_run.
    315 
    316       ghci_script
    317             Runs the current compiler, passing
    318             --interactive and using the specified
    319             script as standard input.
    320 
    321     ''<args>''
    322     is a list of arguments to be passed to <test-fn>.
    323 
    324     For compile, compile_fail and compile_and_run, <args>
    325     is a list with a single string which contains extra
    326     compiler options with which to run the test.  eg.
    327 {{{               
    328     test('tc001', normal, compile, ['-fglasgow-exts'])
    329 }}}
    330     would pass the flag -fglasgow-exts to the compiler
    331     when compiling tc001.
    332 
    333     The multimod_ versions of compile and compile_and_run
    334     expect an extra argument on the front of the list: the
    335     name of the top module in the program to be compiled
    336     (usually this will be 'Main').
    337 
     185The format of these fields is described in the [wiki:Building/RunningTests#Formatofthetestentries next section].
     186
     187 
    338188
    339189A multi-module test is straightforward.  It usually goes in a
     
    350200
    351201For some examples, take a look in tests/ghc-regress/programs.
     202
     203= Format of the test entries =
     204
     205Each test in a `test.T` file is specified by a line the form
     206{{{
     207      test(<name>, <setup>, <test-fn>, <args>)
     208}}}
     209
     210== The <name> field ==
     211
     212''<name>'' is the name of the test, in quotes (' or ").
     213
     214== The <setup> field ==
     215
     216''<setup>''  is a function (i.e. any callable object in Python)
     217which allows the options for this test to be changed.
     218There are many pre-defined functions which can be
     219used in this field:
     220
     221 * '''normal'''                don't change any options from the defaults
     222 * '''skip'''                  skip this test
     223 * '''skip_if_no_ghci'''       skip unless GHCi is available
     224
     225 * '''skip_if_fast'''          skip if "fast" is enabled
     226
     227 * '''omit_ways(ways)'''       skip this test for certain ways
     228
     229 * '''only_ways(ways)'''       do this test certain ways only
     230
     231 * '''extra_ways(ways)'''      add some ways which would normally be disabled
     232
     233 * '''omit_compiler_types(compilers)'''                           skip this test for certain compilers
     234
     235 * '''only_compiler_types(compilers)'''       do this test for certain compilers only
     236
     237 * '''expect_broken(bug)''' this test is a expected not to work due to the indicated trac bug number
     238
     239 * '''expect_broken_for(bug, ways)''' as expect_broken, but only for the indicated ways
     240
     241 * '''if_compiler_type(compiler_type, f)''' Do `f`, but only for the given compiler type
     242
     243 * '''if_platform(plat, f)'''  Do `f`, but only if we are on the specific platform given
     244
     245 * '''if_tag(tag, f)'''        do `f` if the compiler has a given tag
     246
     247 * '''unless_tag(tag, f)'''    do `f` unless the compiler has a given tag
     248
     249 * '''set_stdin(file)'''       use a different file for stdin
     250
     251 * '''no_stdin'''              use no stdin at all (otherwise use `/dev/null`)
     252
     253 * '''exit_code(n)'''          expect an exit code of 'n' from the prog
     254
     255 * '''extra_run_opts(opts)'''  pass some extra opts to the prog
     256
     257 * '''no_clean'''              don't clean up after this test
     258
     259 * '''extra_clean(files)'''    extra files to clean after the test has completed
     260
     261 * '''reqlib(P)'''             requires package P
     262
     263 * '''req_profiling'''         requires profiling
     264
     265 * '''ignore_output'''         don't try to compare output
     266
     267 * '''alone'''                 don't run this test in parallel with anything else
     268
     269 * '''literate'''              look for a `.lhs` file instead of a `.hs` file
     270
     271 * '''c_src'''                 look for a `.c` file
     272
     273 * '''cmd_prefix(string)'''    prefix this string to the command when run
     274
     275 * '''normalise_slashes'''     convert backslashes to forward slashes before comparing the output
     276
     277The following should normally not be used; instead, use the `expect_broken*`
     278functions above so that the problem doesn't get forgotten about, and when we
     279come back to look at the test later we know whether current behaviour is why
     280we marked it as expected to fail:
     281
     282 * '''expect_fail'''           this test is an expected failure, i.e. there is a known bug in the compiler, but we don't want to fix it.
     283
     284 * '''expect_fail_for(ways)''' expect failure for certain ways
     285
     286To use more than one modifier on a test, just put them in a list.
     287For example, to expect an exit code of 3 and omit way 'opt', we could use
     288{{{
     289      [ omit_ways(['opt']), exit_code(3) ]
     290}}}
     291as the `<setup>` argument.
     292
     293== The <test-fn> field ==
     294
     295''<test-fn>''
     296is a function which describes how the test should be
     297run, and determines the form of <args>.  The possible
     298values are:
     299
     300 * '''compile'''  Just compile the program, the compilation should succeed.
     301
     302 * '''compile_fail'''
     303   Just compile the program, the
     304   compilation should fail (error
     305   messages will be in T.stderr).
     306   This kind of failure is mandated by the language definition - it does '''not''' indicate any bug in the compiler.
     307
     308 * '''compile_and_run'''
     309   Compile the program and run it,
     310   comparing the output against the
     311   relevant files.
     312
     313 * '''multimod_compile'''
     314   Compile a multi-module program
     315   (more about multi-module programs
     316   below).
     317
     318 * '''multimod_compile_fail'''
     319   Compile a multi-module program,
     320   and expect the compilation to fail
     321   with error messages in T.stderr.  This kind of failure does '''not''' indicate a bug in the compiler.
     322
     323 * '''multimod_compile_and_run'''
     324   Compile and run a multi-module
     325   program.
     326
     327 * '''compile_and_run_with_prefix'''
     328   Same as compile_and_run, but with command to use to run the execution of the result binary.
     329
     330 * '''multimod_compile_and_run_with_prefix'''
     331   Same as multimod_compile_and_run, but with command to use to run the execution of the result binary.
     332
     333 * '''run_command'''
     334   Just run an arbitrary command.  The output is checked
     335   against `T.stdout` and `T.stderr` (unless `ignore_output`
     336   is used), and the stdin and expected exit code can be
     337   changed in the same way as for compile_and_run.
     338
     339 * '''ghci_script'''
     340   Runs the current compiler, passing
     341   --interactive and using the specified
     342   script as standard input.
     343
     344== The <args> field ==
     345
     346''<args>'' is a list of arguments to be passed to <test-fn>.
     347
     348For compile, compile_fail and compile_and_run, <args>
     349is a list with a single string which contains extra
     350compiler options with which to run the test.  eg.
     351{{{               
     352    test('tc001', normal, compile, ['-fglasgow-exts'])
     353}}}
     354would pass the flag -fglasgow-exts to the compiler
     355when compiling tc001.
     356
     357The multimod_ versions of compile and compile_and_run
     358expect an extra argument on the front of the list: the
     359name of the top module in the program to be compiled
     360(usually this will be 'Main').
     361
     362
    352363
    353364= Sample output files =