Changes between Version 9 and Version 10 of Building/RunningTests/Adding


Ignore:
Timestamp:
May 10, 2012 9:56:27 AM (3 years ago)
Author:
simonmar
Comment:

add info about performance tests

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests/Adding

    v9 v10  
    172172=== The <setup> field === 
    173173 
    174 ''<setup>''  is a function (i.e. any callable object in Python) 
    175 which allows the options for this test to be changed. 
    176 There are many pre-defined functions which can be 
    177 used in this field: 
     174''<setup>''  is a function (i.e. any callable object in Python) which allows the options for this test to be changed. 
     175There are many pre-defined functions which can be used in this field: 
    178176 
    179177 * '''normal'''                don't change any options from the defaults 
     
    248246}}} 
    249247as the `<setup>` argument. 
     248 
     249=== Performance tests === 
     250 
     251Performance tests can specify ranges for certain statistics in the `<setup>` field.  Here's an example test: 
     252 
     253{{{ 
     254test('perf001', 
     255     [ if_wordsize(32, 
     256          compiler_stats_range_field('bytes allocated', 40000000, 10)), 
     257       if_wordsize(64, 
     258          compiler_stats_range_field('bytes allocated', 79110184, 10)) 
     259     ], 
     260     compile,['']) 
     261}}} 
     262 
     263This is testing the performance of GHC itself, and requiring that the statistic 'bytes allocated' for the compiler when compiling the module `perf001.hs` is +/- 10% of 40000000 bytes (on a 32-bit machine; there is a different baseline for 64-bit machines). 
     264 
     265The kinds of constraint that can be used are: 
     266 
     267 * '''compiler_stats_range_field(stat, baseline, deviation)'''  tests the performance of GHC, and should be used with '''compile''' or '''compile_fail''' tests.  '''stat''' is one of the following: `'bytes allocated'`, `'peak_megabytes_allocated'`, or `'max_bytes_used'`; '''baseline''' is the baseline value obtained by running the benchmark, and '''deviation''' is the percentage deviation from the baseline that the framework will allow for the test to pass. 
     268 
     269 * '''stats_range_field(stat, baseline, deviation)'''  is the same, but tests the performance of the ''program'', not the compiler.  It should be used in conjunction with a '''compile_and_run''' test. 
    250270 
    251271=== The <test-fn> field ===