Changes between Version 9 and Version 10 of Building/RunningTests/Adding


Ignore:
Timestamp:
May 10, 2012 9:56:27 AM (3 years ago)
Author:
simonmar
Comment:

add info about performance tests

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests/Adding

    v9 v10  
    172172=== The <setup> field ===
    173173
    174 ''<setup>''  is a function (i.e. any callable object in Python)
    175 which allows the options for this test to be changed.
    176 There are many pre-defined functions which can be
    177 used in this field:
     174''<setup>''  is a function (i.e. any callable object in Python) which allows the options for this test to be changed.
     175There are many pre-defined functions which can be used in this field:
    178176
    179177 * '''normal'''                don't change any options from the defaults
     
    248246}}}
    249247as the `<setup>` argument.
     248
     249=== Performance tests ===
     250
     251Performance tests can specify ranges for certain statistics in the `<setup>` field.  Here's an example test:
     252
     253{{{
     254test('perf001',
     255     [ if_wordsize(32,
     256          compiler_stats_range_field('bytes allocated', 40000000, 10)),
     257       if_wordsize(64,
     258          compiler_stats_range_field('bytes allocated', 79110184, 10))
     259     ],
     260     compile,[''])
     261}}}
     262
     263This is testing the performance of GHC itself, and requiring that the statistic 'bytes allocated' for the compiler when compiling the module `perf001.hs` is +/- 10% of 40000000 bytes (on a 32-bit machine; there is a different baseline for 64-bit machines).
     264
     265The kinds of constraint that can be used are:
     266
     267 * '''compiler_stats_range_field(stat, baseline, deviation)'''  tests the performance of GHC, and should be used with '''compile''' or '''compile_fail''' tests.  '''stat''' is one of the following: `'bytes allocated'`, `'peak_megabytes_allocated'`, or `'max_bytes_used'`; '''baseline''' is the baseline value obtained by running the benchmark, and '''deviation''' is the percentage deviation from the baseline that the framework will allow for the test to pass.
     268
     269 * '''stats_range_field(stat, baseline, deviation)'''  is the same, but tests the performance of the ''program'', not the compiler.  It should be used in conjunction with a '''compile_and_run''' test.
    250270
    251271=== The <test-fn> field ===