GHC Performance Index
|Reported by:||simonmar||Owned by:|
|Type of failure:||Difficulty:||Moderate (less than a day)|
|Test Case:||N/A||Blocked By:|
We want a lightweight way to detect when either (a) the compiler's performance changes, or (b) the performance of compiled code changes. This was triggerred by my noticing that the HEAD nightly builds seem somewhat slower than usual right now.
So the idea is pretty simple. We decide on a fixed set of nofib programs --intiially all of them, but we want the set to remain stable in the future -- and as part of the nightly build we time how long it takes to compile this set with a fixed set of flags (probably -O -fasm, to rule out changes in gcc). Then we time how long it takes to run all those programs, maybe 5 times each. Then we keep track of these two figures for every successful nightly build, and plot the results, giving us a nice way to track GHC's performance over time. We can go back and get some historical figures too. Of course the results are specific to a particular machine, so we'd have to keep track of results per nightly build machine or something.
So basically: some small changes to nofib to allow "just build everything" (I think even NoFibRuns?=0 might do this), and to build a fixed set of programs (the set could be built into the nofib build system, no need to make it dynamic). And some way to collect the results and plot graphs.