Opened 2 years ago

Last modified 7 months ago

#5793 new task

make nofib not suck — at Version 5

Reported by: dterei Owned by: dterei
Priority: normal Milestone:
Component: NoFib benchmark suite Version:
Keywords: Cc: ndmitchell@…, rwbarton@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Difficulty: Unknown
Test Case: Blocked By:
Blocking: #5794 Related Tickets:

Description (last modified by dterei)

Nofib is the standard tool GHC developers use to benchmark changes to the compiler. Its a known fact though that nofib sucks. Its overall design is OK but it's had no love and care for many years and has bittrotted into near uselessness in a lot of situations.

This task isn't to implement an awesome, really useful benchmark suite for Haskell. That would be great but is a huge undertaking. This task is just about making nofib useful again.

The breakdown for this is something like:

  1. Think and maybe fix nofib framework design. It has 'ways' which I think correspond to compilation method but more in the sense of 'dynamic' vs 'static', seems it may not suite being able to use ways for 'fasm' vs 'fllvm'. There is also the concept of 'modes' which corresponds to different benchmark input. So 'normal' and 'slow' for getting different run-times. At moment no easy way to select which benchmark groups to run, so may want to change that. I guess we should just decide, what knobs do we want to be able to easily tweak, and see how well the current design allows that.
  1. Fixup the runtimes for benchmarks to be significant. (A lot of them run in < 1 or < 0.1 seconds). I think trying to get all of them to run in around 10s is a good target.
  1. Above task is to fix normal but we may want to fixup slow as well and perhaps add a 'fast' mode where benchmarks run in around 1 second.
  1. Maybe add more benchmarks to the suite (text, bytestring, performance regressions from ghc testsuite, vector....)

Change History (5)

comment:1 Changed 2 years ago by tibbe

I assume you mean nofib, not fibon.

comment:2 Changed 2 years ago by dterei

  • Blocking 5794 added

comment:3 Changed 2 years ago by dterei

Pushed a whole bunch of fixes so that fibon all compiles now at least. Now need to do some work on framework and make sure all the tests have significant run-times.

comment:4 Changed 2 years ago by dterei

  • Summary changed from make fibon not suck to make nofib not suck

Yes. meant nofib :). With fibon, nofib, nobench... it gets a little jumbled in my head at times.

comment:5 Changed 2 years ago by dterei

  • Description modified (diff)
Note: See TracTickets for help on using tickets.