Compiling DynFlags is jolly slow

[simonpj]

Compiling DynFlags really takes a long time these days.

Ian thinks that it's due to the Read and Show instances he has added (see attached W2.hs.

Simon M suggests: instead of using Read/Show, you could generate some code in mkDerivedConstants to use ReadP and Outputable, which should be much smaller and faster.

This ticket is

• To see if we can speed up compilation of DynFlags
• To check WHY it is so slow. Are there any lessons we can learn or ways to make it compile faster? Is it tickling some asymptotically-bad corner of the compiler?

Simon

[simonpj]

Exhibts long compilation times

[igloo]

    Fri Oct 19 20:23 2012 Time and Allocation Profiling Report  (Final)

       ghc-stage2 +RTS -p -h -RTS -B/home/ian/ghc/git/ghc/inplace/lib -fforce-recomp -c W2.hs

    total time  =       26.08 secs   (26080 ticks @ 1000 us, 1 processor)
    total alloc = 14,601,152,424 bytes  (excludes profiling overheads)

COST CENTRE       MODULE         %time %alloc

OccAnal           SimplCore       27.0   24.5
RegAlloc          AsmCodeGen      19.2   22.4
pprNativeCode     AsmCodeGen      16.8   19.9
regLiveness       AsmCodeGen       8.0    6.3
genMachCode       AsmCodeGen       5.0    5.4
StgCmm            HscMain          4.3    3.5
occAnalBind.assoc OccurAnal        3.3    3.3
tops              CmmPipeline      3.3    3.4
SimplTopBinds     SimplCore        1.7    1.6
do_block          Hoopl.Dataflow   1.1    0.8

[igloo]

    Fri Oct 19 20:32 2012 Time and Allocation Profiling Report  (Final)

       ghc-stage2 +RTS -p -h -RTS -B/home/ian/ghc/git/ghc/inplace/lib -fforce-recomp -O -c W2.hs

    total time  =       76.26 secs   (76258 ticks @ 1000 us, 1 processor)
    total alloc = 45,073,993,600 bytes  (excludes profiling overheads)

COST CENTRE       MODULE      %time %alloc

OccAnal           SimplCore    49.7   45.3
CoreTidy          HscMain      22.3   31.8
SimplTopBinds     SimplCore    12.9    9.9
pprNativeCode     AsmCodeGen    2.0    1.8
RegAlloc          AsmCodeGen    1.9    2.0
sink2             CmmPipeline   1.5    1.8
StgCmm            HscMain       1.0    0.8
occAnalBind.assoc OccurAnal     1.0    1.0

[igloo]

                                                     individual     inherited   
COST CENTRE               MODULE      no.  entries  %time %alloc   %time %alloc

getProxies.fwd_pe         OccurAnal  1776     9226    6.7    0.0    25.5   23.7 
 getProxies.fwd_pe.add1   OccurAnal  1779  9733280    5.8    1.6    18.8   23.7 
  getProxies.fwd_pe.add2  OccurAnal  1781  9733280   13.1   22.1    13.1   22.1 

(without -O)

[igloo]

Simon, I think the conclusion was that you'd look into this. Please bounce it back to me if you'd like me to investigate further.

[igloo]

See also #7450

[simonpj@…]

commit 52e43004f63276c1342933e40a673ad25cf2113a

Author: Simon Peyton Jones <simonpj@microsoft.com>
Date:   Fri Dec 21 17:39:33 2012 +0000

    Use expectP in deriving( Read )
    
    Note [Use expectP]   in TcGenDeriv
    ~~~~~~~~~~~~~~~~~~
    Note that we use
       expectP (Ident "T1")
    rather than
       Ident "T1" <- lexP
    The latter desugares to inline code for matching the Ident and the
    string, and this can be very voluminous. The former is much more
    compact.  Cf Trac #7258, although that also concerned non-linearity in
    the occurrence analyser, a separate issue.

 compiler/prelude/PrelNames.lhs    |    3 +-
 compiler/typecheck/TcGenDeriv.lhs |   43 ++++++++++++++++++++++--------------
 2 files changed, 28 insertions(+), 18 deletions(-)

[simonpj@…]

commit 9ea2b6666cb6684279a120c688e8557bcef3dc73

Author: Simon Peyton Jones <simonpj@microsoft.com>
Date:   Mon Dec 24 11:34:51 2012 +0000

    Simplify the binder-swap transformation
    
    The occurrence analyser implements the "binder-swap" transformation,
    described in Note [Binder swap] in OccAnal. For some reason I had
    implemeted an extremely complicated version, I believe intended to get
    as much as possible done in single simplifier pass.  But it turned
    out (Trac #7258) that the 'getProxies' bit of this complicated code
    scaled rather non-linearly, and all by itself could consume half of
    the entire compile time.
    
    The patch dramatically simplifies the transformation, so that
    we simply swizzle
         case x of y { I# v -> e }
    to
         case x of y { I# v -> let x = y in e }
    
    I can't see any reason not to do this
    
      * Compiler allocation for #7258 with 200 fields goes down by 25%
        and compile time by 20%
    
      * The nofib figures do not budge
    
      * Quite a bit of complicated code goes away

 compiler/simplCore/OccurAnal.lhs |  255 +++++++-------------------------------
 1 files changed, 46 insertions(+), 209 deletions(-)

[simonpj]

OK I have verified that the changes to the occurrence analyser (above) make essentially zero different to nofib numbers. It's a very worthwhile simplification, because it completely gets rid of the getProxies stuff that was eating all the time before.

Alas, compiling W2 is still non-linear. Here's the allocation by the stage-2 compiler

• 50 fields: 1.01 Gbyte
• 100 fields: 2.98 Gbyte
• 200 fields: 9.64 Gbyte

This is after including the improvements to the derived Read code in #7450.

So something is still wrong. Need to do some profilling to find out.

There are some very deeply nested lambdas, which lead to SRTs of ever-increasing size, so there is definitely a quadratic effect there. But I'm not sure that is all.

[simonmar]

I can't compile DynFlags on my 256MB Raspberry Pi. Could we do something about having less auto-generated code in this module please?