Opened 2 years ago

Last modified 23 months ago

#11380 new bug

Compiling a 10.000 line file exhausts memory

Reported by: kennethb Owned by:
Priority: normal Milestone:
Component: Compiler Version: 7.10.3
Keywords: Cc:
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description (last modified by kennethb)

I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn)

Running cabal repl on my laptop with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic.

Problem arises when compiling the giant happy-generated dist/build/Language/Python/Parser/Parse.hs file. The memory usage is flat (~1.5G) within the first 2 mins, and the soars up to over 10GB within seconds.

cabal build and cabal repl --ghc-options=-fobject-code works fine.

Change History (7)

comment:1 Changed 2 years ago by kennethb

Description: modified (diff)
Operating System: Unknown/MultipleLinux

comment:2 Changed 2 years ago by kennethb

Version: 7.10.27.10.3

comment:3 Changed 2 years ago by kennethb

Description: modified (diff)
Last edited 2 years ago by kennethb (previous) (diff)

comment:4 Changed 2 years ago by thomie

Summary: `cabal repl` exhausts memoryCompiling a 10.000 line file exhausts memory
Type of failure: Compile-time crashCompile-time performance bug

Compiling (instead of interpreting) the generated Parser.hs with ghc-7.10.3 (without optimizations) doesn't complete either, when only 3GB of RAM is available (tested with ulimit -v 3000000). The problems seem to begin in the desugaring phase:

*** Parser:
*** Renamer/typechecker:
*** Desugar:
ghc: out of memory (requested 2097152 bytes)

The generated parser is huge (10.000 lines). But it is about the same size as GHC's own parser, which GHC is able to compile with reasonable memory.

I tried making the Parser smaller a bit, but didn't find any obvious rule that might be triggering the problem.

Could you try with other version of GHC (say 7.8.4 and 8.0.1)? Is it a regression, or maybe already fixed?

Last edited 2 years ago by thomie (previous) (diff)

comment:5 Changed 2 years ago by jstolarek

kennethb, what happens when you try to compile your parser with -O0 option (ie. without optimizations)?

comment:6 Changed 2 years ago by thomie

happy has a -a flag:

 -a        --array            generate an array-based parser

Cabal calls happy -agc by default, but I was using just happy in comment:4. It seems to influence the results when trying to compile Parser.hs within 3GB RAM.

ghc-7.10.3 Parser.hs happy happy -a
-O0 not ok ok
-O1 ok ok

@jstolarek: compiling the non-array based parser without optimizations fails. Very strange.

Last edited 2 years ago by thomie (previous) (diff)

comment:7 Changed 23 months ago by rwbarton

Priority: highnormal

Let's stick to the array-based parser since that's what the original issue was about. Then there's no problem compiling, either with or without optimizations. The problem is only with ghci for some reason.

Trying with -v, there seems to be some major early difference when building with ghc vs. ghci.

rwbarton@morphism:/tmp/pyn$ ghc-7.10.1 -fforce-recomp -c dist/build/Language/Python/Pyn/Parser/Parser.hs -O0 -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp
Glasgow Haskell Compiler, Version 7.10.1, stage 2 booted by GHC version 7.8.4
[...]
*** Desugar:
Result size of Desugar (after optimization)
  = {terms: 214,221, types: 3,864,905, coercions: 74,929}
[... everything is fine]
rwbarton@morphism:/tmp/pyn$ ghci-7.10.1 dist/build/Language/Python/Pyn/Parser/Parser.hs -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp
[...]
[10 of 10] Compiling Language.Python.Pyn.Parser.Parser ( dist/build/Language/Python/Pyn/Parser/Parser.hs, interpreted )
*** Parser:
*** Renamer/typechecker:
*** Desugar:
Result size of Desugar (after optimization)
  = {terms: 220,453, types: 414,293,970, coercions: 75,337}
*** Simplifier:
Result size of Simplifier iteration=1
  = {terms: 220,516, types: 419,756,722, coercions: 577,145}
Result size of Simplifier iteration=2
  = {terms: 220,516, types: 419,756,722, coercions: 370,154}
Result size of Simplifier
  = {terms: 220,514, types: 419,756,719, coercions: 370,206}
*** Tidy Core:
Result size of Tidy Core^CInterrupted. [here memory usage exploded]

What would cause this huge difference?

Note: See TracTickets for help on using tickets.