Opened 2 years ago

Last modified 2 months ago

#11380 new bug

Compiling a 10.000 line file exhausts memory

Reported by: kennethb Owned by:
Priority: normal Milestone:
Component: Compiler Version: 7.10.3
Keywords: Cc:
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description (last modified by kennethb)

I have a project that depends on happy/alex to generate parsers. (https://github.com/TOSPIO/pyn)

Running cabal repl on my laptop with 16GiB memory ended up eating up all memory and eventually resulted in kernel panic.

Problem arises when compiling the giant happy-generated dist/build/Language/Python/Parser/Parse.hs file. The memory usage is flat (~1.5G) within the first 2 mins, and the soars up to over 10GB within seconds.

cabal build and cabal repl --ghc-options=-fobject-code works fine.

Change History (8)

comment:1 Changed 2 years ago by kennethb

Description: modified (diff)
Operating System: Unknown/MultipleLinux

comment:2 Changed 2 years ago by kennethb

Version: 7.10.27.10.3

comment:3 Changed 2 years ago by kennethb

Description: modified (diff)
Last edited 2 years ago by kennethb (previous) (diff)

comment:4 Changed 2 years ago by thomie

Summary: `cabal repl` exhausts memoryCompiling a 10.000 line file exhausts memory
Type of failure: Compile-time crashCompile-time performance bug

Compiling (instead of interpreting) the generated Parser.hs with ghc-7.10.3 (without optimizations) doesn't complete either, when only 3GB of RAM is available (tested with ulimit -v 3000000). The problems seem to begin in the desugaring phase:

*** Parser:
*** Renamer/typechecker:
*** Desugar:
ghc: out of memory (requested 2097152 bytes)

The generated parser is huge (10.000 lines). But it is about the same size as GHC's own parser, which GHC is able to compile with reasonable memory.

I tried making the Parser smaller a bit, but didn't find any obvious rule that might be triggering the problem.

Could you try with other version of GHC (say 7.8.4 and 8.0.1)? Is it a regression, or maybe already fixed?

Last edited 2 years ago by thomie (previous) (diff)

comment:5 Changed 2 years ago by jstolarek

kennethb, what happens when you try to compile your parser with -O0 option (ie. without optimizations)?

comment:6 Changed 2 years ago by thomie

happy has a -a flag:

 -a        --array            generate an array-based parser

Cabal calls happy -agc by default, but I was using just happy in comment:4. It seems to influence the results when trying to compile Parser.hs within 3GB RAM.

ghc-7.10.3 Parser.hs happy happy -a
-O0 not ok ok
-O1 ok ok

@jstolarek: compiling the non-array based parser without optimizations fails. Very strange.

Last edited 2 years ago by thomie (previous) (diff)

comment:7 Changed 2 years ago by rwbarton

Priority: highnormal

Let's stick to the array-based parser since that's what the original issue was about. Then there's no problem compiling, either with or without optimizations. The problem is only with ghci for some reason.

Trying with -v, there seems to be some major early difference when building with ghc vs. ghci.

rwbarton@morphism:/tmp/pyn$ ghc-7.10.1 -fforce-recomp -c dist/build/Language/Python/Pyn/Parser/Parser.hs -O0 -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp
Glasgow Haskell Compiler, Version 7.10.1, stage 2 booted by GHC version 7.8.4
[...]
*** Desugar:
Result size of Desugar (after optimization)
  = {terms: 214,221, types: 3,864,905, coercions: 74,929}
[... everything is fine]
rwbarton@morphism:/tmp/pyn$ ghci-7.10.1 dist/build/Language/Python/Pyn/Parser/Parser.hs -ilib -idist/build -v -this-package-key pyn_4LeYrll2NJX6hY3DO8vvbp
[...]
[10 of 10] Compiling Language.Python.Pyn.Parser.Parser ( dist/build/Language/Python/Pyn/Parser/Parser.hs, interpreted )
*** Parser:
*** Renamer/typechecker:
*** Desugar:
Result size of Desugar (after optimization)
  = {terms: 220,453, types: 414,293,970, coercions: 75,337}
*** Simplifier:
Result size of Simplifier iteration=1
  = {terms: 220,516, types: 419,756,722, coercions: 577,145}
Result size of Simplifier iteration=2
  = {terms: 220,516, types: 419,756,722, coercions: 370,154}
Result size of Simplifier
  = {terms: 220,514, types: 419,756,719, coercions: 370,206}
*** Tidy Core:
Result size of Tidy Core^CInterrupted. [here memory usage exploded]

What would cause this huge difference?

comment:8 Changed 2 months ago by AndreasK

This regressed farther in 8.2.2 and 8.0.2 (at -O1)

Compiling the project in GHC-8.2.2 takes forever (~10 Hours on an i7-6700). However this seems to he an issue with recent happy versions.

Ram usage for compile or ghci didn't go over 3GB in the first 10 Minutes but I stopped watching it afterwards. So I can't say for sure if it did run out of ram and started swapping eventually.

Still a significant difference between ghci and compiled code.

Logs for build/ghci with 8.2.2: repo for repro

Compiled (-O1)

[17 of 20] Compiling Language.Python.Pyn.Parser.Parser ( .stack-work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.hs, .stack-work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.o )
*** Checking old interface for Language.Python.Pyn.Lexer (use -ddump-hi-diffs for more details):
*** Deleting temp files:
Deleting:
compile: input file C:\Users\Andi\AppData\Local\Temp\ghc81932_0\ghc_7.hscpp
*** Checking old interface for Language.Python.Pyn.Parser.Parser (use -ddump-hi-diffs for more details):
*** Parser [Language.Python.Pyn.Parser.Parser]:
!!! Parser [Language.Python.Pyn.Parser.Parser]: finished in 937.50 milliseconds, allocated 2647.317 megabytes
*** Renamer/typechecker [Language.Python.Pyn.Parser.Parser]:
!!! Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: finished in 35763203.12 milliseconds, allocated 109977425.131 megabytes
*** Desugar [Language.Python.Pyn.Parser.Parser]:
Result size of Desugar (after optimization)
  = {terms: 83,551,
     types: 2,023,928,
     coercions: 76,061,
     joins: 1/136}
	 
	 
	 
Interpreter:

[17 of 21] Compiling Language.Python.Pyn.Parser.Parser ( C:\test\pyn\.stack-work\dist\5c8418a7\build\Language\Python\Pyn\Parser\Parser.hs, interpreted )
*** Parser [Language.Python.Pyn.Parser.Parser]:
!!! Parser [Language.Python.Pyn.Parser.Parser]: finished in 890.62 milliseconds, allocated 2646.366 megabytes
*** Renamer/typechecker [Language.Python.Pyn.Parser.Parser]:
!!! Renamer/typechecker [Language.Python.Pyn.Parser.Parser]: finished in 35747375.00 milliseconds, allocated 109978920.259 megabytes
*** Desugar [Language.Python.Pyn.Parser.Parser]:
Result size of Desugar (after optimization)
  = {terms: 217,920,
     types: 3,341,879,
     coercions: 76,475,
     joins: 0/653}
Last edited 2 months ago by AndreasK (previous) (diff)
Note: See TracTickets for help on using tickets.