Reduce long-term memory usage of GHCi

If you take moderately sized Haskell project (let's use lens as an example) and decide to work with it using cabal repl, memory usage of GHCi becomes a bit problematic after a few code reloads.

I compared 7.10.3 and 8.0.1 using the following steps:

1. Start GHCi with cabal repl.
2. Wait for all modules to load, check memory usage with htop.
3. Issue find . -name '*.hs' -exec touch {} + in src directory.
4. Issue :r in GHCi so it reloads all modules.
5. Repeat (3) and (4) four more times.

Results:

GHC 7.10.3

• After cabal repl: 874MB
• After reloading all modules (1st): 1304MB
• After reloading all modules (2nd): 1763MB
• After reloading all modules (3rd): 2216MB
• After reloading all modules (4th): 2672MB
• After reloading all modules (5th): 3121MB

Basically, every time modules are reloaded memory usage increases by ~450MB, which is definitely not fun.

GHC 8.0.1

• After cabal repl: 1322MB
• After reloading all modules (1st): 2144MB
• After reloading all modules (2nd): 2201MB
• After reloading all modules (3rd): 2208MB
• After reloading all modules (4th): 2212MB
• After reloading all modules (5th): 2213MB

Here the initial memory usage is much higher as with 7.10.3 and there is a sharp increase after first full reload, but then it more or less plateaus.

I presume this change is the result of 6ae616f5 - upfront memory usage is probably higher due to more things being forced (and the fact that memory usage doesn't increase indefinitely is great).

However, I'm still concerned about the spike after the first reload - would it be possible to get rid of (or at least decrease) it?

Trac metadata

Trac field	Value
Version	8.0.1
Type	FeatureRequest
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	GHCi
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

changed weight to 5

added GHCi Tfeature request Trac import labels

mentioned in issue #14336 (closed)

I have proposed this as an idea for GSoC / HSoC at https://github.com/haskell-org/summer-of-haskell/pull/15/files

If anyone is interesting in mentoring students working on fixing this space leak, please participate in the discussion at https://github.com/haskell-org/summer-of-haskell/issues/29

changed weight to 7

Indeed 400MB is quite a jump. I think this would make for a reasonable SoC project if coupled with a few other bugs. I suspect tracking this down won't take an entire summer, even for a new contributor.

Trac metadata

Trac field	Value
Priority	normal → high

Is https://simonmar.github.io/posts/2018-06-20-Finding-fixing-space-leaks.html related to this ticket?

Indeed that is related, however I just tried GHC 8.6 and this problem still occurs. As in the original report for GHC 8.0.1, the leak stops after a few reloads. This seems very similar to #15111 (closed), I don't know why this is still not fixed.

Trac metadata

Trac field	Value
Related	- → #15111 (closed)

added Phigh label

I can still reproduce this, but with the modification to the find script:

find . -name '*.hs' -exec sh -c 'echo "" >> {}' \;

assigned to @mpickering

changed milestone to %9.4.1

added compiler perf label

The good news is that subsequent reloads of normal packages doesn't leak excessive memory. The bad news is that if your package has module loops that seems to elicit quite bad leaks.

mentioned in merge request !6697 (closed)

Fixed in !6697 (closed)

closed