ghci leaks memory

The following script spawns ghci, and that spawned ghci then goes on to leak memory:

import Control.Concurrent
import Control.Monad
import System.IO
import System.Process

main = do
    (Just hin, Nothing, Nothing, pid) <- createProcess (proc "ghci" ["+RTS","-S"]){std_in=CreatePipe}
    forever $ do
        threadDelay 100000 -- 0.1s
        hPutStrLn hin "\"this is a test of outputting stuff\""
        hFlush hin

This script just writes a string to GHCi, which then echos it back. The +RTS -S is useful to watch the live memory tick up in realtime, but it leaks without it, and the leak can be seen in process explorer (87Mb to 700Mb over about 30 minutes).

While repeatedly writing commands may not be a standard usage of ghci, it is when driven by tools such as ghcid (https://hackage.haskell.org/package/ghcid) and other IDE-like uses.

Trac metadata

Trac field	Value
Version	8.2.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	GHCi
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

changed weight to 5

added GHCi Tbug Trac import labels

Trac metadata

Trac field	Value
CC	- → ndmitchell@gmail.com

Hmm I believe this is not Windows specific, peaking into the heap it seems to be all PAPs and closures so I believe it's a generic GHCi bug, where a reference is being held to the previous repl results and so preventing GC.

I'll find some time to take a look at it, thanks for the report!

I can confirm that this occurs in GHCi, Intero, and HIE on Mac OSX. Someone suggested :set +r could solve this, but it doesn't seem to help. However, :set -fobject-code *seems* to either slow-down the memory leak or stop it altogether.

Other reports of GHCi leaks include #12848 (closed) https://github.com/commercialhaskell/intero/issues/84. The first suggests that loading code leaks memory, which might relate to ticket:14336#comment:145326.

Trac metadata

Trac field	Value
CC	alanz, mnislaih, ndmitchell@gmail.com, saurabhnanda → alanz, mnislaih, ndmitchell@gmail.com, pggiarrusso, saurabhnanda

I have proposed this as an idea for GSoC / HSoC at https://github.com/haskell-org/summer-of-haskell/pull/15/files

If anyone is interesting in mentoring students working on fixing this space leak, please participate in the discussion at https://github.com/haskell-org/summer-of-haskell/issues/29

Just to be clear, I should be looking at the third column increasing? It starts out for me at around 8,569,448 and gradually increases up to 13,360,528 after running 1000 iterations.

Other things I tried.

1. Not creating it bindings as one was created for each line. This didn't seem to make a big difference for small examples but does for big examples. (Max residency 12mb (unmod) vs 13mb (mod))
2. Not running any Haskell code, so changing the string line to `:set -XTypeApplications", the live column still increased but more slowly. (Up to 11219144 after 1000 iterations).
3. Running with :set -fobject-code, seemed to make no difference or make things slightly worse (14051160) (Just once at the start of the loop).
4. Increasing the size of the string (by 100x) makes things much worse 62235312 but not 100x worse, just 6x worse, maximum residency 55mb. BUT, running with my modified compiler which doesn't generate it bindings, only 13mb max residency.

Anyway I don't really know what I am looking for so if someone could point out which number I should be looking at and what it means for my hardware averse brain it would be useful.

Here is the patch I used for no_it\. https://gist.github.com/78366267566cfe8cf85101fc11b169ed

30000 iterations with a short string is about the same max residency with/without no_it. (155mb vs 149mb)

30000 iterations of :set -XTypeApplications only leads to 16mb of maximum residency so perhaps there is no serious leak there. (6mb with 1000 iterations)

The end thing that matters is process memory usage, and given enough time (half an hour or so) that clearly increases, but not linearly (thanks to effects like GC). The use of the RTS columns just gives you an easy way to spot it faster. The columns for the RTS stats are:

    Alloc    Copied     Live     GC     GC      TOT      TOT  Page Flts
    bytes     bytes     bytes   user   elap     user     elap

Of those, Live bytes is the one I was watching (3rd one along)

Doing 1000 iterations with a very big string (approx 20000 chars) results in 47mb max residency with no_it. Doing 1000 iterations with a vanilla compiler causes my computer to oom, 100 iterations requires 488mb

• 100 - no_it 42mb
• 100 - vanilla 488mb
• 1000 - no_it 47mb
• 1000 - vanilla oom

I put the patch up at https://phabricator.haskell.org/D4299 if anyone wants to try it.

mentioned in commit 41afbb3f

changed milestone to %8.6.1

Can someone verify whether there are any leaks remaining after the above patch?

I think there still are leaks but at a much slower rate. I also suspect module reloading still causes big leaks.

Passing -fno-it causes () to be printed to stdout vastly more often. Did this also break that bit? Looking at the section on GHCi Plans I note that the no-it branch is missing the it::() check in step A.

I reasoned that it didn't matter at all if ghci was being invoked programatically. In fact, it is more consistent to print out () rather than omit it.

This was because it wasn't very convenient to insert this check in the no-it version as it worked before by checking if the type of it was ().

Is it breaking your program?

Yes, it breaks my program - but I can work around it - although the workaround is a bit grim.

Anything that wraps ghci into a program that users are still expected to interact with is going to start having weird special cases to undo these changes. For my specific case of ghcid I can remove the () when ghcid itself sends a command in (no real problem), but I also allow the user to interact with the underlying ghci (via the --test flag), and there the surprising change in behaviour is going to be unpleasant and hard to avoid without just filtering out () lines, some of which might be deliberate.

Any chance of getting the behaviour to omit () added back?

Any chance of getting the behaviour to omit () added back?

This doesn't seem to have much to do with the Description!

I have no idea what the old behaviour is, what the new behaviour is, or which patch made the change that you object to.

Would you like give a short ab-initio specification of the change you want? It does not sound controversial or hard to implement. Maybe you can offer a patch too?

Old behaviour: ghci omitted printing out () for things of type ().

New behaviour: ghci, when passed -no-it (which is the solution to memory leaks and introduced in this patch) always prints out ().

Desired behaviour: Have -no-it not print out ().

OK. Sounds plausible. Doesn't sound hard. Can anyone offer a patch?

changed milestone to %8.8.1

These won't be addressed in GHC 8.6.

closed

If this ever makes it into a released GHC (which it will with GHC 8.6) then I need to adjust ghcid to cope with it. Once I've done that, changing the behaviour for a future release has very little benefit to me. As a result, closing this ticket with Fixed as the initial ticket was fixed.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

added runtime perf label

added Pnormal label