Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#10415 closed bug (invalid)

ForeignPtr touched in FFI wrapper is never discarded

Reported by: Deewiant Owned by:
Priority: normal Milestone:
Component: Compiler Version: 7.10.1
Keywords: Cc:
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Runtime performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

If a ForeignPtr is touched in an FFI wrapper (foreign import ccall "wrapper"), it appears to never be discarded. For example, the program below demonstrates a space leak due to multiple mallocForeignPtrBytes results hanging around forever.

import Control.Concurrent (threadDelay)
import Foreign.ForeignPtr
import Foreign.Marshal.Utils (fillBytes)
import Foreign.Ptr (FunPtr)

foreign import ccall "wrapper" wrap :: IO () -> IO (FunPtr (IO ()))

main :: IO ()
main = flip mapM_ [0..50] $ \_ -> do
   let len = 30 * 2^20
   fp <- mallocForeignPtrBytes len
   withForeignPtr fp $ \p -> fillBytes p 0 len
   _ <- wrap $ touchForeignPtr fp
   threadDelay 100000

Each of the 50 iterations allocates 30 megabytes of memory via mallocForeignPtrBytes. None of those allocations is ever freed, resulting in around 1.5 gigabytes of memory use by the time the program terminates.

(Neither fillBytes nor threadDelay are required to reproduce the issue: they are there to respectively touch the memory so that it's fully allocated by the OS and to slow the program down so that one can view the memory usage gradually increasing e.g. with top.)

Example run, with almost 1.6 gigabytes of memory usage reported by GNU time (the "maxresident" bit):

$ ghc --make asdf.hs
[1 of 1] Compiling Main             ( asdf.hs, asdf.o )
Linking asdf ...
$ /usr/bin/time ./asdf
0.12user 0.54system 0:05.77elapsed 11%CPU (0avgtext+0avgdata 1587300maxresident)k
0inputs+0outputs (0major+138078minor)pagefaults 0swaps

People bitten by this can probably work around it without too much trouble by either:

  1. Simply avoiding the trigger, i.e. touchForeignPtr in a wrapper. In my case, that amounts to changing some functions to take Ptr a instead of ForeignPtr a and moving some withForeignPtr calls around, possibly adding manual touchForeignPtr calls somewhere.
  2. Switching to manual memory management with functions like mallocBytes and free.

But the issue seems rather nasty regardless. (At least it took me a while to pin down.) Hopefully it can be fixed without too much trouble.

Change History (2)

comment:1 Changed 3 years ago by rwbarton

Resolution: invalid
Status: newclosed

The FunPtrs produced by a wrapper are like Ptrs, not ForeignPtrs. The functions they point to aren't and can't be managed by GC, because you can pass one directly to a C function and it can squirrel the pointer away in a C data structure somewhere.

So the wrapped functions in this program live forever, and since they refer to the ForeignPtr fp, the memory managed by that pointer lives forever as well.

You can free the result of a wrapper explicitly with freeHaskellFunPtr. If you add such a call to your program, say, after the threadDelay, then the 30 megabyte object does get freed.

Please reopen if I am missing some subtlety here!

comment:2 Changed 3 years ago by Deewiant

No, you're absolutely right. I suspected that the compiler couldn't reasonably do anything here due to the danger of the wrapper being "squirreled away" as you say. :-) What I thought might be possible is some sort of proof based on things like:

  1. In the example, the wrapper is immediately discarded, so there's no need for it to live at all, really.
  2. Perhaps a "safe" foreign call is only made immediately next to the wrapper allocation and the wrapper is not referenced elsewhere, so the compiler could see where its lifetime ends.

But I didn't realize that the wrappers are functions that require manual freeing! freeHaskellFunPtr is something I was missing entirely. I was looking in the FFI sections of the Haskell 2010 report to see if there was anything like that for freeing wrapper functions, but apparently I should've looked elsewhere too.

Thanks, and sorry for the noise.

Note: See TracTickets for help on using tickets.