Opened 2 years ago

Closed 23 months ago

#7726 closed bug (worksforme)

unexpected out of memory error on FreeBSD

Reported by: nejstastnejsistene Owned by: pgj
Priority: normal Milestone: 7.8.1
Component: Runtime System Version: 7.4.2
Keywords: Cc:
Operating System: FreeBSD Architecture: x86
Type of failure: Runtime crash Test Case:
Blocked By: Blocking:
Related Tickets: Differential Revisions:

Description

I wrote a daemon that periodically scrapes information from a webpage that crashes after about 20 hours with the error: "out of memory (requested 1048576 bytes)". The program runs fine on my Ubuntu machine, but always crashes on FreeBSD even though there is plenty of memory to spare/memory stays constant/etc.

I boiled it down to the following code, which consistently crashes after about 4000-6000 iterations when compiled with -O2, and about 35000 without.

import Data.Conduit
import Data.Conduit.List
import qualified Data.ByteString as B
import Network.HTTP.Conduit

main :: IO ()
main = do
    manager <- newManager def
    loop manager 1

loop :: Manager -> Int -> IO ()
loop manager i = do
    putStrLn $ show i
    request <- parseUrl "http://localhost:8000/courselist.html"
    html <- runResourceT $ do
        response <- http request manager
        fmap B.concat $ responseBody response $$+- consume
    loop manager $ i + 1

For debugging, I served a copy of of a typical html sample from localhost using "python -m SimpleHTTPServer". The html file is attached.

Sample Output:

...
35361
35362
35363
35364
35365
main: out of memory (requested 1048576 bytes)

Other information:

$ uname -a
FreeBSD oriskova 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243826: Tue Dec  4 06:55:39 UTC 2012     [email protected]:/usr/obj/usr/src/sys/GENERIC  i386
$ gcc -v
Using built-in specs.
Target: i386-undermydesk-freebsd
Configured with: FreeBSD/i386 system compiler
Thread model: posix[[BR]]
gcc version 4.2.1 20070831 patched [FreeBSD]

I wasn't sure if I should submit this or not since it seemed fairly specific to http-conduit, but the wiki said to go ahead and submit runtime errors, so that's what I did. This is my first time ever submitting a bug report and I tried my best to follow all the instructions at http://hackage.haskell.org/trac/ghc/wiki/ReportABug, but please tell me if there is something I'm doing wrong.

Change History (6)

comment:1 Changed 2 years ago by nejstastnejsistene

I couldn't add the attachment because the file is too big. The html I was going to upload comes from here: http://courselist.wm.edu/wmcourseschedule/courseinfo/searchresults?term_code=201320&status=0&attr=0&levl=0&term_subj=0&sort=crn_key&order=asc

comment:2 Changed 2 years ago by igloo

  • difficulty set to Unknown
  • Milestone set to 7.8.1
  • Owner set to pgj

Thanks for the report. pgj, could you see if you are able to reproduce this please?

comment:3 Changed 2 years ago by pgj

I could not reproduce this problem on FreeBSD/i386 9.1-RELEASE. I followed the instructions from the submitter but the attached source did not cause any memory leak for me -- the memory usage stays constant for me even after 150,000 iterations when compiled with -O2.

Here are my specs:

$ ghc --info
 [("Project name","The Glorious Glasgow Haskell Compilation System")
 ,("GCC extra via C opts"," -fwrapv")
 ,("C compiler command","gcc46")
 ,("C compiler flags"," -fno-stack-protector  -Wl,--hash-size=31 -Wl,--reduce-memory
 ,("ar command","/usr/bin/ar")
 ,("ar flags","clqs")
 ,("ar supports at file","NO")
 ,("touch command","touch")
 ,("dllwrap command","/bin/false")
 ,("windres command","/bin/false")
 ,("perl command","/usr/bin/perl")
 ,("target os","OSFreeBSD")
 ,("target arch","ArchX86")
 ,("target word size","4")
 ,("target has GNU nonexec stack","True")
 ,("target has subsections via symbols","False")
 ,("Project version","7.4.2")
 ,("Booter version","7.4.1")
 ,("Stage","2")
 ,("Build platform","i386-portbld-freebsd")
 ,("Host platform","i386-portbld-freebsd")
 ,("Target platform","i386-portbld-freebsd")
 ,("Have interpreter","YES")
 ,("Object splitting supported","YES")
 ,("Have native code generator","YES")
 ,("Support SMP","YES")
 ,("Unregisterised","NO")
 ,("Tables next to code","YES")
 ,("RTS ways","l debug  thr thr_debug thr_l thr_p  dyn debug_dyn thr_dyn thr_debug_d
 ,("Leading underscore","NO")
 ,("Debug on","False")
 ,("LibDir","/usr/local/lib/ghc-7.4.2")
 ,("Global Package DB","/usr/local/lib/ghc-7.4.2/package.conf.d")
 ,("Gcc Linker flags","[\"-Wl,--hash-size=31\",\"-Wl,--reduce-memory-overheads\"]")
 ,("Ld Linker flags","[\"--hash-size=31\",\"--reduce-memory-overheads\"]")
 ]

There you can see I use lang/ghc from the Ports Collection that is built to invoke GCC 4.6 (perhaps this is the problem?):

$ gcc46 -v
Using built-in specs.
COLLECT_GCC=gcc46
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc46/gcc/i386-portbld-freebsd9.0/4.6.3/lto-wrapper
Target: i386-portbld-freebsd9.0
Configured with: ./../gcc-4.6.3/configure --disable-bootstrap --disable-nls --libdir=/usr/local/lib/gcc46 --libexecdir=/usr/local/libexec/gcc46 --program-suffix=46 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc46/include/c++/ --with-ld=/usr/local/bin/ld --with-libiconv-prefix=/usr/local --with-pkgversion='FreeBSD Ports Collection' --with-system-zlib --enable-languages=c,c++,objc,fortran,java --prefix=/usr/local --mandir=/usr/local/man --infodir=/usr/local/info/gcc46 --build=i386-portbld-freebsd9.0
Thread model: posix
gcc version 4.6.3 (FreeBSD Ports Collection)

I installed http-conduit version 1.9.0 from the Ports Collection (www/hs-http-conduit) to build your test program.

comment:4 Changed 23 months ago by igloo

  • Status changed from new to infoneeded

Thanks for trying pgj.

It's possible that the problem was caused by an old version of e.g. gcc or http-conduit. It's also possible that if it is a GHC bug then it's already been fixed in 7.6 or HEAD.

nejstastnejsistene, are you able to test whether newer versions fix it please?

comment:5 Changed 23 months ago by nejstastnejsistene

Unfortunately I no longer have access to the FreeBSD system I was using. Sorry!

comment:6 Changed 23 months ago by igloo

  • Resolution set to worksforme
  • Status changed from infoneeded to closed

OK, thanks for getting back to us. I think we will have to optimistically assume that this bug has been fixed, then, as we have no way to reproduce it.

Note: See TracTickets for help on using tickets.