#7257 closed bug (fixed)

Regression: pinned memory fragmentation

Reported by: jwlato Owned by: simonmar
Priority: highest Milestone: 7.6.2
Component: Compiler Version: 7.6.1
Keywords: Cc: jwlato@…, tkn.akio@…, bgamari@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Runtime performance bug Difficulty: Unknown
Test Case: perf/should_run/T7257 Blocked By:
Blocking: Related Tickets:

Description

In certain cases, ghc-7.6.1 seems to have much worse memory behavior than 7.4. I have attached a simplified program that I believe demonstrates the problem.

-- compiled with ghc-7.4.2 -rtsopts -O2
./Foo  +RTS -s
200001
   3,549,666,928 bytes allocated in the heap
      55,512,376 bytes copied during GC
     335,555,096 bytes maximum residency (10 sample(s))
       3,787,576 bytes maximum slop
             453 MB total memory in use (0 MB lost due to fragmentation)
-- compiled with ghc-7.6.1 -rtsopts -O2
   2,699,298,272 bytes allocated in the heap
      55,077,544 bytes copied during GC
     327,246,968 bytes maximum residency (11 sample(s))
       3,767,408 bytes maximum slop
             858 MB total memory in use (398 MB lost due to fragmentation)

One of our applications uses 3-4GB more RAM when compiled with ghc-7.6.1 compared to 7.4, all due to memory fragmentation issues.

Attachments (1)

Foo.hs (700 bytes) - added by jwlato 19 months ago.
example

Download all attachments as: .zip

Change History (9)

Changed 19 months ago by jwlato

example

comment:1 Changed 19 months ago by akio

  • Cc tkn.akio@… added

comment:2 Changed 19 months ago by jwlato

Addendum: with ghc-7.6.1, I see the same memory behavior with multiple versions of ByteString? and Data.Vector.Storable, but not Text, Data.Vector, or Data.Vector.Unboxed.

comment:3 Changed 19 months ago by simonmar

  • Difficulty set to Unknown
  • Milestone set to 7.6.2
  • Owner set to simonmar
  • Priority changed from normal to highest

Sorry about this. I know what the problem is, and a fix is on the way. As a bonus, I have made the program go faster by a factor of 10.

comment:4 Changed 19 months ago by marlowsd@…

commit 016fd74d6517512b62b36ff12cdccf2e723a0fb3

Author: Simon Marlow <marlowsd@gmail.com>
Date:   Fri Sep 21 13:18:49 2012 +0100

    Cache the result of countOccupied(gen->large_objects) as gen->n_large_words (#7257)
    
    The program in #7257 was spending 90% of its time counting the live
    data in gen->large_objects.  We already avoid doing this for small
    objects, but in this example the old generation was full of large
    objects (actually pinned ByteStrings).

 includes/rts/storage/GC.h |    1 +
 rts/sm/GC.c               |    5 ++++-
 rts/sm/Storage.c          |    3 ++-
 3 files changed, 7 insertions(+), 2 deletions(-)

comment:5 Changed 19 months ago by simonmar

  • Status changed from new to merge

Please merge the following commits:

commit a68df77ede928e6c7790dacb5925625792a904d3
Author: Simon Marlow <marlowsd@gmail.com>
Date:   Tue Aug 21 11:39:06 2012 +0100

    Reduce fragmentation when using +RTS -H (with or without a size)

commit a8179622f84bbd52e127a9596d2d4a918ca64e0c
Author: Simon Marlow <marlowsd@gmail.com>
Date:   Fri Sep 7 13:36:09 2012 +0100

    Some further tweaks to reduce fragmentation when allocating the nursery


commit 1f5d83648dfda39d999eb8a9e8192339b3eea540
Author: Simon Marlow <marlowsd@gmail.com>
Date:   Fri Sep 21 13:11:22 2012 +0100

    Allow allocNursery() to allocate single blocks (#7257)
    
    Forcing large allocations here can creates serious fragmentation in
    some cases, and since the large allocations are only a small
    optimisation we should allow the nursery to hoover up small blocks
    before allocating large chunks.

commit c19f2e320a575a66d31c83d846ce3cc954c4ab3b
Author: Simon Marlow <marlowsd@gmail.com>
Date:   Fri Sep 21 13:20:37 2012 +0100

    Include pinned memory in the stats for allocated memory
    
    This broke with the changes to the pinned object handling in
    67f4ab7e6b7705a9d617c6109a8c5434ede13cae.

comment:6 Changed 19 months ago by simonmar

  • Test Case set to perf/should_run/T7257

comment:7 Changed 18 months ago by bgamari

  • Cc bgamari@… added
Note: See TracTickets for help on using tickets.