Opened 16 months ago

Closed 8 months ago

#7500 closed bug (fixed)

GHC: internal error: getMBlock: mmap: Operation not permitted

Reported by: guest Owned by:
Priority: normal Milestone: 7.8.1
Component: Compiler Version: 7.4.1
Keywords: Cc:
Operating System: Linux Architecture: Unknown/Multiple
Type of failure: Runtime crash Difficulty: Unknown
Test Case: Blocked By:
Blocking: Related Tickets:

Description

Trying to calculate partition of a number using generating fuction with memoization, like this:

#!/usr/bin/runhaskell

part'' :: Int -> Integer
part'' = ( map part' [0..] !!)

part' :: Int -> Integer
part' 0 = 1
part' 1 = 1
part' n =  
    sum $ map (\k ->
    let kk = fromInteger(k); k1 = floor(1/2*kk*(3*kk-1)); k2 = floor(1/2*kk*(3*kk+1)) in
    (-1)^(k+1) * ( (part (n-k1)) + (part (n-k2))) ) [1..toInteger(n)]

part n = 
    if n < 0 then 0 else part'' n

main = print $ part 20000

Then i watch how memory usage grows up to 2G (RES) and program stops:

./bug.hs
bug.hs: internal error: getMBlock: mmap: Operation not permitted
    (GHC version 7.4.1 for i386_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportab

The kernel is compiled with PAE, and machine has 8G of RAM

Attachments (1)

0001-Treat-EPERM-error-from-mmap-as-an-out-of-memory-cond.patch (1.6 KB) - added by rwbarton 8 months ago.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 16 months ago by ezyang

  • Status changed from new to infoneeded

Returning EPERM from mmap when PROT_EXEC is not specified should be impossible on Linux. What is your kernel version?

comment:2 Changed 16 months ago by igloo

  • Difficulty set to Unknown
  • Resolution set to invalid
  • Status changed from infoneeded to closed

I don't think there's a bug here: on a 32bit platform, a single process is still limited to 2G of memory.

If you still think there's something wrong, please reopen and clarify what behaviour you were expecting.

comment:3 Changed 15 months ago by simonmar

  • Resolution invalid deleted
  • Status changed from closed to new

If this is a simple case of running out of memory, then we ought to emit the correct error message rather than a panic.

comment:4 Changed 12 months ago by igloo

  • Milestone set to 7.8.1

comment:5 Changed 8 months ago by rwbarton

I was able to reproduce this on a 32-bit Linux system by setting mmap_min_addr to 2MB (sudo /sbin/sysctl -w vm.mmap_min_addr=$((2*1048576))). 1MB was not small enough.

mmap_min_addr is a security measure that forbids a user program from mapping pages at or near address 0. (This eliminates an easy mechanism for exploiting kernel NULL (or near-NULL) dereferences: the running program's address space is still mapped when in kernel mode, so an attacker can map a page at 0, trigger a NULL dereference in the kernel and trick the kernel into doing something bad rather than oopsing.)

Based on strace output, it seems that when the hint address to mmap is near mmap_min_addr and there is not enough room left nearby above mmap_min_addr, Linux might either return EPERM or give you a totally different memory block. So this is not exactly a simple case of running out of memory. It's likely that when we receive an EPERM from mmap with a hint address, then falling back to no hint address might give us another block. Then we could report an out-of-memory error if the mmap with no hint address also returned EPERM (though it might just return ENOMEM, instead).

However, in my testing, the test program in the report was able to allocate 3068 MB of virtual address space when mmap_min_addr was not an issue (<= 1MB) and 2929 MB when it got EPERM due to mmap_min_addr (2MB), so there was only 139 MB of virtual address space left that it could have scrounged up with this strategy. I think that simply treating EPERM as an out-of-memory error would be an acceptable fix.

comment:6 Changed 8 months ago by rwbarton

  • Status changed from new to patch

I dug into this some more. In the kernel I have installed (Debian 2.6.32-5-686), mmap selects an address for the new mapping based on the hint address and the process's current mappings, but without regard to mmap_min_addr. Then, if the address selected was less than mmap_min_addr, mmap may return EPERM, depending on the SELinux configuration. I believe the same applies to at least all 32-bit x86 Linux versions.

In my kernel, it doesn't seem that trying to mmap again with no hint address can ever help Linux find the unmapped areas above mmap_min_addr. I'm not sure whether this holds in modern Linux, which has an entirely new algorithm for finding unmapped areas. But in any case, it is easy enough to try a second mmap with no hint address.

I've attached a patch which, on Linux, tries to do a second mmap if it receives EPERM, and treats EPERM as an out-of-memory condition. With the patch, the example program allocates 2929 MB and then exits with the message

part.hs: out of memory (requested 1048576 bytes)

comment:7 Changed 8 months ago by Austin Seipp <aseipp@…>

In acb91b920ebac992c52594adf605b2caf98ab4c0/ghc:

Treat EPERM error from mmap as an OOM (#7500)

Linux can give back EPERM from an mmap call when a user program attempts
to map pages near `mmap_min_addr`, which is a kernel security measure to
prevent people from mapping pages at address 0. We may do this when we
hint to mmap what address to map the pages to.

However, it's theoretically possible we're not actually out of memory -
we could have continuously mapped pages at some other place far away
from `mmap_min_addr` and succeeded instead. So as an added precaution,
if mmap for a given addr gives us EPERM, we'll also attempt to map
*again*, but without the address hint. Maybe the kernel can do the right
thing.

However, while testing #7500, the amount of free address space we could
have otherwise used only turns out to be about 139MB. Which isn't really
a lot. So, given that, we *also* otherwise treat EPERM as an out of
memory error.

This fixes #7500.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

comment:8 Changed 8 months ago by thoughtpolice

  • Resolution set to fixed
  • Status changed from patch to closed

Merged. Thanks Reid!

Note: See TracTickets for help on using tickets.