Runtime error when allocating lots of memory
In GHCI execute
let powers = 2:map (2^) powers
powers
When calculating the fourth element of the list, you get the following error message:
<interactive>: internal error: getMBlock: mmap: Invalid argument
(GHC version 6.12.1 for i386_unknown_linux)
Please report this as a GHC bug: http://www.haskell.org/ghc/reportabug
If you have trouble reproducing the bug, try substituting an other number for 2 in the code, e.g. I have also got that error message when I defined powers as:
let powers = 4:map (4^) powers
and
let powers = 13:map (13^) powers
No child items are currently assigned. Use child items to break down this issue into smaller parts.
Link issues together to show that they're related or that one is blocking others. Learn more.
Activity
- knyblad changed weight to 5
changed weight to 5
- knyblad added Tbug GHCi Trac import labels
added Tbug GHCi Trac import labels
- Simon Marlow changed milestone to %7.2.1
changed milestone to %7.2.1
- Developer
Here's the relevant bit of code:
if (ret == (void *)-1) { if (errno == ENOMEM || (errno == EINVAL && sizeof(void*)==4 && size >= 0xc0000000)) { // If we request more than 3Gig, then we get EINVAL // instead of ENOMEM (at least on Linux). errorBelch("out of memory (requested %lu bytes)", size); stg_exit(EXIT_FAILURE); } else { barf("getMBlock: mmap: %s", strerror(errno)); } }
Perhaps we're getting
EINVAL
under different circumstances. I didn't manage to reproduce it (yet). - Ian Lynagh <igloo@earth.li> changed the description
changed the description
- Ian Lynagh <igloo@earth.li> changed milestone to %7.4.1
changed milestone to %7.4.1
- Ian Lynagh <igloo@earth.li> changed milestone to %7.6.1
changed milestone to %7.6.1
- Ian Lynagh <igloo@earth.li> changed weight to 3
changed weight to 3
Trac metadata
Trac field Value Priority normal → low - Ian Lynagh <igloo@earth.li> changed milestone to %7.6.2
changed milestone to %7.6.2
I was able to reproduce this (Linux 2.6.32-5-686, GHC 7.4.2). Here is the offending mmap call, from strace:
mmap2(0x8d900000, 0, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 EINVAL (Invalid argument)
- .. which is certainly invalid (from the mmap man page: "EINVAL (since Linux 2.6.12) length was 0.")
I'll try to figure out why it is mmapping 0 bytes. Could be an integer overflow, when
size
is computed inosGetMBlocks
...So I investigated further with gdb, and while I no longer have the exact numbers, gmp was indeed asking for an allocation of more than 4095 MB, resulting in
osGetMBlocks(4096)
and an integer overflow when computingsize
resulting insize = 0
.The fix should be to detect when the computation of
size
will overflow asize_t
, and report an out-of-memory error if this occurs.Attached file
0002-Check-for-integer-overflow-in-osGetMBlocks.patch
(download).Attached file
0004-Extra-paranoid-integer-overflow-check-in-osGetMBlock.patch
(download).The first attached patch checks for integer overflow in the computation of
size
inosGetMBlocks
. With the patch the example program now exits with this message:pow: out of memory (requested 4096 MBlocks)
The second and third patches are extra checks for overflow when converting between integer types in
my_mmap
andosGetMBlocks
. AFAIK it *should* always be the case thatW_
>=nat
and thatsize_t
>=W_
on platforms supported by GHC; but on any system where that is the case, the C compiler can eliminate the tests anyways, so I recommend including both tests.Oops, I see there is a missing
%
in the middle patch.Attached file
0003-Extra-paranoid-integer-overflow-check-in-my_mmap.patch
(download).I reverted 48865521 (and its kin) because unfortunately it causes a validate failure on amd64/Linux, where the conditional in patch #2 (closed) is always false, meaning we will always calculate
size = n * MBLOCK_SIZE
.I'll look into this shortly.
Right, the issue is that on amd64/Linux,
sizeof(nat)
is 4 andsizeof(W_)
is 8, so overflow can never occur, which gcc warns about, and validate builds with-Wextra -Werror
. Annoying.Austin and I discussed this on IRC and here are a couple possible workarounds we came up with.
- Wrap the test in an
#if
that testssizeof(nat)
andsizeof(W_)
to determine whether overflow is possible. Portable, but fiddly. - Disable gcc warnings using
#pragma GCC diagnostic
pragmas around the test, themselves conditionalized to gcc. Less fiddly, but GHC doesn't currently use this method to disable any warnings and I somewhat hesitate to be the first. I also haven't tested that this really works.
- Wrap the test in an
- Developer
I'm fairly sure that casting -1 to an unsigned type (W_) is undefined in C, because the value is outside the range of the type. Why not use
HS_WORD_MAX
? Casting to an unsigned integral type is defined to have "modulo 2^n^" semantics by C99 §6.3.1.3, so casting -1 to any unsigned integral type will result in the largest representable value of that type. But
HS_WORD_MAX
is okay too, and probably easier to read; I just didn't think to look for such a thing inHsFFI.h
. (I find the proliferation of identical typesStgWord
,W_
,HsWord
a little confusing, but I'll get over it.)Reid, any word here - would you like you tweak this patch?
- thoughtpolice changed milestone to %7.10.1
changed milestone to %7.10.1
Moving to 7.10.1.
In commit 29ee739e:
Author: Austin Seipp <> Date: Thu Aug 29 17:30:16 2013 -0500 Revert "Check for integer overflow in osGetMBlocks" This reverts commit 48865521de6638240819b3979edbb3d33401dc8e. Signed-off-by: Austin Seipp <aseipp@pobox.com>
Trac metadata
Trac field Value Component GHCi → Runtime System CC hvr → hvr, simonmar - thoughtpolice removed milestone
removed milestone
Moving to 7.12.1 milestone; if you feel this is an error and should be addressed sooner, please move it back to the 7.10.1 milestone.
Trac metadata
Trac field Value TypeOfFailure OtherFailure → GhciCrash - thoughtpolice changed milestone to %8.0.1
changed milestone to %8.0.1
Milestone renamed
- Ben Gamari added Plow label
added Plow label