gcdExtInteger violates assertion

{-# LANGUAGE UnboxedTuples #-}
import GHC.Integer.GMP.Internals

main = let (# _, s #) = gcdExtInteger 2 (2^65 + 1) in print s

fails with

Assertion failed: (sn <= mp_size_abs(xn)), function integer_gmp_gcdext, file libraries/integer-gmp/cbits/wrappers.c, line 316.
Abort trap: 6

It happens because s = -2^64 and does not fit one-limbed BigNat. The implementation of gcdExtInteger x y (https://github.com/ghc/ghc/blob/master/libraries/integer-gmp/src/GHC/Integer/Type.hs#L1392) allocates for s a buffer, equal to size of x (one limb in our case), but according to GMP manual (https://gmplib.org/manual/Number-Theoretic-Functions.html#index-mpz_005fgcdext) it should be equal to size of y (two limbs in our case).

Hopefully, the diff is simple enough for a PR on GitHub (https://github.com/ghc/ghc/pull/163). Otherwise I'll be happy to prepare a patch for Phabricator.

-        s@(MBN# s#) <- newBigNat# (absI# xn#)
+        s@(MBN# s#) <- newBigNat# (absI# yn#)

---

Reopening, because

{-# LANGUAGE UnboxedTuples #-}
import GHC.Integer.GMP.Internals

main = let (# _, s #) = gcdExtInteger (- (2^63 - 1) * 2^63) 0 in print s

fails in GHC 8.6.1 with

Assertion failed: (0 <= gn && gn <= gn0), function integer_gmp_gcdext, file libraries/integer-gmp/cbits/wrappers.c, line 309.
Abort trap: 6

I have not yet understood what is going on.

changed milestone to %8.6.1

changed weight to 5

added Tbug Trac import core libraries labels

changed the description

assigned to @trac-Bodigrim

What's the status here? I see that the change was merged, but I don't see a test. I've tried adding a test, but it's still failing with the same error on my machine. Maybe I'm doing something wrong? Is the plan to release a new version of integer-gmp?

diff --git a/testsuite/tests/lib/integer/integerGmpInternals.hs b/testsuite/tests/lib/integer/integerGmpInternals.hs
index c90df5c90d..e1faf900c7 100644
--- a/testsuite/tests/lib/integer/integerGmpInternals.hs
+++ b/testsuite/tests/lib/integer/integerGmpInternals.hs
@@ -86,6 +86,7 @@ main = do
     print $ gcdExtInteger x (-y)
     print $ gcdExtInteger (-x) y
     print $ gcdExtInteger (-x) (-y)
+    print $ gcdExtInteger 2 (2^65 + 1)  -- see Trac #15350
     print $ powInteger 12345 0
     print $ powInteger 12345 1
     print $ powInteger 12345 30

changed weight to 7

Trac metadata

Trac field	Value
Priority	normal → high

Trac metadata

Trac field	Value
Differential revisions	- → D5042

mentioned in commit c3315921

added 1 deleted label

closed

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

removed 1 deleted label

mentioned in commit 45d5eff8

reopened

changed the description

Trac metadata

Trac field	Value
Version	8.4.3 → 8.6.1
Resolution	ResolvedFixed → Unresolved

added runtime crash label

added Phigh label

Seems it was accidentally fixed in course of !4121 (closed).

closed