Opened 7 months ago

Last modified 29 hours ago

#14381 new bug

Consider making ghc-pkg fill in abi-depends based on depends

Reported by: ezyang Owned by: tdammers
Priority: highest Milestone: 8.4.3
Component: ghc-pkg Version: 8.2.1
Keywords: Cc: ntc2
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D4159
Wiki Page:

Description

In GHC 8.2, we introduced abi-depends to solve #12485. Following the same pattern as all ghc-pkg fields, this field is to be fllled by whoever is performing the registration.

I now suspect designing it this way was a mistake. In https://github.com/haskell/cabal/issues/4728 we have a bug where Cabal is writing nonsense data for abi-depends, ghc-pkg isn't noticing it, and GHC is rejecting the package (with a "shadow" warning) when it gets to the end. The problem is the Cabal aggressively caches the contents of the package database (ostensibly because it is expensive to query ghc-pkg); this means that it is easy to get into a situation where its understanding of the ABIs of its dependencies is out-of-date (because it is not re-reading the database in order to get newer information).

The insult to the injury is, in most cases, ghc-pkg already knows what the ABIs are supposed to be: they're whatever the ABIs of the packages pointed at by 'depends' already in the database are. So ghc-pkg could have computed the abi dependency itself, and prevented this stale data situation from ever happening. This sounds quite attractive to me.

What do people think? Here is one possible proposal (but it is just one in the space):

  • ghc-pkg will be modified to ignore the abi-depends field (perhaps with a warning), to prevent itself from being poisoned by buggy versions of Cabal which are giving bad ABI information
  • Instead, ghc-pkg generates the abi-depends field by looking up dependency IDs from the database. If an ID is not found, it omits the dep from abi-depends (this is equivalent to suspending ABI checking in GHC, so this won't break anything; it will just make ABI checking less robust)
  • Possibly, introduce a new "virtual" field, which can be used to override ghc-pkg default

Change History (14)

comment:1 Changed 7 months ago by thoughtpolice

Milestone: 8.2.2
Owner: set to thoughtpolice

This issue is causing us a large amount of pain at work, and we're currently blocked on a major upgrade to GHC 8.2.1 because of it. So I'm going to be taking a look at fixing this later this week (at least "Part 1" and "Part 2" of the above proposal).

For our case, we're using Nix to control and keep everything working, including GHC -- so we can limp by with approaches that aren't fully upstream-worthy for now, it's easy enough to apply a temporary patch. (I'll of course see it through until it lands in HEAD, just a fore-warning on immediate time constraints.)

Ben, I'm tentatively marking this as slated for 8.2.2, although admittedly I don't know what its current schedule looks like. If it doesn't make it this isn't world-ending for us, but I imagine it would make several people happy anyway. Feel free to push it out if needed.

comment:2 Changed 7 months ago by bgamari

Milestone: 8.2.28.4.1

It seems unlikely that this will happen for 8.2.

comment:3 Changed 7 months ago by thoughtpolice

Differential Rev(s): Phab:D4159
Milestone: 8.4.18.2.3

It'd be nice to get this into the 8.2 branch and the backport is fairly easy; so if there's an 8.2.3, I'd like to queue this patch for its release.

comment:4 Changed 6 months ago by ntc2

Cc: ntc2 added

comment:5 Changed 4 months ago by juhpetersen

I am going to try this patch on 8.2.2, for Fedora 28 development.

comment:6 Changed 4 months ago by juhpetersen

So far seems to be working great on 8.2.2, thanks!

comment:7 Changed 3 months ago by bgamari

Milestone: 8.2.38.4.2
Priority: highhighest

I have encountered this on quite a few occasions. Bumping priority.

comment:8 Changed 10 days ago by bgamari

Owner: changed from thoughtpolice to tdammers

Tobias will have a look at this.

comment:9 Changed 3 days ago by Ben Gamari <ben@…>

In 1cdc14f9/ghc:

ghc-pkg: recompute `abi-depends` for updated packages

See `Note [Recompute abi-depends]` for more information.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Test Plan: `./validate`

Reviewers: bgamari, ezyang

Reviewed By: bgamari

Subscribers: tdammers, juhp, carter, alexbiehl, shlevy, cocreature,
rwbarton, thomie

GHC Trac Issues: #14381

Differential Revision: https://phabricator.haskell.org/D4159

comment:10 Changed 2 days ago by tdammers

OK, I have a patch incoming that should do the trick.

One issue though is that there is a theoretical possibility of the same package database being mentioned multiple times; this would sabotage the "get packages below the current one in the stack" logic, because right now, two package databases are considered "the same" iff their locations are equal, and the "get packages below this one in the stack" logic consists of popping all package db's off the stack that are not the same as the "current" one. So if we have a stack like this one:

[ "foo/bar"
, "baz/quux"
, "foo/bar"
, "pizza/olives"
]

...and we're trying to get all the package databases below the second occurrence of "foo/bar", we would want to get just ["foo/bar", "pizza/olives"], but we will in fact get the entire stack.

Question is whether this is ever going to be a problem in practice - I would assume that having the same package database in the stack more than once would pose all sorts of problems anyway.

comment:11 Changed 41 hours ago by tdammers

As discussed in person: this can produce a lot of warnings that may not be very useful in practice.

ghc-pkg is essentially always doing the "right thing" here, overriding declared dependencies with inferred ones, and we are more confident on the inferred ones than the declared ones (the latter essentially being subject to human error).

The way it works now, we will also see a lot of unnecessary warnings: when the declared and inferred dependencies agree, ghc-pkg will still do the overriding and issue a warning, but that warning will be pointless, because we aren't making any effective changes.

Because of this, we will only raise this warning in debug mode, which should also take care of the now-failing tests.

comment:12 Changed 34 hours ago by bgamari

I have reverted the patch in comment:9 on master as the warning caused validation issues. That being said, we will ship it in 8.4.3 to allow distributions to drop their patches.

comment:13 Changed 30 hours ago by juhpetersen

Thanks a lot this sounds good to me.

Do you want to set the milestone to 8.4.3 then?

comment:14 Changed 29 hours ago by bgamari

Milestone: 8.4.28.4.3

Indeed I do. Good catch.

Note: See TracTickets for help on using tickets.