Opened 14 months ago

Closed 7 months ago

Last modified 6 months ago

#14381 closed bug (fixed)

Consider making ghc-pkg fill in abi-depends based on depends

Reported by: ezyang Owned by: tdammers
Priority: highest Milestone: 8.4.4
Component: ghc-pkg Version: 8.2.1
Keywords: Cc: ntc2
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D4159 Phab:D4729
Wiki Page:

Description

In GHC 8.2, we introduced abi-depends to solve #12485. Following the same pattern as all ghc-pkg fields, this field is to be fllled by whoever is performing the registration.

I now suspect designing it this way was a mistake. In https://github.com/haskell/cabal/issues/4728 we have a bug where Cabal is writing nonsense data for abi-depends, ghc-pkg isn't noticing it, and GHC is rejecting the package (with a "shadow" warning) when it gets to the end. The problem is the Cabal aggressively caches the contents of the package database (ostensibly because it is expensive to query ghc-pkg); this means that it is easy to get into a situation where its understanding of the ABIs of its dependencies is out-of-date (because it is not re-reading the database in order to get newer information).

The insult to the injury is, in most cases, ghc-pkg already knows what the ABIs are supposed to be: they're whatever the ABIs of the packages pointed at by 'depends' already in the database are. So ghc-pkg could have computed the abi dependency itself, and prevented this stale data situation from ever happening. This sounds quite attractive to me.

What do people think? Here is one possible proposal (but it is just one in the space):

  • ghc-pkg will be modified to ignore the abi-depends field (perhaps with a warning), to prevent itself from being poisoned by buggy versions of Cabal which are giving bad ABI information
  • Instead, ghc-pkg generates the abi-depends field by looking up dependency IDs from the database. If an ID is not found, it omits the dep from abi-depends (this is equivalent to suspending ABI checking in GHC, so this won't break anything; it will just make ABI checking less robust)
  • Possibly, introduce a new "virtual" field, which can be used to override ghc-pkg default

Change History (20)

comment:1 Changed 14 months ago by thoughtpolice

Milestone: 8.2.2
Owner: set to thoughtpolice

This issue is causing us a large amount of pain at work, and we're currently blocked on a major upgrade to GHC 8.2.1 because of it. So I'm going to be taking a look at fixing this later this week (at least "Part 1" and "Part 2" of the above proposal).

For our case, we're using Nix to control and keep everything working, including GHC -- so we can limp by with approaches that aren't fully upstream-worthy for now, it's easy enough to apply a temporary patch. (I'll of course see it through until it lands in HEAD, just a fore-warning on immediate time constraints.)

Ben, I'm tentatively marking this as slated for 8.2.2, although admittedly I don't know what its current schedule looks like. If it doesn't make it this isn't world-ending for us, but I imagine it would make several people happy anyway. Feel free to push it out if needed.

comment:2 Changed 14 months ago by bgamari

Milestone: 8.2.28.4.1

It seems unlikely that this will happen for 8.2.

comment:3 Changed 13 months ago by thoughtpolice

Differential Rev(s): Phab:D4159
Milestone: 8.4.18.2.3

It'd be nice to get this into the 8.2 branch and the backport is fairly easy; so if there's an 8.2.3, I'd like to queue this patch for its release.

comment:4 Changed 13 months ago by ntc2

Cc: ntc2 added

comment:5 Changed 11 months ago by juhpetersen

I am going to try this patch on 8.2.2, for Fedora 28 development.

comment:6 Changed 11 months ago by juhpetersen

So far seems to be working great on 8.2.2, thanks!

comment:7 Changed 10 months ago by bgamari

Milestone: 8.2.38.4.2
Priority: highhighest

I have encountered this on quite a few occasions. Bumping priority.

comment:8 Changed 7 months ago by bgamari

Owner: changed from thoughtpolice to tdammers

Tobias will have a look at this.

comment:9 Changed 7 months ago by Ben Gamari <ben@…>

In 1cdc14f9/ghc:

ghc-pkg: recompute `abi-depends` for updated packages

See `Note [Recompute abi-depends]` for more information.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

Test Plan: `./validate`

Reviewers: bgamari, ezyang

Reviewed By: bgamari

Subscribers: tdammers, juhp, carter, alexbiehl, shlevy, cocreature,
rwbarton, thomie

GHC Trac Issues: #14381

Differential Revision: https://phabricator.haskell.org/D4159

comment:10 Changed 7 months ago by tdammers

OK, I have a patch incoming that should do the trick.

One issue though is that there is a theoretical possibility of the same package database being mentioned multiple times; this would sabotage the "get packages below the current one in the stack" logic, because right now, two package databases are considered "the same" iff their locations are equal, and the "get packages below this one in the stack" logic consists of popping all package db's off the stack that are not the same as the "current" one. So if we have a stack like this one:

[ "foo/bar"
, "baz/quux"
, "foo/bar"
, "pizza/olives"
]

...and we're trying to get all the package databases below the second occurrence of "foo/bar", we would want to get just ["foo/bar", "pizza/olives"], but we will in fact get the entire stack.

Question is whether this is ever going to be a problem in practice - I would assume that having the same package database in the stack more than once would pose all sorts of problems anyway.

comment:11 Changed 7 months ago by tdammers

As discussed in person: this can produce a lot of warnings that may not be very useful in practice.

ghc-pkg is essentially always doing the "right thing" here, overriding declared dependencies with inferred ones, and we are more confident on the inferred ones than the declared ones (the latter essentially being subject to human error).

The way it works now, we will also see a lot of unnecessary warnings: when the declared and inferred dependencies agree, ghc-pkg will still do the overriding and issue a warning, but that warning will be pointless, because we aren't making any effective changes.

Because of this, we will only raise this warning in debug mode, which should also take care of the now-failing tests.

comment:12 Changed 7 months ago by bgamari

I have reverted the patch in comment:9 on master as the warning caused validation issues. That being said, we will ship it in 8.4.3 to allow distributions to drop their patches.

comment:13 Changed 7 months ago by juhpetersen

Thanks a lot this sounds good to me.

Do you want to set the milestone to 8.4.3 then?

comment:14 Changed 7 months ago by bgamari

Milestone: 8.4.28.4.3

Indeed I do. Good catch.

comment:15 Changed 7 months ago by tdammers

Differential Rev(s): Phab:D4159Phab:D4159 Phab:D4729
Status: newpatch

comment:16 Changed 7 months ago by bgamari

Phab:D4159 has been merged to GHC 8.4.3 as a stop-gap with 51abb1c88b53e2989a2a8c2939ac4abc04bef194. This may result in somewhat noisy "ignoring (possibly broken) abi-depends field for packages" warnings from the compiler. These can be ignored.

We will merge the more sensible Phab:D4729 to master shortly.

Last edited 7 months ago by bgamari (previous) (diff)

comment:17 Changed 7 months ago by phadej

The warning is indeed noisy, and gives an impression cabal-install (or GHC-8.4.3) is broken.

If there will be 8.4.4, I'd hope it to include the proper fix.

lens master % cabal new-build -w ghc-8.4.3
Build profile: -w ghc-8.4.3 -O1
In order, the following will be built (use -v for more details):
 - cabal-doctest-1.0.6 (lib) (requires build)
 - contravariant-1.4.1 (lib:contravariant) (requires build)
 - exceptions-0.10.0 (lib) (requires build)
 - parallel-3.2.1.1 (lib) (requires build)
 - reflection-2.1.3 (lib) (requires build)
 - semigroups-0.18.4 (lib) (requires build)
 - transformers-base-0.4.5.2 (lib) (requires build)
 - void-0.7.2 (lib) (requires build)
 - distributive-0.5.3 (lib:distributive) (requires build)
 - comonad-5.0.3 (lib:comonad) (requires build)
 - bifunctors-5.5.2 (lib) (requires build)
 - semigroupoids-5.2.2 (lib:semigroupoids) (requires build)
 - profunctors-5.2.2 (lib) (requires build)
 - free-5.0.2 (lib) (requires build)
 - adjunctions-4.4 (lib) (requires build)
 - kan-extensions-5.1 (lib:kan-extensions) (requires build)
 - lens-4.16.1 (lib:lens) (first run)
Configuring contravariant-1.4.1 (all, legacy fallback)...
Configuring cabal-doctest-1.0.6 (lib)...
Configuring parallel-3.2.1.1 (lib)...
Configuring exceptions-0.10.0 (lib)...
Building contravariant-1.4.1 (all, legacy fallback)...
Building exceptions-0.10.0 (lib)...
Building parallel-3.2.1.1 (lib)...
Building cabal-doctest-1.0.6 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring reflection-2.1.3 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring semigroups-0.18.4 (lib)...
Building reflection-2.1.3 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring transformers-base-0.4.5.2 (lib)...
ignoring (possibly broken) abi-depends field for packages
Building semigroups-0.18.4 (lib)...
Configuring void-0.7.2 (lib)...
Building transformers-base-0.4.5.2 (lib)...
Building void-0.7.2 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring distributive-0.5.3 (all, legacy fallback)...
ignoring (possibly broken) abi-depends field for packages
ignoring (possibly broken) abi-depends field for packages
Building distributive-0.5.3 (all, legacy fallback)...
ignoring (possibly broken) abi-depends field for packages
ignoring (possibly broken) abi-depends field for packages
Configuring comonad-5.0.3 (all, legacy fallback)...
Building comonad-5.0.3 (all, legacy fallback)...
ignoring (possibly broken) abi-depends field for packages
Configuring bifunctors-5.5.2 (lib)...
Building bifunctors-5.5.2 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring profunctors-5.2.2 (lib)...
Configuring semigroupoids-5.2.2 (all, legacy fallback)...
Building profunctors-5.2.2 (lib)...
Building semigroupoids-5.2.2 (all, legacy fallback)...
ignoring (possibly broken) abi-depends field for packages
ignoring (possibly broken) abi-depends field for packages
Configuring free-5.0.2 (lib)...
Building free-5.0.2 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring adjunctions-4.4 (lib)...
Building adjunctions-4.4 (lib)...
ignoring (possibly broken) abi-depends field for packages
Configuring kan-extensions-5.1 (all, legacy fallback)...
Building kan-extensions-5.1 (all, legacy fallback)...
ignoring (possibly broken) abi-depends field for packages
Last edited 7 months ago by phadej (previous) (diff)

comment:18 Changed 7 months ago by Ben Gamari <ben@…>

In 1626fe6/ghc:

Handle abi-depends correctly in ghc-pkg

When inferring the correct abi-depends, we now look at all the package
databases in the stack, up to and including the current one, because
these are the ones that the current package can legally depend on. While
doing so, we will issue warnings:

- In verbose mode, we warn about every package that declares
  abi-depends:, whether we actually end up overriding them with the
  inferred ones or not ("possibly broken abi-depends").

- Otherwise, we only warn about packages whose declared abi-depends
  does not match what we inferred ("definitely broken abi-depends").

Reviewers: bgamari

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14381

Differential Revision: https://phabricator.haskell.org/D4729

comment:19 Changed 7 months ago by bgamari

Milestone: 8.4.38.6.1
Resolution: fixed
Status: patchclosed

Fixed correctly in 8.6.1.

comment:20 Changed 6 months ago by bgamari

Milestone: 8.6.18.4.4
Note: See TracTickets for help on using tickets.