GHC 8.6 bundles libraries that don't correspond to Hackage releases

changed weight to 10

process-1.6.3.0 is also affected:

The commit shipped with GHC 8.6 is https://gitlab.haskell.org/ghc/packages/process/tree/36a3ad577e31e8c3336c7464b252fc2c9b01a20c
The Hackage commit for 1.6.3.0 is ghc/packages/process@7c0b5814

changed title from GHC 8.6's bundled version of transformers doesn't correspond to a Hackage release to GHC 8.6 bundles libraries that don't correspond to Hackage releases

Sigh, this is quite bad. We really need to rule this sort of thing out systematically from the release process. I'll work on this.

In the meantime I have bumped process to 1.6.3.0 and transformers to 0.5.5.0 on ghc-8.6.

We should be careful about transformers, since there are actually API-relevant code changes that were introduced in commit ghc/packages/transformers@80557845. In particular, I'm pretty sure that contravariant-1.5 requires these changes to compile (at least, angerman reported an error when he tried to build contravariant-1.5 with the Hackage version of transformers-0.5.5.0).

mentioned in issue #16200

So I've come up with a plan which I think should work. I'll try to whip it up today and see how it performs. I'll need the produces artifacts anyway to get this resolved.

For each of the (external) libraries, we'll run cabal new-sdist (1) Then we'll pull the same one from hackage with cabal get --pristine. (2) And finally run a diff over the unpacked one from (1) and (2).

This should yield an empty set.

I'll try using this approach to generate me the sufficient patches for the packages that are in ghc and for which hackage has different source.

Alright, here we are (for ghc-8.6.3-release)

-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 Win32-2.6.1.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 array-0.5.3.0.patch
-rw-r--r--   1 angerman  staff   642B Jan 18 10:00 binary-0.8.6.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 bytestring-0.10.8.2.patch
-rw-r--r--   1 angerman  staff    70B Jan 18 10:00 containers-0.6.0.1.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 deepseq-1.4.4.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 directory-1.3.3.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 filepath-1.4.2.1.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 haskeline-0.7.4.3.patch
-rw-r--r--   1 angerman  staff   675B Jan 18 10:00 hpc-0.6.0.3.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 mtl-2.2.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 parallel-3.2.2.0.patch
-rw-r--r--   1 angerman  staff   1.4K Jan 18 10:00 parsec-3.1.13.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 pretty-1.1.3.6.patch
-rw-r--r--   1 angerman  staff   6.0K Jan 18 10:00 process-1.6.3.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 stm-2.5.0.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 terminfo-0.4.1.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 text-1.2.3.1.patch
-rw-r--r--   1 angerman  staff   5.4K Jan 18 10:00 time-1.8.0.2.patch
-rw-r--r--   1 angerman  staff    20K Jan 18 10:00 transformers-0.5.5.0.patch
-rw-r--r--   1 angerman  staff   4.8K Jan 18 10:00 unix-2.7.2.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 xhtml-3000.2.2.1.patch

Attached file verify-packages.sh (download).

verify-packages.sh

The verify-packages.sh will generate the cabal sdists, pull the corresponding package from hackage, and diff them. Giving you the output above in the end. Ideally they should all be 0B in size.

Replying to [ticket:16199#comment:167599 angerman]:

For each of the (external) libraries, we'll run cabal new-sdist (1) Then we'll pull the same one from hackage with cabal get --pristine. (2) And finally run a diff over the unpacked one from (1) and (2).

This should yield an empty set.

Fwiw, we can't guarantee this in general, nor do we need to.

There is no requirement that you can actually reinstall a boot package release bundled with GHC. There's no need for that, and there's packages that inevitably fall into this category. So tooling needs to be able to cope with that anyway; and in fact Cabal does.

That being said, the only invariant that actually matters and the only one I care about very strongly is that of **consistent API versioning**:

*If a package foo-1.2.3 is bundled with GHC then if-and-only-if there is a valid build-plan (meeting all involved version constraints), then the foo-1.2.3 from Hackage re-synthesized via Cabal must not to be distinguishable at the API level from the GHC pkg-db pre-installed one. In other words, foo-1.2.3 shall uniquely determine the observable API exposed of that package.**

A very prominent example was the mtl package that shipped with GHC 8.4.1 which broke the API versioning and caused real-world problems which (see https://github.com/haskell-hvr/HsYAML/issues/1 which required some very awkward CPP -- but in general this kind of problem is not always workaroundable by CPP! We must be able to trust the API versioning or we'd have to give up on version numbers altogether)

Anything else, like diverging testsuites/benchmark components or other deltas which have no effect on the exposed API are of very little concern to me.

Avoiding divergences would require enough time between the final release and the preceding release of a GHC last-beta or release-candidate after which boot package maintainers can be encouraged to publish their package release candidates on Hackage to the primary index (and I have to stress: "and no sooner!", as it's strongly discouraged to make releases on Hackage which target or advertise support for unreleased dependencies or tooling, or causing avoidable uploads to Hackage as is clearly specified in the Hackage upload guidelines), after we have ensured (in topological dep ordering) that the package candidates are sound and in sync with the artifacts included in the GHC release, and also fix any minor and not so minor issues detected in the process.

or spelled out differently, the protocol for finalising boot libraries would be more or less like

release a last-beta/release-candidate
go over all boot libraries in topological sorted order and ensure their exposed API matched the Hackage release candidate; ensure that the compatibility advertised via version constraints is accurate; if needed, coordinate fixups with the maintainer
restart 2. in case there's not-so-minor changes which might cause an avalanche effect on their rev-deps
if there were fixups needed, make sure to ./validate again;
when it's clear that the release-candidate has no more issues (otherwise go back to 1.) and we're going to cut the final release; encourage package maintainers to publish their latest Hackage package release candidate (which we verified to match the one contained in GHC's release) to the main index (i.e. press that "publish" button in the Hackage release candidate UI)

I don't know whether this is just regular cabal hell or an instance of this problem, but running:

cabal new-install parsec
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: ghc-8.6.3/installed-8.6... (user goal)
[__1] next goal: process (user goal)
[__1] rejecting: process-1.6.5.0 (conflict: ghc =>
process==1.6.3.0/installed-1.6...)
[__1] rejecting: process-1.6.4.0, process-1.6.3.0/installed-1.6...,
process-1.6.3.0, process-1.6.2.0, process-1.6.1.0, process-1.6.0.0,
process-1.5.0.0, process-1.4.3.0, process-1.4.2.0, process-1.4.1.0,
process-1.4.0.0, process-1.3.0.0, process-1.2.3.0, process-1.2.2.0,
process-1.2.1.0, process-1.2.0.0, process-1.1.0.2, process-1.1.0.1,
process-1.1.0.0, process-1.0.1.5, process-1.0.1.4, process-1.0.1.3,
process-1.0.1.2, process-1.0.1.1, process-1.0.0.0 (constraint from user target
requires ==1.6.5.0)
[__1] fail (backjumping, conflict set: ghc, process)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: process, ghc

If an instance of this problem, then I'd recommend an 8.6.4...

!139 (merged) was merged. Should we close this ticket, or would it behoove us to keep it open until we can verify that 8.6.4 fixes this issue?

closed

changed milestone to %8.6.4

Let's close it.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

It would be great to get 8.6.4 out, if only to fix this bug, which is a real productivity killer as I seem to need to empty my .ghc and .cabal directories and reinstall the libraries needed for whatever my project using to get going. Unless someone knows of a better workaround.

Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: directory-1.3.3.2 (user goal)
[__1] next goal: ghc (user goal)
[__1] rejecting: ghc-8.6.3/installed-8.6... (conflict: directory==1.3.3.2, ghc
=> directory==1.3.3.0/installed-1.3...)
[__1] rejecting: ghc-8.6.1, ghc-8.4.3, ghc-8.4.1, ghc-8.2.2, ghc-8.2.1
(constraint from user target requires ==8.6.3)
[__1] fail (backjumping, conflict set: directory, ghc)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: directory, ghc

mentioned in commit 09120c59

mentioned in commit 6e15ca54

mentioned in commit 9fbcfb97

mentioned in commit 1e4f0a9a

mentioned in merge request !689 (merged)

Transformers bumped in 8.8 via c0661417.

mentioned in commit c0661417

added Phighest label

Trac field	Value
Version	8.6.3
Type	Bug
TypeOfFailure	OtherFailure
Priority	highest
Resolution	Unresolved
Component	libraries (other)
Test case
Differential revisions
BlockedBy
Related
Blocking
CC	angerman, bgamari
Operating system
Architecture

GHC 8.6 bundles libraries that don't correspond to Hackage releases

Child items 0

Activity