Opened 9 months ago

Closed 8 months ago

#9265 closed feature request (fixed)

Create PackageKey to replace PackageId, including version dependency information

Reported by: ezyang Owned by: ezyang
Priority: high Milestone: 7.10.1
Component: Package system Version: 7.9
Keywords: backpack Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Revisions:

Description (last modified by ezyang)

Multiple installed instances of a package has been a long sought after feature to alleviate Cabal hell, and was the subject of an GSoC project that ultimately never got merged into HEAD.

This ticket is our latest attack on the problem, summarized as: add a version dependency hash to the PackageId (NOT the InstalledPackageId) associated with packages in the database. This approach is distinguished from the prior attempts as follows:

  • We make no attempt to support multiple installations of different ways or ABI-compatible versions of a library (e.g. with/without optimizations.) This corresponds to supporting multiple InstalledPackageIds in a database; in our case, we're supporting multiple PackageId.
  • We do not have to deal with conflicting "instances" in GHC's package resolver: multiple instances can be simultaneously loaded and used in a single compiled program. This is because PackageIds are baked into the exported linker symbols, so different versions will have different names and can peacefully coexist.
  • We do not have to deal with the delicate ordering problem of calculating an ABI hash when configuring a package, which is prior to when we actually know it (ABI hash is only known after compilation), because our hash is based ONLY on dependency resolution choice.
  • In absence of preference for previously installed packages, our Cabal dependency resolution stays exactly the same: Cabal picks versions, and from these choices we calculate the dependency hashes. With preference, we will have to do a little more work.
  • We do not attempt to garbage collect old packages. Because we are not dependent on ABI, there is not an explosion of installed pacakges from package development; new installed packages only occur when version numbers are bumped up, or packages are installed in a combinatorially different fashion.

Change History (10)

comment:1 Changed 9 months ago by ezyang

  • Description modified (diff)

comment:2 Changed 9 months ago by rwbarton

We do not have to deal with conflicting "instances" in GHC's package resolver: multiple instances can be simultaneously loaded and used in a single compiled program. This is because PackageIds are baked into the exported linker symbols, so different versions will have different names and can peacefully coexist.

What about packages that statically link against C libraries? Will we automatically mangle those C symbol names with the package version hash also?

comment:3 Changed 9 months ago by ezyang

No, C libraries won't get mangled; they'll be shared (in much the same way if two different Haskell packages ask for the same C library, they'll get the same one linked in.)

comment:4 Changed 9 months ago by ezyang

  • Priority changed from normal to high

comment:5 Changed 9 months ago by rwbarton

What about packages that include "cbits" where the calling convention (or just semantics) of the C function changes from one version to the next? Won't the linker complain or, worse, just pick one? I guess with dynamic linking it might work.

Admittedly, it could already happen that two unrelated packages use clashing names for their incompatible "cbits" C functions, but it seems rather more likely to occur between one version of a package and the next.

comment:6 Changed 9 months ago by ezyang

OK, the situation for internal cbits is different. While an external library is shared, internal C code is not shared. I think this would only work if the functions defined in the cbits are not exported outside of the library boundary.

comment:7 Changed 9 months ago by thoughtpolice

  • Milestone set to 7.10.1

comment:8 Changed 8 months ago by Edward Z. Yang <ezyang@…>

In 4bebab25e4c9a3bfccc491d4dd13c685629cd1de/ghc:

Rename PackageId to PackageKey, distinguishing it from Cabal's PackageId.

Summary:
Previously, both Cabal and GHC defined the type PackageId, and we expected
them to be roughly equivalent (but represented differently).  This refactoring
separates these two notions.

A package ID is a user-visible identifier; it's the thing you write in a
Cabal file, e.g. containers-0.9.  The components of this ID are semantically
meaningful, and decompose into a package name and a package vrsion.

A package key is an opaque identifier used by GHC to generate linking symbols.
Presently, it just consists of a package name and a package version, but
pursuant to #9265 we are planning to extend it to record other information.
Within a single executable, it uniquely identifies a package.  It is *not* an
InstalledPackageId, as the choice of a package key affects the ABI of a package
(whereas an InstalledPackageId is computed after compilation.)  Cabal computes
a package key for the package and passes it to GHC using -package-name (now
*extremely* misnamed).

As an added bonus, we don't have to worry about shadowing anymore.

As a follow on, we should introduce -current-package-key having the same role as
-package-name, and deprecate the old flag.  This commit is just renaming.

The haddock submodule needed to be updated.

Signed-off-by: Edward Z. Yang <[email protected]>

Test Plan: validate

Reviewers: simonpj, simonmar, hvr, austin

Subscribers: simonmar, relrod, carter

Differential Revision: https://phabricator.haskell.org/D79

Conflicts:
	compiler/main/HscTypes.lhs
	compiler/main/Packages.lhs
	utils/haddock

comment:9 Changed 8 months ago by ezyang

  • Summary changed from Extend PackageId to include version dependency information to Create PackageKey to replace PackageId, including version dependency information

comment:10 Changed 8 months ago by ezyang

  • Resolution set to fixed
  • Status changed from new to closed

We'll leave #9351 as further work in this area.

Note: See TracTickets for help on using tickets.