Place shared objects in LIBDIR

changed milestone to %8.0.1

changed weight to 5

added Tbug Trac import package system labels

changed the description

The dynamic linker must look in nearly 25 Haskell library directories to locate every system library! This is madness.

Not only that, but just to spell it all out explicitly,

The dynamic linker has to look in those nearly 25 Haskell library directories for each Haskell library, too, making the behavior quadratic in the number of Haskell packages.
This applies not just to ghc itself, but to any executable that uses dynamic Haskell libraries. Some people have programs that use hundreds of different Haskell packages.

I have wondered about this set up in the past. More precisely, I think having the dynamic libraries for different packages in separate directories with the interface files etc of the package makes a lot of sense. I keeps the $LIBDIR tidy.

However, the dynamic linking set up does appear to be rather inefficient and, at least on OS X, it makes relocating GHC distributions very hard. In the Haskell for Mac build, I hence, post-process all executables and dynamic libraries to optimise the linking process. As my ELF is a bit rusty let me explain it in MACH-O terms. I am setting the RPATH in all executables such that it points to $LIBDIR and I set the library name of each dynamic library to include the package directory. For example, for base, we might have base_GDytRqRVSUX7zckgKqJjgw/libHSbase-4.8.1.0-GDytRqRVSUX7zckgKqJjgw-ghc7.10.2.dylib.

I also set RPATH to be relative to @loader_PATH, which gives me a relocatable set of dynamic libraries and GHC executables.

So, for example, here is what base looks like

LC 03: LC_ID_DYLIB              @rpath/base_GDytRqRVSUX7zckgKqJjgw/libHSbase-4.8.1.0-GDytRqRVSUX7zckgKqJjgw-ghc7.10.2.dylib
...
LC 12: LC_LOAD_DYLIB            @rpath/integ_2aU3IZNMF9a7mQ0OzsZ0dS/libHSinteger-gmp-1.0.0.0-2aU3IZNMF9a7mQ0OzsZ0dS-ghc7.10.2.dylib
LC 13: LC_LOAD_DYLIB            @rpath/ghcpr_8TmvWUcS1U1IKHT0levwg3/libHSghc-prim-0.4.0.0-8TmvWUcS1U1IKHT0levwg3-ghc7.10.2.dylib
...
LC 19: LC_RPATH                 @loader_path/..

This avoids the quadratic explosion of the search space, but still keeps the dynamic libraries in the package directories (and $LIBDIR tidier). This is definitely the better set up on OS X. Can't we do something equivalent on Linux?

Trac metadata

Trac field	Value
CC	ezyang, rwbarton, thomie, trommler → Trommler, chak, ezyang, rwbarton, thomie

I think the real issue is that Haskell's libraries are all in different directories and so the dynamic linker must read at least one directory and one file for each Haskell dynamic library.

The quadratic time required to find a library seems to be small compared to the disk access to read all directories in the RPATH from the hard disk. The second read of a directory would most of the time hit the file system cache.

I tried this experiment on my x86_64 Linux machine (still using spinning hard drives) with a dynamically linked GHC:

$ time ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

real    0m4.805s
user    0m0.044s
sys     0m0.080s
$ time ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

real    0m0.048s
user    0m0.024s
sys     0m0.024s

Given these numbers I am in favour of @bgamari's original suggestion. If we do not want to clutter GHC's $LIBDIR then we could still put all Haskell dynamic libraries into one subdirectory with O(1) time cost.

The solution in ticket:11587#comment:116390 could be implemented in Linux (and ELF in general) too but I think performance would be as bad (at least when disk access is slow). To open foodir/libbar.so the runtime linker still needs to read foodir and then read libbar.so.

IIRC, GHC 8.0 does not encode the ABI hash in the dynamic library's file name anymore but has it only in the package's directory name. We will need to revisit that decision.

changed milestone to %8.2.1

Adding Duncan as this will be primarily a Cabal change, but sadly this won't be happening for 8.0.1.

mentioned in issue #12479 (closed)

Note that some more sharing is possible, given typical layouts.

For example for nix or for cabal new-build we install into a store.

$store/$pkgid-$hash/libHSpkgname-ver.so

(Currently it's actually worse than this since the libname includes hashes too, which should be unnecessary given the separated dirs)

I'm not sure if this is possible with ELF, but if we can include part of the directory into the libname / location, and a separate RUN_PATH then we could use a scheme like:

RUN_PATH /home/me/.cabal/store
SO_NEEDED pkgname-ver-hash/libHSpkgname-ver.so
SO_NEEDED ...

This does appear to be possible with MachO, ie like:

LC_RPATH /home/me/.cabal/store
LC_LOAD_DYLIB @rpath/pkgname-ver-hash/libHSpkgname-ver.dynlib

According to my experiments (I only tested on Debian x86_64 with GNU ld, not gold) you can set up the NEEDED entry in that way if either

libHSpkgname-ver.so was built with -soname pkgname-ver-hash/libHSpkgname-ver.so, or
libHSpkgname-ver.so was built without any -soname set, and you link to it with

-L/home/me/.cabal/store -l:pkgname-ver-hash/libHSpkgname-ver.so

If you set a SONAME when building libHSpkgname-ver.so, then it seems to be impossible to create a NEEDED entry of any other value when building a library that has it as a dependency. So if the SONAME is libHSpkgname-ver.so, you have to add the directory that the library lives in to the run path, as far as I can tell. I don't know how portable any of this behavior is.

GHC sets the SONAME to libHSpkgname-ver.so, originally due to 6efacfe8. I don't know if it would be okay to unset it or to include the directory name in the SONAME. In any case it seems simplest to solve the problem by just putting all the shared libraries in the same directory...

mentioned in issue #12770 (closed)

This sort of happened for 8.0.1 after all: there's a new package description field dynamic-library-dirs for the location of the shared library and by default Cabal now uses the $LIBDIR/lib$PKG_KEY.so layout suggested in this ticket. As I understand it this doesn't yet apply to the libraries distributed with GHC, though; but at least the number of libraries this affects is now constant (~25, rather than however many hundred dependencies a non-GHC program might have).

Trac metadata

Trac field	Value
Component	Package system → Build System

changed milestone to %8.4.1

Given that 8.2.1-rc1 is imminent, I'm bumping these off to the 8.4

mentioned in issue #14031

We have this same problem with static libraries and compile-time linking. See #14031.

Trac metadata

Trac field	Value
Related	- → #12031 (closed)

With ghc 8.0.2, I'm seeing 2500 ENOENTS. That is a large improvement from before, but still expensive. It adds around 200 ms to the startup time.

My program is linked dynamically with 203 haskell libraries, and there are a dozen paths still in RPATH for the libraries bundled with ghc. So, while there are only a few bundled libraries, their RPATHs still multiply badly with the often large numbers of libraries used by haskell programs.

mentioned in issue #14182

changed milestone to %8.6.1

This ticket won't be resolved in 8.4; remilestoning for 8.6. Do holler if you are affected by this or would otherwise like to work on it.

changed milestone to %8.8.1

These won't be fixed for 8.6, bumping to 8.8.

The new Hadrian build system has been merged. Relabeling the tickets concerning the legacy make build system to prevent confusion.

Trac metadata

Trac field	Value
Component	Build System → Build System (make)

changed milestone to %8.10.1

Bumping milestones of low-priority tickets.

marked this issue as related to #12770 (closed)

added linking label

added runtime perf label

mentioned in issue #16822

changed weight to 8

This was done in Hadrian in #12770 (closed).

closed

added Phigh label

mentioned in issue #19065

Place shared objects in LIBDIR

Child items 0

Activity

Place shared objects in LIBDIR

Relates to

Activity