Opened 21 months ago

Last modified 3 months ago

#11587 new bug

Place shared objects in LIBDIR

Reported by: bgamari Owned by:
Priority: normal Milestone: 8.4.1
Component: Build System Version: 7.10.3
Keywords: Cc: thomie, rwbarton, ezyang, Trommler, chak, dcoutts
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Runtime performance bug Test Case:
Blocked By: Blocking:
Related Tickets: #12031 Differential Rev(s):
Wiki Page:

Description (last modified by bgamari)

If one compiles a program with -dynamic, the resulting executable includes in its rpath the library directory of every Haskell package that the program links against. This causes a significant number of excess system calls at program start-up. For instance, in the case of a dynamically linked ghc executable on Debian 8, compiling a trivial "hello world" application produces over 800 open calls, the majority of which originate from the dynamic linker. e.g.,

$ strace -f -e open ghc-7.10.3 -c -fforce-recomp Test.hs 2>&1 | grep open 
...
open("/usr/lib/ghc/bin/../haske_GGvi737nHHfG6zm2y7Rimi/tls/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../haske_GGvi737nHHfG6zm2y7Rimi/tls/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../haske_GGvi737nHHfG6zm2y7Rimi/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../haske_GGvi737nHHfG6zm2y7Rimi/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../termi_6iVf4EBnOgfIaaOCLRs8jl/tls/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../termi_6iVf4EBnOgfIaaOCLRs8jl/tls/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../termi_6iVf4EBnOgfIaaOCLRs8jl/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../termi_6iVf4EBnOgfIaaOCLRs8jl/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../ghc_0AG9TOjDEtx4Ji3wSwHOBe/tls/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../ghc_0AG9TOjDEtx4Ji3wSwHOBe/tls/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/ghc/bin/../ghc_0AG9TOjDEtx4Ji3wSwHOBe/x86_64/libtinfo.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...

The dynamic linker must look in nearly 25 Haskell library directories to locate every system library! This is madness.

Instead of placing each shared library in its own directory, $LIBDIR/$PKG_KEY/lib$PKG_KEY.so as we do currently, why not just place them in $LIBDIR, e.g. $LIBDIR/lib$PKG_KEY.so. This would mean that we need to include only one directory, $LIBDIR, in rpath.

Change History (12)

comment:1 Changed 21 months ago by bgamari

Description: modified (diff)

comment:2 Changed 21 months ago by trommler

Cc: trommler added

comment:3 Changed 21 months ago by rwbarton

The dynamic linker must look in nearly 25 Haskell library directories to locate every system library! This is madness.

Not only that, but just to spell it all out explicitly,

  • The dynamic linker has to look in those nearly 25 Haskell library directories for each Haskell library, too, making the behavior quadratic in the number of Haskell packages.
  • This applies not just to ghc itself, but to any executable that uses dynamic Haskell libraries. Some people have programs that use hundreds of different Haskell packages.

comment:4 Changed 21 months ago by chak

Cc: Trommler chak added; trommler removed

I have wondered about this set up in the past. More precisely, I think having the dynamic libraries for different packages in separate directories with the interface files etc of the package makes a lot of sense. I keeps the $LIBDIR tidy.

However, the dynamic linking set up does appear to be rather inefficient and, at least on OS X, it makes relocating GHC distributions very hard. In the Haskell for Mac build, I hence, post-process all executables and dynamic libraries to optimise the linking process. As my ELF is a bit rusty let me explain it in MACH-O terms. I am setting the RPATH in all executables such that it points to $LIBDIR and I set the library name of each dynamic library to include the package directory. For example, for base, we might have base_GDytRqRVSUX7zckgKqJjgw/libHSbase-4.8.1.0-GDytRqRVSUX7zckgKqJjgw-ghc7.10.2.dylib.

I also set RPATH to be relative to @loader_PATH, which gives me a relocatable set of dynamic libraries and GHC executables.

So, for example, here is what base looks like

LC 03: LC_ID_DYLIB              @rpath/base_GDytRqRVSUX7zckgKqJjgw/libHSbase-4.8.1.0-GDytRqRVSUX7zckgKqJjgw-ghc7.10.2.dylib
...
LC 12: LC_LOAD_DYLIB            @rpath/integ_2aU3IZNMF9a7mQ0OzsZ0dS/libHSinteger-gmp-1.0.0.0-2aU3IZNMF9a7mQ0OzsZ0dS-ghc7.10.2.dylib
LC 13: LC_LOAD_DYLIB            @rpath/ghcpr_8TmvWUcS1U1IKHT0levwg3/libHSghc-prim-0.4.0.0-8TmvWUcS1U1IKHT0levwg3-ghc7.10.2.dylib
...
LC 19: LC_RPATH                 @loader_path/..

This avoids the quadratic explosion of the search space, but still keeps the dynamic libraries in the package directories (and $LIBDIR tidier). This is definitely the better set up on OS X. Can't we do something equivalent on Linux?

comment:5 Changed 21 months ago by trommler

I think the real issue is that Haskell's libraries are all in different directories and so the dynamic linker must read at least one directory and one file for each Haskell dynamic library.

The quadratic time required to find a library seems to be small compared to the disk access to read all directories in the RPATH from the hard disk. The second read of a directory would most of the time hit the file system cache.

I tried this experiment on my x86_64 Linux machine (still using spinning hard drives) with a dynamically linked GHC:

$ time ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

real    0m4.805s
user    0m0.044s
sys     0m0.080s
$ time ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3

real    0m0.048s
user    0m0.024s
sys     0m0.024s

Given these numbers I am in favour of @bgamari's original suggestion. If we do not want to clutter GHC's $LIBDIR then we could still put all Haskell dynamic libraries into one subdirectory with O(1) time cost.

The solution in comment:4 could be implemented in Linux (and ELF in general) too but I think performance would be as bad (at least when disk access is slow). To open foodir/libbar.so the runtime linker still needs to read foodir and then read libbar.so.

IIRC, GHC 8.0 does not encode the ABI hash in the dynamic library's file name anymore but has it only in the package's directory name. We will need to revisit that decision.

comment:6 Changed 20 months ago by bgamari

Cc: dcoutts added
Milestone: 8.0.18.2.1

Adding Duncan as this will be primarily a Cabal change, but sadly this won't be happening for 8.0.1.

comment:7 Changed 14 months ago by duncan

Note that some more sharing is possible, given typical layouts.

For example for nix or for cabal new-build we install into a store.

$store/$pkgid-$hash/libHSpkgname-ver.so

(Currently it's actually worse than this since the libname includes hashes too, which should be unnecessary given the separated dirs)

I'm not sure if this is possible with ELF, but if we can include part of the directory into the libname / location, and a separate RUN_PATH then we could use a scheme like:

RUN_PATH /home/me/.cabal/store
SO_NEEDED pkgname-ver-hash/libHSpkgname-ver.so
SO_NEEDED ...

This does appear to be possible with MachO, ie like:

LC_RPATH /home/me/.cabal/store
LC_LOAD_DYLIB @rpath/pkgname-ver-hash/libHSpkgname-ver.dynlib

comment:8 Changed 14 months ago by rwbarton

According to my experiments (I only tested on Debian x86_64 with GNU ld, not gold) you can set up the NEEDED entry in that way if either

  • libHSpkgname-ver.so was built with -soname pkgname-ver-hash/libHSpkgname-ver.so, or
  • libHSpkgname-ver.so was built without any -soname set, and you link to it with
      -L/home/me/.cabal/store -l:pkgname-ver-hash/libHSpkgname-ver.so
    

If you set a SONAME when building libHSpkgname-ver.so, then it seems to be impossible to create a NEEDED entry of any other value when building a library that has it as a dependency. So if the SONAME is libHSpkgname-ver.so, you have to add the directory that the library lives in to the run path, as far as I can tell. I don't know how portable any of this behavior is.

GHC sets the SONAME to libHSpkgname-ver.so, originally due to 6efacfe8bcbe66dfc3b52397ccbd34a58890520d. I don't know if it would be okay to unset it or to include the directory name in the SONAME. In any case it seems simplest to solve the problem by just putting all the shared libraries in the same directory...

comment:9 Changed 11 months ago by rwbarton

Component: Package systemBuild System

This sort of happened for 8.0.1 after all: there's a new package description field dynamic-library-dirs for the location of the shared library and by default Cabal now uses the $LIBDIR/lib$PKG_KEY.so layout suggested in this ticket. As I understand it this doesn't yet apply to the libraries distributed with GHC, though; but at least the number of libraries this affects is now constant (~25, rather than however many hundred dependencies a non-GHC program might have).

comment:10 Changed 8 months ago by bgamari

Milestone: 8.2.18.4.1

Given that 8.2.1-rc1 is imminent, I'm bumping these off to the 8.4

comment:11 Changed 4 months ago by bgamari

We have this same problem with static libraries and compile-time linking. See #14031.

comment:12 Changed 3 months ago by joeyhess

With ghc 8.0.2, I'm seeing 2500 ENOENTS. That is a large improvement from before, but still expensive. It adds around 200 ms to the startup time.

My program is linked dynamically with 203 haskell libraries, and there are a dozen paths still in RPATH for the libraries bundled with ghc. So, while there are only a few bundled libraries, their RPATHs still multiply badly with the often large numbers of libraries used by haskell programs.

Note: See TracTickets for help on using tickets.