Make it easier to use the gold linker

As pointed out in #4862 (closed), the gold linker is significantly faster than BFD ld. Currently we use whatever linker gcc uses by default. This is an unfortunate situation for users as few packagers take the effort to configure their builds to use gold.

I think we should consider the following,

Introduce a configure flag (to both the source distribution, and the distributed binary distributions), --enable-gold. When enabled, configure will check for the functioning of gcc -fuse-ld=gold. If found to work -fuse-ld=gold would be added to GHC's optlc. The flag would throw an error on non-ELF platforms (which are not supported by gold).

While there is admittedly not a whole lot of precedent for this, the status quo means we are leaving a significant bit of compiler performance on the table in a majority of cases. Given that stack uses GHC's official bindists, we should try to improve this situation.

In fact, I would even weakly suggest that we might consider enabling --enable-gold the default behavior, requiring the user to explicitly pass --disable-gold if they want the current behavior.

mentioned in issue #4862 (closed)

mentioned in issue #11285 (closed)

mentioned in issue #12878 (closed)

changed weight to 5

added Tfeature request Trac import labels

changed the description

Another unfortunate aspect of the status quo is that the user may think that passing LD=gold to ./configure will configure the compiler to use gold, but they would be wrong.

Another approach that may be would be to check whether LD is set to gold and if so check for a functional -fuse-ld=gold.

changed the description

A few things to note:

• gold, does ELF only. So this should only be available and enabled if we are targeting elf.
• I'd also suggest that this is only enabled by default if gcc is used. If clang is used or gcc /is/ clang, don't enable gold.

I still believe that this kind of configuration should be part of the toolchain and not part of ghc though. I'm also a bit confused why gcc would not respect the LD env var. Maybe someone with better knowledge of the gcc toolchain knows.

Gold works perfectly fine with Clang on Linux. Why block that combination?

I might have gone a bit overboard here. I'm not sure if this ticket is related to https://phabricator.haskell.org/D3351, however clang doesn't understand -fuse-ld (at least not the one that comes with the android toolchain, not sure if this statement holds universally), and thus forcing that flag results in a non-functioning compiler.

I certainly don't want to block anything. I just don't want to have anything forced on me by default, which might run counter to my toolchain configuration. In my view this should be part of the toolchain, and ghc should ensure to pick up the configuration from the toolchain proper instead of doing the decision on its own.

I guess a similar case could be made for using lld?

To be clear, I don't think there is any need to block any particular combinations. We simply teach configure to check that -fuse-ld=gold works. If the compiler doesn't support it then the check fails and we either throw an error or proceed with the status quo.

Apparently lld is another factor of three faster than gold, so I suppose if we want to go this route we should ensure the solution will extend to that case as well.

I'd love to hear others opinions on this general direction.

Replying to [ticket:13541#comment:134905 angerman]:

clang doesn't understand -fuse-ld (at least not the one that comes with the android toolchain, not sure if this statement holds universally)

This seems to be Android toolchain specific.

Upstream clang suppports -fuse-ld since 2014: https://github.com/llvm-mirror/clang/commit/bab68c96f15867af6938d9c8f922d59d31351cad

Replying to [ticket:13541#comment:134901 angerman]:

I'm also a bit confused why gcc would not respect the LD env var.

I guess this may be controversial, but I have always liked the fact that cabal and GHC rely as little as possible on environment variables.

It has made it much easier many times for me to debug ghc issues (funnily, especially the linker investigation) because I can see all relevant inputs to a ghc invocation simply in ps or strace, and re-run them to isolate problems without accidentally not replicating the environment correctly.

Trac metadata

Trac field	Value
CC	angerman, geekosaur, hvr, jaffacake, rwbarton → angerman, geekosaur, hvr, jaffacake, nh2, rwbarton

Replying to [ticket:13541#comment:135196 nh2]:

Replying to [ticket:13541#comment:134905 angerman]:

clang doesn't understand -fuse-ld (at least not the one that comes with the android toolchain, not sure if this statement holds universally)

This seems to be Android toolchain specific.

Upstream clang suppports -fuse-ld since 2014: https://github.com/llvm-mirror/clang/commit/bab68c96f15867af6938d9c8f922d59d31351cad

This likely has been a misreading of the error on my side, as noted in https://phabricator.haskell.org/D3351. The LD that comes witht he android toolchain is called <toolchain>-ld or <toolchain>-ld.gold, and both are effectively gold, and there is no tool called just ld.gold, there is one called ld, but that is ld64, and would be horribly wrong to use.

Replying to [ticket:13541#comment:135197 nh2]:

Replying to [ticket:13541#comment:134901 angerman]:

I'm also a bit confused why gcc would not respect the LD env var.

I guess this may be controversial, but I have always liked the fact that cabal and GHC rely as little as possible on environment variables.

It has made it much easier many times for me to debug ghc issues (funnily, especially the linker investigation) because I can see all relevant inputs to a ghc invocation simply in ps or strace, and re-run them to isolate problems without accidentally not replicating the environment correctly.

My personal issue with not respecting env variables is that, without respecting the environment, one has to

• have explicit flags for each possible configuration value, that otherwise would have been taken from the environment.

• by not using the environment, one is posed to break tooling that depends on the environment.

GHC already has a lot of logic to find tools at configuration time and store their paths. FIND_LD already tries to detect gold, (and fails for the android toolchain...)

I am much more in favor of doing tool checking rather than magic. What I mean is this: instead of trying hard to find some tool (say ld), use $LD. However if we know we want gold, try $LD --version to verify it actually *is* gold. And if it's not, put out a warning that $LD is not set to gold and that this is known not to work. However if you want to ignore the error pass --compat-warning-only. Which would then print a warning:

Warning: linker ($LD) does not seem to be gold. Continuing anyway due to --compat-warning-only.

instead of

Error: linker ($LD) does not seem to be gold. bfd is known not to work. To continue anyway, pass --compat-warning-only.

Then again, this makes me wonder why we test for gold, and not against bfd in the first place? Why force gold, if lld is fine as well, when all we want is to make sure we don't use a buggy/broken/slow linker called bfd?

mentioned in issue #13739

mentioned in issue #13810 (closed)

#13810 (closed) demonstrates the terrible brittleness of the current state of affairs, where we rely on users to explicitly set their choice of linker.

Trac metadata

Trac field	Value
Differential revisions	- → D3449
Related	- → #13810 (closed)

I've updated D3449 to reflect my current thinking on this. Namely, we provide a configure flag that indicates that the user doesn't mind if GHC overrides the system's default linker. If this flag is passed, we use either gold or lld, if available. The user can explicitly request one or the other by passing the LD=... variable to configure.

Does this seem reasonable?

It seems that the logic in FIND_LD to install the gcc settings to actually use the linker is only triggered when the user does not pass LD=... I noticed this in Nix where ghc never tells gcc to use gold, despite Nix setting LD=ld.gold in the GHC derivations.

@bgamari do you remember if this was the intended behaviour and why?

/cc https://github.com/NixOS/nixpkgs/pull/24692

/cc @adamse

The user can explicitly request one or the other by passing the LD=... variable to configure.

@bgamari: To clarify, will GHC still at run-time detect which linker is available, and can the user still at run-time tell GHC which linker to use? At configure time, gold may not be installed, or it may be uninstalled afterwards, or the user maybe be using a bindist but for some reasons wants to force GHC to use ld or gold (for example, if gold doesn't work for them for some reason).

The diff doesn't touch the runtime linker-detection logic. It also doesn't provide a way for the user to override the linker choice, but this could be fixed.

However, I do wish we could drop the runtime probing at some point. Currently we start gcc -v and ld -v on every single GHC compilation. On platforms like Windows this can really add up. Even on Linux, where process spawning is relatively fast, it's probably 5 to 10 milliseconds per execution.

Simon Marlow has said that he would really like to see this happen in 8.2.1 due to the regressions in #13739.

Trac metadata

Trac field	Value
Related	#13810 (closed) → #13739, #13810 (closed)

changed milestone to %8.2.1

changed weight to 10

Trac metadata

Trac field	Value
Priority	normal → highest

assigned to @bgamari

mentioned in commit 625143f4

added 1 deleted label

closed

Merged to ghc-8.2 as 2785ef0e.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

removed 1 deleted label

reopened

I found a bug in the above. See D3694 for the fix.

Trac metadata

Trac field	Value
Resolution	ResolvedFixed → Unresolved
Differential revisions	D3449 → D3449, D3694

mentioned in commit 960918bd

closed

Merged to ghc-8.2 with 0798908f.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

mentioned in issue #13958

mentioned in commit fcd2db14

#13541 (closed) merged to ghc-8.2 in c23a84d5.

mentioned in commit d08b9ccd

added Phighest label