Opened 3 years ago

Closed 2 years ago

Last modified 22 months ago

#6063 closed bug (fixed)

GHC's build-time ld-flag checks are problematic

Reported by: parcs Owned by: thoughtpolice
Priority: high Milestone: 7.8.1
Component: Compiler Version: 7.4.1
Keywords: Cc: bos@…, mail@…
Operating System: Linux Architecture: Unknown/Multiple
Type of failure: GHC doesn't work at all Test Case:
Blocked By: Blocking:
Related Tickets: #4862 Differential Revisions:

Description

If the linker that GHC was built against happened to recognize the --hash-size and --reduce-memory-overhead flags, but the current system linker does not, GHC will indiscriminately pass those flags to the current linker when attempting to link anything, and the linking will fail due to an 'unrecognized option' error.

I have experienced this behavior when I upgraded to GHC 7.4.1 on debian. The GHC package in question was built against bfd ld, which recognized those linker flags, but my system linker is gold, which does not recognized those flags. Therefore, I could not build anything until i reverted to bfd ld.

(Relevant commits: 3275b7bd2a803a3adc0b952b6fbfeb738fc15a74 and 9ccb59ed6e5edf73c876e87429e69e8848162497)

Attachments (1)

0001-Detect-linker-information-at-runtime.-Fixes-Trac-606.patch (8.8 KB) - added by thoughtpolice 2 years ago.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 3 years ago by edsko

comment:2 Changed 3 years ago by bos

  • Cc bos@… added
  • Operating System changed from Unknown/Multiple to Linux
  • Type of failure changed from None/Unknown to GHC doesn't work at all

Still an issue with 7.6.

comment:3 Changed 3 years ago by igloo

  • difficulty set to Unknown

comment:4 follow-up: Changed 3 years ago by igloo

  • Milestone set to 7.8.1

The linker flags are detected when a bindist is installed, not when it is built, so they should match the machine that GHC is run on.

Should we check what flags the linker supports every time ghc is run?

comment:5 in reply to: ↑ 4 Changed 3 years ago by parcs

Replying to igloo:

The linker flags are detected when a bindist is installed, not when it is built, so they should match the machine that GHC is run on.

Should we check what flags the linker supports every time ghc is run?

Yeah, that sounds like a reasonable solution. The system linker may change after GHC is installed, and doing so shouldn't result in a non-functioning GHC installation.

comment:6 Changed 3 years ago by igloo

  • Priority changed from normal to high

comment:7 Changed 2 years ago by ezyang

On the other hand, having to shell out to ld every time we run GHC is a bit of a bummer, as far as performance goes. A stop-gap may be to just re-run ld without the lfags if it fails the first time; in the good case, we still manage to only do one invocation.

comment:8 Changed 2 years ago by thoughtpolice

  • Owner set to thoughtpolice

I'll take this over since I've had some complaints from friends about it (and have hit it myself in the past.)

I'll measure any compile time differences, but I imagine checking if the argument is supported through an extra invocation is negligible compared to the actual linking time. On windows, process creation can be expensive, but we'll only be using ld here anyway so we can short-cut it.

comment:9 Changed 2 years ago by nh2

  • Cc mail@… added

Was this not fixed in #4862 9 months ago?

Doesn't seem to be in 7.6.2 though ...

comment:10 Changed 2 years ago by jlebar

Is there a way to work around this in the meantime, short of uninstalling gold? I tried running

# LD=ld.bfd cabal install ...

and that didn't work.

comment:11 Changed 2 years ago by nh2

@jlebar That's not so easy, but you could actually make it *use* gold. See here: http://stackoverflow.com/questions/6952396/why-does-ghc-take-so-long-to-link/16105691#16105691. Might also contain some other relevant info for you (e.g. that -pgml is actually gcc, so you'd have to tell gcc which linker it should call, and I guess that's why your env variable doesn't work).

comment:12 Changed 2 years ago by thoughtpolice

I have a patch for this now. I decided to generalize it a little and added functionality to generally detect linker information at runtime, and distinguish between several cases. I have verified it works and am validating on my Mac OS X machine and my other Linux machines. I will set up a Windows machine as well.

I'd like some code review before I push this patch to the tree directly. If Ian or anyone else would like to give feedback, please do so. If there aren't any complaints I'll push this myself.

@nh2 - no, this was not fixed in that work. The issue is a little more complicated. Please see my commit message and patch which details the issue fairly clearly.

commit e1825358b318f847aca19a230fb0a4377830d767
Author: Austin Seipp <[email protected]>
Date:   Sat May 4 16:07:22 2013 -0500

    Detect linker information at runtime. Fixes Trac #6063
    
    Previously, we did ./configure time checks to see if 'GNU ld' supported
    certain options. If it does, we bake those options into the link step.
    See Trac #5240. Ergo, this is a build-time configuration.
    
    Unfortunately, the linker we use at runtime can change for several
    reasons. One is that the user specifies -pgml 'foo'. The other is if
    /usr/bin/ld or whatnot literally *changes* from when GHC was built.
    Obviously, not every linker supports these options, like GNU gold,
    and that would lead to linking failure. This is Trac #6063.
    
    What this ultimately means is that we need to check at runtime what
    linker we're using. Always. This is actually a little bit complicated
    because we normally use the C compiler as our linker. Also, OS X does
    not support gold OR ld, Windows only has GNU ld, etc.
    
    Finally, this patch also unconditionally gives '--hash-size=31' and
    '--reduce-memory-overheads' to the system linker if it's GNU ld. These
    options have been supported for 8+ years from what I can see, and there
    are probably a lot of *other* reasons why GHC would not work with such
    an ancient binutils, all things considered.
    
    See Note [Run-time linker info] in SysTools for more details. There
    are plenty of comments as well in the surrounding code.
    
    Signed-off-by: Austin Seipp <[email protected]>

comment:13 Changed 2 years ago by thoughtpolice

  • Status changed from new to patch

comment:14 Changed 2 years ago by parcs

I notice two things:

  1. From what I can tell, you don't need to spawn the linker process when os is OSDarwin or OSMinGW32.
  1. Use hang to print the error message:
hang (text "Warning:") 9 $
    text "Couldn't ..." $$
    text "Make ..."

comment:15 Changed 2 years ago by thoughtpolice

Updated per review comments (thanks to int-e on freenode as well.)

comment:16 follow-up: Changed 2 years ago by igloo

Generally looks fine to me. A few comments:

Re this comment:

It can also simply happen 
by using a binary distribution: GHC was built on a machine using GNU ld, 
and installed on a machine using GNU gold.

if by "binary distribution" you mean the bindists that we have on the website, then that's not the case: The info is redetected when you install a bindist.

On compiler/main/SysTools.lhs line 642, case os of doesn't actually match on the os at all AFAICS.

In --make mode, do we sometimes link many times? Even if not, GHC API users might link many times. It might be better to put an IORef (Maybe LinkerInfo) in DynFlags, to cache the info.

comment:17 Changed 2 years ago by igloo

  • Owner thoughtpolice deleted
  • Status changed from patch to new

comment:18 Changed 2 years ago by igloo

  • Owner set to thoughtpolice

comment:19 Changed 2 years ago by aseipp@…

commit 71a194d8ca2efd075a5c000be0b378c8706ca0b3

Author: Austin Seipp <[email protected]>
Date:   Sat May 4 16:07:22 2013 -0500

    Detect linker information at runtime. Fixes Trac #6063
    
    Previously, we did ./configure time checks to see if 'GNU ld' supported
    certain options. If it does, we bake those options into the link step.
    See Trac #5240.
    
    Unfortunately, the linker we use at runtime can change for several
    reasons. One is that the user specifies -pgml 'foo'. The other is if
    /usr/bin/ld or whatnot changes from when GHC was built.  Those options
    mentioned earlier are specific to GNU ld, but many systems support GNU
    gold too. This is Trac #6063.
    
    So we need to check at runtime what linker we're using. This is actually
    a little bit complicated because we normally use the C compiler as our
    linker. Windows and OS X are also special here.
    
    Finally, this patch also unconditionally gives '--hash-size=31' and
    '--reduce-memory-overheads' to the system linker if it's GNU ld. These
    options have been supported for 8+ years from what I can see, and there
    are probably a lot of other reasons why GHC would not work with such an
    ancient binutils, all things considered.
    
    See Note [Run-time linker info] in SysTools for details. There are
    plenty of comments as well in the surrounding code.
    
    Signed-off-by: Austin Seipp <[email protected]>

 aclocal.m4                 |   33 ------------
 compiler/main/DynFlags.hs  |   26 +++++++++-
 compiler/main/SysTools.lhs |  117 ++++++++++++++++++++++++++++++++++++++++++-
 configure.ac               |    2 -
 distrib/configure.ac.in    |    2 -
 5 files changed, 137 insertions(+), 43 deletions(-)

comment:20 Changed 2 years ago by thoughtpolice

  • Resolution set to fixed
  • Status changed from new to closed

comment:21 in reply to: ↑ 16 ; follow-up: Changed 23 months ago by jberryman

Replying to igloo:

if by "binary distribution" you mean the bindists that we have on the website, then that's not the case: The info is redetected when you install a bindist.

Sorry to reply to a closed bug, but can I get clarification that this is also the case in 7.6.3? I created a bindist from the source distribution (with make binary-dist?) and then did a configure && make install after installing binutils-gold. When I build it's still trying to pass those flags to ld and is failing.

comment:22 in reply to: ↑ 21 Changed 22 months ago by gidyn

Replying to jberryman:

Sorry to reply to a closed bug, but can I get clarification that this is also the case in 7.6.3? I created a bindist from the source distribution (with make binary-dist?) and then did a configure && make install after installing binutils-gold. When I build it's still trying to pass those flags to ld and is failing.

Same here. The configure script claims to be checking whether ld understands --hash-size=31, but doesn't do anything with it.

Note: See TracTickets for help on using tickets.