Opened 2 years ago

Closed 10 months ago

Last modified 8 months ago

#6063 closed bug (fixed)

GHC's build-time ld-flag checks are problematic

Reported by: parcs Owned by: thoughtpolice
Priority: high Milestone: 7.8.1
Component: Compiler Version: 7.4.1
Keywords: Cc: bos@…, mail@…
Operating System: Linux Architecture: Unknown/Multiple
Type of failure: GHC doesn't work at all Difficulty: Unknown
Test Case: Blocked By:
Blocking: Related Tickets: #4862

Description

If the linker that GHC was built against happened to recognize the --hash-size and --reduce-memory-overhead flags, but the current system linker does not, GHC will indiscriminately pass those flags to the current linker when attempting to link anything, and the linking will fail due to an 'unrecognized option' error.

I have experienced this behavior when I upgraded to GHC 7.4.1 on debian. The GHC package in question was built against bfd ld, which recognized those linker flags, but my system linker is gold, which does not recognized those flags. Therefore, I could not build anything until i reverted to bfd ld.

(Relevant commits: 3275b7bd2a803a3adc0b952b6fbfeb738fc15a74 and 9ccb59ed6e5edf73c876e87429e69e8848162497)

Attachments (1)

0001-Detect-linker-information-at-runtime.-Fixes-Trac-606.patch (8.8 KB) - added by thoughtpolice 12 months ago.

Download all attachments as: .zip

Change History (23)

comment:1 Changed 21 months ago by edsko

comment:2 Changed 20 months ago by bos

  • Cc bos@… added
  • Operating System changed from Unknown/Multiple to Linux
  • Type of failure changed from None/Unknown to GHC doesn't work at all

Still an issue with 7.6.

comment:3 Changed 20 months ago by igloo

  • Difficulty set to Unknown

comment:4 follow-up: Changed 19 months ago by igloo

  • Milestone set to 7.8.1

The linker flags are detected when a bindist is installed, not when it is built, so they should match the machine that GHC is run on.

Should we check what flags the linker supports every time ghc is run?

comment:5 in reply to: ↑ 4 Changed 19 months ago by parcs

Replying to igloo:

The linker flags are detected when a bindist is installed, not when it is built, so they should match the machine that GHC is run on.

Should we check what flags the linker supports every time ghc is run?

Yeah, that sounds like a reasonable solution. The system linker may change after GHC is installed, and doing so shouldn't result in a non-functioning GHC installation.

comment:6 Changed 18 months ago by igloo

  • Priority changed from normal to high

comment:7 Changed 14 months ago by ezyang

On the other hand, having to shell out to ld every time we run GHC is a bit of a bummer, as far as performance goes. A stop-gap may be to just re-run ld without the lfags if it fails the first time; in the good case, we still manage to only do one invocation.

comment:8 Changed 13 months ago by thoughtpolice

  • Owner set to thoughtpolice

I'll take this over since I've had some complaints from friends about it (and have hit it myself in the past.)

I'll measure any compile time differences, but I imagine checking if the argument is supported through an extra invocation is negligible compared to the actual linking time. On windows, process creation can be expensive, but we'll only be using ld here anyway so we can short-cut it.

comment:9 Changed 12 months ago by nh2

  • Cc mail@… added

Was this not fixed in #4862 9 months ago?

Doesn't seem to be in 7.6.2 though ...

comment:10 Changed 12 months ago by jlebar

Is there a way to work around this in the meantime, short of uninstalling gold? I tried running

# LD=ld.bfd cabal install ...

and that didn't work.

comment:11 Changed 12 months ago by nh2

@jlebar That's not so easy, but you could actually make it *use* gold. See here: http://stackoverflow.com/questions/6952396/why-does-ghc-take-so-long-to-link/16105691#16105691. Might also contain some other relevant info for you (e.g. that -pgml is actually gcc, so you'd have to tell gcc which linker it should call, and I guess that's why your env variable doesn't work).

comment:12 Changed 12 months ago by thoughtpolice

I have a patch for this now. I decided to generalize it a little and added functionality to generally detect linker information at runtime, and distinguish between several cases. I have verified it works and am validating on my Mac OS X machine and my other Linux machines. I will set up a Windows machine as well.

I'd like some code review before I push this patch to the tree directly. If Ian or anyone else would like to give feedback, please do so. If there aren't any complaints I'll push this myself.

@nh2 - no, this was not fixed in that work. The issue is a little more complicated. Please see my commit message and patch which details the issue fairly clearly.

commit e1825358b318f847aca19a230fb0a4377830d767
Author: Austin Seipp <aseipp@pobox.com>
Date:   Sat May 4 16:07:22 2013 -0500

    Detect linker information at runtime. Fixes Trac #6063
    
    Previously, we did ./configure time checks to see if 'GNU ld' supported
    certain options. If it does, we bake those options into the link step.
    See Trac #5240. Ergo, this is a build-time configuration.
    
    Unfortunately, the linker we use at runtime can change for several
    reasons. One is that the user specifies -pgml 'foo'. The other is if
    /usr/bin/ld or whatnot literally *changes* from when GHC was built.
    Obviously, not every linker supports these options, like GNU gold,
    and that would lead to linking failure. This is Trac #6063.
    
    What this ultimately means is that we need to check at runtime what
    linker we're using. Always. This is actually a little bit complicated
    because we normally use the C compiler as our linker. Also, OS X does
    not support gold OR ld, Windows only has GNU ld, etc.
    
    Finally, this patch also unconditionally gives '--hash-size=31' and
    '--reduce-memory-overheads' to the system linker if it's GNU ld. These
    options have been supported for 8+ years from what I can see, and there
    are probably a lot of *other* reasons why GHC would not work with such
    an ancient binutils, all things considered.
    
    See Note [Run-time linker info] in SysTools for more details. There
    are plenty of comments as well in the surrounding code.
    
    Signed-off-by: Austin Seipp <aseipp@pobox.com>

comment:13 Changed 12 months ago by thoughtpolice

  • Status changed from new to patch

comment:14 Changed 12 months ago by parcs

I notice two things:

  1. From what I can tell, you don't need to spawn the linker process when os is OSDarwin or OSMinGW32.
  1. Use hang to print the error message:
hang (text "Warning:") 9 $
    text "Couldn't ..." $$
    text "Make ..."

comment:15 Changed 12 months ago by thoughtpolice

Updated per review comments (thanks to int-e on freenode as well.)

comment:16 follow-up: Changed 10 months ago by igloo

Generally looks fine to me. A few comments:

Re this comment:

It can also simply happen 
by using a binary distribution: GHC was built on a machine using GNU ld, 
and installed on a machine using GNU gold.

if by "binary distribution" you mean the bindists that we have on the website, then that's not the case: The info is redetected when you install a bindist.

On compiler/main/SysTools.lhs line 642, case os of doesn't actually match on the os at all AFAICS.

In --make mode, do we sometimes link many times? Even if not, GHC API users might link many times. It might be better to put an IORef (Maybe LinkerInfo) in DynFlags?, to cache the info.

comment:17 Changed 10 months ago by igloo

  • Owner thoughtpolice deleted
  • Status changed from patch to new

comment:18 Changed 10 months ago by igloo

  • Owner set to thoughtpolice

comment:19 Changed 10 months ago by aseipp@…

commit 71a194d8ca2efd075a5c000be0b378c8706ca0b3

Author: Austin Seipp <aseipp@pobox.com>
Date:   Sat May 4 16:07:22 2013 -0500

    Detect linker information at runtime. Fixes Trac #6063
    
    Previously, we did ./configure time checks to see if 'GNU ld' supported
    certain options. If it does, we bake those options into the link step.
    See Trac #5240.
    
    Unfortunately, the linker we use at runtime can change for several
    reasons. One is that the user specifies -pgml 'foo'. The other is if
    /usr/bin/ld or whatnot changes from when GHC was built.  Those options
    mentioned earlier are specific to GNU ld, but many systems support GNU
    gold too. This is Trac #6063.
    
    So we need to check at runtime what linker we're using. This is actually
    a little bit complicated because we normally use the C compiler as our
    linker. Windows and OS X are also special here.
    
    Finally, this patch also unconditionally gives '--hash-size=31' and
    '--reduce-memory-overheads' to the system linker if it's GNU ld. These
    options have been supported for 8+ years from what I can see, and there
    are probably a lot of other reasons why GHC would not work with such an
    ancient binutils, all things considered.
    
    See Note [Run-time linker info] in SysTools for details. There are
    plenty of comments as well in the surrounding code.
    
    Signed-off-by: Austin Seipp <aseipp@pobox.com>

 aclocal.m4                 |   33 ------------
 compiler/main/DynFlags.hs  |   26 +++++++++-
 compiler/main/SysTools.lhs |  117 ++++++++++++++++++++++++++++++++++++++++++-
 configure.ac               |    2 -
 distrib/configure.ac.in    |    2 -
 5 files changed, 137 insertions(+), 43 deletions(-)

comment:20 Changed 10 months ago by thoughtpolice

  • Resolution set to fixed
  • Status changed from new to closed

comment:21 in reply to: ↑ 16 ; follow-up: Changed 9 months ago by jberryman

Replying to igloo:

if by "binary distribution" you mean the bindists that we have on the website, then that's not the case: The info is redetected when you install a bindist.

Sorry to reply to a closed bug, but can I get clarification that this is also the case in 7.6.3? I created a bindist from the source distribution (with make binary-dist?) and then did a configure && make install after installing binutils-gold. When I build it's still trying to pass those flags to ld and is failing.

comment:22 in reply to: ↑ 21 Changed 8 months ago by gidyn

Replying to jberryman:

Sorry to reply to a closed bug, but can I get clarification that this is also the case in 7.6.3? I created a bindist from the source distribution (with make binary-dist?) and then did a configure && make install after installing binutils-gold. When I build it's still trying to pass those flags to ld and is failing.

Same here. The configure script claims to be checking whether ld understands --hash-size=31, but doesn't do anything with it.

Note: See TracTickets for help on using tickets.