Opened 6 years ago

Closed 5 years ago

Last modified 4 years ago

#5240 closed feature request (fixed)

help GNU ld to use less memory when linking libraries compiled with -split-objs.

Reported by: int-e Owned by: igloo
Priority: high Milestone: 7.4.1
Component: Compiler Version: 7.0.3
Keywords: Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

This is a well-known problem, but I didn't find a previous report here, probably because it is mainly a GNU ld issue. The problem is that with -split-objs, linking, say, ghc, takes a trememndous amount of memory.

Part of this problem will be fixed in ld:

http://sourceware.org/bugzilla/show_bug.cgi?id=12682

In this bug report, Nick Clifton suggests to use '--hash-size=31' and possibly '--reduce-memory-overhead' when invoking ld. Since ghc is invoking gcc for linking, that would mean passing '-Xlinker --hash-size=31' and perhaps '-Xlinker --reduce-memory-overhead' to gcc.

Can we do that? Care needs to be taken if people use the -pgmL option, and we should make sure that gcc is actually using GNU ld.

In my experiments (linking ghc), --hash-size=31 helps, reducing memory usage by about a factor of 1/2; --reduce-memory-overhead without --hash-size=31 also helps (saving about 1/3), but has no noticable effect in connection with --hash-size=31. Based on that I'd suggest to only use --hash-size=31.

Attachments (1)

0001-Rework-fix-for-5240-to-avoid-spamming-the-output-wit.patch (6.6 KB) - added by pgj 5 years ago.

Download all attachments as: .zip

Change History (21)

comment:1 Changed 6 years ago by igloo

Milestone: 7.4.1
Priority: normalhigh

comment:2 Changed 6 years ago by igloo

Owner: set to igloo

comment:3 Changed 5 years ago by igloo@…

commit 3275b7bd2a803a3adc0b952b6fbfeb738fc15a74

Author: Ian Lynagh <igloo@earth.li>
Date:   Sun Nov 27 00:20:59 2011 +0000

    Pass "--hash-size=31 --reduce-memory-overhead" to ld; fixes trac #5240
    
    These reduce the amount of memory that ld takes when linking.

 aclocal.m4                 |    9 +++++++--
 rules/build-package-way.mk |    2 +-
 2 files changed, 8 insertions(+), 3 deletions(-)

comment:4 Changed 5 years ago by igloo

Resolution: fixed
Status: newclosed

Fixed.

comment:5 Changed 5 years ago by pgj

Owner: igloo deleted
Resolution: fixed
Status: closednew

The fix works nicely, however it generates a lot of warnings when linking is not done, which is very annoying. So I created a patch to correct this.

comment:6 Changed 5 years ago by pgj

Status: newpatch

comment:7 Changed 5 years ago by pgj

Owner: set to igloo

comment:8 Changed 5 years ago by igloo

difficulty: Unknown
Owner: igloo deleted

comment:9 Changed 5 years ago by igloo

Owner: set to igloo

comment:10 Changed 5 years ago by igloo

Status: patchinfoneeded

pgj, I'm not sure what you mean by "it generates a lot of warnings when linking is not done". I've tried a few things, but failed to get it to produce any warnings.

Can you give some example commands showing how to create the warnings please?

comment:11 in reply to:  10 Changed 5 years ago by pgj

Replying to igloo:

Can you give some example commands showing how to create the warnings please?

Sorry, it seems I forgot to add that these warnings appear only on FreeBSD 9.0 or later. It is because that there were some changes or updates in the base system toolchain, which make the linker very sensitive about unused flags.

So the best way to reproduce them is to get a working FreeBSD 9 installation (in a virtual machine, for example) and try to build the vanilla GHC sources. Then one should see a lot of messages like this:

gcc: --hash-size=31: linker input file unused because linking not done
gcc: --reduce-memory-overheads: linker input file unused because linking not done

(Such messages tripled the size of the log.) Perhaps this would be fine, but since such flags are also recorded in the "settings" file, the compilation of every Haskell source file will generate the same messages on FreeBSD 9.0 or later.

Note that the builder clients will not yet generate warnings as they are at 8.1, the oldest supported release currently (where the linker does not support the requested features).

comment:12 Changed 5 years ago by igloo

Does this happen if you run:

echo "int f(int x) { return x; }" > j.c
ghc -c j.c

? If so, can you copy/paste the output of

ghc -c j.c -v

please?

comment:13 in reply to:  12 Changed 5 years ago by pgj

Replying to igloo:

Does this happen if you run ..?

No, it does not happen. But let me note that it is because the "settings" file does not contain the "--hash-size=31" and "--reduce-memory-overheads" flags for the linker.

When looking deeper into the cause, it turns out that when configuring the vanilla ghc bindist (7.4.2, built from sources on FreeBSD/amd64 -CURRENT) for installation, all those linker flag tests fail, because LdCmd has no value! (That is, the configure script gives different results for linker flag tests when building the compiler and when installing it!)

However, when I use the "settings" file right from the directory where the compiler was originally built and the information on linker flags is not lost (the same happens when installing the corresponding port):

> ghc -c j.c
gcc: --hash-size=31: linker input file unused because linking not done
gcc: --reduce-memory-overheads: linker input file unused because linking not done
gcc: --hash-size=31: linker input file unused because linking not done
gcc: --reduce-memory-overheads: linker input file unused because linking not done

Is this intentional...?

comment:14 Changed 5 years ago by igloo

Can you show me the output with -v please? And can you also clarify whether your ld supports those flags please?

comment:15 in reply to:  14 Changed 5 years ago by pgj

Replying to igloo:

Can you show me the output with -v please?

Sure. Here it is:

> /usr/local/ghc-7.4.2/bin/ghc -v -c j.c
Glasgow Haskell Compiler, Version 7.4.2, stage 2 booted by GHC version 7.0.4
Using binary package database: /usr/local/ghc-7.4.2/lib/ghc-7.4.2/package.conf.d/package.cache
wired-in package ghc-prim mapped to ghc-prim-0.2.0.0-7d3c2c69a5e8257a04b2c679c40e2fa7
wired-in package integer-gmp mapped to integer-gmp-0.4.0.0-af3a28fdc4138858e0c7c5ecc2a64f43
wired-in package base mapped to base-4.5.1.0-ac318071cf9dcba0758e9123955468c5
wired-in package rts mapped to builtin_rts
wired-in package template-haskell mapped to template-haskell-2.7.0.0-d08d077053eac454ce08e17f2c95c5c8
wired-in package dph-seq not found.
wired-in package dph-par not found.
Hsc static flags: -static
Created temporary directory: /tmp/ghc33022_0
*** C Compiler:
'/usr/bin/gcc' '-fno-stack-protector' '-Wl,--hash-size=31' '-Wl,--reduce-memory-overheads' '-x' 'c' 'j.c' '-o' '/tmp/ghc33022_0/ghc33022_0.s' '-S' '-Wimplicit' '-O' '-D__GLASGOW_HASKELL__=704' '-DTABLES_NEXT_TO_CODE' '-I' '/usr/local/include' '-I' '/usr/local/ghc-7.4.2/lib/ghc-7.4.2/base-4.5.1.0/include' '-I' '/usr/local/ghc-7.4.2/lib/ghc-7.4.2/include'
gcc: --hash-size=31: linker input file unused because linking not done
gcc: --reduce-memory-overheads: linker input file unused because linking not done
*** Assembler:
'/usr/bin/gcc' '-fno-stack-protector' '-Wl,--hash-size=31' '-Wl,--reduce-memory-overheads' '-c' '/tmp/ghc33022_0/ghc33022_0.s' '-o' 'j.o'
gcc: --hash-size=31: linker input file unused because linking not done
gcc: --reduce-memory-overheads: linker input file unused because linking not done
*** Deleting temp files:
Deleting: /tmp/ghc33022_0/ghc33022_0.s
*** Deleting temp dirs:
Deleting: /tmp/ghc33022_0

And can you also clarify whether your ld supports those flags please?

Well, perhaps the version numbers can tell, but I looked it up in the manual page, and ld(1) definitely support these flags.

So:

> ld -v
GNU ld 2.17.50 [FreeBSD] 2007-07-03

> ld --help | grep -E "(hash-size|reduce)"
  --hash-size=<NUMBER>        Set default hash table size close to <NUMBER>
  --reduce-memory-overheads   Reduce memory overheads, possibly taking much longer

comment:16 Changed 5 years ago by igloo

Hmm. What does

/usr/bin/gcc --version

say?

comment:17 in reply to:  16 Changed 5 years ago by pgj

Replying to igloo:

> /usr/bin/gcc --version
gcc (GCC) 4.2.1 20070831 patched [FreeBSD]
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

comment:18 Changed 5 years ago by igloo

Resolution: fixed
Status: infoneededclosed

Aha, looks like this is an old gcc bug that has been fixed:

$ /usr/bin/gcc-4.1 -c j.c -Wl,--foo
gcc-4.1: --foo: linker input file unused because linking not done
$ /usr/bin/gcc-4.3 -c j.c -Wl,--foo
$ 

I don't think it's worth working around this in GHC, as (a) it doesn't cause any real problems, just prints some warnings, and (b) it can be solved by upgrading to a recent gcc.

I have just pushed a patch to fix the problem that LdCmd wasn't getting defined in the bindist configure.ac, though. Thanks for spotting that.

comment:19 in reply to:  18 Changed 5 years ago by pgj

Replying to igloo:

I don't think it's worth working around this in GHC

I think this is not a workaround as the linker flags should not indeed passed when no linking is done. I do not think that the warning is bogus. It is just a more fine-grained way to pass the flags only when they are really needed.

(a) it doesn't cause any real problems, just prints some warnings

...for every Haskell module, which -- as I wrote above -- triples the number of the messages printed to the screen. I do not think that it is good to see such messages all the time.

(b) it can be solved by upgrading to a recent gcc.

It is not by accident that the FreeBSD base system does not have a more recent GCC, folks here are moving to a Clang/LLVM instead. Of course, I could use a newer GCC by adding it as a dependency for the port, but I was trying to avoid this.

Anyways, since GCC is going to be removed from the base system in the next major release, I shall switch to this solution soon... :-)

I have just pushed a patch to fix the problem that LdCmd wasn't getting defined in the bindist configure.ac, though. Thanks for spotting that.

Great, thanks!

comment:20 Changed 4 years ago by aseipp@…

commit 71a194d8ca2efd075a5c000be0b378c8706ca0b3

Author: Austin Seipp <aseipp@pobox.com>
Date:   Sat May 4 16:07:22 2013 -0500

    Detect linker information at runtime. Fixes Trac #6063
    
    Previously, we did ./configure time checks to see if 'GNU ld' supported
    certain options. If it does, we bake those options into the link step.
    See Trac #5240.
    
    Unfortunately, the linker we use at runtime can change for several
    reasons. One is that the user specifies -pgml 'foo'. The other is if
    /usr/bin/ld or whatnot changes from when GHC was built.  Those options
    mentioned earlier are specific to GNU ld, but many systems support GNU
    gold too. This is Trac #6063.
    
    So we need to check at runtime what linker we're using. This is actually
    a little bit complicated because we normally use the C compiler as our
    linker. Windows and OS X are also special here.
    
    Finally, this patch also unconditionally gives '--hash-size=31' and
    '--reduce-memory-overheads' to the system linker if it's GNU ld. These
    options have been supported for 8+ years from what I can see, and there
    are probably a lot of other reasons why GHC would not work with such an
    ancient binutils, all things considered.
    
    See Note [Run-time linker info] in SysTools for details. There are
    plenty of comments as well in the surrounding code.
    
    Signed-off-by: Austin Seipp <aseipp@pobox.com>

 aclocal.m4                 |   33 ------------
 compiler/main/DynFlags.hs  |   26 +++++++++-
 compiler/main/SysTools.lhs |  117 ++++++++++++++++++++++++++++++++++++++++++-
 configure.ac               |    2 -
 distrib/configure.ac.in    |    2 -
 5 files changed, 137 insertions(+), 43 deletions(-)
Note: See TracTickets for help on using tickets.