#11445 closed task (fixed)
Turn on SplitSections by default
Reported by: | ezyang | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | 8.8.1 |
Component: | Build System (make) | Version: | 8.0.1-rc1 |
Keywords: | Cc: | bgamari, olsner, awson | |
Operating System: | Unknown/Multiple | Architecture: | Unknown/Multiple |
Type of failure: | None/Unknown | Test Case: | |
Blocked By: | Blocking: | ||
Related Tickets: | Differential Rev(s): | Phab:D1800 | |
Wiki Page: |
Description (last modified by )
This is a general tracking bug for what is blocking us from turning on SplitSections
by default, implemented in #8405. Fixing this, and mandating use of the gold linker, will help solve #11285 (split objects make linking slow), and will make ghc-split unnecessary, mooting #9832.
I encountered some misinformation while talking to people about this feature, regarding whether or not this feature only supported by Apple-style linkers (i.e. on Mac OS X)? On IRC I heard a claim made that --split-sections
was only supported by Mac OS X. However, olsner pointed out that -split-sections
on GHC was OUR reimplementation of the feature, and likely does not work on Mac OS X. https://github.com/snowleopard/shaking-up-ghc/issues/153#issuecomment-171078800 A later comment mentioned that
So, can we turn on this feature by default? Does it subsume split-objs? Does it work on all the tier-1 platforms? It would be nice if bgamari, olsner and co could weigh in here. Thanks!
Also, there are two different ways we can do this. One is we can turn it on by default in the GHC build system, so all the base libraries and GHC are built with it (but when you invoke ghc, it does NOT have split sections); the other is to have GHC always run split sections, so user code also compiles with it.
Attachments (1)
Change History (28)
comment:1 Changed 3 years ago by
Description: | modified (diff) |
---|
comment:3 Changed 3 years ago by
So on ELF platforms -split-sections should be ready to replace -split-objs; further testing and investigation is necessary on Windows and OS X.
OS X should just use the built-in subsection support.
For non-Linux ELF platforms, an issue is that --gc-sections
(under that name) is only available from GNU ld, other linkers might not have it or might call it something else. That means dropping -split-objs
probably would leave some platforms without dead-code stripping at all.
It should be good to go with GNU ld on non-Windows though. And for unixes without GNU ld we could get configure to check if the linker has something like --gc-sections
and how to activate it. (Looks like e.g. Solaris has a -z discard-unused=sections
flag.)
comment:4 Changed 3 years ago by
Differential Rev(s): | → Phab:D1800 |
---|
comment:5 follow-up: 6 Changed 3 years ago by
I have fully working -split-sections
on Windows.
To make it possible quite recently I've fixed a blocker bug in GHC (thanks to Phyx for putting it in).
But also we need the very latest binutils because of the bug in binutils I've helped to fix just recently.
And the final thing to make it all work are correct linker scripts (generated from the single pep.sc
template), which solve section merging problem. I've incorporated it in my own patched version of binutils but I still didn't propose the corresponding patch to upstream binutils. I have no much spare time at the moment, and this patch also contains an extra bugfix for current binutils, thus some extra work is required here, but if the need is urgent I'll try to wrap it in coming days.
comment:6 Changed 3 years ago by
Replying to awson:
I have fully working
-split-sections
on Windows.
That's awesome! Thank you for working on this.
comment:7 Changed 3 years ago by
comment:8 Changed 2 years ago by
Is there anything left before we can do this?
If not I will go ahead and put a patch up on Phabricator that makes -split-objs
a deprecated synonym for -split-sections
.
comment:9 Changed 2 years ago by
@awson never put his patches up. So at the moment, -split-sections
does not work on PE targets. So no, you can't flip it on yet.
I'm working on getting things up and running.
comment:11 Changed 2 years ago by
Well, things are absolutely OK with -split-sections
for 64-bit PECOFF obj files, but there exist a critical problem for the 32-bit case.
The problem is that to realistically work with -split-sections
(even to build GHC
itself, particularly template-haskell
package, particularly Language.Haskell.TH.Syntax
module compiled in profiling mode) we need big-obj
support (to have more than 32K sections per object file), but binutils
has no big-obj
support for 32-bit PECOFF object files.
Perhaps, the simplest solution would be to have another splitter, which would split the whole assembly to chunks with no more than 32K sections in each.
comment:12 Changed 2 years ago by
If somebody wants to enjoy the benefits of -split-sections
vs -split-objs
(they are tremendous) immediately, she has the following options:
- I could try to put linker script patches to upstream
binutils
. Honestly, I don't want to bother with this ATM; - I could publish these patches for those interested to build their own
binutils
; - I could publish linker scripts which could be used with existing
binutils
.
Since all options require (at least) to modify any existing build scripts, I believe that, perhaps, the option 3 might be the most welcomed one. OTOH, options 2 and 3 require more involvement than the first one.
comment:13 Changed 2 years ago by
awson: could GHC have the new linker script embedded into it, so that GHC can use it automatically? GHC invokes the linker itself, so it can pass any custom linker script necessary.
comment:14 Changed 2 years ago by
Cc: | awson added |
---|
comment:15 Changed 2 years ago by
@awson Please do whatever is easiest for you. If you don't want to bother with integrating your patches into upstream binutils
, maybe someone else is, so please at least open source them.
comment:16 Changed 2 years ago by
Well, I've updated relevant binutils ticket with patches solving the problem.
Changed 2 years ago by
Attachment: | scripts.tar.gz added |
---|
Linker scripts for section merging on Windows
comment:17 Changed 2 years ago by
Scripts attached do section merging on Windows.
Caveat:
As far as I remember, when I worked on this (almost year ago) cabal
(or was it GHC?) didn't respect linker flags when building library's prelinked object file (for GHCi). I may be wrong or cabal
may be changed since that time, but if you will try to use attached linker scripts with existing binutils, please make sure ld
receives the option feeding the linker script to it. If this problem is still there, please ask cabal
(or GHC?) maintainers to fix it. Also it may be a problem that we need to use different linker scripts for different scenarios: "x" for building executables/dlls and "xr" for building prelinked object files.
comment:18 Changed 2 years ago by
The summary (for Windows):
- I always used patched binutils and never used separate linker scripts, thus can't comment on potential problems with the latter.
- On 64-bit GHC I don't use
-split-objs
at all and use-split-sections
exclusively for almost a year and I never had any problems with this. The only important thing one should remember is that we sometimes need to supply-opta-Wa,-mbig-obj
to GHC to make it build correct object files (I know 2 package which require this:template-haskell
when building GHC itself andhaskell-src-exts
). One possible option is to always use this flag if-split-sections
is used. -split-sections
doesn't introduce any noticeable overhead when compiling and thus is dramatically faster than-split-objs
when building packages, it is slightly slower when linking final executables though.- 64-bit GHC itself can be built in release configuration with
-split-objs
replaced by-split-sections
absolutely with no problems (only-opta-Wa,-mbig-obj
should be added toghc-options
intemplate-haskell
cabal file). 32-bit GHC building process is less smooth and requires a special handling oftemplate-haskell
package (one needs to recompileLanguage.Haskell.TH.Syntax
module in profiling configuration with-split-objs
). Overall, building release GHC itself with-split-sections
is much faster than with-split-objs
.
comment:19 Changed 2 years ago by
I assumed the linker problem and the required fix would be much more complicated, but after reading the patches it looks like it could simply be an issue with how we name sections on Windows builds? (See also https://sourceware.org/bugzilla/show_bug.cgi?id=19254#c3)
That means maybe we "only" need:
- unpatched binutils (but a new enough version)
-Wa,-mbig-objs
for the assembler- fixed section names
- a PE-compatible linker script for
ld -r
section merging
comment:20 Changed 2 years ago by
It looks current binutils' linker scripts doesn't handle rodata
sections at all, thus I believe we should patch binutils anyway.
Regarding section namespace separator, indeed gcc
on Windows generates $
instead of .
, and this, perhaps, would be correct to make GHC do the same thing, but even unpatched Windows bintuils have separate handling of .text.*
sections for example. I suspect that this is because there exist another tools from GNU/Unix land, ported to Windows which also use .
for section hierarchy handling.
Also, sections with $
should be sorted according to their suffix names, which, I believe is not what we want in our (GHC) case (and this is not the case for ELF files).
comment:22 Changed 2 years ago by
Milestone: | 8.2.1 → 8.4.1 |
---|
To get any further on this we would need to fix the Windows situation which won't be happening for 8.2. Bumping.
comment:23 Changed 13 months ago by
Milestone: | 8.4.1 → 8.6.1 |
---|
This ticket won't be resolved in 8.4; remilestoning for 8.6. Do holler if you are affected by this or would otherwise like to work on it.
comment:24 Changed 12 months ago by
Blocking: | 11285 removed |
---|
comment:25 Changed 8 months ago by
Milestone: | 8.6.1 → 8.8.1 |
---|
These won't be fixed for 8.6, bumping to 8.8.
comment:26 Changed 7 months ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
The build system now uses -split-sections
to build GHC and the core libraries on supported platforms, see comment:21.
See #12913 and #13939 for remaining work to be done for Windows.
Closing.
comment:27 Changed 4 months ago by
Component: | Build System → Build System (make) |
---|
The new Hadrian build system has been merged. Relabeling the tickets concerning the legacy make
build system to prevent confusion.
The gold linker only supports ELF, so it can't be used on OS X and Windows.
GNU ld didn't support
--gc-sections
for PE/COFF until recently, and this feature still appears to be buggy according to olsner. So we'll need to fix those bugs in ld to make-split-sections
work on Windows. We'll also need to ship with an unreleased version of ld.According to the links you provided, OS X ld doesn't support
--gc-sections
at all, but maybe the-dead_strip
option could be used.So on ELF platforms
-split-sections
should be ready to replace-split-objs
; further testing and investigation is necessary on Windows and OS X.