Linker limit on OS X Sierra breaks builds for big projects

changed weight to 5

For the record, angerman has suggested in the past that we leverage recursive linking to work around this limitation. He has an example here. However, I'll admit I'm not terribly keen on spending our complexity budget working around arbitrary limitations imposed by Apple.

I'll eventually look into the recursive linking part. I've added aggressive name mangling to cabal for macOS a while ago, which help elevate this issue to a certain part. But we will eventually be unable to always stay under the limit.

So a temporary work around (unless you really have excessive amounts of dependencies), might be to use a Cabal build from HEAD. I'm not certain how stack handles this and haven't looked into it.

You are basically looking for any point after https://github.com/haskell/cabal/pull/4656 was merged. (3da83a0 was merged into haskell/cabal on Aug 22 2017).

assigned to @angerman

Trac metadata

Trac field	Value
CC	- → angerman

Note that in nixpkgs we have a solution that will always let us stay under the limit (essentially using intermediate libraries that reexport symbols from all the libraries underneath them) https://github.com/NixOS/nixpkgs/pull/27536

bgamari: I've been asking people to open tickets with Apple bgamari: As they are really the only ones that can really fix this issue

I am quite sure I know what Apple will respond: "You are holding it wrong."

Insider jokes aside, the core problem (as I mentioned before on one of the other tickets) is that GHC tries to use the macOS linker like a Linux linker and it was never meant to be used like that. Hence, Apple will only tell you to use it how it was designed to be used. (This is why I haven't filled a Radar —Apple speak for bug report— against this issue myself.)

As @angerman und @shlevy explained, there are alternatives to GHC's current scheme, which are closer to how library paths are usually managed on macOS. In addition, there is a reason why library names on macOS can include directory paths. In combination with @loader_path and @executable_path that always allowed to drastically shorten GHC's load command size. Haskell for Mac has done this for a long time (and I also mentioned this on a previous ticket) — see https://github.com/haskellformac/GHCframework/blob/master/GHCBuild/BuildGHC.sh for a messy shell script that is part of how the Haskell for Mac build handles this. (The other parts are in that repo as well.)

What I am trying to say here is that Apple will not change this, as they do not consider it a problem caused by macOS, but as a problem caused by GHC's abuse of the MACH-O linker format. Hence, I am afraid, we will have to fix this on our side.

chak, thank you for putting this out here as well!

I still plan to fix this properly in GHC. I'm not sure I will be able to do this for 8.4. The "fix" in cabal was a quick hack to make cabal work, as I had a project which needed to compile and I could not bump GHC. As such cabal needed to learn to be a bit more conservative about RPTAHs.

FWIW nix people use symlinks to lower the command size https://github.com/NixOS/nixpkgs/blob/a2bebdd6cedb58819cc1738282b9f02964d34a9a/pkgs/development/haskell-modules/generic-builder.nix#L239

So, I've recently run into this with a large project (using stack).

I've tried to make a small reproduce case but have not succeeded. I generated a stack project that builds a library along with 400 dummy dependencies, each of which has a long package name. This is a larger number of dependencies than my original project, but it doesn't trigger the panic. So is there something else I need to do to cause this?

Other funny things I've seen: Sometimes the build fails after slack clean, but then builds successfully after running it again. The big difference I see is that in the first case the link command is trying to build '/var/folders/.../ghc4371_0/libghc_100.dylib' when it fails, and in the second case it builds '.stack-work/.../libHSpanictest-0.0.0-8KaC2kNKjFj6cJfkbgZkev-ghc8.2.2.dylib' and succeeds. Is that just a quirk of stack or would it be a meaningful difference? What does it mean to build 'libghc_X.dylib'?

I apologize if this info is too stack specific, but it's hard for me to tell where this problem actually comes from. Also, if anyone could advise me on improving my reproduction case that would be very helpful.

So I believe this was mitigated in stack one way or the other; but your ticket commercialhaskell/stack#2577 makes it seem like it isn't. In cabal it is mitigated by haskell/cabal#4656.

I guess we should just go ahead and fix it properly in GHC. I feel GHC should fall over quicker if you construct a massive dependency tree with long folder and library names.

Of course now that most people using cabal, nix or stack don't hit this issue anymore as hard, the pressure to fix this is not as high anymore.

If you as such can reproduce a sample that always fails (it might be sufficient to double your dependencies)that would be helpful.

Note however: we will not be able to fix this for the case where all dependencies are direct, without some heavier lifting. IOW if it's the indirect (transitive) dependencies that cause the blow up, we might be able to fix that rather easy. If it is the direct dependencies that cause the blow up, this will pose a harder problem as we'd have to potentially inject pseudo aggregation libraries, which I'd rather not do.

Sorry for the delay- I forgot to add myself to CC and didn't see your message.

So I have now successfully reproduced the problem in a clean environment with bitemyapp's help. The missing piece is that it fails during template haskell compilation, but I'm a little confused at to why. It makes sense, of course, that template-haskell compilation would involve some dynamic linking in order to run the code, but if the number of dependencies is the problem, doesn't that imply TH links all of its dependencies dynamically? What is the advantage of dynamic linking here instead of just statically linking for TH?

Here's a github gist for a shell script that generates a giant stack project (with 150 generated dependencies) that triggers the panic: https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59 I'm not sure the best way to do the same thing without using stack, but if you have an idea I can do that.

Thanks!

Trac metadata

Trac field	Value
CC	angerman, chak, lelf → angerman, aosivitz, chak, lelf

Here's another idea: instead of dynamically linking each dep separately for TH codegen, we could combine all the modules into one and then dynamically link in that large blob.

Here's another idea: instead of dynamically linking each dep separately for TH codegen, we could combine all the modules into one and then dynamically link in that large blob.

I fear that may increase compilation time rather significantly, however. Linking tends to be quite expensive.

Hm, so- as I understand it. We create a single libghc_XX.dylib for use by TH, which in turn references each of its dependencies and loads them with 'load commands' (of which there is a hard total size limit on MacOS). What's the reason we take this step instead of just loading each dependency directly? Is that also for performance? If so, could we generate two of these libghc_XX.dylibs? Or 1 per every 50 dependencies?

instead of just loading each dependency directly?

Do you mean directly calling dlopen on each dependency? I suppose we could do this and it might even work. The problem is that it does nothing for the case of executables.

Yea, dlopen for each dependency.

Are you saying there's still a problem with executables that this wouldn't fix? How can I reproduce that? I was only able to reproduce the panic using template haskell.

Oh, I understand. By default GHC will statically link exes, but you can force it to be dynamic and in that case it will fail. Well, I personally am less concerned about that case because I'm not trying to dynamically link any exes, but I see why you would want both to work equally well.

It seems to me like the TH problem (building an extra dynamic lib unnecessarily which happens to cause bad behavior on Mac) is separate from the dynamic exe problem (how can we dynamically link a bunch of libs into an exe on Mac). Is there some implementation reason why those two problems are more tied together than I assume?

Here's a github gist for a shell script that generates a giant stack project (with 150 generated dependencies) that triggers the panic: https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59 I'm not sure the best way to do the same thing without using stack, but if you have an idea I can do that.

This will generate 150 direct dependencies. As such any dylib that will reference those, will have to reference 150 libraries, in the load commands section. I don't think we can do much about this case. If this was 150 transitive dependencies spread over a few levels. we might.

For 150 direct, we could potentially only work around this by using the splitting approach from nix. For 150 transitive dependencies, I still believe recursive linking could solve this. (See https://github.com/angerman/dylib-linking)

We have worked around this issue by following in the Nix folks' footsteps and using a wrapper script around ld to recursively subdivide the dependencies into a tree of re-exporting delegate libraries.

The script, an example using aosivitz repro, and more info about this issue is available here: https://github.com/Simspace/ld-wrapper-macos

The example in the repo shows how the wrapper script can be used to successfully build a project with 750 direct dependencies.

@jship thanks for the info, I tried it with a few work projects and it still fails for some of them, which is very frustrating. It seems like I have to get a tad more familiar with OS X linking tools to get to the bottom of it. I tried to bruteforce my way by adding -optl-fuse-ld=ld-wrapper-macos.sh to all the cabal targets in the repo.

I tried it with a few work projects and it still fails for some of them, which is very frustrating.

Note that the script alters the way in which dylibs are created, not the way in which they are linked. As a consequence, you may have to delete your existing dylib files, otherwise your build tool may choose to keep your old broken dylib files instead of re-creating them using our wrapper script.

Trac metadata

Trac field	Value
Differential revisions	- → D4717

mentioned in commit b592bd98

closed

changed milestone to %8.6.1

The above should fix 99% of these issues. Thanks to angerman for his persistence in searching for a fix.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

added Pnormal label

We ran into this bug at work (mercury.com), and the script provided by simspace seems to fix things.

I'm glad you find it useful! I just want to confirm that at SimSpace, the 99% fix did resolve our issues, so we are no longer using that script.

@trac-gelisam that's good to know, thanks. Unfortunately the script seems to not resolve all of our issues. I'm looking into things more right now

@chessai without knowing the exact setup, how you build/link your software, it's hard to give specific pointers, so here are some generic ones:

In general (when linking dynamically) you only need the top level dependencies and can rely on the linker to lazy load dependencies of dependencies; thus you do not need to embed the whole transitive dependency closure. GHC only links direct dependencies, I'm not absolutely sure if cabal might be trying to be smart and sidestepping this. Ideally cabal would defer linking to ghc.
If all your shared libraries are in the same folder, you only need a single rpath entry to that folder, this can have dramatic effects. If you have every library in a different folder (and even worse those folders are really long--as in /nix/store/...), you end up with lots of rpath entries, which all count towards the load command size. Nix has a hack around this where they symlink all libraries into a common folder.
Same holds for library names, which is why there is this rather ridiculous hack in cabal that mangles library names and removes vowels.

If you really end up with lots of direct dependencies, the script approach, which basically creates fake intermediate libraries that then pull in 1/nth of the direct dependencies each can work. I have not seen a project that has that many direct dependencies though, except for synthetic ones.

@chessai maybe you can provide a bit more context how your build proceeds?

marked this issue as related to #21441 (closed)

mentioned in issue #21441 (closed)

Trac field	Value
Version	8.2.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Compiler (Linking)
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

Linker limit on OS X Sierra breaks builds for big projects

Child items 0

Activity

Linker limit on OS X Sierra breaks builds for big projects

Relates to

Activity