#11617 closed bug (wontfix)

DYLD_LIBRARY_PATH ignored on Mac OS X 10.11.x

Reported by: borsboom Owned by:
Priority: normal Milestone:
Component: hsc2hs Version: 7.10.3
Keywords: Cc: darchon, kazu-yamamoto
Operating System: MacOS X Architecture: x86_64 (amd64)
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: #8266, #8721 Differential Rev(s):
Wiki Page:

Description

This was first reported on the Stack issue tracker at https://github.com/commercialhaskell/stack/issues/1161, but I've traced it to the fact that GHC uses shell script wrappers for many common commands, including ghc, ghci, and hsc2hs.

Background: Mac OS X 10.11 introduced System Integrity Protection, which treats parts of the filesystem specially to prevent tampering with the system. Among those protections is that executables in /bin, such as /bin/sh, have the DYLD_LIBRARY_PATH variable stripped from the environment before being invoked.

Since ghc, hsc2hs, and others are shell scripts that use #!/bin/sh, it means DYLD_LIBRARY_PATH is stripped from the environment when invoking them. I was able to work around this issue by copying /bin/sh to my home directory and then modifying the #! path in the hsc2hs to use the copy.

I'm not sure what the best way to move forward is. Some possibilities I can think of:

  • Have installer tools for OS X (such as Stack, ghcformacosx, and Homebrew) apply the workaround themselves, although that leaves out anyone who installs directly from an official bindist or builds from source, and is a duplication of effort.
  • Have make install on OS X copy /bin/sh and "fix" the wrappers itself.
  • Replace the shell script wrappers with something written in Haskell that reads the paths from a configuration file.

Note: I set the component to hsc2hs since that's where I encountered the problem, but I wouldn't be surprised if it also effected other cases (like ghci, and ghc with Template Haskell). I also hesitated to categorize this a "bug" since it's OS X's weird behaviour, and not actually something I feel GHC is doing wrong, causing this problem. But I doubt OS X is going to change, although it is possible (but not recommended) to disable SIP by booting into recovery mode and running csrutil disable, after which these protections are no longer in place.

Change History (6)

comment:1 Changed 21 months ago by gershomb

Having done a little investigating, it seems to me that the real answer is "you shouldn't need to set DYLD_LIBRARY_PATH" . Indeed I don't think there are any normal workflows on OS X that should require this. The issue would only arise when linking to an external lib that inserts a path on DYLD_LIBRARY_PATH, and such libs are now effectively basically broken throughout the OS X ecosystem.

Here are a few articles that recommend against setting it (ever) as taken from a thread discussing this issue on Oracle bindings for node:

We may want to add a FAQ page for manual workarounds, but my gut says "don't fix, and recommend to those who encounter this that they should reconsider why they need DYLD_LIBRARY_PATH to begin with".

That said, I'm not one of the people "feeling the pain" here, so opinions may vary.

comment:2 Changed 18 months ago by borsboom

I haven't seen any reports from Stack users having trouble with this for months now. I think the world has gotten used to DYLD_LIBRARY_PATH not being useful anymore and adapted. If you want to close this issue, no objection from me.

comment:3 Changed 18 months ago by thomie

Cc: darchon kazu-yamamoto added

@darchon: do you agree with a "wontfix" for this issue?

comment:4 Changed 18 months ago by darchon

Sorry for the slow response. I'm on holiday, and switched to a Linux dev-environment a year ago. Consequently I can't just say whether this is a "won't fix" until I've thought in trough. I'm back from my holiday on the 5th of July, so I hope to give you an answer somewhere next week.

comment:5 Changed 18 months ago by darchon

TL;DR @thomie: After some consideration, I agree with the "wontfix".

I know of one "valid" reason to set the (DY)LD_LIBRARY_PATH environment variable in a "standard" Haskell development environment, and that is when there are dynamically-linked executables in a testing situation (i.e. a dynamically linked executable created by cabal test).

This is based on what I said here: https://github.com/haskell/cabal/issues/2330#issuecomment-69201669

As of Cabal-1.22, Cabal sets all the RPATHs instead of letting GHC handle them. What Cabal does differently from GHC, is that it sets the RPATHs to the final install locations, not the directory where the library currently resides. Now, for installed libraries, the final install location and the directory where the library currently resides are one and the same; however, for a library under development, the final install location is very most likely different from the place it currently resides: ./dist/build.

So, since a dynamically-linked executable created by cabal test is created the same way as any other dynamically-linked executable, I had to find a way for the created test executable to find the library under development. The way I did this is have cabal test run the test executable in an environment where (DY)LD_LIBRARY_PATH includes .dist/build: https://github.com/haskell/cabal/blob/6357bc536f993542fc385e3d1a59dac5f8b61268/Cabal/Distribution/Simple/Test/LibV09.hs#L85

However, since I don't see how the dynamically linked testing executables can end up in one of the protected system locations, their (DY)LD_LIBRARY_PATH environment variables will not be stripped. Consequently, this one "valid" use of (DY)LD_LIBRARY_PATH is not affected by OS X's behaviour.

So for now, I agree with the "wontfix", unless someone can truly point out another valid reason to set (DY)LD_LIBRARY_PATH.

Last edited 18 months ago by darchon (previous) (diff)

comment:6 Changed 18 months ago by thomie

Resolution: wontfix
Status: newclosed

Great explanation, thanks. Good thing GHC/Cabal doesn't rely on DYLD_LIBRARY_PATH (for installed libraries). We would be in a lot of trouble now if it did.

I still don't understand the original problem reported to reddit (https://www.reddit.com/r/haskell/comments/3ooxu4/library_not_loaded_libmariadb2dylib_os_x/, "I cloned the mysql package from bos/mysql and replaced the call to mysql_config program to mariadb_config program."). Maybe there's another bug somewhere, but it is unlikely GHC's fault.

Closing this issue as wontfix.

Note: See TracTickets for help on using tickets.