Loading temp shared object failed: TemplateHaskell and recompilation

In our project, we get from time to time an error like this:

ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.1 for x86_64-apple-darwin):
	Loading temp shared object failed: dlopen(/var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40483_0/libghc_1.dylib, 5): Symbol not found: _ShortText_ShortText_con_info
  Referenced from: /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40483_0/libghc_1.dylib
  Expected in: flat namespace
 in /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40483_0/libghc_1.dylib

I was able to create a somewhat minimal setup for reproducing the problem. I've reproduced the problem on Mac and Linux with GHC 8.0.1, didn't try on Windows.

The root of the problem seems to be an interaction with template haskell and the recompilation manager.

I'll now explain the setup. I've attached an archive which contains all code and a simple build script. Just unpack the archive and execute ./build.sh to reproduce the problem.

The setup consists of four files. (Don't be confused by the module names. They are derived from our project). The problem arises after a change to one of these files and a recompilation.

Here are the initial versions of the four files.

-- src/ShortText.hs
module ShortText where

data ShortText = ShortText String

toString :: ShortText -> String
toString (ShortText s) = s

-- src/Hospital.hs
module Hospital where

foo :: Int -> String
foo i = show i

-- src/Types.hs
{-# LANGUAGE TemplateHaskell #-}
module Types where

import qualified Hospital
import Language.Haskell.TH

genCode :: Q [Dec]
genCode =
    let s = Hospital.foo 5
    in [d|string = s|]

-- src/MetaHandler.hs
{-# LANGUAGE TemplateHaskell #-}
module Main where

import qualified Types

-- splices in "string :: String"
$(Types.genCode)

main :: IO ()
main = putStrLn string

Compiling the project for the first time works without problems:

$ /Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/bin/ghc -O0 -isrc -ibuild -dynamic -outputdir build -package template-haskell --make src/MetaHandler.hs
[1 of 3] Compiling Hospital         ( src/Hospital.hs, build/Hospital.o )
[2 of 3] Compiling Types            ( src/Types.hs, build/Types.o )
[3 of 3] Compiling Main             ( src/MetaHandler.hs, build/Main.o )
Linking src/MetaHandler ...

I then changed the content of src/Hospital.hs to use the ShortText module. Here is the new content:

-- src/Hospital.hs
module Hospital where

import ShortText

foo :: Int -> String
foo i = toString (ShortText (show i))

Then recompilation fails:

[1 of 4] Compiling ShortText        ( src/ShortText.hs, build/ShortText.o )
[2 of 4] Compiling Hospital         ( src/Hospital.hs, build/Hospital.o )
[4 of 4] Compiling Main             ( src/MetaHandler.hs, build/Main.o ) [TH]
ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.1 for x86_64-apple-darwin):
	Loading temp shared object failed: dlopen(/var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40827_0/libghc_7.dylib, 5): Symbol not found: _ShortText_ShortText_con_info
  Referenced from: /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40827_0/libghc_7.dylib
  Expected in: flat namespace
 in /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40827_0/libghc_7.dylib

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Looking at the output when compiling with -v3 gives a clue why:

...
*** Linker:
/usr/bin/gcc -m64 -fno-stack-protector -DTABLES_NEXT_TO_CODE -m64 -dynamiclib -o /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/libghc_7.dylib build/Hospital.o build/Types.o -undefined dynamic_lookup -single_module -install_name '@rpath/libghc_7.dylib' -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/template-haskell-2.11.0.0 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/template-haskell-2.11.0.0 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/pretty-1.1.3.3 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/pretty-1.1.3.3 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/deepseq-1.4.2.0 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/deepseq-1.4.2.0 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/array-0.5.1.1 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/array-0.5.1.1 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/ghc-boot-th-8.0.1 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/ghc-boot-th-8.0.1 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/base-4.9.0.0 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/base-4.9.0.0 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/integer-gmp-1.0.0.1 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/integer-gmp-1.0.0.1 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/ghc-prim-0.5.0.0 -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/ghc-prim-0.5.0.0 -L/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/rts -Wl,-rpath -Wl,/Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/lib/ghc-8.0.1/rts -lHStemplate-haskell-2.11.0.0-ghc8.0.1 -lHSpretty-1.1.3.3-ghc8.0.1 -lHSdeepseq-1.4.2.0-ghc8.0.1 -lHSarray-0.5.1.1-ghc8.0.1 -lHSghc-boot-th-8.0.1-ghc8.0.1 -lHSbase-4.9.0.0-ghc8.0.1 -lHSinteger-gmp-1.0.0.1-ghc8.0.1 -lHSghc-prim-0.5.0.0-ghc8.0.1 -liconv
*** Deleting temp files:
Deleting: /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/ghc_8.rsp /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/libghc_7.dylib
*** Deleting temp dirs:
Deleting: /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0
ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.1 for x86_64-apple-darwin):
	Loading temp shared object failed: dlopen(/var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/libghc_7.dylib, 5): Symbol not found: _ShortText_ShortText_con_info
  Referenced from: /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/libghc_7.dylib
  Expected in: flat namespace
 in /var/folders/8r/25v12lxd02zdm7lpky3xcnh80000gn/T/ghc40913_0/libghc_7.dylib

We can see here that the .o file for ShortText is not passed to gcc when building libghc_7.dylib. I suspect that libghc_7.dylib is used to run template haskell code, but this is only a guess.

Looking at the .hi file for MetaHandler, we can see why gcc does not get the .o file for ShortText: ShortText is not included in the module dependencies:

$ /Users/swehr/.stack/programs/x86_64-osx/ghc-8.0.1/bin/ghc --show-iface build/Main.hi
Magic: Wanted 33214052,
       got    33214052
Version: Wanted [8, 0, 0, 1],
         got    [8, 0, 0, 1]
Way: Wanted [],
     got    [d, y, n]
interface Main 8001
  interface hash: def8085a5dcb4980922089d5d21cf0f9
  ABI hash: 1a46783c6a6c6b308503cf705db45d4d
  export-list hash: d2d75d71ebb6963f6e9c9a5b4050ef71
  orphan hash: 693e9af84d3dfcc71e640e005bdc5e2e
  flag hash: 3468616c2ba87fe39e866ff30d651f01
  sig of: Nothing
  used TH splices: True
  where
exports:
  main
  string
module dependencies: Hospital Types
package dependencies: array-0.5.1.1 base-4.9.0.0 deepseq-1.4.2.0
                      ghc-boot-th-8.0.1 ghc-prim-0.5.0.0 integer-gmp-1.0.0.1
                      pretty-1.1.3.3 template-haskell-2.11.0.0
orphans: GHC.Base GHC.Float

I think the main problem is that src/Types.hs is not recompiled in the second run. I also experimented with compiling each file separately, using the -c flag. When I then compile src/Types.hs, ghc outputs "compilation NOT required" in the second run. If a compile src/Types.hs with -fforce-recomp in the 2nd run, everything works fine.

Trac metadata

Trac field	Value
Version	8.0.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Compiler
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

changed weight to 5

added Tbug Trac import labels

Attached file ghc-bug-12983.tar.gz (download).

I seem to have a similar bug (with 8.0.2) in a Yesod app (the app compiles fine with 7.10.3):

[61 of 61] Compiling Application      ( Application.hs, .stack-work/dist/x86_64-osx/Cabal-1.24.2.0/build/Application.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/1q/4xcpgs9n52sf2j5wbhxlt3r40000gn/T/ghc1009_0/libghc_264.dylib, 5): Symbol not found: _opaleyezm0zi5zi2zi2zm8brufdPJKpl4nCQHKm9Oif_OpaleyeziInternalziRunQuery_zdfQueryRunnerColumnDefaultPGRangePGRange_closure
  Referenced from: /var/folders/1q/4xcpgs9n52sf2j5wbhxlt3r40000gn/T/ghc1009_0/libghc_264.dylib
  Expected in: flat namespace
 in /var/folders/1q/4xcpgs9n52sf2j5wbhxlt3r40000gn/T/ghc1009_0/libghc_264.dylib
             
Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

I noticed that the bug only arises with -O0. With -O2 the app compiles.

Yep, it looks like we are not recompiling even though the module dependencies of a module we import have changed.

BTW, this looks similar to #7277 (closed) but it's not; the module which uses TH is correctly recompiled.

(It occurs to me that fixing this may be pessimal, in the sense that if you add imports to modules, we end up having to recompile all of them to get the module dependencies up to date. This is a tad unfortunate, because most of the time we don't really need this info.)

added TemplateHaskell recompilation checking labels

added compiler crash label

added Pnormal label

assigned to @mpickering

added to epic &2

At some point this has been fixed. I tried with ghc-8.10.2 and HEAD.

mentioned in commit 896c0b02

mentioned in merge request !6057 (closed)

mentioned in commit a02582eb

mentioned in commit 95fa40e2

mentioned in commit 22bc8670

mentioned in commit 69e6ed9b

mentioned in commit 27567379

mentioned in commit 6009fe32

mentioned in commit 6c1ff195

mentioned in commit 1d831090

mentioned in commit 236ac33e

mentioned in commit ee18cab4

mentioned in commit 4d119d54

mentioned in commit ae561594

mentioned in commit a6600a09

mentioned in commit bf15b900

mentioned in commit 128ec0b6

closed with commit 5a31abe3

mentioned in commit 5a31abe3