Opened 9 months ago

Last modified 4 weeks ago

#9176 new bug

GHC not generating dyn_hi files

Reported by: heatsink Owned by:
Priority: normal Milestone:
Component: Compiler Version: 7.9
Keywords: dynamic Cc:
Operating System: Unknown/Multiple Architecture: x86_64 (amd64)
Type of failure: Other Test Case:
Blocked By: Blocking:
Related Tickets: Differential Revisions:

Description (last modified by heatsink)

Overview

In dynamic-too compilation, a module is compiled the normal way, then the backend runs twice to generate normal (.o, .hi) and dynamic (.dyn_o, .dyn_hi) outputs. Compilation uses the .hi files of imported modules. Effectively, the compiler assumes that all the information it gets from a .hi file, used by the normal output, is also true of the corresponding .dyn_hi file, used by the dynamic output. GHC has a few checks to verify this, but they don’t always produce desirable results.

  • When importing a module from a package in --make mode, GHC checks that the normal and dynamic interfaces match, in checkBuildDynamicToo in findAndReadIface. If they don’t match, GHC silently disables dynamic-too, so that no dynamic interface file is produced. No errors or warnings are reported.
  • When importing a module (either from a package, or standalone) in one-shot mode with -c, GHC starts out the same way as above: it checks the imported interfaces, finds out they don't match, and disables dynamic-too output. After generating output the normal way, the compiler pipeline runs again in the dynamic way to generate the dynamic output. This seems to be the "right" thing to do.
  • When importing a standalone module in --make mode, GHC does not examine the dynamic interface at all. It generates both normal and dynamic output.

Of the two --make cases, GHC uses the weaker checks when building a package and stricter checks when using the installed package. The weaker checks allow a package with mismatched normal and dynamic interface files to build and install without errors. After it's installed, the stronger checks suppress the creation of .dyn_hi files whenever the mismatched module is imported. Thus, a package installs fine, but importing from the package prevents dynamic-too compilation from working properly.

Reproducing

The attached code sets up an inconsistent pair of module interfaces and runs both kinds of imports. To compile the first two files, build the cabal package in testcase9176/. To compile the last file, install the package, then run the makefile in user/.

  • Imported.hs is compiled with different optimization flags, setting up a situation where the normal and dyamic module interfaces do not match.
  • User.hs importing directly Imported.hs loads only the static module interface. The dynamic interface is ignored.
  • User.hs importing from the installed package Imported.hs compares the static and dynamic interfaces, turns off dynamic-too compilation, and does not generate a .dyn_hi file. No errors or warnings are produced. This file is a copy of the other User.hs, just compiled differently.

Problems

GHC with -dynamic-too is expected to generate a .dyn_hi file, but it does not when compiling the third file.

When compiling a module, GHC should read the interfaces of any modules that it imports. However, when compiling the second file, GHC does not read the .dyn_hi file describing the imported dynamic module, only the .hi file describing the normal module.

The compiler does not treat mismatched module interfaces consistently in all situations, making it hard to find the root cause of errors.

I'm not sure what the correct behavior should be.

Details

This bug appeared on my system in a Parsec module. Parsec installed with no apparent problems, but other modules importing from Parsec would not compile to .dyn_hi files.

GHC was built from source, commit 9e10963e394680dbb1b964c66cb428a2aa03df09, compiled by GHC 7.6.3 with XCode 5.1.1 on OS X 10.9.3.

Attachments (1)

testcase9176-root.tar.gz (2.2 KB) - added by heatsink 8 months ago.
repro

Download all attachments as: .zip

Change History (11)

comment:1 Changed 9 months ago by heatsink

  • Description modified (diff)

comment:2 Changed 9 months ago by heatsink

  • Description modified (diff)

comment:3 Changed 9 months ago by heatsink

The dyn_hi file goes missing because GHC detects a bad module interface hash and disables dynamic-too. It's disabled in findAndReadIface by assigning False to canGenerateDynamicToo. That flag stops hscWriteIface from writing a dyn_hi file.

GHC will print diagnostic messages if it's run with -ddump-if-trace, so you can see this happening.

IMO, this condition should be an error because GHC does not generate the file it was told to generate.

comment:4 Changed 8 months ago by heatsink

  • Description modified (diff)

Changed 8 months ago by heatsink

repro

comment:5 Changed 8 months ago by heatsink

  • Description modified (diff)

comment:6 Changed 8 months ago by heatsink

  • Description modified (diff)

comment:7 Changed 8 months ago by heatsink

  • Description modified (diff)

comment:8 Changed 8 months ago by heatsink

  • Description modified (diff)

comment:9 follow-up: Changed 4 months ago by thomie

  • Operating System changed from MacOS X to Unknown/Multiple

Thank you for the elaborate report. This is my current understanding of the problems you pointed out, and the steps I took to reproduce them:

$ cd testcase9176

$ make
...

$ ghc --show-iface dist/build/Testcase9176/Imported.hi | grep ABI
ABI hash: 2dc4674632fe6d892f3f2ff2cfa3b411

$ ghc --show-iface dist/build/Testcase9176/Imported.dyn_hi | grep ABI
ABI hash: 786d9e1369def1ee16d1bf9865cce2e0

First observation. The ABI hashes are different, as was the intention. Imported.o is compiled with -O2, whereas Imported.dyn_o is compiled with the default -O1. (Remove this difference, and they have the same ABI.)

$ ghc --show-iface dist/build/Testcase9176/User.hi | grep 'Imported '
import  -/  Testcase9176.Imported 2dc4674632fe6d892f3f2ff2cfa3b411

$ ghc --show-iface dist/build/Testcase9176/User.dyn_hi | grep 'Imported '
import  -/  Testcase9176.Imported 2dc4674632fe6d892f3f2ff2cfa3b411

This shows the first problem (2nd on your list of problems). User.dyn_hi depends on Imported.hi (2dc..), while it should depend on Imported.dyn_hi (786..). GHC never examines the dynamic interface of Imported. This problem still exists in HEAD (ghc-20141113).

Continuing:

# Change extension `.dynlib` to `.so` in `Makefile`.
$ make clean
$ cabal install
$ cd ../user
$ cat Makefile 
User.o User.dyn_o : User.hs
	ghc --make -static -dynamic-too User.hs
$ make
$ ls
Makefile  User.dyn_o  User.hi  User.hs  User.o

This shows the second problem (1st on your list of problems). User.dyn_hi is not generated, even though we request for -dynamic-too. GHC does not show any errors or warnings about this. I am not able to reproduce this problem with HEAD, since cabal install --with-ghc=ghc-7.9.20141113 fails for me with "Bad interface file" (which might be good news for all I know).

Last edited 4 weeks ago by thomie (previous) (diff)

comment:10 in reply to: ↑ 9 Changed 4 months ago by heatsink

Replying to thomie:

Thanks for reproducing this bug. That looks like what I experienced, except for the package not installing with HEAD. The "Bad interface file" message looks like it's refusing to install a package with inconsistent ABI hashes, which may be an improvement.

I don't think anyone has worked out what properties are intended to hold for interface files, such as whether having mismatched ABI hashes is allowed. I brought this up on the mailing and got replies about a few use cases that should be supported. I's not clear to me what this means for the implementation.
https://www.haskell.org/pipermail/ghc-devs/2014-June/005301.html
https://www.haskell.org/pipermail/ghc-devs/2014-July/005475.html

Note: See TracTickets for help on using tickets.