#8919 closed bug (fixed)

Why is xhtml library installed but not exported to users?

Reported by: simons Owned by:
Priority: normal Milestone: 7.8.3
Component: Build System Version: 7.8.2
Keywords: Cc: juhp@…, simons, felixonmars@…, slyfox@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Other Test Case:
Blocked By: Blocking:
Related Tickets: Differential Revisions:

Description (last modified by errge)

GHC 7.8.1-rc2 installs the xhtml library, but doesn't export it, i.e. it's completely invisible to users. Is there a particular reason why this setup is necessary?

If GHC needs that library internally anywhere, would it be possible to link it statically and *not* install it?

Or, if that's not possible, if the library needs to be installed, please make it visible to users.

See comment 17 for a discussion on why this is causing problems for distribution vendors.

Change History (26)

comment:1 Changed 13 months ago by juhpetersen

  • Cc juhp@… added

Right +1

In Fedora, so far I have been working around it by packaging the xhtml library along with the other ghc libraries. Since xhtml is in HP anyway it seems easier/cleaner just to move it to ghc now IMHO.

comment:2 Changed 12 months ago by simons

  • Cc simons added
  • Version changed from 7.6.3 to 7.8.2

comment:3 Changed 12 months ago by hvr

The same applies to terminfo-0.4.0.0 and haskeline-0.7.1.2 which have only the DSOs installed (and no interface files).

comment:4 follow-up: Changed 12 months ago by td123

Just encountered this issue on archlinux since mtl was getting installed but it's dep on transformers wasn't. My journey ended at this ticket.

Here is a full list of packages that are used internally, but contain files when you make install:

haskell-haskeline=0.7.1.2
haskell-hoopl=3.9.0.0
haskell-terminfo=0.4.0.0
haskell-transformers=0.3.0.0
haskell-xhtml=3000.2.1

comment:5 Changed 12 months ago by td123

Investigating this further, it looks like this isn't a case of files getting accidentally included.

I found the following comment in ghc.mk

# If we have built the programs with dynamic libraries, then
# ghc will be dynamically linked against haskeline.so etc, so
# we need the dynamic libraries of everything down to here

Which looks like it includes these libs if DYNAMIC_GHC_PROGRAMS is set to "YES"

comment:6 Changed 12 months ago by felixonmars

  • Cc felixonmars@… added

comment:7 in reply to: ↑ 4 ; follow-up: Changed 12 months ago by hvr

Replying to td123:

Just encountered this issue on archlinux since mtl was getting installed but it's dep on transformers wasn't. My journey ended at this ticket.

Here is a full list of packages that are used internally, but contain files when you make install:

haskell-haskeline=0.7.1.2
haskell-hoopl=3.9.0.0
haskell-terminfo=0.4.0.0
haskell-transformers=0.3.0.0
haskell-xhtml=3000.2.1

Strange, transformers and hoopl shouldn't be in that list (i.e. as mentioned earlier in this ticket, only haskeline,terminfo, and xhtml are problematic), as those are actually needed by the ghc package (and they will show up in ghc-pkg list for a properly performed GHC installation)

comment:8 in reply to: ↑ 7 Changed 12 months ago by td123

Replying to hvr:

Strange, transformers and hoopl shouldn't be in that list (i.e. as mentioned earlier in this ticket, only haskeline,terminfo, and xhtml are problematic), as those are actually needed by the ghc package (and they will show up in ghc-pkg list for a properly performed GHC installation)

Ah, this is my fault for looking at the list of files installed. If we are only talking about ghc-pkg list then I get the following lines on a fresh install:

/usr/lib/ghc-7.8.2/package.conf.d
   Cabal-1.18.1.3
   array-0.5.0.0
   base-4.7.0.0
   bin-package-db-0.0.0.0
   binary-0.7.1.0
   bytestring-0.10.4.0
   containers-0.5.5.1
   deepseq-1.3.0.2
   directory-1.2.1.0
   filepath-1.3.0.2
   ghc-7.8.2
   ghc-prim-0.3.1.0
   haskell2010-1.1.2.0
   haskell98-2.0.0.3
   hoopl-3.10.0.1
   hpc-0.6.0.1
   integer-gmp-0.5.1.0
   old-locale-1.0.0.6
   old-time-1.1.0.2
   pretty-1.1.1.1
   process-1.2.0.0
   rts-1.0
   template-haskell-2.9.0.0
   time-1.4.2
   transformers-0.3.0.0
   unix-2.7.0.1

So I guess the only problems I actually have are the hoopl and transformer packages.

comment:9 Changed 12 months ago by td123

I read the ghc.mk file and it looks like I might know why transformers and hoopl are both getting installed for me.

The following line adds transformers and hoopl to the "regular libs to install" which installs the libs and doesn't do anything special with them.

https://github.com/ghc/ghc/blob/ghc-7.8/ghc.mk#L419

The following line adds a list of packages to install the dynamic libs for since they are dependencies for the compiler:

https://github.com/ghc/ghc/blob/ghc-7.8/ghc.mk#L435

Also, I did some investigation and it looks like hoopl and transformers are really only required for the compiler. Transformers is also a dependency of haskeline. So really both of them should be added to line 435 rather than line 419 so that they belong to REGULAR_INSTALL_DYNLIBS

comment:10 follow-up: Changed 12 months ago by td123

Investigated this some more this morning:

The following commits look like they wanted to make transformers and hoopl installed with ghc on purpose:

"Ship transformers with GHC"
https://github.com/ghc/ghc/commit/71feb1025eed0c3cc849c85e2b00e16bc1a21790

"Merge in new code generator branch" (introduces hoopl)
https://github.com/ghc/ghc/commit/889c084e943779e76d19f2ef5e970ff655f511eb

And looking back at the 7.8.1 release notes, hoopl is mentioned as a package.
http://www.haskell.org/ghc/docs/7.8.1-rc1/html/users_guide/release-7-8-1.html

But it doesn't look like transformers is mentioned.

So I'm not sure if this is an issue anymore. Looks like they were both meant to be added to ghc. Hoopl is mentioned in the release notes. But transformers isn't but it seems like it should be. Thoughts?

comment:11 in reply to: ↑ 10 ; follow-up: Changed 12 months ago by hvr

Replying to td123:

So I'm not sure if this is an issue anymore. Looks like they were both meant to be added to ghc. Hoopl is mentioned in the release notes. But transformers isn't but it seems like it should be. Thoughts?

To be honest, I'm rather confused about your problems with transformers and hoopl, as those have been in fact become officially GHC-bundled (and registered) core libraries since 7.8.1 and 7.2.1 respectively (see also Commentary/Libraries/VersionHistory)

The problems are (afaics) really just only with haskeline ,terminfo, and xhtml, for which only the DSOs get installed but nothing else is done.

comment:12 in reply to: ↑ 11 Changed 12 months ago by td123

All I'm saying there is that the release notes do not include transformers as a library that will be included with ghc.

If you visit: http://www.haskell.org/ghc/docs/7.8.1/html/users_guide/index.html it doesn't mention any transformers library when it should under section 1.5.3

The rest of the packages I mentioned don't seem to be a problem at all.

comment:13 Changed 12 months ago by slyfox

  • Cc slyfox@… added

hoopl/transformers are only used by exported 'ghc' library.
To make users load it into, say, ghci you need to export it completely (ghci -package ghc).

xhtml used by haddock binary, terminfo/haskeline used by ghc binary (--interactive mode, aka GHCi).
Thus they need only dynamic libs, no need to export them.

comment:14 Changed 11 months ago by juhpetersen

Here is the small patch to ghc.mk I have been using for Fedora builds so far:

https://github.com/fedora-haskell/ghc/blob/master/ghc-package-xhtml-terminfo-haskeline.patch

comment:15 Changed 11 months ago by juhpetersen

Sergei asked on ghc-dev list for more context.

This is about shared libaries: of course there is no impact on platforms without them.
Starting with ghc-7.8, ghc's own binaries are also dynamically linked on platforms with dyn Haskell libs.

Shipping the shared library without devel files on Linux seems quite unnatural and awkward.

Last edited 11 months ago by juhpetersen (previous) (diff)

comment:16 Changed 11 months ago by nomeata

Shipping the shared library without devel files on Linux seems quite unnatural and awkward.

I’m not sure about this: If the .so file is put in a private location (not /usr/lib, what’s wrong with that)?

I’d actually prefer if GHC would ship as few packages as little, so that as many packages as possible can be packaged and installed in the “common” way.

comment:17 follow-up: Changed 11 months ago by errge

I've been asked to briefly summarize the issue, since simons didn't write too much about it in the ticket itself.

GHC 7.8 uses the xhtml, terminfo and haskeline packages internally. Because of the dynamic ghci transition, the related .so files have to be shipped with the build. By not shipping the devel files (not making the cabal packages visible), we force the distribution vendors to package up these libraries separately and make them available in their repos. Nothing wrong with that of course (they do this for all the other important libraries, like lens and pipes too). The problem is that they can't ship the .so files in the respective packages, because it's usually not allowed for packages to overwrite each other's files in an adhoc manner. So when they build e.g. the xhtml deb or rpm package, they have to make sure that if the resulting .so files have the same name as the ghc shipped ones, then they don't include it in the package. These kind of workarounds seem extremely kludgy and fragile to me. This naming/overwrite issue causes some headaches for NixOS too, just in a slightly different setting.

I see two solutions going forward.

As nomeata said in comment 16, we can simply ship our .so files in a private directory and make our dynamic programs search there (via RPATH or some other trick).

Or we can simply ship these packages with GHC as we ship base, time and so many others. juhpetersen posted a patch for that in comment 14.

Alternatively, we can ignore it and just force people to go with the workarounds. I'd prefer not doing that. There were also some discussion on the mailing list about discontinuing the dynamic change after 7.10 and going back to static, which would of course make this go away.

comment:18 Changed 11 months ago by errge

  • Description modified (diff)
  • Status changed from new to patch

Setting this to "please review", because two different ways have been analyzed, patch has been proposed for one of them and decision is needed from GHC HQ on how to proceed.

If GHC HQ decides on the solution, I'm happy to prepare/review/test solutions on an expedited bases if it has a chance to go into 7.6.3.

comment:19 follow-up: Changed 11 months ago by nomeata

As nomeata said in comment 16, we can simply ship our .so files in a private directory and make our dynamic programs search there (via RPATH or some other trick).

Or via a „private version number” – it should be sufficient if they have a file name that’s different from any filename resulting from a separately packaged library.

comment:20 in reply to: ↑ 19 ; follow-up: Changed 10 months ago by hvr

Replying to nomeata:

As nomeata said in comment 16, we can simply ship our .so files in a private directory and make our dynamic programs search there (via RPATH or some other trick).

Or via a „private version number” – it should be sufficient if they have a file name that’s different from any filename resulting from a separately packaged library.

If the purpose of those DSO are actually not really shared, what about simply linking them statically so that no orphaned DSOs are left over? Is there any value in having DSOs that are used by only one executable?

comment:21 in reply to: ↑ 20 Changed 10 months ago by slyfox

Replying to hvr:

Replying to nomeata:

As nomeata said in comment 16, we can simply ship our .so files in a private directory and make our dynamic programs search there (via RPATH or some other trick).

Or via a „private version number” – it should be sufficient if they have a file name that’s different from any filename resulting from a separately packaged library.

If the purpose of those DSO are actually not really shared, what about simply linking them statically so that no orphaned DSOs are left over? Is there any value in having DSOs that are used by only one executable?

(not very strong pro) Sometimes DSO linking is preferred (to reduce overall code size/exported amount of symbols) of a single module (as a workaround of host OS).

comment:22 in reply to: ↑ 17 Changed 10 months ago by slyfox

Replying to errge:

I've been asked to briefly summarize the issue, since simons didn't write too much about it in the ticket itself.

GHC 7.8 uses the xhtml, terminfo and haskeline packages internally. Because of the dynamic ghci transition, the related .so files have to be shipped with the build. By not shipping the devel files (not making the cabal packages visible), we force the distribution vendors to package up these libraries separately and make them available in their repos. Nothing wrong with that of course (they do this for all the other important libraries, like lens and pipes too). The problem is that they can't ship the .so files in the respective packages, because it's usually not allowed for packages to overwrite each other's files in an adhoc manner. So when they build e.g. the xhtml deb or rpm package, they have to make sure that if the resulting .so files have the same name as the ghc shipped ones, then they don't include it in the package. These kind of workarounds seem extremely kludgy and fragile to me. This naming/overwrite issue causes some headaches for NixOS too, just in a slightly different setting.

I see two solutions going forward.

As nomeata said in comment 16, we can simply ship our .so files in a private directory and make our dynamic programs search there (via RPATH or some other trick).

Or we can simply ship these packages with GHC as we ship base, time and so many others. juhpetersen posted a patch for that in comment 14.

Alternatively, we can ignore it and just force people to go with the workarounds. I'd prefer not doing that. There were also some discussion on the mailing list about discontinuing the dynamic change after 7.10 and going back to static, which would of course make this go away.

Thanks for the detailed explanation!

ghc's xhtml is already a private lib. You can't accidentally use/link against it:

/usr/lib64/ghc-7.8.2/xhtml-3000.2.1/libHSxhtml-3000.2.1-ghc7.8.2.so

On gentoo we just install external haskell libs in other location:

/usr/lib64/xhtml-3000.2.1/ghc-7.8.2/libHSxhtml-3000.2.1-ghc7.8.2.so

and it's the package registered with cabal/seen with ghci -package xhtml.
(As you see gentoo ships both .so files, allowing user to rebuild xhtml
with any random ghc options out there).

I see a problem in one single case: if you install haskell libs in ghc's private subdir.
I'd strongly suggest against that practice and pick any other random dir out there.

If we would support multiple ghc versions --prefix might be a

/usr/$(libdir)/gentoo-ghc-7.8.2

Anyway, cabal can find them anywhere.

Or i've misunderstood the clash problem completely?

comment:23 Changed 10 months ago by Austin Seipp <austin@…>

In 4caadb7cbee5c176abb99df25c4cc1657ae57f40/ghc:

Ship xhtml, terminfo, haskeline (#8919)

Signed-off-by: Austin Seipp <[email protected]>

comment:24 Changed 10 months ago by thoughtpolice

  • Status changed from patch to merge

comment:25 Changed 10 months ago by slyfox

Ok, so mechanics to break ghc is to install package in ghc's private dir:
http://www.reddit.com/r/haskell/comments/22wu92/problems_installing_ghc_782_on_debian_ubuntu_x86/

comment:26 Changed 10 months ago by thoughtpolice

  • Resolution set to fixed
  • Status changed from merge to closed
Note: See TracTickets for help on using tickets.