Opened 6 years ago

Last modified 6 years ago

#2630 new feature request

installed packages should have a src-dirs field, for access to optionally installed sources

Reported by: claus Owned by:
Priority: normal Milestone:
Component: Package system Version: 6.9
Keywords: Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Test Case:
Blocked By: Blocking:
Related Tickets: Differential Revisions:

Description

In a typical ghc+packages installation, there are no Haskell sources. That makes life difficult for source-code-based tools, such as Haddock or HaRe (as far as I can see from --show-iface output, not even the import hierarchy is visible anymore?).

(1) It would be useful if Ghc Api clients and Ghc users could easily find the matching source for installed packages, via a src-dirs field in the installed package description (if field is empty, no source has been installed).

(2) It would be useful if matching sources could easily be added for installed packages. This would require editing the package description. Should there be an edit command for ghc-pkg?

(3) An alternative to a src-dirs field would be to install sources in a separate source-package database, as long as it is easy to find the source from the binary package.

Change History (14)

comment:1 follow-up: Changed 6 years ago by simonmar

  • difficulty set to Unknown

Sources aren't enough for most uses - you also need the .cabal file, and you might also need other package bits. In short, you really need the whole Cabal sdist to make any sense of the sources.

Perhaps the right thing is to include the URL from which you can download the package sources? cabal-install could automatically set the field, for example.

comment:2 in reply to: ↑ 1 Changed 6 years ago by claus

Replying to simonmar:

Sources aren't enough for most uses - you also need the .cabal file, and you might also need other package bits. In short, you really need the whole Cabal sdist to make any sense of the sources.

Oh, bother. You're right, of course.

Perhaps the right thing is to include the URL from which you can download the package sources? cabal-install could automatically set the field, for example.

.cabal itself already has a package-url field. What I was hoping for was the actual locally available source corresponding to an installed package - the one that a source-based tool might need to find (without going online) if interface files are not enough. There is a slight conflict of interests here, between wanting a pure sdist from which other variations of the package could be recreated, and wanting a pre-configured sdist that exactly matches the installed package. Perhaps pointing to a locally installed sdist, and keeping the configuration in the sdist, might suffice.

Btw, I've filed a corresponding cabal ticket, as well, since cabal and ghc-pkg probably need to coordinate on this one (one providing the field, the other filling it).

comment:3 Changed 6 years ago by simonmar

  • Milestone set to _|_

It's hard to know exactly what to do in this area until we have a prospective client actively asking for something. So I'll drop this into the _|_ milestone until we have more information.

comment:4 Changed 6 years ago by simonmar

  • Architecture changed from Unknown to Unknown/Multiple

comment:5 Changed 6 years ago by simonmar

  • Operating System changed from Unknown to Unknown/Multiple

comment:6 follow-up: Changed 6 years ago by claus

I was of course thinking of a future HaRe/Ghc, but if you dislike vague future references, here are two simple use cases:

(a) just last week, I wanted to trace the import hierarchies of my 6.8.3 installation, for which I don't have sources

(b) even haddock needs package sources long enough to create its *.haddock files, and if any of the dependencies was installed without being haddocked, I need to be able to remedy that

In cases like (a), I'd like to be able to install the package sources once, and then for an import tracer to be able to look up the correct package sources whenever there are cross-package references in the code. In cases like (b), I'd like to be able to add .haddock files to installed packages without having to worry about getting the right sources, because the right sources were installed with the package.

I'm not thinking of anything complex yet, for starters just: "the local sources for this package are in this directory: <dir>" or "local sources for this package have not (yet) been installed" would help. Then the tools would have to deal with those sources, and with associating them with the binary packages, but at least the tools would be able to find the sources easily.

For Ghc Api clients like Ghc, I'd hope that extracting the source dirs from the package database, adding them to the import path and asking for -fforce-recomp would be sufficient, or that -fforce-recomp would look up the sources itself, but that might be optimistic?-)

I'm not sure whether it would be sufficient, but is there any reason not to have a src-dirs field in installed packages descriptions?

comment:7 in reply to: ↑ 6 ; follow-up: Changed 6 years ago by simonmar

Replying to claus:

(a) just last week, I wanted to trace the import hierarchies of my 6.8.3 installation, for which I don't have sources

Well, if we didn't have any orphan instances you wouldn't need to do that :-)

In fact GHC does keep track of the import hierarchy in the .hi files, so you can do this using the GHC API (perhaps not everything you need is exposed via the official parts of the API currently though).

(b) even haddock needs package sources long enough to create its *.haddock files, and if any of the dependencies was installed without being haddocked, I need to be able to remedy that

I don't buy this as an important use case. You should have Haddocked the package when you installed it, or it should be possible to download the package again, 'cabal haddock' and then install just the Haddock docs.

For Ghc Api clients like Ghc, I'd hope that extracting the source dirs from the package database, adding them to the import path and asking for -fforce-recomp would be sufficient, or that -fforce-recomp would look up the sources itself, but that might be optimistic?-)

Sure you can put sources in your import path and then GHC will pick them up instead of the package modules, no -fforce-recomp necessary. But I think you're forgetting about the package metadata again - just having the sources won't be enough in many cases.

I'm not sure whether it would be sufficient, but is there any reason not to have a src-dirs field in installed packages descriptions?

I'm not convinced it's useful.

comment:8 in reply to: ↑ 7 ; follow-up: Changed 6 years ago by claus

Replying to simonmar:

In fact GHC does keep track of the import hierarchy in the .hi files, so you can do this using the GHC API (perhaps not everything you need is exposed via the official parts of the API currently though).

Interesting. From --show-iface, I got the impression that cross-package imports weren't tracked?

(b) even haddock needs package sources long enough to create its *.haddock files, and if any of the dependencies was installed without being haddocked, I need to be able to remedy that

I don't buy this as an important use case. You should have Haddocked the package when you installed it, or it should be possible to download the package again, 'cabal haddock' and then install just the Haddock docs.

There is no guarantee that all installed packages were haddocked. These days, one is locked into a specific haddock version, but I used to experiment with several. When developing packages, I change and reinstall them often, and I do sometimes keep separate package versions installed with separate compiler versions, so I'd have to reconstruct the exact package version and configuration options before trying to add haddocks. If one wanted to apply the haddock shipping with one version of ghc to sources hosted in another version of ghc, one would need access to the sources.

For Ghc Api clients like Ghc, I'd hope that extracting the source dirs from the package database, adding them to the import path and asking for -fforce-recomp would be sufficient, or that -fforce-recomp would look up the sources itself, but that might be optimistic?-)

Sure you can put sources in your import path and then GHC will pick them up instead of the package modules, no -fforce-recomp necessary. But I think you're forgetting about the package metadata again - just having the sources won't be enough in many cases.

I'm not that forgetful;-) cabal calls ghc --make after setting things up, and at that stage, Ghc Api clients either have a chance, or they won't work with any form of the source.

I'm not sure whether it would be sufficient, but is there any reason not to have a src-dirs field in installed packages descriptions?

I'm not convinced it's useful.

The point is to do the source download only once, then to cache the sources somewhere safe, and to make sure that tools can find the sources easily.

I don't understand what is wrong with wanting to have the sources of open-source software available and easily accessible. But if you have alternative suggestions for helping source-based tools along, I'd be interested to hear them.

comment:9 in reply to: ↑ 8 Changed 6 years ago by simonmar

Replying to claus:

Interesting. From --show-iface, I got the impression that cross-package imports weren't tracked?

Try it with 6.10.

There is no guarantee that all installed packages were haddocked. These days, one is locked into a specific haddock version, but I used to experiment with several. When developing packages, I change and reinstall them often, and I do sometimes keep separate package versions installed with separate compiler versions, so I'd have to reconstruct the exact package version and configuration options before trying to add haddocks. If one wanted to apply the haddock shipping with one version of ghc to sources hosted in another version of ghc, one would need access to the sources.

Right, but it's not just the sources - as you say, you need to construct the exact package version and config, so you'd need some kind of preprocessed sources and someone has to know exactly how to invoke Haddock (Cabal doesn't - it knows about Cabal packages, not source caches).

Sure you can put sources in your import path and then GHC will pick them up instead of the package modules, no -fforce-recomp necessary. But I think you're forgetting about the package metadata again - just having the sources won't be enough in many cases.

I'm not that forgetful;-) cabal calls ghc --make after setting things up, and at that stage, Ghc Api clients either have a chance, or they won't work with any form of the source.

I'm afraid I just don't have a clue what you're talking about! What GHC API clients? A chance to do what? Cabal doesn't invoke any GHC API clients other than GHC.

If you want to make some progress here, you'll have to provide a compelling use case, and describe it in very concrete terms. Without that, there's a danger that we'll provide something that nobody wants or can use.

comment:10 Changed 6 years ago by NeilMitchell

I think I understand the kind of things Claus wants to do (since I want to do similar things), but I think Simon is right. A package has little relation to the source code that built it - a package is related to the cabal tarball and configuration.

One thing I (and Tim as well) want GHC to include with all packages is the External Core files. We are going to file a separate ticket aiming for 6.12, but it has a similar feel of maintaining additional information with a package, beyond that needed for compilation - so maybe these things need considering together.

comment:11 Changed 6 years ago by malcolm.wallace@…

Another important use-case is if I have lots of libraries installed by Cabal, then at some later stage I realise that I want profiling versions of all of them. If the original sources were still lying around in a canonical location, this would probably be pretty easy. One thing I definitely don't want is for cabal-install to download and install different versions of those libraries, just because newer ones are available.

comment:12 Changed 6 years ago by NeilMitchell

Another use case would be for people who want to upgrade to GHC HEAD once a day and would like to be able to install all the packages they had yesterday without much trouble. I currently have a script that does this, but its a lot more work than if GHC kept the tarballs around.

comment:13 Changed 6 years ago by simonmar

Both of these sound like they would be easily solved by cabal-install caching the source tarballs for packages it downloads or builds, and having a facility for rebuilding a set of packages. I'm afraid I still don't see how adding anything to the package database will help here.

Suppose there was a cache for package source tarballs somewhere on the system, and for each installed package you will either find a source tarball in the cache, or you can get it from Hackage. Now, you know which package versions are installed, if you know where the cache is you can find the source for any installed package. Isn't that be enough for all of the uses listed so far? And all of this can be managed by cabal-install, with no cooperation from GHC at all.

What's more, it's already there - have a look in ~/.cabal/packages! That's only for downloaded packages currently, but it would be possible to stash the sdist for any package you build in there.

So can't all this be done by cabal-install?

comment:14 Changed 6 years ago by simonmar

  • Component changed from GHC API to Package system
Note: See TracTickets for help on using tickets.