Extend the plugin mechanism to access program representation

mentioned in issue #14443

changed weight to 5

added GHC API Tfeature request Trac import labels

Hmmm, interesting. Adding ezyang and Alan Zimmerman as I know they have both thought a fair amount about tooling.

The description on the wiki page, specifically of the plugin interface itself, seems a bit terse. Do you think you could lay out more precisely what these new plug-ins will be be able to do? Do you anticipate being able to modify the AST or is this strictly for tooling that wishes to I spect the AST?

A few more words on why front-end plug-ins are insufficient would also be helpful; is the problem that you want your plugin to run during the course of normal compilation?

I can imagine tools that modify these representations as well. The main purpose of the plugins being able to change the input is that it seemed a bit arbitrary to limit the plugins in this way. This is also present in Edsko's version.

Yes, basically I would like to be able to run my tool during a normal (enhanced) compilation.

I can imagine tools that modify these representations as well

Sure. The one thing I would really like to see considered is recompilation checking. Currently the existing plugin types are borderline unusable due to the lack of recompilation avoidance (see #7414 (closed)).

While you don't need to solve this issue in your patch (doing so will likely require deeper changes in GHC), we should try to think through which information the plugin will need to provide for the compiler to be able to assess whether recompilation is necessary and design the interface with this need in mind. This will avoid backwards-incompatible interface changes in the future.

Writing down a few motivating examples of applications of the new plugin types may be a good place to start.

The plugin should be able to tell if it requires recompilation due to change in plugin arguments or external factors, and GHC should be aware when the plugin itself changes.

This doesn't really have anything to do with the relative merits/demerits of the proposal, but I just wanted to point out:

Frontend plugins are not applicable, because their usage changes the major mode of the compiler. So if the tool developer wants to go on with the compilation procedure, he must replicate what GHC would do if the frontend plugin was not used. Furthermore, it can't be inserted into a normal build environment, since using the --frontend flag clashes with other mode flags like --make or --interactive.

Out of the box these statements are true, but as I mention in http://blog.ezyang.com/2017/02/how-to-integrate-ghc-api-programs-with-cabal/ there are some relatively simple tricks you can play to solve this problem. As another example, https://github.com/ezyang/ghc-shake comes with a wrapper program which overrides the meaning of --make but otherwise keeps all other major modes the same.

Of course, if the frontend plugin is replacing a mode like --make, it needs to understand all of the same arguments that --make would have understood, but that is what you would expect, no?

Your tutorial is very useful, I also tried it when I was experimenting of using frontend plugins for the same problem that I would like to solve with the source plugins.

I think it is possible to extract the typechecked AST with a frontend plugin and a wrapper program setting up the frontend compiler hook. However for my tool I need other information, including loaded interfaces, renamed AST and splices evaluated. I think other tool developers will find the availability of those useful as well.

From a design perspective, I would like to extend GHC in a way that doesn't alter the compilation process, and I think that normal plugins are better for that than frontend plugins. It is also generally easier to setup the build system to use a few compiler flags than to use a different compiler, and the user will not have problems if he has more than one GHC installed.

Overall you are right in that the wiki page oversimplifies the matter, I will try to improve it. Thank for pointing out!

Replying to [ticket:14709#comment:148077 bgamari]:

Writing down a few motivating examples of applications of the new plugin types may be a good place to start.

Some of these tools exist using external libraries, but by using GHC as the backend they may be applied to more complex projects. I'm mostly interested in editing and code transformation tools, so here is a few example:

**Auto-complete**: Records the names available in a scope using interface files and type checked AST.
**Semantics aware formatting**: Auto-formatter that uses fixity information to break up complex operator expressions. Uses parsed and type checked representation.
**Semantics aware syntax highlight**: Check interfaces for recording extra information about names and use this in an editor to enhance highlighting names.
**Code generator**: Generating pattern matches and expressions in a dependent-typed fashion.
**Auto-correct**: Automatic fix common errors. Uses -fdefer-* flags, extracts type-checked representation. Also checks interfaces files.
**Refactoring tool**: Building a project database of syntactic and semantic information to rewrite source code. Uses all versions of syntax tree, splices, interfaces.

My suggestion for recompilation checking is:

-fplugin change -> recompile

The hash of used plugin changes -> recompile

-fplugin-opt change -> ask the plugin for each module whether recompilation is needed

These examples are useful for discussing the problems with recompilation but I don't think they should stop the progress of this patch.

If we all agree that this is a good direction, which I think we should, we should consider each of the proposed plugins and decide whether each of them is desirable in turn. To summarise, here are the five proposed extensions.

 parsedResultAction :: [CommandLineOption] -> ModSummary -> HsParsedModule 
 needsRenamedSyntax :: [CommandLineOption] -> ModSummary -> Hsc Bool
 typeCheckResultAction :: [CommandLineOption] -> ModSummary -> TcGblEnv 
 spliceRunAction :: [CommandLineOption] -> LHsExpr GhcTc -> TcM (LHsExpr GhcTc)
 interfaceLoadAction :: forall lcl . [CommandLineOption] -> ModIface 
                                       -> IfM lcl ModIface

I think that parsedResultAction, typecheckResultAction and splitRunAction are well motivated hooks which we should definitely include.

I am indifferent towards interfaceLoadAction, I don't see a more direct place to implement this but feel the hook should instead run after the interface is type checked.

I won't accept a patch which includes the needsRenamedSyntax hook which is ad-hoc.

Can other interested parties please comment on these specific so we can move this patch forward in a reasonable time frame?

As a fellow tool-writer, in principle I am in favour of this patch.

One thing that confuses me though, is the motivating examples are to extract information, but the plugins allow changes on the fly.

I am concerned that we open a pandora's box of arbitrary transformations via plugins, that will make it ***harder*** for tool writers, as we now need to know what other things are happening under the hood.

That said, I do believe it would be incredibly useful to be able to make changes to some part of a given AST, and push it down the pipeline to either validate a proposed change to the code, or to gather type information based on the change. And to do that modification is required.

First: thanks, lazac, for the patch. GHC's plugin framework has long been waiting for someone to flesh it out and this patch looks like a great step in that direction. However, GHC's plugin interfaces are part of the compiler's public interface and as such will be supported for the foreseeable future. Consequently, we want to be careful that we don't end up introducing an interface that we will later come to regret.

I discussed your patch with the Simons this morning and there was a general consensus that the proposal and the Haskell tooling community (some of whom have yet to comment here) would both benefit immensely from having the design in written form as a formal proposal. This would allow us to collect enough use-cases to better evaluate your proposed interfaces. I would be happy to lend a hand in developing, proof-reading, or otherwise facilitating this proposal if you like to help.

Yes, GHC's API and plugin interface exists to make it possible (and as easy as possible) to build new tools on top of GHC. It has evolved over time, but needs love, and thoughtful development. So thank you for offering some concrete suggestions.

Clearly we want any new extensions to serve as wide a group of clients as possible. Sharing the proposals as a GHC proposal has proved to be a very effective way to improve ideas by discussing them with others.

It would really help to have a couple of example use-cases to motivate each of the proposed plugin extensions.

Thanks!

I created a formal propasal as it was requested. It is available in the pull request https://github.com/ghc-proposals/ghc-proposals/pull/107. Feel free to contribute to the proposal.

mentioned in issue #14750 (closed)

mentioned in commit c2783ccf

closed

changed milestone to %8.6.1

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed

added ghc proposal label

added Pnormal label

Trac field	Value
Version	8.2.2
Type	FeatureRequest
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	GHC API
Test case
Differential revisions
BlockedBy
Related
Blocking
CC	ulysses4ever
Operating system
Architecture

Extend the plugin mechanism to access program representation

Child items 0

Activity

Extend the plugin mechanism to access program representation

Relates to

Activity