Opened 3 years ago

Last modified 17 months ago

#10077 new bug

Providing type checker plugin on command line results in false cyclic import error

Reported by: jbracker Owned by:
Priority: normal Milestone:
Component: Compiler (Type checker) Version: 7.11
Keywords: typechecker plugin cycle imports Cc: adamgundry
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Incorrect warning at compile-time Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

I am playing around with the new type checker plugins using the current development branch of GHC. When I supply the plugin module to GHC through the command line I get a cyclic import error message, which I think is false. If I supply the same plugin module using a pragma I do not get an error message.

Minimal Example (Command Line)

MyPlugin.hs:

module MyPlugin ( plugin ) where

import Plugins ( Plugin, defaultPlugin )

plugin :: Plugin
plugin = defaultPlugin

Test.hs:

module Test where

main :: IO ()
main = return ()

Command line call of GHC:

~/ghc/inplace/bin/ghc-stage2 \
  --make 
  -package ghc-7.11.20150209 \
  -fplugin=MyPlugin \
  Test.hs

Results in the following error

Module imports form a cycle:
  module ‘MyPlugin’ (./MyPlugin.hs) imports itself

which does not seem reasonable to me understand.

Minimal example (pragma)

On the other hand, if I change Test.hs to the following

{-# OPTIONS_GHC -fplugin MyPlugin #-} 
module Test where

main :: IO ()
main = return ()

and calling GHC like this

~/ghc/inplace/bin/ghc-stage2 \
  --make \
  -package ghc-7.11.20150209 \
  -dynamic \
  Test.hs

it works.

I did not change MyPlugin.hs.

Note

  1. Using -fplugin=MyPlugin vs. -fplugin MyPlugin does not make a difference.
  2. The command line example behaves the same independent of whether I supply the -dynamic flag or not.
  3. I had to add the -dynamic flag, because otherwise GHC will complain that:
    <no location info>:
        cannot find normal object file ‘./MyPlugin.dyn_o’
        while linking an interpreted expression
    
    This might be a long shot, but maybe using the -fplugin option should imply the -dynamic flag?

Change History (10)

comment:1 Changed 3 years ago by jbracker

Cc: adamgundry added

comment:2 Changed 3 years ago by adamgundry

This is arguably correct behaviour, though I agree that it's less than ideal. The combination of --make and -fplugin is often troublesome, because -fplugin MyPlugin essentially imports MyPlugin in every module being compiled. (As usual, when you do ghc --make <flags> Test.hs the dependencies of Test.hs are compiled with the same command-line flags <flags>.) Thus your first example ends up compiling MyPlugin with -fplugin MyPlugin, which is obviously cyclic.

If you're defining and using a plugin in a single package, I recommend using OPTIONS_GHC pragmas in the modules that rely on the plugin, rather than supplying -fplugin on the command line. As you've discovered, this avoids the problem. Alternatively, you can define the plugin in one package and use it in another, then it is easy to compile the first package without -fplugin, and you can use it globally in the second package.

I guess we should make the documentation clearer about how to avoid this problem, and perhaps change the cyclic import message to be more informative in the presence of plugins.

Regarding -dynamic, I guess we should do something similar to #8180, since it's basically the same issue: if you use -fplugin MyPlugin anywhere in the module graph, the compilation manager should make sure that MyPlugin is compiled with -dynamic or -dynamic-too. I don't know how tricky this would be to implement, though.

comment:3 Changed 3 years ago by adamgundry

I've added a section to the typechecker plugins wiki page documenting these issues. Suggestions for improvement are welcome!

comment:4 in reply to:  2 ; Changed 3 years ago by jbracker

Replying to adamgundry:

This is arguably correct behaviour, though I agree that it's less than ideal. The combination of --make and -fplugin is often troublesome, because -fplugin MyPlugin essentially imports MyPlugin in every module being compiled.

I have trouble understanding why a plugin needs to be imported to the module that uses it. As far as I understand GHC dynamically loads the plugin module and uses it, but I don't see why the plugin needs to be imported to the compiled module, since the modules do not have access to the plugins functionality anyway. Is this required because the plugin might import other modules in scope? Or is there some other technical reason?

I guess we should make the documentation clearer about how to avoid this problem, and perhaps change the cyclic import message to be more informative in the presence of plugins.

Yes, I agree. Thank you for extending the wiki page!

Regarding -dynamic, I guess we should do something similar to #8180, since it's basically the same issue: if you use -fplugin MyPlugin anywhere in the module graph, the compilation manager should make sure that MyPlugin is compiled with -dynamic or -dynamic-too. I don't know how tricky this would be to implement, though.

Looking at the changes from that ticket it does not seem to be too much work: e25af05656b496b997c8f3520e5ac8e377a68e7b/ghc Though there might be lingering problems arising from this.

Replying to adamgundry:

I've added a section to the typechecker plugins wiki page documenting these issues. Suggestions for improvement are welcome!

Maybe a link to this ticket, in case someone wants to know more about the issue?

Last edited 3 years ago by jbracker (previous) (diff)

comment:5 in reply to:  4 ; Changed 3 years ago by adamgundry

Replying to jbracker:

Replying to adamgundry:

This is arguably correct behaviour, though I agree that it's less than ideal. The combination of --make and -fplugin is often troublesome, because -fplugin MyPlugin essentially imports MyPlugin in every module being compiled.

I have trouble understanding why a plugin needs to be imported to the module that uses it. As far as I understand GHC dynamically loads the plugin module and uses it, but I don't see why the plugin needs to be imported to the compiled module, since the modules do not have access to the plugins functionality anyway. Is this required because the plugin might import other modules in scope? Or is there some other technical reason?

To try to clarify: it's not exactly true that the plugin needs to be imported into a module that uses it, although I believe the implementation works more-or-less like that, with some cleverness to avoid e.g. typeclass instances being brought in. The point is that if module M uses a plugin in module P, then we have to dynamically load P in order to typecheck M. Hence P cannot depend (directly or indirectly) on M!

When -fplugin=P is used on the command-line, GHC tries to load P for every module being compiled, which might include P itself. This isn't exactly a cyclic import, but it's pretty close.

I've added a section to the typechecker plugins wiki page documenting these issues. Suggestions for improvement are welcome!

Maybe a link to this ticket, in case someone wants to know more about the issue?

Yes. Done.

comment:6 in reply to:  5 Changed 3 years ago by jbracker

Replying to adamgundry:

Thank you for the clarification. I understand now.

Maybe a link to this ticket, in case someone wants to know more about the issue?

Yes. Done.

Thanks!

Last edited 3 years ago by jbracker (previous) (diff)

comment:7 Changed 3 years ago by ezyang

adamgundry: I don't think your diagnosis of the problem is correct. Isn't the case here that, when GHC attempts to compile a plugin, it shouldn't use the -fplugin arguments for the compilation? Simple stage restriction: you can't use a plugin on itself.

comment:8 in reply to:  7 Changed 3 years ago by adamgundry

Replying to ezyang:

Isn't the case here that, when GHC attempts to compile a plugin, it shouldn't use the -fplugin arguments for the compilation? Simple stage restriction: you can't use a plugin on itself.

That's true, but I'm not quite sure what you're suggesting: do you mean that GHC should silently ignore the -fplugin argument when compiling the plugin itself (and its dependencies)? I guess that would be more convenient in this case, although I imagine it would be counterintuitive in other cases (e.g. if you were trying to use a plugin on a module without realising the plugin depended on that module). Perhaps those other cases are sufficiently obscure that it is worth doing, and could perhaps be combined with turning on -dynamic automatically.

comment:9 Changed 17 months ago by osa1

Sorry if I'm missing anything, but I don't think this makes sense and it's not arguable that this is correct behaviour. First, there's no loop here. M uses P, but P doesn't import M. It should just work no matter what.

Second, say there's a loop: M uses P and P imports M. Then I think we _should_ just stop the compilation with the error message currently being printed. The reason is because a Core plugin can generate arbitrary code, so while ignoring the plugin when compiling M may be OK, we can't know in general that this will work (e.g. when the library relies on code generated by the plugin). So maybe we have an excuse here and bailing out is just OK.

Also, -fplugin is a flag, so it can be handled by a build system. So if someone wants to "boot" P with M compiled using the plugin itself, maybe they should do this:

  1. Build M without P (build system doesn't use -fplugin)
  2. Build P with M built in step (1)
  3. Build M again, this time with P (build system uses -fplugin)
  4. Build P again, using M built in step (3)

What do you think? Does that sound reasonable?

I want to fix this at least for the non-recursive case but I don't know anything about this code so it may take some time.

comment:10 Changed 17 months ago by rwbarton

osa1, the cycle is that ghc thinks you have told it to compile module P using the plugin P. It's not literally an import cycle, but it is a dependency cycle of another kind. In your duplicate ticket #12204, ghc thinks it has to recompile Plugin because of the new command-line flag -fplugin=Plugin!

By the way, these examples are illegal according to the documentation, which states the "module must be a member of a package registered in GHC’s package database".

do you mean that GHC should silently ignore the -fplugin argument when compiling the plugin itself (and its dependencies)?

What happens if there are multiple -fplugin arguments? We can't compile each plugin using the other plugins, as that is again a cycle. We could compile each plugin using the plugins specified before it on the command line, but that seems overly complicated and not necessarily what the user wants. Maybe the user wants to compile their program with plugin A (only), but to compile plugin A using plugin B. In general there is no good reason to think that the flags used to build a plugin should have any relation to the flags used to build the program that uses the plugin. In a cross-compilation scenario, even the compiler is different!

I suggest that ghc --make should just not follow dependencies to plugins. If you want to build a plugin and a program that uses the plugin, you need two invocations of ghc --make.

Note: See TracTickets for help on using tickets.