Opened 2 years ago

Closed 11 months ago

Last modified 2 weeks ago

#10598 closed bug (fixed)

DeriveAnyClass and GND don't work well together

Reported by: osa1 Owned by: RyanGlScott
Priority: normal Milestone: 8.2.1
Component: Compiler Version: 7.11
Keywords: Generics, deriving Cc: jpm@…, oerjan
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case: deriving/should_run/T10598_bug
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D2280
Wiki Page:

Description (last modified by bgamari)

I think we definitely have a bug here, but I'm not sure what it really is.

Here's the program:

newtype MyMaybe a = MyMaybe (Maybe a)
  deriving (Functor, Show)

main = print $ MyMaybe $ Just (10 :: Int)

I'm using GHC 7.10.1.

➜  deriveany_bug  ghc --make -fforce-recomp Test.hs -XDeriveAnyClass -XGeneralizedNewtypeDeriving
[1 of 1] Compiling Main             ( Test.hs, Test.o )

Test.hs:2:13:
    Can't make a derived instance of ‘Functor MyMaybe’
      (even with cunning newtype deriving):
      You need DeriveFunctor to derive an instance for this class
      Try GeneralizedNewtypeDeriving for GHC's newtype-deriving extension
    In the newtype declaration for ‘MyMaybe’

Just to try, changing argument order:

➜  deriveany_bug  ghc --make -fforce-recomp Test.hs -XGeneralizedNewtypeDeriving -XDeriveAnyClass
[1 of 1] Compiling Main             ( Test.hs, Test.o )

Test.hs:2:13:
    Can't make a derived instance of ‘Functor MyMaybe’
      (even with cunning newtype deriving):
      You need DeriveFunctor to derive an instance for this class
      Try GeneralizedNewtypeDeriving for GHC's newtype-deriving extension
    In the newtype declaration for ‘MyMaybe’

It works fine if I remove DeriveAnyClass:

➜  deriveany_bug  ghc --make -fforce-recomp Test.hs -XGeneralizedNewtypeDeriving                 
[1 of 1] Compiling Main             ( Test.hs, Test.o )
Linking Test ...

GHC HEAD is failing in exactly the same way.

User manual is saying this in 7.5.6:

In case you try to derive some class on a newtype, and -XGeneralizedNewtypeDeriving is also on, -XDeriveAnyClass takes precedence.

But then why is it telling me to enable GeneralizedNewtypeDeriving in the error message? Even if I already enabled it?

Also, maybe it could try GND when DeriveAnyClass fails? Because the doc is saying DeriveAnyClass has precedence but doesn't specify what happens if it fails.

EDIT: I'd like to work on this myself if experts here help me figuring the right behavior here.

Change History (55)

comment:1 Changed 2 years ago by osa1

Description: modified (diff)

comment:2 Changed 2 years ago by rwbarton

The documentation says "With -XDeriveAnyClass you can derive any other class", where "other" seems to allude to the classes which can be derived by GHC extensions as described in sections 7.5.3 and 7.5.4 (we have to ignore section 7.5.5 here since the interaction with GND is described explicitly), as well as, presumably, the classes which can be derived in Haskell 2010.

And, in fact, if you try to compile your test case with -XDeriveAnyClass only, GHC tells you

Test.hs:2:13:
    Can't make a derived instance of ‘Functor MyMaybe’:
      You need DeriveFunctor to derive an instance for this class
      Try GeneralizedNewtypeDeriving for GHC's newtype-deriving extension
    In the newtype declaration for ‘MyMaybe’

which is consistent with this interpretation (it's the same error you get without -XDeriveAnyClass).

So, I would expect that in all cases of deriving (Functor), the presence of -XDeriveAnyClass should have no effect.

comment:3 Changed 2 years ago by goldfire

It sounds to me, from the original summary, that GHC is behaving as documented, with the exception of a poor choice of error message. Fixing the error message is one way forward. But -- if you're keen on this sort of thing -- what would be even better is to come up with a way for users to direct GHC in this matter.

GHC now has 3 distinct ways of deriving classes: the built-in way (extended beyond Haskell 2010 by various extensions), the GND way, and the DeriveAnyClass way. In your example, they are all applicable (at least with -XDeriveFunctor on). And, I believe they will each produce different instances! What I've wanted here is to have some way to control GHC's choice of deriving mechanism, per instance. Something like

newtype MyMaybe a = Mk (Maybe a)
  deriving( {-# GND #-} Functor
          , {-# BuiltIn #-} Show
          , {-# GND #-} Read
          , {-# Any #-} FromJSON )

Now, the user chooses what facility provides the instances. Note that I've done something currently impossible: I've used GND for the Read class. Normally, we don't want this behavior, and (to my knowledge) there's no way to convince GHC to use GND to derive a Read or Show instance. But maybe some user out there does want it.

This, of course, needs to be extended to standalone-deriving, and all the details (particularly, what are the default choices for the deriving mechanism) need to be worked out. If you (or anyone else) wants to have a go at this, I'd happily lend a hand.

Or, if you're not ready to tackle feature design, we'd gladly welcome a patch just to fix the (clearly broken) error message originally reported. Thanks!

comment:4 Changed 2 years ago by osa1

Sorry for late response, I'm hoping to make progress on this,

I was wondering if we can use a fallback mechanism in instance deriving. For example: If GND and DeriveAnyClass are enabled, we know DeriveAnyClass is tried first, but why not try GND when it fails?

There are couple of things that are very inconvenient with the current approach:

  1. If I want to derive two instances for my newtype, one needs GND and one needs DeriveAnyClass, I can't do that and I have to split things into modules which means orphan instances.
  1. I'm tired of adding dozens of LANGUAGE pragmas in every single file, so I was hoping to move those to cabal file. But I can't do that easily because of problems like this.

About the error message: What would be the correct message here? Also, even with just DeriveAnyClass, the error message is weird:

➜  deriveany_bug  ghc --make -fforce-recomp Test.hs -XDeriveAnyClass
[1 of 1] Compiling Main             ( Test.hs, Test.o )

Test.hs:2:13:
    Can't make a derived instance of ‘Functor MyMaybe’:
      You need DeriveFunctor to derive an instance for this class
      Try GeneralizedNewtypeDeriving for GHC's newtype-deriving extension
    In the newtype declaration for ‘MyMaybe’

It first says that I need DeriveFunctor, but then says I should try GND. Is it trying to say that GND implies DeriveFunctor? (which shouldn't be the case according to the user manual) Otherwise which one should I try? This message is confusing IMO.

comment:5 Changed 2 years ago by goldfire

I'm a little confused here. Why is DeriveAnyClass failing here? From the user manual, it seems DeriveAnyClass should always succeed. I think the first step in understanding this all is to figure that out. I've hunted around (wiki:Commentary/Compiler/GenericDeriving#Usingstandardderivingforgenericfunctions and #5462) but I haven't found a proper specification of the feature. It certainly isn't in the user manual. Between those two links, one says that any class can be derived, and the other requires a default. Perhaps it's the latter. Does anyone out there know? Or do you want to go spelunking through the code? One way or the other, we should disentangle this and write up a specification.

comment:6 Changed 2 years ago by osa1

Oh you're right. I forgot that DeriveAnyClass should never fail, so maybe there are multiple bugs involved.

I managed to modify GHC to make this program work:

➜  deriveany_bug  cat Test.hs
newtype MyMaybe a = MyMaybe (Maybe a)
  deriving (Functor, Show)

main = print $ MyMaybe $ Just (10 :: Int)

➜  deriveany_bug  ghc-stage1 --make Test.hs -fforce-recomp -XDeriveAnyClass
[1 of 1] Compiling Main             ( Test.hs, Test.o )
Linking Test ...

➜  deriveany_bug  ./Test
MyMaybe (Just 10)

➜  deriveany_bug  ghc-stage1 --make Test.hs -fforce-recomp -XDeriveAnyClass -XGeneralizedNewtypeDeriving
[1 of 1] Compiling Main             ( Test.hs, Test.o )

Test.hs:2:13: warning:
    Both DeriveAnyClass and GeneralizedNewtypeDeriving are enabled
    Defaulting to the DeriveAnyClass strategy for instantiating Functor
    In the newtype declaration for ‘MyMaybe’

Test.hs:2:22: warning:
    Both DeriveAnyClass and GeneralizedNewtypeDeriving are enabled
    Defaulting to the DeriveAnyClass strategy for instantiating Show
    In the newtype declaration for ‘MyMaybe’
Linking Test ...

➜  deriveany_bug  ./Test
MyMaybe (Just 10)

mkNewTypeEqn function has some error handling for the cases where we have both GND and DeriveAnyClass, but the logic is buggy, I think. It's easy to fix. (I didn't run the whole test suite though, I'm not 100% sure I didn't break anything)

Should I send a patch for reviews or do you think the problem is something else?

comment:7 Changed 2 years ago by goldfire

Cc: jpm@… added

I'm less worried about implementing the correct behavior than figuring out what the correct behavior should be. You're saying that DeriveAnyClass should never fail... but is that different from what 7.10.1 actually does?

I think issuing a warning in this case is a great step. But I'd still love to see a specification (preferably in the user manual) of how all this works. In any case, submitting a patch for review via Phab is never the wrong thing to do. (And, of course, thanks for writing the patch!)

I'm cc'ing Pedro, who (if memory serves) wrote this feature.

(Full disclosure: I was involved in the design of DeriveAnyClass, and I'm sure that at some point in history, I knew the answers to the questions I'm asking. But it's all lost now, and I'm hoping Pedro can fill in the details.)

comment:8 Changed 2 years ago by kosmikus

Ok, here's my take on this.

The behaviour is clearly inconsistent.

Let's compare ghc-7.8.4 and ghc-7.10.1. There are three flags that play a role here, namely GeneralizedNewtypeDeriving (GND), DeriveFunctor (DF), and DeriveAnyClass (DAC). The DAC flag ist not supported by ghc-7.8.4. We have the following results:

ghc-7.8.4 ghc-7.10.1
no flags error (GND) error (GND)
GND ok (GND) ok (GND)
DF ok (DF) ok (DF)
GND + DF ok (GND) ok (GND)
DAC unsupported error (GND)
DAC + GND unsupported error (GND')
DAC + DF unsupported ok (DF)
DAC + GND + DF unsupported warning (??)

So the good news is that without the use of DAC, both GHC versions behave the same. However, in the presence of DAC, several things go wrong.

In the DAC case, an error is reported suggesting the use of GND instead. That's strange, because doing the same thing for a user-defined MyFunctor class will actually happily work. I know we've had a long discussion about the conditions for using DAC in the past. In principle, I'd like it if it wouldn't work on "any" class, but instead would require the presence of defaults. It doesn't make much sense to derive partial class definitions with missing methods. I'm not entirely sure what the reason may have been to lift this restriction.

The DAC + GND case is probably the worst. We get a completely unhelpful error message, saying that GND doesn't work yet suggesting to enable GND. Also, it's a case where the mere presence of DAC makes things break all of a sudden, because removing DAC will happily make GHC use GND and succeed.

In the DAC + DF, we silently prefer the built-in functor deriving, which at least in this case seems useful, because DAC for the built-in functor would yield a partial class definition.

The DAC + GND + DF case yields a warning that indicates that both DAC and GND are available to derive the Functor instance, and that it is defaulting to DAC! Yet it then seems to use DF to derive the actual Functor instance. So again, the outcome isn't too bad, but the warning is clearly misleading.

What to do about all this? I think there are the following issues:

  • Should DAC work only if defaults are present for all methods, or should it work also in other situations? I cannot think of a good reason not to require defaults for all methods right now?
  • I think that if, given by the flags and imposed restrictions, only one of the three deriving methods is available, that one should clearly be taken.
  • If several deriving methods are available, we might still choose one. I think the natural thing is to choose built-in deriving before GND before DAC. In all such cases, it should probably produce a warning.
  • I very much like Richard's idea to allow PRAGMAs to explicitly indicate the desired deriving method in a deriving clause. This would be both a way to change default behaviour used by GHC and to remove warnings, because you'd explicitly document intent.
  • If no deriving method is available but a deriving clause is used, we should suggest both GND or DAC as options, but only if they're actually available.

It's entirely possible that I've overlooked important cases. Suggestions welcome.

comment:9 Changed 2 years ago by simonpj

Your list sounds plausible to me, Andres. Re "I cannot think of a good reason not to require defaults for all methods right now?", I agree, but I assume you mean "either a generic default method or a polymorphic default method is provided". Remember there are those two forms of default method.

Simon

comment:10 Changed 2 years ago by kosmikus

Yes, I think it makes sense to require at least any sort of default definition for all class methods, regardless of whether it is a normal default or a generic default.

It might also be sensible to take MINIMAL pragmas into account. If an explicit non-empty MINIMAL pragma is specified, then I think the intent of the author is to say that it doesn't make sense to give an empty class instance, and similarly, it would then also not make sense to derive it. But I'm open to suggestions. Even if we allow DeriveAnyClass for classes that have e.g. mutually recursive defaults and a non-empty MINIMAL pragma, I guess we'd at least still get a warning reported for the derived instance.

comment:11 Changed 2 years ago by simonpj

Usually the MINIMAL thing means "you must give non-default definitions for at least these methods". That's easy to understand in the absence of generic-default methods, because the normal (or polymorphic) default methods invariably work by invoking other methods in the same class or superclass.

But given generic-default methods, it's not so clear to me. I suspect that MINIMAL should not complain if you fail to give explicit code for a method that has a generic default, because the generic default is capable of doing type-specific stuff. (I have not checked what really happens.)

The manual entry for MINIMAL does not cover this, and it jolly well should.

Once that is nailed down, it'll become clear what to do for DeriveAnyClass, I think.

comment:12 Changed 2 years ago by osa1

Sorry if this is a digression, but DAC just doesn't make sense to me. I think the only place it's mentioned in the user manual is here: https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/deriving.html and it's just saying "With -XDeriveAnyClass you can derive any other class. The compiler will simply generate an empty instance."

What I found confusing is:

  1. It says "derive any other class", which to me means that GHC first tries other mechanisms to derive implementations, and uses DAC as a last resort, because of the word "other".

E.g. if we have DF and DAC and we're trying to derive Functor, DF should always be used. If we're trying to derive something that's not supported by any other enabled extension, we should try DAC.

  1. It says "the compiler will simply generate an empty instance". This works for all typeclasses, so in a sense this is most general deriving mechanism. So to me it seems like this should be tried last, not first, when we have multiple deriving mechanisms that we can use for a particular deriving (..) statement. E.g. it should start with most specific deriving mechanism and move to more general ones as they fail.

---

@simonpj,

Once that is nailed down, it'll become clear what to do for DeriveAnyClass, I think.

If I understand correctly, you mean we should do something like this:

  1. Update semantics of MINIMAL to make generic implementations counted.
  1. Modify DAC to make working iff all MINIMALs have definitions.
  1. Also fix problems with docs and warnings on the way.

Does that sound right?

Also, it seems like DeriveAnyClass is broken in some other ways(#9821, #9968). Maybe I can fix those on the way.

comment:13 Changed 2 years ago by dreixel

Note that we initially thought of taking MINIMAL pragmas into account for DAC, but eventually decided against that. More background here: https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/GenericDeriving#Usingstandardderivingforgenericfunctions

comment:14 Changed 2 years ago by osa1

I can't see anything about why MINIMALs are not taken into account in that link, where can I learn about this?

comment:15 in reply to:  3 Changed 2 years ago by oerjan

Cc: oerjan added

Replying to goldfire:

Now, the user chooses what facility provides the instances. Note that I've done something currently impossible: I've used GND for the Read class. Normally, we don't want this behavior, and (to my knowledge) there's no way to convince GHC to use GND to derive a Read or Show instance. But maybe some user out there does want it.

I can think of at least one such use case that seems to make sense: Deriving Read and Show for newtypes around standard numeric types.

comment:16 Changed 2 years ago by rwbarton

I'm confused by a lot of this thread. As I mentioned in comment:2, DeriveAnyClass will never derive Functor, even when it is the only deriving extension enabled. So I would expect the presence of DeriveAnyClass to be irrelevant to any attempt to derive Functor, whatever other deriving extensions are enabled. And I think this behavior is (poorly) documented, by the word "other" in the description of DeriveAnyClass.

comment:17 Changed 2 years ago by osa1

@rwbarton, sorry if I'm misunderstand things, but I think DeriveAnyClass isn't deriving Functor because of an implementation bug and not because how it's designed. I think we should reconsider design of DeriveAnyClass and make the documentation more clear about it.

As usual, I'm willing to work on this :)

FWIW, I have a patch that fixes(based on my understanding of DervieAnyClass) some specific cases but without specifying it better it's pointless.

comment:18 Changed 2 years ago by kosmikus

I think the confusion is understandable, because a large design space is being discussed here.

The issue as reported is mainly about interaction between GeneralizedNewtypeDeriving and DeriveAnyClass. There are four options specified here: https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/GenericDeriving#InteractionwithGeneralizedNewtypeDeriving

All four have disadvantages, and unfortunately the document does not say which of the four options actually has been implemented.

It seems from the current behaviour of GHC like the implementation is along the lines of option 2 (and perhaps 3). However, even then we still have the fact that the error messages don't seem to properly reflect what is going on.

comment:19 Changed 2 years ago by oerjan

Hm so here is my "ideal" made-up-on-the-spot system. It somewhat combines all of 2, 3 and 4 from https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/GenericDeriving#InteractionwithGeneralizedNewtypeDeriving. It is intended to be backwards-compatible, except for one added warning.

  • By 3, classes may be annotated (pragma?) to say they prefer GND or DAC deriving. Builtin-derived classes count as annotated for their own style of deriving. To actually derive a class in a module, any extension for the derivation style still needs to be enabled as well.
  • By 4, if GeneralizedNewtypeDeriving is enabled, the newtype keyword may be used to signify that an instance for a newtype should be GND derived, even if this is against the annotated behavior for the class. This might even include builtin-derived classes like Show. (Obviously not Typeable, though.)
  • Also by 4, if both GeneralizedNewtypeDeriving and DeriveAnyClass are enabled (or for least surprise, maybe even with just the latter), the default keyword may be used to signify that an instance for a newtype should not be GND-derived, even if this is the annotated behavior for the class.
  • By 2, if neither the newtype deriving nor the class is annotated, then the behavior depends on which of GeneralizedNewtypeDeriving and DeriveAnyClass is enabled. If both are enabled, then a warning should be given. (This prevents surprises when a user adds both extensions for unrelated instances.) Then it defaults to DAC as today.
  • Although the proper extensions need to be enabled for whichever annotations/derivation styles end up being used, the only case where simply changing the extensions enabled will change code from one legal style of derivation to another should be the one in the previous point. (And thus the warning.)

As of now, I don't remember any classes with builtin-derivations that also are useful with DeriveAnyClass. So I think there isn't much need to be able to distinguish those two cases. Which also means that none of this matters to data declarations, only newtype.

comment:20 Changed 20 months ago by simonpj

Keywords: Generics added

comment:21 Changed 17 months ago by bgamari

Description: modified (diff)
Milestone: 8.2.1

I think we should really try to fix this for 8.2. Currently DeriveAnyClass is nearly useless as in most cases I'd much rather give it up than GeneralizedNewtypeDeriving.

comment:22 Changed 17 months ago by simonpj

I this that maybe Ryan Scott, king of generics, may be able to lead on this?

comment:23 Changed 15 months ago by RyanGlScott

Owner: set to RyanGlScott

I'm working on this now.

Before I get too far, I wanted to ask about a potential design choice for this feature. I like just about everything in comment:19 except for one thing: I don't think we should introduce pragmas for explicitly requesting a "deriving strategy" (the phrase I coined to describe this), if only because it unnecessarily changes the syntax. I think we could just as effectively use type synonyms to denote which deriving strategy you want:

-- | Derive a type class instance using GHC's default behavior.
type Builtin (a :: k) = a

-- | Derive a type class instance by generating an instance with no
-- implementations for any class methods or associated types. This requires the
-- @-XDeriveAnyClass@ extension.
type DAC (a :: k) = a

-- | Derive a type class instance for a newtype by using the underlying type's
-- instance for that class. This requires the @-XGeneralizedNewtypeDeriving@
-- extension.
type GND (a :: k) = a

Then we can specify a deriving strategy (for both deriving clauses and standalone deriving) without any new syntax:

data Foo a = Foo a
  deriving ( Builtin Show
            , DAC (Bar Int)
            )

deriving instance Quux a => GND (Quux (Foo a))

Then GHC simply has to check for the presence of one of these magical types before deciding which deriving mechanism to actually use. Does this sound agreeable?

comment:24 Changed 15 months ago by oerjan

That magic somewhat disturbs me, especially since it means a type synonym is not equivalent to its expansion. But if you think it's better than (backwards-compatibly) extending the parser to reuse keywords... (I guess default is a bit of a stretch anyway.)

In any case, you should also decide how to annotate a class definition, which is where I actually suggested a pragma. (I think this feature is an important part of this: E.g. I think mtl classes are almost always used with newtype deriving while aeson uses generics, and the libraries should be able to specify this.)

comment:25 Changed 15 months ago by RyanGlScott

That magic somewhat disturbs me, especially since it means a type synonym is not equivalent to its expansion.

I'm not proposing making these type synonyms expand to something different than what they're defined as above. The only magic comes when it's used in a deriving statement—that would cause its underlying (expanded) type to have a different derived instance, but the types are honest.

But if you think it's better than (backwards-compatibly) extending the parser to reuse keywords... (I guess default is a bit of a stretch anyway.)

It's not quite backwards-compatible—we'd also have to make a breaking change to template-haskell in order to accommodate the presence of these new pragmas. The type synonym approach, however, is only backportable to GHC 7.6.

(It should be noted that many uses of this feature would not be usable on older GHCs in the first place, so perhaps this isn't a huge concern.)

In any case, you should also decide how to annotate a class definition, which is where I actually suggested a pragma.

I'm not sure what you mean. Are talking about annotating when you should use DeriveAnyClass (DAC)? Or something else?

comment:26 in reply to:  25 Changed 15 months ago by oerjan

Replying to RyanGlScott:

It's not quite backwards-compatible—we'd also have to make a breaking change to template-haskell in order to accommodate the presence of these new pragmas. The type synonym approach, however, is only backportable to GHC 7.6.

Ah, there's always something. Which reminds me, Haddock might possibly want to know about the distinctions, too.

In any case, you should also decide how to annotate a class definition, which is where I actually suggested a pragma.

I'm not sure what you mean. Are talking about annotating when you should use DeriveAnyClass (DAC)? Or something else?

DAC or GND, both should be possible to specify. Neither is equivalent to having no annotation.

The annotations on the class not only tell what you normally should use, but also guide the compiler's selection for a newtype if the deriving clause does not itself have an annotation. Also, annotating the class removes the possibility of a warning if GHC must choose between DAC and GND because a module enables both as language extensions.

To re-summarize my suggested system, when choosing what mechanism to use for deriving a class for a newtype:

  1. Annotations on the deriving clause take top precedence (with the exception of some builtin classes at least including, for safety, Typeable).
  2. Then builtin derivable status of the class.
  3. Then annotations on the class declaration.
  4. Only if none of the above exists are the module's enabled relevant language extensions (GND or DAC) used to choose. If both GND and DAC are enabled, a warning is given, and DAC is chosen.
  5. No matter what is chosen in the above, the module must have enabled any relevant language extensions.

The guiding principles here being backwards compatibility (2,4), the ability to state intended usage at both class declaration and deriving sites (1,2,3), and reducing undetected surprises from irrelevant changes (4,5).

comment:27 Changed 15 months ago by simonpj

An alternative would be to require an instance declaration for DAD, thus

instance C a => C (T a)

That's all that is required, provided C has suitable generic default methods and T is an instance of Generic. And if DAC always used this route, there'd be less ambiguity in the deriving clause of a data type declaration.

I think (but I am not sure) that we don't allow standalone deriving for DAC; thus

deriving instance C a => C (T a)

would not work for DAC. Is that right? It seems reasonable to disallow it, because it's one word longer than the ordinary instance declaration. And if it is disallowed, then builtin and GND are already treated differently to DAC. So we'd get

  • GND/builtin: deriving clauses and deriving instance declarations
  • DAC: always a plain instance decl

I know that's not backward compatible..

On Ryan's synonym thing, like oerjan I'm uncomfortable with pressing synonyms into service like this.

comment:28 in reply to:  27 Changed 15 months ago by RyanGlScott

Replying to simonpj:

An alternative would be to require an instance declaration for DAD, thus

instance C a => C (T a)

That's all that is required, provided C has suitable generic default methods and T is an instance of Generic. And if DAC always used this route, there'd be less ambiguity in the deriving clause of a data type declaration.

I would strongly object to this! Literally the whole reason why I want this feature in the first place is so I can combine GeneralizedNewtypeDeriving with DeriveAnyClass in the presence of deriving clauses (which very conveniently don't require providing context). And moreover, if type out an instance manually like that, it's no longer DeriveAnyClass, it's pure Haskell 98! So imposing this "requirement" for DeriveAnyClass is tantamount to completely losing its utility, in my opinion.

I think (but I am not sure) that we don't allow standalone deriving for DAC; thus

deriving instance C a => C (T a)

would not work for DAC. Is that right? It seems reasonable to disallow it, because it's one word longer than the ordinary instance declaration. And if it is disallowed, then builtin and GND are already treated differently to DAC. So we'd get

  • GND/builtin: deriving clauses and deriving instance declarations
  • DAC: always a plain instance decl

I know that's not backward compatible..

We do currently allow StandaloneDeriving to be used with any deriving strategy in existence, including DeriveAnyClass. You're right that using DeriveAnyClass in this fashion is a tad redundant (you can just as well drop the deriving part and get the same thing), but I don't think there's any reason to explicitly disallow it—after all, that would be another breaking change, and it does seem a bit ad hoc to disallow one form of deriving here when it could easily work with the ideas proposed in this ticket.

On Ryan's synonym thing, like oerjan I'm uncomfortable with pressing synonyms into service like this.

OK, that's two votes against that idea, so I'll stop pursuing it :) In that case, pragmas looks like the path forward. Do the following names sound reasonable for pragmas?

  • {-# BUILTIN #-}
  • {-# GND #-}
  • {-# DAC #-} (goldfire proposed {-# ANY #-} above, but I'd argue that is a bit too ambiguous)

comment:29 Changed 15 months ago by RyanGlScott

Let me also address oerjan's proposal to annotate class definitions with these pragmas as well to guide which deriving mechanism is chosen.

I'll admit the idea makes me a bit uncomfortable, since it's not backwards compatible. For example, if there's a class ToJSON in use with current GHCs that becomes annotated with class {-# DAC #-} ToJSON, then the following code:

{-# LANGUAGE GeneralizedNewtypeDeriving #-}

newtype Foo = Foo Bar deriving ToJSON

would do two different things depending on which GHC is used! If an older GHC is used, it'll pick GeneralizedNewtypeDeriving, but if a more recent GHC is used, it'll pick DeriveAnyClass. Not to mention GHC will now complain that you need to enable DeriveAnyClass for that code to compile in the first place, which would be a pretty confusing breakage. This seems deeply wrong to me—in my opinion, one should be able to tell from the module in which the deriving statement is in which mechanism will be picked.

Also, I'm not sure what would happen if you have class {-# GND #-} Baz and tried to do data Quux = Quux deriving Baz. When you explicitly annotate a deriving statement for a data type with {-# GND #-}, that's an obvious error. But when the pragma is tucked away in a class definition (possibly in a far-flung module), it might not be intuitive as why that code would error.

Last edited 15 months ago by RyanGlScott (previous) (diff)

comment:30 Changed 15 months ago by oerjan

As Ryan implies, Simon's suggestion seems precisely equivalent to abolishing DeriveAnyClass altogether. And the entire point of that extension is to extend deriving clauses to allow user-defined classes.

OK, so the class annotation idea is not backwards compatible across actually adding those annotations. At least that would seem to need a major version change. Maybe they could still be useful to enable warnings if the program is implicitly using the non-recommended derivation type. This could easily happen if a module lists the wrong GND/DAC language extension.

Your ToJSON example is instructive, in that a programmer might very well want either version, and in that there's already a problem today if the module lists the wrong language extensions.

As for your last example, I meant for all of this to have no effect at all on data types. But on the other hand, an annotation of {-# GND #-} might strongly imply that a class is not designed to support DAC, and so suggest a warning.

Which now makes me realize that it may be reasonable for some classes to support both, at least if they are equivalent in result (and then, probably more efficient with GND.)

It's getting pretty obvious here that some of my design goals are inconsistent with each other. I'd suggest erring on the side of giving warnings unless the user is being explicit.

comment:31 Changed 15 months ago by RyanGlScott

I would propose making the class-annotation stuff its own ticket—I think there's further discussion that would need to be had before considering such a change.

In any case, I'm getting close to having a Diff ready for this. I originally implemented it using the type synonyms approach, but since that was shot down, I'll need to do some refactoring to use pragmas instead (hopefully just plumbing).

comment:32 Changed 15 months ago by RyanGlScott

Differential Rev(s): Phab:D2280
Status: newpatch

See also https://github.com/haskell/haddock/pull/520 for the necessary Haddock changes.

comment:33 Changed 15 months ago by goldfire

On Phab:D2280, kosmikus said:

In principle: I think this is great and definitely needed so that we can more reliably use all the different deriving-mechanisms, so thanks a lot for trying to move this forward.

But: I think this should not be done via pragmas. (I know I've suggested using pragmas in the past myself, but I've changed my opinon.) I'm not sure whether there is a clear guideline for what should be a pragma and what not, but to me, I think stuff affecting optimizations or performance are perfectly fine, stuff that can make a difference between a program compiling at all or not (OVERLAPPING, OVERLAPPABLE) are critical, and stuff that changes the semantics/result of a program (this change) is too much. I'd much rather see this implemented via actual language syntax. It would also hopefully make it easier to extend it with more flexibility in subsequent versions.

(Copying here because this is a design issue, not an implementation one.)

I agree fully with this, including the bit that I, too, have suggested pragmas but now change my stance. I think we should have a guiding principle about pragmas. I propose this:

  • A pragma should either:
    1. Behave precisely like options that could be passed in at the command line, OR
    2. Have no effect on the semantics (static or dynamic) of the program

Ideally, actually, we'd just have (2), but LANGUAGE doesn't qualify for (2).

To be clear, by "semantics (static or dynamic)", I mean typability and runtime behavior.

Browsing through the current set of accepted pragmas (in the manual, if that's complete) all pragmas meet this guideline except these:

  • The OVERLAPPING/OVERLAPS/OVERLAPPABLE/INCOHERENT pragmas, as these affect static semantics.
  • RULES, as these can affect dynamic semantics.

Indeed, I'd be in favor of phasing out these pragmas in favor of other syntax. One could argue that RULES should be a pragma, because they shouldn't, if written correctly, affect runtime behavior. I would disagree with that argument, I think, but perhaps I'd be in the minority opinion.

I know this is a larger discussion than on just this ticket, but I thought I'd mention this here to gauge response. If others like this idea at all, I'll post a fresh ticket.

comment:34 Changed 15 months ago by RyanGlScott

I don't really care what color we paint the bikeshed with, but I will caution that we're dealing with a perilous part of the parser here, and that cramming more syntax into deriving clauses might lead to more trouble than it's worth.

The parser rule for deriving clauses is currently this:

deriv_types :: { [LHsSigType RdrName] }
        : typedoc                       { [mkLHsSigType $1] }

        | typedoc ',' deriv_types       {% addAnnotation (gl $1) AnnComma (gl $2)
                                           >> return (mkLHsSigType $1 : $3) }

In other words, arbitrary types. This poses a problem for trying to introduce new syntax in front of a derived type, since:

  1. Uppercase identifiers can be confused for type constructors
  2. Lowercase identifiers can be confused for type variables

To my knowledge, no one has proposed any alternatives to the suggested designs above, but if you do have a suggestion, please details how you would resolve this ambiguity. The pragma approach, while perhaps stretching the powers that most pragmas have, do not introduce any ambiguities into the parser.

Also, I'm not sure why one pragmas wouldn't be amenable to "extend it with more flexibility in subsequent versions". In fact, that's precisely one of the reasons why I like pragmas, since it's cheap to add more of them in the future, and they're wildly configurable. In fact, later I hope to explore extending the GHC plugin mechanism to allow programmers to write their own deriving mechanisms, with something like:

data Foo = Foo Bar Baz deriving {-# CUSTOM myDeriv #-} C

myDeriv :: SrcSpan -> TyCon -> TcM (LHsBinds RdrName, BagDerivStuff)
myDeriv = ...

It would be dead simple to add this functionality via a pragma. With custom syntax, however, we'd have to reserve even more keywords... at least, I assume. I suppose I should ask: what exactly are folks proposing to use in place of pragmas? It's hard to have this discussion without something concrete to reference.

comment:35 Changed 15 months ago by oerjan

This may be off topic, but: Before the whole OVERLAPPABLE etc. stuff got included, I had somehow got the impression that pragmas were supposed to follow two simple principles, that improved portability between different compilers:

  • If a compiler does not understand a pragma other than a LANGUAGE pragma, it can safely ignore it, and if the program still compiles, its semantics must be the same.
  • If a compiler does not understand a LANGUAGE pragma, it should bail out with an error.

The Haskell 2010 report seems to recommend this behavior, although it doesn't quite seem to require it.

At some point things started to look inconsistent, so in a vain attempt to keep making sense of it I thought of an additional principle for added flexibility:

  • A LANGUAGE pragma might imply the existence of other pragmas, which must then also be understood.

However, the OVERLAPPABLE etc. pragmas aren't as far as I know governed by any particular LANGUAGE pragma. Perhaps they still follow the first principle, I suspect there may be corner cases.

Notably, I think the current Phab:D2280 implementation does obey the last principle. Any use of DAC or GND requires a corresponding language pragma, and Builtin never changes semantics.

comment:36 Changed 14 months ago by RyanGlScott

oerjan, I greatly appreciate your effort to salvage the current pragma-based approach, but enough GHC devs have objected to the point that I don't think it's worth pursuing anymore.

It seems that we need (1) some new syntax for demarcating which deriving strategy to use, and (2) a new language extension to hide it behind. Here is my proposal:

  1. In deriving clauses, one can optionally indicate a deriving strategy with the following syntax:
newtype Foo = Foo Bar deriving ( Eq
                               , Ord <- builtin
                               , Read <- newtype
                               , Show <- default
                               )

And in standalone deriving declarations:

newtype Foo = Foo Int
deriving instance Eq Foo
deriving builtin instance Ord Foo
deriving newtype instance Read Foo
deriving default instance Show Foo

where builtin corresponds to BUILTIN, newtype corresponds to GND, and default corresponds to DAC in my pragma-based proposal. I'm not terribly attached to these names, so if you have better suggestions, I'm all ears.

Importantly, I think we need some way to separate the derived type from the strategy in a deriving clause, or parsing it will be a nightmare for the reasons I laid out in this comment. Standalone deriving declarations won't be as tricky since we can just put the strategy between deriving and instance without too much trouble, although we could also require this for consistency:

deriving instance Eq Foo
deriving instance Ord Foo <- builtin
deriving instance Read Foo <- newtype
deriving instance Show Foo <- default

The choice of <- as a delimeter is also completely arbitrary. I think we'll need some kind of non-alphanumeric thing, but I'm not sure what's sensible.

  1. Require the use of {-# LANGUAGE DerivingStrategies #-} to use this feature.

Comments?

comment:37 Changed 14 months ago by RyanGlScott

Another possibility: we allow the use of multiple derivings after a datatype. Something like:

newtype Foo = Foo Bar
  deriving         (A, B, C)
  deriving builtin (D, E, F)
  deriving newtype (G, H, I)
  deriving default (J, K, L)

Pros: this would be a lot more consistent with the treatment for standalone deriving declarations. Cons: we'd have to significantly rework the way we parse data/newtype declarations.

comment:38 Changed 14 months ago by simonpj

FWIW, I like comment:37 and its uniformity with standalone deriving. I dislike the <- notation.

I doubt that parsing will be hard.

comment:39 Changed 14 months ago by RyanGlScott

Thus far, I have received a vote in favor of the option presented in comment:37 from Simon (and Matthew Pickering on #ghc). Since no one has objected to it, I think I'll proceed forward with it. I'd like to make one critique of my own proposal, though: I don't really like the use of the default keyword to signify DeriveAnyClass, especially since default could very easily be mistaken for what builtin is supposed to signify. I'd much rather use the word anyclass, as inspired by a ghc-devs post by kosmikus.

Also, to ensure that users can't write something like

newtype Foo = Foo Bar
  deriving (A, B, C)
  deriving (D, E, F)
  deriving (G, H, I)
  deriving (J, K, L)

in regular Haskell, we'd need to guard the use of multiple deriving clauses for one datatype behind a language extension. I propose just making that language extension also be DerivingStrategies.

comment:40 Changed 13 months ago by oerjan

After reading the ghc-devs discussion that decided to rename builtin to bespoke, I have a question.

Is it still the case that using any other deriving strategy than bespoke for a class that supports bespoke requires using this new extension? The summary of the new system in the mailing list discussion seemed to imply not in some cases, but that could be just my misunderstanding.

If not, then that could lead to confusing errors, like accidentally deriving Functor with anyclass because DeriveAnyClass is enabled but DeriveFunctor has been forgotten, and would probably warrant some more warnings.

On the other hand, I think GND would give equivalent result to bespoke for many classes, so it might be convenient not to warn in those cases. E.g. when someone wants to derive Functor, Applicative and Monad for their monad stack newtype, it seems almost reasonable that only GND needs to be enabled, and not DeriveFunctor.

comment:41 in reply to:  40 Changed 13 months ago by RyanGlScott

Replying to oerjan:

Is it still the case that using any other deriving strategy than bespoke for a class that supports bespoke requires using this new extension?

The short answer is: yes. The longer, more nuanced answer is that GHC sometimes picks to derive certain "standard" classes using GND when they would produce the same instance (see "The deriving strategy resolution algorithm" section of this wiki page for more info). For example, newtype Foo = Foo Int deriving Eq actually uses GND to derive the Eq instance, not the bespoke instance you'd normally get for Eq. Most of the time, though, if you derive a standard class without a keyword, it'll default to the bespoke strategy.

If not, then that could lead to confusing errors, like accidentally deriving Functor with anyclass because DeriveAnyClass is enabled but DeriveFunctor has been forgotten, and would probably warrant some more warnings.

This is a valid concern, but luckily that doesn't happen, even with the current GHC. Currently, GHC's outlook is that DeriveAnyClass should never kick in for standard classes. As a result, if you try compiling this program:

{-# LANGUAGE DeriveAnyClass #-}
newtype T a = T (Maybe a) deriving Functor

it will error with:

    Can't make a derived instance of ‘Functor T’:
      You need DeriveFunctor to derive an instance for this class
      Try GeneralizedNewtypeDeriving for GHC's newtype-deriving extension
    In the newtype declaration for ‘T’

whereas for nonstandard classes, DeriveAnyClass kicks in. Since none of the standard classes use DefaultSignatures, I think this special case is probably acceptable (and if you really want to derive your Functor instance using DeriveAnyClass for some bizarre reason, you'll be able to with DerivingStrategies).

On the other hand, I think GND would give equivalent result to bespoke for many classes, so it might be convenient not to warn in those cases. E.g. when someone wants to derive Functor, Applicative and Monad for their monad stack newtype, it seems almost reasonable that only GND needs to be enabled, and not DeriveFunctor.

It is indeed reasonable, and GHC does sometimes choose to pick GeneralizedNewtypeDeriving over the DeriveFunctor algorithm when the last type variable can be eta-reduced (again, see this wiki page for the full story). Emphatically, Phab:D2280 will not change any of this existing behavior. The only things it does are (1) fix the bug reported in the original comment and (2) give you the ability to choose explicitly which strategy to use, and hence bypass the confusing algorithm GHC uses to choose a strategy implicitly in the absence of a strategy keyword.

comment:42 Changed 13 months ago by oerjan

Hm. Commenting on that wiki algorithm:

  • Might it be simpler to treat the bespoke -> newtype optimization as a post-step, independent of the rest? Or is it ever important not to apply it? Even for step 1.
  • Is 2(a) missing Enum?
  • Why is Traversable in 2(b)? I would have thought that the bespoke -> newtype optimization would apply to it. I guess there's some technical difference.
  • All the examples in 2(c) are listed in a or b, leaving the strange impression that it can never be triggered. Although I assume Functor and Foldable belong there. (Which also tells me the last paragraph in my previous comment has been taken care of).
  • The phrase "can be successfully used with GeneralizedNewtypeDeriving" is needed in 2(c) as well. Should 2(d) apply or not if that check falls? If it does then sometimes standard classes could get anyclassed. If it does not, then should the classes fall back to bespoke, at least if their extensions are enabled?
  • It seems that Functor and Foldable never get newtype derived if that extension is not enabled, even when it would be safe to do so.

I think the algorithm can be simplified a bit: remove point 2(c) entirely, and say explicitly in point 2(d) that it doesn't apply to standard derivable classes.

Lastly, while thinking about this, I made a table of the (currently four-way) classification of the "standard/bespoke" classes:

GND equivalence No extension Extension
Always 2(a) Eq Ord Enum Ix Bounded
Requires check 2(c) Functor Foldable
Never 2(b) Read Show 2(b) Data Generic Generic1 Typeable Traversable(?) Lift

comment:43 in reply to:  42 ; Changed 13 months ago by RyanGlScott

oerjan, hopefully this answers all of your questions:

Replying to oerjan:

  • Might it be simpler to treat the bespoke -> newtype optimization as a post-step, independent of the rest? Or is it ever important not to apply it? Even for step 1.

Yes, there are cases where you don't want to replace the bespoke strategy with the newtype one. Consider the following example:

newtype Foo a = Foo (Maybe a)

What would happen if you tried implementing a Traversable instance for Foo? As it turns out, trying to define traverse = coerce is impossible at the moment:

    • Couldn't match representation of type ‘f (Maybe b)’
                               with that of ‘f (Foo b)’
        arising from the coercion of the method ‘traverse’
          from type ‘forall (f :: * -> *) a b.
                     Applicative f =>
                     (a -> f b) -> Maybe a -> f (Maybe b)’
            to type ‘forall (f :: * -> *) a b.
                     Applicative f =>
                     (a -> f b) -> Foo a -> f (Foo b)’
      NB: We cannot know what roles the parameters to ‘f’ have;
        we must assume that the role is nominal
    • When deriving the instance for (Traversable Foo)

This is because the type signature of traverse leads to an ill-roled coercion (see here for the full story). Until #9123 is fixed, GHC will refuse to apply the newtype strategy to derived Traversable instances (unless you explicitly ask for it).

BTW, the full list of classes that GHC avoids using the newtype strategy for can be found here.

  • Is 2(a) missing Enum?

No. The full list of "standard" classes for which newtype kicks in by default (without the presence of -XGeneralizedNewtypeDeriving) can be found here. As the comments there indicate, Enum isn't in that list because by default, deriving Enum for a newtype would fail since it checks for a datatype with all nullary constructors. Therefore, you have to enable -XGeneralizedNewtypeDeriving (or use the newtype keyword) to derive Enum for a newtype.

  • Why is Traversable in 2(b)? I would have thought that the bespoke -> newtype optimization would apply to it. I guess there's some technical difference.

See my answer above.

  • All the examples in 2(c) are listed in a or b, leaving the strange impression that it can never be triggered. Although I assume Functor and Foldable belong there. (Which also tells me the last paragraph in my previous comment has been taken care of).

Sorry, I should have explicitly enumerated the remaining classes not listed in 2(a) or 2(b). They are Functor, Foldable, and Enum.

  • The phrase "can be successfully used with GeneralizedNewtypeDeriving" is needed in 2(c) as well. Should 2(d) apply or not if that check falls? If it does then sometimes standard classes could get anyclassed. If it does not, then should the classes fall back to bespoke, at least if their extensions are enabled?

Yes, I should use that phrase, thank you for noticing that omission. I certainly don't want standard classes to be derived with the anyclass strategy.

  • It seems that Functor and Foldable never get newtype derived if that extension is not enabled, even when it would be safe to do so.

You have to explicitly enable -XGeneralizedNewtypeDeriving to derive Functor or Foldable that way because, by default, deriving them is guarded behind -XDeriveFunctor and -XDeriveFoldable. You can't just say newtype Foo a = Foo (Maybe a) deriving Functor and expect it to work without extensions, unlike Eq, Ord, Ix, and Bounded.

I think the algorithm can be simplified a bit: remove point 2(c) entirely, and say explicitly in point 2(d) that it doesn't apply to standard derivable classes.

I disagree. 2(a), (b), and (c) are all special cases for standard derivable classes, each with enough nuances that trying to cram one of them into 2(d) would make it even more confusing (than it already is).

Lastly, while thinking about this, I made a table of the (currently four-way) classification of the "standard/bespoke" classes:

Thank you, this is fantastic! I've updated the wiki page with the corrections above and incorporating your table (with some slight corrections).

comment:44 in reply to:  43 ; Changed 13 months ago by oerjan

Replying to RyanGlScott:

oerjan, hopefully this answers all of your questions:

I'm afraid not quite, also now I have some new ones :)

Replying to oerjan:

  • Might it be simpler to treat the bespoke -> newtype optimization as a post-step, independent of the rest? Or is it ever important not to apply it? Even for step 1.

Yes, there are cases where you don't want to replace the bespoke strategy with the newtype one. Consider the following example:

Although that answered my later question about Traversable, what I meant here was whether to treat it as a common post-step for the classes that can be newtype derived. In particular Functor and Foldable, see below.

What would happen if you tried implementing a Traversable instance for Foo? As it turns out, trying to define traverse = coerce is impossible at the moment:

Ah, similar to that "Monad not getting join as a method" situation.

Until #9123 is fixed, GHC will refuse to apply the newtype strategy to derived Traversable instances (unless you explicitly ask for it).

Surely with the role situation, it will always fail even if you ask for it? Also, I noticed in the code you linked a comment that it wouldn't be equivalent in any case for law-breaking Applicatives.

Enum isn't in that list because by default, deriving Enum for a newtype would fail since it checks for a datatype with all nullary constructors.

Ah, I misremembered what types can derive Enum.

Sorry, I should have explicitly enumerated the remaining classes not listed in 2(a) or 2(b). They are Functor, Foldable, and Enum.

By what you said otherwise, shouldn't Enum be in 2(b)?

  • The phrase "can be successfully used with GeneralizedNewtypeDeriving" is needed in 2(c) as well. Should 2(d) apply or not if that check falls? If it does then sometimes standard classes could get anyclassed. If it does not, then should the classes fall back to bespoke, at least if their extensions are enabled?

Yes, I should use that phrase, thank you for noticing that omission. I certainly don't want standard classes to be derived with the anyclass strategy.

My point in the rest of that comment was that even adding that phrase isn't enough, because the wording and fallthrough still implies that anyclass would be used in this case:

{-# LANGUAGE GeneralizedNewtypeDeriving, DeriveAnyClass #-}

newtype F x = F ([x], Maybe x) deriving Functor

In fact, even assuming anyclass is not used, that example is a bit worrisome: What exactly does GHC do in this case? Does it succeed in deriving Functor? (I assume No.) If it fails, does it give an error message that doesn't confuse the user about why it fails?

  • It seems that Functor and Foldable never get newtype derived if that extension is not enabled, even when it would be safe to do so.

You have to explicitly enable -XGeneralizedNewtypeDeriving to derive Functor or Foldable that way because, by default, deriving them is guarded behind -XDeriveFunctor and -XDeriveFoldable. You can't just say newtype Foo a = Foo (Maybe a) deriving Functor and expect it to work without extensions, unlike Eq, Ord, Ix, and Bounded.

What I meant here, relevant to my post-step question, is that unlike with e.g. Eq and Ord, the algorithm description implies that the newtype deriving optimization is not automatic: it's not applied when deriving Functor or Foldable with DeriveFunctor/DeriveFoldable but not GND enabled.

Lastly, while thinking about this, I made a table of the (currently four-way) classification of the "standard/bespoke" classes:

Thank you, this is fantastic! I've updated the wiki page with the corrections above and incorporating your table (with some slight corrections).

You're welcome, although the table was meant to summarize all the data for what determines automatic strategy selection, not just whether GND can be used. (I'm a bit unsure whether it was a good idea to separate the first two rows.) In particular, "Requires -XGeneralizedNewtypeDeriving" only applies to the middle right cell, not the whole column, since only the first two rows even allow GND to be automatically selected.

I tried rewriting the description from a more class/table-centric perspective. (In the process, I seem to have changed "standard derivable class" into "class which has a bespoke strategy".)

  1. Look for a deriving strategy. If one is present, use that.
  1. If deriving a class which has a bespoke strategy:

(a) If deriving Eq, Ord, Ix, or Bounded for a newtype, use the GeneralizedNewtypeDeriving strategy (even if the language extension isn't enabled).

(b) If deriving Functor, Foldable, or Enum(?) for a newtype, the datatype can be successfully used with GeneralizedNewtypeDeriving, and -XGeneralizedNewtypeDeriving has been enabled, use the GeneralizedNewtypeDeriving strategy.

(c) Otherwise, if deriving a class which has a bespoke strategy, and the corresponding language extension is enabled (if necessary), use the bespoke strategy. If the language extension is not enabled, throw an error.

  1. If deriving a class without a bespoke strategy:

(a) If deriving an instance for a newtype and both -XGeneralizedNewtypeDeriving and -XDeriveAnyClass are enabled, default to DeriveAnyClass, but emit a warning stating the ambiguity.

(b) Otherwise, if -XDeriveAnyClass is enabled, use DeriveAnyClass.

(c) Otherwise, if deriving an instance for a newtype, the datatype and typeclass can be successfully used with GeneralizedNewtypeDeriving, and -XGeneralizedNewtypeDeriving is enabled, do so.

(d) Otherwise, throw an error.

comment:45 in reply to:  44 Changed 13 months ago by RyanGlScott

I like your algorithm much better, since it has no jumps/gotos. I've updated the wiki page to use it. I've also updated the table to use more accurate labels. Thank you for the peer editing!

As for your other questions:

Surely with the role situation, it will always fail even if you ask for it? Also, I noticed in the code you linked a comment that it wouldn't be equivalent in any case for law-breaking Applicatives.

Yes, it will always fail. But that's the point of -XDerivingStrategies—by explicitly using the newtype keyword, you take on the burden of ensuring that your typeclass can actually be derived, and you must accept the consequences if you try something which won't work.

My point in the rest of that comment was that even adding that phrase isn't enough, because the wording and fallthrough still implies that anyclass would be used in this case:

{-# LANGUAGE GeneralizedNewtypeDeriving, DeriveAnyClass #-}

newtype F x = F ([x], Maybe x) deriving Functor

In fact, even assuming anyclass is not used, that example is a bit worrisome: What exactly does GHC do in this case? Does it succeed in deriving Functor? (I assume No.) If it fails, does it give an error message that doesn't confuse the user about why it fails?

That is a very good point, and one which your refactored table takes into account, so now the table reflects the reality that -XDeriveAnyClass does not fall through in this case. In case you're curious, here's the error message that GHC gives for that example:

    • Can't make a derived instance of ‘Functor F’
        (even with cunning GeneralizedNewtypeDeriving):
        You need DeriveFunctor to derive an instance for this class
    • In the newtype declaration for ‘F’

That seems like a sensible error to me. I'll add a test case for this in Phab:D2280 to be safe.

Last edited 13 months ago by RyanGlScott (previous) (diff)

comment:46 Changed 13 months ago by oerjan

Still a few comments:

  • I really don't think Enum belongs in 2(b), which is why I put a question mark on it in the first place. I think it should also be moved to the bottom left cell in the table.
  • Even though it's explained below, I have a hunch the phrase "bespoke typeclass instance" could be misinterpreted as referring to the selected strategy. It's a little longer, but "instance for a bespoke typeclass" feels less ambiguous.
  • The paragraph starting "Step 2.(b) deserves some explanation." doesn't make sense with the new algorithm, since the issue no longer applies with the new control flow. (After all, one of the things simplifying it is that step 2 doesn't need to consider anyclass any more, and step 3 doesn't need to consider bespoke.)

comment:47 in reply to:  46 ; Changed 13 months ago by RyanGlScott

Replying to oerjan:

  • I really don't think Enum belongs in 2(b), which is why I put a question mark on it in the first place. I think it should also be moved to the bottom left cell in the table.

I don't follow. Why should it be in the "never" category? As noted above, there are scenarios when GHC will derive Enum for newtypes, and they are perfectly captured in 2(b).

  • Even though it's explained below, I have a hunch the phrase "bespoke typeclass instance" could be misinterpreted as referring to the selected strategy. It's a little longer, but "instance for a bespoke typeclass" feels less ambiguous.

This is why I hate choosing syntax :)

I'm going strictly by the dictionary definition of "bespoke" here, which means "tailor-made" or "custom-fit". That means the phrase "bespoke typeclass" doesn't make sense, since "bespoke" is a property of the instance, not the typeclass.

  • The paragraph starting "Step 2.(b) deserves some explanation." doesn't make sense with the new algorithm, since the issue no longer applies with the new control flow. (After all, one of the things simplifying it is that step 2 doesn't need to consider anyclass any more, and step 3 doesn't need to consider bespoke.)

I've reworded it to make it a little clearer, hopefully.

comment:48 in reply to:  47 ; Changed 13 months ago by oerjan

Replying to RyanGlScott:

I don't follow. Why should it be in the "never" category? As noted above, there are scenarios when GHC will derive Enum for newtypes, and they are perfectly captured in 2(b).

Oh, I'd missed that. I guess removing the question mark was fine, then.

However, that means Enum alone doesn't fit properly into my intended table scheme, since it's sort of a hybrid between lower left and middle right (actually, upper right, since there's no check needed on the shape of the newtype). Which means the current table indeed only describes when to do GND.

Possibly, it could fit in the middle left if the table is slightly relabeled, so that the rows tell what to do about GND and the columns tell about general use. Like so:

No extension required Requires language extension to use
GND when possible 2(a) Eq, Ord, Ix, Bounded
GND with extension 2(b) Enum 2(b) Functor, Foldable
Never select GND 2(c) Read, Show 2(c) Data, Generic, Generic1, Typeable, Traversable, Lift

Which reminds me, I think maybe the word "silly" in step 1 should be "impossible", since after all some very silly things are allowed, like using anyclass on a class that doesn't support it.

  • Even though it's explained below, I have a hunch the phrase "bespoke typeclass instance" could be misinterpreted as referring to the selected strategy. It's a little longer, but "instance for a bespoke typeclass" feels less ambiguous.

This is why I hate choosing syntax :)

I'm going strictly by the dictionary definition of "bespoke" here, which means "tailor-made" or "custom-fit". That means the phrase "bespoke typeclass" doesn't make sense, since "bespoke" is a property of the instance, not the typeclass.

Right. (I have no prior relationship with that word myself.) Anyway my point is that at this point in the algorithm, you don't want to branch on the (not yet) selected instance, but only on the class. This means you might not want to use a syntax ("bespoke typeclass instance") that could imply that "bespoke" is a property of that instance.

I've reworded it to make it a little clearer, hopefully.

My quibbling mind says that "when it would otherwise derive a bespoke instance" should be something like "when it supports bespoke instances", otherwise there are still corner cases that aren't obviously caught, such as deriving Foldable for a newtype with GND, deriving Enum for a newtype, or even (when interpreted intuitively) deriving an instance for which bespoke is selected, but the derivation fails for some other reason.

comment:49 in reply to:  48 Changed 13 months ago by RyanGlScott

Replying to oerjan:

Possibly, it could fit in the middle left if the table is slightly relabeled, so that the rows tell what to do about GND and the columns tell about general use. Like so:

No extension required Requires language extension to use
GND when possible 2(a) Eq, Ord, Ix, Bounded
GND with extension 2(b) Enum 2(b) Functor, Foldable
Never select GND 2(c) Read, Show 2(c) Data, Generic, Generic1, Typeable, Traversable, Lift

Which reminds me, I think maybe the word "silly" in step 1 should be "impossible", since after all some very silly things are allowed, like using anyclass on a class that doesn't support it.

Sounds good to me. Updated with both suggestions.

Right. (I have no prior relationship with that word myself.) Anyway my point is that at this point in the algorithm, you don't want to branch on the (not yet) selected instance, but only on the class. This means you might not want to use a syntax ("bespoke typeclass instance") that could imply that "bespoke" is a property of that instance.

My quibbling mind says that "when it would otherwise derive a bespoke instance" should be something like "when it supports bespoke instances", otherwise there are still corner cases that aren't obviously caught, such as deriving Foldable for a newtype with GND, deriving Enum for a newtype, or even (when interpreted intuitively) deriving an instance for which bespoke is selected, but the derivation fails for some other reason.

I think I can live with the phrase "when deriving a class which supports bespoke instances". I've updated the wiki to use it.

comment:50 Changed 11 months ago by nomeata

Motivated by the lightning talk, a note on the wording “bespoke”. To me, “bespoke” would be a instance that is tailored to my needs (e.g. “ignore field x in the Eq instance, because it is just a cached value that depends on the other”). The derived instance is not bespoke in that sense. It is more of a stock instance. How about deriving stock Eq instead?

comment:51 in reply to:  50 Changed 11 months ago by RyanGlScott

Replying to nomeata:

How about deriving stock Eq instead?

I spent so long looking for builtin synonyms that I had never considered stock as an option. I have to say, I really like that suggestion. I've posted a follow-up to the ghc-devs mailing list thread in which I argue in favor of stock. If nobody raises a serious objection, I think I'll make the switch to use that keyword instead.

comment:52 Changed 11 months ago by Ben Gamari <ben@…>

In 9e862765/ghc:

Implement deriving strategies

Allows users to explicitly request which approach to `deriving` to use
via keywords, e.g.,

```
newtype Foo = Foo Bar
  deriving Eq
  deriving stock    Ord
  deriving newtype Show
```

Fixes #10598. Updates haddock submodule.

Test Plan: ./validate

Reviewers: hvr, kosmikus, goldfire, alanz, bgamari, simonpj, austin,
erikd, simonmar

Reviewed By: alanz, bgamari, simonpj

Subscribers: thomie, mpickering, oerjan

Differential Revision: https://phabricator.haskell.org/D2280

GHC Trac Issues: #10598

comment:53 Changed 11 months ago by RyanGlScott

Resolution: fixed
Status: patchclosed
Test Case: deriving/should_run/T10598_bug

comment:54 Changed 8 months ago by Ben Gamari <ben@…>

In 5349d648/ghc:

Rename TH constructors for deriving strategies

After talking to Richard, he and I concluded that choosing the rather
common name `Newtype` to represent the corresponding deriving strategy
in Template Haskell was a poor choice of name. I've opted to rename it
to something less common (`NewtypeStrategy`) while we still have time. I
also renamed the corrsponding datatype in the GHC internals so as to
match it.

Reviewers: austin, goldfire, hvr, bgamari

Reviewed By: bgamari

Subscribers: thomie, mpickering

Differential Revision: https://phabricator.haskell.org/D2814

GHC Trac Issues: #10598

comment:55 Changed 2 weeks ago by RyanGlScott

Keywords: deriving added
Note: See TracTickets for help on using tickets.