GHC: Ticket #8423: Less conservative compatibility check for closed type families
http://ghc.haskell.org/trac/ghc/ticket/8423
<p>
attached is an example where the type checker isn't "computing" in the closed type families, or at least doing a one step reduction.
</p>
<p>
this makes sense, given that type families only compute when instantiated...
But if we could partially evaluate closed type families (or at least do a one step reduction), the attached example code would type check!
</p>
<p>
interestingly, depending on what order the cases for the PSum are written, the type error changes!
</p>
<p>
I guess I want an "eager matching" closed type family, that when partially instantiated will patch on the first pattern it satisfies, to complement ordered type families.
</p>
<p>
attached is a toy example where I otherwise need an unsafeCoerce to make things type check
</p>
en-usGHChttp://ghc.haskell.org/trac/ghc/chrome/site/ghc_logo.png
http://ghc.haskell.org/trac/ghc/ticket/8423
Trac 1.0.9carterTue, 08 Oct 2013 22:03:07 GMTattachment set
http://ghc.haskell.org/trac/ghc/ticket/8423
http://ghc.haskell.org/trac/ghc/ticket/8423
<ul>
<li><strong>attachment</strong>
set to <em>Fancy.hs</em>
</li>
</ul>
<p>
closed families are too weak
</p>
TicketcarterTue, 08 Oct 2013 22:03:57 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:1
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:1
<p>
so maybe this is a "feature request" idea rather than a bug, but i'd like more folks to chime in please :)
</p>
TicketcarterWed, 09 Oct 2013 01:41:51 GMTtype, milestone changed
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:2
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:2
<ul>
<li><strong>type</strong>
changed from <em>bug</em> to <em>feature request</em>
</li>
<li><strong>milestone</strong>
changed from <em>7.8.1</em> to <em>7.10.1</em>
</li>
</ul>
<p>
Though I imagine theres some subtleties to adding such machinery, but it might be a great way to allow more "proofs by computation" to work out in Haskell
</p>
TicketgoldfireWed, 09 Oct 2013 02:18:06 GMTrelated set
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:3
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:3
<ul>
<li><strong>related</strong>
set to <em>#4259</em>
</li>
</ul>
<p>
Yes, closed type families are too weak. But for a good reason.
</p>
<p>
Here is the test case boiled down:
</p>
<pre class="wiki">data PNat = S !PNat | Z
type family PSum (a :: PNat) (b:: PNat) :: PNat where
PSum 'Z b = b -- 1
PSum a 'Z = a -- 2
PSum a ('S b) = 'S (PSum a b) -- 3
PSum ('S a) b = 'S (PSum a b) -- 4
</pre><p>
At some point, the type checker wants to know whether <tt>(PSum (S r2) b) ~ (PSum r2 (S b))</tt>. To do this, both sides of the <tt>~</tt> need to reduce one step. But neither can reduce in practice. Why? An equation in a closed type family can fire (that is, be used in a reduction) only when GHC can be absolutely positive of one of two things:
</p>
<ol><li>No previous equation can fire, or
</li><li>Any previous equation that can fire will reduce, in one step, to the same thing.
</li></ol><p>
(Technically, (2) subsumes (1), but it's easier to think about the cases separately.)
</p>
<p>
Let's consider reducing <tt>PSum r2 (S b)</tt>. Clearly, neither of the first two equations are applicable. Equally clearly, the third equation is quite tempting. Now, we check for these conditions. We quickly see that equation (1) might fire, if <tt>r2</tt> ends up becoming <tt>Z</tt>. So, now we need to satisfy (2). If equation (1) can fire, then we really are in the case <tt>PSum Z (S b)</tt>. If equation (1) fires, we get <tt>S b</tt>. If equation (3) fires, we get <tt>S (PSum Z b)</tt>. These are <strong>not</strong> the same. So, GHC remains on the fence about the whole affair and prudently refuses to take action.
</p>
<p>
An obvious question at this point is: Why the one-step restriction? The answer: anything else is not obviously well-founded. (Note: I did not say "obviously not well-founded", which is quite a different claim!) The general case is to check that the reducts (that is, the right-hand sides) are <em>confluent</em>. (The current check looks to see if they are <em>coincident</em>.) But, to check confluence means reasoning about type family reductions... including perhaps the one we are defining... whose reduction behavior would depend on the confluence of the equations' right-hand sides. Oops; we've just fallen into the rabbit hole.
</p>
<p>
There might be a way out of this morass, but I go cross-eyed when I think about it for too long. Not that it isn't worth it -- I think that allowing code like Carter's to be accepted would be a great boon to GHC! I just don't know a way of thinking about this that will solve the problem.
</p>
<p>
For more reading on the subject, check out <a class="new ticket" href="http://ghc.haskell.org/trac/ghc/ticket/4259" title="feature request: Relax restrictions on type family instance overlap (new)">#4259</a>, which is all about this problem. It's possible that closed type families address some of the examples there, but the core problem discussed in that bug is the same as the one here. However, because that bug is about open families and this one is about closed, I will leave this one open. It's (barely) conceivable that they have different solutions.
</p>
TicketcarterWed, 09 Oct 2013 04:48:47 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:4
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:4
<p>
excellent points that i probably agree with (and i'll have to dig into that other ticket as time permits)
</p>
<p>
Hrm, another way of looking at it is that i'd be ok with having to come up with sort of induction principles or the like that let me give the type checker the proofs! Currently we dont have a decent story that plays well with equality constraints.
</p>
<p>
so I either need to "lie about the types" and do something like
</p>
<pre class="wiki">
reverseShape :: Shape r -> Shape r
reverseShape Nil = Nil
reverseShape shs@(anIx :* rest) = go shs Nil
where
--- too weakly typed
go :: Shape a -> Shape b -> Shape r -- want r= PSum a b
go Nil res = unsafeCoerce $ res -- 0 + b = b ==> b=r
go (ix :* more ) res = go more (ix :* res) --- 1+a + b = r
</pre><p>
(which has a cast in the base case, but is ok othewise)
</p>
<p>
OR try to have some sort of "proof" mapping, which honestly dones't work, and to even "write" the terms required
</p>
<pre class="wiki">{-# LANGUAGE AllowAmbiguousTypes #-}
</pre><p>
that attempt looked like the following
</p>
<pre class="wiki">assocSumSucc :: Shape (PSum ('S r2) b) -> Shape ( PSum r2 ('S b))
assocSumSucc = unsafeCoerce
assocSumSuccBad :: Shape (PSum a b) -> Shape ( PSum c d)
assocSumSuccBad = unsafeCoerce
</pre><p>
this doesn't quite work, because neither interacts with the equality constraint solving!
</p>
<p>
So the only sane thing of that sort is
</p>
<pre class="wiki">shapeCoerce :: Shape a -> Shape b
shapeCoerce = unsafeCoerce
</pre><p>
which is honestly pretty bad! (and apply this in the recursive case)
</p>
<p>
these as all (unlike the other solution inline above) pretty bad, because they break tail calls.
</p>
<p>
I guess what i want is a way have having "proof principles" or something that i could build, which would interact well with the types.
</p>
TicketcarterWed, 09 Oct 2013 04:49:50 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:5
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:5
<p>
relatedly: Ranjit Jhala took the motiviating code and demo'd checking it with liquid haskell! Which is pretty darn awesome
</p>
<p>
<a class="ext-link" href="http://goto.ucsd.edu:8090/index.html#?demo=permalink%2F1381284690.hs"><span class="icon"></span>http://goto.ucsd.edu:8090/index.html#?demo=permalink%2F1381284690.hs</a>
</p>
TicketcarterWed, 09 Oct 2013 04:51:22 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:6
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:6
<p>
honestly i just want a way to check things, my ticket here is just noting limits that happen with type families. I'm open to anything that has a sane usability story and a sane engineering story (on the ghc and users sides both).
</p>
TicketgoldfireWed, 09 Oct 2013 14:51:12 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:7
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:7
<p>
If you just want a way to check things, please see <a class="ext-link" href="https://gist.github.com/goldfirere/6902431"><span class="icon"></span>here</a>. The key step is that you do in fact have to write proofs! These proofs may, sadly, have runtime significance. But, from the optimization results in the "Deferred type errors" <a class="ext-link" href="http://research.microsoft.com/en-us/um/people/simonpj/papers/ext-f/icfp12.pdf"><span class="icon"></span>paper</a> lead me to wonder if these runtime traces are optimized away. (I haven't checked -- doing so is beyond the scope of my morning Haskell stretches.) I did check, though, that the inliner will remove calls to <tt>gcoerce</tt> and make <tt>go</tt> properly tail-recursive (ignoring the type-cast, which I know is removed at runtime).
</p>
<p>
Thanks for posting this! It's made me realize that <tt>gcoerce</tt> from that example should probably be in the Data.Type.Equality module anyway.
</p>
TicketcarterWed, 09 Oct 2013 19:45:43 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:8
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:8
<p>
thanks! this is great! Would I want to compare the core? (or just use criterion to benchmark?). I'll add that to my infinite todo list.
</p>
<p>
This example seems to work only in ghc 7.8, how would i adapt those ideas to work on 7.6? (can I even?)
</p>
<p>
that said, having more examples like this about how to prove things in haskell is probably a good idea! Also, figuring out how to erase such proofs as much as possible also would likely be worth while. I'm happy to help contribute a few braincells to chewing on that post 7.8
</p>
TicketgoldfireThu, 10 Oct 2013 16:18:56 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:9
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:9
<p>
This should work just fine in 7.6. Just change the closed type family <tt>(+)</tt> to be an open one. Everything else should just work. I've update <a class="ext-link" href="https://gist.github.com/goldfirere/6902431#file-shape-7-6-hs"><span class="icon"></span>the gist</a> with a 7.6-compatible version. (I also had to inline a few definitions that aren't available in the 7.6 libraries.)
</p>
<p>
Examining the core won't work, because even core needs the proofs to type-check! You might be able to examine something after core, but GHC's behavior after it's done with core is black magic to me. You might have to benchmark it.
</p>
TicketgoldfireThu, 27 Mar 2014 13:05:28 GMT
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:10
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:10
<p>
After being prodded via email to consider this issue further, I've made a tiny bit of progress, but nowhere near an actual solution.
</p>
<p>
It turns out that the closed type family case and the open type family case can indeed be considered separately. Why? Because the compatibility check can usefully be turned off in the closed type family case, but not in the open one. Turning off the compatibility check for closed families reduces the number of reductions possible, but doesn't threaten soundness. On the other hand, turning off the compatibility check for unordered, open families destroys soundness.
</p>
<p>
So, I thought about this: during the compatibility check, I normalized both substituted RHSs, but, during the normalization, I ignored compatibility. This seems like a nice, conservative check, but less conservative than the current one, requiring coincidence.
</p>
<p>
It turned out that implementing the check above was non-trivial (it required delicate ordering among family instances), so I did the following (even better) thing instead (this is available in the <tt>wip/recurs-compat</tt> branch):
</p>
<ul><li>The existing compatibility check was left unchanged.
</li><li>During type family simplification, an extra check was added, in parallel with the apartness check. (This follows more closely what is written up in <a class="ext-link" href="http://www.cis.upenn.edu/~eir/papers/2014/axioms/axioms-extended.pdf"><span class="icon"></span>the paper</a>.) Say the current, matching equation is <tt>e</tt> and the previous one under consideration is <tt>d</tt>. Suppose the target is <tt>t</tt>.
<ul><li>We unify <tt>d</tt> with <tt>t</tt>, possibly producing unifying substitution <tt>ds</tt>. (If unification fails, the new check fails, and no reduction happens. But, note that if <tt>d</tt> and <tt>t</tt> are apart, then the apartness check succeeds, allowing reduction.)
</li><li>Let <tt>es</tt> be the substitution (sure to exist) that witnesses that <tt>e</tt> matches <tt>t</tt>.
</li><li>Now, we check if <tt>normalize(ds(d_rhs)) = normalize(ds(es(e_rhs)))</tt>. If this check succeeds, we allow reduction by equation <tt>e</tt>.
</li><li>The <tt>normalize</tt> function here is special: if it is reducing a closed type family and needs to do a compatibility check, that check <em>does not normalize</em>. Without this restriction, we would often loop, as demonstrated below.
</li></ul></li></ul><p>
There are a few natural questions at this point:
</p>
<ul><li><strong>Is this type safe?</strong> I haven't proved anything. But, I believe that it is type safe as long as all type families terminate. Why? Because the type safety proof (appearing in the paper) doesn't seem to be disturbed by this change. That proof requires only <em>local</em> confluence, meaning that it can take an arbitrary number of steps to recombine two terms after they have diverged. However, I believe that this is <strong>not</strong> type-safe in the presence of non-terminating type families. Why? Because the proof in the non-terminating case requires a <em>local diamond</em> property, which requires single-step recombining after divergence. The normalization step explicitly throws that out. I don't know of a way to repair the proof, so I'm led to think that the property is indeed broken. I have no counter-example, though.
</li></ul><ul><li><strong>Does it work in practice?</strong> A bit. Take the <tt>PSum</tt> family in <a class="ticket" href="http://ghc.haskell.org/trac/ghc/ticket/8423#comment:3" title="Comment 3">comment:3</a>. In GHC 7.8, the third equation won't fire because it conflicts with the second. With the extra check as described here, the third equation can indeed fire. But, the fourth can't, because the compatibility check against the third requires normalization internally to work. I don't see an easy solution to that problem.
</li></ul><ul><li><strong>What about axioms in Core?</strong> My work didn't touch axioms. So, <tt>-dcore-lint</tt> would fail in my branch if the new checks were used. A real solution here would require changes to Core, essentially to record exactly how two equations are compatible. Given that the proof of compatibility (with normalization) might be arbitrarily large, it would seem to require some new form that is an explicit witness of compatibility. Of course, we could just bake the normalization step into the Core type-checking algorithm, but normalization is potentially non-terminating, so doing so breaks the tenet of "type-checking Core is simple and decidable".
</li></ul><ul><li><strong>Where to go from here?</strong> I believe that the core problem is that we're currently finding some sort of least fixed point of compatibility, where we really want the greatest fixed point. That is, if we assume that the third and fourth equations of <tt>PSum</tt> are compatible, then we can prove they're compatible. This recursive proof would be productive (that is, there would be a new <tt>'S</tt> constructor), so I think the idea isn't entirely silly. I haven't worked out any details, though.
</li></ul><ul><li><strong>What's the looping example?</strong> Check this out:
</li></ul><pre class="wiki">type family F a where
F [a] = F a
F b = b
</pre><blockquote>
<p>
This is a simple, terminating type family. Yet, if we try to simplify <tt>F c</tt> (which should be stuck), a fully recursive compatibility check would loop, as the check would, in turn, need to simplify <tt>F a</tt>, which is isomorphic to what we started with.
</p>
</blockquote>
<p>
Given the complications here (especially the thought of how to update Core), I'm tabling this for now.
</p>
TicketjstolarekFri, 04 Jul 2014 09:32:05 GMTcc set
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:11
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:11
<ul>
<li><strong>cc</strong>
<em>jan.stolarek@…</em> added
</li>
</ul>
TicketthoughtpoliceTue, 23 Dec 2014 13:34:10 GMTmilestone changed
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:12
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:12
<ul>
<li><strong>milestone</strong>
changed from <em>7.10.1</em> to <em>7.12.1</em>
</li>
</ul>
<p>
Moving to 7.12.1 milestone; if you feel this is an error and should be addressed sooner, please move it back to the 7.10.1 milestone.
</p>
TicketthoughtpoliceFri, 11 Sep 2015 08:38:26 GMTmilestone changed
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:13
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:13
<ul>
<li><strong>milestone</strong>
changed from <em>7.12.1</em> to <em>8.0.1</em>
</li>
</ul>
<p>
Milestone renamed
</p>
TicketjstolarekFri, 18 Sep 2015 09:06:48 GMTcc changed
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:14
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:14
<ul>
<li><strong>cc</strong>
<em>jstolarek</em> added; <em>jan.stolarek@…</em> removed
</li>
</ul>
TicketgoldfireFri, 18 Sep 2015 15:36:13 GMTsummary, milestone changed
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:15
http://ghc.haskell.org/trac/ghc/ticket/8423#comment:15
<ul>
<li><strong>summary</strong>
changed from <em>contraint solver doesn't reduce reducible closed type family expressions (even with undecidable instances!)</em> to <em>Less conservative compatibility check for closed type families</em>
</li>
<li><strong>milestone</strong>
changed from <em>8.0.1</em> to <em>⊥</em>
</li>
</ul>
<p>
New title and milestone to reflect the realities here.
</p>
Ticket