Opened 3 years ago

Closed 21 months ago

Last modified 17 months ago

#10382 closed feature request (fixed)

Template Haskell (non-quasi) quotes should work with stage 1 compiler

Reported by: ezyang Owned by:
Priority: high Milestone: 8.0.1
Component: Template Haskell Version: 7.11
Keywords: Cc: ekmett
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D876
Wiki Page:

Description (last modified by ezyang)

Right now, you must enable the TemplateHaskell extension to use (non-quasi) quotes, and TemplateHaskell is not supported by the stage 1 compiler. But there actually is no good reason why this should be the case: (non-quasi) quoting doesn't require any user-written code to be loaded up and run, so it should be doable by the stage 1 compiler. I propose adding a new extension, Quotes, which turns on JUST quotation (NOT quasiquoting) and works with the stage 1 compiler.

This will solve len's problem https://github.com/ekmett/lens/issues/496 where they want to rename some syntax using quotation, but don't want break compilation on a stage 1 compiler.

See also #10279.

Change History (28)

comment:1 Changed 3 years ago by ezyang

Differential Rev(s): Phab:D876

comment:2 Changed 3 years ago by ezyang

Description: modified (diff)

comment:3 Changed 3 years ago by simonpj

I'm not sure about this.

Minor point: we already have a language extension Opt_QuasiQuotes. Surely we don't need another!

Medium point: this change really only relates to building GHC itself, so it's fairly parochial. For example if we have [qq| blah |] in GHC itself, then to compile stage 2 we must have qq defined in some library that is compiled with stage 1; a boot library in other words. And the boot libraries are fairly limited.

Main thing: It does add complexity. If you write [qq| blah |] then:

  • In stage 1 we must find and dynamically link function qq, against the bootstrap compiler
  • In stage 2 we must find and dynamically link function qq, against the stage 1 compiler

Moreover, if HsSyn has changed, then function qq must change too.

And if you can manage all this version skew stuff, what's to stop us allowing all Template Haskell splices in stage 1? I thought it was obvious, but now I can't quite see it!

Simon

comment:4 Changed 3 years ago by ezyang

To be clear, I am NOT suggesting quasi-quotes be supported by this; just plain old brackets! (Apologies for the bad names; perhaps "expression quotation" is a more accurate name?) With plain quotes, we don't have to find and link qq with this complexity. See the diff that's been attached, which I know works and is not all that complicated.

Whether or not the change is parochial, I think it relates to more than just GHC; it applies to any platform where we can compile GHC, but dynamic code loading is not supported (i.e. no GHCi support). Now, you might wonder, "why would anyone need to use TH quotes, without any TH splices?" but Edward Kmett gives an example of one such case, where he wants to put his TH library code (no splices) in a main library, while still supporting platforms without GHCi, without macroing the business.

Unusual? Maybe. But someone wants it, and it doesn't seem too hard to give them, so let's do it!

If we don't like the language extension, we could alternate make TemplateHaskell not immediately error on stage 1 compilation, and just error when we actually reach a splice site.

comment:5 Changed 3 years ago by ezyang

Description: modified (diff)
Summary: Template Haskell quotes should work with stage 1 compilerTemplate Haskell (non-quasi) quotes should work with stage 1 compiler

comment:6 Changed 3 years ago by ekmett

The current situation:

On one hand I have a number of packages that provide TemplateHaskell convenience functions for users, lens being the go-to example. But on the other hand, I have a bunch of folks who maintain releases for my packages on platforms where only a stage1 compiler is available, Joachim Breitner and the Debian folks used to maintain patches and versions of my code that removed large chunks of the libraries to build on those platforms.

When i found out, I offered to support stage1 compilers myself more directly. This ensured a more uniform API, and that different distributions weren't crippling my code in different ways, making it so certain combinators or modules just weren't available on certain platforms.

But then we ran into an issue, we needed to generate names that linked to the right places within our code. So we manually construct all of our names ourselves, using the cabal-supplied version number or package key when needed:

https://github.com/ekmett/lens/blob/9a247b52ed20e578d9c843d8cc6dad5433a1c186/src/Control/Lens/Internal/TH.hs#L104

Then we just don't use the TemplateHaskell extension ourselves, despite linking against the template-haskell package and everything is good enough for us to limp along.

I would love to eventually be able to drop this set of hoops, but I have few opinions on the best way to get there. That said, Edward's suggestion of making TemplateHaskell just bomb when you reach a splice site in stage1 rather than immediately would avoid introducing any new flags and sounds pretty simple.

comment:7 Changed 3 years ago by goldfire

Seems sensible to me, if we just piggyback on the TemplateHaskell extension. (I, too, don't like Quotes!) If Phab is to be believed that almost all the code changes there are code movement and not code changes, it does look indeed very simple to implement.

comment:8 Changed 3 years ago by ezyang

I can rewrite the code motion to be added ifdef macros, but it seemed cleaner to just hoist them all out.

comment:9 Changed 3 years ago by angerman

Sorry for being a little late to the discussion. Ideally, I would prefer that a stage1 compiler would try and go ahead with compiling template haskell, until it figures it can't. Motivation being: when ever I find time to hack on out-of-process-template-haskell (effectively trying to bring over luites th approach from ghcjs), I have to inject lots of ifdefs into the ghc code to even get ghc to the point where it starts to complain that it can't compile the splice instead of refusing to even try it right from the start.

I haven't had much time to work on it throughout the last half year but would love to continue with it. I'll leave some of the relevant links here:

From my understanding, and please correct me if I'm wrong, th splices fall into three categories:

  • (a) IO - can be anything includiing arbitarary IO actions at compile time.
  • (b) Target dependent - reading target intrinsic values (maxBound :: Int)
  • (c) everything else that does not include (a) or (b).

where (c) should be fine to compile on the host, (b) could probably be also compiled on the host if all the target intrinsic information is taken care of. (a) is going to be interesting, as at least with the out-of-process-th appraoch, you would be running the code on a device that might not have the same environment. Such that thinks like looking up git revisions would have to fail.

comment:10 Changed 3 years ago by Edward Z. Yang <ezyang@…>

In 28257cae77023f2ccc4cc1c0cd1fbbd329947a00/ghc:

Support stage 1 Template Haskell (non-quasi) quotes, fixes #10382.

Summary:
This commit adds stage 1 support for Template Haskell
quoting, e.g. [| ... expr ... |], which is useful
for authors of quasiquoter libraries that do not actually
need splices.  The TemplateHaskell extension now does not
unconditionally fail; it only fails if the renamer encounters
a splice that it can't run.

In order to make sure the referenced data structures
are consistent, template-haskell is now a boot library.

In the following patches, there are:

    - A few extra safety checks which should be enabled
      in stage1
    - Separation of the th/ testsuite into quotes/ which
      can be run on stage1

Note for reviewer: big diff changes are simply code
being moved out of an ifdef; there was no other substantive
change to that code.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Test Plan: validate

Reviewers: simonpj, austin, goldfire

Subscribers: bgamari, thomie

Differential Revision: https://phabricator.haskell.org/D876

GHC Trac Issues: #10382

comment:11 Changed 3 years ago by Edward Z. Yang <ezyang@…>

In 21c72e7d38c96ac80d31addf67ae4b3c7a6c3bbb/ghc:

Split off quotes/ from th/ for tests that can be done on stage1 compiler.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Test Plan: run these tests with stage1

Reviewers: simonpj, austin

Subscribers: bgamari, thomie

Differential Revision: https://phabricator.haskell.org/D877

GHC Trac Issues: #10382

comment:12 Changed 3 years ago by Edward Z. Yang <ezyang@…>

In eb0ed4030374af542c0a459480d32c8d4525e48d/ghc:

RnSplice's staging test should be applied for quotes in stage1.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Test Plan: validate

Reviewers: simonpj, austin

Subscribers: bgamari, thomie

Differential Revision: https://phabricator.haskell.org/D878

GHC Trac Issues: #10382

comment:13 Changed 3 years ago by ezyang

Resolution: fixed
Status: newclosed

comment:14 Changed 3 years ago by Edward Z. Yang <ezyang@…>

In 5c459eefcb17ff97beebdc08ccfca21bd8fa5201/ghc:

Revert stage 1 template-haskell. This is a combination of 5 commits.

Revert "Quick fix: drop base bound on template-haskell."

This reverts commit 3c70ae032e4361b203dfcf22b0a424e8838a5037.

Revert "Always do polymorphic typed quote check, c.f. #10384"

This reverts commit 9a43b2c1f78b3cf684646af64b9b67dc8079f58f.

Revert "RnSplice's staging test should be applied for quotes in stage1."

This reverts commit eb0ed4030374af542c0a459480d32c8d4525e48d.

Revert "Split off quotes/ from th/ for tests that can be done on stage1 compiler."

This reverts commit 21c72e7d38c96ac80d31addf67ae4b3c7a6c3bbb.

Revert "Support stage 1 Template Haskell (non-quasi) quotes, fixes #10382."

This reverts commit 28257cae77023f2ccc4cc1c0cd1fbbd329947a00.

comment:15 Changed 3 years ago by ezyang

There was a bug: template-haskell needs to build with GHC 7.8, which it does not right now.

comment:16 Changed 3 years ago by Edward Z. Yang <ezyang@…>

In f16ddcee0c64a92ab911a7841a8cf64e3ac671fd/ghc:

Support stage 1 Template Haskell (non-quasi) quotes, fixes #10382.

Summary:
This commit adds stage 1 support for Template Haskell
quoting, e.g. [| ... expr ... |], which is useful
for authors of quasiquoter libraries that do not actually
need splices.  The TemplateHaskell extension now does not
unconditionally fail; it only fails if the renamer encounters
a splice that it can't run.

In order to make sure the referenced data structures
are consistent, template-haskell is now a boot library.
There are some minor BC changes to template-haskell to make it boot
on GHC 7.8.

Note for reviewer: big diff changes are simply code
being moved out of an ifdef; there was no other substantive
change to that code.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Test Plan: validate

Reviewers: simonpj, austin, goldfire

Subscribers: bgamari, thomie

Differential Revision: https://phabricator.haskell.org/D876

GHC Trac Issues: #10382

comment:17 Changed 2 years ago by thoughtpolice

Milestone: 7.12.18.0.1

Milestone renamed

comment:18 Changed 2 years ago by osa1

I was trying to update TH parts of user manual about which quotes can be run by stage1 compiler, and on the way I found out that tests in quotes/ directory are currently not running under stage1.

I tried this:

  • I run a test using top-level Makefile, e.g. make TEST=T2632 in testsuite/.
  • I realized it's running using stage2 compiler. But the whole point of moving these tests to a new directory was to test them using stage1 compiler.
  • When I tried compiling the test using stage1 compiler(I copied the command and replaced stage2 with stage1), it failed with this:
$ ./tests/quotes &&  "/home/omer/haskell/ghc/inplace/test   spaces/ghc-stage1" -c T2632.hs -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-warn-missed-specialisations -fno-ghci-history -XTemplateHaskell -package template-haskell

T2632.hs:10:9: error:
    Can't find interface-file declaration for type constructor or class DecsQ
      Probable cause: bug in .hi-boot file, or inconsistent .hi file
      Use -ddump-if-trace to get an idea of which file caused the error
    In the expression: [d| func = 0 `op` 3 |]
    In an equation for ‘decl1’: decl1 = [d| func = 0 `op` 3 |]

T2632.hs:12:9: error:
    Can't find interface-file declaration for type constructor or class DecsQ
      Probable cause: bug in .hi-boot file, or inconsistent .hi file
      Use -ddump-if-trace to get an idea of which file caused the error
    In the expression:
      [d| op x y = x
          func = 0 `op` 3 |]
    In an equation for ‘decl2’:
        decl2
          = [d| op x y = x
                func = 0 `op` 3 |]

I also tried the same thing on some manually created simple TH code and they all failed with same error. It seems like there's a bug somewhere.

comment:19 Changed 2 years ago by ezyang

Owner: ezyang deleted
Resolution: fixed
Status: closednew

comment:20 Changed 2 years ago by ezyang

We also need a way to force the test suite to run stage1 in some cases, so we can catch this regression next time.

comment:21 in reply to:  20 Changed 2 years ago by thomie

Replying to ezyang:

We also need a way to force the test suite to run stage1 in some cases, so we can catch this regression next time.

I see three options:

  • Option 1 (change validate)
    • create a directory with tests that should be tested with the stage1 compiler.
    • In ./validate, add something like make -C testsuite/tests/stage1-tests stage=1 (The full command would also include: $MAKE_TEST_TARGET $BINDIST $TEST_VERBOSITY THREADS=$threads SUMMARY_FILE=...)
    • Make sure the grepping for results in testsuite_summary.txt still does the right thing (bit fiddly)
    • Minor point: define target synonyms fasttest and slowtest in testsuite/Makefile and testsuite/mk/test.mk)
  • Option 2 (driver support)
    • add a new setup function to testsuite/driver/testlib.py: something like def stage(n) or def stage1()
    • assign the STAGE1_GHC variable from testsuite/mk/boilerplate.mk to a new option config.stage1 (add --config 'stage1=$(call quote_path,$(STAGE1_GHC))' to RUNTEST_OPTS in testsuite/mk/test.mk, and change testsuite/driver/runtests.py and testsuite/driver/testglobals.py).
    • add a compiler field to TestOptions (testsuite/driver/testglobals.py). By default it should contain config.compiler (do so in def test in testsuite/driver/testlib.py, not sure..)
    • the setup function stage1() sets opts.compiler to config.stage1, but only when the stage1 compiler is available. When TEST_HC is specified or when TEST_HC is implied to be an installed compiler, STAGE1_GHC won't be defined.
    • change all uses of config.compiler to opts.compiler (grep also for '{compiler}').
  • Option 3 (Makefile)
    • define the test in a Makefile. Refer to STAGE1_GHC instead of TEST_HC. Skip if empty.

The drawback of option 1 is that a failing validate might confuse GHC developers, because they can't reproduce the test failure when running the test locally (they'd have to specify make stage=1, which isn't obvious). Many questions on IRC/Phabricator/mail.

Last edited 2 years ago by thomie (previous) (diff)

comment:22 Changed 2 years ago by ezyang

Reading these options, an alternative comes to mind: what if we just made stage1 another WAY just like prof, etc.?

comment:23 Changed 2 years ago by osa1

As far as I can see WAY only effects the arguments passed to GHC, so it doesn't help unless we do some refactoring in the test driver, and the test driver is ... messy.

Can we do this with less amount of work by adding a new field to testglobals.TestOptions, something like stage, and then in testlib.simple_build() we check the option and override compiler variable?

Then we could set the stage in all.T using setTestOpts().

Last edited 2 years ago by osa1 (previous) (diff)

comment:24 Changed 21 months ago by ezyang

Priority: normalhigh
Status: newpatch

comment:25 Changed 21 months ago by Ben Gamari <ben@…>

In bbfff229/ghc:

Unconditionally handle TH known key names.

Previously, we didn't add Template Haskell key names to the list
of known uniques when building a stage 1 compiler.  But with
f16ddcee0c64a92ab911a7841a8cf64e3ac671fd we may refer to TH
names even in stage 1, and this was causing uniques to not
be setup properly.

Signed-off-by: Edward Z. Yang <ezyang@cs.stanford.edu>

Test Plan: validate and run stage1 test suite

Reviewers: osa1, austin, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1520

GHC Trac Issues: #10382

comment:26 Changed 21 months ago by bgamari

Status: patchmerge

Hasn't been merged yet.

comment:27 Changed 21 months ago by bgamari

Resolution: fixed
Status: mergeclosed

comment:28 Changed 17 months ago by Thomas Miedema <thomasmiedema@…>

In d94c405/ghc:

Testsuite: validate the tests/stage1 directory with the stage1 compiler

* See `Note [Why is there no stage1 setup function?]`.
* Move T2632 to the tests/stage1 directory (#10382).

Reviewed by: ezyang, nomeata, bgamari

Differential Revision: https://phabricator.haskell.org/D2341

GHC Trac Issues: #12197
Note: See TracTickets for help on using tickets.