Opened 19 months ago

Last modified 7 months ago

#11551 new task

Get doctests into testsuite

Reported by: lwm Owned by:
Priority: normal Milestone:
Component: Core Libraries Version: 7.10.3
Keywords: documentation, doctest Cc: ekmett
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Documentation bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

It seems like a bit of a shame not to have Mr.Orlitzky's lovely doctests not running with the testsuite. Or perhaps to at least have some sort of make doctest. If there is some interest to get this working, I'd like to take a look at submitting a patch.

Here are the previous documentation patches I can find: https://phabricator.haskell.org/p/mjo/

I can see the following files that have doctests:

ruby-2.2.1 ~/work/contributing/ghc (master) [12:35:44]
lwm$ grep -R "=== __Examples__" libraries/base/ | uniq
grep: libraries/base/Data/Maybe.hs:-- ==== __Examples__
libraries/base/Data/Either.hs:-- ==== __Examples__
libraries/base/Data/List.hs:-- ==== __Examples__
libraries/base/Data/Functor.hs:-- ==== __Examples__
libraries/base/Data/Bool.hs:-- ==== __Examples__
libraries/base/Data/Char.hs:-- ==== __Examples__
libraries/base/GHC/Unicode.hs:-- ==== __Examples__
libraries/base/Text/Printf.hs:-- ==== __Examples__

Change History (7)

comment:1 Changed 19 months ago by thomie

Go for it.

Either create a separate doctest target in the testsuite (see testsuite/Makefile, testsuite/mk/test.mk and testsuite/tests/Makefile), and have validate call that. Or add it simply as a run_command test.

Think about what should happen when the user doesn't have doctest installed.

comment:2 Changed 12 months ago by bgamari

lwm, is the trouble just integration into the build system or is there a deeper issue here? I quickly tried running doctest on some modules in base and very quickly ran into issues with multiple base versions (e.g. GHC desugared do syntax to imply a constraint of the Monad class provided by the compiler, not the Monad class provided by the base library which I was building.

This suggests to me that even getting doctest running against base will take some effort by someone. Have you had different experiences, lwm?

comment:3 Changed 12 months ago by lwm

bgamari, thanks for the input.

Have you had different experiences, lwm?

No, that's pretty much exactly what happened to me. I didn't get as deep as you did with regards to the desugared output.

I have since been playing with trying to run doctest against base. I think I will get some logs to illustrate the problem and create an issue over at the doctest repo. It's definitely been blocking me. Otherwise, do you have some recommendations?

Last edited 12 months ago by lwm (previous) (diff)

comment:4 Changed 7 months ago by phadej

you have to compile doctests against the same compiler you are testing with.

Examples: 44  Tried: 44  Errors: 0  Failures: 0

Maybe, Char and Either pass, Functor got:

### Failure in libraries/base/Data/Functor.hs:49: expression `show <$> Nothing'
expected: "Nothing"
 but got: ""
          "\ESC[;1m<interactive>:31:6: \ESC[;1m\ESC[31merror:\ESC[;1m"
          "    Ambiguous occurrence \8216<$>\8217"
          "    It could refer to either \8216Data.Functor.<$>\8217,"
          "                             defined at libraries/base/Data/Functor.hs:73:1"
          "                          or \8216Prelude.<$>\8217,"
          "                             imported from \8216Prelude\8217 (and originally defined in \8216Data.Functor\8217)\ESC[0m"

nasty colorcodes!

comment:5 Changed 7 months ago by Ben Gamari <ben@…>

In 5ef956e/ghc:

Fix doctests in Data.Functor

Reviewers: austin, hvr, bgamari

Reviewed By: bgamari

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D2920

GHC Trac Issues: #11551

comment:6 Changed 7 months ago by phadej

With https://gist.github.com/phadej/fdd74f4b4b40dc0a758ca6192cd66e83 I can successfully run the doctests in base. There's a hack, it picks files based on whether there is $setup, which IMHO is not so bad assumption.

If someone could give a pointer how to integrate it with validate, I probably can do it. Yet, i'd like to have ability to run the doctests in standalone way, as editing docs / doctests shouldn't require rebuilds of the base etc.

comment:7 Changed 7 months ago by lwm

Oh, that is awesome phadej, well done!

Yes, would be nice for it to be stand alone from base.

I'll try and get a doctest run on my local with your Gist ...

Note: See TracTickets for help on using tickets.