Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss plans post-7.10.2.
GHC 7.10.2 release
GHC 7.10.2 has been released!
As always, if you suspect that you have found a regression don't hesitate to open a Trac ticket. We are especially interested in performance regressions with fairly minimal reproduction cases.
GHC 7.10.2 and the
A few days ago a report came in of long compilations times under 7.10.2 on a program with many
Text literals (#10528). This ended up being due to a change in the simplifier which caused it to perform rule rewrites on the left-hand-side of other rules. While this is questionable (read "buggy") behavior, it doesn't typically cause trouble so long as rules are properly annotated with phase control numbers to ensure they are performed in the correct order. Unfortunately, it turns out that the rules provided by the
text package for efficiently handling string literals did not include phase control annotations. This resulted in a rule from
base being performed on the literal rules, which rendered the literal rules ineffective. The simplifier would then expend a great deal of effort trying to simplify the rather complex terms that remained.
Thankfully, the fix is quite straightforward: ensure that the the text literal rules fire in the first simplifier phase (phase 2). This avoids interference from the
base rules, allowing them to fire as expected.
This fix is now present in
text-220.127.116.11. Users of GHC 7.10.2 should be use this release if at all possible. Thanks to
text's maintainer, Bryan O'Sullivan for taking time out of his vacation to help me get this new release out.
While this mis-behaviour was triggered by a bug in GHC, a similar outcome could have arisen even without this bug. This highlights the importance of including phase control annotations on
RULE pragmas: Without them the compiler may choose the rewrite in an order that you did not anticipate. This has also drawn attention to a few shortcomings in the current rewrite rule mechanism, which lacks the expressiveness to encode complex ordering relationships between rules. This limitation pops up in a number of places, including when trying to write rules on class-overloaded functions. Simon Peyton Jones is currently pondering possible solutions to this on #10595.
This week we merged the long-anticipated
-XStrictData extension (Phab:D1033) by Adam Sandberg Ericsson. This implements a subset of the [StrictPragma] proposal initiated by Johan Tibell.In particular,
StrictData allows a user to specify that datatype fields should be strict-by-default on a per-module basis, greatly reducing the syntactic noise introduced by this common pattern. In addition to implementing a useful feature, the patch ended up being a nice clean-up of the GHC's handling of strictness annotations.
What remains of this proposal is the more strong
-XStrict extension which essentially makes all bindings strict-by-default. Adam has indicated that he may take up this work later this summer.
AMP-related performance regression
In late May Herbert Valerio Riedel opened Phab:D924, which removed an explicit definition for
mapM in the
Traversable instance, as well as redefined
mapM_ in terms of
traverse_ to bring consistency with the post-AMP world. The patch remains unmerged, however, due to a failing ghci testcase. It turns out the regression is due to the redefinition of
mapM_, which uses
(>>) was once used. This tickles poor behavior in ghci's
ByteCodeAsm module. The problem can be resolved by defining
(*>) = (>>) in the
Applicative Assembler instance (e.g. Phab:1097). That being said, the fact that this change has already exposed performance regressions raises doubts as to whether it is prudent.
GHC Performance work
Over the last month or so I have been working on nailing down a variety of performance issues in GHC and the code it produces. This has resulted in a number of patches which in some cases dramatically improve compilation time (namely Phab:1012 and Phab:D1041). Now since 7.10.2 is out I'll again be spending most of my time on these issues. We have heard a number of reports that GHC 7.10 has regressed on real-world programs. If you have a reproducible performance regression that you would like to see addressed please open a Trac ticket.
- Phab:D1028: Fixity declarations are now allowed for infix data constructors in GHCi (thanks to Thomas Miedema)
- Phab:D1061: Fix a long-standing correctness issue arising when pattern matching on floating point values
- Phab:D1085: Allow programs to run in environments lacking iconv (thanks to Reid Barton)
- Phab:D1094: Improve code generation in
integer-gmp(thanks to Reid Barton)
- Phab:D1068: Implement support for the
MachOpin the LLVM backend (thanks to Michael Terepeta)
- Phab:D524: Improve runtime system allocator performance with two-step allocation (thanks to Simon Marlow)
That's all for this time. Enjoy your week!
Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss the status of the imminent 7.10.2 release.
In the past weeks we have been busily tracking down a number of regressions in
ghc-7.10 branch. At this point we have built up an impressive
list of fixes.
Thanks to everyone who has helped in this process!
In addition to resolving a number of simplifier regressions and portability issues (some mentioned in the Weekly News from 6 July), GHC 7.10.2 should be the first release which works out-of-the-box with GHCJS, thanks to a fix from Luite Stegeman. Moreover, we will have support for running in even the most minimal container environment (#10623).
At this point we have successfully tested the pre-release against Stackage (thanks for Michael Snoyman and Herbert Valerio Riedel for making this possible) and have found no major issues. A source tarball will be finalized and sent to the binary builders today. With luck we will have releasable binaries by the end of the week.
Thomas Miedema has been doing some amazing work fixing some long-standing
validation failures on the
branch. Moreover, he has been examining
the viability of enabling
larger swaths of the testsuite in
In addition, Thomas has been doing some great work smoothing over a variety of rough edges in the build system and generally keeping things running smoothly. Thanks Thomas!
Another subject of discussion in this week's GHC call was
Phabricator D757, a long-languishing
change which moves the generation of
Typeable instances from types' use sites to
their definition sites. This change involves a trade-off as it moves compilation
effort to the defining module, which will adversely affect compilation times of
modules defining many types. Moreover, in the event that
Typeable is never actually used
this time is wasted effort. That being said, the current design of generating
Typeable instances at the point of use makes the class a bit of a black sheep
at the moment.
Runtime system matters
This week Simon Marlow will merge his D524, a rework of the runtime system's allocator which he reports has improved performance significantly in his workloads. This commit splits the concerns of reserving address space and requesting backing memory for this address space. While the former is relatively cheap, the latter can be quite expensive due to page-table setup. Consequently, his rework allocates a large chunk of addressing space up front and then incrementally commits it as needed. Interested readers are encouraged to look at the patch, which offers a nice glimpse into the inner workings of GHC's memory allocator.
Simon also has finished Phab:D1076, which should improve garbage collector performance by reworking the logic responsible for scavenging static objects. This work will be merged shortly.
Also discussed was the profiler mis-attribution issue mentioned in the Weekly News from 6 July 2015 (#10007). Peter Wortmann is currently looking at this issue, which ends up being due to an infelicity in the semantics implemented by the runtime system. Simon Marlow expressed that he did not know of a way to resolve this that isn't quite invasive. We will have to see what Peter proposes.
For some time now Simon Marlow has been working on implementing the ApplicativeDo proposal. Today in the call we discussed the status of this work and concluded that while some refactoring is likely possible, the work can be merged as-is. Hopefully at some point in the next few weeks this work will land.
Haddock comments for GADT constructors
It came to our attention that the GHC parser was unable to parse Haddock comments attached to GADT constructors. As it turns out, this is a rather long-standing problem. Despite this fact, the fix ended up being quite straightforward and will be in 7.12.
In general one should be able to compile the current GHC
master branch with
the last two compiler releases. Recently, however, the reality is a bit less
clear-cut: while the current
ghc-7.10 branch GHC will technically build with
GHC 7.6 and 7.8, the tree does not necessarily pass the
validate script due
to a variety of imports rendered redundant by AMP and FTP. Moreover, the
official policy on backwards compatibility is rather well-hidden on the
This was discussed in today's meeting and it was decided that we will in the future
maintain full validation-compatibility with the previous two releases. To ease this
we will relax the use of
-Werror when building the stage 1 compiler.
On a related note, this week Thomas Miedema
ripped out some
#ifdefs for GHC 7.6
compatibility from the
master branch. Be aware that you will now need GHC 7.8 or newer to build
Mac OS X El Capitan support
It is well-known that the next Mac OS X release, El Capitan, will default to
"root-less" mode, a security feature which restricts the operations available to
root user. As a result of this feature some system calls in the El Capitan developer preview fail with
EPERM instead of the usual
change uncovered a bug in the
unix library where
EPERM was not treated
EACCES. This was
fixed in November 2014, a fix which
is included in GHC 7.10.1.
However, there have been a
ghc-devs for a bugfix release of the 7.8 series including the updated
discussed this in the call today and concluded that we would not make such a
release. Given that El Capitan is not yet released and the issue is fixed in 7.10, it doesn't seem worthwhile to put more developer time into 7.8. We would suggest that any El
Capitan user unable to update to GHC 7.10.1 or newer disable root-less mode for
the time being. This can be accomplished with,
sudo nvram boot-args="rootless=0"
Recently it came to our attention that one of the build machines used by Harbormaster (Phabricator's build mechanism) was still running GHC 7.6. If you have seen strange validation issues on Harbormaster builds in the past, this is may be the cause. As of today this is fixed; all Harbormaster validations are now being built with GHC 7.8.4.
Harbormaster has historically had trouble working with Differentials which
changed submodule revisions. This has made testing revisions involving submodules quite tricky. Thanks to a
patch from Adam Sandberg Eriksson Harbormaster can now grab submodule commits from non-upstream repositories if set in
Herbert Valerio Riedel has been making great strides improving the responsiveness of Trac. A Trac upgrade, a move to Postresql, and some fiddling with the WSGI configuration should result in a much better experience for everyone.
Have a great week!
Welcome for the latest entry in the GHC Weekly News. The past week, GHC HQ met up to discuss the status of the approaching 7.10.2 release.
After quite some delay due to a number of tricky regressions in 7.10.1, 7.10.2 is nearing the finish line. Austin cut release candidate 2 on Friday and so far the only reports of trouble appear to be some validation issues, most of which have already been fixed thanks to Richard Eisenberg.
7.10.2 will include a number of significant bug-fixes. These include,
- #10521, where overlap of floating point STG registers weren't properly accounted for, resulting in incorrect results in some floating point computations. This was fixed by the amazing Reid Barton.
- #10534, a type-safety hole enabling a user to write
unsafeCoercewith data families and
coerce. Fix due to the remarkable Richard Eisenberg.
- #10538, where some programs would cause the simplifier to emit an empty case, resulting in runtime crashes. Fix due to the industrious Simon Peyton Jones.
- #10527, where the simplifier would expend a great deal of effort simplifying arguments which were never demanded by the callee.
- #10414, where a subtle point of the runtime system's black-holing mechanism resulting in hangs on a carefully constructed testcase.
- #10236, where incorrect DWARF annotations would be generated, resulting in incorrect backtraces. Fixed by Peter Wortmann
- #7450, where cached free variable information was being unnecessarily dropped by the specialiser, resulting in non-linear compile times for some programs.
See the status page for a complete listing of issues fixed in this release.
In the coming days we will being working with FP Complete to test the pre-release against Stackage. While Hackage tends to be too large to build in bulk, the selection of packages represented in Stackage is feasible to build and is likely to catch potential regressions. Hopefully this sort of large-scale validation will become common-place for future releases.
If all goes well, 7.10.2 will mark the end of the 7.10 series. However, there is always the small possibility that a major regression will be found. In this case we will cut a 7.10.3 release which will include a few patches which didn't make it into 7.10.2.
It has been suggested in #10601 that GHC builds should ship with DWARF symbols for the base libraries and runtime system. While we all want to see this new capability in users' hands, 7.10.2 will, like 7.10.1, not be shipping with debug symbols. GHC HQ will be discussing the feasibility of including debug symbols with 7.12 in the future. In the meantime, we will be adding options to
build.mk to make it easier for users to build their own compilers with debug-enabled libraries.
In this week's GHC meeting the effort to port GHC's build system to the Shake? build system briefly came up. Despite the volume of updates on the Wiki Simon reports that the project is still moving along. The current state of the Shake-based build system can be found on Github.
While debugging #7540 it became clear that there may be trouble lurking in the profiler. Namely when profiling GHC itself
lintAnnots is showing up strongly where it logically should not. This was previously addressed in #10007, which was closed after a patch by Simon Marlow was merged. As it appears that this did not fully resolve the issue I'll be looking further into this.