Posts for the month of July 2015

GHC Weekly News - 2015/07/29

Hi *,

Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss plans post-7.10.2.

GHC 7.10.2 release

GHC 7.10.2 has been released!

Feel free to grab a tarball and enjoy! See the release notes for discussion of what has changed.

As always, if you suspect that you have found a regression don't hesitate to open a Trac ticket. We are especially interested in performance regressions with fairly minimal reproduction cases.

GHC 7.10.2 and the text package

A few days ago a report came in of long compilations times under 7.10.2 on a program with many Text literals (#10528). This ended up being due to a change in the simplifier which caused it to perform rule rewrites on the left-hand-side of other rules. While this is questionable (read "buggy") behavior, it doesn't typically cause trouble so long as rules are properly annotated with phase control numbers to ensure they are performed in the correct order. Unfortunately, it turns out that the rules provided by the text package for efficiently handling string literals did not include phase control annotations. This resulted in a rule from base being performed on the literal rules, which rendered the literal rules ineffective. The simplifier would then expend a great deal of effort trying to simplify the rather complex terms that remained.

Thankfully, the fix is quite straightforward: ensure that the the text literal rules fire in the first simplifier phase (phase 2). This avoids interference from the base rules, allowing them to fire as expected.

This fix is now present in text-1.2.1.2. Users of GHC 7.10.2 should be use this release if at all possible. Thanks to text's maintainer, Bryan O'Sullivan for taking time out of his vacation to help me get this new release out.

While this mis-behaviour was triggered by a bug in GHC, a similar outcome could have arisen even without this bug. This highlights the importance of including phase control annotations on INLINE and RULE pragmas: Without them the compiler may choose the rewrite in an order that you did not anticipate. This has also drawn attention to a few shortcomings in the current rewrite rule mechanism, which lacks the expressiveness to encode complex ordering relationships between rules. This limitation pops up in a number of places, including when trying to write rules on class-overloaded functions. Simon Peyton Jones is currently pondering possible solutions to this on #10595.

StrictData

This week we merged the long-anticipated -XStrictData extension (Phab:D1033) by Adam Sandberg Ericsson. This implements a subset of the [StrictPragma] proposal initiated by Johan Tibell.In particular, StrictData allows a user to specify that datatype fields should be strict-by-default on a per-module basis, greatly reducing the syntactic noise introduced by this common pattern. In addition to implementing a useful feature, the patch ended up being a nice clean-up of the GHC's handling of strictness annotations.

What remains of this proposal is the more strong -XStrict extension which essentially makes all bindings strict-by-default. Adam has indicated that he may take up this work later this summer.

AMP-related performance regression

In late May Herbert Valerio Riedel opened Phab:D924, which removed an explicit definition for mapM in the [] Traversable instance, as well as redefined mapM_ in terms of traverse_ to bring consistency with the post-AMP world. The patch remains unmerged, however, due to a failing ghci testcase. It turns out the regression is due to the redefinition of mapM_, which uses (*>) where (>>) was once used. This tickles poor behavior in ghci's ByteCodeAsm module. The problem can be resolved by defining (*>) = (>>) in the Applicative Assembler instance (e.g. Phab:1097). That being said, the fact that this change has already exposed performance regressions raises doubts as to whether it is prudent.

GHC Performance work

Over the last month or so I have been working on nailing down a variety of performance issues in GHC and the code it produces. This has resulted in a number of patches which in some cases dramatically improve compilation time (namely Phab:1012 and Phab:D1041). Now since 7.10.2 is out I'll again be spending most of my time on these issues. We have heard a number of reports that GHC 7.10 has regressed on real-world programs. If you have a reproducible performance regression that you would like to see addressed please open a Trac ticket.

Merged patches

  • Phab:D1028: Fixity declarations are now allowed for infix data constructors in GHCi (thanks to Thomas Miedema)
  • Phab:D1061: Fix a long-standing correctness issue arising when pattern matching on floating point values
  • Phab:D1085: Allow programs to run in environments lacking iconv (thanks to Reid Barton)
  • Phab:D1094: Improve code generation in integer-gmp (thanks to Reid Barton)
  • Phab:D1068: Implement support for the MO_U_Mul2 MachOp in the LLVM backend (thanks to Michael Terepeta)
  • Phab:D524: Improve runtime system allocator performance with two-step allocation (thanks to Simon Marlow)

That's all for this time. Enjoy your week!

Cheers,

  • Ben

GHC Weekly News - 2015/07/21

Hi *,

Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss the status of the imminent 7.10.2 release.

7.10.2 Status

In the past weeks we have been busily tracking down a number of regressions in the ghc-7.10 branch. At this point we have built up an impressive list of fixes. Thanks to everyone who has helped in this process!

In addition to resolving a number of simplifier regressions and portability issues (some mentioned in the Weekly News from 6 July), GHC 7.10.2 should be the first release which works out-of-the-box with GHCJS, thanks to a fix from Luite Stegeman. Moreover, we will have support for running in even the most minimal container environment (#10623).

At this point we have successfully tested the pre-release against Stackage (thanks for Michael Snoyman and Herbert Valerio Riedel for making this possible) and have found no major issues. A source tarball will be finalized and sent to the binary builders today. With luck we will have releasable binaries by the end of the week.

7.11 Things

Testing

Thomas Miedema has been doing some amazing work fixing some long-standing validation failures on the master branch. Moreover, he has been examining the viability of enabling larger swaths of the testsuite in Harbormaster validations.

In addition, Thomas has been doing some great work smoothing over a variety of rough edges in the build system and generally keeping things running smoothly. Thanks Thomas!

Typeable implementation

Another subject of discussion in this week's GHC call was Phabricator D757, a long-languishing change which moves the generation of Typeable instances from types' use sites to their definition sites. This change involves a trade-off as it moves compilation effort to the defining module, which will adversely affect compilation times of modules defining many types. Moreover, in the event that Typeable is never actually used this time is wasted effort. That being said, the current design of generating Typeable instances at the point of use makes the class a bit of a black sheep at the moment.

Runtime system matters

This week Simon Marlow will merge his D524, a rework of the runtime system's allocator which he reports has improved performance significantly in his workloads. This commit splits the concerns of reserving address space and requesting backing memory for this address space. While the former is relatively cheap, the latter can be quite expensive due to page-table setup. Consequently, his rework allocates a large chunk of addressing space up front and then incrementally commits it as needed. Interested readers are encouraged to look at the patch, which offers a nice glimpse into the inner workings of GHC's memory allocator.

Simon also has finished Phab:D1076, which should improve garbage collector performance by reworking the logic responsible for scavenging static objects. This work will be merged shortly.

Also discussed was the profiler mis-attribution issue mentioned in the Weekly News from 6 July 2015 (#10007). Peter Wortmann is currently looking at this issue, which ends up being due to an infelicity in the semantics implemented by the runtime system. Simon Marlow expressed that he did not know of a way to resolve this that isn't quite invasive. We will have to see what Peter proposes.

Applicative do

For some time now Simon Marlow has been working on implementing the ApplicativeDo proposal. Today in the call we discussed the status of this work and concluded that while some refactoring is likely possible, the work can be merged as-is. Hopefully at some point in the next few weeks this work will land.

Haddock comments for GADT constructors

It came to our attention that the GHC parser was unable to parse Haddock comments attached to GADT constructors. As it turns out, this is a rather long-standing problem. Despite this fact, the fix ended up being quite straightforward and will be in 7.12.

Backwards Compatibility

In general one should be able to compile the current GHC master branch with the last two compiler releases. Recently, however, the reality is a bit less clear-cut: while the current ghc-7.10 branch GHC will technically build with GHC 7.6 and 7.8, the tree does not necessarily pass the validate script due to a variety of imports rendered redundant by AMP and FTP. Moreover, the official policy on backwards compatibility is rather well-hidden on the Commentary/CodingStyle page.

This was discussed in today's meeting and it was decided that we will in the future maintain full validation-compatibility with the previous two releases. To ease this we will relax the use of -Werror when building the stage 1 compiler.

On a related note, this week Thomas Miedema ripped out some #ifdefs for GHC 7.6 compatibility from the master branch. Be aware that you will now need GHC 7.8 or newer to build master.

Mac OS X El Capitan support

It is well-known that the next Mac OS X release, El Capitan, will default to "root-less" mode, a security feature which restricts the operations available to even the root user. As a result of this feature some system calls in the El Capitan developer preview fail with EPERM instead of the usual EACCES. This change uncovered a bug in the unix library where EPERM was not treated similarly to EACCES. This was fixed in November 2014, a fix which is included in GHC 7.10.1.

However, there have been a few calls on ghc-devs for a bugfix release of the 7.8 series including the updated unix. We discussed this in the call today and concluded that we would not make such a release. Given that El Capitan is not yet released and the issue is fixed in 7.10, it doesn't seem worthwhile to put more developer time into 7.8. We would suggest that any El Capitan user unable to update to GHC 7.10.1 or newer disable root-less mode for the time being. This can be accomplished with,

sudo nvram boot-args="rootless=0"

Infrastructure

Recently it came to our attention that one of the build machines used by Harbormaster (Phabricator's build mechanism) was still running GHC 7.6. If you have seen strange validation issues on Harbormaster builds in the past, this is may be the cause. As of today this is fixed; all Harbormaster validations are now being built with GHC 7.8.4.

Harbormaster has historically had trouble working with Differentials which changed submodule revisions. This has made testing revisions involving submodules quite tricky. Thanks to a patch from Adam Sandberg Eriksson Harbormaster can now grab submodule commits from non-upstream repositories if set in .gitmodules.

Herbert Valerio Riedel has been making great strides improving the responsiveness of Trac. A Trac upgrade, a move to Postresql, and some fiddling with the WSGI configuration should result in a much better experience for everyone.

Have a great week!

~ Ben

  • Posted: 2015-07-21 19:20 (Updated: 2015-07-21 22:37)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)

GHC Weekly News - 2015/07/06

Hi *,

Welcome for the latest entry in the GHC Weekly News. The past week, GHC HQ met up to discuss the status of the approaching 7.10.2 release.

7.10.2 status

After quite some delay due to a number of tricky regressions in 7.10.1, 7.10.2 is nearing the finish line. Austin cut release candidate 2 on Friday and so far the only reports of trouble appear to be some validation issues, most of which have already been fixed thanks to Richard Eisenberg.

7.10.2 will include a number of significant bug-fixes. These include,

  • #10521, where overlap of floating point STG registers weren't properly accounted for, resulting in incorrect results in some floating point computations. This was fixed by the amazing Reid Barton.
  • #10534, a type-safety hole enabling a user to write unsafeCoerce with data families and coerce. Fix due to the remarkable Richard Eisenberg.
  • #10538, where some programs would cause the simplifier to emit an empty case, resulting in runtime crashes. Fix due to the industrious Simon Peyton Jones.
  • #10527, where the simplifier would expend a great deal of effort simplifying arguments which were never demanded by the callee.
  • #10414, where a subtle point of the runtime system's black-holing mechanism resulting in hangs on a carefully constructed testcase.
  • #10236, where incorrect DWARF annotations would be generated, resulting in incorrect backtraces. Fixed by Peter Wortmann
  • #7450, where cached free variable information was being unnecessarily dropped by the specialiser, resulting in non-linear compile times for some programs.

See the status page for a complete listing of issues fixed in this release.

In the coming days we will being working with FP Complete to test the pre-release against Stackage. While Hackage tends to be too large to build in bulk, the selection of packages represented in Stackage is feasible to build and is likely to catch potential regressions. Hopefully this sort of large-scale validation will become common-place for future releases.

If all goes well, 7.10.2 will mark the end of the 7.10 series. However, there is always the small possibility that a major regression will be found. In this case we will cut a 7.10.3 release which will include a few patches which didn't make it into 7.10.2.

Other matters

It has been suggested in #10601 that GHC builds should ship with DWARF symbols for the base libraries and runtime system. While we all want to see this new capability in users' hands, 7.10.2 will, like 7.10.1, not be shipping with debug symbols. GHC HQ will be discussing the feasibility of including debug symbols with 7.12 in the future. In the meantime, we will be adding options to build.mk to make it easier for users to build their own compilers with debug-enabled libraries.

In this week's GHC meeting the effort to port GHC's build system to the Shake? build system briefly came up. Despite the volume of updates on the Wiki Simon reports that the project is still moving along. The current state of the Shake-based build system can be found on Github.

While debugging #7540 it became clear that there may be trouble lurking in the profiler. Namely when profiling GHC itself lintAnnots is showing up strongly where it logically should not. This was previously addressed in #10007, which was closed after a patch by Simon Marlow was merged. As it appears that this did not fully resolve the issue I'll be looking further into this.

~ Ben

  • Posted: 2015-07-06 16:25 (Updated: 2015-07-07 09:35)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)