Recent posts (max 20) - Browse or Archive for more

Meet Jenkins: GHC's new CI and build infrastructure

While Phabricator is generally well-liked among GHC developers, GHC's interaction with Harbormaster, Phabricator's continuous integration component, has been less than rosy. The problem is in large part a mismatch between Harbormaster's design assumptions and GHC's needs, but it's also in part attributable to the somewhat half-finished state in which Harbormaster seems to linger. Regardless, we won't go into detail here; these issues are well covered elsewhere.

Suffice it to say that, after having looked at a number of alternatives to Harbormaster (including buildbot, GitLab's Pipelines, Concourse, and home-grown solutions), Jenkins seems to be the best option at the moment. Of course, this is not to say that it is perfect; as we have learned over the last few months it is very far from perfect. However, it has the maturity and user-base to be almost-certainly able to handle what we need of it on the platforms that we care about.

Let's see what we get out of this new bit of infrastructure:

Pre-merge testing

Currently there are two ways that code ends up in master,

  • a Differential is opened, built with Harbormaster, and eventually landed (hopefully, but not always, after Harbormaster successfully finishes)
  • someone pushes commits directly

Bad commits routinely end up merged via both channels. This means that authors of patches failing CI often need to consider whether *their* patch is incorrect or whether they rather simply had the misfortune of basing their patch on a bad commit. Even worse, if the commit isn't quickly reverted or fixed GHC will end up with a hole in its commit history where neither bisection nor performance tracking will be possible. For these reasons, we want to catch these commits before they make it into master.

To accomplish this we have developed some tooling to run CI on commits *before* they are finally merged to master. By making CI the only path patches can take to get to master, improve our changes of rejecting bad patches before they turn the tree red.

Automation of the release builds

Since the 7.10.3 release we have been gradually working towards automating GHC's release process. Thanks to this work, today a single person can build binary distributions for all seven tier-1 configurations in approximately a day, most of which is spent simply waiting. This has allowed us to take responsibility (starting in 8.2.1) for the OpenBSD, FreeBSD, ARMv7 and AArch64 builds in addition to the traditional tier-1 platforms, allowing us to eliminate the week-long wait between source distribution availability and the binary distribution announcement previously needed for correspondence with binary build contributors..

However, we are far from done: our new Jenkins-based build infrastructure (see #13716) will allow us to produce binary distributions directly from CI, reducing the cost of producing release builds to nearly nothing.

Testing of GHC against user packages

While GHC is already tested against Hackage and Stackage prior to release candidate availability, these builds have been of limited use as packages low on the dependency tree (think hashable and lens) often don't build prior to the first release candidate. While we do our best to fix these packages up, the sheer number of them makes this a losing battle for a small team such as GHC's.

Having the ability to cheaply produce binary distributions means that we can produce and validate nightly snapshot releases. This gives users a convenient way to test pre-release compilers and fix their libraries accordingly. We hope this will spread the maintenance effort across a larger fraction of the Haskell community and over a longer period of time, meaning there will be less to do at release time and consequently pre-release Stackage builds will be more fruitful.

Once the Jenkins infrastructure is stable, we can consider introducing nightly builds of user packages as well. While building a large population such as Stackage would likely not be productive, working with a smaller sample of popular, low-dependency-count packages would be quite possible. For testing against larger package repositories, leaning on a dedicated tool such as the Hackage Matrix Builder will likely be a more productive path.

Expanded platform coverage of CI

While GHC targets a wide variety of architectures and operating systems (and don't forget cross-compilation targets), by far the majority of developers use Linux, Darwin, or Windows on amd64. This means that breakage often only comes to light long after the culpable patch was merged.

Of course, GHC, being a project with modest financial resources, can't test each commit on every supported platform. We can, however, shrink the time between a bad commit being merged and the breakage being found by testing these "unusual" platforms on a regular (e.g. nightly) basis.

By catching regressions early, we hope to reduce the amount of time spent bisecting and fixing bugs around release time.

Tracking core libraries

Keeping GHC's core library dependencies (e.g. directory, process) up-to-date with their respective upstreams is important to ensure that tools that link against the ghc library (e.g. ghc-mod) can build easily. However, it also requires that we work with nearly a dozen upstream maintainers at various points in their own release cycles to arrange that releases are made prior to the GHC release. Moreover, there is inevitably a fair amount of work propagating verion bounds changes down the dependency tree. While this work takes relatively little effort in terms of man-hours,

Jenkins can help us here by allowing us to automate integration testing of upstream libraries, catching bounds issues and other compatibility issues well before they are in the critical path of the release.

Improved debugging tools

One of the most useful ways to track down a bugs in GHC is bisection. This is especially true for regressions found in release candidates, where you have at most a few thousand commits to bisect through. Nevertheless, GHC builds are long and developer time scarce so this approach isn't used as often as it could be.

Having an archive of nightly GHC builds will free the developer from having to build dozens of compilers during bisection, making the process a significantly more enjoyable experience than it is today. This will allow us to solve more bugs in less time and with far fewer grey hairs.

Status of Jenkins effort

The Jenkins CI overhaul has been an on-going project throughout the spring and summer and is nearing completion. The Jenkins configuration can be seen in the wip/jenkins branch on git.haskell.org (gitweb). At the moment the prototype is running on a few private machines but we will be setting up a publicly accessible test instance in the coming weeks. Jenkins will likely coexist with our current Harbormaster infrastructure for a month or so while we validate that things are stable.

Reflections on GHC's release schedule

Looking back on GHC's past release schedule reveals a rather checkered past,

Release Date Time to next major release
6.12.1 mid December 2009
12 months
7.0.1 mid November 2010
9.5 months
7.2.1 early August 2011
6 months
7.4.1 early February 2012
7 months
7.6.1 early September 2012
19 months
7.8.1 early April 2014
13 months
7.10.1 late March 2015
14 months
8.0.1 late May 2016
14 months
8.2.1 late July 2017
-
8.4.1 TDB and the topic of this post

There are a few things to notice here:

  • release cadence has swung rather wildly
  • the release cycle has stretched in the last several releases
  • time-between-releases generally tends to be on the order of a year

While GHC is far from the only compiler with such an extended release schedule, others (namely LLVM, Go, and, on the extreme end, Rust) have shown that shorter cycles are possible. I personally think that a more stable, shorter release cycle would be better for developers and users alike,

  • developers have a tighter feedback loop, inducing less pressure to get new features and non-critical bugfixes into minor releases
  • release managers have fewer patches to cherry-pick
  • users see new features and bugfixes more quickly

With 8.2.1 at long last behind us, now is a good time to reflect on why these cycles are so long, what release schedule we would like to have, and what we can change to realize such a schedule. On the way we'll take some time to examine the circumstances that lead to the 8.2.1 release which, while not typical, remind us that there is a certain amount of unpredictability inherent in developing large systems like GHC; a fact that must be born in mind when considering release policy.

Let's dig in...

The release process today

Cutting a GHC release is a fairly lengthy process involving many parties and a significant amount of planning. The typical process for a major release looks something like this,

  1. (a few months after the previous major release) A set of release priorities are defined determining which major features we want in the coming release
  2. wait until all major features are merged to the master branch
  3. when all features are merged, cut a stable branch
  4. in parallel:
    1. coordinate with core library authors to determine which library versions the new release should ship
    2. prepare release documentation
    3. do preliminary testing against Hackage and Stackage to identify and fix early bugs
    4. backport significant fixes merged to master
  5. when the tasks in (4) are sufficiently advanced, cut a source release for a release candidate
  6. produce tier-1 builds and send source tarballs to binary packagers, wait a week to prepare binary builds; if anyone finds the tree is unbuildable, go back to (5)
  7. upload release artifacts, announce release candidate
  8. wait a few weeks for testing
  9. if there are significant issues: fix them and return to (5)
  10. finalize release details (e.g. release notes, last check over core library versions)
  11. cut source tarball, send to binary build contributors, wait a week for builds
  12. announce final release, celebrate!

Typically the largest time-sinks in this process are waiting for regression fixes and coordinating with core library authors. In particular, the coordination involved in the latter isn't difficult, but merely high latency.

In the case of 8.2.1, the timeline looked something like this,

Time Event
Fall 2016 release priorities for 8.2 discussed
Early March 2017 stable branch cut
Early April 2017 most core library versions set
release candidate 1 cut
Mid May 2017 release candidate 2 cut
Early July 2017 release candidate 3 cut
Late July 2017 final release cut

Unexpected set-backs

This timeline was a bit more extended than desired for a few reasons.

The first issues were #13426 and #13535, compile-time performance regressions which came to light shortly after the branch and after the first release candidate, respectively. In #13535 it was observed that the testsuite of the vector package (already known for its propensity to reveal compiler regressions) increased by nearly a factor of five in compile-time allocations over 8.0.2.

While a performance regression would rarely classify as a release blocker, both the severity of the regressions combined with the fact that 8.2 was intended to be a performance-oriented release made releasing before fixes were available quite unappealing. For this reason David Feuer, Reid Barton, and I invested significant effort to try to track down the culprits. Unfortunately, the timescale on which this sort of bug is resolved span days, stretching to weeks when time is split with other responsibilities. While Reid's valiant efforts lead to the resolution of #13426, we were eventually forced to set #13535 aside as the release cycle wore on.

The second setback came in the form of two quite grave correctness issues (#13615, #13916) late in the cycle. GHC being a compiler, we take correctness very seriously: Users' confidence that GHC will compile their programs faithfully is crucial for language adoption, yet also very easily shaken. Consequently, while neither of these issues were regressions from 8.0, we deemed it important to hold the 8.2 release until these issues were resolved (which ended up being significant efforts in their own right; a blog post on this will be coming soon).

Finally, there was the realization (#13739) after release candidate 2 that some BFD linker releases suffered from very poor performance when linking with split-sections enabled (the default behavior in 8.2.1). This served as a forcing function to act on #13541, which we originally planned for 8.4. As expected, it took quite some time to follow through on this in a way that satisfied users and distribution packagers in a portable manner.

Moving forward: Compressing the release schedule

Collectively the above issues set the release back by perhaps six or eight weeks in total, including the additional release candidate necessary to validate the raft of resulting patches. While set-backs due to long-standing bugs are hard to avoid, there are a few areas where we can do better,

  1. automate the production of release artifacts
  2. regularly test GHC against user packages in between releases
  3. expand continuous integration of GHC to less common platforms to ensure that compatibility problems are caught before the release candidate stage
  4. regularly synchronize with core library maintainers between releases to reduce need for version bound bumps at release time
  5. putting in place tools to ease bisection, which is frequently a useful debugging strategy around release-time

As it turns out, nearly all of these are helped by our on-going effort to move GHC's CI infrastructure to Jenkins (see #13716). As this is a rather deep topic in its own right, I'll leave this more technical discussion for a second post (blog:jenkins-ci).

With the above tooling and process improvements, I think it would be feasible to get the GHC release cycle down to six months or shorter if we so desired. Of course, shorter isn't necessarily better: we need to be careful to balance the desire for a short release cycle against the need for an adequate post-release "percolation" time. This time is crucial to allow the community to adopt the new release, discover and fix its regressions. In fact, the predictability that a short release schedule (hopefully) affords is arguably more important than the high cadence itself.

Consequently, we are considering tightening up the release schedule for future GHC releases in a slow and measured manner. Given that we are now well into the summer, I think positioning the 8.4 release around February 2018, around seven months from now, would be a sensible timeline. However, we would like to hear your opinions.

Here are some things to think about,

  1. Do you feel that it takes too long for GHC features to make it to users' hands?
  2. How many times per year do you envision upgrading your compiler before the process becomes too onerous? Would the current load of interface changes per release be acceptable under a faster release cadence?
  3. Should we adjust the three-release policy to counteract a shorter GHC release cycle?
  4. Would you feel more likely to contribute to GHC if your work were more quickly available in a release?

We would love to hear your thoughts. Be sure to mention whether you are a user, GHC contributor, or both.

GHC 8.2.1 is available

The Glasgow Haskell Compiler -- version 8.2.1

The GHC developers are very happy to announce the long-awaited 8.2.1 release of Glasgow Haskell Compiler. Binary and source distributions can be found at <https://downloads.haskell.org/~ghc/8.2.1/>.

This is the second release in the 8.0 series. As such, the focus of this release is performance, stability, and consolidation. Consequently numerous cleanups can be seen throughout the compiler including,

  • Significant improvements in compiler performance
  • More robust support for levity polymorphism
  • Reliable DWARF debugging information
  • Improved runtime system performance on NUMA systems
  • Retooling of the cost-center profiler, including support for live streaming of profile data via the GHC event log
  • Interface file determinism
  • More robust treatment of join points, enabling significantly better code generation in many cases
  • Numerous improvements in robustness on Windows
  • and the resolution of over 500 other tickets

In addition, there are a number of new features,

  • A new, more type-safe type reflection mechanism
  • The long-awaited Backpack module system
  • Deriving strategies to disambiguate between GHC's various instance deriving mechanisms
  • Unboxed sum types, for efficient unpacked representation of sum data types
  • Compact regions, allowing better control over garbage collection in the presence of large heaps containing many long-lived objects.
  • Colorful messages and caret diagnostics for more legible errors

A more thorough list of the changes in this release can be found in the [[https://haskell.org/ghc/docs/8.2.1/html/users_guide/8.2.1-notes.html |release notes]].

There are a few changes in release-engineering that should be noted,

  • Binary distributions for 32-bit CentOS 6.7 have been dropped. Moreover, there are no dedicated CentOS 7.0 distributions as CentOS 7 can use can use Debian 8 binaries. If you would like us to continue to produce 32-bit CentOS 6.7 distributions please let us know.
  • GHC HQ now builds FreeBSD and OpenBSD distributions for amd64; this comes after many years of these distributions being faithfully provided by Karel Gardas and Pali Gabor Janos, who we should heartily thank for their contributions.

GHC HQ building these distributions ourselves will allow us to more quickly ship distributions to users by eliminating the need for a long lag time between source release availability and having all binary distributions available.

  • There is a technology-preview of an AArch64 Linux binary distribution, as well as an ARM Linux build. AArch64 support is quite preliminary but should be stable in 8.4 thanks to further linker fixes by Moritz Angerman. ARM should be stable.
  • GHC now tries to use the gold and lld linkers by default. These linkers are significantly faster than the BFD linker implementation that most Linux distributions use by default. If gold or lld are not available GHC will use the system's default linker. GHC can be forced to use the default linker by passing --disable-ld-override to configure.

This release has been the result of over a year of hard work by over 150 code contributors. Thanks to everyone who has helped in writing patches, testing, reporting bugs, and offering feedback over the last year.

This release cycle was admittedly quite drawn out, significantly longer than expected or desired. While we are confident that the result is worth the wait, we have been steadily working on infrastructure which should help shrink future release cycles and give us better testing between releases. More details on this coming soon.

As always, let us know if you encounter trouble.

How to get it

Both the source tarball and binary distributions for a wide variety of platforms are available here.

Background

Haskell is a standard lazy functional programming language.

GHC is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating efficient code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces. GHC is distributed under a BSD-style open source license.

A wide variety of Haskell related resources (tutorials, libraries, specifications, documentation, compilers, interpreters, references, contact information, links to research groups) are available from the Haskell home page (see below).

On-line GHC-related resources

Relevant URLs on the World-Wide Web:

Supported Platforms

The list of platforms we support, and the people responsible for them, is here:

http://ghc.haskell.org/trac/ghc/wiki/Contributors

Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform:

http://ghc.haskell.org/trac/ghc/wiki/Building

Developers

We welcome new contributors. Instructions on accessing our source code repository, and getting started with hacking on GHC, are available from the GHC's developer's site run by Trac.

Community Resources

There are mailing lists for GHC users, develpoers, and monitoring bug tracker activity; to subscribe, use the Mailman web interface.

There are several other Haskell and GHC-related mailing lists on haskell.org; for the full list, see the lists page.

Some GHC developers hang out on the #ghc and #haskell of the Freenode IRC network, too. See the Haskell wiki for details.

Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here.

GHC 8.0.2 is available!

The GHC team is happy to at last announce the 8.0.2 release of the Glasgow Haskell Compiler. Source and binary distributions are available at

http://downloads.haskell.org/~ghc/8.0.2/

This is the second release of the 8.0 series and fixes nearly two-hundred bugs. These include,

  • Interface file build determinism (#4012).
  • Compatibility with macOS Sierra and GCC compilers which compile position-independent executables by default
  • Compatibility with systems which use the gold linker
  • Runtime linker fixes on Windows (see #12797)
  • A compiler bug which resulted in undefined reference errors while compiling some packages (see #12076)
  • A number of memory consistency bugs in the runtime system
  • A number of efficiency issues in the threaded runtime which manifest on larger core counts and large numbers of bound threads.
  • A typechecker bug which caused some programs using -XDefaultSignatures to be incorrectly accepted.
  • More than two-hundred other bugs. See Trac for a complete listing.
  • #12757, which lead to broken runtime behavior and even crashes in the presence of primitive strings.
  • #12844, a type inference issue affecting partial type signatures.
  • A bump of the directory library, fixing buggy path canonicalization behavior (#12894). Unfortunately this required a major version bump in directory and minor bumps in several other libraries.
  • #12912, where use of the select system call would lead to runtime system failures with large numbers of open file handles.
  • #10635, wherein -Wredundant-constraints was included in the -Wall warning set

A more detailed list of the changes included in this release can be found in the release notes.

Please note that this release breaks with our usual tendency to avoid major version bumps of core libraries in minor GHC releases by including an upgrade of the directory library to 1.3.0.0.

Also note that, due to a rather serious bug (#13100) affecting Windows noticed late in the release cycle, the Windows binary distributions were produced using a slightly patched source tree. Users compiling from source for Windows should be certain to include this patch in their build.

This release is the result of six months of effort by the GHC development community. We'd like to thank everyone who has contributed code, bug reports, and feedback to this release. It's only due to their efforts that GHC remains a vibrant and exciting project.

How to get it

Both the source tarball and binary distributions for a wide variety of platforms are available here.

Background

Haskell is a standardized lazy functional programming language.

The Glasgow Haskell Compiler (GHC) is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating efficient code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces. GHC is distributed under a BSD-style open source license.

Supported Platforms

The list of platforms we support, and the people responsible for them, can be found on the GHC wiki

Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform.

Developers

We welcome new contributors. Instructions on getting started with hacking on GHC are available from GHC's developer site.

Community Resources

There are mailing lists for GHC users, develpoers, and monitoring bug tracker activity; to subscribe, use the Mailman web interface.

There are several other Haskell and GHC-related mailing lists on haskell.org; for the full list, see the lists page.

Some GHC developers hang out on the #ghc and #haskell of the Freenode IRC network, too. See the Haskell wiki for details.

Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here.

Call for Nominations: GHC Steering Committee

Hello everyone,

As you likely know, over the last few months we have been discussing options for reforming the process for proposing language and compiler changes to GHC. After much discussion, we have a process which, while not perfect, is acceptable to a majority of our contributor base and will be an improvement over the status quo. While we invite suggestions for future improvements, we are at a point where we can move ahead with implementation.

Consequently, we are seeking nominations for the initial GHC steering committee. This body is responsible for overseeing the progression of proposals through the process, working with authors on refining their ideas, and evaluating proposals for acceptance. The committee will consist of five to eight members of diverse backgrounds.

We would like to offer the following as a criteria for membership. Note that a candidate is not expected to satisfy all or even most of these criteria, but a good candidate should satisfy at least one:

  • A history of contributions to the design of new language features
  • Experience developing Haskell libraries and applications
  • A demonstrated track record of contributing code to GHC
  • A pedagogical background, with experience in either teaching or authoring educational materials
  • Experience in compiler development
  • Knowledge of functional programming language theory

I would like to emphasize that committee membership is as much a duty as it is a privilege. Membership is not intended to be a platform to be used by members to drive their own ideas; rather it is a way of serving the Haskell community by helping other community members refine and advance their proposals. This, of course, requires an investment of time and effort, which you should be willing and able to consistently put forth.

If you would like to be considered for committee membership then please write a statement describing why you feel you are well-qualified to serve, in terms of the criteria above and any others that you would like to offer. Please send your statements to ben at well-typed.com by September 30th. The initial committee selection will be made by the Simons soon thereafter.

Thanks to everyone for their feedback and cooperation so far!

Cheers,

~ Ben

  • Posted: 2016-09-05 18:56 (Updated: 2016-09-05 18:57)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)

Rethinking GHC's approach to managing proposals

Recently there has been a fair bit of discussion around the mechanisms by which proposed changes to GHC are evaluated. While we have something of a formal proposal protocol, it is not clearly documented, inconsistently applied, and may be failing to serve a significant fraction of GHC's potential contributor pool.

Over the last few weeks, I have been doing a fair amount of reading, thinking, and discussing to try to piece together a proposal scheme which better serves our community.

The resulting proposal is strongly inspired by the RFC process in place in the Rust community, the leaders of which have thought quite hard about fostering community growth and participation. While no process is perfect, I feel like the Rust process is a good starting point for discussion, offering enough structure to guide new contributors through the process while requiring only a modest investment of developer time.

To get a sense for how well this will work in our community, I propose that we attempt to self-host the proposed process. To this end I have setup a ghc-proposals repository and opened a pull request for discussion of the process proposal. Let's see how this goes.

Cheers,

~ Ben

Contributing to GHC

This post is a response to Anthony's blog post about contributing to GHC, and the subsequent Reddit discussion. You'll find it easier to follow my comments below if you read Anthony's post first. Short summary: many of Anthony's criticisms are accurate, but they are not easy to address, especially in a volunteer-only project. However we will do something about the feature-request process.

Barrier to entry

I am ashamed that GHC seems difficult to contribute to. The details don't matter; the fact that it made you feel that way is by definition bad. I'm really sorry. We should find a way to do better.

An underlying issue is one of effort budget. GHC has essentially no full-timers, unlike many open-source projects. In particular, Simon Marlow and I are volunteers like everyone else, and like everyone else we have a day job. Microsoft Research generously pays Well Typed for front-line support, manifested in the heroic form of Ben and Austin, totalling around one FTE. But their effort is fully absorbed by managing releases, reviewing patches, maintaining the infrastructure, and so on.

It's a real challenge to maintain a low barrier to entry for a large complex project, whose motive force is primarily volunteers. It means that any initiative will only fly if someone steps up to drive it.

Technical documentation

The questions of scattered and inconsistent documentation are difficult to address. GHC is twenty-five years old; it has hundreds of authors and documentation that was once accurate falls out of date. I would love it to have consistent, well-explained documentation. But I don't know how to achieve it.

GHC's technical documentation is either in the code itself, or on GHC's wiki. An advantage of a wiki is that anyone can edit it. Yes, instructions and technical descriptions go out of date. Who will fix them? You, gentle reader! There is no one else.

But in practice few do. Perhaps that's because such work is invisible, so no one even knows you've done it? What would make people feel "I'd really like to improve those instructions or that documentation?"

I will argue for two things. First, I find Notes incredibly helpful. They live with the code, so they are less likely to go out of date. They don't mess up the flow of the code. They can be referred to from multiple places. They are the single most successful code documentation mechanism we have. (To be fair to Anthony, I don't think was complaining about Notes, just expressing surprise.)

Second, I really do think that each new proposed feature needs a description, somewhere, of what it is, why it's a good thing, and as precisely as possible what its specification is. Plus perhaps some implementation notes. It's very important when reading an implementation to know what it is that the code is seeking to implement. So yes, I do frequently ask for a specification.

Arc, github, phabricator, etc

Personally I carry no torch for arc. I do whatever I'm told, workflow-wise. If GitHub has got a lot better since we took the Arc decision, and there's a consensus that we should shift, I'm all for it provided someone explains to me what my new workflows should be. I am absolutely in favour of reducing barriers to contribution. (Other members of the GHC team have much stronger and better informed views than me, though.)

But there are costs to moving, especially if the move implies friction in the future. In particular, those that worry me most are those surrounding issue tracking. Github's issue tracker simply doesn't seem sufficient for a project as large and multi-faceted as GHC; in particular, tags as the only form of metadata is extremely limiting.

One alternative would be to use Github and Trac side-by-side, but then we face the issue of conflicting ticket number spaces as both systems insist on ticket numbers of the form #NNNN.

Coding style

Coding style is rather a personal thing, and we have been reluctant to enforce a single uniform style. (Quite apart from the task of reformatting 150k lines of Haskell.) Personally I don't find it an obstacle to reading other people's code.

Feature requests

That leaves the most substantial issue that Anthony poses: the process of making a contribution.

For fixing a bug, I think that (aside from the arc/Github debate) things are not too bad. On the arc/Github question, Austin has been working with the Phabricator maintainers to try to have them introduce a workflow which might be more familiar and more convenient for Github experts.

But for offering a new feature, Anthony argues that the process is unacceptably arduous. There are lots of things to think about

  • GHC has tons of features implemented by talented and motivated folk... who have since moved on. So when a new feature is proposed, my baseline guess is that I will personally be responsible for maintaining it in five years time. So I want to understand what the feature is. I want to understand how the implementation works. I want to be reasonably sure that it doesn't add a bunch of complexity to an already very complicated code base. And since any new feature adds some complexity, I want to have some confidence that the feature commands broad support -- even when it's behind a language extension flag.

So actually I think it's reasonable that the process should be somewhat arduous. A new feature imposes costs on every single person who works on that code in the future. We don't really make this point explicitly, but the contributor guidelines for Phabricator do. It might be helpful if we articulated similar guidelines.

  • "Any proposal needs a lightweight way to gauge broad support, then a period of constructive refinement". Indeed! What would be such a lightweight way? The trouble is that there is an enormous silent majority, and discussions are typically driven by a handful of articulate and well-informed contributors. All credit to them, but they may very well be an unrepresentative sample. I simply don't know how to gauge true broad support.
  • There is a problem with my own personal lack of bandwidth. I am one of the main gatekeepers for some big chunks of GHC, the renamer, typechecker and optimisation passes. That is good in a way, because if GHC lacked gatekeepers, it would soon lose conceptual integrity and become a huge ball of mud. But it is bad in other ways. I review a lot of code; but not fast enough! In prioritising I am guided by my (doubtless flawed) perceptions of things that lots of people are eagerly awaiting. The same thing goes for Simon Marlow, especially in the runtime system. We both have other day jobs. Even writing this post means that I'm not reviewing someone's code.

But I am acutely aware that "Simon and Simon are busy" is pretty cold comfort to someone awaiting a review. Maybe there should be a bunch of other people playing this role. That would be great. For example, Richard Eisenberg has taken responsibility for Template Haskell, which is totally fantastic. I would love to hear from people who are willing to take overall responsibility for a piece of GHC.

None of this is an argument for the status quo. Simon, Ben, Austin, Herbert, and I have been talking about a rather more systematic process for new features. We'd like to learn from experience elsewhere, rather than reinvent the wheel, such as the Rust process. Please suggest processes that you have seen working well elsewhere.

GHC 8.0.1 is available!

The GHC developers are very pleased to announce the release of the first new super-major version of our Haskell compiler in six years, GHC 8.0.1. This release features dozens of exciting developments including,

  • A more refined interface for implicit call-stacks, allowing libraries to provide more helpful runtime error messages to users
  • The introduction of the DuplicateRecordFields language extension, allowing multiple datatypes to declare fields of the same name
  • Significant improvements in error message readability and content, including facilities for libraries to provide custom error messages, more aggressive warnings for fragile rewrite rules, and more helpful errors for missing imports
  • A rewritten and substantially more thorough pattern match checker, providing more precise exhaustiveness checking in GADT pattern matches
  • More reliable debugging information including experimental backtrace support, allowing better integration with traditional debugging tools
  • Support for desugaring do-notation to use Applicative combinators, allowing the intuitive do notation to be used in settings which previously required the direct use of Applicative combinators
  • The introduction of Strict and StrictData language extensions, allowing modules to be compiled with strict-by-default evaluation of bindings
  • Great improvements in portability, including more reliable linking on Windows, a new PPC64 code generator, support for the AIX operating system, unregisterised m68k support, and significant stabilization on ARM targets
  • A greatly improved user's guide, with beautiful and modern PDF and HTML output
  • Introduction of type application syntax, reducing the need for proxies
  • More complete support for pattern synonyms, including record pattern synonyms and the ability to export patterns "bundled" with a type, as you would a data constructor
  • Support for injective type families and recursive superclass relationships
  • An improved generics representation leveraging GHC's support for type-level literals
  • The TypeInType extension, which unifies types and kinds, allowing GHC to reason about kind equality and enabling promotion of more constructs to the type level
  • ...and more!

A more thorough list of the changes included in this release can be found in the release notes,

As always, we have collected various points of interest for users of previous GHC releases on the GHC 8.0 migration page, Please let us know if you encounter anything missing or unclear on this page.

This release is the culmination of nearly eighteen months of effort by over one hundred contributors. We'd like to thank everyone who has contributed code, bug reports, and feedback over the past year. It's only because of their efforts that GHC continues to evolve.

How to get it

Both the source tarball and binary distributions for a wide variety of platforms are available here.

Background

Haskell is a standardized lazy functional programming language.

The Glasgow Haskell Compiler (GHC) is a state-of-the-art programming suite for Haskell. Included is an optimising compiler generating efficient code for a variety of platforms, together with an interactive system for convenient, quick development. The distribution includes space and time profiling facilities, a large collection of libraries, and support for various language extensions, including concurrency, exceptions, and foreign language interfaces. GHC is distributed under a BSD-style open source license.

Supported Platforms

The list of platforms we support, and the people responsible for them, can be found on the GHC wiki

Ports to other platforms are possible with varying degrees of difficulty. The Building Guide describes how to go about porting to a new platform.

Developers

We welcome new contributors. Instructions on getting started with hacking on GHC are available from GHC's developer site.

Community Resources

There are mailing lists for GHC users, develpoers, and monitoring bug tracker activity; to subscribe, use the Mailman web interface.

There are several other Haskell and GHC-related mailing lists on haskell.org; for the full list, see the lists page.

Some GHC developers hang out on the #ghc and #haskell of the Freenode IRC network, too. See the Haskell wiki for details.

Please report bugs using our bug tracking system. Instructions on reporting bugs can be found here.

GHC Weekly News - 2015/10/16

Hi *,

Welcome for the latest entry in the GHC Weekly News, which has been somewhat irregularly scheduled - but we're as busy as ever!

8.0.1 release roadmap

We're still steaming ahead on GHC 8.0.1 - any interested participants are suggested to look at the wiki:Status/GHC-8.0.1 page, to see where we're currently at.

These past few weeks have been good: we've gotten the first part of the overloaded record fields work in, and we have plans to land the kind equalities work in November. Furthermore, Simon Marlow, Peter Wortmann and Ben are working on reviewing all of the DWARF improvements, and hopefully the start of this work will land next week.

But 8.0 isn't the only thing that'll be getting released...

And some other minor releases

In a total, unprecedented upset - we're aiming to do three GHC releases in a fairly short amount of time.

7.10.3

Ben Gamari has been working on steadily hacking away at GHC 7.10.3, and the hopes are that we'll be able to ship it soon. This will fix several dozen bugs, some of which are critical for our users.

You can keep up to date by following the wiki:Status/GHC-7.10.3 page.

7.8.5

Simultaneously, your author will also be working on a GHC 7.8.5 release. While we were alerted a few months ago to this breakage, it seems rather unfortunate for the 7.8 series to remain broken on such a popular OS.

Furthermore, the "Three GHC Release Policy" for many authors - to support the last three major versions of GHC - would mean that 7.8 would be broken for OS X developers for an entire year until GHC 8.2.1. Which is a pretty unfortunate circumstance.

It's not expected the 7.8.5 release will contain any other fixes, however.

List chatter

(Over the past two weeks)

Noteworthy commits

(Over the past two weeks)

Closed tickets

(Over the past two weeks)

#10392, #7883, #10475, #10745, #10926, #9238, #10700, #10810, #10342, #365(!), #10361, #10929, #10563, #9907, #10513, #10868, #10932, #8920, #10516, #10416, #5966, #8335, #10520, #10687, #10571, #9058, #10939, #10938, #9590, #10949, #10153, #10947, #10948, #10936, #1883, #5289, #10733, #10950, #10611, #10959, #10960, #10831, #10796, #10890, #8010, #10216, #10965, #10953, #10964, #10931, #10714, #10888, #10633, #8652, #3971, #10882, #10977, #10267, and #10911.

GHC Weekly News - 2015/09/17

Hi *,

Welcome for the latest entry in the GHC Weekly News. It's been a little while, but here we are!

And your author has finally returned from his 8 week sabbatical, too! So without any futher ado, lets get going...

8.0.1 release roadmap

So HEAD has been steaming along pretty smoothly for the past few months now. After talking with Simon last week, we decided that the best course of action would be to release 8.0.1 (a super-major release) sometime around late February, which were the plans for 7.10 (modulo a few weeks due to the FTP debates). The current schedule is roughly:

  • November: Fork the new ghc-8.0 STABLE branch
    • At this point, master development will probably slow as we fix bugs.
    • This gives us 2 months or so until branch, from Today.
    • This is nice as the branch is close to the first RC.
  • December: First release candidate
  • Mid/late February: Final release.

"Why call it 8.0.1?", you ask? Because we have a lot of excellent features planned! I'm particularly partial to Richard's work for merging types and kinds (Phab:D808). But there's a lot of other stuff.

For all the nitty gritty details, be sure to check 8.0.1 status page to keep track of everything - it will be our prime source of information and coordination. And be sure to read my email to `ghc-devs` for more info.

... and a 7.10.3 release perhaps?

On top of this, we've been wondering if another release in the 7.10 branch should be done. Ben did the release shortly after I left, and for the most part looked pretty great. But there have been some snags, as usual.

So we're asking: who needs GHC 7.10.3? We'd really like to know of any major showstoppers you've found with 7.10 that are preventing you from using it. Especially if you're stuck or there's no clear workaround.

Currently, we're still not 100% committed to this course of action (since the release will take time away from other things). However, we'll keep the polls open for a while - so please get in touch with us if you need it! (Be sure to read my email for specific information.)

List chatter

(Over the past two weeks)

Noteworthy commits

(Over the past several weeks)

Closed tickets

(Over the past two weeks)

#10834, #10830, #10047, #9943, #1851, #1477, #8229, #8926, #8614, #10777, #8596, #10788, #9500, #9087, #10157, #10866, #10806, #10836, #10849, #10869, #10682, #10863, #10880, #10883, #10787, #8552, #10884, #7305, #5757, #9389, #8689, #10105, #8168, #9925, #10305, #4438, #9710, #10889, #10885, #10825, #10821, #10790, #10781, #9855, #9912, #10033, #9782, #10035, #9976, #10847, and #10865.

GHC Weekly News - 2015/08/06

GHC Weekly News - 6 Aug 2015

Hello *,

Here is a rather belated Weekly News which I found sitting nearly done on my work-queue. I hope this will make for a good read despite its age. The next edition of the Weekly News will be posted soon.

Warnings for missed specialization opportunities

Simon Peyton Jones recently [a4261549afaee56b00fbea1b4bc1a07c95e60929 introduced] a warning in master to alert users when the compiler was unable to specialize an imported binding despite it being marked as INLINEABLE. This change was motivated by #10720, where the reporter observed poor runtime performance despite taking care to ensure his binding could be inlined. Up until now, ensuring that the compiler's optimizations meet the user's expectation would require a careful look at the produced Core. With this change the user is notified of exactly where the compiler had to stop specializing, along with a helpful hint on where to add a INLINABLE pragma.

Ticky-Ticky profiling

Recently I have been looking into breathing life back into GHC's ticky-ticky profiling mechanism. When enabled, ticky-ticky maintains low-level counters of various runtime-system events. These include closure entries, updates, and allocations. While ticky doesn't provide nearly the detail that the cost-center profiler allows, it is invisible to the Core-to-Core optimization passes and has minimal runtime overhead (manifested as a bit more memory traffic due to counter updates). For this reason, the ticky-ticky profiler can be a useful tool for those working on the Core simplifier.

Sadly, ticky-ticky has fallen into quite a state of disrepair in recent years as the runtime system and native code generator have evolved. As the beginning of an effort to resuscitate the ticky-ticky profiler I've started putting together a list of the counters currently implemented and whether they can be expected to do something useful. Evaluating the functionality of these counters is non-trivial, however, so this will be an on-going effort.

One of our goals is to eventually do a systematic comparison of the heap allocation numbers produced by the ticky-ticky profiler, the cost-center profiler, and ticky-ticky. While this will help validate some of the more coarse-grained counters exposed by ticky, most of them will need a more thorough read-through of the runtime system to verify.

integer-gmp Performance

Since the 7.10.2 release much of my effort has been devoted to characterizing the performance of various benchmarks over various GHC versions. This is part of an effort to find places where we have regressed in the past few versions. One product of this effort is a complete comparison of results from our nofib benchmark suite ranging from 7.4.2 to 7.10.1.

The good news is there are essentially no disastrous regressions. Moreover, on the mean runtimes are over 10% faster than they were in 7.4.2. There are, however, a few cases which have regressed. The runtime of the integer test, for instance, has increased by 7%. Looking at the trend across versions, it becomes apparent that the regression began with 7.10.1.

One of the improvements that was introduced with 7.10 was a rewrite of the integer-gmp library, which this benchmark tests heavily. To isolate this potential cause, I recompiled GHC 7.10.1 with the old integer-gmp-0.5. Comparing 7.10.1 with the two integer-gmp versions reveals a 4% increase in allocations.

While we can't necessarily attribute all of the runtime increase to these allocations, they are something that should be addressed if possible. Herbert Valerio Riedel, the author of the integer-gmp rewrite, believes that the cause may be due to the tendency for the rewrite to initially allocate a conservatively-sized backing ByteArray# for results. This leads to increased allocations due to the reallocations that are later required to accommodate larger-than-expected results.

While being more liberal in the initial allocation sizes would solve the reallocation issue, this approach may substantially increase working-set sizes and heap fragmentation for integer-heavy workloads. For this reason, Herbert will be looking into exploiting a feature of our heap allocator. Heap allocations in GHC occur by bumping a pointer into an allocation block. Not only is this a very efficient means of allocating, it potentially allows one to efficiently grow an existing allocation. In this case, if we allocate a buffer and soon after realize that our request was too small we can simply bump the heap pointer by the size deficit, so long as no other allocations have occurred since our initial allocation. We can do this since we know that the memory after the heap pointer is available; we merely need to ensure that the current block we are allocating into is large enough.

Simon Marlow and Herbert will be investigating this possibility in the coming weeks.

D924: mapM_ and traverse_

As discussed in the most recent Weekly News, one issue on our plate at the moment is Phab:D924, which attempted to patch up two remaining facets of the Applicative-Monad Proposal,

  1. Remove the override of mapM for the [] Traversal instance
  2. Rewrite mapM_ in terms of traverse_

While (1) seems like an obvious cleanup, (2) is a bit tricky. As noted last time, traverse_ appears to give rise to non-linear behavior in this context.

akio has contributed an insightful [analysis https://ghc.haskell.org/trac/ghc/timeline?from=2015-08-01T10%3A00%3A33Z&precision=second] shedding light on the cause of this behavior. Given that the quadratic behavior is intrinsic to the Applicative formulation, we'll be sending this matter back to the Core Libraries Committee to inform their future design decisions.

That is all for this week!

Cheers,

~ Ben

  • Posted: 2015-08-25 21:14 (Updated: 2015-08-25 21:14)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)

GHC Weekly News - 2015/07/29

Hi *,

Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss plans post-7.10.2.

GHC 7.10.2 release

GHC 7.10.2 has been released!

Feel free to grab a tarball and enjoy! See the release notes for discussion of what has changed.

As always, if you suspect that you have found a regression don't hesitate to open a Trac ticket. We are especially interested in performance regressions with fairly minimal reproduction cases.

GHC 7.10.2 and the text package

A few days ago a report came in of long compilations times under 7.10.2 on a program with many Text literals (#10528). This ended up being due to a change in the simplifier which caused it to perform rule rewrites on the left-hand-side of other rules. While this is questionable (read "buggy") behavior, it doesn't typically cause trouble so long as rules are properly annotated with phase control numbers to ensure they are performed in the correct order. Unfortunately, it turns out that the rules provided by the text package for efficiently handling string literals did not include phase control annotations. This resulted in a rule from base being performed on the literal rules, which rendered the literal rules ineffective. The simplifier would then expend a great deal of effort trying to simplify the rather complex terms that remained.

Thankfully, the fix is quite straightforward: ensure that the the text literal rules fire in the first simplifier phase (phase 2). This avoids interference from the base rules, allowing them to fire as expected.

This fix is now present in text-1.2.1.2. Users of GHC 7.10.2 should be use this release if at all possible. Thanks to text's maintainer, Bryan O'Sullivan for taking time out of his vacation to help me get this new release out.

While this mis-behaviour was triggered by a bug in GHC, a similar outcome could have arisen even without this bug. This highlights the importance of including phase control annotations on INLINE and RULE pragmas: Without them the compiler may choose the rewrite in an order that you did not anticipate. This has also drawn attention to a few shortcomings in the current rewrite rule mechanism, which lacks the expressiveness to encode complex ordering relationships between rules. This limitation pops up in a number of places, including when trying to write rules on class-overloaded functions. Simon Peyton Jones is currently pondering possible solutions to this on #10595.

StrictData

This week we merged the long-anticipated -XStrictData extension (Phab:D1033) by Adam Sandberg Ericsson. This implements a subset of the [StrictPragma] proposal initiated by Johan Tibell.In particular, StrictData allows a user to specify that datatype fields should be strict-by-default on a per-module basis, greatly reducing the syntactic noise introduced by this common pattern. In addition to implementing a useful feature, the patch ended up being a nice clean-up of the GHC's handling of strictness annotations.

What remains of this proposal is the more strong -XStrict extension which essentially makes all bindings strict-by-default. Adam has indicated that he may take up this work later this summer.

AMP-related performance regression

In late May Herbert Valerio Riedel opened Phab:D924, which removed an explicit definition for mapM in the [] Traversable instance, as well as redefined mapM_ in terms of traverse_ to bring consistency with the post-AMP world. The patch remains unmerged, however, due to a failing ghci testcase. It turns out the regression is due to the redefinition of mapM_, which uses (*>) where (>>) was once used. This tickles poor behavior in ghci's ByteCodeAsm module. The problem can be resolved by defining (*>) = (>>) in the Applicative Assembler instance (e.g. Phab:1097). That being said, the fact that this change has already exposed performance regressions raises doubts as to whether it is prudent.

GHC Performance work

Over the last month or so I have been working on nailing down a variety of performance issues in GHC and the code it produces. This has resulted in a number of patches which in some cases dramatically improve compilation time (namely Phab:1012 and Phab:D1041). Now since 7.10.2 is out I'll again be spending most of my time on these issues. We have heard a number of reports that GHC 7.10 has regressed on real-world programs. If you have a reproducible performance regression that you would like to see addressed please open a Trac ticket.

Merged patches

  • Phab:D1028: Fixity declarations are now allowed for infix data constructors in GHCi (thanks to Thomas Miedema)
  • Phab:D1061: Fix a long-standing correctness issue arising when pattern matching on floating point values
  • Phab:D1085: Allow programs to run in environments lacking iconv (thanks to Reid Barton)
  • Phab:D1094: Improve code generation in integer-gmp (thanks to Reid Barton)
  • Phab:D1068: Implement support for the MO_U_Mul2 MachOp in the LLVM backend (thanks to Michael Terepeta)
  • Phab:D524: Improve runtime system allocator performance with two-step allocation (thanks to Simon Marlow)

That's all for this time. Enjoy your week!

Cheers,

  • Ben

GHC Weekly News - 2015/07/21

Hi *,

Welcome for the latest entry in the GHC Weekly News. Today GHC HQ met to discuss the status of the imminent 7.10.2 release.

7.10.2 Status

In the past weeks we have been busily tracking down a number of regressions in the ghc-7.10 branch. At this point we have built up an impressive list of fixes. Thanks to everyone who has helped in this process!

In addition to resolving a number of simplifier regressions and portability issues (some mentioned in the Weekly News from 6 July), GHC 7.10.2 should be the first release which works out-of-the-box with GHCJS, thanks to a fix from Luite Stegeman. Moreover, we will have support for running in even the most minimal container environment (#10623).

At this point we have successfully tested the pre-release against Stackage (thanks for Michael Snoyman and Herbert Valerio Riedel for making this possible) and have found no major issues. A source tarball will be finalized and sent to the binary builders today. With luck we will have releasable binaries by the end of the week.

7.11 Things

Testing

Thomas Miedema has been doing some amazing work fixing some long-standing validation failures on the master branch. Moreover, he has been examining the viability of enabling larger swaths of the testsuite in Harbormaster validations.

In addition, Thomas has been doing some great work smoothing over a variety of rough edges in the build system and generally keeping things running smoothly. Thanks Thomas!

Typeable implementation

Another subject of discussion in this week's GHC call was Phabricator D757, a long-languishing change which moves the generation of Typeable instances from types' use sites to their definition sites. This change involves a trade-off as it moves compilation effort to the defining module, which will adversely affect compilation times of modules defining many types. Moreover, in the event that Typeable is never actually used this time is wasted effort. That being said, the current design of generating Typeable instances at the point of use makes the class a bit of a black sheep at the moment.

Runtime system matters

This week Simon Marlow will merge his D524, a rework of the runtime system's allocator which he reports has improved performance significantly in his workloads. This commit splits the concerns of reserving address space and requesting backing memory for this address space. While the former is relatively cheap, the latter can be quite expensive due to page-table setup. Consequently, his rework allocates a large chunk of addressing space up front and then incrementally commits it as needed. Interested readers are encouraged to look at the patch, which offers a nice glimpse into the inner workings of GHC's memory allocator.

Simon also has finished Phab:D1076, which should improve garbage collector performance by reworking the logic responsible for scavenging static objects. This work will be merged shortly.

Also discussed was the profiler mis-attribution issue mentioned in the Weekly News from 6 July 2015 (#10007). Peter Wortmann is currently looking at this issue, which ends up being due to an infelicity in the semantics implemented by the runtime system. Simon Marlow expressed that he did not know of a way to resolve this that isn't quite invasive. We will have to see what Peter proposes.

Applicative do

For some time now Simon Marlow has been working on implementing the ApplicativeDo proposal. Today in the call we discussed the status of this work and concluded that while some refactoring is likely possible, the work can be merged as-is. Hopefully at some point in the next few weeks this work will land.

Haddock comments for GADT constructors

It came to our attention that the GHC parser was unable to parse Haddock comments attached to GADT constructors. As it turns out, this is a rather long-standing problem. Despite this fact, the fix ended up being quite straightforward and will be in 7.12.

Backwards Compatibility

In general one should be able to compile the current GHC master branch with the last two compiler releases. Recently, however, the reality is a bit less clear-cut: while the current ghc-7.10 branch GHC will technically build with GHC 7.6 and 7.8, the tree does not necessarily pass the validate script due to a variety of imports rendered redundant by AMP and FTP. Moreover, the official policy on backwards compatibility is rather well-hidden on the Commentary/CodingStyle page.

This was discussed in today's meeting and it was decided that we will in the future maintain full validation-compatibility with the previous two releases. To ease this we will relax the use of -Werror when building the stage 1 compiler.

On a related note, this week Thomas Miedema ripped out some #ifdefs for GHC 7.6 compatibility from the master branch. Be aware that you will now need GHC 7.8 or newer to build master.

Mac OS X El Capitan support

It is well-known that the next Mac OS X release, El Capitan, will default to "root-less" mode, a security feature which restricts the operations available to even the root user. As a result of this feature some system calls in the El Capitan developer preview fail with EPERM instead of the usual EACCES. This change uncovered a bug in the unix library where EPERM was not treated similarly to EACCES. This was fixed in November 2014, a fix which is included in GHC 7.10.1.

However, there have been a few calls on ghc-devs for a bugfix release of the 7.8 series including the updated unix. We discussed this in the call today and concluded that we would not make such a release. Given that El Capitan is not yet released and the issue is fixed in 7.10, it doesn't seem worthwhile to put more developer time into 7.8. We would suggest that any El Capitan user unable to update to GHC 7.10.1 or newer disable root-less mode for the time being. This can be accomplished with,

sudo nvram boot-args="rootless=0"

Infrastructure

Recently it came to our attention that one of the build machines used by Harbormaster (Phabricator's build mechanism) was still running GHC 7.6. If you have seen strange validation issues on Harbormaster builds in the past, this is may be the cause. As of today this is fixed; all Harbormaster validations are now being built with GHC 7.8.4.

Harbormaster has historically had trouble working with Differentials which changed submodule revisions. This has made testing revisions involving submodules quite tricky. Thanks to a patch from Adam Sandberg Eriksson Harbormaster can now grab submodule commits from non-upstream repositories if set in .gitmodules.

Herbert Valerio Riedel has been making great strides improving the responsiveness of Trac. A Trac upgrade, a move to Postresql, and some fiddling with the WSGI configuration should result in a much better experience for everyone.

Have a great week!

~ Ben

  • Posted: 2015-07-21 19:20 (Updated: 2015-07-21 22:37)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)

GHC Weekly News - 2015/07/06

Hi *,

Welcome for the latest entry in the GHC Weekly News. The past week, GHC HQ met up to discuss the status of the approaching 7.10.2 release.

7.10.2 status

After quite some delay due to a number of tricky regressions in 7.10.1, 7.10.2 is nearing the finish line. Austin cut release candidate 2 on Friday and so far the only reports of trouble appear to be some validation issues, most of which have already been fixed thanks to Richard Eisenberg.

7.10.2 will include a number of significant bug-fixes. These include,

  • #10521, where overlap of floating point STG registers weren't properly accounted for, resulting in incorrect results in some floating point computations. This was fixed by the amazing Reid Barton.
  • #10534, a type-safety hole enabling a user to write unsafeCoerce with data families and coerce. Fix due to the remarkable Richard Eisenberg.
  • #10538, where some programs would cause the simplifier to emit an empty case, resulting in runtime crashes. Fix due to the industrious Simon Peyton Jones.
  • #10527, where the simplifier would expend a great deal of effort simplifying arguments which were never demanded by the callee.
  • #10414, where a subtle point of the runtime system's black-holing mechanism resulting in hangs on a carefully constructed testcase.
  • #10236, where incorrect DWARF annotations would be generated, resulting in incorrect backtraces. Fixed by Peter Wortmann
  • #7450, where cached free variable information was being unnecessarily dropped by the specialiser, resulting in non-linear compile times for some programs.

See the status page for a complete listing of issues fixed in this release.

In the coming days we will being working with FP Complete to test the pre-release against Stackage. While Hackage tends to be too large to build in bulk, the selection of packages represented in Stackage is feasible to build and is likely to catch potential regressions. Hopefully this sort of large-scale validation will become common-place for future releases.

If all goes well, 7.10.2 will mark the end of the 7.10 series. However, there is always the small possibility that a major regression will be found. In this case we will cut a 7.10.3 release which will include a few patches which didn't make it into 7.10.2.

Other matters

It has been suggested in #10601 that GHC builds should ship with DWARF symbols for the base libraries and runtime system. While we all want to see this new capability in users' hands, 7.10.2 will, like 7.10.1, not be shipping with debug symbols. GHC HQ will be discussing the feasibility of including debug symbols with 7.12 in the future. In the meantime, we will be adding options to build.mk to make it easier for users to build their own compilers with debug-enabled libraries.

In this week's GHC meeting the effort to port GHC's build system to the Shake? build system briefly came up. Despite the volume of updates on the Wiki Simon reports that the project is still moving along. The current state of the Shake-based build system can be found on Github.

While debugging #7540 it became clear that there may be trouble lurking in the profiler. Namely when profiling GHC itself lintAnnots is showing up strongly where it logically should not. This was previously addressed in #10007, which was closed after a patch by Simon Marlow was merged. As it appears that this did not fully resolve the issue I'll be looking further into this.

~ Ben

  • Posted: 2015-07-06 16:25 (Updated: 2015-07-07 09:35)
  • Author: bgamari
  • Categories: (none)
  • Comments (0)

GHC Weekly News - 2015/06/10

Hi *,

Welcome for the latest entry in the GHC Weekly News. The past week, GHC HQ met up for a quick catch up on 7.10.2 (which you'll want to read up on, see below), and some other technical nonsense about some work we've been doing. As a result the current weekly notes have been slow - the current priority is the next release though, which leads us to...

7.10.2 status

7.10.2 is going to be out soon - our current plan is to have a release candidate on the weekend of Saturday the 13th, and the final release the next week. That means if you want something fixed, you'd better hollar very soon, or we're just not going to get to it!

If you're wondering what's currently been fixed/scheduled, the status page shows the current set of tickets we've fixed and plan on fixing.

List chatter

  • Jan Stolarek asks: in some cases, GHC will generate default instances or values, but that source code has no textual information location (for example, consider an instance clause without the where) - what do people think about fixing this, and are there anythings to look out for? https://mail.haskell.org/pipermail/ghc-devs/2015-June/009202.html
  • David Luposchainsky has opened a new thread - about moving fail out of Monad and into its own typeclass, MonadFail. This change is a request that's very long in the tooth (older than the AMP or FTP changes by a landslide), but David's proposal has a clearly set out goal to tackle compatibility, warnings, and implementation. https://mail.haskell.org/pipermail/ghc-devs/2015-June/009186.html

Noteworthy commits

Closed tickets

#10460, #7672, #9252, #9506, #10294, #5316, #10408, #10386, #9960, #10145, #9259, #10386, #9507, #8723, #10442, #5014, #4215, #10443, #8244, #10499, #10500, #10428, #10488, #10489, #10406, #10501, #10441, #10406, #10502, #9101, #9663, and #9945.

GHC Weekly News - 2015/06/03

Hi *,

It's that time once again - to get some info on what's happening in the world of GHC! It's been a quiet few weeks as a UK Holiday punted one of GHC HQ's meetings off, and this week we were only partially there.

The main point of discussion was 7.10.2, and continuing work on compiler performance. The good news is, the past few weeks have seen good work on both these fronts!

7.10.2 status

7.10.2 is swimming along very nicely - the status page shows the current set of tickets we've fixed and plan on fixing.

Not much has changed from last time, except we've fixed even more bugs! We're currently sitting at about 85 bugs fixed, some of them pretty important - code generation bugs, compiler performance fixes, some RTS and event manager work. Your author is actually quite happy with what GHC 7.10.2 looks like, at this rate.

List chatter

  • Facundo Dominguez asks: sometimes we want to create a static pointer in a function with a local definition, how can we do that? The current problem is the desugarer gets in the way and current approaches are currently rejected, but Facundo has some ideas/questions about a fix. https://mail.haskell.org/pipermail/ghc-devs/2015-May/009110.html

Noteworthy commits

Closed tickets

#10407, #10408, #10177, #10359, #10403, #10248, #9579, #10415, #10419, #10427, #10429, #10397, #10422, #10335, #10366, #10110, #10397, #10349, #10244, #8555, #8799, #9131, #10396, #10354, #10278, #9899, #3533, #9950, #10092, #9950, #10430, #9682, #9584, #10446, #10410, #10298, #10449, #10399, #7695, #10261, #8292, #10360, #10126, #10317, #10101, #10322, #10313, #10471, #10473, #7170, #10473, #10423, #10466, #8695, #10461, #10052, #10370, #10425, #10452, #10474,

GHC Weekly News - 2015/05/11

Hi *,

It's been a few weeks since the last news bulletin - this is the result of mostly quietness on behalf of the list and developers, and some sickness on behalf of your editor for several days there. But now there's actually some things to write here!

The past few weeks, GHC HQ has been having some quiet meetings mostly about bugfixes for a 7.10.2 release - as well as noodling about compiler performance. Austin has begun compiling his preliminary notes on the wiki, under the CompilerPerformance page, where we'll be trying to keep track of the ongoing performance story. Hopefully, GHC 7.12.1 will boast a bit better performance numbers.

There are a lot of users who are interested in this particular pain point, so please file tickets and CC yourself on bugs (like #10370), or feel free to help out!

7.10.2 status

There's been a bit of chatter about the lists about something on many peoples mind: the release of GHC 7.10.2. Most prominently, Mark Lentczner popped in to ask when the next GHC release will happen - in particular, he'd like to make a Haskell Platform release in lockstep with it (see below for a link to Mark's email).

Until recently, the actual desire for 7.10.2 wasn't totally clear, and at this point, GHC HQ hasn't firmly committed to the 7.10.2 release date. But if milestone:7.10.2 is any indicator, we've already closed over three dozen bugs, several of them high priority - and they keep coming in. So it seems likely people will want these fixes in their hands relatively soon.

Just remember: if you need a fix for 7.10.2, or have a bug you need us to look at, please email the ghc-devs list, file a ticket, and get our attention! Just be sure to set the milestone to 7.10.2.

List chatter

  • Niklas Hambüchen announced that he's backported the recent lightweight stack-trace support in GHC HEAD to GHC 7.10 and GHC 7.8 - meaning that users of these stable release can have informative call stack traces, even without profiling! FP Complete was interested in this feature, so they'd probably love to hear user input. https://mail.haskell.org/pipermail/ghc-devs/2015-April/008862.html
  • David Terei has written up a proposal on reconciling the existence of Roles with Safe Haskell, which caused us a lot of problems during the 7.8 release cycle. In particular, concerning the ability to break module abstractions and requiring programmers to safeguard abstractions through careful use of roles - and David's written a proposal to address that. https://mail.haskell.org/pipermail/ghc-devs/2015-April/008902.html
  • Mark Lentczner started a thread about the 7.10.2 release schedule - because this time, he wants to do a concurrent Haskell Platform release! The thread ended up with a good amount of discussion concerning if 7.10.2 is even needed - but at this rate, it looks like it will ship sometime soon. https://mail.haskell.org/pipermail/ghc-devs/2015-May/008904.html
  • Mateusz Kowalczyk posted to ghc-devs hoping to get some help with a tricky, long-standing issue: #4012, which concerns the determinism of GHC binaries. It turns out GHC isn't entirely deterministic when it calculates package IDs, meaning things get really bad when you mix prebuilt binary packages for systems. This in particular has become a real problem for the Nix package manager and users of Haskell applications. Mateusz asks if anyone would be willing to help look into it - and a lot of people would appreciate the help! https://mail.haskell.org/pipermail/ghc-devs/2015-May/008992.html

Noteworthy commits

Closed tickets

#10293, #10273, #10021, #10209, #10255, #10326, #9745, #10314, #8928, #8743, #10182, #10281, #10325, #10297, #10292, #10304, #10260, #9204, #10121, #10329, #9920, #10308, #10234, #10356, #10351, #10364, #9564, #10306, #10108, #9581, #10369, #9673, #10288, #10260, #10363, #10315, #10389, #9929, #10384, #10382, #10400, #10256, #10254, #10277, #10299, #10268, #10269, #10280, #10312, #10209, #10109, #10321, #10285, #9895, #10395, #10263, #10293, #10210, #10302, #10206, #9858, #10045, and #9840.

GHC Weekly News - 2015/04/14

Hi *,

It's been a few weeks since the last news bulletin - your editor apologizes about that. It's actually been a relatively slow few weeks here too, and busy-ness outside of GHC has attracted some of my attention. Despite that, GHC 7.10.1 was released, a new HP alpha is out, and things are moving along smoothly. Now that the release is done, things are quitely moving along in HEAD - with people committing code with reckless abandon, of course.

This week, GHC HQ met up, but it's been very light since the 7.10.1 release. Currently there isn't anything pushing us to do a 7.10.2 release at least for a few more weeks it looks like - but see below.

  • We puzzled a bit about the release status of 7.10.2, and thought: it's only holding up people who are depending on it. So, who's depending on it, and what do they need fixed? See below for more.
  • We also talked a bit about performance - it seems the compiler has been getting much slower over time since the 7.8.x release series, and it's time to reign it in. Austin will be spending his week investigating a bit of this, and the causes.

7.10.2 Status

So, you may be wondering when the 7.10.2 release is. The trick is it happens when you tell us it should happen!

So far, taking a look at milestone:7.10.2, we've fixed about half the bugs we currently have marked down to fix. But we're probably going to punt some of those - and we're not sure all the ones that are there should be.

So this is a call: If you need something to be fixed during 7.10.2, please file a ticket, set the milestone, and alert us. The sooner the better, because it'll inform us as to when we should release. Emailing ghc-devs@haskell.org is also a sure-fire way to get our attention.

And remember: you can always find out the latest about the next release at the Status page (in this case, for 7.10.2) - https://ghc.haskell.org/trac/ghc/wiki/Status/GHC-7.10.2

Call for help: DocBook to AsciiDoc

The GHC team needs some help. A few months ago, we put out a poll to convert our DocBook-based manual to AsciiDoc.

The poll had a mostly lukewarm reception, with the idea that it will A) make life easier for people who frequently modify the users guide, and B) make life easier for people who add things irregularly, as a lower barrier to entry.

It looks like we still want to do this - but alas, many of us don't have time!

So, we're asking the public: Is anyone willing to step up and help here? For example, it may be possible to get a long ways with just pandoc, but we need someone to finish it - and in return, we'll help along the way!

List chatter

  • A GHC user, Dave, asked the list about some questions with Cross Compilation, as he's attempting to get GHC to work natively inside the Open Embedded build environment. Unfortunately, things haven't been going well so far, and any input from enterprising hackers is appreciated: https://mail.haskell.org/pipermail/ghc-devs/2015-April/008774.html
  • Dan Aloni has started a discussion about improving GHC's error messages, spurred by a popular blog post he wrote and posted on Reddit about some Happy/GHC improvements he's made. This is a difficult area (error messages in general are hard) to work on, so thanks to Dan for helping! https://mail.haskell.org/pipermail/ghc-devs/2015-April/008778.html
  • Tamar Christina started a thread about replacing ghc-split, an old Perl script inside GHC, but he wanted to know: what do we do about a regex replacement? Mikhail Glushenkov spoke up about a similar decision the LLVM developers used: to use the OpenBSD regex implementation. https://mail.haskell.org/pipermail/ghc-devs/2015-April/008785.html

Noteworthy commits

Closed tickets

#10222, #10219, #8057, #10226, #10220, #9723, #10230, #10208, #10236, #10213, #10231, #10240, #10243, #10237, #10224, #8811, #10197, #10252, #9958, #10253, #8248, #10207, #10214, #9964, #10194, #10251, #10188, #10257, #10247, #10247, #9160, #10259, #9965, #10265, #10264, #10286, #10282, #10290, #10291, #10300, #9929, #8276, #10218, #10148, #10232, #10274, #10275, #10195, and #10233.

GHC Weekly News - 2015/03/24

Hi *,

It's time for the GHC weekly news. We've had an absence of the last one, mostly due to a lot of hustle to try and get 7.10 out the door (more on that shortly throughout this post). But now we're back and everything seems to be taken care of.

This week, in the wake of the GHC 7.10 release (which is occuring EOD, hopefully), GHC HQ met up for a brief chat and caught up:

  • This week GHC HQ met for only a very short time to discuss the pending release - it looks like all the blocking bugs have been fixed, and we've got everything triaged appropriately. You'll hopefully see the 7.10 announcement shortly after reading this.

We've also had small amounts of list activity (the past 6-8 weeks have been very, very quiet it seems):

  • Yitzchak Gale revived a thread he started a while back, which puttered out: bootstrapping GHC 7.8 with GHC 7.10. The long and short of it is, it should just about work - although we still haven't committed to this policy, it looks like Yitz and some others are quite adamant about it. https://mail.haskell.org/pipermail/ghc-devs/2015-March/008531.html

Some noteworthy commits that went into ghc.git in the past two weeks include:

Closed tickets this past week include: #9122, #10099, #10081, #9886, #9722, #9619, #9920, #9691, #8976, #9873, #9541, #9619, #9799, #9823, #10156, #1820, #6079, #9056, #9963, #10164, #10138, #10166, #10115, #9921, #9873, #9956, #9609, #7191, #10165, #10011, #8379, #10177, #9261, #10176, #10151, #9839, #8078, #8727, #9849, #10146, #9194, #10158, #7788, #9554, #8550, #10079, #10139, #10180, #10181, #10170, #10186, #10038, #10164, and #8976.

GHC Weekly News - 2015/03/10

Hi *,

It's that time again! Today GHC HQ met on a Tuesday to avoid some scheduling conflicts, and that means it's time to send some news to people.

Just a quick reminder from last week: we're hoping to make our third GHC 7.10.1 release candidate on Friday, March 13th, with the final release on Friday, March 27th.

  • Today, GHC HQ mostly talked about 7.10 bugs; HEAD is steaming along as usual with no impediments, but we've got several critical bugs we plan on landing fixes for this week; see milestone:7.10.1 for more.

But we've also had a little more list activity this week than we did before:

  • Karel Gardas wrote to the list about how to better parallelize the GHC build. For background, he's using a UltraSPARC T1-CPU/T2000 server with 32 hardware threads over 8 cores, where parallelism is a bigger win than raw single-threaded performance. But despite that, the performance is of the parallel GHC build is relatively abysmal - and Karel wants help brainstorming ideas to fix it. https://mail.haskell.org/pipermail/ghc-devs/2015-March/008474.html

Some noteworthy commits that went into ghc.git in the past week include:

Closed tickets the past week include: #7854, #10118, #10119, #3321, #10132, #9987, #10126, #9707, #10142, #10147, #10113, #9524, #10058, #10100, #2991, #10140, and #9122.