wiki:Building/GettingTheSources

Getting the GHC Sources

There are two ways to get sources to GHC: download a source distribution, or get the sources directly from our repository using git.

Source distributions

A source distribution is a file like ghc-7.8.3-src.tar.xz, which contains a complete snapshot of the source tree for a particular version of GHC. Source distributions for all versions of GHC are available from the download page.

Source distributions are easier to build, because we also include the output from running certain external tools like Happy, so you don't need to install these tools (see below for more).

Using git: preliminary setup

Please make sure that you have the correct name and email address set for Git that you want your commits to be recorded as. Make sure you use the same name and email on all your machines so we can easily track a single author. This can be done in Git by running:

$ git config --global user.name "Firstname Lastname"
$ git config --global user.email "[email protected]"

Then, if you are on Windows, ensure that git handles line-endings sanely by running:

$ git config --global core.autocrlf false

Quickly getting the GHC source repository

The first thing to do is install git. You'll also need some tools - see Building/Preparation for details.

Note: The following directions are valid for cloning GHC 7.9 or newer. For cloning GHC 7.8 or earlier, see below.

On Unix-like systems (Linux, OS X), a complete GHC source tree can be obtained (located in ghc) by saying:

$ git clone --recursive git://git.haskell.org/ghc.git

On Windows, you need to additionally fetch an extra repository containing some build tools. After running clone:

$ cd ghc/
$ git clone git://git.haskell.org/ghc-tarballs.git

A source tree consists of more than one repository: at the top level there is the main GHC repository, and certain subdirectories contain separate git repositories (for a list see Repositories).

And then read Git Working Conventions for instructions on how to use Git with GHC development.

  • git.haskell.org is reachable via IPv6 as well as IPv4.
  • If you're behind a firewall blocking port 9418 (or git clone git://... fails for some other reason), try replacing git:// by http:// or https:// in the instructions above.

Making a local clone

TODO investigate&document --reference-based git clone which may work better with submodules and doesn't require resetting the origin Repo back to git.haskell.org

You can make a local clone of a GHC tree with

 $ git clone ~/ghc ~/ghc-branch

where ~/ghc is the repository you want to branch and ~/ghc-branch is where you want to put the branch. Then use sync-all (see Building/SyncAll) to clone the rest of the repositories. Note that the origin for the local clone will point back to the repository that it was cloned from - if you want origin to point back to the main GHC repo then do this:

  $ ./sync-all -r git://git.haskell.org remote set-url origin

Making a new workdir

I (ezyang) use Git workdirs to manage all of my GHC checkouts. It is quite nice: there is logically only one local repository, and just many checkouts of it, so it's very easy to share patches between various checkouts and your branches are all in one centralized place. However, sharing all of your submodules too takes a bit of work to setup, since Git doesn't natively support this workflow.

Here's what I do:

  1. Start by making some pristine GHC checkout (call it ghc-pristine) which is an ordinary Git clone of the main GHC repository.
  1. Remove all of the submodules that Git recursively created. This is because they are in the wrong format. You can do it with this command: for i in `git submodule status | cut -d' ' -f3`; do rm -rf $i; done
  1. Re-checkout the submodules using a normal git clone, rather than the submodule tool. This can be done with this command: for i in `git submodule status | cut -d' ' -f2`; do git clone git://git.haskell.org/`echo "$i" | sed s/libraries/packages/ | sed s/utils\\///`.git $i; done (On OS X you might have to escape the backslash two more times)
  1. Finish off the configuration by running git submodule init and git submodule update

Now, to create a new workdir, run git-new-workdir ghc-pristine ghc-newdir, and then inside ghc-newdir, run for i in `git submodule status | cut -d' ' -f2`; do rmdir $i; git-new-workdir ../ghc-pristine/$i $i; done to recursively make workdirs of all the submodules.

ToDo: Someone should scriptify this.

Getting a branch (GHC 7.9 or later)

The above instructions will get the HEAD, the main trunk of GHC development. There is also a branch for each stable release line, as well as branches for development of major new features. The active branches are listed on ActiveBranches.

To get a branch, you need to get from a repo that contains the branch; in particular, local clones of the central repo will normally not include the branches that are in the central repo.

You can clone a specific branch via:

  $ git clone -b <branchname> --recursive git://git.haskell.org/ghc.git ghc-<branchname>

and switch between branches on an existing clone by

  $ git checkout <other-branchname>
  $ git submodule update --init

Note: The instructions above apply to branches that contain the commit [db19c665ec5055c2193b2174519866045aeff09a/ghc] which converted all sub-repos into submodules. To clone a branch prior to that commit, follow the instructions below instead. It is best not to attempt to cross that commit with git checkout; instead make a fresh clone of the desired branch directly.

Getting a branch (GHC 7.8 or earlier)

It is best to check out the desired branch directly in a fresh clone, as follows:

  $ git clone -b ghc-7.8 git://git.haskell.org/ghc.git ghc-7.8
  $ cd ghc-7.8
  $ ./sync-all get -b ghc-7.8

Getting a tag

The commands given below may soon be outdated, starting with GHC 7.10.1, you can simply clone a specific tag via:

git clone -b ghc-7.10.1-release --recursive git://git.haskell.org/ghc.git ghc-7.10.1

Each release is tagged in the git repository, making it possible to check out an old version of GHC by tag. To see all available tags, run git tag in the GHC repository.

To check out a specific version of GHC, run

  $ git clone http://git.haskell.org/ghc.git
  $ cd ghc
  $ git checkout <tag>
  $ ./sync-all get

For checking out a tag after you have already done ./sync-all [OPTIONS] get

  $ shopt -s extglob
  $ rm -rf !(.git)
  $ git checkout -f <tag>
  $ ./sync-all get

Getting a GHC repository from GitHub

The official mirror for GHC on GitHub is located at https://github.com/ghc/ghc.

New method

You don't need ./sync-all anymore and can use Git commands directly, if you configure the following Git url rewrites to account for the different naming scheme on GitHub (due to GitHub not supporting / in repository names) before cloning (those rules are persisted in ${HOME}/.gitconfig so you need to perform it only once):

git config --global url."git://github.com/ghc/packages-".insteadOf     git://github.com/ghc/packages/ 
git config --global url."http://github.com/ghc/packages-".insteadOf    http://github.com/ghc/packages/ 
git config --global url."https://github.com/ghc/packages-".insteadOf   https://github.com/ghc/packages/ 
git config --global url."ssh://[email protected]/ghc/packages-".insteadOf ssh://[email protected]/ghc/packages/ 
git config --global url."[email protected]:/ghc/packages-".insteadOf      [email protected]:/ghc/packages/ 

and then simply proceed by

git clone --recursive git://github.com/ghc/ghc

Old method

  $ git clone git://github.com/ghc/ghc
  $ cd ghc
  $ ./sync-all get

If you want to clone your own fork instead, add an argument to sync-all to tell it where it can find the other repositories it needs.

  $ git clone <your preferred github.com GHC fork URL> ghc
  $ cd ghc
  $ ./sync-all -r git://github.com/ghc get

Tracking the full repository state (pre-March 2014, e.g., the 7.8.* branches)

Since db19c665e/ghc, all used repositories are submodules, so this is only relevant to track older states, e.g., for GHC 7.8.*. The full state of a GHC repository includes the current state of the repositories for all of the GHC boot libraries that are used to to build GHC (list of boot libraries). For GHC 7.8.* the repositories for these libraries are fetched and updated by the sync-all script. To record the full repository state (including boot libraries), git submodules can be used starting from March 2014, but for 7.8.* they are not fully usable, so fingerprints are provided instead in annotated & gpg-signed release tags, such as http://git.haskell.org/ghc.git/tag/refs/tags/ghc-7.8.3-release. That way you're able to restore via the fingerprint for a given release, as in:

$ ./utils/fingerprint/fingerprint.py restore -f <(git show ghc-7.8.3-release | grep -F '|')

Here is how such fingerprints are created. The fingerprint.py script in utils/fingerprint/ can create a "fingerprint" to uniquely identify a GHC repository state by recording the current commits of the GHC and boot library repositories. This fingerprint can be used later to restore the state of all repositories to the state captured by the fingerprint. To create a new fingerprint, run the create command in the top level ghc repo. The fingerprint can also be created from a Builder log that contains the appropriate output from the sync-all command by passing the log file to the create command with the -l flag.

$ ./utils/fingerprint/fingerprint.py create 
$ ./utils/fingerprint/fingerprint.py create -l builder.log 

This command will create a new fingerprint, which is just pairs of repositories and commits. To restore a fingerprint use the restore command and pass either a fingerprint file with the -f flag or a builder log file with the -l flag.

$ ./utils/fingerprint/fingerprint.py restore -f 2011-05-23.fp 
$ ./utils/fingerprint/fingerprint.py restore -l builder.log 

This command will read the fingerprint and perform a checkout of the appropriate commit for each repository found in the fingerprint. By default, the restore command will create a new branch in the top level ghc repository and add an entry to git config that sets the new branch's remote to origin. The config options are added so that future sync-all commands will work as expected. Passing the -n flag will cause the fingerprint script not to create a new branch. To "unrestore" a fingerprint, simply use sync-all to checkout the master branch in each repository

$ ./sync-all checkout master 

To make the best use of fingerprinting, you need to collect fingerprints for the states you may wish to restore. To ease the automatic collection of fingerprints, the fingerprint.py script allows a -d DIR option that will output a fingerprint to the directory DIR with the current time stamp as a file name. The -g DIR option tells the script to run in the ghc repository pointed to by DIR. These options can be useful for collecting fingerprints as a cron job or on a post-commit hook. To restore the repository state as of a particular date, you can grab a fingerprint emitted by one of the nightly builds. The nightly build emails are sent to the ghc-builds mailing list, and have subjects like "[nightly] 06-Oct-2012 build of HEAD on x86_64-unknown-linux (cam-04-unx)". Copy and paste the fingerprint section, which looks something like this:

Respository hashes: 
.|4a138b708463a99a1087ce2d8a70239de3aa04e4 
ghc-tarballs|18e0c37f8023abf469af991e2fc2d3b024319c27 
libraries/Cabal|31c3f404dceb221d7281467a66bf68cc166fc2d9 
libraries/Win32|21335a30161c099da79ae9619c9782e5e32e4644 
libraries/array|e4b800af700cd7a5d42286086f17713ec98698a0 
libraries/base|219ea0e539aca217fe1819baf02d4cdd1d670f51 
... and so on 

Put it in a file, and run ./utils/fingerprint.py restore -f <file> to restore your repository tree to that state.

Last modified 5 months ago Last modified on Dec 10, 2014 12:22:35 AM