wiki:BuildBot

Version 5 (modified by simonmar, 7 years ago) (diff)

--

Setting up a nightly build

The GHC buildbot builds GHC on various platforms in various different ways each night, runs the test suite and performance benchmarks, and mails the results to the cvs-ghc@haskell.org mailing list. We're always keen to add more build slaves to the setup, especially if you have a platform that doesn't already have a build slave, so if you'd like to join the fun, please let us know at cvs-ghc@…. If a platform is represented in the nightly builds, it's more likely we'll be able to identify and fix problems specific to that platform quickly. To see the current status of the builds:

http://darcs.haskell.org:8010/

To create a new build slave

First you, as a buildbot client, need to agree a buildbot username (myUser) and password (myPass) with the buildbot admins (just pick a username and password and send it to igloo@earth.li). You'll also need to decide:

  • when the build(s) should happen
  • HEAD or branch builds
  • full build (up to stage 3, with extra-libs, full testsuite, and 5 nofib runs) or a fast build (stage 2, no extra-libs, fast testsuite, no nofib runs), or something in-between

Finally, if there is anything special that needs to be done for the client (e.g. if gcc is in an unusual place) then you'll need to let the admins know.

Then you'll need to install buildbot and its dependencies on the machine that will be doing the nightly build; see the BuildBot website for details. NB. if you're on Windows, you'll need to install BuildBot under Cygwin using the Cygwin Python; there are various problems getting the GHC build to work via BuildBot using the native Win32 Python, so we've given up on that route for now.

Now create and enter the directory you want the buildbot client to work in

$ mkdir /buildbot/ghc
$ cd /buildbot/ghc

and tell buildbot to set up a slave there

$ buildbot create-slave . darcs.haskell.org:9989 myUser myPass

This will print a few lines asking you to fill in info/admin and info/host. In the latter file, please include information on what operating system and architecture the machine is running.

It also created Makefile.sample; we recommend renaming this to Makefile. You can now start the buildbot client with make start and stop it with make stop.

You can watch what your slave is doing by looking at the twistd.log file in the directory in which you're running your slave.

Automating startup: Unix

The easiest way to make the client start up automatically is to use cron. Type crontab -e, and add this line to your crontab file:

@reboot cd /buildbot/ghc && make start

Remember to change /buildbot/ghc to your buildbot directory.

Cron will run the command in a minimal environment: it won't execute your normal shell startup files, so you won't have your usual PATH settings, for example. To get the right PATH and other environment variables, we suggest adding them to the make start rule in the Makefile. FOr example, my setup looks something like this:

start:
	PATH=/usr/bin:/bin:/home/simonmar/bin \
	http_proxy=http://157.58.63.38:80 \
	twistd --no_save -y buildbot.tac

It might be a good idea to have the buildbot restarted once a day before your build is due to start, just in case it has died for any reason. I have another line in my crontab that looks like this:

0 17 * * * cd /64playpen/buildbot && (make stop; make start)

To restart the client at 17.00, before the builds start at 18.00.

It's a good idea to test that running the client via a cron job actually works, so test it: setup a temporary cron job to start the client in a couple of minutes time, check that the client is up and running, and maybe force a build via the status page to check that the build environment is working.

Automating startup: Windows

I did it the following way. Create a script in <buildbotdir>/restart.sh:

PATH=/bin:/usr/bin
cd <buildbotdir>
make stop
make start

(don't forget to create the script as a Unix text file, not a DOS text file, otherwise strange things will probably happen, they did to me anyway).

Create a new "Scheduled Task" via Control Panel->Scheduled Tasks. The command you want to run is

c:\cygwin\bin\sh <buildbotdir>/restart.sh

Schedule the task to run (a) at startup and possibly also (b) once a day, before your build is due to start. You can add multiple schedulers for a task by checking the box at the bottom of the "Schedule" page of the scheduled task settings.

Admin steps

(for the admins only...)

Pull the buildbot master configuration:

$ darcs get buildbot@darcs.haskell.org:/home/buildbot/master
$ cd master

Edit master.cfg. Add new entries to slaves, schedulers, and builders as necessary. Record and push the changes. Then restart the build master:

$ ssh buildbot@darcs.haskell.org "cd master; make reconfig"

If there is anything unusual about the machine the build is being run on, e.g. the path to gcc is different, then you will need to add a field for the unusual thing to GhcDefaultConfig and alter the build steps to make use of it. Then make a special factory for the build client you are adding with this field changed as appropriate.

Did it work?

Once the master is reconfiged and the client is started, the client should become visible on http://darcs.haskell.org:8010/

At present there is no way to force an immediate test build.