wiki:SoC2007Hackage

Version 9 (modified by sboehme, 7 years ago) (diff)

--

Enhancing the HackageDB Web Interface

Sascha Böhme, Ross Paterson { Simon Marlow, BryanOS, Bjorn Bringert, Alex Jacobson }

(Original application to Google)

Automated building of packages and generation of Haddock documentation

The plan is for a daemon to build HackageDB packages and add the resulting build logs and Haddock documentation to the package pages. Since building Cabal packages may execute arbitrary code, it must be done in a protected environment (e.g. chroot). Currently, there are two tools for this purpose, a bash script to set up an appropriate chroot environment and a Haskell application to perform the generation of Haddock documentation.

The general idea is as follows. In the secure environment, an unprivileged user downloads a package (the work unit), inspects it dependencies from the package's Cabal file and downloads necessary dependency packages, if not already available on the system. After the dependencies have been built and their documentation was generated, the work unit is processed in the same way. During that, a build log is generated which is uploaded to the corresponding package's HackageDB website along with the package's Haddock documentation.

Other extensions of the HackageDB website

See HackageDB ToDo list, especially Package properties and Queries.

Progress

2007-08-02

The development continues. The autodoc tool was adjusted to the changes of Cabal-1.1.7 (configurations). Additionally, the incremental mode of autodoc was abolished. Now, every run of autodoc tries to build all available packages. This will immediately reflect the changes of different package dependency resolvings.

  • Todo list
    • add more needed libraries to the chroot environment

2007-07-28

My computer does not start anymore. Since the autodoc tool was run exclusively there, no build longs will be generated for now. It probably takes some time until I might continue.

2007-07-20

The autodoc tool can now be downloaded:

darcs get http://community.haskell.org/~sboehme/autodoc

The setup_chroot script will be available soon, too.

Out of the 316 packages on HackageDB (on 19th July 2007), the autodoc tool managed to build 104 HackageDB packages and detected build errors of 31 packages; 65 packages cannot be build in the current system (mostly depending on the setup of the chroot environment), and the other packages cannot be build due to missing dependencies. An overview and the results can be obtained from overview page.

  • Todo list
    • add more needed libraries to the chroot environment
    • solve problems caused by different versions of Cabal used in the chroot environment
  • Possible extensions
    • allow for different compilers and compiler versions

2007-07-13

The autodoc tool was completely redesigned. It now tries to build all HackageDB packages which have not been successfully built on the current system. Building a package includes configuring it, building it, running its test suit and generating Haddock and Hoogle documentation. Along with building, a build log is generated to be exposed on the package's web site on HackageDB. The autodoc tool is now less platform dependent as before because it does not rely anymore on a working chroot environment but can be run anywhere (with corresponding security problems due to execution of untrusted code).

  • Todo list
    • identify C libraries needed to build packages (e.g. SDL, X11, OpenGL, ...) and add them to what the setup_chroot script is already requiring
    • debug and document the autodoc tool and run it on all HackageDB packages
    • push the sources to the public darcs repository
  • Possible extensions
    • allow for different compilers and compiler versions

2007-07-06

The setup_chroot script, based on debootstrap, allows to create a basic environment to build Cabal packages. It consists of GHC 6.6, Cabal 1.1.7 (taken from the darcs repository) and Haddock 0.8, along with Alex and Happy and other necessary tools. It lacks several libraries needed to build all packages currently available at HackageDB.

The autodoc tool already fulfills the above the requirements (ignoring the packages whose dependencies cannot be met, i.e. SDL and alike). In a test run, 102 package documentations could be built. For now, the tool uses the approach of cabal_install, that is, it builds all dependency packages before processing the actual package. This can be sped up by storing already built packages in the global package repository. Additionally, the autodoc tool currently relies explicitly on a chroot environment, although this is not strictly needed.

  • Todo list
    • identify C libraries needed to build packages (e.g. SDL, X11, OpenGL, ...) and add them to what the setup_chroot script is already requiring
    • rewrite the autodoc tool to use a bottom-up approach for building packages, that is, successfully built packages are stored in the system and not rebuilt again afterwards
    • remove the dependency on a chroot environment from the autodoc tool, thus improve platform-independency
    • improve autodoc output, especially more readable build logs
  • Possible extensions
    • Hoogle output
    • run test suites of packages
    • allow for different compilers and compiler versions