Changes between Version 15 and Version 16 of Design/BuildSystem


Ignore:
Timestamp:
Nov 21, 2008 10:50:40 AM (7 years ago)
Author:
simonpj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Design/BuildSystem

    v15 v16  
    111111   package metadata or the code that processes it, only the build rules.
    112112
    113 == Detailed plan ==
    114 
    115  * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
    116 
    117  * The version of Cabal used by ghc-cabal.hs does not need to be an
    118    up-to-the-minute bleeding-edge version. It should be stable and vary
    119    slowly. We suck a new version of Cabal into the GHC build system
    120    manually, rather than mirroring the Cabal HEAD.
    121 
    122  * Rather than installing things in-place all over the build tree, we
    123    will have a single inplace directory at the root of the tree. The
    124    structure inside this directory will match that of the normal install,
    125    which will simplify various things. There are two slight wrinkles:
    126    * The tree will not be complete; for example, the libraries will be
    127      registered in-place in their dist directories
    128    * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123]
    129    Tools like genprimopcode, genapply etc. will probably also go into
    130    inplace/bin, in order to make the makefiles more consistent.
    131 
    132  * The build order looks something like:
     113== Avoiding recursive makefiles ==
     114
     115Currently, with recursive make, this means we jump around between
     116Makefiles a lot, which isn't good for parallelism in the build.  For example, the current build order looks like this:
    133117     * With bootstrapping compiler:
    134118       * Build libraries/{filepath,Cabal}
     
    148132       * libraries/*
    149133       * compiler
    150    Currently, with recursive make, this means we jump around between
    151    Makefiles a lot, which isn't good for parallelism in the build.
    152    Instead, we want to move all the logic and dependencies into the root
    153    Makefile (or files that get included into it) so that make sees all of
    154    it together.
    155 
    156    One concern is that make may take a long time thinking if it can see
    157    the rules for the whole system, even when only asked to build a single
    158    file. We will have to see how well it performs in practice.
    159 
    160  * But we still want "make" to work in subdirectories, so for example the
    161    Makefile (actually GNUmakefile, to avoid colliding with Makefile in
    162    libraries like Cabal) in libraries/base might look like
     134
     135Instead, following [http://miller.emu.id.au/pmiller/books/rmch/?ref=DDiyet.Com Recursive make considered harmful], we want to move all the logic and dependencies into the root
     136Makefile (or files that get included into it) so that make sees all of
     137it together.  Advantages:
     138
     139   * We get to specify dependencies between different parts of the tree much
     140     more easily and precisely.  BIG WIN.  Have you noticed how often you
     141     need to `make distclean` in a GHC tree to make sure everything is up to
     142     date? (well I rarely do this, because I have the dependencies in my
     143     head and I can rebuild manually, but I imagine this isn't the case for
     144     most people!)
     145
     146   * We get more parallelism, more easily.  I'd argue this is a big win too,
     147     right now a validate only uses about 1.3 out of 2 cores.
     148
     149We don't lose modularity: different parts of the build system are still in
     150different files, it's just that make sees them all at once.  Right now
     151every make invocation already reads thousands of lines of boilerplate
     152Makefile code, and we'd be invoking make only once rather than many times.
     153
     154So we will probably have to worry about efficiency.  For example, it takes
     155tens of seconds on Windows for make to discover that compiler/ is up to
     156date.  We don't want that happening every time you rebuild some small part
     157of the tree, so we plan to cut a few dependencies on purpose.  But I'm
     158hoping it'll be necessary to do this in a few well-defined places only, and
     159only when building in subdirectories.  For example, If you say 'make' in
     160libraries/base, then we won't try to rebuild the stage1 compiler, we'll
     161just fail if it does't exist.
     162
     163
     164We still want "make" to work in subdirectories, so for example the
     165Makefile (actually GNUmakefile, to avoid colliding with Makefile in
     166libraries like Cabal) in libraries/base might look like
    163167
    164168{{{
     
    174178}}}
    175179
    176    (ghc.mk is discussed later). In actual fact, GNUmakefile will want to
    177    be more complicated, to handle "make way=v", "make way=p", "make doc",
    178    etc. Where possible, the make code will be "include"d in, rather than
    179    generated, so as to make it easier to deal with.
    180 
    181    We need the .NOTPARALLEL or if you say "make foo bar" then the two
    182    recursive make calls might both make "quux" (a dependency of foo and
    183    bar) at the same time. The main Makefile will be able to do work in
    184    parallel when building each of foo and bar, though. The common case,
    185    where you only specify 0 or 1 targets, doesn't lose any parallelism.
     180(ghc.mk is discussed later). In actual fact, GNUmakefile will want to
     181be more complicated, to handle "make way=v", "make way=p", "make doc",
     182etc. Where possible, the make code will be "include"d in, rather than
     183generated, so as to make it easier to deal with.
     184
     185We need the .NOTPARALLEL or if you say "make foo bar" then the two
     186recursive make calls might both make "quux" (a dependency of foo and
     187bar) at the same time. The main Makefile will be able to do work in
     188parallel when building each of foo and bar, though. The common case,
     189where you only specify 0 or 1 targets, doesn't lose any parallelism.
     190
     191
     192== Detailed plan ==
     193
     194 * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/
     195
     196 * The version of Cabal used by ghc-cabal.hs does not need to be an
     197   up-to-the-minute bleeding-edge version. It should be stable and vary
     198   slowly. We suck a new version of Cabal into the GHC build system
     199   manually, rather than mirroring the Cabal HEAD.
     200
     201 * Rather than installing things in-place all over the build tree, we
     202   will have a single inplace directory at the root of the tree. The
     203   structure inside this directory will match that of the normal install,
     204   which will simplify various things. There are two slight wrinkles:
     205   * The tree will not be complete; for example, the libraries will be
     206     registered in-place in their dist directories
     207   * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123]
     208   Tools like genprimopcode, genapply etc. will probably also go into
     209   inplace/bin, in order to make the makefiles more consistent.
     210
    186211
    187212 * In e.g. utils/ghc-pkg, the default target will be