Changes between Version 15 and Version 16 of Design/BuildSystem


Ignore:
Timestamp:
Nov 21, 2008 10:50:40 AM (5 years ago)
Author:
simonpj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Design/BuildSystem

    v15 v16  
    111111   package metadata or the code that processes it, only the build rules.  
    112112 
    113 == Detailed plan ==  
    114  
    115  * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/ 
    116  
    117  * The version of Cabal used by ghc-cabal.hs does not need to be an 
    118    up-to-the-minute bleeding-edge version. It should be stable and vary 
    119    slowly. We suck a new version of Cabal into the GHC build system 
    120    manually, rather than mirroring the Cabal HEAD.  
    121  
    122  * Rather than installing things in-place all over the build tree, we 
    123    will have a single inplace directory at the root of the tree. The 
    124    structure inside this directory will match that of the normal install, 
    125    which will simplify various things. There are two slight wrinkles: 
    126    * The tree will not be complete; for example, the libraries will be 
    127      registered in-place in their dist directories 
    128    * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123] 
    129    Tools like genprimopcode, genapply etc. will probably also go into 
    130    inplace/bin, in order to make the makefiles more consistent. 
    131  
    132  * The build order looks something like: 
     113== Avoiding recursive makefiles == 
     114 
     115Currently, with recursive make, this means we jump around between 
     116Makefiles a lot, which isn't good for parallelism in the build.  For example, the current build order looks like this: 
    133117     * With bootstrapping compiler: 
    134118       * Build libraries/{filepath,Cabal} 
     
    148132       * libraries/* 
    149133       * compiler 
    150    Currently, with recursive make, this means we jump around between 
    151    Makefiles a lot, which isn't good for parallelism in the build. 
    152    Instead, we want to move all the logic and dependencies into the root 
    153    Makefile (or files that get included into it) so that make sees all of 
    154    it together. 
    155  
    156    One concern is that make may take a long time thinking if it can see 
    157    the rules for the whole system, even when only asked to build a single 
    158    file. We will have to see how well it performs in practice. 
    159  
    160  * But we still want "make" to work in subdirectories, so for example the 
    161    Makefile (actually GNUmakefile, to avoid colliding with Makefile in 
    162    libraries like Cabal) in libraries/base might look like 
     134 
     135Instead, following [http://miller.emu.id.au/pmiller/books/rmch/?ref=DDiyet.Com Recursive make considered harmful], we want to move all the logic and dependencies into the root 
     136Makefile (or files that get included into it) so that make sees all of 
     137it together.  Advantages: 
     138 
     139   * We get to specify dependencies between different parts of the tree much 
     140     more easily and precisely.  BIG WIN.  Have you noticed how often you 
     141     need to `make distclean` in a GHC tree to make sure everything is up to 
     142     date? (well I rarely do this, because I have the dependencies in my 
     143     head and I can rebuild manually, but I imagine this isn't the case for 
     144     most people!) 
     145 
     146   * We get more parallelism, more easily.  I'd argue this is a big win too, 
     147     right now a validate only uses about 1.3 out of 2 cores. 
     148 
     149We don't lose modularity: different parts of the build system are still in 
     150different files, it's just that make sees them all at once.  Right now 
     151every make invocation already reads thousands of lines of boilerplate 
     152Makefile code, and we'd be invoking make only once rather than many times. 
     153 
     154So we will probably have to worry about efficiency.  For example, it takes 
     155tens of seconds on Windows for make to discover that compiler/ is up to 
     156date.  We don't want that happening every time you rebuild some small part 
     157of the tree, so we plan to cut a few dependencies on purpose.  But I'm 
     158hoping it'll be necessary to do this in a few well-defined places only, and 
     159only when building in subdirectories.  For example, If you say 'make' in 
     160libraries/base, then we won't try to rebuild the stage1 compiler, we'll 
     161just fail if it does't exist. 
     162 
     163 
     164We still want "make" to work in subdirectories, so for example the 
     165Makefile (actually GNUmakefile, to avoid colliding with Makefile in 
     166libraries like Cabal) in libraries/base might look like 
    163167 
    164168{{{ 
     
    174178}}} 
    175179 
    176    (ghc.mk is discussed later). In actual fact, GNUmakefile will want to 
    177    be more complicated, to handle "make way=v", "make way=p", "make doc", 
    178    etc. Where possible, the make code will be "include"d in, rather than 
    179    generated, so as to make it easier to deal with. 
    180  
    181    We need the .NOTPARALLEL or if you say "make foo bar" then the two 
    182    recursive make calls might both make "quux" (a dependency of foo and 
    183    bar) at the same time. The main Makefile will be able to do work in 
    184    parallel when building each of foo and bar, though. The common case, 
    185    where you only specify 0 or 1 targets, doesn't lose any parallelism. 
     180(ghc.mk is discussed later). In actual fact, GNUmakefile will want to 
     181be more complicated, to handle "make way=v", "make way=p", "make doc", 
     182etc. Where possible, the make code will be "include"d in, rather than 
     183generated, so as to make it easier to deal with. 
     184 
     185We need the .NOTPARALLEL or if you say "make foo bar" then the two 
     186recursive make calls might both make "quux" (a dependency of foo and 
     187bar) at the same time. The main Makefile will be able to do work in 
     188parallel when building each of foo and bar, though. The common case, 
     189where you only specify 0 or 1 targets, doesn't lose any parallelism. 
     190 
     191 
     192== Detailed plan ==  
     193 
     194 * Rename cabal-bin.hs to ghc-cabal.hs, and move it into utils/ghc-cabal/ 
     195 
     196 * The version of Cabal used by ghc-cabal.hs does not need to be an 
     197   up-to-the-minute bleeding-edge version. It should be stable and vary 
     198   slowly. We suck a new version of Cabal into the GHC build system 
     199   manually, rather than mirroring the Cabal HEAD.  
     200 
     201 * Rather than installing things in-place all over the build tree, we 
     202   will have a single inplace directory at the root of the tree. The 
     203   structure inside this directory will match that of the normal install, 
     204   which will simplify various things. There are two slight wrinkles: 
     205   * The tree will not be complete; for example, the libraries will be 
     206     registered in-place in their dist directories 
     207   * Rather than inplace/bin/ghc, we will have inplace/bin/ghc-stage[123] 
     208   Tools like genprimopcode, genapply etc. will probably also go into 
     209   inplace/bin, in order to make the makefiles more consistent. 
     210 
    186211 
    187212 * In e.g. utils/ghc-pkg, the default target will be