Opened 4 years ago

Last modified 5 weeks ago

#5987 new bug

Too many symbols in ghc package DLL

Reported by: igloo Owned by: Phyx-
Priority: normal Milestone: 8.2.1
Component: Compiler Version: 7.5
Keywords: Cc: bill@…, george.colpitts@…, tkn.akio@…, shelarcy@…, RyanGlScott
Operating System: Windows Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: #10352 Blocking: #5355
Related Tickets: Differential Rev(s):
Wiki Page:

Description

In HEAD, the libHSghc DLL currently contains about 75000 symbols, but DLLs can only have about 65000 symbols. This means that when you try to run anything that uses the libHSghc DLL, references get resolved incorrectly and the GCer segfaults.

Max ran into this before, and temporarily fixed it with:

commit dd391759500ab2b6abd23d502ade7ff8946c780f
Author: Max Bolingbroke <batterseapower@hotmail.com>
Date:   Wed Jul 6 18:01:02 2011 +0100

    Don't export the _info symbol for the data constructor worker bindings

but the number of symbols has since crept up.

Three possible solutions:

  • Split the ghc package up into ghc-base, ghc-codegen, ghc-nativegen, etc.
  • Export fewer things. We'd first have to do this at the Haskell level (i.e. stop exposing some modules in the .cabal file; this may upset GHC API users), and then thread this info through so that we can tell the linker which symbols we do/don't want it to expose. Unfortunately, re-exports at the Haskell level mean that it isn't as simple as just not exporting anything from modules that aren't exposed.
  • Automatically split the package into 2 DLLs. The tricky bit here is that we'd need dependencies to go only in one direction, i.e. if libHSghc1.dll uses symbols from libHSghc2.dll, then libHSghc2.dll doesn't use any symbols from libHSghc1.dll. Ideally Cabal would have a way to do this automatically, and the GHC build system would use the split that it calculates. However, i don't think Cabal has enough info to do this currently.

It would also be good if we can detect that building a DLL has gone wrong, and give an error.

Change History (60)

comment:1 Changed 4 years ago by batterseapower

I reported a bug to binutils over 5 months ago asking them to make "ld" error out when this happens, but there is no indication that anyone has even read the bug. I even provided a patch, so this is a bit disappointing :-(

The bug is at http://sourceware.org/bugzilla/show_bug.cgi?id=12969

comment:2 Changed 4 years ago by igloo

Simon Marlow says:

There is one problem that I know of with breaking up the compiler into separate packages: GHC will do too much recompilation when things change, which will make development somewhat painful. I discovered this when making changes in hoopl: a small change in hoopl causes recompilation of nearly every module in GHC. The reason is the way that GHC records dependencies across packages: rather than the fine-grained dependencies we get within a package, a dependency on a package module records only the ABI hash of the whole module.

comment:3 Changed 4 years ago by simonpj

I mis-read Max's comment. I thought he was saying that we (the GHC crowd) have been ignoring the ticket. But actually he's saying that he offered a patch to the binutils folk, but they have ignored it so far. Does anyone know anyone who who might help un-glue the binutils process?

Simon

comment:4 Changed 4 years ago by igloo

Note that the binutils patch doesn't actually fix the problem: It just means that the GHC build will fail, rather than silently making a bogus DLL.

comment:5 Changed 4 years ago by igloo

  • Owner set to igloo

comment:6 Changed 4 years ago by igloo@…

commit 6c29fad6a1ab20e8cfed5ad4af406a821ee74291

Author: Ian Lynagh <igloo@earth.li>
Date:   Fri Jun 29 17:47:32 2012 +0100

    Don't build the ghc DLL on Windows; works around trac #5987
    
    We also don't build DLLs for the dph-lifted-* packages as they depend
    on ghc.

 compiler/ghc.mk        |    8 ++++++++
 ghc.mk                 |   12 ++++++++++++
 rules/build-package.mk |    2 +-
 3 files changed, 21 insertions(+), 1 deletions(-)

comment:7 Changed 4 years ago by igloo@…

commit 5ddc841a547d2d37428280b04c071a1d4b83e10d

Author: Ian Lynagh <igloo@earth.li>
Date:   Fri Jun 29 01:33:17 2012 +0100

    On Windows, detect if DLLs have too many symbols; trac #5987
    
    The test isn't very pretty; it involves trawling through the
    objdump -p output. I couldn't find an easier way, unfortuantely.

 configure.ac               |    7 +++++++
 mk/config.mk.in            |    1 +
 rules/build-package-way.mk |    3 +++
 3 files changed, 11 insertions(+), 0 deletions(-)

comment:8 Changed 4 years ago by igloo

  • Milestone changed from 7.6.1 to 7.8.1

This isn't fixed, but (a) we now detect it if it happens, and (b) we no longer build a broken ghc package DLL (because we don't build the DLL at all).

So we still need a proper fix, but this will do for now.

comment:9 Changed 4 years ago by igloo

  • Owner igloo deleted

comment:10 Changed 4 years ago by marlowsd@…

commit 99fd2469fba1a38b2a65b4694f337d92e559df01

Merge: d260d91... 0d19922...
Author: Simon Marlow <marlowsd@gmail.com>
Date:   Wed Jul 4 10:34:48 2012 +0100

    Merge remote-tracking branch 'origin/master' into newcg
    
    * origin/master: (756 commits)
      don't crash if argv[0] == NULL (#7037)
      -package P was loading all versions of P in GHCi (#7030)
      Add a Note, copying text from #2437
      improve the --help docs a bit (#7008)
      Copy Data.HashTable's hashString into our Util module
      Build fix
      Build fixes
      Parse error: suggest brackets and indentation.
      Don't build the ghc DLL on Windows; works around trac #5987
      On Windows, detect if DLLs have too many symbols; trac #5987
      Add some more Integer rules; fixes #6111
      Fix PA dfun construction with silent superclass args
      Add silent superclass parameters to the vectoriser
      Add silent superclass parameters (again)
      Mention Generic1 in the user's guide
      Make the GHC API a little more powerful.
      tweak llvm version warning message
      New version of the patch for #5461.
      Fix Word64ToInteger conversion rule.
      Implemented feature request on reconfigurable pretty-printing in GHCi (#5461)
      ...
    
    Conflicts:
    	compiler/basicTypes/UniqSupply.lhs
    	compiler/cmm/CmmBuildInfoTables.hs
    	compiler/cmm/CmmLint.hs
    	compiler/cmm/CmmOpt.hs
    	compiler/cmm/CmmPipeline.hs
    	compiler/cmm/CmmStackLayout.hs
    	compiler/cmm/MkGraph.hs
    	compiler/cmm/OldPprCmm.hs
    	compiler/codeGen/CodeGen.lhs
    	compiler/codeGen/StgCmm.hs
    	compiler/codeGen/StgCmmBind.hs
    	compiler/codeGen/StgCmmLayout.hs
    	compiler/codeGen/StgCmmUtils.hs
    	compiler/main/CodeOutput.lhs
    	compiler/main/HscMain.hs
    	compiler/nativeGen/AsmCodeGen.lhs
    	compiler/simplStg/SimplStg.lhs

 .authorspellings                                   |   29 -
 .darcs-boring                                      |  256 ---
 .gitignore                                         |    1 +
 README                                             |   11 +-
 aclocal.m4                                         |   96 +-
 bindisttest/Makefile                               |    4 +-
 bindisttest/ghc.mk                                 |    4 +-
 compiler/HsVersions.h                              |   13 +-
 compiler/basicTypes/BasicTypes.lhs                 |   44 +-
 compiler/basicTypes/DataCon.lhs                    |   33 +-
 compiler/basicTypes/Demand.lhs                     |    3 -
 compiler/basicTypes/Id.lhs                         |   21 +-
 compiler/basicTypes/IdInfo.lhs                     |   17 +-
 compiler/basicTypes/Literal.lhs                    |   56 +-
 compiler/basicTypes/MkId.lhs                       |   74 +-
 compiler/basicTypes/MkId.lhs-boot                  |    3 +
 compiler/basicTypes/Name.lhs                       |    5 +-
 compiler/basicTypes/NameEnv.lhs                    |   32 +-
 compiler/basicTypes/OccName.lhs                    |   13 +-
 compiler/basicTypes/RdrName.lhs                    |   46 +-
 compiler/basicTypes/SrcLoc.lhs                     |  279 ++--
 compiler/basicTypes/UniqSupply.lhs                 |    9 +-
 compiler/basicTypes/Unique.lhs                     |   37 +-
 compiler/basicTypes/Var.lhs                        |   40 +-
 compiler/cmm/Bitmap.hs                             |    2 +-
 compiler/cmm/CLabel.hs                             |  117 +-
 compiler/cmm/Cmm.hs                                |   10 +-
 compiler/cmm/CmmBuildInfoTables.hs                 |   38 +-
 compiler/cmm/CmmCvt.hs                             |    2 +-
 compiler/cmm/CmmExpr.hs                            |    4 +
 compiler/cmm/CmmInfo.hs                            |    4 +-
 compiler/cmm/CmmLint.hs                            |   71 +-
 compiler/cmm/CmmMachOp.hs                          |    9 +-
 compiler/cmm/CmmOpt.hs                             |   31 +-
 compiler/cmm/CmmParse.y                            |   10 +-
 compiler/cmm/CmmPipeline.hs                        |   22 +-
 compiler/cmm/CmmProcPoint.hs                       |    2 +-
 compiler/cmm/CmmStackLayout.hs                     |  591 +++++
 compiler/cmm/CmmType.hs                            |    6 +
 compiler/cmm/CmmUtils.hs                           |    7 +-
 compiler/cmm/OldCmm.hs                             |    9 +-
 compiler/cmm/OldCmmLint.hs                         |   70 +-
 compiler/cmm/OldCmmUtils.hs                        |    1 +
 compiler/cmm/OldPprCmm.hs                          |   99 +-
 compiler/cmm/OptimizationFuel.hs                   |    1 +
 compiler/cmm/PprC.hs                               |   76 +-
 compiler/cmm/PprCmm.hs                             |  109 +-
 compiler/cmm/PprCmmDecl.hs                         |   97 +-
 compiler/cmm/PprCmmExpr.hs                         |  117 +-
 compiler/codeGen/CgBindery.lhs                     |   27 +-
 compiler/codeGen/CgCase.lhs                        |    1 -
 compiler/codeGen/CgClosure.lhs                     |   29 +-
 compiler/codeGen/CgCon.lhs                         |   11 +-
 compiler/codeGen/CgExpr.lhs                        |    2 +-
 compiler/codeGen/CgForeignCall.hs                  |    8 +-
 compiler/codeGen/CgInfoTbls.hs                     |    5 +-
 compiler/codeGen/CgMonad.lhs                       |    1 +
 compiler/codeGen/CgParallel.hs                     |   69 +-
 compiler/codeGen/CgPrimOp.hs                       |  186 ++-
 compiler/codeGen/CgProf.hs                         |    3 +-
 compiler/codeGen/CgStackery.lhs                    |    3 +-
 compiler/codeGen/CgTailCall.lhs                    |    1 +
 compiler/codeGen/CgTicky.hs                        |   13 +-
 compiler/codeGen/CgUtils.hs                        |   13 +-
 compiler/codeGen/ClosureInfo.lhs                   |   50 +-
 compiler/codeGen/CodeGen.lhs                       |    3 +-
 compiler/codeGen/StgCmm.hs                         |    5 +-
 compiler/codeGen/StgCmmBind.hs                     |   47 +-
 compiler/codeGen/StgCmmClosure.hs                  |   68 +-
 compiler/codeGen/StgCmmCon.hs                      |    4 +-
 compiler/codeGen/StgCmmEnv.hs                      |    1 -
 compiler/codeGen/StgCmmExpr.hs                     |    2 +-
 compiler/codeGen/StgCmmForeign.hs                  |    7 +-
 compiler/codeGen/StgCmmGran.hs                     |   57 +-
 compiler/codeGen/StgCmmHeap.hs                     |   10 +-
 compiler/codeGen/StgCmmLayout.hs                   |   27 +-
 compiler/codeGen/StgCmmMonad.hs                    |   12 +-
 compiler/codeGen/StgCmmPrim.hs                     |   17 +-
 compiler/codeGen/StgCmmProf.hs                     |    3 +-
 compiler/codeGen/StgCmmTicky.hs                    |   19 +-
 compiler/codeGen/StgCmmUtils.hs                    |   14 +-
 compiler/coreSyn/CoreArity.lhs                     |   10 +-
 compiler/coreSyn/CoreFVs.lhs                       |   20 +-
 compiler/coreSyn/CoreLint.lhs                      |  531 ++---
 compiler/coreSyn/CorePrep.lhs                      |  481 ++--
 compiler/coreSyn/CoreSubst.lhs                     |    5 +-
 compiler/coreSyn/CoreSyn.lhs                       |   92 +-
 compiler/coreSyn/CoreTidy.lhs                      |    4 +-
 compiler/coreSyn/CoreUnfold.lhs                    |  165 +-
 compiler/coreSyn/CoreUtils.lhs                     |  275 ++-
 compiler/coreSyn/MkCore.lhs                        |   54 +-
 compiler/coreSyn/MkExternalCore.lhs                |  199 +-
 compiler/coreSyn/PprCore.lhs                       |  261 ++--
 compiler/coreSyn/TrieMap.lhs                       |  137 +-
 compiler/deSugar/Coverage.lhs                      |  510 +++--
 compiler/deSugar/Desugar.lhs                       |    9 +-
 compiler/deSugar/DsArrows.lhs                      |    4 +-
 compiler/deSugar/DsBinds.lhs                       |  208 ++-
 compiler/deSugar/DsCCall.lhs                       |   13 +-
 compiler/deSugar/DsExpr.lhs                        |   63 +-
 compiler/deSugar/DsForeign.lhs                     |  136 +-
 compiler/deSugar/DsListComp.lhs                    |   50 +-
 compiler/deSugar/DsMeta.hs                         | 1005 +++++----
 compiler/deSugar/DsMonad.lhs                       |   19 +-
 compiler/deSugar/DsUtils.lhs                       |    4 +-
 compiler/deSugar/Match.lhs                         |    2 +-
 compiler/deSugar/MatchCon.lhs                      |   17 +-
 compiler/ghc.cabal.in                              |   42 +-
 compiler/ghc.mk                                    |   28 +-
 compiler/ghci/ByteCodeAsm.lhs                      |  599 +++---
 compiler/ghci/ByteCodeGen.lhs                      |  276 ++-
 compiler/ghci/ByteCodeItbls.lhs                    |   31 +-
 compiler/ghci/ByteCodeLink.lhs                     |   14 +-
 compiler/ghci/Debugger.hs                          |   11 +-
 compiler/ghci/DebuggerUtils.hs                     |    4 +-
 compiler/ghci/LibFFI.hsc                           |    4 +-
 compiler/ghci/Linker.lhs                           |  705 +++---
 compiler/ghci/ObjLink.lhs                          |   41 +-
 compiler/ghci/RtClosureInspect.hs                  |  180 +-
 compiler/hsSyn/Convert.lhs                         |  353 ++--
 compiler/hsSyn/HsBinds.lhs                         |   44 +-
 compiler/hsSyn/HsDecls.lhs                         |  419 ++--
 compiler/hsSyn/HsExpr.lhs                          |   46 +-
 compiler/hsSyn/HsImpExp.lhs                        |   19 +-
 compiler/hsSyn/HsPat.lhs                           |   11 +-
 compiler/hsSyn/HsSyn.lhs                           |    3 +-
 compiler/hsSyn/HsTypes.lhs                         |  406 ++--
 compiler/hsSyn/HsUtils.lhs                         |   94 +-
 compiler/iface/BinIface.hs                         |  169 +-
 compiler/iface/BuildTyCl.lhs                       |   15 +-
 compiler/iface/FlagChecker.hs                      |   46 +-
 compiler/iface/IfaceEnv.lhs                        |   25 +-
 compiler/iface/IfaceSyn.lhs                        |   39 +-
 compiler/iface/IfaceType.lhs                       |  154 +-
 compiler/iface/LoadIface.lhs                       |   15 +-
 compiler/iface/MkIface.lhs                         |  254 ++-
 compiler/iface/TcIface.lhs                         |  198 ++-
 compiler/llvmGen/Llvm.hs                           |    1 -
 compiler/llvmGen/Llvm/PpLlvm.hs                    |  106 +-
 compiler/llvmGen/LlvmCodeGen.hs                    |   41 +-
 compiler/llvmGen/LlvmCodeGen/Base.hs               |   16 +-
 compiler/llvmGen/LlvmCodeGen/CodeGen.hs            |   57 +-
 compiler/llvmGen/LlvmCodeGen/Ppr.hs                |   13 +-
 compiler/main/Annotations.hs                       |  107 +
 compiler/main/Annotations.lhs                      |  108 -
 compiler/main/CmdLineParser.hs                     |    4 +-
 compiler/main/CodeOutput.lhs                       |   13 +-
 compiler/main/Constants.lhs                        |    4 +
 compiler/main/DriverMkDepend.hs                    |   10 +-
 compiler/main/DriverPipeline.hs                    |  106 +-
 compiler/main/DynFlags.hs                          |  412 +++-
 compiler/main/DynFlags.hs-boot                     |   13 +
 compiler/main/DynamicLoading.hs                    |   21 +-
 compiler/main/ErrUtils.lhs                         |  119 +-
 compiler/main/ErrUtils.lhs-boot                    |    1 +
 compiler/main/GHC.hs                               |  192 ++-
 compiler/main/GhcMake.hs                           | 1306 ++++++------
 compiler/main/HeaderInfo.hs                        |   78 +-
 compiler/main/HscMain.hs                           |  325 ++--
 compiler/main/HscStats.hs                          |  160 ++
 compiler/main/HscStats.lhs                         |   12 +-
 compiler/main/HscTypes.lhs                         |  105 +-
 compiler/main/InteractiveEval.hs                   |   99 +-
 compiler/main/PackageConfig.hs                     |   50 +-
 compiler/main/Packages.lhs                         |  437 ++--
 compiler/main/StaticFlagParser.hs                  |   38 +-
 compiler/main/StaticFlags.hs                       |   68 +-
 compiler/main/SysTools.lhs                         |   25 +-
 compiler/main/TidyPgm.lhs                          |  749 ++++---
 compiler/nativeGen/AsmCodeGen.lhs                  |  647 +++---
 compiler/nativeGen/PIC.hs                          |   93 +-
 compiler/nativeGen/PPC/CodeGen.hs                  |   33 +-
 compiler/nativeGen/PPC/Ppr.hs                      |  567 +++---
 compiler/nativeGen/PPC/Regs.hs                     |   14 +-
 compiler/nativeGen/PprBase.hs                      |   17 -
 compiler/nativeGen/RegAlloc/Graph/Main.hs          |  609 +++---
 compiler/nativeGen/RegAlloc/Graph/SpillCost.hs     |   10 +-
 compiler/nativeGen/RegAlloc/Graph/Stats.hs         |   30 +-
 compiler/nativeGen/RegAlloc/Graph/TrivColorable.hs |   56 +-
 compiler/nativeGen/RegAlloc/Linear/FreeRegs.hs     |   14 +-
 compiler/nativeGen/RegAlloc/Linear/Main.hs         |   44 +-
 compiler/nativeGen/RegAlloc/Liveness.hs            |   68 +-
 compiler/nativeGen/SPARC/CodeGen.hs                |  511 ++++-
 compiler/nativeGen/SPARC/CodeGen/CCall.hs          |  343 ---
 compiler/nativeGen/SPARC/CodeGen/CondCode.hs       |    7 +-
 compiler/nativeGen/SPARC/CodeGen/Gen64.hs          |    3 +-
 compiler/nativeGen/SPARC/CodeGen/Sanity.hs         |   10 +-
 compiler/nativeGen/SPARC/Imm.hs                    |    5 +-
 compiler/nativeGen/SPARC/Ppr.hs                    |   94 +-
 compiler/nativeGen/TargetReg.hs                    |   70 +-
 compiler/nativeGen/X86/CodeGen.hs                  |  458 +++--
 compiler/nativeGen/X86/Instr.hs                    |    3 +
 compiler/nativeGen/X86/Ppr.hs                      |  155 +-
 compiler/nativeGen/X86/Regs.hs                     |   43 +-
 compiler/parser/Lexer.x                            |   75 +-
 compiler/parser/Parser.y.pp                        |  143 +-
 compiler/parser/ParserCore.y                       |   34 +-
 compiler/parser/RdrHsSyn.lhs                       |  330 ++--
 compiler/prelude/ForeignCall.lhs                   |   56 +-
 compiler/prelude/PrelNames.lhs                     |  136 +-
 compiler/prelude/PrelRules.lhs                     |  212 ++-
 compiler/prelude/PrimOp.lhs                        |    3 -
 compiler/prelude/PrimOp.lhs-boot                   |    7 +
 compiler/prelude/TysPrim.lhs                       |  114 +-
 compiler/prelude/TysWiredIn.lhs                    |   88 +-
 compiler/prelude/primops.txt.pp                    |   39 +
 compiler/profiling/CostCentre.lhs                  |    6 +-
 compiler/profiling/ProfInit.hs                     |    9 +-
 compiler/profiling/SCCfinal.lhs                    |    1 +
 compiler/rename/RnBinds.lhs                        |   80 +-
 compiler/rename/RnEnv.lhs                          |  470 +++--
 compiler/rename/RnExpr.lhs                         |   80 +-
 compiler/rename/RnHsSyn.lhs                        |  159 --
 compiler/rename/RnNames.lhs                        |  194 +--
 compiler/rename/RnPat.lhs                          |   41 +-
 compiler/rename/RnSource.lhs                       |  563 +++---
 compiler/rename/RnTypes.lhs                        |  652 ++++--
 compiler/simplCore/CSE.lhs                         |   30 +-
 compiler/simplCore/CoreMonad.lhs                   |   61 +-
 compiler/simplCore/FloatIn.lhs                     |   18 +-
 compiler/simplCore/OccurAnal.lhs                   |  116 +-
 compiler/simplCore/SetLevels.lhs                   |   41 +-
 compiler/simplCore/SimplCore.lhs                   |   92 +-
 compiler/simplCore/SimplEnv.lhs                    |    1 +
 compiler/simplCore/SimplMonad.lhs                  |    3 +-
 compiler/simplCore/SimplUtils.lhs                  |  234 +--
 compiler/simplCore/Simplify.lhs                    |  131 +-
 compiler/simplStg/SRT.lhs                          |    4 +-
 compiler/simplStg/SimplStg.lhs                     |   14 +-
 compiler/simplStg/UnariseStg.lhs                   |  167 ++
 compiler/specialise/Rules.lhs                      |  584 +++---
 compiler/specialise/SpecConstr.lhs                 |   13 +-
 compiler/specialise/Specialise.lhs                 | 1070 +++++-----
 compiler/stgSyn/CoreToStg.lhs                      |   41 +-
 compiler/stgSyn/StgLint.lhs                        |   71 +-
 compiler/stgSyn/StgSyn.lhs                         |   48 +-
 compiler/stranal/DmdAnal.lhs                       |   17 +-
 compiler/stranal/WorkWrap.lhs                      |   88 +-
 compiler/stranal/WwLib.lhs                         |   56 +-
 compiler/typecheck/FamInst.lhs                     |   78 +-
 compiler/typecheck/Inst.lhs                        |  104 +-
 compiler/typecheck/TcArrows.lhs                    |    4 +-
 compiler/typecheck/TcBinds.lhs                     |  485 ++---
 compiler/typecheck/TcCanonical.lhs                 | 1384 ++++++-------
 compiler/typecheck/TcClassDcl.lhs                  |  124 +-
 compiler/typecheck/TcDeriv.lhs                     |  292 ++-
 compiler/typecheck/TcEnv.lhs                       |  194 +-
 compiler/typecheck/TcErrors.lhs                    |  388 ++--
 compiler/typecheck/TcEvidence.lhs                  |  159 ++-
 compiler/typecheck/TcExpr.lhs                      |   88 +-
 compiler/typecheck/TcForeign.lhs                   |  104 +-
 compiler/typecheck/TcGenDeriv.lhs                  |   12 +-
 compiler/typecheck/TcGenGenerics.lhs               |  515 ++++-
 compiler/typecheck/TcHsSyn.lhs                     |  184 +-
 compiler/typecheck/TcHsType.lhs                    | 1528 +++++++------
 compiler/typecheck/TcInstDcls.lhs                  |  433 +++--
 compiler/typecheck/TcInteract.lhs                  | 1171 +++++-----
 compiler/typecheck/TcMType.lhs                     |  828 ++++----
 compiler/typecheck/TcMatches.lhs                   |   70 +-
 compiler/typecheck/TcPat.lhs                       |   77 +-
 compiler/typecheck/TcRnDriver.lhs                  |  231 ++-
 compiler/typecheck/TcRnMonad.lhs                   |   80 +-
 compiler/typecheck/TcRnTypes.lhs                   |  326 ++--
 compiler/typecheck/TcRules.lhs                     |  197 ++-
 compiler/typecheck/TcSMonad.lhs                    | 1493 ++++++++------
 compiler/typecheck/TcSimplify.lhs                  | 1115 ++++++----
 compiler/typecheck/TcSplice.lhs                    |  157 +-
 compiler/typecheck/TcSplice.lhs-boot               |    7 +-
 compiler/typecheck/TcTyClsDecls.lhs                |  848 ++++----
 compiler/typecheck/TcTyDecls.lhs                   |   18 +-
 compiler/typecheck/TcType.lhs                      |  131 +-
 compiler/typecheck/TcUnify.lhs                     |  237 +--
 compiler/types/Class.lhs                           |    3 -
 compiler/types/Coercion.lhs                        |   22 +-
 compiler/types/FamInstEnv.lhs                      |   12 +-
 compiler/types/FunDeps.lhs                         |   91 +-
 compiler/types/IParam.lhs                          |   41 -
 compiler/types/IParam.lhs-boot                     |   10 -
 compiler/types/InstEnv.lhs                         |   71 +-
 compiler/types/Kind.lhs                            |  240 +--
 compiler/types/OptCoercion.lhs                     |    1 +
 compiler/types/TyCon.lhs                           |  871 ++++----
 compiler/types/Type.lhs                            |  303 ++--
 compiler/types/TypeRep.lhs                         |  172 +-
 compiler/types/Unify.lhs                           |   59 +-
 compiler/utils/Binary.hs                           |    3 -
 compiler/utils/Digraph.lhs                         |   17 +-
 compiler/utils/Exception.hs                        |    6 +-
 compiler/utils/FastString.lhs                      |    5 -
 compiler/utils/GraphColor.hs                       |  597 +++---
 compiler/utils/GraphOps.hs                         |  923 ++++----
 compiler/utils/ListSetOps.lhs                      |   79 +-
 compiler/utils/MonadUtils.hs                       |   18 -
 compiler/utils/Outputable.lhs                      |  268 +--
 compiler/utils/Outputable.lhs-boot                 |    7 +
 compiler/utils/Panic.lhs                           |  197 ++-
 compiler/utils/Platform.hs                         |   14 +-
 compiler/utils/Pretty.lhs                          |    9 +-
 compiler/utils/UniqFM.lhs                          |    4 +-
 compiler/utils/Util.lhs                            |  211 +-
 compiler/vectorise/Vectorise.hs                    |   21 +-
 compiler/vectorise/Vectorise/Builtins/Base.hs      |    2 +-
 .../vectorise/Vectorise/Builtins/Initialise.hs     |    2 +-
 compiler/vectorise/Vectorise/Convert.hs            |    9 +-
 compiler/vectorise/Vectorise/Env.hs                |    9 +-
 compiler/vectorise/Vectorise/Exp.hs                |  716 +++++--
 .../vectorise/Vectorise/Generic/Description.hs     |    3 +-
 compiler/vectorise/Vectorise/Generic/PADict.hs     |   41 +-
 compiler/vectorise/Vectorise/Generic/PAMethods.hs  |   27 +-
 compiler/vectorise/Vectorise/Generic/PData.hs      |    4 +-
 compiler/vectorise/Vectorise/Monad.hs              |   20 +-
 compiler/vectorise/Vectorise/Monad/Base.hs         |   41 +-
 compiler/vectorise/Vectorise/Monad/Global.hs       |    9 +-
 compiler/vectorise/Vectorise/Monad/InstEnv.hs      |   18 +-
 compiler/vectorise/Vectorise/Type/Classify.hs      |    1 +
 compiler/vectorise/Vectorise/Type/TyConDecl.hs     |   29 +-
 compiler/vectorise/Vectorise/Type/Type.hs          |    1 +
 compiler/vectorise/Vectorise/Utils/Base.hs         |   19 +-
 compiler/vectorise/Vectorise/Utils/PADict.hs       |   74 +-
 config.guess                                       |  482 ++---
 configure.ac                                       |  156 +-
 distrib/compare/compare.hs                         |   23 +-
 docs/coding-style.html                             |   60 +-
 docs/comm/genesis/modules.html                     |    2 +-
 docs/comm/rts-libs/threaded-rts.html               |    2 +-
 docs/ext-core/Makefile                             |    3 -
 docs/ext-core/a4wide.sty                           |   39 -
 docs/ext-core/code.sty                             |   83 -
 docs/ext-core/core.bib                             |  124 -
 docs/ext-core/core.tex                             |  779 -------
 docs/ext-core/ghc.mk                               |   15 -
 docs/index.html.in                                 |   22 +-
 docs/users_guide/bugs.xml                          |   26 +-
 docs/users_guide/external_core.xml                 | 1807 +++++++++++++++
 docs/users_guide/ffi-chap.xml                      |   52 +
 docs/users_guide/flags.xml                         |  103 +-
 docs/users_guide/ghci.xml                          |  119 +-
 docs/users_guide/glasgow_exts.xml                  |  694 ++++--
 docs/users_guide/packages.xml                      |  108 +-
 docs/users_guide/phases.xml                        |   25 +-
 docs/users_guide/runghc.xml                        |    2 +-
 docs/users_guide/runtime_control.xml               |   43 +-
 docs/users_guide/ug-book.xml.in                    |    1 +
 docs/users_guide/ug-ent.xml.in                     |    2 +
 docs/users_guide/using.xml                         | 2354 +++++++++++---------
 docs/vh/vh.xml                                     |  593 +++---
 driver/ghci/ghc.mk                                 |    4 +-
 driver/utils/cwrapper.c                            |    6 +-
 ghc.mk                                             |   39 +-
 ghc/GhciMonad.hs                                   |   48 +-
 ghc/GhciTags.hs                                    |   42 +-
 ghc/InteractiveUI.hs                               |  616 ++++--
 ghc/Main.hs                                        |   19 +-
 ghc/ghc-bin.cabal.in                               |    4 +-
 ghc/ghc.mk                                         |   11 +-
 ghc/hschooks.c                                     |    5 +-
 includes/Cmm.h                                     |    8 +-
 includes/HaskellConstants.hs                       |   12 +-
 includes/MachDeps.h                                |   27 +-
 includes/Rts.h                                     |   26 +-
 includes/RtsAPI.h                                  |    2 +-
 includes/Stg.h                                     |   10 +-
 includes/ghc.mk                                    |   31 +-
 includes/mkDerivedConstants.c                      |   23 +-
 includes/rts/Constants.h                           |   50 +-
 includes/rts/EventLogFormat.h                      |   27 +-
 includes/rts/FileLock.h                            |    6 +-
 includes/rts/Flags.h                               |   12 +-
 includes/rts/Hooks.h                               |    6 +-
 includes/rts/Messages.h                            |   15 +-
 includes/rts/Threads.h                             |    4 +
 includes/rts/Types.h                               |    6 +-
 includes/rts/storage/GC.h                          |    5 +-
 includes/rts/storage/InfoTables.h                  |   22 +-
 includes/rts/storage/TSO.h                         |    2 +-
 includes/stg/DLL.h                                 |   25 +-
 includes/stg/MachRegs.h                            |  209 +-
 includes/stg/MiscClosures.h                        |    2 +
 includes/stg/Types.h                               |   29 +-
 libffi/ghc.mk                                      |   11 +-
 .../Distribution/InstalledPackageInfo/Binary.hs    |    4 +-
 libraries/gen_contents_index                       |   37 +-
 mk/config.mk.in                                    |   21 +-
 mk/tree.mk                                         |    2 +-
 mk/validate-settings.mk                            |   18 +-
 packages                                           |    3 +-
 rts/Adjustor.c                                     |  477 +++--
 rts/Capability.c                                   |   50 +-
 rts/Capability.h                                   |    6 +-
 rts/ClosureFlags.c                                 |   18 +-
 rts/Disassembler.c                                 |   18 +-
 rts/FileLock.c                                     |  144 ++
 rts/FileLock.h                                     |   15 +
 rts/GetTime.h                                      |    3 +
 rts/HeapStackCheck.cmm                             |   12 +-
 rts/Interpreter.c                                  |   48 +-
 rts/Linker.c                                       |  826 +++++++-
 rts/LinkerInternals.h                              |    6 +-
 rts/PosixSource.h                                  |    4 +
 rts/Prelude.h                                      |    8 +-
 rts/PrimOps.cmm                                    |   39 +-
 rts/Printer.c                                      |   26 +-
 rts/ProfHeap.c                                     |    2 +-
 rts/RaiseAsync.c                                   |   77 +-
 rts/RetainerProfile.c                              |    4 +-
 rts/RtsDllMain.c                                   |    4 +-
 rts/RtsDllMain.h                                   |    2 +-
 rts/RtsFlags.c                                     |   20 +-
 rts/RtsMain.c                                      |    7 +-
 rts/RtsProbes.d                                    |   37 +-
 rts/RtsStartup.c                                   |   20 +-
 rts/Schedule.c                                     |   38 +-
 rts/Stats.c                                        |  163 +-
 rts/Stats.h                                        |   19 +-
 rts/StgCRun.c                                      |   87 +-
 rts/StgMiscClosures.cmm                            |    6 +-
 rts/StgStartup.cmm                                 |   12 +-
 rts/Task.c                                         |  104 +-
 rts/Task.h                                         |   30 +-
 rts/Threads.c                                      |   54 +-
 rts/Ticky.c                                        |   26 +-
 rts/Trace.c                                        |  148 ++-
 rts/Trace.h                                        |  263 ++-
 rts/eventlog/EventLog.c                            |  201 ++-
 rts/eventlog/EventLog.h                            |   34 +
 rts/ghc.mk                                         |   23 +-
 rts/hooks/MallocFail.c                             |    2 +-
 rts/hooks/OutOfHeap.c                              |    2 +-
 rts/hooks/StackOverflow.c                          |    2 +-
 rts/package.conf.in                                |    8 +-
 rts/posix/Clock.h                                  |   35 +
 rts/posix/FileLock.c                               |  145 --
 rts/posix/FileLock.h                               |   15 -
 rts/posix/GetTime.c                                |   50 +-
 rts/posix/Itimer.c                                 |   13 +-
 rts/posix/OSMem.c                                  |    2 +-
 rts/posix/OSThreads.c                              |    1 -
 rts/posix/Select.c                                 |   81 +-
 rts/posix/Select.h                                 |    2 +-
 rts/sm/Compact.c                                   |    2 +-
 rts/sm/Evac.c                                      |    2 +-
 rts/sm/GC.c                                        |   25 +-
 rts/sm/Sanity.c                                    |   18 +-
 rts/sm/Scav.c                                      |    6 +-
 rts/sm/Storage.c                                   |  178 +-
 rts/sm/Storage.h                                   |    3 +-
 rts/win32/AwaitEvent.c                             |    4 +-
 rts/win32/GetTime.c                                |   74 +-
 rts/win32/IOManager.c                              |    2 +-
 rts/win32/OSMem.c                                  |   20 +-
 rts/win32/ThrIOManager.c                           |    6 +-
 rts/win32/libHSbase.def                            |    1 -
 rts/win32/libHSghc-prim.def                        |    1 +
 rts/win32/seh_excn.c                               |    5 +-
 rules/build-package-data.mk                        |    8 +-
 rules/build-package-way.mk                         |   15 +-
 rules/build-package.mk                             |    2 +-
 rules/build-prog.mk                                |    4 +-
 rules/c-suffix-rules.mk                            |   22 +-
 rules/cmm-suffix-rules.mk                          |   12 +-
 rules/distdir-way-opts.mk                          |   46 +-
 rules/hs-suffix-rules-srcdir.mk                    |   30 +-
 rules/hs-suffix-rules.mk                           |    8 +-
 rules/package-config.mk                            |   15 +-
 rules/pretty_commands.mk                           |   13 +
 sync-all                                           |   63 +
 utils/compare_sizes/Main.hs                        |    1 -
 utils/fingerprint/fingerprint.py                   |    2 +-
 utils/genapply/GenApply.hs                         |  650 +++---
 utils/genprimopcode/Main.hs                        |  433 ++--
 utils/genprimopcode/Parser.y                       |    1 -
 utils/genprimopcode/ParserM.hs                     |    1 -
 utils/genprimopcode/Syntax.hs                      |   20 +-
 utils/ghc-cabal/Main.hs                            |   44 +-
 utils/ghc-cabal/ghc.mk                             |    4 +-
 utils/ghc-pkg/Main.hs                              |  108 +-
 utils/ghc-pkg/ghc-pkg.wrapper                      |    2 +-
 utils/ghc-pkg/ghc.mk                               |    8 +-
 utils/ghctags/Main.hs                              |   16 +-
 utils/hp2ps/Key.c                                  |   23 +-
 utils/runghc/ghc.mk                                |    3 +
 utils/runghc/runghc.hs                             |   12 +-
 validate                                           |   27 +-
 483 files changed, 33989 insertions(+), 25880 deletions(-)

comment:11 Changed 4 years ago by thorkilnaur

The commit 5ddc841a547d2d37428280b04c071a1d4b83e10d causes tn23 (a Mac OS X builder) to fail with

configure: error: cannot find objdump in your PATH

See http://darcs.haskell.org/ghcBuilder/builders/tn23/655/8.html. Best regards Thorkil

comment:12 Changed 4 years ago by igloo

  • Blocking 3658 added

comment:13 follow-up: Changed 4 years ago by duncan

Have we looked into whether we can avoid exporting so many symbols by hiding all package-internal symbols. My intuition is that there must be lots of them that are only needed for intra-package linkage and are not needed in the external interface. The only symbols that should need to be external are the ones for things accessable via the .hi files of the exposed modules. Though if basically every module is exposed then that would not help much.

comment:14 in reply to: ↑ 13 Changed 4 years ago by simonmar

Replying to duncan:

Have we looked into whether we can avoid exporting so many symbols by hiding all package-internal symbols. My intuition is that there must be lots of them that are only needed for intra-package linkage and are not needed in the external interface. The only symbols that should need to be external are the ones for things accessable via the .hi files of the exposed modules. Though if basically every module is exposed then that would not help much.

We should do this, but it wouldn't help in the case of the GHC package: as you say, every module is exposed.

It would be good to have a per-module flag to say "this is a hidden module", and then we could give the symbols the correct linker hint so that they don't populate the .so file's symbol table.

comment:15 Changed 4 years ago by rassilon

  • Cc bill@… added

Here's a pretty nutty idea: What if we split the DLL in parts by automatically forcing the .o files into partitions and create a .def file for each partition thus breaking the import library cycle.

GHC already uses .def files to break the cycle between the RTS and the base DLLs.

Then you would "just" modify Cabal to understand partitioned DLLs. Unfortunately, this might end up increasing the total number of partitions if you're not careful because internal symbols would now need to be exported from the partitions. :(

comment:16 Changed 4 years ago by igloo@…

commit 56353e3da9d5718dfd25e25ccf61c78b25deefe8

Author: Ian Lynagh <igloo@earth.li>
Date:   Tue Mar 12 22:12:38 2013 +0100

    Finish adding support for 2 DLLs in the ghc package; fixes #5987

 ghc.mk                     |    3 ---
 rules/build-package-way.mk |   20 +++++++++++++++++---
 utils/ghc-cabal/Main.hs    |   27 +++++++++++++++++++++++++--
 3 files changed, 42 insertions(+), 8 deletions(-)

comment:17 Changed 4 years ago by igloo

  • Resolution set to fixed
  • Status changed from new to closed

Fixed. We now have

libHSghc-7.7.20130310-0-ghc7.7.20130310.dll (42123 symbols)
libHSghc-7.7.20130310-ghc7.7.20130310.dll   (47675 symbols)

comment:18 Changed 4 years ago by simonpj

Great. Are there some Wiki notes or comments in the code to explain the thinking, what the problem is, how it is solved, where the partioning is done, how you managed to factor GHC into two? Etc. This kind of global info can be hard to find.

Indeed is there a wiki page surveying the whole dynamic-linking question and how we address it? I can't find it if so.

Simon

comment:19 Changed 3 years ago by george.colpitts

For the wiki page surveying the whole dynamic-linking question I believe it is http://ghc.haskell.org/trac/ghc/wiki/DynamicGhcPrograms

comment:20 Changed 3 years ago by george.colpitts

  • Cc george.colpitts@… added

comment:21 Changed 3 years ago by awson

Now x86_64 build fails on Windows again:

Number of symbols in compiler/stage2/build/libHSghc-7.7.20130915-0-ghc7.7.20130915.dll: 65583
...
Too many symbols in DLL compiler/stage2/build/libHSghc-7.7.20130915-0-ghc7.7.20130915.dll

comment:22 Changed 3 years ago by akio

  • Cc tkn.akio@… added

comment:23 Changed 3 years ago by awson

  • Resolution fixed deleted
  • Status changed from closed to new

comment:24 Changed 3 years ago by thoughtpolice

  • Owner set to thoughtpolice

comment:25 Changed 3 years ago by tomberek

compiling master:

# Now check that the DLL doesn't have too many symbols. See trac #5987.
SYMBOLS=`c:/Haskell/ghc/inplace/mingw/bin/objdump.exe -p compiler/stage2/build/l
ibHSghc-7.7.20131018-0-ghc7.7.20131018.dll | sed -n "1,/^.Ordinal\/Name Pointer/
 D; p; /^$/ q" | tail -n +2 | wc -l`; echo "Number of symbols in compiler/stage2
/build/libHSghc-7.7.20131018-0-ghc7.7.20131018.dll: $SYMBOLS"
Number of symbols in compiler/stage2/build/libHSghc-7.7.20131018-0-ghc7.7.201310
18.dll: 70721
case `c:/Haskell/ghc/inplace/mingw/bin/objdump.exe -p compiler/stage2/build/libH
Sghc-7.7.20131018-0-ghc7.7.20131018.dll | sed -n "1,/^.Ordinal\/Name Pointer/ D;
 p; /^$/ q" | grep "\[ *0\]" | wc -l` in 1) echo DLL compiler/stage2/build/libHS
ghc-7.7.20131018-0-ghc7.7.20131018.dll OK;; 0) echo No symbols in DLL compiler/s
tage2/build/libHSghc-7.7.20131018-0-ghc7.7.20131018.dll; exit 1;; [0-9]*) echo T
oo many symbols in DLL compiler/stage2/build/libHSghc-7.7.20131018-0-ghc7.7.2013
1018.dll; c:/Haskell/ghc/inplace/mingw/bin/objdump.exe -p compiler/stage2/build/
libHSghc-7.7.20131018-0-ghc7.7.20131018.dll | sed -n "1,/^.Ordinal\/Name Pointer
/ D; p; /^$/ q" | tail; exit 1;; *) echo bad DLL compiler/stage2/build/libHSghc-
7.7.20131018-0-ghc7.7.20131018.dll; exit 1;; esac
Too many symbols in DLL compiler/stage2/build/libHSghc-7.7.20131018-0-ghc7.7.201
31018.dll
        [5176] ghczm7zi7zi20131018_Var_zdfUniquableVar_closure
        [5177] ghczm7zi7zi20131018_Var_zdfUniquableVar_info
        [5178] ghczm7zi7zi20131018_Var_zdwsetVarUnique_closure
        [5179] ghczm7zi7zi20131018_Var_zdwsetVarUnique_info
        [5180] ghczm7zi7zi20131018_Var_zdwupdateTyVarKindM_closure
        [5181] ghczm7zi7zi20131018_Var_zdwupdateTyVarKindM_info
        [5182] ghczm7zi7zi20131018_Var_zdwzdcgmapMp_closure
        [5183] ghczm7zi7zi20131018_Var_zdwzdcgmapMp_info
        [5184] setHeapSize

make[1]: *** [compiler/stage2/build/libHSghc-7.7.20131018-0-ghc7.7.20131018.dll]
 Error 1
make[1]: *** Deleting file `compiler/stage2/build/libHSghc-7.7.20131018-0-ghc7.7
.20131018.dll'
make: *** [all] Error 2

Windows 7, MSYS, ld 2.23.2

The stage1 of the compiler seems to work as expected on non-TH code. Let me know if any other information is needed.

Last edited 3 years ago by tomberek (previous) (diff)

comment:26 Changed 3 years ago by shelarcy

  • Cc shelarcy@… added

comment:27 Changed 3 years ago by Austin Seipp <austin@…>

In 9bb909bbcf280f137c7a379d05acf5bc21268e5b/ghc:

Temporarily lower unfolding threshold on Windows

This is a very temporary, very unsatisfactory hack to fix #5987
(for now.)

The included comments essentially say it all: we lower the unfolding
threshold to minimize some amount of exported symbols from the GHC
stage2 DLL. I unfortunately had to lower it quite substantially for the
dynamic stage2 build to pass.

As of this writing, the DLL split between ghc.dll and ghc-0.dll is
something like 26,000 vs 63,000 exported symbols, respectively. So we're
still quite in danger of tripping it, but I think we will be OK at this
exact moment.

Signed-off-by: Austin Seipp <austin@well-typed.com>

comment:28 Changed 3 years ago by thoughtpolice

  • Milestone changed from 7.8.1 to 7.10.1

I've temporarily fixed this in HEAD by lowering the unfolding threshold for the stage2 build on windows. We're still extremely close to the symbol limit, but it should stay OK for 7.8.

I'm leaving this highest priority and bumping to 7.10.1 - we must find a better fix.

comment:29 Changed 3 years ago by Austin Seipp <austin@…>

In 4af1e76c701a7698ebd9b5ca3fb1394dd8b56c8d/ghc:

Add Windows to NoSharedLibsPlatformList

We're punting on full -dynamic and -dynamic-too support for Windows
right now, since it's still unstable. Also, ensure "Support dynamic-too"
in `ghc --info` is set to "NO" for Cabal.

See issues #7134, #8228, and #5987

Signed-off-by: Austin Seipp <austin@well-typed.com>

comment:30 Changed 3 years ago by carter

is there some way we could do a hack to do split DLLs and allow cycles? (would probably mean some sort of indirect jump penalty in one direction of the cycle).

One extreme would be to use some sort of graph clustering software tool like METIS http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

do .hi files have enough information to sort out the directed graph of imports exports? I'd be happy to help do some graph clustering experimentation on the object files (that piece isn't windows specific, so i could hack it out on my current machine)

comment:31 Changed 3 years ago by carter

alternatively, if the free version of visual studio, visual studio express, http://www.visualstudio.com/en-us/products/visual-studio-express-vs.aspx comes with the Visual Studio linker, doesn't that have some support for dealing with cyclic deps in linking?

comment:32 Changed 3 years ago by thoughtpolice

  • Milestone changed from 7.10.1 to 7.8.2

comment:33 Changed 2 years ago by thoughtpolice

  • Milestone changed from 7.8.2 to 7.8.3

comment:34 Changed 2 years ago by thoughtpolice

  • Priority changed from highest to high

comment:35 Changed 2 years ago by thoughtpolice

  • Milestone changed from 7.8.3 to 7.8.4

Moving to 7.8.4.

comment:36 Changed 2 years ago by thoughtpolice

  • Owner changed from thoughtpolice to non-existant-user

comment:37 Changed 2 years ago by thoughtpolice

  • Owner changed from non-existant-user to thoughtpolice

comment:38 Changed 2 years ago by thoughtpolice

  • Blocking 3658 removed

comment:39 Changed 2 years ago by thomie

  • Operating System changed from Unknown/Multiple to Windows

comment:40 Changed 2 years ago by thoughtpolice

  • Milestone changed from 7.8.4 to 7.10.1

Moving (in bulk) to 7.10.4

comment:41 Changed 22 months ago by thomie

  • Blocking 7389 added

comment:42 Changed 22 months ago by thomie

  • Blocking 5355 added

See also the discussion in #5292.

Last edited 22 months ago by thomie (previous) (diff)

comment:43 Changed 20 months ago by thoughtpolice

  • Milestone changed from 7.10.1 to 7.12.1

Moving to 7.12.1

comment:44 Changed 15 months ago by Phyx-

  • Blocking 7389 removed

comment:45 Changed 13 months ago by thoughtpolice

  • Milestone changed from 7.12.1 to 8.0.1

Milestone renamed

comment:46 follow-up: Changed 10 months ago by bgamari

  • Owner thoughtpolice deleted
  • Priority changed from high to normal

Demoting priority as we have disabled dynamic linking on Windows (see comment:29).

My understanding of this situation is that it is not an intrinsic limitation of PE. The relevant data structures are the .edata and .idata tables which are defined in sections 63 and 64 of the PE/COFF specification. The specification specifies that one means of referring to exported symbols is via a 16-bit ordinal. However, this is not the only way; if the lookup fails the loader will fall back to a standard name search. Indeed, Microsoft even explicitly says that ordinals are a legacy artifact and their use is discouraged in new code (this document describes the .def format, but the idea still applies),

You can use @ordinal to specify that a number, and not the function name, will go into the DLL's export table. Many Windows DLLs export ordinals to support legacy code. It was common to use ordinals in 16-bit Windows code, because it can help minimize the size of a DLL. We don’t recommend exporting functions by ordinal unless your DLL’s clients need it for legacy support. Because the .LIB file will contain the mapping between the ordinal and the function, you can use the function name as you normally would in projects that use the DLL.

For this reason I believe the problem is in fact that ld.bfd is producing DLLs with symbols exported by (or perhaps objects that are imported by) ordinal.

comment:47 Changed 9 months ago by Ben Gamari <ben@…>

In cdeefa4/ghc:

ghc.mk: Add reference to Trac #5987

comment:48 in reply to: ↑ 46 Changed 9 months ago by rassilon

Replying to bgamari:

My understanding of this situation is that it is not an intrinsic limitation of PE. The relevant data structures are the .edata and .idata tables which are defined in sections 63 and 64 of the PE/COFF specification. The specification specifies that one means of referring to exported symbols is via a 16-bit ordinal. However, this is not the only way; if the lookup fails the loader will fall back to a standard name search. Indeed, Microsoft even explicitly says that ordinals are a legacy artifact and their use is discouraged in new code (this document describes the .def format, but the idea still applies),

Your understanding doesn't match my understanding of the PE/COFF specification. Page 75 of the 02/06/2013 version of the spec which is the page where the Export Ordinal Table of the .edata section is described.

The section includes pseudo-code explaining how the SymbolRVA of a named symbol is determined:

i = Search_ExportNameTable(ExportName);

ordinal = ExportOrdinalTable[i];

SymbolRVA = ExportAddressTable[ordinal - ordinalBase];

So, the problem isn't with this pseudo-code, but the pseudo-code combined with the Export Ordinal Table being defined as:

The export ordinal table is an array of 16-bit indexes into the address table.

This is especially sad, since the Address Table Entries field of the Export Directory Table is 32 bits of data.

The text you quote below is from the syntax of .DEF files explaining why 16-bit DLLs, didn't include exported symbol string names and utilized the ordinal number as the ONLY API between DLLs. The documentation you quote is recommending usage of string symbols for providing a cross-DLL API resolution and only using the ordinal API for legacy reasons. The documentation doesn't mean to imply that ordinals are optional (based on my reading of the PE/COFF sections about .edata).

You can use @ordinal to specify that a number, and not the function name, will go into the DLL's export table. Many Windows DLLs export ordinals to support legacy code. It was common to use ordinals in 16-bit Windows code, because it can help minimize the size of a DLL. We don’t recommend exporting functions by ordinal unless your DLL’s clients need it for legacy support. Because the .LIB file will contain the mapping between the ordinal and the function, you can use the function name as you normally would in projects that use the DLL.

comment:49 follow-up: Changed 8 months ago by thomie

rassilon: you seem to know stuff. How to fix dynamic linking on Windows? Some more problems here: #4824 #5620 #7665 #8228.

comment:50 in reply to: ↑ 49 Changed 8 months ago by rassilon

Replying to thomie:

rassilon: you seem to know stuff. How to fix dynamic linking on Windows? Some more problems here: #4824 #5620 #7665 #8228.

The "easiest" way of fixing this issue is by refactoring the GHC internals in such a way as to minimize the dependencies between the partitions of the GHC base DLL. (in short, removing all need for Haskell source level cyclic imports could help a bunch here)

Either that, or come up with an automatic partitioning scheme that could handle cyclic references between the partition DLLs.

The PE/COFF limitation of only being able to export 16bits worth of APIs from a dynamic library is the biggest problem of all of these issues. #5620 would only be worth addressing when we have a solution to this issue one way or another.

Given that the root of the 16bit exported API problem stems from the PE/COFF specification, it would seem that any alternatives we come up would be conceptually equivalent to our current usages of non-dlopen(or equivalent) dynamic loading/linking that RTS/GHCi do today. In fact the code necessary to fix this issue would probably make that logic even more convoluted and difficult to understand/maintain. (Since the code would need to understand whatever additional sections/per-DLL API functions/etc.. that would allow the code to be relocated and linked properly.)

Such a scheme would have additional downsides as well:

  • It would be (by necessity) be Windows specific.
  • It would probably completely shatter the mechanism for calling Haskell functions from C/C++ either in the same DLL or another DLL.

Of the possible above approaches, I would think an automatic partitioning approach that handled cyclic references between the partitions would be the approach that had the best long term chances of successfully being maintained.

Having said that of course, even automatic partitioning is still a non trivial amount of work. Since ideally, Cabal would know about DLL partitioning as well so that when anyone referenced a Haskell package with partitions, Cabal would help (or direct) GHC to link against all of the partition DLLs of that package.

Ugh. Maybe it'll be easier once Microsoft lets us push a pull request to Window's dynamic linker to support more than 16bits of ordinals for Windows 20. ;)

comment:51 Changed 8 months ago by thomie

  • Milestone 8.0.1 deleted

comment:52 Changed 8 months ago by simonmar

I propose:

  • Do not dynamically-link GHC on Windows
  • Do not build a dynamic version of the GHC package
  • Continue to build dynamic versions of everything else
  • Use Remote GHCi (-fexternal-interpreter) to get dynamic linking in GHCi
  • Close this ticket :)

comment:53 Changed 8 months ago by rassilon

I'm not sure I completely understand your GHCi suggestion (but that is unnecessary) , but I certainly agree with the rest of your suggestions. If someone cares enough, they can fund the necessary engineering. :)

comment:54 Changed 3 months ago by Phyx-

  • Owner set to Phyx-

I have recently taken a look at this and have an almost working version that should solve the problem once and for all.

First, we don't actually have that many symbols to go over the limit. Or it seems we don't. If I measure amount of symbols in the input object files going into the link and the amount coming out, the difference is huge.

Looking at it further this is because of two things. We never explicitly use __declspec, we just change the names of the functions to match the conventions that __declspec would use. This is fine, but it means that binutil's default of --export-all-symbols is still enabled. Which means, we'll re-export any symbol we link from archives as well.

Given that -dynamic on Windows always produces an import library .dll.a and the search order for ld is

libxxx.dll.a
xxx.dll.a
libxxx.a
cygxxx.dll (*)
libxxx.dll
xxx.dll

Then we always end up picking the import lib. This is recursive, we link against gmp, base etc. By the time it gets to GHC the resulting import lib is huge and hence we blow passed the number of symbols. Also kernel and gdi32 and mingwex etc are all import libraries for GCC. So we accumulate a ton of symbols from there as well.

So the first thing my changes do is only export symbols defined in the input object files.

This not only drastically reduces the size of the resulting DLLs and import libraries, it also pushes the number of symbols way way below the limit.

In fact I got rid of dll-split all together and allow all symbols to go into the same dll and we end up with

$ nm -g "R:\ghc\libHSghc-8.1-ghc8.1.20160617.dll" | wc -l
49610

This down from ~240,000 (mingwex and mingw32 are huge for instance).

The second thing my build changes do is that in order to prevent this from happening again, I implemented an automatic partitioning scheme which requires no special treatment of the split dlls.

In case we hit the limit again, the build script will automatically detect this and do the following:

It will split the symbols up per object file input. So that all symbols of the same object file are in the same DLL.

Like @rassilon suggested before, I'm using import libraries to break the dependencies. So the specific grouping doesn't matter.

The import libraries point to the location of the dll which contains the symbol:

LIBRARY "libHSCabal-1.25.0.0-ghc8.1.20160617-pt2.dll"
EXPORTS
    "__stginit_Cabalzm1zi25zi0zi0_DistributionziCompatziBinary"
    "__stginit_Cabalzm1zi25zi0zi0_DistributionziCompatziCopyFile"
...

And these are used to break the dependencies.

We then end up with smaller dlls with the suffix -pt<num>.dll and their import libraries.

The next step is to produce one large/merged import library with the name of the dll we were originally supposed to create. libHSCabal-1.25.0.0-ghc8.1.20160617.dll.a which is just a merging of the different -pt import files.

This has the effect that when -lHSCabal-1.25.0.0-ghc8.1.20160617 is used as the link argument (which we do), the import lib is found and the linker puts a reference to the right dlls. No extra/special handling is needed by any other tool.

Using the import libraries essentially removes the limit, since each symbol is an object file in the archive.

(note that while I recently added support for import libraries to GHCi, this support only extends to single dll import libraries. It needs some minor modifications to support this too but LD should work fine.)

This works fine, and I can successfully compile a dynamic version of GHC and the program runs (but segfaults due to a piece of bit rotted code I'm looking at).

comment:55 Changed 3 months ago by simonpj

Phyx, that sounds amazing! I have not idea what you are talking about -- I do not know enough about Windows and import libraries -- but for the first time we have someone who DOES actually understand looking hard at it. I always HATED the dll-split thing. Thank you.

Maybe before you are done you can write a Duffers Guide to what people need to know about Windows linking to make sense of the implementation strategies you are using.

With thanks

Simon

comment:56 Changed 3 months ago by Phyx-

Hi @simonpj,

Sure that won't be a problem, I was already planning on writing an extended explanation of the implementation so make the code review easier and for if someone else needs to take a look at the implementation in the future :)

comment:57 Changed 3 months ago by simonmar

I second what @simonpj said, amazing job, thanks for working on this!

comment:58 Changed 3 months ago by Phyx-

  • Blocked By 10352 added

Quick update, dynamic linking now works again:

$ ghc-stage2 -fforce-recomp test.hs -dynamic && ./test
[1 of 1] Compiling Main             ( test.hs, test.o )
Linking test.exe ...
Hello World

And loader logs:

Process Started: 0000000000400000 E:\temp\dynamic\test.exe
...
DLL Loaded: 0000000070D00000 R:\GHC\libHSghc-prim-0.5.0.0-ghc8.1.20160628.dll
DLL Loaded: 0000000067240000 R:\GHC\libHSrts-ghc8.1.20160628.dll
...
DLL Loaded: 0000000069F80000 R:\GHC\libHSbase-4.9.0.0-ghc8.1.20160628.dll
...
DLL Loaded: 000000006B740000 R:\GHC\libffi-6.dll
...
DLL Loaded: 0000000069780000 R:\GHC\libHSinteger-gmp-1.0.0.1-ghc8.1.20160628.dll

Approach is written https://ghc.haskell.org/trac/ghc/wiki/WindowsDynamicLinking here and I'll keep updating it as I go.

Code is not done but changes can be seen https://github.com/Mistuke/ghc/pull/5 on my Github.

The major limitation in this version is that only the normal rts can be used. Due to the fact that we can't link with unknown symbols on Windows in the final link. After talking to rwbarton I believe this can be solved by a Windows version of #10352 and so temporarily stopping work on this so I can take a look at #10352 instead.

Remaining:

  • Figure out the ASSERT failures in compiler/stgSyn/CoreToStg.hs
  • Fix dist building to include split dlls if split occurs
  • Implement mechanism to select right rts version.
  • Clean up code

For reference, the sizes of the produced dlls are now also a tad smaller

  695960 /r/GHC/libHSarray-0.5.1.1-ghc8.1.20160628.dll
11997540 /r/GHC/libHSbase-4.9.0.0-ghc8.1.20160628.dll
  770941 /r/GHC/libHSbinary-0.8.3.0-ghc8.1.20160628.dll
 1222357 /r/GHC/libHSbytestring-0.10.8.1-ghc8.1.20160628.dll
17275289 /r/GHC/libHSCabal-1.25.0.0-ghc8.1.20160628.dll
 2327994 /r/GHC/libHScontainers-0.5.7.1-ghc8.1.20160628.dll
  183756 /r/GHC/libHSdeepseq-1.4.2.0-ghc8.1.20160628.dll
  386058 /r/GHC/libHSdirectory-1.2.6.2-ghc8.1.20160628.dll
  224319 /r/GHC/libHSfilepath-1.4.1.0-ghc8.1.20160628.dll
47179898 /r/GHC/libHSghc-8.1-ghc8.1.20160628.dll
  583840 /r/GHC/libHSghc-boot-8.1-ghc8.1.20160628.dll
  476365 /r/GHC/libHSghc-boot-th-8.1-ghc8.1.20160628.dll
 2004675 /r/GHC/libHSghci-8.1-ghc8.1.20160628.dll
 3144663 /r/GHC/libHSghc-prim-0.5.0.0-ghc8.1.20160628.dll
 1721465 /r/GHC/libHShaskeline-0.7.2.3-ghc8.1.20160628.dll
  894721 /r/GHC/libHShoopl-3.10.2.1-ghc8.1.20160628.dll
  318805 /r/GHC/libHShpc-0.6.0.3-ghc8.1.20160628.dll
  761724 /r/GHC/libHSinteger-gmp-1.0.0.1-ghc8.1.20160628.dll
  668894 /r/GHC/libHSpretty-1.1.3.3-ghc8.1.20160628.dll
  264811 /r/GHC/libHSprocess-1.4.2.0-ghc8.1.20160628.dll
 1680082 /r/GHC/libHSrts_debug-ghc8.1.20160628.dll
 2182245 /r/GHC/libHSrts_l-ghc8.1.20160628.dll
 1796551 /r/GHC/libHSrts_thr_debug-ghc8.1.20160628.dll
 2466036 /r/GHC/libHSrts_thr_l-ghc8.1.20160628.dll
 2206968 /r/GHC/libHSrts_thr-ghc8.1.20160628.dll
 1949321 /r/GHC/libHSrts-ghc8.1.20160628.dll
 4638331 /r/GHC/libHStemplate-haskell-2.11.0.0-ghc8.1.20160628.dll
 1448186 /r/GHC/libHStime-1.6.0.1-ghc8.1.20160628.dll
 1494791 /r/GHC/libHStransformers-0.5.2.0-ghc8.1.20160628.dll
 2644212 /r/GHC/libHSWin32-2.3.1.1-ghc8.1.20160628.dll

comment:59 Changed 6 weeks ago by RyanGlScott

  • Cc RyanGlScott added

comment:60 Changed 5 weeks ago by Phyx-

  • Milestone set to 8.2.1

This will be finished by 8.2.1 (probably earlier as well).

Though it requires some thinking about how the bindist should go.

Note: See TracTickets for help on using tickets.