Opened 2 years ago

Closed 2 years ago

Last modified 23 months ago

#10777 closed bug (fixed)

Overlong linker arguments on Windows leads to broken builds with confusing error messages

Reported by: snoyberg Owned by: snoyberg
Priority: normal Milestone: 8.0.1
Component: Compiler (Linking) Version: 7.10.2
Keywords: Cc:
Operating System: Windows Architecture: Unknown/Multiple
Type of failure: Compile-time crash Test Case:
Blocked By: Blocking:
Related Tickets: #8596 #9685 Differential Rev(s):
Wiki Page:

Description

I've seen three different issues on Github about this: two in the stack repo for stack users, and one in the yesod repo for a cabal sandbox user:

https://github.com/yesodweb/yesod/issues/1020 https://github.com/commercialhaskell/stack/issues/466 https://github.com/commercialhaskell/stack/issues/795

The problem comes down to this: Windows has a limit of 32k on the size of a command string, which can be tripped by GHC's link phase when using a large number of libraries. The ideal situation would be for GHC to have a different way of passing large argument lists to the linker; I don't know how feasible this is, but I wanted to open up a ticket before I started investigating myself in case others have seen this.

Attachments (1)

cabal-rsp.patch (4.2 KB) - added by awson 2 years ago.
Quick and dirty Cabal patch to make it use response files in a couple of ar and ld invocations

Download all attachments as: .zip

Change History (13)

comment:1 Changed 2 years ago by awson

I have a patch for Cabal which introduces response files usage for ar and ld. This speeds things up dramatically in some cases and eliminates some problems with prelinked object files creation when split-objs and gcc > 4.8 are used.

This technique can clearly be applied to gcc invocations also because gcc on windows also supports response files (for ages).

The problem with this patch is that:

  1. I am lazy, and since I use this patch only internally I didn't bother to factor common code out and simply almost copypasted (added a very minor edit) the same code both for ar and ld.
  2. I didn't test anything on the platforms other than windows -- perhaps something could be broken there. OTOH, the patch is very simple and could easily be polished to play nicely with a platform specificity.
  3. It deals with Cabal, not with ghc. I didn't read linked content thoroughly, do we need to modify ghc, or modifying Cabal would suffice? Btw, feature request for adding it to ghc exists.

Anyway, if anybody is interested to look at how this works I can attach the patch.

comment:2 Changed 2 years ago by snoyberg

I'd very much like to see the Cabal diff, and I'm sure other users would too.

Changed 2 years ago by awson

Attachment: cabal-rsp.patch added

Quick and dirty Cabal patch to make it use response files in a couple of ar and ld invocations

comment:3 Changed 2 years ago by awson

More or less relevant is the part behind withTempFile.

comment:4 Changed 2 years ago by Phyx-

The limits are a bit all over the place. For the record:

  • The maximum command line length for the CreateProcess function is 32767 characters. This limitation comes from the UNICODE_STRING structure. CreateProcess is the core function for creating processes, so if you are talking directly to Win32, then that's the only limit you have to worry about. But if you are reaching CreateProcess by some other means, then the path you travel through may have other limits.
  • If you are using the CMD.EXE command processor, then you are also subject to the 8192 character command line length limit imposed by CMD.EXE.
  • If you are using the ShellExecute/Ex function, then you become subject to the INTERNET_MAX_URL_LENGTH (around 2048) command line length limit imposed by the ShellExecute/Ex functions. (If you are running on Windows 95, then the limit is only MAX_PATH.)

[1] http://blogs.msdn.com/b/oldnewthing/archive/2003/12/10/56028.aspx

[2] https://support.microsoft.com/en-us/kb/830473

comment:5 Changed 2 years ago by snoyberg

Owner: set to snoyberg

comment:6 Changed 2 years ago by snoyberg

I've sent a diff for this:

https://phabricator.haskell.org/D1158

Unfortunately, I haven't been successful yet at building GHC on Windows, so I haven't tested there yet. If someone who's experienced this problem has success building with that change, please report back.

comment:7 Changed 2 years ago by Ben Gamari <ben@…>

In 296bc70/ghc:

Use a response file for linker command line arguments #10777

On Windows, we're constrained to 32k bytes total for command line
arguments.  When building large projects, this limit can be exceeded.
This patch changes GHC to always use response files for linker
arguments, a feature first used by Microsoft compilers and added to GCC
(over a decade ago).

Alternatives here include:

* Only use this method on Windows systems
* Check the length of the command line arguments and use that to decide
  whether to use this method

I did not pursue either of these, as I believe it would make the patch
more likely to break in less tested situations.

Test Plan:
Confirm that linking still works in general. Ideally: compile a very
large project on Windows with this patch. (I am attempting to do that
myself now, but having trouble getting the Windows build tool chain up
and running.)

Reviewers: goldfire, hvr, rwbarton, austin, thomie, bgamari, Phyx

Reviewed By: thomie, bgamari, Phyx

Subscribers: erikd, awson, #ghc_windows_task_force, thomie

Differential Revision: https://phabricator.haskell.org/D1158

GHC Trac Issues: #8596, #10777

comment:8 Changed 2 years ago by ezyang

Status: newmerge

Merge maybe?

comment:9 Changed 2 years ago by JohnWiegley

Component: CompilerCompiler (Linking)

comment:10 Changed 2 years ago by bgamari

Resolution: fixed
Status: mergeclosed

comment:11 Changed 2 years ago by bgamari

This was merged to ghc-7.10 as 6b08e42ad99bb7857b631f28db869def046bff35.

comment:12 Changed 23 months ago by thomie

Milestone: 8.0.1
Note: See TracTickets for help on using tickets.