linker fails to load package with binding to foreign library

mentioned in issue #1883 (closed)

mentioned in issue #2283 (closed)

mentioned in issue #3242 (closed)

mentioned in issue #3658 (closed)

changed weight to 5

added RTS Tbug Trac import labels

Attached file winsock-load-faiure.hs ($744).

WSACleanup is declared __declspec(dllimport) in the MinGW CRT headers. That makes the object file to refer to __imp__WSACleanup@0, to be resolved through a stub library of MinGW named libws2_32.a. ws2_32 is mentioned in the extra-libraries field of the network package entry in the ghc package configuration database, so -lws2_32 is added to the linker command arguments. From the linker documentation:

     For instance, when ld is called with the argument `-lxxx' it will
     attempt to find, in the first directory of its search path,

          libxxx.dll.a
          xxx.dll.a
          libxxx.a
          xxx.lib
          cygxxx.dll (*)
          libxxx.dll
          xxx.dll

     before moving on to the next directory in the search path.

So the linker finds libws2_32.a in one of the search paths passed on on the command line (through the built-in gcc specs, cf. link_gcc_c_sequence).

Trac metadata

Trac field	Value
Related	→ #2283 (closed)

When linking the binary, how does the linker know it also needs ws2_32.dll?

I wonder why GHCi doesn't also find libws2_32.a? Maybe it isn't on the search path for some reason.

Replying to [ticket:7097#comment:63254 simonmar]:

When linking the binary, how does the linker know it also needs ws2_32.dll?

It doesn't. Only the stub file is linked in. The stub file is a peculiar to Windows way to produce an indirection/relocation. When the produced binary is executed, the Windows dynamic loader takes care of the __imp__WSACleanup@0 symbol redirection/resolution.

I wonder why GHCi doesn't also find libws2_32.a? Maybe it isn't on the search path for some reason.

It doesn't search for libxxx.a, and the path searched does not include the location of the stub file. Also, I believe the PEI386 handling in rts/Linker.c currently would be unable to load such stub files. Please, pardon my brevity, still looking into this.

libws2_32.a in MinGW-w64 is an archive combining objects created from a .def file and the C sources `src_libws2_32`. The .def file declares the indirection:

;
; Definition file of WS2_32.dll
; Automatic generated by gendef
; written by Kai Tietz 2008
;
LIBRARY "WS2_32.dll"
EXPORTS
[...snip...]
WSACleanup@0
[...snip...]

$ objdump -x /c/mingw32/i686-w64-mingw32/lib/libws2_32.a
[...snip...]
dajhs00026.o:     file format pe-i386
rw-r--r-- 1000/100    619 Apr 30 06:54 2012 dajhs00026.o
architecture: i386, flags 0x00000039:
HAS_RELOC, HAS_DEBUG, HAS_SYMS, HAS_LOCALS
start address 0x00000000

Characteristics 0x104
        line numbers stripped
        32 bit words
[...snip...]
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000008  00000000  00000000  0000012c  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  00000000  00000000  00000000  2**2
                  ALLOC, LOAD, DATA
  2 .bss          00000000  00000000  00000000  00000000  2**2
                  ALLOC
  3 .idata$7      00000004  00000000  00000000  00000134  2**2
                  CONTENTS, RELOC
  4 .idata$5      00000004  00000000  00000000  00000138  2**2
                  CONTENTS, RELOC
  5 .idata$4      00000004  00000000  00000000  0000013c  2**2
                  CONTENTS, RELOC
  6 .idata$6      0000000e  00000000  00000000  00000140  2**1
                  CONTENTS
SYMBOL TABLE:
[  0](sec  1)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .text
[  1](sec  2)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .data
[  2](sec  3)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .bss
[  3](sec  4)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .idata$7
[  4](sec  5)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .idata$5
[  5](sec  6)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .idata$4
[  6](sec  7)(fl 0x00)(ty   0)(scl   3) (nx 0) 0x00000000 .idata$6
[  7](sec  1)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 _WSACleanup@0
[  8](sec  5)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__WSACleanup@0
[  9](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __head_lib32_libws2_32_a


RELOCATION RECORDS FOR [.text]:
OFFSET   TYPE              VALUE
00000002 dir32             .idata$5


RELOCATION RECORDS FOR [.idata$7]:
OFFSET   TYPE              VALUE
00000000 rva32             __head_lib32_libws2_32_a


RELOCATION RECORDS FOR [.idata$5]:
OFFSET   TYPE              VALUE
00000000 rva32             .idata$6


RELOCATION RECORDS FOR [.idata$4]:
OFFSET   TYPE              VALUE
00000000 rva32             .idata$6

Contents of section .text:
 0000 ff250000 00009090                    .%......
Contents of section .idata$7:
 0000 00000000                             ....
Contents of section .idata$5:
 0000 00000000                             ....
Contents of section .idata$4:
 0000 00000000                             ....
Contents of section .idata$6:
 0000 1a005753 41436c65 616e7570 0000      ..WSACleanup..

The produced binary thus depends on WS2_32.DLL:

$ objdump -x winsock-load-faiure.exe
winsock-load-faiure.exe:     file format pei-i386
winsock-load-faiure.exe
architecture: i386, flags 0x0000013a:
EXEC_P, HAS_DEBUG, HAS_SYMS, HAS_LOCALS, D_PAGED
start address 0x004014d0
[...snip...]
The Import Tables (interpreted .idata section contents)
 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
  000fe000       000fe08c 00000000 00000000 000ff404 000fe404

        DLL Name: KERNEL32.dll
        vma:  Hint/Ord Member-Name Bound-To
        fe77c      69  CloseHandle
[...snip...]
 000fe050       000fe37c 00000000 00000000 000ff68c 000fe6f4

        DLL Name: WS2_32.dll
        vma:  Hint/Ord Member-Name Bound-To
        ff164      26  WSACleanup
[...snip...]
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         000bfab8  00401000  00401000  00000600  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, CODE, DATA
  1 .data         0000b1d4  004c1000  004c1000  000c0200  2**5
                  CONTENTS, ALLOC, LOAD, DATA
  2 .rodata       00014750  004cd000  004cd000  000cb400  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  3 .rdata        000158d4  004e2000  004e2000  000dfc00  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .eh_frame     00000168  004f8000  004f8000  000f5600  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  5 .bss          00004200  004f9000  004f9000  00000000  2**5
                  ALLOC
  6 .idata        000016a8  004fe000  004fe000  000f5800  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  7 .CRT          00000034  00500000  00500000  000f7000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  8 .tls          00000020  00501000  00501000  000f7200  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  9 .rsrc         0000025c  00502000  00502000  000f7400  2**2
                  CONTENTS, ALLOC, LOAD, DATA
[...snip...]
SYMBOL TABLE:
[  0](sec -2)(fl 0x00)(ty   0)(scl 103) (nx 1) 0x0000002d crtexe.c
[...snip...]
[28681](sec  7)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x000006f4 __imp__WSACleanup@0
[...snip...]
[33283](sec -1)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 _WSACleanup@0
[...snip...]

To recap, the final linkage is done by the Windows dynamic loader using the DLL dependency information in the executable import tables. The partially linked object file HSnetwork-2.3.0.14.o which GHCi tries to load for the network package has no such information. In order for GCHi to be able to load such a package, it at least needs to:

be supplied with that information (perhaps via extra-libraries/extra-ghci-libraries fields in the package's .cabal file);
find the MinGW stub libraries the partially linked file depends upon (cf. compiler/ghci/Linker.lhs);
know how to load the stub libraries (currently no .idata sections processing in rts/Linker.c).

Ehm, wasn't thinking straight, there's no need to load the stub libraries, as they're already linked into the partially linked object file. So s/MinGW stub libraries/DLLs/ in (2) and s/stub libraries/DLLs/ in (3).

The confusion source is a similar bug #3242 (closed), where a reference to an undefined symbol _copysignf is found in the partially linked object file HSieee754-0.7.3.o of the ieee754 package. The undefined symbol reference doesn't come directly from a MinGW-w64 stub library, though.

Trac metadata

Trac field	Value
Related	#2283 (closed) → #2283 (closed), #3242 (closed)

Notice that in #3242 (closed) the undefined reference to _copysignf is still expected to be satisfied from a MinGW-w64 stub library named libmingwex.a which happens to implement the actual function, as MSVCRT doesn't. As a result GHCi fails to load the ieee754 package.

So, there're two issues to be resolved:

(for #3242 (closed)) arrange for GHCi to satisfy undefined references to symbols also via MinGW-w64 stub libraries (the list of which to be supplied by package authors via extra-libraries/extra-ghci-libraries);
implement the indirect __imp__* symbols resolution in GHCi.

Could you explain what is special about the WSACleanup symbol here? Why is it different from any other symbol that we import from a DLL? After all, we call a lot of functions that live in DLLs.

Replying to [ticket:7097#comment:63356 simonmar]:

Could you explain what is special about the WSACleanup symbol here? Why is it different from any other symbol that we import from a DLL?

WSACleanup just happens to come prior to other such symbols in the symbol table of the PLOF:

$ objdump -x network-2.3.0.14/ghc-7.4.2/HSnetwork-2.3.0.14.o |grep __imp__
[733](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__WSACleanup@0
[1548](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__accept@12
[1787](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__inet_ntoa@4
[2047](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__GetLastError@0
[2632](sec  0)(fl 0x00)(ty   0)(scl   2) (nx 0) 0x00000000 __imp__WSAStartup@8

After all, we call a lot of functions that live in DLLs.

We only manage to do that for the fully linked by GNU ld executable images, which get loaded into the memory by the Windows dynamic loader, which, in turn, specially processes these 'import' symbols.

It's worth noting that the Windows dynamic linker doesn't actually care how the symbol names are prefixed. What matters is what PE/COFF section the symbols with the prefixes end up after the partial linkage by GNU ld. Cf. "The Imports Section" in "An In-Depth Look into the Win32 Portable Executable File Format, Part 2" and "The .idata Section" in "Microsoft PE and COFF Specification".

Trac metadata

Trac field	Value
Related	#2283 (closed), #3242 (closed) → #1883 (closed), #2283 (closed), #3242 (closed)

Replying to [ticket:7097#comment:63356 simonmar]:

Could you explain what is special about the WSACleanup symbol here? Why is it different from any other symbol that we import from a DLL? After all, we call a lot of functions that live in DLLs.

Rephrasing that made the issue become more apparent, thanks. Why is it that only some packages that import from foreign libraries get references to the __imp_ prefixed symbols and most others don't? Most foreign functionality gets explictly declared and refered to using the Haskell FFI foreign import declarations. Some functionality, though, gets declared and refered to implicitly, via C headers, as in the case with the network package: network-2.3.0.14/Network/Socket/Internal.hsc:

[...snip...]
withSocketsDo :: IO a -> IO a
#if !defined(WITH_WINSOCK)
withSocketsDo x = x
#else
withSocketsDo act = do
    x <- initWinSock
[...snip...]
foreign import ccall unsafe "initWinSock" initWinSock :: IO Int
foreign import ccall unsafe "shutdownWinSock" shutdownWinSock :: IO ()

#endif
[...snip...]

network-2.3.0.14/cbits/initWinSock.c:

#include "HsNet.h"
#include "HsFFI.h"

#if defined(HAVE_WINSOCK2_H) && !defined(__CYGWIN__)
[...snip...]
int
initWinSock ()
{
[...snip...]
    err = WSAStartup ( wVersionRequested, &wsaData );
[...snip...]
      WSACleanup();
[...snip...]

network-2.3.0.14/include/HsNet.h:

[...snip...]
#if defined(HAVE_WINSOCK2_H) && !defined(__CYGWIN__)
#include <winsock2.h>
[...snip...]

i686-w64-mingw32/include/winsock2.h:

[...snip...]
#define WINSOCK_API_LINKAGE     DECLSPEC_IMPORT
[...snip...]
  WINSOCK_API_LINKAGE int WSAAPI WSAStartup(WORD wVersionRequested,LPWSADATA lpWSAData);
  WINSOCK_API_LINKAGE int WSAAPI WSACleanup(void);
[...snip...]

i686-w64-mingw32/include/winnt.h:

[...snip...]
#if defined(_X86_) || defined(__ia64__) || defined(__x86_64)
#define DECLSPEC_IMPORT __declspec(dllimport)
#else
#define DECLSPEC_IMPORT
#endif
[...snip...]

GCC: Declaring Attributes of Functions:

  On Microsoft Windows and Symbian OS targets, the dllimport attribute causes the compiler to reference a function or variable via a global pointer to a pointer that is set up by the DLL exporting the symbol. The attribute implies extern. On Microsoft Windows targets, the pointer name is formed by combining _imp__ and the function or variable name.

The codegen has no control over the symbol name emitted, unlike the explicitly declared imports cases:

$ cabal build network
Building network-2.3.0.14...
[...snip...]
Building C Sources...
creating dist\build
("c:\\mnt\\data1\\ghc32b\\bin\\ghc.exe",["-Idist\\build","-Iinclude","-no-user-package-conf","-package-id","base-4.5.1.0-724fe8b5495aa663f22c79d0a09b2abb","-package-id","bytestring-0.9.2.1-0c74e8abeebb3c3d794fd93f5313ffd8","-package-id","parsec-3.1.3-2f82092583ddf7939f1cf390c7cadc1c","-optc-DCALLCONV=stdcall","-optc-O2","-odir","dist\\build","-v","-c","cbits/initWinSock.c"])
[...snip...]
Linking...
("c:\\mnt\\data1\\ghc32b\\mingw\\bin\\ar.exe",["-r","-v","dist\\build\\libHSnetwork-2.3.0.14.a","dist\\build\\Network.o","dist\\build\\Network\\BSD.o","dist\\build\\Network\\Socket.o","dist\\build\\Network\\Socket\\ByteString.o","dist\\build\\Network\\Socket\\ByteString\\Lazy.o","dist\\build\\Network\\Socket\\Internal.o","dist\\build\\Network\\URI.o","dist\\build\\Network\\Socket\\ByteString\\Internal.o","dist\\build\\cbits/initWinSock.o","dist\\build\\cbits/winSockErr.o","dist\\build\\cbits/asyncAccept.o","dist\\build\\cbits/HsNet.o"])
a - dist\build\Network.o
a - dist\build\Network\BSD.o
a - dist\build\Network\Socket.o
a - dist\build\Network\Socket\ByteString.o
a - dist\build\Network\Socket\ByteString\Lazy.o
a - dist\build\Network\Socket\Internal.o
a - dist\build\Network\URI.o
a - dist\build\Network\Socket\ByteString\Internal.o
a - dist\build\cbits/initWinSock.o
a - dist\build\cbits/winSockErr.o
a - dist\build\cbits/asyncAccept.o
a - dist\build\cbits/HsNet.o
("c:\\mnt\\data1\\ghc32b\\mingw\\bin\\ld.exe",["-x","--hash-size=31","--reduce-memory-overheads","-r","-o","dist\\build\\HSnetwork-2.3.0.14.o","dist\\build\\Network.o","dist\\build\\Network\\BSD.o","dist\\build\\Network\\Socket.o","dist\\build\\Network\\Socket\\ByteString.o","dist\\build\\Network\\Socket\\ByteString\\Lazy.o","dist\\build\\Network\\Socket\\Internal.o","dist\\build\\Network\\URI.o","dist\\build\\Network\\Socket\\ByteString\\Internal.o","dist\\build\\cbits/initWinSock.o","dist\\build\\cbits/winSockErr.o","dist\\build\\cbits/asyncAccept.o","dist\\build\\cbits/HsNet.o"])
[...snip...]

mentioned in issue #7207 (closed)

Trac metadata

Trac field	Value
BlockedBy	→ #3658 (closed)

changed milestone to %7.8.1

changed weight to 7

Trac metadata

Trac field	Value
Priority	normal → high

mentioned in issue #7568 (closed)

Also reported as #7568 (closed) (includes a repro case using hashable).

I'd want to clarify things a bit here.

Suppose we have some compiled dll exporting some function foo. Suppose we write a client program in C calling this foo. Regardless of how this foo is prototyped: as bare foo or as __declspec(dllimport) foo a client program will happily call it, but depending on a prototype in different ways: in the first case this will be call foo and if this symbol is external a linker will generate a thunk for this foo as jmp __imp_foo, in the second case a C compiler always generates call __imp_foo.

Thus the single thing matters here: which prototype of a function was visible to C compiler when it compiled a code calling that function. If that prototype had __declspec(dllimport) attribute attached, a C compiler generated a call against __imp_foo, otherwise - a call against bare foo. GHC FFI of modern era (with no C backend) always generates the second variant AFAIK.

This also explains the difference between 32 and 64 GHCs for Windows. 64-bit GHC is distributed with mingw-w64 headers which use __declspec(dllimport) extensively, and many packages fail to compile. 32-bit GHC is distributed with mingw headers which has almost (or entirely?) no __declspec(dllimport) declared APIs.

To make 64-bit GHC usable on Windows I manually removed all __declspec(dllimport) from mingw-w64 headers and now it all works smoothly.

Hope this helps.

Is this now fixed with the Windows linker fixes in 7.8.1?

Trac metadata

Trac field	Value
CC	bos@serpentine.com, mikesteele81@gmail.com → bos@serpentine.com, mikesteele81@gmail.com, simonmar

closed

Yes, it is.

Trac metadata

Trac field	Value
Resolution	Unresolved → ResolvedFixed
BlockedBy	#3658 (closed) →

mentioned in issue #9218 (closed)

added Phigh label

linker fails to load package with binding to foreign library

Child items 0

Activity

Trac field	Value
Version	7.4.2
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Runtime System
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

linker fails to load package with binding to foreign library

Relates to

Activity