Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#8935 closed bug (fixed)

Obscure linker bug leads to crash in GHCi

Reported by: simonmar Owned by:
Priority: high Milestone: 7.10.1
Component: Runtime System Version: 7.8.1-rc2
Keywords: Cc: igloo, simonmar, jwlato@…, idhameed@…, tkn.akio@…, trommler
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: GHCi crash Test Case:
Blocked By: Blocking:
Related Tickets: #9186, #9480 Differential Rev(s): Phab:D349
Wiki Page:

Description

I have a build of GHC (with DYNAMIC_GHC_PROGRAMS=NO) that exhibits the following crash:

  $ ghc -e 'System.Environment.getEnvironment'
  <segfault>

I tracked it down, eventually, to a bad reference to the symbol environ from __hscore_environ in libraries/base/includes/HsBase.h. Somehow, environ had got linked to the wrong address.

Lots more investigation lead me to discover this: internal_dlsym() in Linker.c tries to look up a symbol in all the different shared libraries we have loaded so far, one by one. (see be497c202b790999c3fd0ddc4a4176b8cf6acf7e). Unfortunately, this seems to break things in my case. Here's a simple test program that works on Ubuntu 12.04:

#include <dlfcn.h>
#include <stdio.h>

char *so = "/usr/lib/x86_64-linux-gnu/libgmp.so";
char *so2 = "/usr/lib/x86_64-linux-gnu/libpthread.so";

extern char**environ;

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  printf("environ = %p\n", &environ);
  printf("dlsym(deflt, \"environ\") = %p\n", dlsym(deflt,"environ"));
  hdl = dlopen(so, RTLD_LAZY);
  printf("dlsym(\"libgmp\", \"environ\") = %p\n", dlsym(hdl,"environ"));
  hdl = dlopen(so2, RTLD_LAZY);
  printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ"));
}

And the output:

environ = 0x601040
dlsym(deflt, "environ") = 0x601040
dlsym("libgmp", "environ") = 0x2aaaab290568
dlsym("libpthread", "environ") = 0x601040

Note that the value we get from looking up environ in libgmp is different to the others. The correct value is 0x601040. gdb thinks that 0x2aaaab290568 is also environ:

(gdb) p4 0x2aaaab290568
0x2aaaab290580 <buflen.9817>:   0x0
0x2aaaab290578: 0x0
0x2aaaab290570 <miss_F_GETOWN_EX>:      0x0
0x2aaaab290568 <environ>:       0x0

but note that it contains zero. The real one is:

(gdb) p4 0x601040
0x601058:       0x0
0x601050 <dtor_idx.6533>:       0x0
0x601048 <completed.6531>:      0x0
0x601040 <environ@@GLIBC_2.2.5>:        0x7fffffffe268

In GHC we're loading libgmp when we load the integer-gmp package, and this causes future references to environ to go wrong.

I've locally fixed this by changing internal_dlsym to dlsym, but since there was a reason to make this change in the first place I haven't pushed it to master. Suggestions welcome.

Attachments (6)

0001-Fix-obscure-linker-bug-with-weak-symbols-part-1.patch (1.1 KB) - added by trommler 4 years ago.
Look up symbols in RTLD_DEFAULT first
0002-Do-not-put-symbols-in-the-global-scope.patch (965 bytes) - added by trommler 4 years ago.
Part two: Put symbols in LOCAL scope.
0003-Remove-redundant-parameter-in-internal_dlsym.patch (1.7 KB) - added by trommler 4 years ago.
Refactor parameter list of internal_dlsym
0004-Do-not-add-NULL-handle-to-list-of-openedSOs.patch (1.2 KB) - added by trommler 4 years ago.
Do not add a NULL handle to SO list for failed dlopen
0001-Fix-GHCi-crash-on-access-to-process-environment.patch (13.3 KB) - added by trommler 3 years ago.
All of the above plus fix for regressions.
0001-Fix-GHCi-linking-of-files-compiled-to-object-code.patch (2.2 KB) - added by trommler 3 years ago.
Patch most likely required for OS X.

Download all attachments as: .zip

Change History (72)

comment:1 Changed 4 years ago by simonmar

Version: 7.6.37.8.1-rc2

comment:2 Changed 4 years ago by trommler

I tried the test program on openSUSE 12.3:

environ = 0x601060
dlsym(deflt, "environ") = 0x601060
dlsym("libgmp", "environ") = 0x601060
dlsym("libpthread", "environ") = 0x601060

Here is my libgmp:

ls /usr/lib64/libgmp.so*
/usr/lib64/libgmp.so  /usr/lib64/libgmp.so.10  /usr/lib64/libgmp.so.10.0.5

IIRC POSIX.1 (but not C!) reserves variable environ. So I'd say it is a bug in Ubuntu's libgmp.

Last edited 4 years ago by trommler (previous) (diff)

comment:3 Changed 4 years ago by thoughtpolice

Milestone: 7.8.27.8.3

comment:4 in reply to:  2 Changed 4 years ago by simonmar

Replying to trommler:

I tried the test program on openSUSE 12.3:

environ = 0x601060
dlsym(deflt, "environ") = 0x601060
dlsym("libgmp", "environ") = 0x601060
dlsym("libpthread", "environ") = 0x601060

Here is my libgmp:

ls /usr/lib64/libgmp.so*
/usr/lib64/libgmp.so  /usr/lib64/libgmp.so.10  /usr/lib64/libgmp.so.10.0.5

IIRC POSIX.1 (but not C!) reserves variable environ. So I'd say it is a bug in Ubuntu's libgmp.

The original distro I encountered the problem on was CentOS. I don't think it is to do with gmp specifically, IIRC there were other libraries that cause the behaviour too (but I'd need to go back and check again).

My partial theory is that because environ is a symbol with R_COPY relocation, the linker is confused and has returned the original location (before the contents were copied and the symbol relocated).

comment:5 Changed 4 years ago by dagit

I haven't tested if I have the crash on my system, but suspect you're seeing something to do with weak symbols. On my system, Fedora 19, I see that glibc defines environ as a weak symbol and gives it the same address that your libgmp check reports.

I modified your C program slightly so that it takes the paths in argv[1] and argv[2]:

$ ./check-environ /usr/lib64/libgmp.so /usr/lib64/libpthread.so
environ = 0x601058
dlsym(deflt, "environ") = 0x601058
dlsym("libgmp", "environ") = 0x31e45bd508
dlsym("libpthread", "environ") = 0x601058

My libgmp doesn't define environ so it must find it in libc, and indeed:

$ readelf -Wa /usr/lib64/libc.so.6 | grep environ
00000031e45b9dd8  0000011900000006 R_X86_64_GLOB_DAT      00000031e45bd508 _environ + 0
00000031e45b9ea0  0000050000000006 R_X86_64_GLOB_DAT      00000031e45bd508 __environ + 0
   281: 00000031e45bd508     8 OBJECT  WEAK   DEFAULT   33 _environ@@GLIBC_2.2.5
   956: 00000031e45bd508     8 OBJECT  WEAK   DEFAULT   33 environ@@GLIBC_2.2.5
  1280: 00000031e45bd508     8 OBJECT  GLOBAL DEFAULT   33 __environ@@GLIBC_2.2.5
   349: 00000031e45bbe70     8 OBJECT  LOCAL  DEFAULT   33 last_environ
  1813: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS environ.c
  4539: 00000031e4238a10  1122 FUNC    LOCAL  DEFAULT   12 __add_to_environ
  5707: 00000031e45bd508     8 OBJECT  WEAK   DEFAULT   33 _environ
  6374: 00000031e45bd508     8 OBJECT  GLOBAL DEFAULT   33 __environ
  6483: 00000031e45bd508     8 OBJECT  WEAK   DEFAULT   33 environ

I suspect that the way you're searching for the symbol needs to look for a (the?) strong binding first. I hope that helps.

comment:6 Changed 4 years ago by dagit

I just realized my previous reply probably leaves people wondering: Since libpthread also DOES NOT define environ why does it give the right address?

The difference is that libpthread depends on ld-linux.so as a shared object dependency (grep the readelf -Wa output for NEEDED) and ld-linux.so defines a LOCAL symbol for environ. As I understand it, that means the dynamic linker thinks libpthread defines environ as LOCAL (so it goes to look for a different definition and ends up with the right one). For libgmp it looks in libc.so and finds a WEAK definition and uses it because there is no other GLOBAL definition.

To help demonstrate this, consider this example. On my system, libz.so has dependencies similar to libgmp.so:

$ readelf -Wa /usr/lib64/libgmp.so.10 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
$ readelf -Wa /usr/lib64/libz.so.1 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

And look at this:

$ ./check-environ /usr/lib64/libgmp.so /usr/lib64/libz.so.1
environ = 0x601058
dlsym(deflt, "environ") = 0x601058
dlsym("libgmp", "environ") = 0x31e45bd508
dlsym("libpthread", "environ") = 0x31e45bd508

I'm still fuzzy on some of the details, but it seems that WEAK and LOCAL mean very different things to the dynamic linker. WEAK is the same as GLOBAL but the link priority is lower, while LOCAL means the symbol should not be exported. As far as I can tell, when it finds LOCAL it continues the search whereas with WEAK it stops earlier because technically it has found an exported symbol.

comment:7 Changed 4 years ago by jwlato

Cc: jwlato@… added

comment:8 Changed 4 years ago by ihameed

Cc: idhameed@… added

comment:9 Changed 4 years ago by akio

Cc: tkn.akio@… added

comment:10 in reply to:  6 Changed 4 years ago by trommler

Replying to dagit:

The difference is that libpthread depends on ld-linux.so as a shared object dependency (grep the readelf -Wa output for NEEDED) and ld-linux.so defines a LOCAL symbol for environ. As I understand it, that means the dynamic linker thinks libpthread defines environ as LOCAL (so it goes to look for a different definition and ends up with the right one). For libgmp it looks in libc.so and finds a WEAK definition and uses it because there is no other GLOBAL definition.

To help demonstrate this, consider this example. On my system, libz.so has dependencies similar to libgmp.so:

$ readelf -Wa /usr/lib64/libgmp.so.10 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
$ readelf -Wa /usr/lib64/libz.so.1 | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

I'm still fuzzy on some of the details, but it seems that WEAK and LOCAL mean very different things to the dynamic linker. WEAK is the same as GLOBAL but the link priority is lower, while LOCAL means the symbol should not be exported. As far as I can tell, when it finds LOCAL it continues the search whereas with WEAK it stops earlier because technically it has found an exported symbol.

I checked on openSUSE 13.1 (x86_64 and powerpc64) and I have the same NEEDED for libc.so.

environ is a WEAK symbol in my libc.so, too.

Here is my libc.so.6 on powerpc64:

00000000001d7ce0  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d7d00  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d7ee8  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d7ef0  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d8018  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d84a0  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d84a8  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d84b0  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d8570  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d8928  0000056600000026 R_PPC64_ADDR64         00000000001da1f0 __environ + 0
00000000001d81e8  0000013100000026 R_PPC64_ADDR64         00000000001da1f0 _environ + 0
   305: 00000000001da1f0     8 OBJECT  WEAK   DEFAULT   32 _environ@@GLIBC_2.3
  1026: 00000000001da1f0     8 OBJECT  WEAK   DEFAULT   32 environ@@GLIBC_2.3
  1382: 00000000001da1f0     8 OBJECT  GLOBAL DEFAULT   32 __environ@@GLIBC_2.3
   178: 00000000001d8fa8     8 OBJECT  LOCAL  DEFAULT   32 last_environ
  2904: 00000000001c9628  1436 FUNC    LOCAL  DEFAULT   28 __add_to_environ
  4138: 00000000001da1f0     8 OBJECT  WEAK   DEFAULT   32 _environ
  4843: 00000000001da1f0     8 OBJECT  GLOBAL DEFAULT   32 __environ
  4957: 00000000001da1f0     8 OBJECT  WEAK   DEFAULT   32 environ

On x86_64 it looks similar to comment 5 except the line with environ.c is missing.

Do openSUSE patch their system linker? I'll go ask on their mailing list and report back.

comment:11 Changed 4 years ago by trommler

Status: newinfoneeded

Could you post the output of LD_DEBUG=symbols ./check-environ

comment:12 Changed 4 years ago by dagit

I hope this helps! This is from the Fedora 19 system I mentioned earlier.

$ LD_DEBUG=symbols ./check-environ /usr/lib64/libgmp.so /usr/lib64/libpthread.so
      5823:	symbol=_res;  lookup in file=./check-environ [0]
      5823:	symbol=_res;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_res;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=stderr;  lookup in file=./check-environ [0]
      5823:	symbol=stderr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stderr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=error_one_per_line;  lookup in file=./check-environ [0]
      5823:	symbol=error_one_per_line;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=error_one_per_line;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__malloc_initialize_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__malloc_initialize_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__malloc_initialize_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_dl_starting_up;  lookup in file=./check-environ [0]
      5823:	symbol=_dl_starting_up;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_dl_starting_up;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_dl_starting_up;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__morecore;  lookup in file=./check-environ [0]
      5823:	symbol=__morecore;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__morecore;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__key_encryptsession_pk_LOCAL;  lookup in file=./check-environ [0]
      5823:	symbol=__key_encryptsession_pk_LOCAL;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__key_encryptsession_pk_LOCAL;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__progname_full;  lookup in file=./check-environ [0]
      5823:	symbol=__progname_full;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__progname_full;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__ctype32_tolower;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype32_tolower;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype32_tolower;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_environ;  lookup in file=./check-environ [0]
      5823:	symbol=_rtld_global;  lookup in file=./check-environ [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__progname;  lookup in file=./check-environ [0]
      5823:	symbol=__progname;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__progname;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=argp_err_exit_status;  lookup in file=./check-environ [0]
      5823:	symbol=argp_err_exit_status;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=argp_err_exit_status;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=mallwatch;  lookup in file=./check-environ [0]
      5823:	symbol=mallwatch;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=mallwatch;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__rcmd_errstr;  lookup in file=./check-environ [0]
      5823:	symbol=__rcmd_errstr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__rcmd_errstr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__vdso_clock_gettime;  lookup in file=./check-environ [0]
      5823:	symbol=__vdso_clock_gettime;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__vdso_clock_gettime;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=svcauthdes_stats;  lookup in file=./check-environ [0]
      5823:	symbol=svcauthdes_stats;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=svcauthdes_stats;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__libc_enable_secure;  lookup in file=./check-environ [0]
      5823:	symbol=__libc_enable_secure;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__libc_enable_secure;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__libc_enable_secure;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_res_hconf;  lookup in file=./check-environ [0]
      5823:	symbol=_res_hconf;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_res_hconf;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=malloc;  lookup in file=./check-environ [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=getdate_err;  lookup in file=./check-environ [0]
      5823:	symbol=getdate_err;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=getdate_err;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__tzname;  lookup in file=./check-environ [0]
      5823:	symbol=__tzname;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__tzname;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__timezone;  lookup in file=./check-environ [0]
      5823:	symbol=__timezone;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__timezone;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=./check-environ [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=optarg;  lookup in file=./check-environ [0]
      5823:	symbol=optarg;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=optarg;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__ctype_tolower;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype_tolower;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype_tolower;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__ctype_toupper;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype_toupper;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype_toupper;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=svc_max_pollfd;  lookup in file=./check-environ [0]
      5823:	symbol=svc_max_pollfd;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=svc_max_pollfd;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__ctype_b;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype_b;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype_b;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=argp_program_version_hook;  lookup in file=./check-environ [0]
      5823:	symbol=argp_program_version_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=argp_program_version_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__after_morecore_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__after_morecore_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__after_morecore_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__environ;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype32_b;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype32_b;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype32_b;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__curbrk;  lookup in file=./check-environ [0]
      5823:	symbol=__curbrk;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__curbrk;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=argp_program_version;  lookup in file=./check-environ [0]
      5823:	symbol=argp_program_version;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=argp_program_version;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__daylight;  lookup in file=./check-environ [0]
      5823:	symbol=__daylight;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__daylight;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__memalign_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__memalign_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__memalign_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__malloc_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__malloc_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__malloc_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__free_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__free_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__free_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=svc_pollfd;  lookup in file=./check-environ [0]
      5823:	symbol=svc_pollfd;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=svc_pollfd;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_nl_domain_bindings;  lookup in file=./check-environ [0]
      5823:	symbol=_nl_domain_bindings;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_nl_domain_bindings;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_nl_msg_cat_cntr;  lookup in file=./check-environ [0]
      5823:	symbol=_nl_msg_cat_cntr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_nl_msg_cat_cntr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=argp_program_bug_address;  lookup in file=./check-environ [0]
      5823:	symbol=argp_program_bug_address;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=argp_program_bug_address;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__key_decryptsession_pk_LOCAL;  lookup in file=./check-environ [0]
      5823:	symbol=__key_decryptsession_pk_LOCAL;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__key_decryptsession_pk_LOCAL;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=h_errlist;  lookup in file=./check-environ [0]
      5823:	symbol=h_errlist;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=h_errlist;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=program_invocation_short_name;  lookup in file=./check-environ [0]
      5823:	symbol=program_invocation_short_name;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=program_invocation_short_name;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=optind;  lookup in file=./check-environ [0]
      5823:	symbol=optind;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=optind;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=stdout;  lookup in file=./check-environ [0]
      5823:	symbol=stdout;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stdout;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=obstack_alloc_failed_handler;  lookup in file=./check-environ [0]
      5823:	symbol=obstack_alloc_failed_handler;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=obstack_alloc_failed_handler;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=error_print_progname;  lookup in file=./check-environ [0]
      5823:	symbol=error_print_progname;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=error_print_progname;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=optopt;  lookup in file=./check-environ [0]
      5823:	symbol=optopt;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=optopt;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_IO_funlockfile;  lookup in file=./check-environ [0]
      5823:	symbol=_IO_funlockfile;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_IO_funlockfile;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=obstack_exit_failure;  lookup in file=./check-environ [0]
      5823:	symbol=obstack_exit_failure;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=obstack_exit_failure;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=error_message_count;  lookup in file=./check-environ [0]
      5823:	symbol=error_message_count;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=error_message_count;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=svc_fdset;  lookup in file=./check-environ [0]
      5823:	symbol=svc_fdset;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=svc_fdset;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=program_invocation_name;  lookup in file=./check-environ [0]
      5823:	symbol=program_invocation_name;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=program_invocation_name;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=loc1;  lookup in file=./check-environ [0]
      5823:	symbol=loc1;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=loc1;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=free;  lookup in file=./check-environ [0]
      5823:	symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=loc2;  lookup in file=./check-environ [0]
      5823:	symbol=loc2;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=loc2;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__fpu_control;  lookup in file=./check-environ [0]
      5823:	symbol=__fpu_control;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__fpu_control;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=re_syntax_options;  lookup in file=./check-environ [0]
      5823:	symbol=re_syntax_options;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=re_syntax_options;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=stdin;  lookup in file=./check-environ [0]
      5823:	symbol=stdin;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stdin;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__check_rhosts_file;  lookup in file=./check-environ [0]
      5823:	symbol=__check_rhosts_file;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__check_rhosts_file;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=opterr;  lookup in file=./check-environ [0]
      5823:	symbol=opterr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=opterr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__ctype32_toupper;  lookup in file=./check-environ [0]
      5823:	symbol=__ctype32_toupper;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__ctype32_toupper;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__realloc_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__realloc_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__realloc_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_dl_argv;  lookup in file=./check-environ [0]
      5823:	symbol=_dl_argv;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_dl_argv;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_dl_argv;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=rpc_createerr;  lookup in file=./check-environ [0]
      5823:	symbol=rpc_createerr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=rpc_createerr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_IO_2_1_stderr_;  lookup in file=./check-environ [0]
      5823:	symbol=_IO_2_1_stderr_;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_IO_2_1_stderr_;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_IO_2_1_stdout_;  lookup in file=./check-environ [0]
      5823:	symbol=_IO_2_1_stdout_;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_IO_2_1_stdout_;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_IO_2_1_stdin_;  lookup in file=./check-environ [0]
      5823:	symbol=_IO_2_1_stdin_;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_IO_2_1_stdin_;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=realloc;  lookup in file=./check-environ [0]
      5823:	symbol=realloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=realloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=memset;  lookup in file=./check-environ [0]
      5823:	symbol=memset;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=memset;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=malloc;  lookup in file=./check-environ [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=./check-environ [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=memmove;  lookup in file=./check-environ [0]
      5823:	symbol=memmove;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=memmove;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=memalign;  lookup in file=./check-environ [0]
      5823:	symbol=memalign;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=memalign;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=calloc;  lookup in file=./check-environ [0]
      5823:	symbol=calloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=calloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=free;  lookup in file=./check-environ [0]
      5823:	symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__vdso_time;  lookup in file=linux-vdso.so.1 [0]
      5823:	symbol=__vdso_gettimeofday;  lookup in file=linux-vdso.so.1 [0]
      5823:	symbol=__pthread_key_create;  lookup in file=./check-environ [0]
      5823:	symbol=__pthread_key_create;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__pthread_key_create;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__pthread_key_create;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=./check-environ [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__pthread_getspecific;  lookup in file=./check-environ [0]
      5823:	symbol=__pthread_getspecific;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__pthread_getspecific;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__pthread_getspecific;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmon_start__;  lookup in file=./check-environ [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=./check-environ [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_rtld_global_ro;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__pthread_once;  lookup in file=./check-environ [0]
      5823:	symbol=__pthread_once;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__pthread_once;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__pthread_once;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_libc_intl_domainname;  lookup in file=./check-environ [0]
      5823:	symbol=_libc_intl_domainname;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_libc_intl_domainname;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=./check-environ [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__environ;  lookup in file=./check-environ [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=./check-environ [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__pthread_setspecific;  lookup in file=./check-environ [0]
      5823:	symbol=__pthread_setspecific;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__pthread_setspecific;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__pthread_setspecific;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_rtld_global;  lookup in file=./check-environ [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_rtld_global;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=./check-environ [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmon_start__;  lookup in file=./check-environ [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__environ;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__environ;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_r_debug;  lookup in file=./check-environ [0]
      5823:	symbol=_r_debug;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_r_debug;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_r_debug;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=free;  lookup in file=./check-environ [0]
      5823:	symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__libc_memalign;  lookup in file=./check-environ [0]
      5823:	symbol=__libc_memalign;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__libc_memalign;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=malloc;  lookup in file=./check-environ [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=malloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=./check-environ [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__tls_get_addr;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=calloc;  lookup in file=./check-environ [0]
      5823:	symbol=calloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=calloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=realloc;  lookup in file=./check-environ [0]
      5823:	symbol=realloc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=realloc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=free;  lookup in file=./check-environ [0]
      5823:	symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	
      5823:	calling init: /lib64/ld-linux-x86-64.so.2
      5823:	
      5823:	
      5823:	calling init: /lib64/libc.so.6
      5823:	
      5823:	symbol=__vdso_clock_gettime;  lookup in file=linux-vdso.so.1 [0]
      5823:	symbol=__vdso_getcpu;  lookup in file=linux-vdso.so.1 [0]
      5823:	
      5823:	calling init: /lib64/libdl.so.2
      5823:	
      5823:	symbol=__libc_start_main;  lookup in file=./check-environ [0]
      5823:	symbol=__libc_start_main;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__libc_start_main;  lookup in file=/lib64/libc.so.6 [0]
      5823:	
      5823:	initialize program: ./check-environ
      5823:	
      5823:	
      5823:	transferring control: ./check-environ
      5823:	
      5823:	symbol=dlopen;  lookup in file=./check-environ [0]
      5823:	symbol=dlopen;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=printf;  lookup in file=./check-environ [0]
      5823:	symbol=printf;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=printf;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=dlsym;  lookup in file=./check-environ [0]
      5823:	symbol=dlsym;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_dl_sym;  lookup in file=./check-environ [0]
      5823:	symbol=_dl_sym;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_dl_sym;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=environ;  lookup in file=./check-environ [0]
      5823:	symbol=obstack_vprintf;  lookup in file=./check-environ [0]
      5823:	symbol=obstack_vprintf;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=obstack_vprintf;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=vfprintf;  lookup in file=./check-environ [0]
      5823:	symbol=vfprintf;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=vfprintf;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_asprintf_memory;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_asprintf_memory;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_asprintf_memory;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_asprintf_memory;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_asprintf_memory;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_asprintf_reps;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_asprintf_reps;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_asprintf_reps;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_asprintf_reps;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_asprintf_reps;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_asprintf_final;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_asprintf_final;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_asprintf_final;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_asprintf_final;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_asprintf_final;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__isoc99_fscanf;  lookup in file=./check-environ [0]
      5823:	symbol=__isoc99_fscanf;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__isoc99_fscanf;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=fgetc;  lookup in file=./check-environ [0]
      5823:	symbol=fgetc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=fgetc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=ungetc;  lookup in file=./check-environ [0]
      5823:	symbol=ungetc;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=ungetc;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_randget_mt;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_randget_mt;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_randget_mt;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_randget_mt;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_randget_mt;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_randclear_mt;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_randclear_mt;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_randclear_mt;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_randclear_mt;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_randclear_mt;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_randiset_mt;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_randiset_mt;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_randiset_mt;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_randiset_mt;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_randiset_mt;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=./check-environ [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=stdout;  lookup in file=./check-environ [0]
      5823:	symbol=stdout;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stdout;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_reallocate_func;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_reallocate_func;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_reallocate_func;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_reallocate_func;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_reallocate_func;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=stdin;  lookup in file=./check-environ [0]
      5823:	symbol=stdin;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stdin;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fib_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_fib_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_fib_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fib_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_fib_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_fprintf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_fprintf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_fprintf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fprintf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_fprintf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmpz_sub;  lookup in file=./check-environ [0]
      5823:	symbol=__gmpz_sub;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmpz_sub;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmpz_sub;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmpz_sub;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_allocate_func;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_allocate_func;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_allocate_func;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_allocate_func;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_allocate_func;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_oddfac_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_oddfac_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_oddfac_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_oddfac_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_oddfac_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_junk;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_junk;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_junk;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_junk;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_junk;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_0;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_0;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_0;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_0;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_0;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_asprintf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_asprintf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_asprintf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_asprintf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_asprintf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_jacobi_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_jacobi_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_jacobi_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_jacobi_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_jacobi_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_limbroots_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_limbroots_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_limbroots_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_limbroots_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_limbroots_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_errno;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_errno;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_errno;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_errno;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_errno;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmon_start__;  lookup in file=./check-environ [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_reallocate;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_default_reallocate;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_default_reallocate;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_default_reallocate;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_reallocate;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_digit_value_tab;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_digit_value_tab;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_digit_value_tab;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_digit_value_tab;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_digit_value_tab;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_odd2fac_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_odd2fac_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_odd2fac_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_odd2fac_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_odd2fac_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmpn_bases;  lookup in file=./check-environ [0]
      5823:	symbol=__gmpn_bases;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmpn_bases;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmpn_bases;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmpn_bases;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_sprintf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_sprintf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_sprintf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_sprintf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_sprintf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_binvert_limb_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_binvert_limb_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_binvert_limb_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_binvert_limb_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_binvert_limb_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_default_fp_limb_precision;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_default_fp_limb_precision;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_default_fp_limb_precision;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_default_fp_limb_precision;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_fp_limb_precision;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmpn_gcdext_hook;  lookup in file=./check-environ [0]
      5823:	symbol=__gmpn_gcdext_hook;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmpn_gcdext_hook;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmpn_gcdext_hook;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmpn_gcdext_hook;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_rands;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_rands;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_rands;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_rands;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_rands;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_sscanf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_sscanf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_sscanf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_sscanf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_sscanf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_fac2cnt_table;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_fac2cnt_table;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_fac2cnt_table;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fac2cnt_table;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_fac2cnt_table;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmpz_add;  lookup in file=./check-environ [0]
      5823:	symbol=__gmpz_add;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmpz_add;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmpz_add;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmpz_add;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_default_allocate;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_default_allocate;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_default_allocate;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_default_allocate;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_allocate;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=./check-environ [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_Jv_RegisterClasses;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_free_func;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_free_func;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_free_func;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_free_func;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_free_func;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_snprintf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_snprintf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_snprintf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_snprintf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_snprintf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_default_free;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_rands_initialized;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_rands_initialized;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_rands_initialized;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_rands_initialized;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_rands_initialized;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=__gmp_obstack_printf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_obstack_printf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_obstack_printf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_obstack_printf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_obstack_printf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=./check-environ [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=./check-environ [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=stderr;  lookup in file=./check-environ [0]
      5823:	symbol=stderr;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=stderr;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fscanf_funs;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_fscanf_funs;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_fscanf_funs;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_fscanf_funs;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_fscanf_funs;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	
      5823:	calling init: /usr/lib64/libgmp.so
      5823:	
      5823:	symbol=environ;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=environ;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=free;  lookup in file=./check-environ [0]
      5823:	symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=environ;  lookup in file=./check-environ [0]
      5823:	
      5823:	calling fini: ./check-environ [0]
      5823:	
      5823:	
      5823:	calling fini: /lib64/libdl.so.2 [0]
      5823:	
      5823:	symbol=__cxa_finalize;  lookup in file=./check-environ [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libc.so.6 [0]
      5823:	
      5823:	calling fini: /usr/lib64/libgmp.so [0]
      5823:	
      5823:	symbol=__cxa_finalize;  lookup in file=./check-environ [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__cxa_finalize;  lookup in file=/lib64/libc.so.6 [0]
environ = 0x601058
dlsym(deflt, "environ") = 0x601058
dlsym("libgmp", "environ") = 0x31e45bd508
dlsym("libpthread", "environ") = 0x601058

comment:13 in reply to:  12 Changed 4 years ago by trommler

Replying to dagit: Great, thanks!

      5823:	symbol=environ;  lookup in file=/usr/lib64/libgmp.so [0]
      5823:	symbol=environ;  lookup in file=/lib64/libc.so.6 [0]

That looks like we have found the reason, why openSUSE behaves differently here: On openSUSE libgmp.so is never checked for environ because ld.so looks in check-environ first. check-environ has symbol environ, because of the extern char** environ; in line 7.

According to gABI http://refspecs.linuxbase.org/elf/gabi4+/ch5.dynamic.html this is the default behavior. To override that one can set set the flag DF_SYMBOLIC in the shared library's header and I suspect Fedora does that.

The behavior is defined as follows in gABI:

DF_SYMBOLIC

If this flag is set in a shared object library, the dynamic linker's symbol resolution algorithm for references within the library is changed. Instead of starting a symbol search with the executable file, the dynamic linker starts from the shared object itself. If the shared object fails to supply the referenced symbol, the dynamic linker then searches the executable file and other shared objects as usual.

Could you post the output of readelf -h /usr/lib64/libgmp.so.10 to confirm. Replace 10 with the version on your system.

comment:14 Changed 4 years ago by dagit

$ readelf -h /usr/lib64/libgmp.so.10
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x31fba0c080
  Start of program headers:          64 (bytes into file)
  Start of section headers:          488144 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         31
  Section header string table index: 30

comment:15 in reply to:  14 Changed 4 years ago by trommler

Replying to dagit: My bad! The flags we need to check are in the dynamic section and not in the ELF header. Could you run readelf -d /usr/lib64/libgmp.so.10 instead.

comment:16 Changed 4 years ago by dagit

Hmm...I still don't see the flag you're looking for.

$ readelf -d /usr/lib64/libgmp.so.10

Dynamic section at offset 0x6dce0 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libgmp.so.10]
 0x000000000000000c (INIT)               0x31fba0aab8
 0x000000000000000d (FINI)               0x31fba5e5f0
 0x0000000000000019 (INIT_ARRAY)         0x31fbc6d8e0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x31fbc6d8e8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x31fba001f0
 0x0000000000000005 (STRTAB)             0x31fba05170
 0x0000000000000006 (SYMTAB)             0x31fba01570
 0x000000000000000a (STRSZ)              10633 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x31fbc6e000
 0x0000000000000002 (PLTRELSZ)           8232 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x31fba08a90
 0x0000000000000007 (RELA)               0x31fba08070
 0x0000000000000008 (RELASZ)             2592 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x31fba08000
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x31fba07afa
 0x000000006ffffff9 (RELACOUNT)          52
 0x000000006ffffdf8 (CHECKSUM)           0xcc8edcb1
 0x000000006ffffdf5 (GNU_PRELINKED)      2013-12-20T11:44:15
 0x0000000000000000 (NULL)               0x0

comment:17 Changed 4 years ago by simonmar

libgmp is not symbolic on my system either.

Symbol information and relocs for the program:

$ objdump --syms --dynamic-syms --reloc --dynamic-reloc ./a.out | grep envir                   
0000000000000000 l    df *ABS*  0000000000000000              check-environ.c
0000000000601060  w    O .bss   0000000000000008              environ@@GLIBC_2.2.5
0000000000601060 g     O .bss   0000000000000008              __environ@@GLIBC_2.2.5
0000000000601060  w   DO .bss   0000000000000008  GLIBC_2.2.5 _environ
0000000000601060  w   DO .bss   0000000000000008  GLIBC_2.2.5 environ
0000000000601060 g    DO .bss   0000000000000008  GLIBC_2.2.5 __environ
0000000000601060 R_X86_64_COPY     __environ

Note there are 3 symbols at the same location (environ, _environ, and __environ). The first two are weak, the third is global. The third is also a copy reloc, which means its contents is copied to this location when the program starts up.

Symbol information and relocs for libc:

$ objdump --syms --dynamic-syms --reloc --dynamic-reloc /lib/x86_64-linux-gnu/libc-2.17.so | grep envir    
00000000003c44e8  w   DO .bss   0000000000000008  GLIBC_2.2.5 _environ
00000000003c44e8  w   DO .bss   0000000000000008  GLIBC_2.2.5 environ
00000000003c44e8 g    DO .bss   0000000000000008  GLIBC_2.2.5 __environ
00000000003c0de0 R_X86_64_GLOB_DAT  _environ
00000000003c0ea8 R_X86_64_GLOB_DAT  __environ

Again we have 3 symbols at the same location, all 8 bytes long.

There are no mentions of any environ symbol in libgmp's symbol table or relocations.

LD_DEBUG=all shows a normal lookup like this:

     23419:	symbol=environ;  lookup in file=./a.out [0]
     23419:	binding file ./a.out [0] to ./a.out [0]: normal symbol `environ'

but the lookup in libgmp looks like this:

     23419:	symbol=environ;  lookup in file=/usr/lib/x86_64-linux-gnu/libgmp.so [0]
     23419:	symbol=environ;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
     23419:	binding file /usr/lib/x86_64-linux-gnu/libgmp.so [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `environ'

I still have no idea why the lookup in libgmp is behaving differently from the others.

comment:18 Changed 4 years ago by dagit

Simon, did you read my explanation above about LOCAL vs. GLOBAL?

comment:19 Changed 4 years ago by simonmar

I read it, but I don't think I understand the difference between local and global. So do you have an explanation for what we're seeing? The impression I got from what you wrote above is that there was still some guesswork involved.

comment:20 Changed 4 years ago by simonmar

Also, on my system ld-linux.so doesn't have a symbol for environ, local or otherwise.

comment:21 Changed 4 years ago by dagit

Simon, at the moment I'm just trying to understand this whole thing better myself. My uncertainty is in these areas:

  • How the dlsym() search should work. For example, I'm not sure if dlsym() is supposed to look in the NEEDED dependencies or not. It seems to look in them.
  • Whether glibc implements the above search correctly.
  • Finally, I have no idea what is the right way to fix ghc's linker.

I assume you're already familiar with the elf linking spec, but here is the link just to make sure we're looking at the same doc: http://refspecs.linuxbase.org/elf/elf.pdf

It says:

  • STB_LOCAL Local symbols are not visible outside the object file containing their definition. Local symbols of the same name may exist in multiple files without interfering with each other.
  • STB_GLOBAL Global symbols are visible to all object files being combined. One file's definition of a global symbol will satisfy another file's undefined reference to the same global symbol.
  • STB_WEAK Weak symbols resemble global symbols, but their definitions have lower precedence.

I'd say, based on what I see on my machine that WEAK seems to be even lower priority than LOCAL.

It's interesting that your ld-linux.so doesn't have environ, when I grep mine I get these hits:

$ readelf -Wa /usr/lib64/ld-linux-x86-64.so.2 | grep environ
   143: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS dl-environ.c
   179: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS environ.c
   268: 00000031e4022168     8 OBJECT  LOCAL  DEFAULT   21 _environ
   275: 00000031e4022168     8 OBJECT  LOCAL  DEFAULT   21 __environ
   300: 00000031e4022168     8 OBJECT  LOCAL  DEFAULT   21 environ

For your libgmp.so does environ get defined in any of the deps listed by ldd?

The dlsym manpage has this (potentially) interesting comment:

External references in the library are resolved using the libraries in that library's dependency list and any other libraries previously opened with the RTLD_GLOBAL flag. If the executable was linked with the flag "-rdynamic" (or, synonymously, "--export-dynamic"), then the global symbols in the executable will also be used to resolve references in a dynamically loaded library.

Please let me know if there is more I can do to help.

comment:22 Changed 4 years ago by trommler

Let's look at comment:12 again. We see that even for symbols defined in libgmp.so the dynamic linker searches ./check-environ (and libdl and libc and ld-linux) before it finally finds the symbol in libgmp.so

      5823:	symbol=__gmp_default_free;  lookup in file=./check-environ [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/libdl.so.2 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/libc.so.6 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
      5823:	symbol=__gmp_default_free;  lookup in file=/usr/lib64/libgmp.so [0]

And then for environ the search starts in libgmp.so. Perhaps because environ is a weak symbol.

It seems that openSUSE ignores the weak attribute of symbols in the dynamic linker and treats them as strong and hence starts the search in ./check-environ. This seems to be correct as Drepper http://www.akkadia.org/drepper/dsohowto.pdf says on p. 6: "Note that a definition in a DSO being weak has no effects. Weak definitions only play a role in static linking." It seems Ubuntu and Fedora do not agree.

There is also an environment variable LD_DYNAMIC_WEAK (see ld.so(8)) to allow overriding weak symbols. Is that environment variable set on your systems?

comment:23 Changed 4 years ago by dagit

The value of LD_DYNAMIC_WEAK doesn't make a difference on my system. I've also tried printing the result of dlerror() before/after calls to dlsym().

comment:24 in reply to:  23 Changed 4 years ago by trommler

Replying to dagit:

The value of LD_DYNAMIC_WEAK doesn't make a difference on my system. I've also tried printing the result of dlerror() before/after calls to dlsym().

What happens if there is no LD_DYNAMIC_WEAK in the environment?

comment:25 Changed 4 years ago by trommler

Arrrg! Now I see what I was doing wrong! The libraries were specified with paths that do not exist on openSUSE. With this program I can reproduce the problem:

#include <dlfcn.h>
#include <stdio.h>

char *so = "libgmp.so";
char *so2 = "libpthread.so";

extern char**environ;

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  printf("&environ = %p, environ = %p\n", &environ, environ);
  char ***dflt_env = dlsym(deflt,"environ");
  printf("dlsym(deflt, \"environ\") = %p, *dflt_env = %p\n", dflt_env, *dflt_env );
  hdl = dlopen(so, RTLD_LAZY);
  char ***env = dlsym(hdl,"environ");
  printf("dlsym(\"libgmp\", \"environ\") = %p\n", env);
  printf("*env = %p\n", *env);
  hdl = dlopen(so2, RTLD_LAZY);
  printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ"));
}

I also added code that shows the content of the pointer returned from dlsym called with libgmp.so.

peter@montebre:~/tmp> ./a.out
&environ = 0x601060, environ = 0x7fffc82ce4b8
dlsym(deflt, "environ") = 0x601060, *dflt_env = 0x7fffc82ce4b8
dlsym("libgmp", "environ") = 0x7f9dcb870fd8
*env = (nil)
dlsym("libpthread", "environ") = 0x601060

Now, playing some more, I removed extern char **environ; and the print statements and the NULL pointer issue is gone:

#include <dlfcn.h>
#include <stdio.h>

char *so = "libgmp.so";
char *so2 = "libpthread.so";

// extern char**environ;

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  //  printf("&environ = %p, environ = %p\n", &environ, environ);
  char ***dflt_env = dlsym(deflt,"environ");
  printf("dlsym(deflt, \"environ\") = %p, *dflt_env = %p\n", dflt_env, *dflt_env );
  hdl = dlopen(so, RTLD_LAZY);
  char ***env = dlsym(hdl,"environ");
  printf("dlsym(\"libgmp\", \"environ\") = %p\n", env);
  printf("*env = %p\n", *env);
  hdl = dlopen(so2, RTLD_LAZY);
  printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ"));
}
peter@montebre:~/tmp> ./a.out
dlsym(deflt, "environ") = 0x7f4a0181bfd8, *dflt_env = 0x7fff2d021848
dlsym("libgmp", "environ") = 0x7f4a0181bfd8
*env = 0x7fff2d021848
dlsym("libpthread", "environ") = 0x7f4a0181bfd8

comment:26 Changed 4 years ago by trommler

Adding error handling:

  hdl = dlopen(so2, RTLD_LAZY);
  if (hdl == NULL) {
    printf("failed to open libpthread: %s\n", dlerror());
    return 1;
  }
  printf("dlsym(\"libpthread\", \"environ\") = %p\n", dlsym(hdl,"environ"));

I get:

&environ = 0x601068, environ = 0x7fffd170d688
dlsym(deflt, "environ") = 0x601068, *dflt_env = 0x7fffd170d688
dlsym("libgmp", "environ") = 0x7fcc61a82fd8
*env = (nil)
failed to open libpthread: /usr/lib64/libpthread.so: invalid ELF header

libgmp.so is not special after all. On openSUSE libpthread.so is a linker script. dlopen() cannot open it and returns NULL. RTLD_DEFAULT is defined as (void *) 0 so NULL so dlsym did not fail but return the symbol from the default SO (the program itself).

Trying libpthread.so.0 instead shows the same problem as libgmp.so.

comment:27 Changed 4 years ago by trommler

I have the following fix in mind: In internal_dlsym search the default handle first and then the list of SOs opened so far.

But there are a few things in Linker.c that I don't understand yet:

  1. hdl is dl_prog_handle in all two call sites. Do we need that parameter or could we instead use the static variable dl_prog_handle directly?
  2. With hdl being dl_prog_handle: In internal_dlsym the SO pointed to by hdl is searched last. Why can I not look there first?
  3. In internal_dlopen the SO is opened with RTLD_GLOBAL but we want to be able to override symbols later when we reload a library. Why is RTLD_GLOBAL needed?

comment:28 Changed 4 years ago by trommler

And another question: From internal_dlopen():

   if (hdl == NULL) {
      /* dlopen failed; return a ptr to the error msg. */
      errmsg = dlerror();
      if (errmsg == NULL) errmsg = "addDLL: unknown error";
      errmsg_copy = stgMallocBytes(strlen(errmsg)+1, "addDLL");
      strcpy(errmsg_copy, errmsg);
      errmsg = errmsg_copy;
   }
   o_so = stgMallocBytes(sizeof(OpenedSO), "addDLL");
   o_so->handle = hdl;
   o_so->next   = openedSOs;
   openedSOs    = o_so;

If dlopen() fails we are adding a NULL handle to openedSOs. Which on my system means RTLD_DEFAULT. Is that what we want?

comment:29 Changed 4 years ago by simonmar

My apologies for making a buggy test program :-) Thanks for eliminating the strangeness there. So it seems that dlsym always returns a bogus result for environ.

Your proposed fix only works if we load libraries with RTLD_LOCAL rather than RTLD_GLOBAL, I think. That is probably fine, but it needs testing. This commit that introduced internal_dlsym explains the rationale: be497c202b790999c3fd0ddc4a4176b8cf6acf7e

I'm fine with refactoring away the hdl argument.

comment:30 in reply to:  29 Changed 4 years ago by trommler

Status: infoneedednew

Replying to simonmar:

My apologies for making a buggy test program :-) Thanks for eliminating the strangeness there.

No problem! I learned a lot about dynamic linking!

So it seems that dlsym always returns a bogus result for environ.

No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at *defl_env and *env in the second program in comment:25. The addresses match and I checked they really point to the environment array!

In the first program the static linker creates the reference for environ because we take its address &environ to print it. So there is a static symbol already there and that confuses the dynamic linker when it searches in libgmp and then finds environ in libc.

Once we have fixed this bug I will read the ELF specs again and see if giving us a defunct symbol when we load a weak symbol again is really allowed by the specification.

Your proposed fix only works if we load libraries with RTLD_LOCAL rather than RTLD_GLOBAL, I think. That is probably fine, but it needs testing. This commit that introduced internal_dlsym explains the rationale: be497c202b790999c3fd0ddc4a4176b8cf6acf7e

I'm fine with refactoring away the hdl argument.

There is a branch on my github https://github.com/trommler/ghc/tree/rtld-8935 where I implemented that and what I proposed above. While my AMD 64 bit and my old PowerMac G5 build ghc, I will read up on the rationale for internal_dlsym.

If validate succeeds and I am sufficiently confident I understand what internal_dlsym needs to do. I'll post my patches.

We can remove the info needed.

comment:31 Changed 4 years ago by trommler

Validating on x86_64 Linux:

Unexpected failures:
   codeGen/should_run  cgrun051 [bad exit code] (normal)
   ffi/should_compile  cc004 [stderr mismatch] (normal)
   ffi/should_run      T2276_ghci [bad stdout or stderr] (ghci)
   ghci/scripts        T8696 [bad stderr] (ghci)
   perf/compiler       T3064 [stat not good enough] (normal)
   perf/compiler       T6048 [stat not good enough] (optasm)

So two dynamic linking related test failures: T2276_ghci and T8696.

T2276_ghci:

=====> T2276_ghci(ghci) 3154 of 3962 [0, 0, 0] 
cd ./ffi/should_run && $MAKE -s --no-print-directory T2276_ghci_setup

T2276_ghci_c.c:4:1:
     warning: ‘stdcall’ attribute ignored [-Wattributes]
     {
     ^
cd ./ffi/should_run && '/local/home/peter/ghc-validate/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-ghci-history T2276_ghci.hs --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS -fobject-code T2276_ghci_c.o  <T2276_ghci.genscript 1>T2276_ghci.interp.stdout 2>T2276_ghci.interp.stderr
Actual stderr output differs from expected:
--- /dev/null   2014-05-06 16:04:29.720620754 +0200
+++ ./ffi/should_run/T2276_ghci.run.stderr      2014-05-06 21:35:41.193750997 +0200
@@ -0,0 +1,6 @@
+ghc-stage2: panic! (the 'impossible' happened)
+  (GHC version 7.9.20140506 for x86_64-unknown-linux):
+       Loading temp shared object failed: /tmp/ghc18005_0/ghc18005_5.so: undefined symbol: test
+

The warning about stdcall is strange.

and T8696:

=====> T8696(ghci) 3619 of 3962 [0, 0, 0] 
cd ./ghci/scripts && HC='/local/home/peter/ghc-validate/inplace/bin/ghc-stage2' HC_OPTS='-dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-ghci-history ' '/local/home/peter/ghc-validate/inplace/bin/ghc-stage2' --interactive -v0 -ignore-dot-ghci -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-ghci-history     <T8696.script >T8696.run.stdout 2>T8696.run.stderr
Actual stderr output differs from expected:
--- /dev/null   2014-05-06 16:04:29.720620754 +0200
+++ ./ghci/scripts/T8696.run.stderr     2014-05-06 21:37:54.034299041 +0200
@@ -0,0 +1,6 @@
+ghc-stage2: panic! (the 'impossible' happened)
+  (GHC version 7.9.20140506 for x86_64-unknown-linux):
+       Loading temp shared object failed: /tmp/ghc18096_0/ghc18096_4.so: undefined symbol: T8696A_a_closure
+

The performance tests pass when I run them from the command line (I validated with CPUS=4) and I seem to remember that cgrun051 exit code was discussed on ghc-devs recently. I have no idea what to make of cc004 and the warning about stdcall calling convention not implemented on my platform.

And last but not least: ghc -e 'System.Environment.getEnvironment' does not segfault but prints the actual environment.

I'll prepare a patch and attach it to the ticket.

Changed 4 years ago by trommler

Look up symbols in RTLD_DEFAULT first

Changed 4 years ago by trommler

Part two: Put symbols in LOCAL scope.

Changed 4 years ago by trommler

Refactor parameter list of internal_dlsym

Changed 4 years ago by trommler

Do not add a NULL handle to SO list for failed dlopen

comment:32 Changed 4 years ago by trommler

Status: newpatch

I attached four patches for review.

I'll check if I see the test failures also in HEAD.

comment:33 Changed 4 years ago by simonmar

No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at *defl_env and *env in the second program in comment:25. The addresses match and I checked they really point to the environment array!

Well, every time we call dlsym passing the handle of a dlopen'd library, we get a bogus result. This happens for both libgmp and libpthread (after modifying the libpthread path to point to the real .so, not the linker script).

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

char *so = "/usr/lib/x86_64-linux-gnu/libgmp.so";
char *so2 = "/lib/x86_64-linux-gnu/libpthread.so.0";

extern char**environ;

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  char ***env;

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  printf("&environ = %p, environ = %p\n", &environ, environ);

  env = dlsym(deflt,"environ");
  printf("dlsym(deflt, \"environ\") = %p, *env = %p\n", env, *env );

  hdl = dlopen(so, RTLD_LAZY);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  env = dlsym(hdl,"environ");
  printf("dlsym(\"libgmp\", \"environ\") = %p, *env = %p\n", env, *env);

  hdl = dlopen(so2, RTLD_LAZY);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  env = dlsym(hdl,"environ");
  printf("dlsym(\"libpthread\", \"environ\") = %p, *env = %p\n", env, *env);
}

output:

$ ./a.out                     
&environ = 0x601078, environ = 0x7fffc44c2008
dlsym(deflt, "environ") = 0x601078, *env = 0x7fffc44c2008
dlsym("libgmp", "environ") = 0x2ba8772b64e8, *env = (nil)
dlsym("libpthread", "environ") = 0x2ba8772b64e8, *env = (nil)

comment:34 in reply to:  33 Changed 4 years ago by trommler

Replying to simonmar:

No, no, it does not look bad at all. The address looks strange but that is caused by the fact that it was created by the dynamic linker. Look at *defl_env and *env in the second program in comment:25. The addresses match and I checked they really point to the environment array!

Well, every time we call dlsym passing the handle of a dlopen'd library, we get a bogus result. This happens for both libgmp and libpthread (after modifying the libpthread path to point to the real .so, not the linker script).

Yes, that is consistent with what I expect from our previous experiments.

I don't say I understand yet why the dynamic linker gives us a broken symbol and if this behavior is correct. What I know is that the dynamic linker treats weak symbols the same as strong symbols. And then there is a conflict: In the program there is a symbol environ created by the static linker and now there is another symbol environ and that is not permitted and instead of failing we get this NULL pointer.

I bet all libraries that depend on libc directly or indirectly will exhibit the behavior we see.

But that was not my point. I wanted to say, when we look for environ in RTLD_DEFAULT we will get a properly set up pointer to the environment. My second test program (in comment:25) was meant to demonstrate that this is also true when we do not refer to environ in the program itself.

comment:35 in reply to:  32 Changed 4 years ago by trommler

Replying to trommler:

I attached four patches for review.

I'll check if I see the test failures also in HEAD.

The validate on HEAD just finished.

The two failures I reported earlier are regressions. I will look into these on Thursday.

comment:36 Changed 4 years ago by dagit

Here is the test program I use:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

int jason;
extern char**environ;

char envname[]  = "environ";
char jasname[]  = "jason";
char weakname[] = "weak_jason";

void dlsym_check(void *handle, char * lib, char * sym){
    void * e = dlsym(handle,sym);
    fprintf(stderr, "dlsym(\"%s\", \"%s\") = %p\n", lib, sym, e);
    char * error = dlerror();
    if( error != NULL )
    fprintf(stderr, "Errors: %s\n", error);
}

int main(int argc, char *argv[])
{
  if( argc < 3 ){
    printf("usage: ./check-environ <path to shared object> <path to shared object>\n");
    exit(1);
  }
  char* so  = argv[1];
  char* so2 = argv[2];
  void *deflt, *hdl, *hdl2;

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  dlsym_check(deflt, so, envname);
  hdl = dlopen(so, RTLD_LAZY | RTLD_GLOBAL);
  if( hdl == NULL ){
    fprintf(stderr, "Error: %s", dlerror());
    exit(1);
  }
  hdl2 = dlopen(so2, RTLD_LAZY | RTLD_GLOBAL);
  if( hdl2 == NULL ){
    fprintf(stderr, "Error: %s", dlerror());
    exit(1);
  }

  fprintf(stderr, "\nFind a symbol that is located here and libc\n");
  dlsym_check(deflt,        so, envname);
  dlsym_check(hdl,          so, envname);
  dlsym_check(RTLD_DEFAULT, so, envname);
  dlsym_check(RTLD_NEXT,    so, envname);

  fprintf(stderr, "\nFind a symbol that is only defined in this executable\n");
  dlsym_check(deflt,        so, jasname);
  dlsym_check(hdl,          so, jasname);
  dlsym_check(RTLD_DEFAULT, so, jasname);
  dlsym_check(RTLD_NEXT,    so, jasname);

  fprintf(stderr, "\nFind a symbol that is only defined weak in a shared object\n");
  dlsym_check(deflt,        so2, weakname);
  dlsym_check(hdl2,         so2, weakname);
  dlsym_check(RTLD_DEFAULT, so2, weakname);
  dlsym_check(RTLD_NEXT,    so2, weakname);
}

That lives in check-environ.c, so I build that with:

LDFLAGS=-ldl CFLAGS=-rdynamic make check-environ

(No actual build file required, I just use implicit rules.)

I also have test.c, containing just one line:

int __attribute__((weak)) weak_jason;

Built with: gcc -shared -fpic test.c -o test.so

Notice I've added a symbol "jason" that is not defined anywhere but in the executable. I use -rdynamic as the manpage for dlsym suggests. The test.so shared object has a weak symbol definition for weak_jason (you should double check me). I've also duplicated the handle options to dlsym just so we can see how they change the behavior.

When I run it:

$  ./check-environ libgmp.so ./test.so 2>&1
dlsym("libgmp.so", "environ") = 0x31e45bd508

Find a symbol that is located here and libc
dlsym("libgmp.so", "environ") = 0x31e45bd508
dlsym("libgmp.so", "environ") = 0x31e45bd508
dlsym("libgmp.so", "environ") = 0x31e45bd508
dlsym("libgmp.so", "environ") = 0x31e45bd508

Find a symbol that is only defined in this executable
dlsym("libgmp.so", "jason") = 0x60208c
dlsym("libgmp.so", "jason") = (nil)
Errors: /lib64/libgmp.so: undefined symbol: jason
dlsym("libgmp.so", "jason") = 0x60208c
dlsym("libgmp.so", "jason") = (nil)

Find a symbol that is only defined weak in a shared object
dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c
dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c
dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c
dlsym("./test.so", "weak_jason") = 0x7f8de2f1e02c

I then ran it with LD_DEBUG=all, and looked at the output for environ.

 Find a symbol that is located here and libc
     21283:     symbol=environ;  lookup in file=./check-environ [0]
     21283:     symbol=environ;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=environ;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ'
dlsym("libgmp.so", "environ") = 0x31e45bd508
     21283:     symbol=environ;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=environ;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file /lib64/libgmp.so [0] to /lib64/libc.so.6 [0]: normal symbol `environ'
dlsym("libgmp.so", "environ") = 0x31e45bd508
     21283:     symbol=environ;  lookup in file=./check-environ [0]
     21283:     symbol=environ;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=environ;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ'
dlsym("libgmp.so", "environ") = 0x31e45bd508
     21283:     symbol=environ;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=environ;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file ./check-environ [0] to /lib64/libc.so.6 [0]: normal symbol `environ'
dlsym("libgmp.so", "environ") = 0x31e45bd508

In each cases, once it found the definition in libc it stopped looking (even though it's WEAK). The search for weak_jason is similar, but it has to keep looking until it gets to test.so. I think it's safe to say that the behavior is not specific to environ or libc:

Find a symbol that is only defined weak in a shared object
     21283:     symbol=weak_jason;  lookup in file=./check-environ [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libc.so.6 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=weak_jason;  lookup in file=./test.so [0]
     21283:     binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason'
dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c
     21283:     symbol=weak_jason;  lookup in file=./test.so [0]
     21283:     binding file ./test.so [0] to ./test.so [0]: normal symbol `weak_jason'
dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c
     21283:     symbol=weak_jason;  lookup in file=./check-environ [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libc.so.6 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=weak_jason;  lookup in file=./test.so [0]
     21283:     binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason'
dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c
     21283:     symbol=weak_jason;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libc.so.6 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21283:     symbol=weak_jason;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=weak_jason;  lookup in file=./test.so [0]
     21283:     binding file ./check-environ [0] to ./test.so [0]: normal symbol `weak_jason'
dlsym("./test.so", "weak_jason") = 0x7f0e99fbd02c

In the case of the jason symbol:

Find a symbol that is only defined in this executable
     21283:     symbol=jason;  lookup in file=./check-environ [0]
     21283:     binding file ./check-environ [0] to ./check-environ [0]: normal symbol `jason'
dlsym("libgmp.so", "jason") = 0x60208c
     21283:     symbol=jason;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=jason;  lookup in file=/lib64/libc.so.6 [0]
     21283:     symbol=jason;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21283:     /lib64/libgmp.so: error: symbol lookup error: undefined symbol: jason (fatal)
dlsym("libgmp.so", "jason") = (nil)
     21283:     symbol=__dcgettext;  lookup in file=./check-environ [0]
     21283:     symbol=__dcgettext;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=__dcgettext;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `__dcgettext' [GLIBC_2.2.5]
     21283:     symbol=__asprintf;  lookup in file=./check-environ [0]
     21283:     symbol=__asprintf;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=__asprintf;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `__asprintf' [GLIBC_2.2.5]
     21283:     symbol=free;  lookup in file=./check-environ [0]
     21283:     symbol=free;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=free;  lookup in file=/lib64/libc.so.6 [0]
     21283:     binding file /lib64/libdl.so.2 [0] to /lib64/libc.so.6 [0]: normal symbol `free' [GLIBC_2.2.5]
Errors: /lib64/libgmp.so: undefined symbol: jason
     21283:     symbol=jason;  lookup in file=./check-environ [0]
     21283:     binding file ./check-environ [0] to ./check-environ [0]: normal symbol `jason'
dlsym("libgmp.so", "jason") = 0x60208c
     21283:     symbol=jason;  lookup in file=/lib64/libdl.so.2 [0]
     21283:     symbol=jason;  lookup in file=/lib64/libc.so.6 [0]
     21283:     symbol=jason;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     21283:     symbol=jason;  lookup in file=/lib64/libgmp.so [0]
     21283:     symbol=jason;  lookup in file=./test.so [0]
dlsym("libgmp.so", "jason") = (nil)

The point though, is that dlsym is doing exactly what it should: It returns the first global definition it finds.

comment:37 Changed 4 years ago by simonmar

The point though, is that dlsym is doing exactly what it should: It returns the first global definition it finds.

That's right, but let me add that weak symbols are a red herring. Let me demonstrate this without using weak symbols or environ.

foo.c:

int bar = 1;

int getbar(void)
{
    return bar;
}

Compile it like this:

$ gcc -fPIC foo.c -shared -o libfoo.so

Now the test program, test.c:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

extern int bar;
extern int getbar(void);

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  int *pbar;
  int (*pgetbar)(void);

  bar = 2;

  printf("&bar = %p, bar = %d, getbar() = %d\n", &bar, bar, getbar());

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  pbar = dlsym(deflt, "bar");
  printf("dlsym(deflt, \"bar\") = %p, *pbar = %d\n", pbar, *pbar);
  pgetbar = dlsym(deflt, "getbar");
  printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());

  hdl = dlopen("libfoo.so", RTLD_LAZY);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  pbar = dlsym(hdl, "bar");
  printf("dlsym(\"./libfoo.so\", \"bar\") = %p, *pbar = %d\n", pbar, *pbar);
  pgetbar = dlsym(hdl, "getbar");
  printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());
}

Note that we have extern references for both the data variable bar and the function getbar. Compile and run it like this:

$ gcc test.c -ldl libfoo.so
$ LD_LIBRARY_PATH=. ./a.out
$ LD_LIBRARY_PATH=. ./a.out
&bar = 0x601060, bar = 2, getbar() = 2
dlsym(deflt, "bar") = 0x601060, *pbar = 2
dlsym(deflt, "getbar") = 0x2b9809ab259c, pgetbar() = 2
dlsym("./libfoo.so", "bar") = 0x2b9809cb3010, *pbar = 1
dlsym(deflt, "getbar") = 0x2b9809ab259c, pgetbar() = 2

Note that:

  • the address of the function getbar is the same, regardless of whether we look it up in the main program or libfoo.so
  • the data variable bar has one address in the main program, and another one in libfoo.so.
  • the version in libfoo.so has the initial value 1, it wasn't updated by the assignment bar = 2 in the main program
  • Calling getbar() returns the correct value 2, because the refernce to bar in libfoo.so has been relocated to point to the version of bar in the main program, not the one in libfoo.so.

So I'm pretty sure that this is all due to the copy semantics that moves data variables into the main program. It means that if you use dlsym to find the address of a data variable, you might get the wrong answer.

In any case, the correct fix for GHC ought to be to use RTLD_LOCAL and look up in the main program first.

comment:38 Changed 4 years ago by dagit

Great and thanks for the examples!

By the way, weak not making a difference is the other half of the point I wanted to make :) I'm glad we're on the same page. I feel like I've learned something useful that I can use elsewhere.

You might also look at RTLD_DEEPBIND for doing the initial symbol match up (when you use RTLD_LOCAL). I can't be certain, but I think it may help when reloading a shared object (presumably with new definitions inside). As far as I can tell, that option was added specifically for uses like this (I found some mailing list threads about it). I suspect the point with that option is that it allows consistent behavior even if something you don't have control over passes RTLD_GLOBAL to dlopen. My prediction is that if you switch to using RTLD_LOCAL, then you won't need RTLD_DEEPBIND in 99% of cases.

Also, I played with RTLD_NEXT, to see if I could locate the last symbol loaded, but I couldn't figure out when I had reached the end and I also wasn't able to lookup the symbol in the executable that way. I wish I understood what they mean by next.

Thanks!

comment:39 Changed 4 years ago by dagit

After playing with your example a bit I see what you mean about the copy semantics. I found this article: http://netwinder.osuosl.org/users/p/patb/public_html/elf_relocs.html

Interestingly, glibc has these comments:

/*
 * Process the special R_X86_64_COPY relocations in the main program.  These
 * copy data from a shared object into a region in the main program's BSS
 * segment.
 *
 * Returns 0 on success, -1 on failure.
 */
int
do_copy_relocations(Obj_Entry *dstobj)

And:

case R_X86_64_COPY:
		/*
		 * These are deferred until all other relocations have
		 * been done.  All we do here is make sure that the COPY
		 * relocation is not in a shared library.  They are allowed
		 * only in executable files.
		 */

The elf interpreter delays processing of the R_COPY's so that it gets the correct behavior. I think this supports your earlier hypothesis.

At this point, I noticed that I can switch environ in the executable between R_X86_64_GLOB_DAT and R_X86_64_COPY by adding -fPIC (you probably already knew that). Specifically, without -fPIC I get copy and with it I get glob dat.

Test program again:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

extern char**environ;

char envname[]  = "environ";

void dlsym_check(void *handle, char * lib, char * sym){
    void * e = dlsym(handle,sym);
    fprintf(stderr, "dlsym(\"%s\", \"%s\") = %p\n", lib, sym, e);
    char * error = dlerror();
    if( error != NULL )
    fprintf(stderr, "Errors: %s\n", error);
}

int main(int argc, char *argv[])
{
  if( argc < 2 ){
    printf("usage: ./check-environ <path to shared object>\n");
    exit(1);
  }
  char* so  = argv[1];
  void *deflt, *hdl;

  fprintf(stderr, "environ = %p\n", &environ);

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  fprintf(stderr, "\nFind a symbol after passing NULL to dlopen:\n");
  dlsym_check(deflt, NULL, envname);
  hdl = dlopen(so, RTLD_LAZY | RTLD_GLOBAL);
  if( hdl == NULL ){
    fprintf(stderr, "Error: %s", dlerror());
    exit(1);
  }

  fprintf(stderr, "\nFind a symbol that is located here and libc (handle comes from dlopen on %s)\n", so);
  dlsym_check(hdl,          so, envname);
}
$ ./check-environ-without-pic /usr/lib64/libgmp.so.10
environ = 0x601070

Find a symbol after passing NULL to dlopen:
dlsym("(null)", "environ") = 0x601070

Find a symbol that is located here and libc (handle comes from dlopen on /usr/lib64/libgmp.so.10)
dlsym("/usr/lib64/libgmp.so.10", "environ") = 0x31e45bd508

$ ./check-environ-with-pic /usr/lib64/libgmp.so.10
environ = 0x31e45bd508

Find a symbol after passing NULL to dlopen:
dlsym("(null)", "environ") = 0x31e45bd508

Find a symbol that is located here and libc (handle comes from dlopen on /usr/lib64/libgmp.so.10)
dlsym("/usr/lib64/libgmp.so.10", "environ") = 0x31e45bd508

comment:40 Changed 4 years ago by trommler

Thanks for the great examples!

Coming back to comment:31 where I reported two failing tests: I think some changes in the creation of the SOs that ghci loads are required. Could you tell me where in the code those SOs are created?

May I temporarily own the ticket for that fix?

comment:41 Changed 4 years ago by simonmar

Owner: changed from simonmar to trommler

@trommler, giving you the ticket.

Thinking about it, I'm less sure that RTLD_LOCAL will work now, because it won't allow us to load new libraries that depend on libraries we've previously loaded, and we definitely need to be able to do that.

comment:42 in reply to:  41 Changed 4 years ago by trommler

Replying to simonmar:

@trommler, giving you the ticket.

Thanks!

Thinking about it, I'm less sure that RTLD_LOCAL will work now, because it won't allow us to load new libraries that depend on libraries we've previously loaded, and we definitely need to be able to do that.

For libraries (SOs) coming from installed packages everything should be fine. The have a NEEDED tag for all dependencies and the system linker will dlopen() them and resolve the symbols.

So for new libraries temporary SOs need to be produced the same way SOs in packages are, with NEEDED tags for all previously loaded libraries. The new library could potentially depend on all previously loaded libraries.

I am thinking of something along the lines of:

ld -shared temp.o ... < -l for each library loaded so far >

Perhaps we could remove libraries from the list that are not needed by the new library by using --as-needed but I want to check first if this flag is GNU ld specific.

comment:43 Changed 4 years ago by simonmar

For libraries (SOs) coming from installed packages everything should be fine. The have a NEEDED tag for all dependencies and the system linker will dlopen() them and resolve the symbols.

But won't that reload all the dependencies every time we load a library? We'll end up with multiple copies of ghc-prim, base, etc. etc. It's not just a question of wasted memory, but multiple copies of base causes problems because we have multiple stdouts and other CAFs.

comment:44 Changed 4 years ago by trommler

No, the library would not be loaded again.

POSIX says:

Only a single copy of an object file is brought into the address space, even if dlopen() is invoked multiple times in reference to the file, and even if different pathnames are used to reference the file.

The Linux man page:

If the same library is loaded again with dlopen(), the same file handle is returned. The dl library maintains reference counts for library handles, so a dynamic library is not deallocated until dlclose() has been called on it as many times as dlopen() has succeeded on it. The _init() routine, if present, is called only once. But a subsequent call with RTLD_NOW may force symbol resolution for a library earlier loaded with RTLD_LAZY.

So I think we will be good.

comment:45 Changed 4 years ago by simonmar

I didn't believe you, so I made a test program, and now I believe you :) It's somewhat surprising that a library we loaded with RTLD_LOCAL can be used to satisfy a dependency of another library we loaded with RTLD_LOCAL, but it does indeed seem to be the case.

comment:46 Changed 4 years ago by dagit

Simon, that is surprising. Do you still have your test setup? Could I take a look?

comment:47 Changed 4 years ago by simonmar

bar.c:

static int bar = 1;

int _getbar(void)
{
    return bar;
}

void _setbar(int x)
{
    bar = x;
}

foo.c:

extern int _getbar(void);
extern void _setbar(int);

int getbar(void)
{
    return _getbar();
}

int setbar(int x)
{
    _setbar(x);
}

test.c:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

extern int getbar(void);
extern void setbar(int);

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  int (*pgetbar)(void);
  int (*psetbar)(int);

  setbar(2);

  printf("getbar() = %d\n", getbar());

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  if (deflt == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  pgetbar = dlsym(deflt, "getbar");
  printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());

  hdl = dlopen("libbar.so", RTLD_LOCAL | RTLD_LAZY);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  pgetbar = dlsym(hdl, "_getbar");
  psetbar = dlsym(hdl, "_setbar");
  printf("dlsym(deflt, \"_getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());
  (*psetbar)(3);
  printf("(*psetbar)(3); pgetbar() = %d\n", (*pgetbar)());

  hdl = dlopen("libfoo.so", RTLD_LOCAL | RTLD_LAZY);
  if (hdl == NULL) {
      printf("%s\n", dlerror());
      exit(1);
  }
  pgetbar = dlsym(hdl, "getbar");
  psetbar = dlsym(hdl, "setbar");
  printf("dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());
  (*psetbar)(4);
  printf("(*psetbar)(4); pgetbar() = %d\n", (*pgetbar)());

}
$ gcc -shared -fPIC bar.c -o libbar.so                   
$ gcc -shared -fPIC foo.c -o libfoo.so -L. -lbar 
$ gcc test.c -L. -lfoo -lbar -ldl -g
$ LD_LIBRARY_PATH=. ./a.out         
getbar() = 2
dlsym(deflt, "getbar") = 0x2b4660dcc61c, pgetbar() = 2
dlsym(deflt, "_getbar") = 0x2b46615a658c, pgetbar() = 2
(*psetbar)(3); pgetbar() = 3
dlsym(deflt, "getbar") = 0x2b4660dcc61c, pgetbar() = 3
(*psetbar)(4); pgetbar() = 4

comment:48 Changed 4 years ago by dagit

I think you test depends on how you link the program and libraries.

I'm using your setup, but I made the following tweaks.

test.c, in this version dlopen of libbar is RTLD_GLOBAL:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

extern int getbar(void);
extern void setbar(int);

int main(int argc, char *argv[])
{
  void *deflt, *hdl;
  int (*pgetbar)(void);
  int (*psetbar)(int);

  //setbar(2);

  //fprintf(stderr,"getbar() = %d\n", getbar());

  deflt = dlopen(NULL, RTLD_LAZY | RTLD_GLOBAL);
  if (deflt == NULL) {
      fprintf(stderr,"%s\n", dlerror());
      exit(1);
  }
  //pgetbar = dlsym(deflt, "getbar");
  //fprintf(stderr,"dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());

  hdl = dlopen("libbar.so", RTLD_GLOBAL | RTLD_LAZY);
  if (hdl == NULL) {
      fprintf(stderr,"%s\n", dlerror());
      exit(1);
  }
  pgetbar = dlsym(hdl, "_getbar");
  char* e;
  e = dlerror();
  if( e != NULL ){
    fprintf(stderr,"%s\n", e);
  }
  psetbar = dlsym(hdl, "_setbar");
  e = dlerror();
  if( e != NULL ){
    fprintf(stderr,"%s\n", e);
  }
  fprintf(stderr,"dlsym(deflt, \"_getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());
  (*psetbar)(3);
  fprintf(stderr,"(*psetbar)(3); pgetbar() = %d\n", (*pgetbar)());

  hdl = dlopen("libfoo.so", RTLD_LOCAL | RTLD_LAZY);
  if (hdl == NULL) {
      fprintf(stderr,"%s\n", dlerror());
      exit(1);
  }
  pgetbar = dlsym(hdl, "getbar");
  e = dlerror();
  if( e != NULL ){
    fprintf(stderr,"%s\n", e);
  }
  psetbar = dlsym(hdl, "setbar");
  e = dlerror();
  if( e != NULL ){
    fprintf(stderr,"%s\n", e);
  }
  fprintf(stderr,"dlsym(deflt, \"getbar\") = %p, pgetbar() = %d\n", pgetbar, (*pgetbar)());
  (*psetbar)(4);
  fprintf(stderr,"(*psetbar)(4); pgetbar() = %d\n", (*pgetbar)());

  return 0;
}

Here, I don't link together the libraries at compile time, not even for test.c (I'm trying to be careful not to call this static linking, because as we'll see later it's not):

gcc -shared -fPIC bar.c -o libbar.so
gcc -shared -fPIC foo.c -o libfoo.so
gcc -Wall test.c -ldl -g
LD_LIBRARY_PATH=. ./a.out

Now when I run the program I get:

dlsym(deflt, "_getbar") = 0x7fddcc3a5698, pgetbar() = 1
(*psetbar)(3); pgetbar() = 3
dlsym(deflt, "getbar") = 0x7fddcc1a3728, pgetbar() = 3
(*psetbar)(4); pgetbar() = 4

Okay, as we would expect libfoo uses the definition in libbar. Now, switch the dlopen to RTLD_LOCAL, and run it again:

dlsym(deflt, "_getbar") = 0x7f056f698698, pgetbar() = 1
(*psetbar)(3); pgetbar() = 3
./a.out: symbol lookup error: ./libfoo.so: undefined symbol: _getbar

Now if we link libbar and the final program:

gcc -shared -fPIC bar.c -o libbar.so
gcc -shared -fPIC foo.c -o libfoo.so
gcc -Wall test.c -L. -lbar -ldl -g
LD_LIBRARY_PATH=. ./a.out
dlsym(deflt, "_getbar") = 0x7f75b9914698, pgetbar() = 1
(*psetbar)(3); pgetbar() = 3
dlsym(deflt, "getbar") = 0x7f75b96f9728, pgetbar() = 3
(*psetbar)(4); pgetbar() = 4

Now it finds the symbol again. The difference seems to be this:

readelf -Wa a.out.withbar | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libbar.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
readelf -Wa a.out.withoutbar | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

In other words, mentioning libbar on the gcc line for test.c adds an entry that the elf interpreter sees and it resolves the symbol, lazily, before we get into main. I would expect the same behavior if libfoo had been linked to libbar instead of with the main application. And indeed, that's what we get:

gcc -shared -fPIC bar.c -o libbar.so
gcc -shared -fPIC foo.c -o libfoo.so -L. -lbar
gcc -Wall test.c -ldl -g -o a.out.withoutbar
LD_LIBRARY_PATH=. ./a.out.withoutbar
dlsym(deflt, "_getbar") = 0x7fc75e77c698, pgetbar() = 1
(*psetbar)(3); pgetbar() = 3
dlsym(deflt, "getbar") = 0x7fc75e57a728, pgetbar() = 3
(*psetbar)(4); pgetbar() = 4

readelf -Wa libfoo.so | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libbar.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

comment:49 Changed 4 years ago by simonmar

@dagit: that's right, the point of that test is to show that a library loaded with RTLD_LOCAL is used to satisfy a dependency of another library loaded with RTLD_LOCAL. The dependency is expressed by the DT_NEEDED tag, and is set up when you link the library. @trommler's point earlier was that if we use RTLD_LOCAL then we'll need to ensure that all the dependencies are specified with DT_NEEDED tags.

comment:50 Changed 4 years ago by dagit

Excellent. I must have missed trommler saying that. Thanks!

Changed 3 years ago by trommler

All of the above plus fix for regressions.

Changed 3 years ago by trommler

Patch most likely required for OS X.

comment:51 Changed 3 years ago by trommler

Owner: changed from trommler to simonmar

The attached patches change the way shared libraries are loaded (dlopen) by the runtime system and the way temporary shared libraries are linked for files compiled to object code in GHCi.

The first patch https://ghc.haskell.org/trac/ghc/attachment/ticket/8935/0001-Fix-GHCi-crash-on-access-to-process-environment.patch implements the changes for Linux and I validated those on x86_64 (openSUSE 13.1). The second patch https://ghc.haskell.org/trac/ghc/attachment/ticket/8935/0001-Fix-GHCi-linking-of-files-compiled-to-object-code.patch might be needed for OS X. This patch is not validated as I do not have access to an OS X machine at the moment.

Simon, please have a look at the patches and the commit messages and let me know what could be improved.

comment:52 Changed 3 years ago by Simon Marlow <marlowsd@…>

In 2f8b4c9330b455d4cb31c186c747a7db12a69251/ghc:

Fix obscure problem with using the system linker (#8935)

See Note [RTLD_LOCAL] for a summary of the problem and solution, and

comment:53 Changed 3 years ago by simonmar

Status: patchmerge

Committed with a few changes and comments by me. I'm not sure whether the second patch is required, but I refactored that bit anyway. Please re-open if there's still a problem.

comment:54 Changed 3 years ago by thoughtpolice

Resolution: fixed
Status: mergeclosed

Merged.

comment:55 Changed 3 years ago by thoughtpolice

Milestone: 7.8.37.8.4
Owner: simonmar deleted
Resolution: fixed
Status: closednew

Re-opening, the patch to fix this (2f8b4c9330b455d4cb31c186c747a7db12a69251) caused the failure in #9186, which has been reverted on the GHC 7.8 branch. I'll be reverting it on HEAD as well soon.

comment:56 Changed 3 years ago by thoughtpolice

Owner: set to simonmar

comment:57 Changed 3 years ago by Austin Seipp <austin@…>

In aed1723f97e0539d5ab35222b180c1552a5f4cfc/ghc:

Revert "Fix obscure problem with using the system linker (#8935)"

This reverts commit 2f8b4c9330b455d4cb31c186c747a7db12a69251.

Signed-off-by: Austin Seipp <austin@well-typed.com>

comment:58 Changed 3 years ago by rwbarton

I think the patch to fix this (which has been reverted) would fix #9480, too.

comment:59 Changed 3 years ago by trommler

Cc: trommler added

comment:60 Changed 3 years ago by thoughtpolice

Milestone: 7.8.47.10.1

Moving (in bulk) to 7.10.4

comment:61 Changed 3 years ago by trommler

Owner: changed from simonmar to trommler

I have a fix for 2f8b4c9330b455d4cb31c186c747a7db12a69251 that validates and also does not introduce the regression reported in #9186. I'll clean up the patch and post it on phabricator.

Analysis: On some systems the order of linker options and especially the order libraries are passed to ld matters. That behaviour is caused by a patch binutils to change the default of --no-as-needed to --as-needed. The library ordering issue is the reason for the regression we saw in #9186.

On openSUSE you can control the default behaviour of ld by setting SUSE_ASNEEDED=1 in the environment. Given the name of the environment variable I doubt it works on other distributions :-)

comment:62 Changed 3 years ago by trommler

Differential Rev(s): Phab:D349
Status: newpatch

Here is my patch. I validated on openSUSE 13.1 running on x86_64.

Additionally I back ported the patch to 7.8.3 built, all of Haskell Platform and pandoc and verified I can build compdata-0.8.1.0

comment:63 Changed 3 years ago by trommler

Owner: trommler deleted

comment:64 Changed 3 years ago by Austin Seipp <austin@…>

In 383733b9191a36e2d3f757700842dbc3855911d9/ghc:

Fix obscure problem with using the system linker (#8935)

Summary:
In a statically linked GHCi symbol `environ` resolves to NULL when
called from a Haskell script.

When resolving symbols in a Haskell script we need to search the
executable program and its dependent (DT_NEEDED) shared libraries
first and then search the loaded libraries.

We want to be able to override functions in loaded libraries later.
Libraries must be opened with local scope (RTLD_LOCAL) and not global.
The latter adds all symbols to the executable program's symbols where
they are then searched in loading order. We want reverse loading order.

When libraries are loaded with local scope the dynamic linker
cannot use symbols in that library when resolving the dependencies
in another shared library. This changes the way files compiled to
object code must be linked into temporary shared libraries. We link
with the last temporary shared library created so far if it exists.
Since each temporary shared library is linked to the previous temporary
shared library the dynamic linker finds the latest definition of a
symbol by following the dependency chain.

See also Note [RTLD_LOCAL] for a summary of the problem and solution.

Cherry-picked commit 2f8b4c

Changed linker argument ordering

On some ELF systems GNU ld (and others?) default to
--as-needed and the order of libraries in the link
matters.

The last temporary shared library, must appear
before all other libraries. Switching the position
of extra_ld_inputs and lib_path_objs does that.

Fixes #8935 and #9186

Reviewers: austin, hvr, rwbarton, simonmar

Reviewed By: simonmar

Subscribers: thomie, carter, simonmar

Differential Revision: https://phabricator.haskell.org/D349

GHC Trac Issues: #8935, #9186, #9480

comment:65 Changed 3 years ago by thoughtpolice

Resolution: fixed
Status: patchclosed

OK, marking as fixed (crosses fingers).

comment:66 Changed 3 years ago by Austin Seipp <austin@…>

In 0fcc454329c4e3e0dc4474412bff599d0e9bdfcd/ghc:

Dynamically link all loaded packages in new object

Summary:
As a result of fixing #8935 we needed to open shared libraries
with RTLD_LOCAL and so symbols from packages loaded earlier
cannot be found anymore. We need to include in the link all
packages loaded so far.

This fixes #10058

Test Plan: validate

Reviewers: hvr, simonmar, austin

Reviewed By: austin

Subscribers: rwbarton, thomie

Differential Revision: https://phabricator.haskell.org/D676

GHC Trac Issues: #10058
Note: See TracTickets for help on using tickets.