Opened 19 months ago

Last modified 7 months ago

#12075 infoneeded bug

Fails to build on powerpcspe because of inline assembly

Reported by: glaubitz Owned by:
Priority: normal Milestone: 8.4.1
Component: Build System Version: 8.0.1
Keywords: Cc: trommler, slyfox
Operating System: Linux Architecture: powerpc
Type of failure: Building GHC failed Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D3540
Wiki Page:

Description (last modified by glaubitz)

Hello!

I'm currently bootstrapping GHC on powerpcspe (e500v2) which is a PowerPC architecture without an FPU but with signal processing extensions (hence SPE).

When building GHC on this architecture, it detects the host architecture as powerpc-linux-gnuspe or short "powerpc" which works most of the time.

However, GHC contains some inline assembly in rts/StgCRunAsm.s which uses FPU instructions not available on e500v2 CPUs resulting in assembler error, here with GHC 7.10.3:

"inplace/bin/ghc-stage1" -optc-fno-stack-protector -optc-Wall -optc-Wextra -optc-Wstrict-prototypes -optc-Wmissing-prototypes -optc-Wmissing-declarations -optc-Winline -optc-Waggregate-return -optc-Wpointer-arith -optc-Wmissing-noreturn -optc-Wnested-externs -optc-Wredundant-decls -optc-Iincludes -optc-Iincludes/dist -optc-Iincludes/dist-derivedconstants/header -optc-Iincludes/dist-ghcconstants/header -optc-Irts -optc-Irts/dist/build -optc-DCOMPILING_RTS -optc-DUSE_LIBFFI_FOR_ADJUSTORS -optc-fno-strict-aliasing -optc-fno-common -optc-Irts/dist/build/autogen -optc-O2 -optc-fomit-frame-pointer -optc-g -optc-DRtsWay=\"rts_v\" -optc-w -static  -H32m -O -lffi -optl-pthread -lffi -optl-pthread -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Irts -Irts/dist/build -DCOMPILING_RTS -this-package-key rts -dcmm-lint      -i -irts -irts/dist/build -irts/dist/build/autogen -Irts/dist/build -Irts/dist/build/autogen           -O2    -c rts/StgCRun.c -o rts/dist/build/StgCRun.o
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/tmp/ghcf20d_0/ghc_1.s: Assembler messages:

/tmp/ghcf20d_0/ghc_1.s:24:0:  Error: unrecognized opcode: `stfd'

(...)

/tmp/ghcf20d_0/ghc_1.s:67:0:  Error: unrecognized opcode: `lfd'

/tmp/ghcf20d_0/ghc_1.s:68:0:  Error: unrecognized opcode: `lfd'

/tmp/ghcf20d_0/ghc_1.s:69:0:  Error: unrecognized opcode: `lfd'
rts/ghc.mk:236: recipe for target 'rts/dist/build/StgCRun.o' failed
make[3]: *** [rts/dist/build/StgCRun.o] Error 1
Makefile:71: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/usr/src/ghc-7.10.3'
dh_auto_build: make -j1 returned exit code 2
debian/rules:109: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 2
make[1]: Leaving directory '/usr/src/ghc-7.10.3'
debian/rules:47: recipe for target 'binary-arch' failed
make: *** [binary-arch] Error 2
(sid2-powerpcspe-sbuild)root@atlantis:/usr/src/ghc-7.10.3#

This can be verified by commenting out the assembly which will make the above compile command execute without any problems. Trying to continue compiling GHC will still fail with:

"inplace/bin/ghc-stage1" -static  -H32m -O -lffi -optl-pthread -lffi -optl-pthread -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header -Iincludes/dist-ghcconstants/header -Irts -Irts/dist/build -DCOMPILING_RTS -this-package-key rts -dcmm-lint      -i -irts -irts/dist/build -irts/dist/build/autogen -Irts/dist/build -Irts/dist/build/autogen           -O2    -c rts/HeapStackCheck.cmm -o rts/dist/build/HeapStackCheck.o
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/tmp/ghcca51_0/ghc_3.s: Assembler messages:

/tmp/ghcca51_0/ghc_3.s:525:0:  Error: unrecognized opcode: `lfs'

/tmp/ghcca51_0/ghc_3.s:538:0:  Error: unrecognized opcode: `lfd'

/tmp/ghcca51_0/ghc_3.s:592:0:  Error: unrecognized opcode: `stfs'

/tmp/ghcca51_0/ghc_3.s:604:0:  Error: unrecognized opcode: `stfd'
rts/ghc.mk:236: recipe for target 'rts/dist/build/HeapStackCheck.o' failed
make[3]: *** [rts/dist/build/HeapStackCheck.o] Error 1
Makefile:71: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/usr/src/ghc-7.10.3'
dh_auto_build: make -j1 returned exit code 2
debian/rules:109: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 2
make[1]: Leaving directory '/usr/src/ghc-7.10.3'
debian/rules:47: recipe for target 'binary-arch' failed
make: *** [binary-arch] Error 2
(sid2-powerpcspe-sbuild)root@atlantis:/usr/src/ghc-7.10.3#

So, there is obviously more assembly code that needs to be disabled on powerpcspe.

For that, I suggest testing whether the compiler sets NO_FPRSwhich is only set on powerpc hosts without FP registers:

root@atlantis:~# echo | gcc -E -dM -|grep -i FP
#define __NO_FPRS__ 1
root@atlantis:~# dpkg --print-architecture
powerpcspe
root@atlantis:~# uname -m
ppc
root@atlantis:~#

For reference, on a real powerpc host, the output is as follows:

(sid_powerpc-dchroot)glaubitz@partch:~$ echo | gcc -E -dM -|grep -i FP
#define __FP_FAST_FMAF 1
#define __FP_FAST_FMA 1
(sid_powerpc-dchroot)glaubitz@partch:~$ dpkg --print-architecture
powerpc
(sid_powerpc-dchroot)glaubitz@partch:~$ uname -m
ppc
(sid_powerpc-dchroot)glaubitz@partch:~$

If you need any more input, please let me know.

For reference, the list of supported instructions on e500(v2) CPUs can be found here:

http://www.nxp.com/files/32bit/doc/ref_manual/E500CORERM.pdf (page 3-66 or 196)

Adrian

Attachments (1)

config.log (27.0 KB) - added by glaubitz 18 months ago.
Output of configure without options on a e500v2 machine

Download all attachments as: .zip

Change History (17)

comment:1 Changed 19 months ago by glaubitz

Description: modified (diff)

comment:2 Changed 18 months ago by bgamari

Cc: Trommler added
Milestone: 8.2.1

CCing Peter Trommler, who has done a great deal of work on PPC.

comment:3 Changed 18 months ago by thomie

Cc: trommler slyfox added; Trommler removed

comment:4 Changed 18 months ago by trommler

Component: CompilerCompiler (NCG)
Status: newinfoneeded

I think we need to be more specific when we parse the host triple in configure.ac and introduce instruction set variants. Right now powerpc (ArchPPC in Haskell code) refers to a server (not embedded) PowerPC. Could you please attach the output of ./configure to this ticket so we can fix architecture detection in configure.ac.

The native code generator assumes a server PPC with FPU so your proposed fix would only be a first step to support e500 CPUs . That is where you see your second failure.

I don't have access to e500 hardware but I could help with coaching and code reviews if you would like to add support for embedded PPCs. Meanwhile you could also build an unregisterised compiler.

Changed 18 months ago by glaubitz

Attachment: config.log added

Output of configure without options on a e500v2 machine

comment:5 in reply to:  4 ; Changed 18 months ago by glaubitz

Replying to trommler:

I think we need to be more specific when we parse the host triple in configure.ac and introduce instruction set variants. Right now powerpc (ArchPPC in Haskell code) refers to a server (not embedded) PowerPC. Could you please attach the output of ./configure to this ticket so we can fix architecture detection in configure.ac.

Attaching the output of configure (config.log) which was run on native hardware without options (7.10.3).

The native code generator assumes a server PPC with FPU so your proposed fix would only be a first step to support e500 CPUs . That is where you see your second failure.

I was actually not asking for NGC right now, but just wanted to build unregisterised.

I don't have access to e500 hardware but I could help with coaching and code reviews if you would like to add support for embedded PPCs. Meanwhile you could also build an unregisterised compiler.

That's exactly the problem. The inline assembly is there and used even when "--enable-unregisterised" was passed to "configure". Apparently, it just checks for "powerpc" and ignores the "--enable-unregistrised".

I am happy to provide access for you on a e500v2 machine for debugging and testing. Just send me your public SSH key in a private, GPG-signed mail.

Adrian

Last edited 18 months ago by glaubitz (previous) (diff)

comment:6 Changed 8 months ago by bgamari

Milestone: 8.2.18.4.1

Given that 8.2.1-rc1 is imminent, I'm bumping these off to the 8.4

comment:7 Changed 7 months ago by glaubitz

So, this actually seems to be fairly easy to fix but I cannot come up with a satisfiable patch.

Basically, we just need to disable rts/AdjustorAsm.S in rts/ghc.mk if the C compiler defines __NO_FPRS__. When this macro is defined, the CPU is using FPU emulation instead of a hardware FPU, so basically the same as for ARM soft-float.

Looking at the main ghc.mk, it does something similar for ARM but it actually defines a Haskell constructor with "SOFT" defined for armABI in case of FPU emulation.

Can someone more familiar with the build system suggest a patch which I can test?

comment:8 Changed 7 months ago by bgamari

glaubitz, see Phab:D3519 for a sketch of such a patch.

comment:9 in reply to:  8 Changed 7 months ago by glaubitz

Replying to bgamari:

glaubitz, see Phab:D3519 for a sketch of such a patch.

Thanks. This is pretty much what I had in my head, perfect!

Since I am building unregisterised, it takes a few days to get GHC build on this embedded CPU, but I will definitely test and report back.

comment:10 Changed 7 months ago by glaubitz

Correction: My previous explanation in comment 7 contains a mistake. It's not AdjustorAsm.S which needs to be disabled but StgCRun.S as it also becomes obvious from error messages listed in the original message to this bug report.

The reason why I suspected AdjustorAsm.S is because I commented out the whole block around 'ifneq "$(PORTING_HOST)" "YES"' and just read the following condition 'ifneq "$(findstring $(TargetArch_CPP), powerpc64le)" ""' as "match for powerpc64le only' but, of course, it's the other way around. It matches for all architectures except powerpc64le. Furthermore, AdjustorAsm.S doesn't even contain any code which is built on powerpc32/linux, just powerpc32 code which builds on non-Linux targets.

Thus, could you move the guarding 'ifneq "$(POWERPC_NO_FPRS)" "YES" ... endif' away from AdjustorAsm.S and put it around StgCRun.S? This is also what I did for the suggested patch for the Debian package [1].

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861806

comment:11 in reply to:  5 ; Changed 7 months ago by trommler

Component: Compiler (NCG)Build System

Replying to glaubitz:

That's exactly the problem. The inline assembly is there and used even when "--enable-unregisterised" was passed to "configure". Apparently, it just checks for "powerpc" and ignores the "--enable-unregistrised".

I wonder why the assembly code in StgCRun.c causes a problem here. In an unregisterised compiler USE_MINIINTERPRETER is defined in includes/ghc.mk and ordinary C code should be used for StgRun() and StgReturn(). I don't see that define in the command lines in the ticket. To be sure we have an issue with the build system, could you clean your source tree and configure the compiler with --enable-unregisterised and then compile again.

BTW StgCRunAsm.S should not matter in your case at all. It has code only for AIX and PowerPC 64-bit Little Endian. It does not, however, handle USE_MINIINTERPRETER which is incorrect on AIX and 64-bit LE. I will post a patch for StgCRunAsm.S on Phab shortly.

comment:12 in reply to:  11 Changed 7 months ago by trommler

Replying to trommler:

BTW StgCRunAsm.S should not matter in your case at all. It has code only for AIX and PowerPC 64-bit Little Endian. It does not, however, handle USE_MINIINTERPRETER which is incorrect on AIX and 64-bit LE. I will post a patch for StgCRunAsm.S on Phab shortly.

The patch is Phab:D3540.

comment:13 in reply to:  11 ; Changed 7 months ago by glaubitz

Replying to trommler:

That's exactly the problem. The inline assembly is there and used even when "--enable-unregisterised" > BTW StgCRunAsm.S should not matter in your case at all. It has code only for AIX and PowerPC 64-bit Little Endian. It does not, however, handle USE_MINIINTERPRETER which is incorrect on AIX and 64-bit LE. I will post a patch for StgCRunAsm.S on Phab shortly.

Hmm, you're right. I was pretty though I was building with --enable-unregisterised. I will recheck the issue and report back. But, please bear with me as building on the embedded system takes some time with an unregisterised build.

comment:14 in reply to:  13 Changed 7 months ago by trommler

Replying to glaubitz:

Hmm, you're right. I was pretty though I was building with --enable-unregisterised. I will recheck the issue and report back. But, please bear with me as building on the embedded system takes some time with an unregisterised build.

Building on a 14 years old PowerMac G5 is also pretty slow :-)

If you see a failure again could you compile StgCRun.c and HeapStackCheck.cmm with an additional -v3 flag added to the command line so we can check ghc is passing the correct set of options to gcc.

comment:15 Changed 7 months ago by bgamari

Differential Rev(s): Phab:D3540
Status: infoneededpatch

comment:16 in reply to:  15 Changed 7 months ago by trommler

Status: patchinfoneeded

Replying to bgamari:

Note: Phab:D3540 does not fix @glaubitz' issue in this ticket but fixes a bug that I discovered while I investigated the bug on powerpc64le Linux. AIX is also affected by the same bug, I think.

As I said in comment:11 no inline assembly from StgCRun.c nor any code from StgCRunAsm.S should be passed to the assembler on a powerpc (32-bit) configured with --enable-unregsisterised.

I also don't understand why HeapStackCheck.cmm fails. It does not contain any inline assembly. I wonder whether ghc does not pass the options required for @glaubitz' system (soft float implementation) to gcc.

I set status infoneeded again while we are waiting for the input requested in comment:11 and comment:14.

Note: See TracTickets for help on using tickets.