Application (warp) server crashing periodically with "TSO object entered"

We are seeing one of our warp servers periodically crash with the following error:

Apr 06 10:44:28 stdouterr.log: binary-name-elided: internal error: TSO object entered!
Apr 06 10:44:28 stdouterr.log:     (GHC version 8.0.2 for x86_64_unknown_linux)

We are sometimes seeing CPU/memory spikes concurrently with this error, but not always.

We are running very similar servers for other applications and they are not experiencing this error. But this server is one of our highest-load ones.

Other tickets I found with this error:

• #12383
• #8316

But these seem to be happening in ghci and ghc itself, not in a binary compiled by ghc.

Questions I have:

• What exactly is this error?
• Is there anything else we can do on our side to help debug? Any artifacts that should be produced upon crash?
• Should we consider upgrading our GHC version from 8.0.2 to get a bugfix?

Trac metadata

Trac field	Value
Version	8.0.2
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Runtime System
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

changed milestone to %8.4.3

changed weight to 5

added RTS Tbug Trac import labels

The error is an internal error in the runtime system or compiled code, and doesn't tell us very much unfortunately. A wide variety of different kinds of problems can lead to this symptom. I'm not aware of any known bugs in this area.

To narrow this down we'll need to be able to reproduce it and debug it. Anything you could do to produce a standalone reproduction for the bug would be really helpful. Failing that if you could provide a binary and a core dump, that might also let us dig around and get more information.

Just saw the error happen again. I'll investigate the server to see if I can find any more artifacts.

We did find some extra logs in messages:

Apr 12 09:43:07 stdouterr.log:  binary-name-elided: internal error: TSO object entered!
Apr 12 09:43:07 stdouterr.log:      (GHC version 8.0.2 for x86_64_unknown_linux)
Apr 12 09:43:07 stdouterr.log:      Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
Apr 12 09:43:07 messages:  Apr 12 16:43:07 ip-10-10-28-23 kernel: [413908.905615] traps: ghc_worker[23036] general protection ip:7f5b72a24196 sp:7f5b64de2c10 error:0
Apr 12 09:43:07 messages:  Apr 12 16:43:07 ip-10-10-28-23 kernel: [413908.911110]  in libc-2.23.so[7f5b729ed000+1c0000]

There was also an associated spike in GHC GC memory use (+ also Docker container memory use due to that): https://i.imgur.com/1RiCPuP.png

The spike begins around the time of these error messages, and then seems to self-resolve a little while later (maybe due to the Docker container restarting?)

The blue line is current_bytes_used as reported by GHC RTS metrics

Can you offer any information about the nature of your application? Does it use concurrency? STM? Make heavy use of exceptions? Any additional information would be helpful.

Of course, a core dump would also be extremely helpful.

changed milestone to %8.6.1

changed milestone to %8.8.1

These won't be addressed in GHC 8.6.

changed milestone to %8.10.1

Bumping milestones of low-priority tickets.

I hit something similar last night. Unfortunately I don't have a way to make a repro but will provide what information I do have in case it helps eventually. Our app is a server, although not a web server. We make extensive use of the STM and there may be hundreds of threads running at a time. I would say there are a moderate amount of exceptions, mostly forced on us from their use in other libraries, as we tend not to use them. Happy to provide more specific info if I can.

App was started with the following parameters: +RTS -N -qg1 -A5m

added runtime crash label

added Pnormal label

Hard to debug this without a reproducer.

added info needed label

added Phigh label and removed Pnormal label

removed milestone