Concurrent executions of ghc-pkg can cause inconstant package.cache files

changed weight to 5

added Tbug Trac import package system labels

The comment "Big fat hairy race condition" might have something to do with this.

Replying to [ticket:8591#comment:91364 thomie]:

The comment "Big fat hairy race condition" might have something to do with this.

That comment is in the MINGW32 host or target branch of the #if, dealing with the case where renameFile can't rename over an existing file on the Win32 platform. The real race is depending on renameFile at all, as in the #else case of of that #if, as is the case on the platform I'm using, FreeBSD.

The "atomic property" referred to in the comment here is for single-process execution, the race I'm talking about is the race between two processes to call renameFile.

A better approach would be for the process to open the targetFile with O_EXCL (and create it with O_CREAT if it doesn't exist), and hold it open while doing the atomic rename on top of the target. That way the targetFile is used a lock to synchronise updates between concurrent processes.

assigned to @trac-pgj

Wow, I did not even know that such ticket exists. Sorry for missing this.

Replying to [ticket:8591#comment:91366 janm]:

A better approach would be for the process to open the targetFile with O_EXCL (and create it with O_CREAT if it doesn't exist), and hold it open while doing the atomic rename on top of the target. That way the targetFile is used a lock to synchronise updates between concurrent processes.

Does the same problem persist with GHC 7.8.x? Unfortunately, I do not own such a beefy hardware, although I may be able to get some help in that respect from the FreeBSD Project. What is the version of the operating system (that is, ideally OSVERSION)? I guess the architecture is amd64.

I am running a test build now but code inspection shows that the same race is present. I expect it to fail.

We resolved the issue by single-threading all package installation in the build process. We are running on 9.3-p5 at the moment -- It was probably 9.2-RELEASE or 9-STABLE at the time of the original bug report.

I also see that the FreeBSD port for lang/ghc limits concurrency to 4 jobs, added in SVN rev 348842. Assuming you're also pgj@freebsd you committed the change, so probably know! This was in response to FreeBSD bug ports/186829 (see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186829 ).

The same problem seems to be the underlying cause. The error trace in the bug report shows cabal failing to install a package, just like I saw in my system build processes. Resolving the problem in ghc-pkg should resolve this problem and allow lang/ghc builds to run with full parallelism.

In principle, the withFileAtomic function should (in order):

take a lock on the target file (creating it if it is not present)
create the temporary file
write to the temporary file
close the temporary file
rename the temporary file on top of the locked target file
close the original target file which is now no longer present in the file system.

Concurrent execution will now be serialised around the lock on the target file.

This doesn't resolve the "big hairy race condition" on Windows, but it should resolve the big hairy race condition I'm hitting that isn't mentioned code comments.

Not reading this thread closely, but since there's also a comment that says "copied from Cabal's Distribution.Simple.Utils", there might be a function in there now that does exactly what you need: https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Utils.hs

Thanks for the reply. A quick code inspection of the renameFile function shows the same race. The function is atomic for single threaded execution (ie. all or nothing for the updated file), but updates can be lost when there is concurrent execution of two processes.

Replying to [ticket:8591#comment:91606 janm]:

I am running a test build now but code inspection shows that the same race is present. I expect it to fail.

All right, excellent, thanks.

I also see that the FreeBSD port for lang/ghc limits concurrency to 4 jobs, added in SVN rev 348842. Assuming you're also pgj@freebsd you committed the change, so probably know! This was in response to FreeBSD bug ports/186829.

Well, to be honest, I did not realize that it was a related issue. In the past, the GHC build system already had problems with parallel builds on FreeBSD due to e.g. low-resolution timestamps in the file system.

Replying to [ticket:8591#comment:91610 thomie]:

Not reading this thread closely, but since there's also a comment that says "copied from Cabal's Distribution.Simple.Utils", there might be a function in there now that does exactly what you need: https://github.com/haskell/cabal/blob/master/Cabal/Distribution/Simple/Utils.hs

Yeah, I know that -- this is the issue that also causes me headaches on my Windows builders recently.

Replying to [ticket:8591#comment:91624 pgj]:

Replying to [ticket:8591#comment:91606 janm]:

I am running a test build now but code inspection shows that the same race is present. I expect it to fail.

All right, excellent, thanks.

Interestingly enough it didn't fail. I suspect that it is because we have switch to pkg from the old-style packaging system on FreeBSD. We have local modifications to retry database access due to contention during package installation. This may be serialising execution of the Haskell installation processes.

I will remove the JOBS limit in lang/ghc and rerun -- that should fail because there is no pkg database.

janm: are you installing the packages with cabal or manually?

The GHC build system (in ghc.mk) contains the following comment:

# register the boot packages in strict sequence, because running
# multiple ghc-pkgs in parallel doesn't work (registrations may get
# lost).

Presumably cabal -j also registers the packages sequentially, or there'd be many more reported issues.

Changing just withFileAtomic as you propose in ticket:8591#comment:91366 and ticket:8591#comment:91606 wouldn't solve the problem. Two concurrent ghc-pkg processes still wouldn't know about each other's modifications to the package database, so the last one to call withFileAtomic (edit: to write the cache file) would still win.

Trac metadata

Trac field	Value
Related	→ #10205 (closed)
Operating system	FreeBSD → Unknown/Multiple

added Pnormal label

I believe this was addressed in #13194.

closed

mentioned in issue #21311

Trac field	Value
Version	7.6.3
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Package system
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system	FreeBSD
Architecture

Concurrent executions of ghc-pkg can cause inconstant package.cache files

Child items ...

Activity

Concurrent executions of ghc-pkg can cause inconstant package.cache files

Relates to

Activity