Broken tests on Windows

I'll be marking these as broken:

Unexpected results from:
TEST="T10598_TH T11072gcc T12530 T13618 T15324 T15858 T15904 T2276_ghci T7477 T8759 T9086 TH_NestedSplices dynbrk004 plugin-recomp-change plugin-recomp-flags plugin-recomp-impure plugins09 plugins10 plugins11 print009 print012 print023 result001"

SUMMARY for test run started at Mon Jan  7 01:57:14 2019 GST
 1:57:17 spent to go through
    6747 total tests, which gave rise to
   19416 test cases, of which
   12221 were skipped

      47 had missing libraries
    6945 expected passes
     180 expected failures

       0 caused framework failures
       0 caused framework warnings
       0 unexpected passes
      23 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   ffi/should_run/T2276_ghci.run        T2276_ghci [bad exit code] (ghci)
   ghci/linking/dyn/T11072gcc.run       T11072gcc [bad exit code] (normal)
   ghc-e/should_run/T9086.run           T9086 [bad exit code] (normal)
   ghci.debugger/scripts/print009.run   print009 [bad exit code] (ghci-ext)
   ghci.debugger/scripts/print012.run   print012 [bad exit code] (ghci-ext)
   ghci.debugger/scripts/print023.run   print023 [bad exit code] (ghci-ext)
   hp2ps/T15904.run                     T15904 [bad exit code] (normal)
   ghci.debugger/scripts/dynbrk004.run  dynbrk004 [bad exit code] (ghci-ext)
   ghci.debugger/scripts/result001.run  result001 [bad exit code] (ghci-ext)
   plugins/plugin-recomp-change.run     plugin-recomp-change [bad exit code] (normal)
   plugins/T15858.run                   T15858 [bad exit code] (ghci)
   th/TH_NestedSplices.run              TH_NestedSplices [exit code non-0] (normal)
   th/T7477.run                         T7477 [exit code non-0] (ext-interp)
   th/T8759.run                         T8759 [exit code non-0] (ext-interp)
   th/T10598_TH.run                     T10598_TH [exit code non-0] (normal)
   th/T12530.run                        T12530 [exit code non-0] (normal)
   th/T13618.run                        T13618 [exit code non-0] (normal)
   th/T15324.run                        T15324 [exit code non-0] (ext-interp)
   plugins/plugins09.run                plugins09 [bad stdout] (normal)
   plugins/plugins10.run                plugins10 [bad stdout] (normal)
   plugins/plugins11.run                plugins11 [bad stdout] (normal)
   plugins/plugin-recomp-impure.run     plugin-recomp-impure [bad exit code] (normal)
   plugins/plugin-recomp-flags.run      plugin-recomp-flags [bad exit code] (normal)

Trac metadata

Trac field	Value
Version	8.6.3
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Compiler
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system	Unknown/Multiple
Architecture

mentioned in issue #14271 (closed)

changed weight to 5

added Tbug Trac import labels

Trac metadata

Trac field	Value
Operating system	Unknown/Multiple → Windows

Actually, it looks like most of these merely timed out. Specifically:

Test	Reason
`foreignInterruptible(ghci)`	saw exit code 1, expected 0
`T2276_ghci`	timed out
`T11072gcc`	see #16157
`T9086(normal)`	timed out
`print009(ghci-ext)`	timed out
`print012(ghci-ext)`	timed out
`print023(ghci-ext)`	timed out
`T15904(normal)`	#16158
`dynbrk004(ghci-ext)`	timed out
`result001(ghci-ext)`	timed out
`plugin-recomp-change(normal)`	#16159
`T15858(ghci)`	timed out
`heapprof001(ghci-ext-prof)`	saw exit code 1, expected 0
`scc001(ghci-ext-prof)`	saw exit code 1, expected 0
`scc002(ghci-ext-prof)`	saw exit code 1, expected 0
`scc003(ghci-ext-prof)`	saw exit code 1, expected 0
`scc005(ghci-ext-prof)`	saw exit code 1, expected 0
`T680(ghci-ext-prof)`	saw exit code 1, expected 0
`T2552(ghci-ext-prof)`	saw exit code 1, expected 0
`prof-doc-fib(ghci-ext-prof)`	saw exit code 1, expected 0
`prof-doc-last(ghci-ext-prof)`	saw exit code 1, expected 0
`T5559(ghci-ext-prof)`	saw exit code 1, expected 0
`T5363(ghci-ext-prof)`	saw exit code 1, expected 0
`profinline001(ghci-ext-prof)`	saw exit code 1, expected 0
`T12962(ghci-ext-prof)`	saw exit code 1, expected 0
`TH_NestedSplices(normal)`	#16160
`T7477(ext-interp)`	timed out
`T8759(ext-interp)`	timed out
`T10598_TH(normal)`	timed out
`T12530(normal)`	timed out
`T13618(normal)`	timed out
`T15324(ext-interp)`	timed out
`plugin-recomp-impure(normal)`	#16161
`plugin-recomp-flags(normal)`	#16159

mentioned in issue #16162

I've opened #16162 to track the many external-interpreter issues.

Did you try rerunning the testsuite again with only these tests single threaded @bgamari? There shouldn't be this many failing, but the testsuite has been exhibiting threading related issues for a while now.

Hasn't been very high on my list to look at I must admit... Should probably do..

I ran a vanilla build of on a fresh Windows instance and saw the following failures:

Unexpected results from:
TEST="T15904 plugin-recomp-flags plugin-recomp-impure plugins09 plugins10 plugins11"

SUMMARY for test run started at 01/09/19 21:10:29 Greenwich Standard Time
 0:14:31 spent to go through
      23 total tests, which gave rise to
      67 test cases, of which
      32 were skipped

       0 had missing libraries
      29 expected passes
       0 expected failures

       0 caused framework failures
       0 caused framework warnings
       0 unexpected passes
       6 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   hp2ps/T15904.run                  T15904 [bad exit code] (normal)
   plugins/plugins09.run             plugins09 [bad exit code] (normal)
   plugins/plugins10.run             plugins10 [bad exit code] (normal)
   plugins/plugins11.run             plugins11 [bad stdout] (normal)
   plugins/plugin-recomp-impure.run  plugin-recomp-impure [bad stdout] (normal)
   plugins/plugin-recomp-flags.run   plugin-recomp-flags [bad exit code] (normal)

Consequently I suspect few of these failures reported in the ticket description are real.

Phyx, you mean running the testsuite driver with multiple threads has caused threads to fail? I'll admit I haven't tried running it without threading but I'm not sure we can really afford to do so; the testsuite runs already take altogether far too long on Windows.

@bgamari yes, at least on mine and Simon's machines. A concurrency issue was introduced somewhere which causes more tests to fail when running in parallel. I was mostly wondering if these failures were caused by that as well as it's an indication that it's not the tests themselves that are broken.

added Pnormal label

Most of these are either fixed or marked as broken due to other tickets. Closing.

closed