Opened 3 years ago

Closed 2 years ago

Last modified 23 months ago

#9389 closed bug (fixed)

Full Test Suite Failures

Reported by: jrp Owned by:
Priority: normal Milestone: 8.0.1
Component: Test Suite Version: 7.8.3
Keywords: Cc:
Operating System: MacOS X Architecture: Unknown/Multiple
Type of failure: Other Test Case:
Blocked By: Blocking:
Related Tickets: #9286,#8756,#7876,#7877 Differential Rev(s):
Wiki Page:

Description

This is just to record progress on running the full test suite on OS X for the current HEAD using a BuildFlavour = perf build.

I'll check through these (some of which are configuration issues because of the non-standard build or some required libraries are not being pulled in).

OVERALL SUMMARY for test run started at Thu Jul 31 23:03:39 2014 BST
 0:32:31 spent to go through
    4036 total tests, which gave rise to
   19262 test cases, of which
    4218 were skipped

     358 had missing libraries
   14485 expected passes
     135 expected failures

       0 caused framework failures
       0 unexpected passes
      66 unexpected failures

Unexpected failures:
   ../../libraries/base/tests    enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum03 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests    signals004 [bad exit code] (threaded2)
   concurrent/should_run         T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run         conc012 [bad exit code] (ghci)
   driver/objc                   objc-hi [bad profile] (profthreaded)
   driver/objc                   objc-hi [bad heap profile] (profasm)
   driver/objc                   objcpp-hi [bad profile] (profthreaded)
   driver/objc                   objcpp-hi [bad heap profile] (profasm)
   ffi/should_run                ffi002 [bad exit code] (threaded2)
   indexed-types/should_compile  T7837 [stderr mismatch] (profasm)
   perf/compiler                 T4801 [stat too good] (normal)
   perf/haddock                  haddock.base [stat not good enough] (normal)
   perf/should_run               T9203 [stat too good] (normal)
   profiling/should_run          callstack001 [bad stdout] (prof)
   quasiquotation/qq007          qq007 [exit code non-0] (profasm)
   quasiquotation/qq008          qq008 [exit code non-0] (profasm)
   rts                           T5435_dyn_asm [bad stdout] (normal)
   rts                           T7919 [exit code non-0] (profasm,profthreaded)
   rts                           T9078 [exit code non-0] (profasm)
   rts                           derefnull [bad profile] (profasm,profthreaded)
   rts                           divbyzero [bad profile] (profasm,profthreaded)
   rts                           overflow1 [bad exit code] (hpc,optasm,profasm,threaded2,dyn,profthreaded,optllvm)
   rts                           overflow2 [bad profile] (profasm,profthreaded)
   rts                           overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile      T5550 [exit code non-0] (profasm)
   simplCore/should_compile      T7702 [exit code non-0] (profasm)

Change History (24)

comment:1 Changed 3 years ago by jrp

Today's snapshot:

OVERALL SUMMARY for test run started at Sat Aug  2 11:06:07 2014 BST
 0:30:58 spent to go through
    4042 total tests, which gave rise to
   19285 test cases, of which
    4222 were skipped

     358 had missing libraries
   14494 expected passes
     135 expected failures

       0 caused framework failures
       0 unexpected passes
      76 unexpected failures

Unexpected failures:
   ../../libraries/base/tests    enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    topHandler03 [bad exit code] (ghci)
   ../../libraries/unix/tests    signals004 [bad exit code] (threaded2)
   concurrent/should_run         T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run         T9379 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   concurrent/should_run         T9379 [bad stdout or stderr] (ghci)
   concurrent/should_run         conc012 [bad exit code] (ghci)
   driver/objc                   objc-hi [bad profile] (profthreaded)
   driver/objc                   objc-hi [bad heap profile] (profasm)
   driver/objc                   objcpp-hi [bad profile] (profthreaded)
   driver/objc                   objcpp-hi [bad heap profile] (profasm)
   ghc-api                       ghcApi [exit code non-0] (optasm)
   indexed-types/should_compile  T7837 [stderr mismatch] (profasm)
   perf/compiler                 T4801 [stat too good] (normal)
   perf/should_run               T9203 [stat too good] (normal)
   profiling/should_run          callstack001 [bad stdout] (prof)
   quasiquotation/qq007          qq007 [exit code non-0] (profasm)
   quasiquotation/qq008          qq008 [exit code non-0] (profasm)
   rts                           T5435_dyn_asm [bad stdout] (normal)
   rts                           T7919 [exit code non-0] (profasm,profthreaded)
   rts                           T9078 [exit code non-0] (profasm)
   rts                           derefnull [bad profile] (profasm,profthreaded)
   rts                           divbyzero [bad profile] (profasm,profthreaded)
   rts                           overflow1 [bad exit code] (hpc,optasm,profasm,threaded2,dyn,profthreaded,optllvm)
   rts                           overflow2 [bad profile] (profasm,profthreaded)
   rts                           overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile      T5550 [exit code non-0] (profasm)
   simplCore/should_compile      T7702 [exit code non-0] (profasm)

Some of these (eg, topHandler03, signals004) are the result of running the tests in parallel (or perhaps running against bindist). Running the failing tests against the stage 2 compiler results in:

OVERALL SUMMARY for test run started at Sat Aug  2 11:45:30 2014 BST
 0:09:55 spent to go through
    4068 total tests, which gave rise to
   19536 test cases, of which
   19333 were skipped

       0 had missing libraries
     138 expected passes
       2 expected failures

       0 caused framework failures
       0 unexpected passes
      63 unexpected failures

Unexpected failures:
   ../../libraries/base/tests    enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests    enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
   ../../libraries/base/tests    enum03 [bad stdout or stderr] (ghci)
   concurrent/should_run         T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run         conc012 [bad exit code] (ghci)
   driver/objc                   objc-hi [bad profile] (profthreaded)
   driver/objc                   objc-hi [bad heap profile] (profasm)
   driver/objc                   objcpp-hi [bad profile] (profthreaded)
   driver/objc                   objcpp-hi [bad heap profile] (profasm)
   indexed-types/should_compile  T7837 [stderr mismatch] (profasm)
   perf/compiler                 T4801 [stat too good] (normal)
   perf/should_run               T9203 [stat too good] (normal)
   profiling/should_run          callstack001 [bad stdout] (prof)
   quasiquotation/qq007          qq007 [exit code non-0] (profasm)
   quasiquotation/qq008          qq008 [exit code non-0] (profasm)
   rts                           T5435_dyn_asm [bad stdout] (normal)
   rts                           T7919 [exit code non-0] (profasm,profthreaded)
   rts                           T9078 [exit code non-0] (profasm)
   rts                           derefnull [bad profile] (profasm,profthreaded)
   rts                           divbyzero [bad profile] (profasm,profthreaded)
   rts                           overflow1 [bad exit code] (hpc,optasm,profasm,threaded2,dyn,profthreaded,optllvm)
   rts                           overflow2 [bad profile] (profasm,profthreaded)
   rts                           overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile      T5550 [exit code non-0] (profasm)
   simplCore/should_compile      T7702 [exit code non-0] (profasm)

Last edited 3 years ago by jrp (previous) (diff)

comment:2 Changed 3 years ago by jrp

Today's results from a fast bindist build.

Unexpected results from:
TEST="T367_letnoescape T9379 conc012 overflow1 T5435_dyn_asm T5321FD T5030 T4801 T6048 T5631 T783 T5642 T9020 T3064 parsing001 T1969 T5321Fun T5837 T3294 cabal06 enum01 topHandler03 enum03 enum02"

OVERALL SUMMARY for test run started at Sun Aug 10 11:25:55 2014 BST
 0:32:58 spent to go through
    4059 total tests, which gave rise to
   15828 test cases, of which
    3379 were skipped

     299 had missing libraries
   11938 expected passes
     153 expected failures

       3 caused framework failures
       0 unexpected passes
      56 unexpected failures

Unexpected failures:
   ../../libraries/base/tests  enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  topHandler03 [bad exit code] (ghci)
   cabal/cabal06               cabal06 [bad stdout] (normal)
   concurrent/should_run       T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run       T9379 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   concurrent/should_run       T9379 [bad stdout or stderr] (ghci)
   concurrent/should_run       conc012 [bad exit code] (ghci)
   perf/compiler               T1969 [stat not good enough] (normal)
   perf/compiler               T3064 [stat not good enough] (normal)
   perf/compiler               T3294 [stat not good enough] (normal)
   perf/compiler               T4801 [stat not good enough] (normal)
   perf/compiler               T5030 [stat not good enough] (normal)
   perf/compiler               T5321FD [stat not good enough] (normal)
   perf/compiler               T5321Fun [stat not good enough] (normal)
   perf/compiler               T5631 [stat not good enough] (normal)
   perf/compiler               T5642 [stat not good enough] (normal)
   perf/compiler               T5837 [stat not good enough] (normal)
   perf/compiler               T6048 [stat not good enough] (optasm)
   perf/compiler               T783 [stat not good enough] (normal)
   perf/compiler               T9020 [stat not good enough] (optasm)
   perf/compiler               parsing001 [stat not good enough] (normal)
   rts                         T5435_dyn_asm [bad stdout] (normal)
   rts                         overflow1 [bad exit code] (hpc,optasm,threaded2,dyn,optllvm)

Some of these tests pass if I run fulltst with the build tree, rather than bindist, version:

Unexpected results from:
TEST="T367_letnoescape conc012 overflow1 T5435_dyn_asm T5321FD T5030 T4801 T6048 T5631 T5837 T5642 T9020 T3064 parsing001 T1969 T5321Fun T783 T3294 cabal06 enum01 topHandler03 enum03 enum02"

OVERALL SUMMARY for test run started at Sun Aug 10 12:02:27 2014 BST
 0:07:21 spent to go through
    4085 total tests, which gave rise to
   16029 test cases, of which
   15949 were skipped

       0 had missing libraries
      32 expected passes
       0 expected failures

       0 caused framework failures
       0 unexpected passes
      48 unexpected failures

Unexpected failures:
   ../../libraries/base/tests  enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  topHandler03 [bad exit code] (ghci)
   cabal/cabal06               cabal06 [bad stdout] (normal)
   concurrent/should_run       T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run       conc012 [bad exit code] (ghci)
   perf/compiler               T1969 [stat not good enough] (normal)
   perf/compiler               T3064 [stat not good enough] (normal)
   perf/compiler               T3294 [stat not good enough] (normal)
   perf/compiler               T4801 [stat not good enough] (normal)
   perf/compiler               T5030 [stat not good enough] (normal)
   perf/compiler               T5321FD [stat not good enough] (normal)
   perf/compiler               T5321Fun [stat not good enough] (normal)
   perf/compiler               T5631 [stat not good enough] (normal)
   perf/compiler               T5642 [stat not good enough] (normal)
   perf/compiler               T5837 [stat not good enough] (normal)
   perf/compiler               T6048 [stat not good enough] (optasm)
   perf/compiler               T783 [stat not good enough] (normal)
   perf/compiler               T9020 [stat not good enough] (optasm)
   perf/compiler               parsing001 [stat not good enough] (normal)
   rts                         T5435_dyn_asm [bad stdout] (normal)
   rts                         overflow1 [bad exit code] (hpc,optasm,threaded2,dyn,optllvm)

comment:3 Changed 3 years ago by jrp

Today's results:

Unexpected results from:
TEST="T367_letnoescape conc012 joao-circular enum01 topHandler03 enum03 enum02 hpc_markup_001 T5321FD T5030 T4801 T6048 T5631 T783 T5642 T9020 T3064 parsing001 T1969 T5321Fun T5837 T3294 cabal06 T5435_dyn_asm"

OVERALL SUMMARY for test run started at Sat Aug 16 23:05:03 2014 BST
 0:35:47 spent to go through
    4088 total tests, which gave rise to
   16032 test cases, of which
    3412 were skipped

     315 had missing libraries
   12108 expected passes
     149 expected failures

       3 caused framework failures
       0 unexpected passes
      45 unexpected failures

Unexpected failures:
   ../../libraries/base/tests             enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests             enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests             enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests             enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             topHandler03 [bad exit code] (ghci)
   ../../libraries/hpc/tests/simple/tixs  hpc_markup_001 [bad stdout] (normal)
   cabal/cabal06                          cabal06 [bad stdout] (normal)
   concurrent/should_run                  T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run                  conc012 [bad exit code] (ghci)
   perf/compiler                          T1969 [stat not good enough] (normal)
   perf/compiler                          T3064 [stat not good enough] (normal)
   perf/compiler                          T3294 [stat not good enough] (normal)
   perf/compiler                          T4801 [stat not good enough] (normal)
   perf/compiler                          T5030 [stat not good enough] (normal)
   perf/compiler                          T5321FD [stat not good enough] (normal)
   perf/compiler                          T5321Fun [stat not good enough] (normal)
   perf/compiler                          T5631 [stat not good enough] (normal)
   perf/compiler                          T5642 [stat not good enough] (normal)
   perf/compiler                          T5837 [stat not good enough] (normal)
   perf/compiler                          T6048 [stat not good enough] (optasm)
   perf/compiler                          T783 [stat not good enough] (normal)
   perf/compiler                          T9020 [stat not good enough] (optasm)
   perf/compiler                          parsing001 [stat not good enough] (normal)
   programs/joao-circular                 joao-circular [exit code non-0] (hpc)
   rts                                    T5435_dyn_asm [bad stdout] (normal)

comment:4 Changed 3 years ago by jrp

Today's results:

Unexpected results from:
TEST="T367_letnoescape conc012 T5435_dyn_asm T5321FD T5030 T4801 T6048 T5631 T783 T5642 T9020 T3064 parsing001 T1969 T5321Fun T5837 T3294 cabal06 enum01 topHandler03 enum03 enum02"

OVERALL SUMMARY for test run started at Wed Aug 27 19:57:51 2014 BST
 0:34:19 spent to go through
    4094 total tests, which gave rise to
   16045 test cases, of which
    3420 were skipped

     315 had missing libraries
   12114 expected passes
     150 expected failures

       3 caused framework failures
       0 unexpected passes
      43 unexpected failures

Unexpected failures:
   ../../libraries/base/tests  enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests  enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests  topHandler03 [bad exit code] (ghci)
   cabal/cabal06               cabal06 [bad stdout] (normal)
   concurrent/should_run       T367_letnoescape [bad exit code] (optllvm)
   concurrent/should_run       conc012 [bad exit code] (ghci)
   perf/compiler               T1969 [stat not good enough] (normal)
   perf/compiler               T3064 [stat not good enough] (normal)
   perf/compiler               T3294 [stat not good enough] (normal)
   perf/compiler               T4801 [stat not good enough] (normal)
   perf/compiler               T5030 [stat not good enough] (normal)
   perf/compiler               T5321FD [stat not good enough] (normal)
   perf/compiler               T5321Fun [stat not good enough] (normal)
   perf/compiler               T5631 [stat not good enough] (normal)
   perf/compiler               T5642 [stat not good enough] (normal)
   perf/compiler               T5837 [stat not good enough] (normal)
   perf/compiler               T6048 [stat not good enough] (optasm)
   perf/compiler               T783 [stat not good enough] (normal)
   perf/compiler               T9020 [stat not good enough] (optasm)
   perf/compiler               parsing001 [stat not good enough] (normal)
   rts                         T5435_dyn_asm [bad stdout] (normal)

comment:5 Changed 3 years ago by jrp

A significant increase in the number of failures since the last run:

Unexpected results from:
TEST="conc059 conc012 process011 T1735 TcTypeNatSimpleRun T3500b T5751 enum01 enum03 enum02 topHandler01 T7891 tc124 T5321FD T5030 T4801 T6048 T5631 T5837 T5642 T9020 T3064 parsing001 T1969 T5321Fun T783 T3294 cabal06 T5435_dyn_asm signals002"

OVERALL SUMMARY for test run started at Fri Sep 19 19:43:39 2014 BST
 0:07:28 spent to go through
    4107 total tests, which gave rise to
   16097 test cases, of which
   15983 were skipped

       0 had missing libraries
      28 expected passes
       0 expected failures

       0 caused framework failures
       0 unexpected passes
      86 unexpected failures

Unexpected failures:
   ../../libraries/base/tests     enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests     enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests     enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   ../../libraries/base/tests     enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler01 [bad exit code] (threaded1,threaded2)
   ../../libraries/base/tests     topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/process/tests  process011 [bad exit code] (threaded1,threaded2)
   ../../libraries/process/tests  process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests     signals002 [bad stdout] (threaded1)
   ../../libraries/unix/tests     signals002 [bad stdout or stderr] (ghci)
   cabal/cabal06                  cabal06 [bad stdout] (normal)
   concurrent/should_run          conc012 [bad exit code] (ghci)
   concurrent/should_run          conc059 [bad stderr] (threaded1,threaded2)
   perf/compiler                  T1969 [stat not good enough] (normal)
   perf/compiler                  T3064 [stat not good enough] (normal)
   perf/compiler                  T3294 [stat not good enough] (normal)
   perf/compiler                  T4801 [stat not good enough] (normal)
   perf/compiler                  T5030 [stat not good enough] (normal)
   perf/compiler                  T5321FD [stat not good enough] (normal)
   perf/compiler                  T5321Fun [stat not good enough] (normal)
   perf/compiler                  T5631 [stat not good enough] (normal)
   perf/compiler                  T5642 [stat not good enough] (normal)
   perf/compiler                  T5837 [stat not good enough] (normal)
   perf/compiler                  T6048 [stat not good enough] (optasm)
   perf/compiler                  T783 [stat not good enough] (normal)
   perf/compiler                  T9020 [stat not good enough] (optasm)
   perf/compiler                  parsing001 [stat not good enough] (normal)
   rts                            T5435_dyn_asm [bad stdout] (normal)
   typecheck/should_compile       T7891 [exit code non-0] (hpc,optasm,optllvm)
   typecheck/should_compile       tc124 [exit code non-0] (hpc,optasm,optllvm)
   typecheck/should_run           T1735 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   typecheck/should_run           T1735 [bad stdout or stderr] (ghci)
   typecheck/should_run           T3500b [exit code non-0] (hpc,optasm,threaded2,dyn,optllvm)
   typecheck/should_run           T5751 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   typecheck/should_run           T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run           TcTypeNatSimpleRun [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn,optllvm)
   typecheck/should_run           TcTypeNatSimpleRun [bad stdout or stderr] (ghci)

comment:6 Changed 3 years ago by thomie

comment:7 Changed 3 years ago by thomie

comment:8 Changed 3 years ago by thomie

comment:9 Changed 3 years ago by thomie

comment:10 Changed 3 years ago by merijn

./validate --fast on OSX 10.10 produced the following today:

Unexpected results from:
TEST="cabal01 T5435_dyn_asm subsections_via_symbols cabal06 T4801"

OVERALL SUMMARY for test run started at Sat Nov 22 21:35:31 2014 PST
0:07:30 spent to go through
   4230 total tests, which gave rise to
  13843 test cases, of which
   9956 were skipped

     53 had missing libraries
   3763 expected passes
     66 expected failures

      0 caused framework failures
      0 unexpected passes
      4 unexpected failures
      1 unexpected stat failures

Unexpected failures:
  cabal/cabal01                            cabal01 [bad exit code] (normal)
  cabal/cabal06                            cabal06 [bad stdout] (normal)
  llvm/should_run/subsections_via_symbols  subsections_via_symbols [bad exit code] (normal)
  rts                                      T5435_dyn_asm [bad stdout] (normal)

Unexpected stat failures:
  perf/compiler  T4801 [stat too good] (normal)

comment:11 Changed 3 years ago by jrp

Here are today's results (OS X 10.10.1):

OVERALL SUMMARY for test run started at Thu Dec  4 19:00:11 2014 GMT
 0:37:10 spent to go through
    4338 total tests, which gave rise to
   14096 test cases, of which
    2691 were skipped

     275 had missing libraries
   10905 expected passes
     145 expected failures

       3 caused framework failures
       0 unexpected passes
      62 unexpected failures
      15 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests               enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests               enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests               enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests               enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests               enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests               enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests               topHandler01 [bad exit code] (threaded1,threaded2)
   ../../libraries/base/tests               topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/process/tests            process011 [bad exit code] (threaded1,threaded2)
   ../../libraries/process/tests            process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests               signals002 [bad stdout] (threaded1,threaded2)
   ../../libraries/unix/tests               signals002 [bad stdout or stderr] (ghci)
   cabal/cabal01                            cabal01 [bad exit code] (normal)
   cabal/cabal06                            cabal06 [bad stdout] (normal)
   cabal/sigcabal01                         sigcabal01 [bad stderr] (normal)
   concurrent/should_run                    allocLimit4 [bad exit code] (ghci)
   concurrent/should_run                    conc012 [bad exit code] (ghci)
   concurrent/should_run                    conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run                       dsrun014 [bad exit code] (ghci)
   ffi/should_run                           T2276_ghci [bad stdout or stderr] (ghci)
   ghci/prog003                             prog003 [bad stderr] (ghci)
   ghci/scripts                             T8696 [bad stderr] (ghci)
   ghci/scripts                             ghci024 [bad stdout] (normal)
   ghci/scripts                             ghci058 [bad stderr] (ghci)
   llvm/should_run/subsections_via_symbols  subsections_via_symbols [bad exit code] (normal)
   numeric/should_run                       add2 [bad exit code] (ghci)
   numeric/should_run                       mul2 [bad exit code] (ghci)
   numeric/should_run                       quotRem2 [bad exit code] (ghci)
   parser/should_run                        readRun004 [bad exit code] (ghci)
   rts                                      T5435_dyn_asm [bad stdout] (normal)
   rts                                      T7919 [bad exit code] (optasm,dyn)
   typecheck/should_compile                 T7891 [exit code non-0] (hpc,optasm)
   typecheck/should_compile                 tc124 [exit code non-0] (hpc,optasm)
   typecheck/should_run                     T7861 [bad exit code] (normal,hpc,optasm,ghci,threaded1,threaded2,dyn)

Unexpected stat failures:
   perf/compiler  T1969 [stat not good enough] (normal)
   perf/compiler  T3064 [stat not good enough] (normal)
   perf/compiler  T3294 [stat not good enough] (normal)
   perf/compiler  T4801 [stat not good enough] (normal)
   perf/compiler  T5030 [stat not good enough] (normal)
   perf/compiler  T5321FD [stat not good enough] (normal)
   perf/compiler  T5321Fun [stat not good enough] (normal)
   perf/compiler  T5631 [stat not good enough] (normal)
   perf/compiler  T5642 [stat not good enough] (normal)
   perf/compiler  T5837 [stat not good enough] (normal)
   perf/compiler  T6048 [stat not good enough] (optasm)
   perf/compiler  T783 [stat not good enough] (normal)
   perf/compiler  T9020 [stat not good enough] (optasm)
   perf/compiler  T9675 [stat not good enough] (optasm)
   perf/compiler  parsing001 [stat not good enough] (normal)

and ./validate --fast gives

OVERALL SUMMARY for test run started at Thu Dec  4 20:00:43 2014 GMT
 0:05:17 spent to go through
    4338 total tests, which gave rise to
   14092 test cases, of which
   10097 were skipped

      53 had missing libraries
    3859 expected passes
      70 expected failures

       0 caused framework failures
       0 unexpected passes
      11 unexpected failures
       2 unexpected stat failures

Unexpected failures:
   cabal/cabal01                            cabal01 [bad exit code] (normal)
   cabal/cabal06                            cabal06 [bad stdout] (normal)
   cabal/sigcabal01                         sigcabal01 [bad stderr] (normal)
   ffi/should_run                           T2276_ghci [bad stdout or stderr] (ghci)
   ghci/prog003                             prog003 [bad stderr] (ghci)
   ghci/scripts                             T8696 [bad stderr] (ghci)
   ghci/scripts                             ghci058 [bad stderr] (ghci)
   ghci/should_run                          T2589 [bad stdout or stderr] (ghci)
   ghci/should_run                          ghcirun004 [bad stdout or stderr] (ghci)
   llvm/should_run/subsections_via_symbols  subsections_via_symbols [bad exit code] (normal)
   rts                                      T5435_dyn_asm [bad stdout] (normal)

Unexpected stat failures:
   perf/compiler  T4801 [stat too good] (normal)
   perf/compiler  T9675 [stat not good enough] (optasm)

comment:12 Changed 3 years ago by nomeata

What are the settings for the first set of results? Note that performance test results are only valid for the default validate settings, which are *not* the default or the release settings (as discussed in #9315).

The others are definitely relevant, but not very helpful without full build logs. How about you create a git repo somewhere where you add your build logs, then you can link them from her, like https://github.com/nomeata/ghc-speed-logs.

comment:13 Changed 3 years ago by jrp

I am using the default build file with the recommended quick selected. I'll have a look at the links and putting up a github repo.

Last edited 3 years ago by jrp (previous) (diff)

comment:14 Changed 3 years ago by jrp

Today, quick build gives (sorry, no github repo yet):

OVERALL SUMMARY for test run started at Thu Dec 11 20:47:57 2014 GMT
 0:41:48 spent to go through
    4359 total tests, which gave rise to
   14160 test cases, of which
    2699 were skipped

     275 had missing libraries
   10958 expected passes
     145 expected failures

       3 caused framework failures
       0 unexpected passes
      62 unexpected failures
      18 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests     enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler01 [bad exit code] (threaded1)
   ../../libraries/base/tests     topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/process/tests  process011 [bad exit code] (threaded1,threaded2)
   ../../libraries/process/tests  process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests     forkprocess01 [bad exit code] (ghci)
   ../../libraries/unix/tests     signals002 [bad stdout] (threaded1,threaded2)
   ../../libraries/unix/tests     signals002 [bad stdout or stderr] (ghci)
   cabal/cabal01                  cabal01 [bad exit code] (normal)
   cabal/sigcabal01               sigcabal01 [bad stderr] (normal)
   codeGen/should_run             CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run          allocLimit4 [bad exit code] (ghci)
   concurrent/should_run          conc012 [bad exit code] (ghci)
   concurrent/should_run          conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run             DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run             dsrun014 [bad exit code] (ghci)
   ffi/should_run                 T2276_ghci [bad stdout or stderr] (ghci)
   ghci.debugger/scripts          print021 [bad exit code] (ghci)
   ghci/prog003                   prog003 [bad stderr] (ghci)
   ghci/scripts                   T8696 [bad stderr] (ghci)
   ghci/scripts                   ghci024 [bad stdout] (normal)
   ghci/scripts                   ghci058 [bad stderr] (ghci)
   numeric/should_run             add2 [bad exit code] (ghci)
   numeric/should_run             mul2 [bad exit code] (ghci)
   numeric/should_run             quotRem2 [bad exit code] (ghci)
   parser/should_run              readRun004 [bad exit code] (ghci)
   rts                            GcStaticPointers [bad stdout or stderr] (ghci)
   rts                            ListStaticPointers [bad stdout or stderr] (ghci)
   rts                            T7919 [bad exit code] (optasm,threaded2,dyn)
   th                             TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_run           T7861 [bad exit code] (normal,hpc,optasm,ghci,threaded1,threaded2,dyn)

An unadorned validate gives:

OVERALL SUMMARY for test run started at Thu Dec 11 22:03:04 2014 GMT
 0:05:02 spent to go through
    4359 total tests, which gave rise to
   14156 test cases, of which
   10141 were skipped

      53 had missing libraries
    3884 expected passes
      70 expected failures

       0 caused framework failures
       0 unexpected passes
       6 unexpected failures
       2 unexpected stat failures

Unexpected failures:
   cabal/cabal01     cabal01 [bad exit code] (normal)
   cabal/sigcabal01  sigcabal01 [bad stderr] (normal)
   ffi/should_run    T2276_ghci [bad stdout or stderr] (ghci)
   ghci/prog003      prog003 [bad stderr] (ghci)
   ghci/scripts      T8696 [bad stderr] (ghci)
   ghci/scripts      ghci058 [bad stderr] (ghci)

Unexpected stat failures:
   perf/compiler  T4801 [stat too good] (normal)
   perf/haddock   haddock.base [stat not good enough] (normal)

comment:15 Changed 3 years ago by jrp

An unadorned validate gives

Unexpected results from:
TEST="sigcabal01"

OVERALL SUMMARY for test run started at Mon Jan  5 22:57:44 2015 GMT
 0:05:44 spent to go through
    4388 total tests, which gave rise to
   14199 test cases, of which
   10157 were skipped

      52 had missing libraries
    3919 expected passes
      70 expected failures

       0 caused framework failures
       0 unexpected passes
       1 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   cabal/sigcabal01  sigcabal01 [bad stderr] (normal)

Cool. I'll put up full results tomorrow.

Last edited 3 years ago by jrp (previous) (diff)

comment:16 Changed 3 years ago by jrp

The result of make fulltest THREADS=8 with BuildFlavour = quick

OVERALL SUMMARY for test run started at Tue Jan  6 20:00:16 2015 GMT
 0:45:55 spent to go through
    4389 total tests, which gave rise to
   14206 test cases, of which
    2703 were skipped

     267 had missing libraries
   10980 expected passes
     148 expected failures

       3 caused framework failures
       0 unexpected passes
      86 unexpected failures
      19 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests             enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests             enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests             enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests             enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             topHandler01 [bad exit code] (threaded1,threaded2)
   ../../libraries/base/tests             topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             topHandler03 [bad exit code] (ghci)
   ../../libraries/hpc/tests/simple/tixs  hpc_markup_001 [bad stdout] (normal)
   ../../libraries/hpc/tests/simple/tixs  hpc_markup_002 [bad stdout] (normal)
   ../../libraries/process/tests          process011 [bad exit code] (threaded1,threaded2)
   ../../libraries/process/tests          process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests             signals002 [bad stdout] (threaded1,threaded2)
   ../../libraries/unix/tests             signals002 [bad stdout or stderr] (ghci)
   cabal/sigcabal01                       sigcabal01 [bad stderr] (normal)
   codeGen/should_run                     CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run                  allocLimit4 [bad exit code] (ghci)
   concurrent/should_run                  conc012 [bad exit code] (ghci)
   concurrent/should_run                  conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run                     DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run                     dsrun014 [bad exit code] (ghci)
   driver                                 T9938 [bad exit code] (normal)
   driver/retc001                         retc001 [bad stderr] (normal)
   ghci/scripts                           ghci024 [bad stdout] (normal)
   numeric/should_run                     add2 [bad exit code] (ghci)
   numeric/should_run                     mul2 [bad exit code] (ghci)
   numeric/should_run                     quotRem2 [bad exit code] (ghci)
   parser/should_run                      readRun004 [bad exit code] (ghci)
   rts                                    GcStaticPointers [bad stdout or stderr] (ghci)
   rts                                    ListStaticPointers [bad stdout or stderr] (ghci)
   rts                                    T7919 [bad exit code] (optasm,threaded2,dyn)
   th                                     TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_compile               T9939 [stderr mismatch] (normal,hpc,optasm)
   typecheck/should_run                   T3500a [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run                   T3500a [bad stdout or stderr] (ghci)
   typecheck/should_run                   T5751 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run                   T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run                   T7126 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run                   T7126 [bad stdout or stderr] (ghci)
   typecheck/should_run                   T7861 [bad exit code] (normal,hpc,optasm,ghci,threaded1,threaded2,dyn)
   typecheck/should_run                   tcrun045 [stderr mismatch] (normal)

Unexpected stat failures:
   perf/compiler  T1969 [stat not good enough] (normal)
   perf/compiler  T3064 [stat not good enough] (normal)
   perf/compiler  T3294 [stat not good enough] (normal)
   perf/compiler  T4801 [stat not good enough] (normal)
   perf/compiler  T5030 [stat not good enough] (normal)
   perf/compiler  T5321FD [stat not good enough] (normal)
   perf/compiler  T5321Fun [stat not good enough] (normal)
   perf/compiler  T5631 [stat not good enough] (normal)
   perf/compiler  T5642 [stat not good enough] (normal)
   perf/compiler  T5837 [stat not good enough] (normal)
   perf/compiler  T6048 [stat not good enough] (optasm)
   perf/compiler  T783 [stat not good enough] (normal)
   perf/compiler  T9020 [stat not good enough] (optasm)
   perf/compiler  T9675 [stat not good enough] (optasm)
   perf/compiler  T9872a [stat not good enough] (normal)
   perf/compiler  T9872b [stat not good enough] (normal)
   perf/compiler  T9872c [stat not good enough] (normal)
   perf/compiler  T9872d [stat not good enough] (normal)
   perf/compiler  parsing001 [stat not good enough] (normal)

comment:17 Changed 3 years ago by jrp

... and here with BuildFlavour = perf

OVERALL SUMMARY for test run started at Tue Jan  6 21:42:54 2015 GMT
 0:39:46 spent to go through
    4389 total tests, which gave rise to
   17976 test cases, of which
    3602 were skipped

     328 had missing libraries
   13767 expected passes
     139 expected failures

       0 caused framework failures
       0 unexpected passes
     134 unexpected failures
       6 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests             enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests             enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests             enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests             enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             topHandler01 [bad exit code] (threaded1,threaded2,profthreaded)
   ../../libraries/base/tests             topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests             topHandler03 [bad exit code] (ghci)
   ../../libraries/hpc/tests/simple/tixs  hpc_markup_001 [bad stdout] (normal)
   ../../libraries/process/tests          process011 [bad exit code] (threaded1,threaded2,profthreaded)
   ../../libraries/process/tests          process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests             signals002 [bad stdout] (threaded2,profthreaded)
   ../../libraries/unix/tests             signals002 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests             signals004 [bad stderr] (threaded2)
   cabal/sigcabal01                       sigcabal01 [bad stderr] (normal)
   codeGen/should_run                     CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run                  T5611 [bad stderr] (normal)
   concurrent/should_run                  allocLimit4 [bad exit code] (ghci)
   concurrent/should_run                  conc012 [bad exit code] (ghci)
   concurrent/should_run                  conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run                     DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run                     dsrun014 [bad exit code] (ghci)
   driver                                 T9938 [bad exit code] (normal)
   driver/objc                            objc-hi [bad profile] (profthreaded)
   driver/objc                            objc-hi [bad heap profile] (profasm)
   driver/objc                            objcpp-hi [bad profile] (profthreaded)
   driver/objc                            objcpp-hi [bad heap profile] (profasm)
   driver/recomp001                       recomp001 [bad stderr] (normal)
   driver/retc001                         retc001 [bad stderr] (normal)
   ghci/scripts                           ghci024 [bad stdout] (normal)
   numeric/should_run                     add2 [bad exit code] (ghci)
   numeric/should_run                     mul2 [bad exit code] (ghci)
   numeric/should_run                     quotRem2 [bad exit code] (ghci)
   parser/should_run                      readRun004 [bad exit code] (ghci)
   profiling/should_run                   T2552 [bad profile] (profasm,profthreaded)
   profiling/should_run                   callstack001 [bad stdout] (prof)
   profiling/should_run                   ioprof [bad profile] (profasm,profthreaded)
   profiling/should_run                   prof-doc-fib [bad profile] (profasm,profthreaded,prof)
   profiling/should_run                   scc001 [bad profile] (prof)
   quasiquotation/qq007                   qq007 [exit code non-0] (profasm)
   quasiquotation/qq008                   qq008 [exit code non-0] (profasm)
   rts                                    GcStaticPointers [bad stdout or stderr] (ghci)
   rts                                    ListStaticPointers [bad stdout or stderr] (ghci)
   rts                                    T7919 [bad exit code] (optasm,threaded2,dyn)
   rts                                    T7919 [exit code non-0] (profasm,profthreaded)
   rts                                    T9078 [exit code non-0] (profasm)
   rts                                    derefnull [bad profile] (profasm,profthreaded)
   rts                                    divbyzero [bad profile] (profasm,profthreaded)
   rts                                    overflow1 [bad profile] (profasm,profthreaded)
   rts                                    overflow2 [bad profile] (profasm,profthreaded)
   rts                                    overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile               T5550 [exit code non-0] (profasm)
   th                                     TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_compile               T9939 [stderr mismatch] (normal,hpc,optasm,profasm)
   typecheck/should_run                   T3500a [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run                   T3500a [bad stdout or stderr] (ghci)
   typecheck/should_run                   T5751 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run                   T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run                   T7126 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run                   T7126 [bad stdout or stderr] (ghci)
   typecheck/should_run                   T7861 [bad exit code] (normal,hpc,optasm,profasm,ghci,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run                   tcrun045 [stderr mismatch] (normal)

Unexpected stat failures:
   perf/compiler             T1969 [stat not good enough] (normal)
   perf/compiler             T5642 [stat not good enough] (normal)
   perf/compiler             T6048 [stat not good enough] (optasm)
   perf/compiler             T783 [stat not good enough] (normal)
   perf/should_run           T9203 [stat too good] (normal)
   simplCore/should_compile  T7702 [stat too good] (profasm)

comment:18 Changed 3 years ago by simonpj

Very helpful, thank you.

It would be great if someone could look through these, case by case, and figure out what is going wrong. Maybe GHC is fine, and the test need adjusting. Maybe there's a bug. Maybe it's a test-harness issue. etc.

Open tickets for bugs!

Simon

comment:19 Changed 3 years ago by jrp

This week, again with the perf Buildflavour, 18 fewer failures.

OVERALL SUMMARY for test run started at Tue Jan 13 21:20:19 2015 GMT
 0:35:49 spent to go through
    4393 total tests, which gave rise to
   17970 test cases, of which
    3594 were skipped

     328 had missing libraries
   13787 expected passes
     139 expected failures

       0 caused framework failures
       0 unexpected passes
     116 unexpected failures
       6 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests     enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler01 [bad exit code] (threaded1,threaded2,profthreaded)
   ../../libraries/base/tests     topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/process/tests  process011 [bad exit code] (threaded1,threaded2,profthreaded)
   ../../libraries/process/tests  process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests     signals002 [bad stdout] (threaded1,threaded2,profthreaded)
   ../../libraries/unix/tests     signals002 [bad stdout or stderr] (ghci)
   cabal/sigcabal01               sigcabal01 [bad stderr] (normal)
   codeGen/should_run             CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run          allocLimit4 [bad exit code] (ghci)
   concurrent/should_run          conc012 [bad exit code] (ghci)
   concurrent/should_run          conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run             DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run             dsrun014 [bad exit code] (ghci)
   driver/objc                    objc-hi [bad profile] (profthreaded)
   driver/objc                    objc-hi [bad heap profile] (profasm)
   driver/objc                    objcpp-hi [bad profile] (profthreaded)
   driver/objc                    objcpp-hi [bad heap profile] (profasm)
   driver/recomp001               recomp001 [bad stderr] (normal)
   driver/retc001                 retc001 [bad stderr] (normal)
   ghci/scripts                   ghci024 [bad stdout] (normal)
   numeric/should_run             add2 [bad exit code] (ghci)
   numeric/should_run             mul2 [bad exit code] (ghci)
   numeric/should_run             quotRem2 [bad exit code] (ghci)
   parser/should_run              readRun004 [bad exit code] (ghci)
   profiling/should_run           T2552 [bad profile] (profasm,profthreaded)
   profiling/should_run           callstack001 [bad stdout] (prof)
   profiling/should_run           ioprof [bad profile] (profasm,profthreaded)
   profiling/should_run           prof-doc-fib [bad profile] (profasm,profthreaded,prof)
   profiling/should_run           scc001 [bad profile] (prof)
   quasiquotation/qq007           qq007 [exit code non-0] (profasm)
   quasiquotation/qq008           qq008 [exit code non-0] (profasm)
   rts                            GcStaticPointers [bad stdout or stderr] (ghci)
   rts                            ListStaticPointers [bad stdout or stderr] (ghci)
   rts                            T7919 [bad exit code] (optasm,dyn)
   rts                            T7919 [exit code non-0] (profasm,profthreaded)
   rts                            T9078 [exit code non-0] (profasm)
   rts                            derefnull [bad profile] (profasm,profthreaded)
   rts                            divbyzero [bad profile] (profasm,profthreaded)
   rts                            overflow1 [bad profile] (profasm,profthreaded)
   rts                            overflow2 [bad profile] (profasm,profthreaded)
   rts                            overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile       T5550 [exit code non-0] (profasm)
   th                             TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_run           T3500a [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T3500a [bad stdout or stderr] (ghci)
   typecheck/should_run           T5751 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run           T7126 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T7126 [bad stdout or stderr] (ghci)
   typecheck/should_run           tcrun045 [stderr mismatch] (normal)

Unexpected stat failures:
   perf/compiler             T1969 [stat not good enough] (normal)
   perf/compiler             T5642 [stat not good enough] (normal)
   perf/compiler             T783 [stat not good enough] (normal)
   perf/compiler             T9675 [stat not good enough] (optasm)
   perf/should_run           T9203 [stat too good] (normal)
   simplCore/should_compile  T7702 [stat too good] (profasm)

and a validate leads to a couple of regressions:

OVERALL SUMMARY for test run started at Tue Jan 13 22:36:23 2015 GMT
 0:05:53 spent to go through
    4393 total tests, which gave rise to
   14198 test cases, of which
   10151 were skipped

      52 had missing libraries
    3920 expected passes
      71 expected failures

       0 caused framework failures
       0 unexpected passes
       3 unexpected failures
       1 unexpected stat failures

Unexpected failures:
   cabal/sigcabal01       sigcabal01 [bad stderr] (normal)
   driver/retc001         retc001 [bad stderr] (normal)
   ghci.debugger/scripts  print021 [bad exit code] (ghci)

Unexpected stat failures:
   perf/compiler  T9675 [stat not good enough] (optasm)
Last edited 3 years ago by jrp (previous) (diff)

comment:20 Changed 3 years ago by jrp

And here are some more detailed diagnostics from the perf BuildFlavour

=====> sigcabal01(normal) 95 of 4394 [0, 0, 0] 
cd ./cabal/sigcabal01 && $MAKE -s --no-print-directory sigcabal01 CLEANUP=1    </dev/null >sigcabal01.run.stdout 2>sigcabal01.run.stderr
Actual stderr output differs from expected:
--- ./cabal/sigcabal01/sigcabal01.stderr	2014-12-03 23:14:05.000000000 +0000
+++ ./cabal/sigcabal01/sigcabal01.run.stderr	2015-01-14 20:45:55.000000000 +0000
@@ -1,2 +1,4 @@
 Setup: Missing signature implementations for these modules: Map
+/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: dist/build/libHSp_LUw0YzhH9mm7VMqZOQq5o8.a-51161/libHSp_LUw0YzhH9mm7VMqZOQq5o8.a(Map.o) has no symbols
+/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: dist/build/libHSp_HXXcYbYUHJg90CX6clNHdD.a-51218/libHSp_HXXcYbYUHJg90CX6clNHdD.a(Map.o) has no symbols
 Main: Prelude.undefined
*** unexpected failure for sigcabal01(normal)

(retc001 and print021 pass today)

=====> topHandler01(ghci) 4165 of 4394 [0, 0, 0] 
cd ../../libraries/base/tests && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history topHandler01.hs --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS   <topHandler01.genscript 1>topHandler01.interp.stdout 2>topHandler01.interp.stderr
Actual stderr output differs from expected:
--- /dev/null	2015-01-14 20:55:02.000000000 +0000
+++ ../../libraries/base/tests/topHandler01.run.stderr	2015-01-14 20:55:07.000000000 +0000
@@ -0,0 +1 @@
+Interrupted.
*** unexpected failure for topHandler01(ghci)
=====> topHandler01(threaded1) 4165 of 4394 [0, 1, 0] 
cd ../../libraries/base/tests && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history -o topHandler01 topHandler01.hs -threaded -debug   >topHandler01.comp.stderr 2>&1
cd ../../libraries/base/tests && ./topHandler01    </dev/null >topHandler01.run.stdout 2>topHandler01.run.stderr
Wrong exit code (expected 0 , actual 130 )
Stdout:
Success: caught UserInterrupt

Stderr:

*** unexpected failure for topHandler01(threaded1)
=====> topHandler01(threaded2) 4165 of 4394 [0, 2, 0] 
cd ../../libraries/base/tests && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history -o topHandler01 topHandler01.hs -O -threaded -eventlog   >topHandler01.comp.stderr 2>&1
cd ../../libraries/base/tests && ./topHandler01  +RTS -N2 -ls -RTS  </dev/null >topHandler01.run.stdout 2>topHandler01.run.stderr
Wrong exit code (expected 0 , actual 130 )
Stdout:
Success: caught UserInterrupt

Stderr:

*** unexpected failure for topHandler01(threaded2)
=====> T7126(normal) 4061 of 4394 [0, 0, 0] 
cd ./typecheck/should_run && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history -o T7126 T7126.hs    >T7126.comp.stderr 2>&1
Compile failed (status 256) errors were:
[1 of 1] Compiling Main             ( T7126.hs, T7126.o )

T7126.hs:27:10:
    Could not deduce (Class1 a)
      arising from the superclasses of an instance declaration
    from the context: Class3 a
      bound by the instance declaration at T7126.hs:27:10-29
    Possible fix:
      add (Class1 a) to the context of the instance declaration
    In the instance declaration for ‘Class2 a’

*** unexpected failure for T7126(normal)
=====> T5751(normal) 4063 of 4394 [0, 0, 0] 
cd ./typecheck/should_run && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history -o T5751 T5751.hs    >T5751.comp.stderr 2>&1
Compile failed (status 256) errors were:

T5751.hs:1:79: Warning:
    -XOverlappingInstances is deprecated: instead use per-instance pragmas OVERLAPPING/OVERLAPPABLE/OVERLAPS
[1 of 1] Compiling Main             ( T5751.hs, T5751.o )

T5751.hs:28:10:
    Could not deduce (XMLGenerator m)
      arising from the superclasses of an instance declaration
    from the context: MonadRender m
      bound by the instance declaration at T5751.hs:28:10-35
    Possible fix:
      add (XMLGenerator m) to the context of the instance declaration
    In the instance declaration for ‘Widgets m’

*** unexpected failure for T5751(normal)
=====> T3500a(normal) 4052 of 4394 [0, 0, 0] 
cd ./typecheck/should_run && '/Volumes/LaCie/Users/jrp/Projects/haskell/ghc/inplace/bin/ghc-stage2' -fforce-recomp -dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-history -o T3500a T3500a.hs    >T3500a.comp.stderr 2>&1
Compile failed (status 256) errors were:
[1 of 1] Compiling Main             ( T3500a.hs, T3500a.o )

T3500a.hs:11:10:
    Constraint is no smaller than the instance head
      in the constraint: C (F a)
    (Use UndecidableInstances to permit this)
    In the instance declaration for ‘C (a, b)’

*** unexpected failure for T3500a(normal)

comment:21 Changed 3 years ago by jrp

Today's results (perf BuildFlavour):

OVERALL SUMMARY for test run started at Thu Jan 22 19:36:52 2015 GMT
 0:34:34 spent to go through
    4405 total tests, which gave rise to
   18011 test cases, of which
    3603 were skipped

     328 had missing libraries
   13819 expected passes
     139 expected failures

       0 caused framework failures
       0 unexpected passes
     116 unexpected failures
       6 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests     enum01 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum02 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum03 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   ../../libraries/base/tests     enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler01 [bad exit code] (threaded1)
   ../../libraries/base/tests     topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/process/tests  process011 [bad exit code] (threaded1,threaded2,profthreaded)
   ../../libraries/process/tests  process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests     signals002 [bad stdout] (threaded1,threaded2,profthreaded)
   ../../libraries/unix/tests     signals002 [bad stdout or stderr] (ghci)
   annotations/should_run         annrun01 [bad stdout or stderr] (ghci)
   cabal/sigcabal01               sigcabal01 [bad stderr] (normal)
   codeGen/should_run             CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run          T5611 [bad stdout] (profasm)
   concurrent/should_run          allocLimit4 [bad exit code] (ghci)
   concurrent/should_run          conc012 [bad exit code] (ghci)
   concurrent/should_run          conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run             DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run             dsrun014 [bad exit code] (ghci)
   driver/objc                    objc-hi [bad profile] (profthreaded)
   driver/objc                    objc-hi [bad heap profile] (profasm)
   driver/objc                    objcpp-hi [bad profile] (profthreaded)
   driver/objc                    objcpp-hi [bad heap profile] (profasm)
   driver/retc001                 retc001 [bad stderr] (normal)
   ghci/scripts                   ghci024 [bad stdout] (normal)
   numeric/should_run             add2 [bad exit code] (ghci)
   numeric/should_run             mul2 [bad exit code] (ghci)
   numeric/should_run             quotRem2 [bad exit code] (ghci)
   parser/should_run              readRun004 [bad exit code] (ghci)
   profiling/should_run           T2552 [bad profile] (profasm,profthreaded)
   profiling/should_run           callstack001 [bad stdout] (prof)
   profiling/should_run           ioprof [bad profile] (profasm,profthreaded)
   profiling/should_run           prof-doc-fib [bad profile] (profasm,profthreaded,prof)
   profiling/should_run           scc001 [bad profile] (prof)
   quasiquotation/qq007           qq007 [exit code non-0] (profasm)
   quasiquotation/qq008           qq008 [exit code non-0] (profasm)
   rts                            GcStaticPointers [bad stdout or stderr] (ghci)
   rts                            ListStaticPointers [bad stdout or stderr] (ghci)
   rts                            T7919 [bad exit code] (optasm,dyn)
   rts                            T7919 [exit code non-0] (profasm,profthreaded)
   rts                            T9078 [exit code non-0] (profasm)
   rts                            derefnull [bad profile] (profasm,profthreaded)
   rts                            divbyzero [bad profile] (profasm,profthreaded)
   rts                            linker_unload [bad exit code] (normal)
   rts                            overflow1 [bad profile] (profasm,profthreaded)
   rts                            overflow2 [bad profile] (profasm,profthreaded)
   rts                            overflow3 [bad profile] (profasm,profthreaded)
   simplCore/should_compile       T5550 [exit code non-0] (profasm)
   th                             TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_run           T3500a [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T3500a [bad stdout or stderr] (ghci)
   typecheck/should_run           T5751 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run           T7126 [exit code non-0] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded)
   typecheck/should_run           T7126 [bad stdout or stderr] (ghci)
   typecheck/should_run           tcrun045 [stderr mismatch] (normal)

Unexpected stat failures:
   perf/compiler             T1969 [stat not good enough] (normal)
   perf/compiler             T5642 [stat not good enough] (normal)
   perf/compiler             T783 [stat not good enough] (normal)
   perf/compiler             T9675 [stat not good enough] (optasm)
   perf/should_run           T9203 [stat too good] (normal)
   simplCore/should_compile  T7702 [stat too good] (profasm)

and validate --slow gives:

OVERALL SUMMARY for test run started at Thu Jan 22 21:16:02 2015 GMT
 0:25:08 spent to go through
    4405 total tests, which gave rise to
   14234 test cases, of which
    2726 were skipped

     267 had missing libraries
   11018 expected passes
     149 expected failures

       0 caused framework failures
       0 unexpected passes
      74 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   ../../libraries/base/tests     enum01 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum02 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum02 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     enum03 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   ../../libraries/base/tests     enum03 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler01 [bad exit code] (threaded1)
   ../../libraries/base/tests     topHandler01 [bad stdout or stderr] (ghci)
   ../../libraries/base/tests     topHandler03 [bad exit code] (ghci)
   ../../libraries/process/tests  process011 [bad exit code] (threaded1,threaded2)
   ../../libraries/process/tests  process011 [bad stdout or stderr] (ghci)
   ../../libraries/unix/tests     signals002 [bad stdout] (threaded1,threaded2)
   ../../libraries/unix/tests     signals002 [bad stdout or stderr] (ghci)
   cabal/sigcabal01               sigcabal01 [bad stderr] (normal)
   codeGen/should_run             CgStaticPointers [bad stdout or stderr] (ghci)
   concurrent/should_run          allocLimit4 [bad exit code] (ghci)
   concurrent/should_run          conc012 [bad exit code] (ghci)
   concurrent/should_run          conc059 [bad stderr] (threaded1,threaded2)
   deSugar/should_run             DsStaticPointers [bad stdout or stderr] (ghci)
   deSugar/should_run             dsrun014 [bad exit code] (ghci)
   driver/retc001                 retc001 [bad exit code] (normal)
   driver/retc002                 retc002 [bad exit code] (normal)
   ghci/scripts                   ghci024 [bad stdout] (normal)
   numeric/should_run             add2 [bad exit code] (ghci)
   numeric/should_run             mul2 [bad exit code] (ghci)
   numeric/should_run             quotRem2 [bad exit code] (ghci)
   parser/should_run              readRun004 [bad exit code] (ghci)
   rts                            GcStaticPointers [bad stdout or stderr] (ghci)
   rts                            ListStaticPointers [bad stdout or stderr] (ghci)
   rts                            T7919 [bad exit code] (optasm,threaded2,dyn)
   rts                            linker_unload [bad exit code] (normal)
   th                             TH_StaticPointers [bad stdout or stderr] (ghci)
   typecheck/should_run           T3500a [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run           T3500a [bad stdout or stderr] (ghci)
   typecheck/should_run           T5751 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run           T5751 [bad stdout or stderr] (ghci)
   typecheck/should_run           T7126 [exit code non-0] (normal,hpc,optasm,threaded1,threaded2,dyn)
   typecheck/should_run           T7126 [bad stdout or stderr] (ghci)
   typecheck/should_run           tcrun045 [stderr mismatch] (normal)

whereas validate --fast paints a rosier picture:

OVERALL SUMMARY for test run started at Thu Jan 22 22:10:03 2015 GMT
 0:06:07 spent to go through
    4405 total tests, which gave rise to
   14230 test cases, of which
   10172 were skipped

      52 had missing libraries
    3932 expected passes
      71 expected failures

       0 caused framework failures
       0 unexpected passes
       3 unexpected failures
       0 unexpected stat failures

Unexpected failures:
   cabal/sigcabal01  sigcabal01 [bad stderr] (normal)
   driver/retc001    retc001 [bad stderr] (normal)
   rts               linker_unload [bad exit code] (normal)

comment:22 Changed 2 years ago by thomie

Resolution: fixed
Status: newclosed

jrp: maybe you could do another ./validate run, and send the result to the mailing list? GHC on Mac OSX is in need of a new maintainer. It would be nice if somebody could restart the OSX builder also: http://haskell.inf.elte.hu/builders/

I'm closing this ticket here, because the information is most likely outdated.

comment:23 Changed 2 years ago by bgamari

For the record, I believe I've cleaned up the remaining validation issues on Darwin for the 8.0 release.

comment:24 Changed 23 months ago by thomie

Milestone: 8.0.1
Note: See TracTickets for help on using tickets.