Code generated by GHC 8.2.1 faster than 8.0.1 but still somewhat slower than 7.10.3

Unicode normalization library (https://github.com/harendra-kumar/unicode-transforms) shows around 10% improvement with GHC 8.2.1 when compared to GHC 8.0.1, across most benchmarks. However, it is still somewhat slower when compared to GHC 7.10.3. On the other hand, when compared to GHC 7.8.4, GHC 8.2.1 is marginally better.

This library has been optimized with an objective to match or better (in some cases it is better) the performance of the equivalent ICU C++ library (compare only decompose normalization). Some of the last hurdles to match the best case of C++ were in the code generated by GHC. I filed a couple of ghc trac tickets earlier on that.

You can reproduce these results easily as follows:

$ git clone https://github.com/harendra-kumar/unicode-transforms.git
$ git checkout -b v0.3.3 v0.3.3
$ echo ghc-8.0
$ stack bench 
$ stack bench --stack-yaml stack-7.10.yaml
$ stack bench --stack-yaml stack-8.2.yaml

Here are some excerpts from the benchmarks:

GHC 7.10.3:

benchmarking unicode-transforms-text/NFD/English
time                 4.823 ms   (4.765 ms .. 4.902 ms)

benchmarking unicode-transforms-text/NFD/Deutsch
time                 5.607 ms   (5.522 ms .. 5.673 ms)

GHC 8.0.1

benchmarking unicode-transforms-text/NFD/English
time                 6.330 ms   (6.232 ms .. 6.439 ms)

benchmarking unicode-transforms-text/NFD/Deutsch
time                 7.109 ms   (7.034 ms .. 7.183 ms)

GHC 8.2.1

benchmarking unicode-transforms-text/NFD/English
time                 5.659 ms   (5.594 ms .. 5.740 ms)

benchmarking unicode-transforms-text/NFD/Deutsch
time                 6.406 ms   (6.329 ms .. 6.478 ms)

Trac metadata

Trac field	Value
Version	8.2.1
Type	Bug
TypeOfFailure	OtherFailure
Priority	normal
Resolution	Unresolved
Component	Compiler
Test case
Differential revisions
BlockedBy
Related
Blocking
CC
Operating system
Architecture

changed weight to 5

added Tbug Trac import labels

I filed a couple of ghc trac tickets earlier on that.

Can you be specific? Which tickets?

In a quest to extract every last drop of perf in this library so that I can beat the gcc version, I investigated the GHC generated code and filed these tickets #12231 and #12232.

added runtime perf label

added Pnormal label

I've had a look at the current unicode-transforms master, but couldn't find a performance regression from GHC 7.10.3 onwards:

7.10.3:

benchmarked unicode-transforms-text/NFD/English
time                 1.931 ms   (1.929 ms .. 1.934 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.934 ms   (1.932 ms .. 1.937 ms)
std dev              8.370 μs   (6.777 μs .. 11.04 μs)

8.0.2:

benchmarked unicode-transforms-text/NFD/English
time                 1.929 ms   (1.925 ms .. 1.934 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.940 ms   (1.937 ms .. 1.947 ms)
std dev              15.91 μs   (10.49 μs .. 23.61 μs)

8.2.2:

benchmarked unicode-transforms-text/NFD/English
time                 1.708 ms   (1.700 ms .. 1.716 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.724 ms   (1.719 ms .. 1.733 ms)
std dev              22.34 μs   (16.78 μs .. 31.26 μs)

8.10.4:

benchmarked unicode-transforms-text/NFD/English
time                 1.508 ms   (1.502 ms .. 1.516 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.514 ms   (1.509 ms .. 1.518 ms)
std dev              16.98 μs   (14.24 μs .. 21.54 μs)

At the v0.3.3 tag I can reproduce the perf regression from 7.10.3 to 8.2.2. It seems to be fixed as of 8.10.4 though:

7.10.3:

benchmarking unicode-transforms-text/NFD/English
time                 3.987 ms   (3.979 ms .. 3.994 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 3.999 ms   (3.993 ms .. 4.008 ms)
std dev              23.74 μs   (15.35 μs .. 40.67 μs)

8.0.2:

benchmarking unicode-transforms-text/NFD/English
time                 4.853 ms   (4.836 ms .. 4.869 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 4.880 ms   (4.865 ms .. 4.913 ms)
std dev              64.10 μs   (29.62 μs .. 112.6 μs)

8.2.2:

benchmarking unicode-transforms-text/NFD/English
time                 4.570 ms   (4.561 ms .. 4.579 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 4.583 ms   (4.577 ms .. 4.591 ms)
std dev              21.09 μs   (17.02 μs .. 29.73 μs)

8.10.4 (with --allow-newer=path,path-io,criterion,optparse-applicative):

benchmarking unicode-transforms-text/NFD/English
time                 3.620 ms   (3.608 ms .. 3.636 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 3.619 ms   (3.616 ms .. 3.625 ms)
std dev              13.96 μs   (10.38 μs .. 19.97 μs)

I think the regression is probably not worth investigating further. It would be interesting to know whether some of the optimizations that you have apparently applied since v0.3.3 could be applied by GHC on its own though. Does anything come to mind, @harendra?

This may not be worth pursuing any more, too many changes have gone in GHC since then.

closed

This may not be worth pursuing any more, too many changes have gone in GHC since then.

Hmm. Regardless of how many changes, it'd be good to know if 9.6 (say) is sill slower than 7.10. The differences you report above are significant. They clearly indicate that there is money on table. (But maybe we have grabbed that money since.)

Would it be possible to try 7.10 against 9.6?

@sjakobi reported above that 8.10.4 is in fact faster than 7.10.3 so it seems to have been fixed. We can check 9.6 against 8.10 then.

Also, unicode-transforms has changed a lot since then. We are better than the C++ version except in a few cases - see the benchmark results here https://github.com/composewell/unicode-transforms. We can try these on recent compilers and compare if there is any regression.