Opened 3 weeks ago

Last modified 34 hours ago

#15929 new task

Explore whether adding XRay attributes to LLVM IR is worthwhile

Reported by: mpickering Owned by:
Priority: normal Milestone:
Component: Compiler Version: 8.6.2
Keywords: newcomer Cc: adamse
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

LLVM has support for XRay which is a tracing framework developed by Google. It might be possible to add the relevant function attributes to the IR we generate to get access to detailed traces which can be analysed using existing tooling.

https://llvm.org/docs/XRay.html

Change History (5)

comment:1 Changed 3 weeks ago by mpickering

I tried this out. The diff to add support was very simple.

https://github.com/mpickering/ghc/commit/42c3df674e684fc33bcd472db8b55ace3d4385ca

I then tried to compile the following program and to enable the XRay trace.

module Main where                                                               
                                                                                
main :: IO ()                                                                   
main = print "abc" >> print (f 1000) >> print (g 1000) >> print (qux B)         
                                                                                
data T = A | B deriving Show                                                    
                                                                                
f 0 = 1                                                                         
f x = f (x -1) + 1                                                              
                                                                                
g 1 = 0                                                                         
g x = g (x - 1) + f (x - 1)                                                     
                                                                                
{-# NOINLINE qux #-}                                                            
qux A = A                                                                       
qux B = qux A 

The first thing to note was that it was necessary to use clang-7.0.0 which I achieved by using this diff to ghc.nix. https://gist.github.com/mpickering/f1317ea6aac955eb9ecb1e2e1ef0fba7

Then I compiled with the following sequence of commands:

inplace/bin/ghc-stage2 -fllvm -ddump-llvm llvm.hs -opta="-v" -pgma=clang -pgmc=clang -pgml=clang -optl="-fxray-instrument" -optl="-fxray-instruction-threshold=1" -O2 -fforce-recomp
XRAY_OPTIONS="patch_premain=true xray_mode=xray-basic verbosity=1" ./llvm

Then the log file was analysed using the following command

llvm-xray account  -sort=sum -sortorder=dsc -instr_map ./llvm  xray-log.llvm.XEyh0y 

Which produced the very unhelpful profiling output.

Functions with latencies: 7
   funcid      count [      min,       med,       90p,       99p,       max]       sum  function    
        7         18 [ 0.000006,  0.000006,  0.000021,  0.000040,  0.000040]  0.000175  <invalid>:0:0: @(422fd0)
        5         14 [ 0.000006,  0.000006,  0.000014,  0.000021,  0.000021]  0.000110  <invalid>:0:0: @(422e90)
       10          8 [ 0.000006,  0.000006,  0.000042,  0.000042,  0.000042]  0.000104  <invalid>:0:0: @(4231a8)
        8         10 [ 0.000006,  0.000006,  0.000040,  0.000040,  0.000040]  0.000096  <invalid>:0:0: @(423070)
        6         14 [ 0.000006,  0.000006,  0.000006,  0.000006,  0.000006]  0.000086  <invalid>:0:0: @(422f30)
       11         13 [ 0.000006,  0.000006,  0.000006,  0.000008,  0.000008]  0.000082  <invalid>:0:0: @(423260)
        9          7 [ 0.000006,  0.000006,  0.000014,  0.000014,  0.000014]  0.000055  <invalid>:0:0: @(423110)

Should be easy now for someone else more familiar with LLVM to investigate.

comment:2 Changed 3 weeks ago by mpickering

Keywords: newcomer added

comment:3 Changed 5 days ago by bgamari

Milestone: 8.6.3

Ticket retargeted after milestone closed

comment:4 Changed 34 hours ago by mpickering

I investigated some more and get the basic support working. None of the call stack stuff is useful for GHC as every call is a tail call so the call hierarchy is completely flat.

Here is the branch: https://github.com/mpickering/ghc/tree/llvm-xray

See this thread on llvm-devs for some of the problems and suggestions of the LLVM devs: http://lists.llvm.org/pipermail/llvm-dev/2018-December/128184.html

Here's an example comment I was using to compile which correctly instrumented a binary.

_build/stage1/bin/ghc -fllvm -ddump-llvm llvm.hs -opta="-v" -pgma=clang -pgmc=clang -pgml=clang -optl="-fxray-instrument" -optl="-fxray-instruction-threshold=1" -optlo="-O0" -O2 -g3 -fforce-recomp nofib/real/fulsom/Main.hs  -inofib/real/fulsom/

Problems

  1. You have to pass -O0 to opt as otherwise the LLVM verifier complains about our debugging information. See the email list for more information about this.
  2. It didn't seem like every call was actually counted in some simple tests but I didn't look too closely.

comment:5 Changed 34 hours ago by adamse

Cc: adamse added
Note: See TracTickets for help on using tickets.