Intel Processor Trace: How to use it

To be noted is that “Processor Trace” is a feature of recent Intel x86 processor (eg skylake):

https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing

First clone this:

https://github.com/01org/processor-trace

And do a “sudo make install” to install the libipt.so libraries.

Next do a clone of Andi Kleen pt tracing tool:

git clone github.com:andikleen/simple-pt

If you do “make install” is to make the kernel module, and then followed by “sudo insmod simple-pt.ko” to load it.

Next is to “make user” to make all the relevant binaries (see Makefile, which is essentially sptdump, fastdecode, sptdecode and ptfeature).

During compilation you make encounter some errors on missing files, and the following are the ones I need to install before successful “make user” (but your mileage may differ):

sudo apt-get install libelf-dev
 sudo apt-get install libdw-dev dwarfdump
 sudo apt-get install libdwarf-dev

There are other binaries like “rdmsr” which the tester program depends as well.

Next is to run tester as root: “sudo ./tester”, and the output is listed here:

http://pastebin.com/d3QeVNsV

And looking further into “stest.out”:

0 [+0] [+   1] native_write_msr_safe+18
 [+  31] trace_event_raw_event_msr+116 -> trace_event_buffer_reserve
 [+  21] trace_event_buffer_reserve+143 -> trace_event_buffer_lock_reserve
 [+  23] trace_event_buffer_lock_reserve+67 -> trace_buffer_lock_reserve
 [+  13] trace_buffer_lock_reserve+33 -> ring_buffer_lock_reserve
 [+  41] ring_buffer_lock_reserve+184 -> rb_reserve_next_event
 [+  36] rb_reserve_next_event+180 -> trace_clock_local
 [+   5] trace_clock_local+20 -> sched_clock
 [+   4] sched_clock+12 -> native_sched_clock
 [+   7] trace_buffer_lock_reserve+65 -> ring_buffer_event_data
 [+   4] ring_buffer_event_data+12 -> rb_event_data
 [+   6] trace_buffer_lock_reserve+90 -> tracing_generic_entry_update
 [+   7] trace_event_buffer_reserve+176 -> ring_buffer_event_data
 [+   4] ring_buffer_event_data+12 -> rb_event_data
 [+  11] trace_event_raw_event_msr+164 -> trace_event_buffer_commit
 [+  33] trace_event_buffer_commit+124 -> filter_check_discard
 [+  11] trace_event_buffer_commit+226 -> trace_buffer_unlock_commit
 [+  15] trace_buffer_unlock_commit+45 -> ring_buffer_unlock_commit
 [+  16] ring_buffer_unlock_commit+54 -> rb_update_write_stamp
 [+   7] trace_buffer_unlock_commit+115 -> ftrace_trace_userstack

It is generated by the sptdecode command:

sptdecode --sideband ${PREFIX}.sideband --pt ${PREFIX}.0 $DARGS > ${PREFIX}.out

The function “trace_event_raw_event_msr” is nowhere to be found inside simple-pt.c, nor part of the kernel symbols. But “trace_event_buffer_reserve” is part of the kernel symbol (sudo cat /proc/kallsyms |grep trace_event_buffer_reserve).

So now we shall disassemble the APIs shown in the stest.out from the live-running kernel (every is still read-only, so is safe, no modification is possible, but you will need root):

To see the APIs live in action, first identify the “vmlinux” where the kernel is build. Mine is build by myself, and so do this (sudo is needed before /proc/kcore is root read-only):

sudo gdb ./vmlinux /proc/kcore ==> gdb prompt.

And next is to identify the source directory of the simple-pt.ko kernel module, and its offset in memory:

sudo cat /proc/modules |grep simple
 simple_pt 61440 0 - Live 0xffffffffa1021000 (OE)
 

And so add the following inside the gdb prompt:

 add-symbol-file /home/tteikhua/simple-pt/simple-pt.ko 0xffffffffa1021000

Now you can list in assembly by name:

(gdb) x /10i trace_event_raw_event_msr
 0xffffffffa1022c70 <trace_event_raw_event_msr>: push %rbp
 0xffffffffa1022c71 <trace_event_raw_event_msr+1>: mov %rsp,%rbp
 0xffffffffa1022c74 <trace_event_raw_event_msr+4>: push %r15
 0xffffffffa1022c76 <trace_event_raw_event_msr+6>: push %r14
 0xffffffffa1022c78 <trace_event_raw_event_msr+8>: push %r13
<snip>
 0xffffffffa1022ccd <trace_event_raw_event_msr+93>: lea -0x58(%rbp),%rdi
 0xffffffffa1022cd1 <trace_event_raw_event_msr+97>: mov $0x20,%edx
 0xffffffffa1022cd6 <trace_event_raw_event_msr+102>: mov %r12,%rsi
 0xffffffffa1022cd9 <trace_event_raw_event_msr+105>: addq $0x1,0x7d27(%rip) # 0xffffffffa102aa08
 0xffffffffa1022ce1 <trace_event_raw_event_msr+113>: mov %ecx,-0x5c(%rbp)
 0xffffffffa1022ce4 <trace_event_raw_event_msr+116>: 
callq 0xffffffff81205060 <trace_event_buffer_reserve>  
0xffffffffa1022ce9 <trace_event_raw_event_msr+121>: addq $0x1,0x7d1f(%rip) # 0xffffffffa102aa10

From the “stest.out” output above, we can also see that the line by line output correspond to each “basic blocks” (https://en.wikipedia.org/wiki/Basic_block) in the assembly listing.

References:

https://lwn.net/Articles/576551/

https://lwn.net/Articles/584539/

https://lwn.net/Articles/654705/

https://lwn.net/Articles/648154/

How to create Xen HVM domU guest in Ubuntu 14.04 (not with xenbr0 but with legacy bridge)

I have so many network bridges in my system arising from legacies: docker0, br0, virbr0, and the last thing is to create another new xenbr0.   (just “ls -al /sys/class/net/*/* | grep ‘bridge:’” to identify all the bridges)

So how to setup the bridge networking for a HVM Xen domU guest using one of these bridges without xenbr0?

First, install Ubuntu 14.04 64-bit LTS. Next is to install the Xen hypervisor (see https://help.ubuntu.com/community/Xen):

sudo apt-get install xen-hypervisor-amd64

And after rebooting into Xen hypervisor, the dom0 should be running Ubuntu.   Open up a terminal and enter “sudo xl list” to confirm that dom0 is running.

Next is the bridge setup:

Inside /etc/xen/xl.conf, enter the name of the bridge to be used:

vif.default.bridge="virbr0"

and inside the /etc/xen/xen-config.sxp enter the following:

(network-script 'network-bridge bridge=virbr0')

And checking on the bridging:

sudo brctl addif virbr0 eth0

sudo brctl show

bridge name bridge id STP enabled interfaces
docker0 8000.33333 no
virbr0 8000.444444 yes eth0

And so now virbr0 is the bridge to be used by xen.

Do a “sudo service xen restart” and “sudo service xendomains restart” to restart all the services using the modified configuration.

Next is to create HVM domU running CentOS.

Copy the /etc/xen/xlexample.hvm sample config file (for HVM) into your home directory.

And inside this file modify the name, and memory must be at least 512 (MB) otherwise it will hang during installation (happened twice in two different machine for me). “vcpus” can set to 1 to minimize contention with the host. And for VGA console, I prefer “SDL=1”. For disk:

disk = [ 'file:/sda11/xenhvm.dd,xvda,rw', 'file:/sda11/CentOS-7-x86_64-DVD-1511.iso,hdc:cdrom,r' ]

where the file /sda11/xenhvm.dd is created via:

dd if=/dev/zero of=/sda11/xenhvm.dd bs=1M count 8196

which is 8GB in disk storage space. CentOS will need to be at least about 1.5GB, and 8 GB seemed to avoid all the latencies and slowness issues at runtime.

But if you want to pass an entire disk media directly to the domU, then specify it as:

'phys:/dev/sda, xvda,w' and inside the guest OS the /dev/sda will be accessed as /dev/xvda instead.

In summary the following file is the configuration file (xenhvmguest.cfg) after modification:

 ###vif = [ 'mac=00:16:3e:00:00:00,bridge=xenbr0' ]

# =====================================================================
 # Example HVM guest configuration
 # =====================================================================
 #
 # This is a fairly minimal example of what is required for an
 # HVM guest. For a more complete guide see xl.cfg(5)

# This configures an HVM rather than PV guest
 builder = "hvm"now virbr0 is the bridge.

# Guest name
 name = 'HVM_domU'

# 128-bit UUID for the domain as a hexadecimal number.
 # Use "uuidgen" to generate one if required.
 # The default behavior is to generate a new UUID each time the guest is started.
 #uuid = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"

# Enable Microsoft Hyper-V compatibile paravirtualisation /
 # enlightenment interfaces. Turning this on can improve Windows guest
 # performance and is therefore recommended
 #viridian = 1

# Initial memory allocation (MB)
 memory = 512

# Maximum memory (MB)
 # If this is greater than `memory' then the slack will start ballooned
 # (this assumes guest kernel support for ballooning)
 #maxmem = 512

# Number of VCPUS
 vcpus = 1

# Network devices
 # A list of 'vifspec' entries as described in
 # docs/misc/xl-network-configuration.markdown
 vif = [ '' ]

# Disk Devices
 # A list of `diskspec' entries as described in
 # docs/misc/xl-disk-configuration.txt
 ## disk = [ '/dev/vg/guest-volume,raw,xvda,rw' ]
 disk = [ 'file:/sda11/xenhvm.dd,xvda,rw', 'file:/sda11/CentOS-7-x86_64-DVD-1511.iso,hdc:cdrom,r' ]

# Guest VGA console configuration, either SDL or VNC
 sdl = 1
 #vnc = 1
 #vnc = 1
 #vnclisten = '0.0.0.0'
 #vncdisplay = 1

Now is to create the domU using the above configuration file (xenhvmguest.cfg).

sudo xl list
sudo xl shutdown HVM_domU
sudo xl destroy HVM_domU
sudo xl create xenhvmguest.cfg
sudo xl list

Questions:

a. how to generate traces of calls from guest to host kernel?

b. How to do USB passthrough? PCI passthrough? SCSI passthrough?