Applications of eBPF

There are many applications of BPF:

Allowing non-root, user-customizable firewall:

https://sweetness.hmmz.org/2015-07-25-fun-with-bpf-or-shutting-down-a-tcp-listening.html

"It is a rare situation where decades of undisciplined tinkering with Linux esoterica occasionally pay out, but this was such an occasion. Unlike in BSD, where Berkeley Packet Filter is implemented as a root-only device that attaches to entire network interfaces, on Linux it is implemented in terms of a socket optionthat usually attaches to AF_PACKET or AF_RAW sockets, however it is a little known fact you can also attach such filters to AF_INET sockets, and better yet, the ability to do so does not require root. Essentially, Linux allows non-root programs to configure their own little private firewall."

https://www.youtube.com/watch?v=apU5sg0Ui5U

Stracing and understanding the flow of syscalls + bpf() calls + argument: complimenting stracing via BPF?

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

https://suchakra.wordpress.com/2017/05/23/an-entertaining-ebpf-xdp-adventure/ (archtiecture of Netroname smart NIC)

https://developers.redhat.com/blog/2018/12/06/achieving-high-performance-low-latency-networking-with-xdp-part-1/

https://lwn.net/Articles/719850/

https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

Kernel Path understanding and tracing:

https://developers.redhat.com/blog/2018/12/03/network-debugging-with-ebpf/ (Network debugging)

https://elinux.org/images/d/dc/Kernel-Analysis-Using-eBPF-Daniel-Thompson-Linaro.pdf (Kernel analysis)

https://www.ntop.org/wp-content/uploads/2018/10/Sabella.pdf (network traffic analysis)

Network packet analysis and processing (with speed):

https://blog.cloudflare.com/epbf_sockets_hop_distance/

https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/

https://www.youtube.com/watch?v=apU5sg0Ui5U

https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://lwn.net/Articles/747551/ https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

https://forum.openwrt.org/t/what-is-the-difference-between-nftables-and-bpfilter/24602

Performance Analysis:

https://www.memsql.com/blog/bpf-linux-performance/

AppSwtich:

https://suchakra.wordpress.com/2018/03/31/so-whats-this-appswitch-thing-going-around/

http://appswitch.io/

https://jpetazzo.github.io/2018/03/13/appswitch-hyperlay-network-stack-future/

https://hub.docker.com/r/appswitch/ax/

Containers analysis:

https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/

Intrusion Detection:

https://www.slideshare.net/AlexMaestretti/security-monitoring-with-ebpf

https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/

https://blog.sflow.com/2016/05/berkeley-packet-filter-bpf.html

https://www.netresec.com/?page=Blog&month=2015-11&post=BPF-is-your-Friend

Tracing – both network and processes:

http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html

https://twitter.com/daniel_bilar/status/881570223840780289

https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf

https://suchakra.wordpress.com/2015/05/18/bpf-internals-i/

https://www.iovisor.org/blog/2015/10/15/bpf-internals-ii

https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf

https://www.youtube.com/watch?v=w8nFRoFJ6EQ

https://www.usenix.org/conference/atc17/program/presentation/gregg-superpowers

https://elixir.bootlin.com/linux/v4.14.8/source/kernel/trace/bpf_trace.c#L212

https://elixir.bootlin.com/linux/v4.20-rc1/source/kernel/trace/bpf_trace.c

What are all the technologies/foundation which BPF used:

bpf stacks: http://www.brendangregg.com/blog/2016-01-18/ebpf-stack-trace-hack.html

bpf output:

https://lwn.net/Articles/649965/

https://stackoverflow.com/questions/47914219/output-from-bpf-printk

bpf + kprobes:

https://lwn.net/Articles/636976/

https://elixir.bootlin.com/linux/v4.20/source/kernel/kprobes.c

https://lwn.net/Articles/132196/ (Intro to Kprobes)

https://www.kernel.org/doc/Documentation/kprobes.txt

https://github.com/weaveworks/tcptracer-bpf

https://www.ibm.com/developerworks/library/l-kprobes/index.html

https://vjordan.info/log/fpga/how-linux-kprobes-works.html

https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt

bpf + tracepoints:

https://lwn.net/Articles/683504/

http://thread.gmane.org/gmane.linux.kernel.api/8127/focus=8165

bpf + userspace tracepoints:

https://lwn.net/Articles/753601/

bpf + systemtap:

https://developers.redhat.com/blog/2018/04/23/systemtaps-bpf-backend-tracepoint-support/

How to turn any syscall into an event: Introducing eBPF Kernel probes:

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

Running examples from BPF samples in linux kernel source directory

Summarizing the installation needed before running make on the kernel source:

sudo apt-get install build-essential bison flex
sudo apt-get install clang llvm
sudo apt-get install libelf-dev
sudo apt-get install elfutils-libelf-devel
sudo apt-get install clang
sudo apt-get install llc
sudo apt-get install llvm

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 4052245BD4284CDD
echo “deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main” | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
sudo apt-get install netsniff-ng
sudo apt-get install libpci-dev

sudo apt-get install libreadline6-dev

sudo apt-get install libcap-dev
sudo apt-get install libcap-ng-dev
sudo apt-get install libmount-dev
sudo apt-get install libxen-dev

sudo apt-get install linux-headers-4.15.0-43-generic
sudo apt-get install binutils-dev
sudo apt-get install libpopt-dev

sudo apt-get install libnuma-dev libfuse-dev

sudo apt-get install netperf

sudo apt-get install gcc-multilib libc6-i386 libc6-dev-i386

cd to the linux kernel source’s sample directory and “make all”:

Testing the sample binaries as “sudo”:

Doing a “make V=1 all” let you see the details of compilation:

So this means that sockex2 is compiled from sockex2_user.c and the libbpf.a library, which I compiled earlier separately at the tools subsidrectory.

Now this is the sockex2_user.c:

https://gist.github.com/tthtlc/7d179f08fc8f82fa2485efd81b070f59

And we can see that it is loading a sockex2_kern.o file (https://github.com/torvalds/linux/blob/master/samples/bpf/sockex2_kern.c):

https://gist.github.com/tthtlc/c1238c8e4c7470784c4ae9aa32179d63

via load_bpf_file() API, and followed by open_raw_sock() and setsockopt() to execute the kernel module.

So we can see from the sockex2_kern.c:

bpf_prog2()->flow_dissector()->which will probe into the SKB packets and identify the protocols bits to be filtered.

A key structure in this is the BPF map:

https://prototype-kernel.readthedocs.io/en/latest/bpf/ebpf_maps_types.html

https://prototype-kernel.readthedocs.io/en/latest/bpf/ebpf_maps.html#creating-a-map

More understanding about eBPF maps, and all the bpf_XXX API will be needed.

 

 

 

 

 

 

 

 

 

 

Exploring eBPF Tracing from userspace to kernel

First can my Ubuntu 16.04’s latest bootup image be able to run BPF?

According to “uname -a” and checking the bootup config file:

/boot/config-4.15.0-43-generic

And answer is yes, the kernel is able to support it.   Next is to install the userland tools:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

 

How it worked?

Let’s take a look at a BPF sample:

First is creating the BPF map:

“a generic data structure that allows data to be passed back and forth within the kernel or between the kernel and user space. As the name “map” implies, data is stored and retrieved using a key.”

Each map is defined by four values: a type, a maximum number of elements, a value size in bytes, and a key size in bytes.

And the types of map:

• BPF_MAP_TYPE_HASH: a hash table
• BPF_MAP_TYPE_ARRAY: an array map, optimized for fast lookup speeds, often used for counters
• BPF_MAP_TYPE_PROG_ARRAY: an array of file descriptors corresponding to eBPF programs; used to implement jump tables and sub-programs to handle specific packet protocols
• BPF_MAP_TYPE_PERCPU_ARRAY: a per-CPU array, used to implement histograms of latency
• BPF_MAP_TYPE_PERF_EVENT_ARRAY: stores pointers to struct perf_event, used to read and store perf event counters
• BPF_MAP_TYPE_CGROUP_ARRAY: stores pointers to control groups
• BPF_MAP_TYPE_PERCPU_HASH: a per-CPU hash table
• BPF_MAP_TYPE_LRU_HASH: a hash table that only retains the most recently used items
• BPF_MAP_TYPE_LRU_PERCPU_HASH: a per-CPU hash table that only retains the most recently used items
• BPF_MAP_TYPE_LPM_TRIE: a longest-prefix match trie, good for matching IP addresses to a range
• BPF_MAP_TYPE_STACK_TRACE: stores stack traces
• BPF_MAP_TYPE_ARRAY_OF_MAPS: a map-in-map data structure
• BPF_MAP_TYPE_HASH_OF_MAPS: a map-in-map data structure
• BPF_MAP_TYPE_DEVICE_MAP: for storing and looking up network device references
• BPF_MAP_TYPE_SOCKET_MAP: stores and looks up sockets and allows socket redirection with BPF helper functions

We have create map=>programming codes=>load_program=>pushed into kernel network socket for execution.

And this map can also be shared across different tracing event:

After creating the map is the raw BPF language programming:  bpf_insn_prog[].

Instead of coding in the raw BPF language it is also possible to have the program generated by a bpf compiler:

bhttps://www.systutorials.com/docs/linux/man/8-bpfc/

Or alternatively you can use bpf_asm_compile() to generate the assembly.

And then setsockopt() is used to attached the BPF program to the execution path of setsockopt().

http://man7.org/linux/man-pages/man2/bpf.2.html

In summary, what are all the operations you can do with bpf() APIs:

create map, lookup based on key, load BPF program, lookup and delete/updae element.

(Picture below is from:  https://github.com/iovisor/bpf-docs/blob/master/bpf-internals-2.md)

Here is another description from userspace to kernel flow:

Some user-customizable programming codes is inserted into the kernel – to be compiled at the userspace level via LLVM and passed into the kernel (via bpf() syscall) and then verified (for code integrity) before executed inside the kernel.

Lots of sample codes available in the linux kernel source codes:

https://github.com/torvalds/linux/tree/master/samples/bpf

Here is the BPF language specification (for 64bit):

And for a more comprehensive list please refer to:

https://github.com/iovisor/bpf-docs/blob/master/eBPF.md

How BPF has been used inside the Linux kernel is documented here:

https://www.kernel.org/doc/Documentation/networking/filter.txt

And its use for performance measurement has been used in the BCC tools:

http://www.brendangregg.com/ebpf.html

https://www.iovisor.org/technology/bcc

https://github.com/iovisor/bpf-docs/blob/master/bpf-internals-2.md

And its use in Facebook:

https://facebookmicrosites.github.io/bpf/

Berkeley Packet Filters from Kernel TLV

https://www.slideshare.net/vh21/meet-cutebetweenebpfandtracing?qid=3413bd02-3d22-454e-bc92-bf80d168e1f6&v=&b=&from_search=2

https://www.youtube.com/watch?v=JRFNIKUROPE

https://lwn.net/Articles/740157/

http://www.adelzaalouk.me/2017/security-bpf-docker-cillium/

https://www.dropbox.com/s/hhhieyxchlbyxgj/BPF_tracing_superpowers.pptx?dl=0

https://www.iovisor.org/technology/ebpf

https://www.iovisor.org/blog/2016/04/12/exploring-ebpf-io-visor-and-beyond

https://www.slideshare.net/AffanSyed1/ebpf-and-io-visor-the-what-how-and-what-next

https://www.slideshare.net/suselab/ebpf-trace-from-kernel-to-userspace

Click to access Room%20211%20-%20IOVisor%20-%20SCaLE%2014x.pdf

https://blog.cloudflare.com/bpf-the-forgotten-bytecode/

https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/

https://stackoverflow.com/questions/49577061/reading-bpf-assembly

Applications of BPF:

https://www.iovisor.org/blog/2016/04/12/exploring-ebpf-io-visor-and-beyond

http://www.adelzaalouk.me/2017/security-bpf-docker-cillium/

http://www.brendangregg.com/ebpf.html

https://events.static.linuxfound.org/sites/events/files/slides/ebpf_on_the_mainframe_lcon_2015.pdf

http://www.brendangregg.com/Slides/PerformanceAtScale2016_LinuxBPFSuperpowers.pdf

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

RISC-V: a preliminary study

Following through the recent talk:

https://www.youtube.com/watch?v=4OKkHCg7El0

https://www.youtube.com/watch?v=57Qgif69LBE

And RISC-V presentation material in LPC 2018:

https://linuxplumbersconf.org/event/2/sessions/34/#20181115

http://heshamelmatary.blogspot.com/2017/06/sel4testrisc-v-unofficial-release-priv.html

http://lkml.iu.edu/hypermail/linux/kernel/1709.1/02783.html

http://palms.ee.princeton.edu/system/files/HPCA2015_1_introduction.pdf

https://cloudfront.escholarship.org/dist/prd/content/qt7zj0b3m7/qt7zj0b3m7.pdf?t=odiqm3

https://content.riscv.org/wp-content/uploads/2017/05/riscv-privileged-v1.10.pdf

https://content.riscv.org/wp-content/uploads/2017/12/Tue0942-riscv-hypervisor-waterman.pdf

https://content.riscv.org/wp-content/uploads/2018/07/0930-19.07.18-Atish-Patra-Western-Digital.pdf

https://fosdem.org/2019/schedule/event/riscvsbi/

https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.md

https://linuxplumbersconf.org/event/2/contributions/233/

https://lkml.org/lkml/2018/10/31/431

https://lwn.net/Articles/734866/

https://news.ycombinator.com/item?id=17590905

https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.1.pdf

https://content.riscv.org/wp-content/uploads/2017/05/riscv-privileged-v1.10.pdf

https://www.linuxplumbersconf.org/event/2/contributions/196/attachments/127/159/Atish_SBI.pdf

https://github.com/riscv/riscv-sbi-doc/blob/master/riscv-sbi.md

https://linuxplumbersconf.org/event/2/contributions/233/

https://fosdem.org/2019/schedule/event/riscvsbi/

http://palms.ee.princeton.edu/system/files/HPCA2015_1_introduction.pdf

https://people.eecs.berkeley.edu/~krste/papers/riscv-privileged-v1.9.1.pdf

https://news.ycombinator.com/item?id=17590905

https://content.riscv.org/wp-content/uploads/2017/12/Tue0942-riscv-hypervisor-waterman.pdf

https://rv8.io

https://www.linux-kvm.org/images/6/6a/02x04B-QEMU-Support_for_the_RISC-V_Instruction_Set_Architecture.pdf

https://www.linuxplumbersconf.org/event/2/contributions/196/attachments/127/159/Atish_SBI.pdf

https://www.passeidireto.com/arquivo/44014224/the-risc-v-instruction-set-manual-ingles/3

Corrode: Cabal to Haskell to Rust

Firstly Corrode is a C to RUST translator.

https://github.com/jameysharp/corrode

And it is written in Haskell. So after downloading Corrode how to compile?

First, we shall be using docker.

So firstly is to create a docker image:

docker run -it ubuntu:16.04

and inside the container:

apt-get update && apt-get install git ghc haskell-stack -y
git clone https://github.com/jameysharp/corrode.git
cd corrode && stack build && stack install

During the “stack build” you will encounter this error:

So follow this to install GHC 8.0.2:

sudo apt-get update
sudo apt-get install -y software-properties-common
sudo add-apt-repository -y ppa:hvr/ghc
sudo apt-get update
sudo apt-get install -y ghc-8.0.2

which will also install ghc into the non-standard directory /opt:

So add this to your path: /opt/ghc/bin.   And now “stack build” give you new error:

So according to this:

https://github.com/commercialhaskell/intero/issues/252

the issue is caused by stack version.

First attempt “stack upgrade” does not succeed and this is the error:

So reading this:

https://docs.haskellstack.org/en/stable/README/

The solution is:

curl -sSL https://get.haskellstack.org/ | sh

(but because stack has been installed before, I have to uninstall it via “apt-get remove haskell-stack” before this.)

And after installation it complained that /root/.local/bin is not in $PATH, and so it installed stack in /usr/local/bin.

Ok, now spawning a new shell to corrode, and the issuing “stack build” again:

Notice that now the new version of stack automatically install ghc8.0.2 (because I forgotten to add /opt/ghc/bin into the $PATH), in addition to resolving the “invalid package ID” as encountered earlier.

Nevertheless, the compilation complete to success:

and this:

And the binary corrode is installed in:

/corrode/.stack-work/install/x86_64-linux/lts-8.5/8.0.2/bin

From this:

#include <stdio.h>
#include <string.h>

void functionA(char *str) {
char buf[1024];
strcpy(str, buf);
printf(“%s”, buf);
}

void main() {
functionA(“hello no hello”);
}

“corrode main.c” will generate this rust program:

extern {
fn printf(__format : *const u8, …) -> i32;
fn strcpy(__dest : *mut u8, __src : *const u8) -> *mut u8;
}

#[no_mangle]
pub unsafe extern fn functionA(mut str : *mut u8) {
let mut buf : [u8; 1024];
strcpy(str,buf.as_mut_ptr() as (*const u8));
printf((*b”%s\0″).as_ptr(),buf.as_mut_ptr());
}

fn main() {
let ret = unsafe { _c_main() };
::std::process::exit(ret);
}

#[no_mangle]
pub unsafe extern fn _c_main() {
functionA((*b”hello no hello\0″).as_ptr() as (*mut u8));
0
}

And the compilation by rustc produce the errors:

More “rust” debugging needed here.