How to configure project to use cmake?

How do you use cmake in a project?    Suppose the project is named as “myllvm” and has only one file:  log.c and you have decided to use “gcc” to compile the C file.

This is the simplest CMakeLists.txt to get started:

The directory only have two files: log.c and CMakeLists.txt

To build the binary: either clang or gcc can be used, how to configure that?

First “mkdir build” and “cd build” and then issue the following command to use gcc to compile:

Otherwise, you can use clang (ie, replace the “gcc” with “clang”).

But it is also possible to modifiy CMakeLists.txt to have “gcc” specified inside.

How to add new directories with new C files?   (the subdirectory is call “core”):

add_subdirectory(core)

How to add header directory into the compilation?

include_directories(include)
How to add multiple C or CPP files in “src” subdirectory?
#set(SOURCES src/mainapp.cpp src/Student.cpp)
#or just a widcard:
file(GLOB SOURCES “src/*.cpp”)
For more complete list of commands (for CMakeLists.txt) we can dig into cmake source codes and looking into “Help/command” directory.   Or the commands are listed here:
And when you do a “cmake -G <generator> these are the generate currently available (under Help/generator directory):
Selection_022
How to make use of the available cmake recipes written for each different platform:
For example here some Linux related cmake recipes:
Selection_028
And here are the Windows related cmake templates:
Selection_029
How to use the cmake recipes for different common tasks (eg, finding nvidia cuda compiler, find header files, etc) which are called Modules.
For example of a subtasks, these are all the “finding compiler” related modules (left hand side number is the number of lines for the file – a good representation of the complexity of the tasks it is handling):
Selection_023
Here are a complete list of all the tasks:
For more complete tutorial examples:

https://gitlab.kitware.com/cmake/community/wikis/Home

https://cmake.org/cmake/help/latest/index.html

https://stackoverflow.com/questions/44722714/including-c-file-linking-with-cmake-isnt-working-with-c-cannot-include-funct

Advertisements

Understanding Amazon Web Services Security

First is to highlight the security issues that AWS occasionally have:

https://alas.aws.amazon.com/alas2.html

https://alas.aws.amazon.com

https://aws.amazon.com/security/security-bulletins/

In particular is the ROBOT attack:

https://security.stackexchange.com/questions/177337/what-makes-servers-vulnerable-to-return-of-bleichenbachers-oracle-threat-robot

https://www.usenix.org/system/files/conference/usenixsecurity18/sec18-bock.pdf

https://www.usenix.org/conference/usenixsecurity18/presentation/bock

https://robotattack.org/

Amazon Web Services: Overview of Security Processes:

https://d0.awsstatic.com/whitepapers/Security/Intro_Security_Practices.pdf

https://cloudacademy.com/blog/amazon-web-services-security/

https://d36cz9buwru1tt.cloudfront.net/aws-gov-summit-2011/Security_Keynote_CJ_Moses.pdf

https://i.crn.com/sites/default/files/ckfinderimages/userfiles/images/crn/custom/2018/AlienVault_360_Q418_AWS_security_monitoring_for_beginners.pdf

https://1xt03y2cptxl2pmtse29x65s-wpengine.netdna-ssl.com/wp-content/uploads/2018/05/08-AWS-Security-State-of-the-Union_final.pdf

https://d1.awsstatic.com/whitepapers/aws-security-best-practices.pdf

https://d1.awsstatic.com/whitepapers/Security/AWS_Security_Checklist.pdf

https://www.fortinet.com/content/dam/fortinet/assets/white-papers/wp-why-your-aws-cloud-environments-need-advanced-security.pdf

https://www.isc2denver.org/sites/default/files/AWS_Security_Best_Practices.pdf

https://www.owasp.org/images/3/33/Aws_security_joel_leino.pdf

https://www.threatstack.com/blog/what-you-need-to-know-about-the-top-7-aws-security-issues

https://aws.amazon.com/security/

Tips for AWS firewall:

https://aws.amazon.com/blogs/compute/automating-security-group-updates-with-aws-lambda/

https://blog.trendmicro.com/top-10-aws-security-tips-5-create-restrictive-firewall-policies/

AWS IoT Security:

Application Development How to:

https://medium.com/@CharlesBordet/how-to-deploy-a-shiny-app-on-aws-part-1-4893d0a7432f

https://medium.com/capgemini-norway/hosting-your-react-application-with-aws-in-30-minutes-623f6539d92d

https://towardsdatascience.com/deploying-a-python-web-app-on-aws-57ed772b2319

https://www.thinxtream.com/whitepapers/thinxtream-iot-security-wp-005.pdf

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/index.html

Development APIs and setup:

https://aws.amazon.com/tools/

https://aws.amazon.com/sdk-for-node-js/

https://github.com/aws/aws-sdk-js

https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-nodejs.html

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/index.html

Fuzzer:

https://github.com/MozillaSecurity/octo

https://github.com/MozillaSecurity/funfuzz

https://github.com/MozillaSecurity

https://nth10sd.github.io/js-fuzzing-in-mozilla/

Characteristics of eBPF

a. it never can modify or read aribitrary parts of kernel memory, but only through known channels or mechanism.

b. if it uses FTRACE for tracing, then the dynamic instrumentation is delayed as long as possible – only when the code execution path is encountered. So if the user requested ebpf tracing of kernel APIs, many times it will not be instrumented until the point it is executed.

c. instead of directly running binary codes, it run a virtual machine to interpret binary codes. since this intermediate language must not have loops, it is verified so, and so is many other characteristics verified as well.

d. whenever it read parts of kernel memory, the data are only snapshots, and constantly changing. data-misses may happen, and this statistical by nature. memory buffer may be too small and reused quickly (ring buffer) – especially for network packet collections.

e. eBPF ==> no userspace dependencies, like FTRACE, and unlike SYSTEMTAP, so python, go, RUST, and so many other variations of usage of eBPF have been spawned. (and so similarly you have FTRACE + python, FTRACE + go and so on)

f. what is seccomp bpf?

https://lwn.net/Articles/740157/

"BPF does not define itself by only providing its instruction set, but also by offering further infrastructure around it such as maps which act as efficient key / value stores, helper functions to interact with and leverage kernel functionality, tail calls for calling into other BPF programs, security hardening primitives, a pseudo file system for pinning objects (maps, programs), and infrastructure for allowing BPF to be offloaded, for example, to a network card.

LLVM provides a BPF back end, so that tools like clang can be used to compile C into a BPF object file, which can then be loaded into the kernel. BPF is deeply tied to the Linux kernel and allows for full programmability without sacrificing native kernel performance.

Last but not least, also the kernel subsystems making use of BPF are part of BPF’s infrastructure. The two main subsystems discussed throughout this document are tc and XDP where BPF programs can be attached to. XDP BPF programs are attached at the earliest networking driver stage and trigger a run of the BPF program upon packet reception. By definition, this achieves the best possible packet processing performance since packets cannot get processed at an even earlier point in software. However, since this processing occurs so early in the networking stack, the stack has not yet extracted metadata out of the packet. On the other hand, tc BPF programs are executed later in the kernel stack, so they have access to more metadata and core kernel functionality. Apart from tc and XDP programs, there are various other kernel subsystems as well which use BPF such as tracing (kprobes, uprobes, tracepoints, etc)."

How can eBPF be compromised by vulnerabilities?

What are the security risks for allowing any runtimes codes to be executed in the kernel?

According to:

https://hardenedlinux.github.io/system-security/2016/12/13/kernel_mitigation_checklist.html

We can have:

  1. Privilege escalation
  2. Buffer overflow (You can see how spender tried to protect all the buffer for BPF processing here: https://grsecurity.net/~spender/jit_prot.diff, for example by using MACROS instead of some haphazard numbers like 64). Another to overflow the buffer is via “JIT spraying”: (https://lwn.net/Articles/525609/)
  3. Information leakage

Actual bug history:

https://www.openwall.com/lists/oss-security/2018/10/17/9

https://tools.cisco.com/security/center/viewAlert.x?alertId=56415 (information leakage from kernel to userland)

https://blog.aquasec.com/ebpf-vulnerability-cve-2017-16995-when-the-doorman-becomes-the-backdoor

https://secuniaresearch.flexerasoftware.com/advisories/68979/

But BPF is a double edged sword:

http://www.brendangregg.com/Slides/BSidesSF2017_BPF_security_monitoring.pdf

https://www.openwall.com/lists/oss-security/2017/12/24/1

https://lwn.net/Articles/742170/

https://www.exploit-db.com/exploits/42048

https://ricklarabee.blogspot.com/2018/07/ebpf-and-analysis-of-get-rekt-linux.html

https://lkml.org/lkml/2018/1/3/895

 

Applications of eBPF

There are many applications of BPF:

Allowing non-root, user-customizable firewall:

https://sweetness.hmmz.org/2015-07-25-fun-with-bpf-or-shutting-down-a-tcp-listening.html


"It is a rare situation where decades of undisciplined tinkering with Linux esoterica occasionally pay out, but this was such an occasion. Unlike in BSD, where Berkeley Packet Filter is implemented as a root-only device that attaches to entire network interfaces, on Linux it is implemented in terms of a socket optionthat usually attaches to AF_PACKET or AF_RAW sockets, however it is a little known fact you can also attach such filters to AF_INET sockets, and better yet, the ability to do so does not require root. Essentially, Linux allows non-root programs to configure their own little private firewall."

https://www.youtube.com/watch?v=apU5sg0Ui5U

Stracing and understanding the flow of syscalls + bpf() calls + argument: complimenting stracing via BPF?

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

https://suchakra.wordpress.com/2017/05/23/an-entertaining-ebpf-xdp-adventure/ (archtiecture of Netroname smart NIC)

https://developers.redhat.com/blog/2018/12/06/achieving-high-performance-low-latency-networking-with-xdp-part-1/

https://lwn.net/Articles/719850/

https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

Kernel Path understanding and tracing:

https://developers.redhat.com/blog/2018/12/03/network-debugging-with-ebpf/ (Network debugging)

https://elinux.org/images/d/dc/Kernel-Analysis-Using-eBPF-Daniel-Thompson-Linaro.pdf (Kernel analysis)

https://www.ntop.org/wp-content/uploads/2018/10/Sabella.pdf (network traffic analysis)

Network packet analysis and processing (with speed):

https://blog.cloudflare.com/epbf_sockets_hop_distance/

https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/

https://www.youtube.com/watch?v=apU5sg0Ui5U

https://www.netronome.com/blog/bpf-ebpf-xdp-and-bpfilter-what-are-these-things-and-what-do-they-mean-enterprise/ https://lwn.net/Articles/747551/ https://cilium.io/blog/2018/04/17/why-is-the-kernel-community-replacing-iptables/

https://forum.openwrt.org/t/what-is-the-difference-between-nftables-and-bpfilter/24602

Performance Analysis:

https://www.memsql.com/blog/bpf-linux-performance/

AppSwtich:

https://suchakra.wordpress.com/2018/03/31/so-whats-this-appswitch-thing-going-around/

http://appswitch.io/

https://jpetazzo.github.io/2018/03/13/appswitch-hyperlay-network-stack-future/

https://hub.docker.com/r/appswitch/ax/

Containers analysis:

https://kubernetes.io/blog/2017/12/using-ebpf-in-kubernetes/

Intrusion Detection:

https://www.slideshare.net/AlexMaestretti/security-monitoring-with-ebpf

https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/

https://blog.sflow.com/2016/05/berkeley-packet-filter-bpf.html

https://www.netresec.com/?page=Blog&month=2015-11&post=BPF-is-your-Friend

Tracing – both network and processes:

http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html

https://twitter.com/daniel_bilar/status/881570223840780289

https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf

https://suchakra.wordpress.com/2015/05/18/bpf-internals-i/

https://www.iovisor.org/blog/2015/10/15/bpf-internals-ii

https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf

https://www.youtube.com/watch?v=w8nFRoFJ6EQ

https://www.usenix.org/conference/atc17/program/presentation/gregg-superpowers

https://elixir.bootlin.com/linux/v4.14.8/source/kernel/trace/bpf_trace.c#L212

https://elixir.bootlin.com/linux/v4.20-rc1/source/kernel/trace/bpf_trace.c

What are all the technologies/foundation which BPF used:

bpf stacks: http://www.brendangregg.com/blog/2016-01-18/ebpf-stack-trace-hack.html

bpf output:

https://lwn.net/Articles/649965/

https://stackoverflow.com/questions/47914219/output-from-bpf-printk

bpf + kprobes:

https://lwn.net/Articles/636976/

https://elixir.bootlin.com/linux/v4.20/source/kernel/kprobes.c

https://lwn.net/Articles/132196/ (Intro to Kprobes)

https://www.kernel.org/doc/Documentation/kprobes.txt

https://github.com/weaveworks/tcptracer-bpf

https://www.ibm.com/developerworks/library/l-kprobes/index.html

https://vjordan.info/log/fpga/how-linux-kprobes-works.html

https://www.kernel.org/doc/Documentation/trace/kprobetrace.txt

bpf + tracepoints:

https://lwn.net/Articles/683504/

http://thread.gmane.org/gmane.linux.kernel.api/8127/focus=8165

bpf + userspace tracepoints:

https://lwn.net/Articles/753601/

bpf + systemtap:

https://developers.redhat.com/blog/2018/04/23/systemtaps-bpf-backend-tracepoint-support/

How to turn any syscall into an event: Introducing eBPF Kernel probes:

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

Running examples from BPF samples in linux kernel source directory

Summarizing the installation needed before running make on the kernel source:

sudo apt-get install build-essential bison flex
sudo apt-get install clang llvm
sudo apt-get install libelf-dev
sudo apt-get install elfutils-libelf-devel
sudo apt-get install clang
sudo apt-get install llc
sudo apt-get install llvm

sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys 4052245BD4284CDD
echo “deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main” | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)
sudo apt-get install netsniff-ng
sudo apt-get install libpci-dev

sudo apt-get install libreadline6-dev

sudo apt-get install libcap-dev
sudo apt-get install libcap-ng-dev
sudo apt-get install libmount-dev
sudo apt-get install libxen-dev

sudo apt-get install linux-headers-4.15.0-43-generic
sudo apt-get install binutils-dev
sudo apt-get install libpopt-dev

sudo apt-get install libnuma-dev libfuse-dev

sudo apt-get install netperf

sudo apt-get install gcc-multilib libc6-i386 libc6-dev-i386

cd to the linux kernel source’s sample directory and “make all”:

Testing the sample binaries as “sudo”:

image 16

Doing a “make V=1 all” let you see the details of compilation:

So this means that sockex2 is compiled from sockex2_user.c and the libbpf.a library, which I compiled earlier separately at the tools subsidrectory.

Now this is the sockex2_user.c:

https://gist.github.com/tthtlc/7d179f08fc8f82fa2485efd81b070f59

And we can see that it is loading a sockex2_kern.o file (https://github.com/torvalds/linux/blob/master/samples/bpf/sockex2_kern.c):

https://gist.github.com/tthtlc/c1238c8e4c7470784c4ae9aa32179d63

via load_bpf_file() API, and followed by open_raw_sock() and setsockopt() to execute the kernel module.

So we can see from the sockex2_kern.c:

bpf_prog2()->flow_dissector()->which will probe into the SKB packets and identify the protocols bits to be filtered.

A key structure in this is the BPF map:

https://prototype-kernel.readthedocs.io/en/latest/bpf/ebpf_maps_types.html

https://prototype-kernel.readthedocs.io/en/latest/bpf/ebpf_maps.html#creating-a-map

More understanding about eBPF maps, and all the bpf_XXX API will be needed.

 

 

 

 

 

 

 

 

 

 

Exploring eBPF Tracing from userspace to kernel

First can my Ubuntu 16.04’s latest bootup image be able to run BPF?

According to “uname -a” and checking the bootup config file:

/boot/config-4.15.0-43-generic

And answer is yes, the kernel is able to support it.   Next is to install the userland tools:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/iovisor.list
sudo apt-get update
sudo apt-get install bcc-tools libbcc-examples linux-headers-$(uname -r)

 

How it worked?

Let’s take a look at a BPF sample:

Selection_106

First is creating the BPF map:

“a generic data structure that allows data to be passed back and forth within the kernel or between the kernel and user space. As the name “map” implies, data is stored and retrieved using a key.”

Each map is defined by four values: a type, a maximum number of elements, a value size in bytes, and a key size in bytes.

And the types of map:

  • BPF_MAP_TYPE_HASH: a hash table
  • BPF_MAP_TYPE_ARRAY: an array map, optimized for fast lookup speeds, often used for counters
  • BPF_MAP_TYPE_PROG_ARRAY: an array of file descriptors corresponding to eBPF programs; used to implement jump tables and sub-programs to handle specific packet protocols
  • BPF_MAP_TYPE_PERCPU_ARRAY: a per-CPU array, used to implement histograms of latency
  • BPF_MAP_TYPE_PERF_EVENT_ARRAY: stores pointers to struct perf_event, used to read and store perf event counters
  • BPF_MAP_TYPE_CGROUP_ARRAY: stores pointers to control groups
  • BPF_MAP_TYPE_PERCPU_HASH: a per-CPU hash table
  • BPF_MAP_TYPE_LRU_HASH: a hash table that only retains the most recently used items
  • BPF_MAP_TYPE_LRU_PERCPU_HASH: a per-CPU hash table that only retains the most recently used items
  • BPF_MAP_TYPE_LPM_TRIE: a longest-prefix match trie, good for matching IP addresses to a range
  • BPF_MAP_TYPE_STACK_TRACE: stores stack traces
  • BPF_MAP_TYPE_ARRAY_OF_MAPS: a map-in-map data structure
  • BPF_MAP_TYPE_HASH_OF_MAPS: a map-in-map data structure
  • BPF_MAP_TYPE_DEVICE_MAP: for storing and looking up network device references
  • BPF_MAP_TYPE_SOCKET_MAP: stores and looks up sockets and allows socket redirection with BPF helper functions

We have create map=>programming codes=>load_program=>pushed into kernel network socket for execution.

And this map can also be shared across different tracing event:

Selection_110

After creating the map is the raw BPF language programming:  bpf_insn_prog[].

Instead of coding in the raw BPF language it is also possible to have the program generated by a bpf compiler:

bhttps://www.systutorials.com/docs/linux/man/8-bpfc/

Or alternatively you can use bpf_asm_compile() to generate the assembly.

And then setsockopt() is used to attached the BPF program to the execution path of setsockopt().

http://man7.org/linux/man-pages/man2/bpf.2.html

In summary, what are all the operations you can do with bpf() APIs:

create map, lookup based on key, load BPF program, lookup and delete/updae element.

Selection_140

(Picture below is from:  https://github.com/iovisor/bpf-docs/blob/master/bpf-internals-2.md)

Here is another description from userspace to kernel flow:

Some user-customizable programming codes is inserted into the kernel – to be compiled at the userspace level via LLVM and passed into the kernel (via bpf() syscall) and then verified (for code integrity) before executed inside the kernel.

Lots of sample codes available in the linux kernel source codes:

https://github.com/torvalds/linux/tree/master/samples/bpf

Here is the BPF language specification (for 64bit):

Selection_104

And for a more comprehensive list please refer to:

https://github.com/iovisor/bpf-docs/blob/master/eBPF.md

How BPF has been used inside the Linux kernel is documented here:

https://www.kernel.org/doc/Documentation/networking/filter.txt

And its use for performance measurement has been used in the BCC tools:

http://www.brendangregg.com/ebpf.html

https://www.iovisor.org/technology/bcc

https://github.com/iovisor/bpf-docs/blob/master/bpf-internals-2.md

And its use in Facebook:

https://facebookmicrosites.github.io/bpf/

https://www.slideshare.net/vh21/meet-cutebetweenebpfandtracing?qid=3413bd02-3d22-454e-bc92-bf80d168e1f6&v=&b=&from_search=2

https://www.youtube.com/watch?v=JRFNIKUROPE

https://lwn.net/Articles/740157/

http://www.adelzaalouk.me/2017/security-bpf-docker-cillium/

https://www.dropbox.com/s/hhhieyxchlbyxgj/BPF_tracing_superpowers.pptx?dl=0

https://www.iovisor.org/technology/ebpf

https://www.iovisor.org/blog/2016/04/12/exploring-ebpf-io-visor-and-beyond

https://www.slideshare.net/AffanSyed1/ebpf-and-io-visor-the-what-how-and-what-next

https://www.slideshare.net/suselab/ebpf-trace-from-kernel-to-userspace

https://www.socallinuxexpo.org/sites/default/files/presentations/Room%20211%20-%20IOVisor%20-%20SCaLE%2014x.pdf

https://blog.cloudflare.com/bpf-the-forgotten-bytecode/

https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/

https://stackoverflow.com/questions/49577061/reading-bpf-assembly

Applications of BPF:

https://www.iovisor.org/blog/2016/04/12/exploring-ebpf-io-visor-and-beyond

http://www.adelzaalouk.me/2017/security-bpf-docker-cillium/

http://www.brendangregg.com/ebpf.html

https://events.static.linuxfound.org/sites/events/files/slides/ebpf_on_the_mainframe_lcon_2015.pdf

http://www.brendangregg.com/Slides/PerformanceAtScale2016_LinuxBPFSuperpowers.pdf

https://blog.yadutaf.fr/2016/03/30/turn-any-syscall-into-event-introducing-ebpf-kernel-probes/

Astr0baby's not so random thoughts _____ rand() % 100;

@astr0baby on Twitter for fresh randomness

WildML

Artificial Intelligence, Deep Learning, and NLP

The Data Explorer

playing around with open data to learn some cool stuff about data analysis and the world

Conorsblog

Data | ML | NLP | Python | R

quyv

Just a thought

IFT6266 - H2017 Deep Learning

A Graduate Course Offered at Université de Montréal

Deep Learning IFT6266-H2017 UdeM

Philippe Paradis - My solutions to the image inpainting problem

JOURNEY CALLED: MACHINE LEARNING AND DEEP LEARNING

Taking the required Steps for learning

IFT6266 – H2017 DEEP LEARNING

Pulkit's thoughts on the course project

ML/AI

Machine learning. Artificial Intelligence

the morning paper

an interesting/influential/important paper from the world of CS every weekday morning, as selected by Adrian Colyer

The Spectator

Shakir's Machine Learning Blog

Everything about Data Analytics

big data, data analytics

%d bloggers like this: