How to understand the crashdump and identify the source codes in Linux Kernel?

Problem:

Got this in my crash dump, what does it mean? How to identify exactly where in the assembly listing this dump comes from?

[   90.984131] dump_stack+0xe6/0x15b
[   90.984346] ? _atomic_dec_and_lock+0x165/0x165
[   90.984628] ? do_raw_spin_lock+0xf7/0x1e0
[   90.984899] ubsan_epilogue+0x12/0x53
[   90.985131] __ubsan_handle_type_mismatch+0x1f1/0x2c6
[   90.985443] ? ubsan_epilogue+0x53/0x53
[   90.985683] ? debug_lockdep_rcu_enabled+0x26/0x40
[   90.985979] ? ubsan_epilogue+0x53/0x53
[   90.986219] ? __build_skb+0x87/0x320
[   90.986452] dev_gro_receive+0x1b2b/0x1c00
[   90.986805] ? trace_event_raw_event_fib_table_lookup_nh+0x8/0x4b0
[   90.987205] napi_gro_receive+0xa9/0x410
[   90.987446] e1000_clean_rx_irq+0x76f/0x1670
[   90.987709] ? e1000_alloc_jumbo_rx_buffers+0xd80/0xd80
[   90.988052] e1000_clean+0x8c2/0x2b60
[   90.988284] ? trace_hardirqs_on+0xd/0x10
[   90.988527] ? __lock_acquire+0xf4/0x39c0
[   90.988794] ? e1000_clean_tx_ring+0x3c0/0x3c0
[   90.989076] ? __ubsan_handle_type_mismatch+0x29e/0x2c6
[   90.989396] ? time_hardirqs_on+0x2a/0x380
[   90.989712] ? ubsan_epilogue+0x53/0x53
[   90.989947] ? net_rx_action+0x22e/0xe90
[   90.990207] net_rx_action+0x818/0xe90
[   90.990438] ? napi_complete_done+0x440/0x440
[   90.990726] ? debug_lockdep_rcu_enabled+0x26/0x40
[   90.991050] ? __do_softirq+0x18a/0xbc4
[   90.991293] ? time_hardirqs_on+0x2a/0x380
[   90.991560] ? do_raw_spin_trylock+0xd0/0xd0
[   90.991818] ? __do_softirq+0x18a/0xbc4
[   90.992063] __do_softirq+0x1ae/0xbc4
[   90.992336] irq_exit+0x149/0x1d0
[   90.992540] do_IRQ+0x92/0x1b0
[   90.992730] common_interrupt+0x90/0x90
[   90.992963] RIP: 0010:ext4_da_write_begin+0x5fa/0xee0

Firstly, I will assumed you know that the crashdump above are all return addresses of a “call xxxx” command. And so if you mapped it to the assembly listing in the “vmlinux” files when linux kernel is generated, you will see a “call xxx” just before the addresses above.

The cause of the crashdump is somewhere at the top, as it is the cause of triggering the UBSAN compiled kernel, and thus leading to calling dump_stack() function.

Therefore start reading from the top:

First we will need to convert the “vmlinux” output to its assembly listing:

objdump -d vmlinux > vmlinux.S

Then go direct to the “dump_stack” symbol:

So the address is “0xffffffff81e0ae65” for dump_stack():

dump_stack+0xe6/0x15b ==> 0xffffffff81e0ae65 + 0xe6 = 0xffffffff81e0af4b

So now looking at “0xffffffff81e0af4b”:

So we know the dump address is the address immediately after “call show_stack()” function call.

Therefore, using “addr2line -e vmlinux 0xffffffff81e0af4b” we get:

/home/tteikhua/linux-4.12/lib/dump_stack.c:54

Now we need to direct to the source:

So line 54 is exactly after the “__dump_stack()” call, and we can now do the same for all other calls.

cat crash.addr | sed 's/.* //' | sed 's/^M//' |sed 's//.*//' |sed 's/+/ /' > /tmp/tmp$$
 backtrace=20
 forwardtrace=40
 cat /tmp/tmp$$ | while
 read data data1
 do
 addr=`grep "<${data}.:$" vmlinux.S | sed 's/ .*//'`
 output=`addr2line -e ./vmlinux 0x$addr | sed 's/ .*//'`
 addr1=`bash ./addhex.bash 0x$addr $data1`
 output=`addr2line -e ./vmlinux 0x$addr1 | sed 's/ .*//'`
 echo "${output}"
 source=`echo $output | tail -1`
 filename=`echo $source | sed 's/:.*//'`
 line=`echo $source | sed 's/.*://'`
 line1=`expr ${line} - ${backtrace}`

echo "==========================================================="
 cat $filename | nl -ba | sed -n "${line1},+${forwardtrace}p" | sed "${backtrace}s/^/>>/"

echo "==========================================================="
 done

As the automated script above contained some control codes, the attached file is better (output as below):

https://drive.google.com/open?id=0B1hg6_FvnEa8cGpvZEdxTFNCVDA

https://drive.google.com/open?id=0B1hg6_FvnEa8MHEwNU10VDRXQzA

References:

http://helenfornazier.blogspot.sg/2015/07/linux-kernel-memory-corruption-debug.html

https://serverfault.com/questions/605946/kernel-stack-trace-to-source-code-lines

https://stackoverflow.com/questions/6151538/addr2line-on-kernel-module

http://elinux.org/Addr2line_for_kernel_debugging

http://www.linuxdevcenter.com/cmd/cmd.csp?path=a/addr2line

https://plus.google.com/+TheodoreTso/posts/h8H9z4EopkS

How to extend the disk size in my VMware guest running Centos 7

To increase the disk size inside the VMware guest running CentOS7 (7.3, 1611) these are the steps needed:

a. Remove all snapshots from your Vmware image, or if you cannot afford to remove the snapshot, shutdown the guest OS, and "clone from snapshot" (you can select any specific snapshot to clone) to create a new Vmware image. This new one is totally independent, and will not have any of the snapshot.

b. Now go to VMware settings, hardware, and "hard disk" to select "Expand Disk" option.

c. After expanding disk, reboot into your CentOS.

d. Issue "sudo fdisk /dev/sda" to repartition the /dev/sda inside the guest OS:

And here is the partition deletion and recreation part (be careful) – the starting block number must be the same, but the end will default to the largest possible block number.

Now remember to REBOOT, as the partition table will not be updated until after reboot.

And now check using pvdisplay – it is not updated yet, still 75G. Use "pvresize" to extend it.

So now the PV occupies 200G.

Now check with lvdisplay:

So we need to extend the LV to 200G:

Followed by xfs_growfs:

Check now:

Updated.

Linux kernel commandline bootup option

In your /boot/grub/grub.cfg kernel bootup setup, there is a line where you can add a lot of kernel option (see the “linux” line below):

Or if you startup your kernel using QEMU, the place where you can add the kernel options are inside the “-append” line (in bold):

qemu-system-x86_64 -m 3048 -net nic -net user,host=10.0.2.10,hostfwd=tcp::11398-:22 -display none -serial stdio -no-reboot -numa node,nodeid=0,cpus=0-1 -numa node,nodeid=1,cpus=2-3 -smp sockets=2,cores=2,threads=1 -enable-kvm -usb -usbdevice mouse -usbdevice tablet -soundhw all -hda /home/user/syz4123/wheezy_4.12.img -snapshot -initrd /home/tteikhua/syz4129/initrd -kernel /home/tteikhua/syz4129/bzImage -append “console=ttyS0 slub_debug=FZ vsyscall=native rodata=n oops=panic nmi_watchdog=panic panic_on_warn=1 panic=86400 ftrace_dump_on_oops=orig_cpu earlyprintk=serial net.ifnames=0 biosdevname=0 kvm-intel.nested=1 kvm-intel.unrestricted_guest=1 kvm-intel.vmm_exclusive=1 kvm-intel.fasteoi=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 root=/dev/sda”

Or you can add it inside the /etc/default/grub file, and modify the “GRUB_CMDLINE_LINUX_DEFAULT” parameter, and run “update-grub” to modify the /boot/grub/grub.cfg file:

In all these scenario, what are all the available kernel options which you can enable?

In the latest count, there are about 460+ kernel options available (full list is here: https://tthtlc.wordpress.com/a-list-of-kernel-command-line-option-as-collected-from-kernel-source-code/):

Too many, so what are most useful ones?

Cannot answer myself either. But for debugging:

These seemed to be useful.

In particular, one “slub_debug” is used for debugging a recently discoverd vulnerability (CVE-2017-7533):

https://bugzilla.redhat.com/attachment.cgi?id=1296934

The kernel function for handling “slub_debug” (from above) is “setup_slub_debug” (inside mm/slub.c) from where we can read and understand how the option is handled by the kernel: