How to understand the crashdump and identify the source codes in Linux Kernel?

Problem:

Got this in my crash dump, what does it mean? How to identify exactly where in the assembly listing this dump comes from?

[   90.984131] dump_stack+0xe6/0x15b
[   90.984346] ? _atomic_dec_and_lock+0x165/0x165
[   90.984628] ? do_raw_spin_lock+0xf7/0x1e0
[   90.984899] ubsan_epilogue+0x12/0x53
[   90.985131] __ubsan_handle_type_mismatch+0x1f1/0x2c6
[   90.985443] ? ubsan_epilogue+0x53/0x53
[   90.985683] ? debug_lockdep_rcu_enabled+0x26/0x40
[   90.985979] ? ubsan_epilogue+0x53/0x53
[   90.986219] ? __build_skb+0x87/0x320
[   90.986452] dev_gro_receive+0x1b2b/0x1c00
[   90.986805] ? trace_event_raw_event_fib_table_lookup_nh+0x8/0x4b0
[   90.987205] napi_gro_receive+0xa9/0x410
[   90.987446] e1000_clean_rx_irq+0x76f/0x1670
[   90.987709] ? e1000_alloc_jumbo_rx_buffers+0xd80/0xd80
[   90.988052] e1000_clean+0x8c2/0x2b60
[   90.988284] ? trace_hardirqs_on+0xd/0x10
[   90.988527] ? __lock_acquire+0xf4/0x39c0
[   90.988794] ? e1000_clean_tx_ring+0x3c0/0x3c0
[   90.989076] ? __ubsan_handle_type_mismatch+0x29e/0x2c6
[   90.989396] ? time_hardirqs_on+0x2a/0x380
[   90.989712] ? ubsan_epilogue+0x53/0x53
[   90.989947] ? net_rx_action+0x22e/0xe90
[   90.990207] net_rx_action+0x818/0xe90
[   90.990438] ? napi_complete_done+0x440/0x440
[   90.990726] ? debug_lockdep_rcu_enabled+0x26/0x40
[   90.991050] ? __do_softirq+0x18a/0xbc4
[   90.991293] ? time_hardirqs_on+0x2a/0x380
[   90.991560] ? do_raw_spin_trylock+0xd0/0xd0
[   90.991818] ? __do_softirq+0x18a/0xbc4
[   90.992063] __do_softirq+0x1ae/0xbc4
[   90.992336] irq_exit+0x149/0x1d0
[   90.992540] do_IRQ+0x92/0x1b0
[   90.992730] common_interrupt+0x90/0x90
[   90.992963] RIP: 0010:ext4_da_write_begin+0x5fa/0xee0

Firstly, I will assumed you know that the crashdump above are all return addresses of a “call xxxx” command. And so if you mapped it to the assembly listing in the “vmlinux” files when linux kernel is generated, you will see a “call xxx” just before the addresses above.

The cause of the crashdump is somewhere at the top, as it is the cause of triggering the UBSAN compiled kernel, and thus leading to calling dump_stack() function.

Therefore start reading from the top:

First we will need to convert the “vmlinux” output to its assembly listing:

objdump -d vmlinux > vmlinux.S

Then go direct to the “dump_stack” symbol:

So the address is “0xffffffff81e0ae65” for dump_stack():

dump_stack+0xe6/0x15b ==> 0xffffffff81e0ae65 + 0xe6 = 0xffffffff81e0af4b

So now looking at “0xffffffff81e0af4b”:

So we know the dump address is the address immediately after “call show_stack()” function call.

Therefore, using “addr2line -e vmlinux 0xffffffff81e0af4b” we get:

/home/tteikhua/linux-4.12/lib/dump_stack.c:54

Now we need to direct to the source:

So line 54 is exactly after the “__dump_stack()” call, and we can now do the same for all other calls.

cat crash.addr | sed 's/.* //' | sed 's/^M//' |sed 's//.*//' |sed 's/+/ /' > /tmp/tmp$$
 backtrace=20
 forwardtrace=40
 cat /tmp/tmp$$ | while
 read data data1
 do
 addr=`grep "<${data}.:$" vmlinux.S | sed 's/ .*//'`
 output=`addr2line -e ./vmlinux 0x$addr | sed 's/ .*//'`
 addr1=`bash ./addhex.bash 0x$addr $data1`
 output=`addr2line -e ./vmlinux 0x$addr1 | sed 's/ .*//'`
 echo "${output}"
 source=`echo $output | tail -1`
 filename=`echo $source | sed 's/:.*//'`
 line=`echo $source | sed 's/.*://'`
 line1=`expr ${line} - ${backtrace}`

echo "==========================================================="
 cat $filename | nl -ba | sed -n "${line1},+${forwardtrace}p" | sed "${backtrace}s/^/>>/"

echo "==========================================================="
 done

As the automated script above contained some control codes, the attached file is better (output as below):

https://drive.google.com/open?id=0B1hg6_FvnEa8cGpvZEdxTFNCVDA

https://drive.google.com/open?id=0B1hg6_FvnEa8MHEwNU10VDRXQzA

References:

http://helenfornazier.blogspot.sg/2015/07/linux-kernel-memory-corruption-debug.html

https://serverfault.com/questions/605946/kernel-stack-trace-to-source-code-lines

https://stackoverflow.com/questions/6151538/addr2line-on-kernel-module

http://elinux.org/Addr2line_for_kernel_debugging

http://www.linuxdevcenter.com/cmd/cmd.csp?path=a/addr2line

https://plus.google.com/+TheodoreTso/posts/h8H9z4EopkS

Advertisements

One response to this post.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: