Archive for the ‘filesystem’ Category

The reflink(2) system call v2. LWN.net

https://lwn.net/Articles/332802/

https://pkalever.wordpress.com/2016/01/22/xfs-reflinks-tutorial/

https://lwn.net/Articles/331808/

https://blogs.oracle.com/devpartner/entry/whats_new_in_oracle_linux

https://blogs.oracle.com/wim/entry/ocfs2_reflink

https://lkml.org/lkml/2016/10/12/176

https://bbs.archlinux.org/viewtopic.php?id=212825

https://patchwork.kernel.org/patch/9480871/

Advertisements

Operating Systems: File-System Implementation

A very good writeup on Filesystem Implementation internals:

https://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/12_FileSystemImplementation.html

12_11_IO_wo_BufferCache.jpg

Figure 12.11 – I/O without a unified buffer cache.

12_12_UnifiedBufferCache.jpg

Figure 12.12 – I/O using a unified buffer cache.

  • Page replacement strategies can be complicated with a unified cache, as one needs to decide whether to replace process or file pages, and how many pages to guarantee to each category of pages. Solaris, for example, has gone through many variations, resulting in priority paging giving process pages priority over file I/O pages, and setting limits so that neither can knock the other completely out of memory.
  • Another issue affecting performance is the question of whether to implement synchronous writes or asynchronous writes. Synchronous writes occur in the order in which the disk subsystem receives them, without caching; Asynchronous writes are cached, allowing the disk subsystem to schedule writes in a more efficient order ( See Chapter 12. ) Metadata writes are often done synchronously. Some systems support flags to the open call requiring that writes be synchronous, for example for the benefit of database systems that require their writes be performed in a required order.
  • The type of file access can also have an impact on optimal page replacement policies. For example, LRU is not necessarily a good policy for sequential access files. For these types of files progression normally goes in a forward direction only, and the most recently used page will not be needed again until after the file has been rewound and re-read from the beginning, ( if it is ever needed at all. ) On the other hand, we can expect to need the next page in the file fairly soon. For this reason sequential access files often take advantage of two special policies:
    • Free-behind frees up a page as soon as the next page in the file is requested, with the assumption that we are now done with the old page and won’t need it again for a long time.
    • Read-ahead reads the requested page and several subsequent pages at the same time, with the assumption that those pages will be needed in the near future. This is similar to the track caching that is already performed by the disk controller, except it saves the future latency of transferring data from the disk controller memory into motherboard main memory.
  • The caching system and asynchronous writes speed up disk writes considerably, because the disk subsystem can schedule physical writes to the disk to minimize head movement and disk seek times. ( See Chapter 12. ) Reads, on the other hand, must be done more synchronously in spite of the caching system, with the result that disk writes can counter-intuitively be much faster on average than disk reads.

BTRFS utility result in kernel crash

I have installed Qubes OS, which comes installed with BTRFS filesystem.

Have it dual boot with CentOS (by installing CentOS first, and Qubes will autodetect that).

To debug a infinite loop in Qubes, I have it bootup in CentOS, which have kdump enabled (as below):

http://linuxsysconfig.com/2013/03/kdump-on-centos-6

And after "sudo yum install btrfs-prog", and then using "mount /dev/sda6 /mnt" where /dev/sda6 is a BTRFS formatted Qubes partition, it immediately result in kdump generated crash.

http://wiki.1tux.org/wiki/RHEL/Btrfs

https://btrfs.wiki.kernel.org/index.php/Getting_started

KERNEL: /boot/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
CPUS: 8
DATE: Sun Sep 14 16:27:27 2014
UPTIME: 00:16:21
LOAD AVERAGE: 0.08, 0.09, 0.08
TASKS: 395
NODENAME: tthtlchp
RELEASE: 2.6.32-358.el6.x86_64
VERSION: #1 SMP Fri Feb 22 00:31:26 UTC 2013
MACHINE: x86_64 (2294 Mhz)
MEMORY: 7.9 GB
PANIC: "Oops: 0000 [#1] SMP " (check log for details)
PID: 4430
COMMAND: "mount"
TASK: ffff88023e317540 [THREAD_INFO: ffff880236520000]
CPU: 0
STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 4430 TASK: ffff88023e317540 CPU: 0 COMMAND: "mount"
#0 [ffff880236521820] machine_kexec at ffffffff81035b7b
#1 [ffff880236521880] crash_kexec at ffffffff810c0db2
#2 [ffff880236521950] oops_end at ffffffff815111d0
#3 [ffff880236521980] no_context at ffffffff81046bfb
#4 [ffff8802365219d0] __bad_area_nosemaphore at ffffffff81046e85
#5 [ffff880236521a20] bad_area at ffffffff81046fae
#6 [ffff880236521a50] __do_page_fault at ffffffff81047760
#7 [ffff880236521b70] do_page_fault at ffffffff8151311e
#8 [ffff880236521ba0] page_fault at ffffffff815104d5
[exception RIP: selinux_set_mnt_opts+63]
RIP: ffffffff8122a16f RSP: ffff880236521c58 RFLAGS: 00010292
RAX: ffffffffa079da73 RBX: ffff880236521ce8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880236521ce8 RDI: ffff880258c18000
RBP: ffff880236521cd8 R8: 0000000000000000 R9: 000000000000001d
R10: 0000000000000003 R11: 0000000000031d84 R12: ffff880258c18000
R13: 0000000000000000 R14: ffff880258c18000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#9 [ffff880236521ce0] superblock_doinit at ffffffff8122a711
#10 [ffff880236521d30] selinux_sb_kern_mount at ffffffff8122a7c9
#11 [ffff880236521df0] security_sb_kern_mount at ffffffff8121b8b6
#12 [ffff880236521e00] vfs_kern_mount at ffffffff81183847
#13 [ffff880236521e50] do_kern_mount at ffffffff811839c2
#14 [ffff880236521ea0] do_mount at ffffffff811a3c12
#15 [ffff880236521f20] sys_mount at ffffffff811a42a0
#16 [ffff880236521f80] system_call_fastpath at ffffffff8100b072
RIP: 00007f582966109a RSP: 00007ffffb177f88 RFLAGS: 00010206
RAX: 00000000000000a5 RBX: ffffffff8100b072 RCX: ffffffffc0ed0000
RDX: 00007f582a9a4640 RSI: 00007f582a99b850 RDI: 00007f582a99b830
RBP: 00000000c0ed0000 R8: 0000000000000000 R9: 00007f582a9a4640
R10: ffffffffc0ed0000 R11: 0000000000000206 R12: 0000000000000000
R13: 00007f582a99b850 R14: 00007f582a99b830 R15: 0000000000000000
ORIG_RAX: 00000000000000a5 CS: 0033 SS: 002b
crash>

Above indicated crash at selinux_set_mnt_opts:

crash> dis selinux_set_mnt_opts+63 100
0xffffffff8122a16f <selinux_set_mnt_opts+63>: mov 0x0(%r13),%rax
0xffffffff8122a173 <selinux_set_mnt_opts+67>: mov 0x68(%rax),%rax
0xffffffff8122a177 <selinux_set_mnt_opts+71>: mov 0x10(%rax),%rax
0xffffffff8122a17b <selinux_set_mnt_opts+75>: mov 0x230(%rax),%rax
0xffffffff8122a182 <selinux_set_mnt_opts+82>: mov %rax,-0x58(%rbp)
0xffffffff8122a186 <selinux_set_mnt_opts+86>: lea 0x30(%r13),%rax
0xffffffff8122a18a <selinux_set_mnt_opts+90>: mov (%rsi),%rbx
0xffffffff8122a18d <selinux_set_mnt_opts+93>: mov 0x8(%rsi),%r15
0xffffffff8122a191 <selinux_set_mnt_opts+97>: mov 0x10(%rsi),%esi
0xffffffff8122a194 <selinux_set_mnt_opts+100>: mov %rax,%rdi
0xffffffff8122a197 <selinux_set_mnt_opts+103>: mov %rax,-0x50(%rbp)
0xffffffff8122a19b <selinux_set_mnt_opts+107>: mov %esi,-0x44(%rbp)

where %r13 is completely NULL.

A "dmesg" command in crash show the place of crash as well:

TECH PREVIEW: btrfs may not be fully supported.
Please review provided documentation for limitations.
Btrfs loaded
device label qubes_dom0 devid 1 transid 154 /dev/sda6
btrfs: disk space caching is enabled
BTRFS: couldn't mount because of unsupported optional features (40).
btrfs: open_ctree failed
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [] selinux_set_mnt_opts+0x3f/0x580
PGD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda/dev
CPU 0
Modules linked in: btrfs(T) zlib_deflate lzo_decompress lzo_compress libcrc32c fuse ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge autofs4 sunrpc 8021q garp stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun kvm_intel kvm uinput arc4 hp_accel lis3lv02d input_polldev iwlwifi mac80211 cfg80211 rfkill r8169 mii sg uvcvideo videodev v4l2_compat_ioctl32 microcode i2c_i801 iTCO_wdt iTCO_vendor_support shpchp snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc xhci_hcd ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom ahci i915 nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi video output wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 4430, comm: mount Tainted: G --------------- T 2.6.32-358.el6.x86_64 #1 Hewlett-Packard HP Pavilion dv6 Notebook PC/181B
RIP: 0010:[] [] selinux_set_mnt_opts+0x3f/0x580
RSP: 0018:ffff880236521c58 EFLAGS: 00010292
RAX: ffffffffa079da73 RBX: ffff880236521ce8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880236521ce8 RDI: ffff880258c18000
RBP: ffff880236521cd8 R08: 0000000000000000 R09: 000000000000001d
R10: 0000000000000003 R11: 0000000000031d84 R12: ffff880258c18000
R13: 0000000000000000 R14: ffff880258c18000 R15: 0000000000000000
FS: 00007f582a38f7e0(0000) GS:ffff88002c200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000236600000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process mount (pid: 4430, threadinfo ffff880236520000, task ffff88023e317540)
Stack:
ffff880256306400 0000000000000006 ffffffffa079da73 ffff880254be1c00
ffff880232de2530 ffff880258c2b5c0 ffff880258c18000 ffff880232de2000
0000000000000000 ffffffffffffffea ffff880236521cc8 ffff880236521ce8
Call Trace:
[] superblock_doinit+0x61/0xd0
[] ? deactivate_locked_super+0x5e/0x90
[] selinux_sb_kern_mount+0x49/0xd0
[] security_sb_kern_mount+0x16/0x20
[] vfs_kern_mount+0xa7/0x1b0
[] do_kern_mount+0x52/0x130
[] do_mount+0x2d2/0x8d0
[] ? strndup_user+0x64/0xc0
[] sys_mount+0x90/0xe0
[] system_call_fastpath+0x16/0x1b
Code: 00 00 65 48 8b 04 25 c0 cb 00 00 48 8b 80 48 06 00 00 49 89 fe 48 89 45 98 48 8b 47 30 4c 8b af c0 00 00 00 48 8b 00 48 89 45 90 8b 45 00 48 8b 40 68 48 8b 40 10 48 8b 80 30 02 00 00 48 89
RIP [] selinux_set_mnt_opts+0x3f/0x580
RSP
CR2: 0000000000000000

Searching for the btrfs-prog source code:

yumdb info btrfs-progs
Loaded plugins: fastestmirror, refresh-packagekit
btrfs-progs-0.20-0.2.git91d9eec.el6.x86_64
checksum_data = 689790cd69d452ee13936f47a96f308d3ae876245477c523e37f5f0b195136db
checksum_type = sha256

http://vault.centos.org/6.4/os/Source/SPackages/

The same package name was found.

After downloading the source, and using ctags to generate the tags file (or just “make tags” inside linux kernel untarred directory), and using vi to navigate the source code (using ctrl-]), the following code was identified:

static int superblock_doinit(struct super_block *sb, void *data)
{
int rc = 0;
char *options = data;
struct security_mnt_opts opts;

security_init_mnt_opts(&opts);

if (!data)
goto out;

BUG_ON(sb->s_type->fs_flags & FS_BINARY_MOUNTDATA);

rc = selinux_parse_opts_str(options, &opts);
if (rc)
goto out_err;

out:
rc = selinux_set_mnt_opts(sb, &opts);

And according to Linux kernel calling convention (http://eli.thegreenplace.net/2011/09/06/stack-frame-layout-on-x86-64/), %RDI is the first argument, (value is 0xffff880258c18000):

crash> x /100x 0xffff880258c18000
0xffff880258c18000: 0x58c18000 0xffff8802 0x58c18000 0xffff8802
0xffff880258c18010: 0x0000001d 0x00000000 0x00001000 0x00000000
0xffff880258c18020: 0x0000000c 0x00000000 0xffffffff 0x7fffffff
0xffff880258c18030: 0xa07ac8e0 0xffffffff 0xa0799080 0xffffffff
0xffff880258c18040: 0x81623040 0xffffffff 0x816230e0 0xffffffff
0xffff880258c18050: 0xa079cfc0 0xffffffff 0x20810000 0x00000000
0xffff880258c18060: 0x9123683e 0x00000000 0x00000000 0x00000000
0xffff880258c18070: 0x00000000 0x00000000 0x00000000 0x00000000

Oh….well, if u traverse down the binaries, it is possible to find out where is the fault (one of the following lines):

static int selinux_set_mnt_opts(struct super_block *sb,
struct security_mnt_opts *opts)
{
const struct cred *cred = current_cred();
int rc = 0, i;
struct superblock_security_struct *sbsec = sb->s_security;
const char *name = sb->s_type->name;
struct inode *inode = sbsec->sb->s_root->d_inode;
struct inode_security_struct *root_isec = inode->i_security;
u32 fscontext_sid = 0, context_sid = 0, rootcontext_sid = 0;
u32 defcontext_sid = 0;
char **mount_options = opts->mnt_opts;
int *flags = opts->mnt_opts_flags;
int num_opts = opts->num_mnt_opts;

Exploring BTRFS filesystem internals (userspace + kernel)

 
Starting with a design documentation:
 
 
A good intro writeup:
 
 
Lots of technical details:
 
 
 
 
 
 
 
 
Elaborating on the COW mechanism in BTRFS (similar to ZFS):
 
 
 
Performance comparison:
 
 
Now some hands-on (a good system admin guide:   http://docs.oracle.com/cd/E37670_01/E37355/html/ol_use_case7_btrfs.html):
 
First create a file to be used as a virtual disk:
 
truncate -s 10G test-image.1
 
Next ensure the userspace tools for BTRFS are installed (I am on Ubuntu 12.04 LTS 64-bit):
 
sudo apt-get install btrfs-tools
 
Next make BTRFS filesystem on the virtual disk:
 
mkfs.btrfs test-image.1
 
Is the btrfs kernel module loaded?
 
lsmod|grep btr
 
No.   Check if it is compiled into the kernel?
 
grep -i btrfs /boot/config-3.8.0-29-generic 
 
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
 
Oh, yes, it is compiled into kernel as a loadable module.   So we shall load it:
 
sudo modprobe btrfs
 
Now use that to mount a BTRFS filesystem:
 
sudo mkdir  /btrfs
sudo mount test-image.1 /btrfs
sudo chmod 777 /btrfs/
Next we can start doing filesystem testing on /btrfs, using tools from below:
 
 
And to understand the filesystem internals, we can use ftrace, the sequence of operation are captured essentially as follows:
 
#!/bin/bash
sudo mkdir /debug
sudo mount -t debugfs nodev /debug
echo 0 >/debug/tracing/tracing_on
echo ‘btrfs_*’ ‘journal_*’ > /debug/tracing/set_ftrace_filter
 
cat /debug/tracing/enabled_functions 
 
We can see many BTRFS functions which is partially listed here:
 
btrfs_create_qgroup [btrfs] (1)
btrfs_remove_qgroup [btrfs] (1)
btrfs_limit_qgroup [btrfs] (1)
btrfs_qgroup_record_ref [btrfs] (1)
btrfs_qgroup_account_ref [btrfs] (1)
btrfs_run_qgroups [btrfs] (1)
btrfs_qgroup_inherit [btrfs] (1)
btrfs_qgroup_reserve [btrfs] (1)
btrfs_qgroup_free [btrfs] (1)
btrfs_ioctl_send [btrfs] (1)
btrfs_init_dev_replace [btrfs] (1)
btrfs_after_dev_replace_commit [btrfs] (1)
btrfs_dev_replace_is_ongoing [btrfs] (1)
btrfs_dev_replace_lock [btrfs] (1)
btrfs_dev_replace_unlock [btrfs] (1)
btrfs_resume_dev_replace_async [btrfs] (1)
btrfs_dev_replace_finishing [btrfs] (1)
btrfs_dev_replace_suspend_for_unmount [btrfs] (1)
btrfs_dev_replace_cancel [btrfs] (1)
btrfs_dev_replace_status [btrfs] (1)
btrfs_dev_replace_kthread [btrfs] (1)
btrfs_dev_replace_start [btrfs] (1)
btrfs_run_dev_replace [btrfs] (1)
btrfs_set_acl [btrfs] (1)
btrfs_xattr_acl_set [btrfs] (1)
btrfs_get_acl [btrfs] (1)
btrfs_xattr_acl_get [btrfs] (1)
btrfs_init_acl [btrfs] (1)
btrfs_acl_chmod [btrfs] (1)
btrfs_interface_exit [btrfs] (1)
btrfs_find_highest_objectid [btrfs] (1)
 
echo function >/debug/tracing/current_tracer
echo 1 >/debug/tracing/tracing_on
ls -alR /btrfs
cp /bin/l* /btrfs
sleep 3
echo 0 >/debug/tracing/tracing_on
cat /debug/tracing/trace
 
And looking into the output:
 
  1.            <…>-7497  [003] ….  8751.390285: btrfs_real_readdir <-vfs_readdir
  2.            <…>-7497  [003] ….  8751.390286: btrfs_alloc_path <-btrfs_real_readdir
  3.            <…>-7497  [003] ….  8751.390288: btrfs_get_delayed_items <-btrfs_real_readdir
  4.            <…>-7497  [003] ….  8751.390288: btrfs_get_delayed_node <-btrfs_get_delayed_items
  5.            <…>-7497  [003] ….  8751.390289: btrfs_search_slot <-btrfs_real_readdir
And the full trace is here (for pid=7497):
 
 
And some of the userspace command features of BTRFS:
 

btrfs
Usage:
btrfs subvolume snapshot <source> [<dest>/]<name>
Create a writable snapshot of the subvolume <source> with
the name <name> in the <dest> directory.
btrfs subvolume delete <subvolume>
Delete the subvolume <subvolume>.
btrfs subvolume create [<dest>/]<name>
Create a subvolume in <dest> (or the current directory if
not passed).
btrfs subvolume list <path>
List the snapshot/subvolume of a filesystem.
btrfs subvolume find-new <path> <last_gen>
List the recently modified files in a filesystem.
btrfs filesystem defragment [-vcf] [-s start] [-l len] [-t size] <file>|<dir> [<file>|<dir>…]
Defragment a file or a directory.
btrfs subvolume set-default <id> <path>
Set the subvolume of the filesystem <path> which will be mounted
as default.
btrfs filesystem sync <path>
Force a sync on the filesystem <path>.
btrfs filesystem resize [+/-]<newsize>[gkm]|max <filesystem>
Resize the file system. If ‘max’ is passed, the filesystem
will occupe all available space on the device.
btrfs filesystem show [<uuid>|<label>]
Show the info of a btrfs filesystem. If no <uuid> or <label>
is passed, info of all the btrfs filesystem are shown.
btrfs filesystem df <path>
Show space usage information for a mount point
.
btrfs filesystem balance <path>
Balance the chunks across the device.
btrfs device scan [<device> [<device>..]
Scan all device for or the passed device for a btrfs
filesystem.
btrfs device add <dev> [<dev>..] <path>
Add a device to a filesystem.
btrfs device delete <dev> [<dev>..] <path>
Remove a device from a filesystem.

btrfs help|–help|-h
Show the help.http://docs.oracle.com/cd/E37670_01/E37355/html/ol_use_case7_btrfs.html

Btrfs Btrfs v0.19

 
And some other commands:
 
btrfsctl
btrfs-map-logical
btrfs-vol
btrfsck btrfs-debug-tree
btrfs-show
btrfs-convert
btrfs-image
btrfstune
 
 
 

How to do kernel debugging via gdb over serial port via QEMU?

The host OS is Ubuntu 12.04 LTS 64-bit, with qemu binaries installed (this include qemu-img, qemu-system-x86_64 etc). The Qemu guest will be running the linux kernel to be debugged. It is also running CentOS 64-bit as the distribution.

First create the Qemu image if needed:

qemu-img create centos64.img 20G

Install CentOS (as the guest OS) inside the Qemu guest (assuming that the CentOS-6.4-x86_64-minimal.iso has been downloaded):

qemu-system-x86_64 -m 2048 -net user -net nic -enable-kvm -hda centos64.img -cdrom CentOS-6.4-x86_64-minimal.iso

For cases of no serial debugging is needed, this is the way to use the quest:

qemu-system-x86_64 -m 2048 -net user -net nic -enable-kvm centos64.img

If additional harddisk image (eg, ext4.img) is needed, just append “-hdb ext4.img” to the above line. Inside the guest, you may need to issue “dhclient” acquire a TCP IP address for the guest before it can connect to outside world.

For cases of serial debugging is needed:

qemu-system-x86_64 -s -S -m 2048 -net user -net nic -enable-kvm centos64.img

The “-s -S” will stop the guest from execution just before executing the first instruction in the guest. Doing a “man qemu”:

-S Do not start CPU at startup (you must type 'c' in the monitor).

-gdb dev
Wait for gdb connection on device dev. Typical connections will
likely be TCP-based, but also UDP, pseudo TTY, or even stdio are
reasonable use case. The latter is allowing to start qemu from
within gdb and establish the connection via a pipe:

(gdb) target remote | exec qemu -gdb stdio ...

-s Shorthand for -gdb tcp::1234, i.e. open a gdbserver on TCP port
1234.

So in the Ubuntu host issue the gdb command:

gdb vmlinux

where “vmlinux” is the uncompressed image generated from the compiled linux kernel running INSIDE the guest machine. In other words, after installing the guest OS, download the linux kernel and compile it, as some Linux distribution does not provide the uncompress vmlinux image.

In Ubuntu this is not a problem:

http://superuser.com/questions/62575/where-is-vmlinux-on-my-ubuntu-installation

But in CentOS it is not available:

https://www.centos.org/forums/viewtopic.php?t=1491

So the best option is to compile the kernel inside the guest, modify the grub.conf to boot up into that kernel version (not the default one!!), and then copy out the vmlinux image to the host machine for the use of gdb command.

After “gdb vmlinux“, enter “target remote localhost:1234” to connect to the gdbserver inside the Qemu guest, and then “cont” to continue the gdb execution.

And doing a “bt” inside gdb:

(gdb) bt
Remote 'g' packet reply is too long: 000000000000000000000000000000000000000000000000000000000000000001000000000000002802de81ffffffffc81ea081ffffffffc81ea081ffffffff0000000000000000000000000000000002000000000000000000000000000000c075c081ffffffff0000000000000000ffffffffffffffff50370900000000002bb90381ffffffff4602000010000000180000001800000018000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f03000000000000000000000000000000000000000000000000000000000000000000000000ff0000000000000000ff6564645f6964002d2d6578706f727400000000000000000000000000000000000000000000000000ff0000000000ffff20000000000000004100000000000000404040404040404040404040404040405b5b5b5b5b5b5b5b5b5b5b5b5b5b5b5b202020202020202020202020202020200000000000000000000000000000000000000000000000ff0000000000ffffff20202020202000202020202020202000ffffffffffffffffffffffffffffffff000000000000000000000000000000000000000000000000000000 000000000 00000000000000000000000000000000000000000000000000000000000000000801f0000

What is this error? Googling revealed the answer:

http://stackoverflow.com/questions/4896316/problem-gdb-remote-debugging-remote-g-packet-reply-is-too-long

And so:

(gdb) set architecture i386:x86-64:intel
The target architecture is assumed to be i386:x86-64:intel

Trying again, and now the backtrace beautifully displayed:

(gdb) bt
#0 native_safe_halt ()
at /root/arch/x86/include/asm/irqflags.h:50
#1 0xffffffff81014a5d in raw_safe_halt ()
at /root/arch/x86/include/asm/paravirt.h:110
#2 default_idle () at arch/x86/kernel/process.c:328
#3 0xffffffff81009fc6 in cpu_idle () at arch/x86/kernel/process_64.c:140
#4 0xffffffff814f160a in rest_init () at init/main.c:483
#5 0xffffffff81c27f7b in start_kernel () at init/main.c:740
#6 0xffffffff81c2733a in x86_64_start_reservations (
real_mode_data=<optimized out>) at arch/x86/kernel/head64.c:123
#7 0xffffffff81c27438 in x86_64_start_kernel (
real_mode_data=0x93750 <Address 0x93750 out of bounds>)
at arch/x86/kernel/head64.c:94
#8 0x0000000000000000 in ?? ()

Next you can try “rbreak vfs_*” to breakpoint on all the vfs_* functions:

Breakpoint 50, vfs_dq_drop (inode=0xffff88003786ca48) at fs/quota/dquot.c:1327
1327 fs/quota/dquot.c: No such file or directory.
(gdb) bt
#0 vfs_dq_drop (inode=0xffff88003786ca48) at fs/quota/dquot.c:1327
#1 0xffffffff8119c8b3 in clear_inode (inode=0xffff88003786ca48)
at fs/inode.c:334
#2 0xffffffff8119d0f3 in generic_forget_inode (inode=0xffff88003786ca48)
at fs/inode.c:1306
#3 generic_drop_inode (inode=0xffff88003786ca48) at fs/inode.c:1321
#4 0xffffffff8119bf72 in iput_final (inode=0xffff88003786ca48)
at fs/inode.c:1343
#5 iput (inode=0xffff88003786ca48) at fs/inode.c:1361
#6 0xffffffff81198bc0 in dentry_iput (dentry=0xffff880037a84cc0)
at fs/dcache.c:118
#7 0xffffffff81198d21 in d_kill (dentry=0xffff880037a84cc0) at fs/dcache.c:177
#8 0xffffffff8119a8cc in dput (dentry=0xffff880037a84cc0) at fs/dcache.c:256
#9 0xffffffff81182189 in __fput (file=0xffff88007caa1140)
at fs/file_table.c:266
#10 0xffffffff81182235 in fput (file=<optimized out>) at fs/file_table.c:199
#11 0xffffffff8117d68d in filp_close (filp=0xffff88007caa1140,
id=0xffff880037c24e80) at fs/open.c:971
#12 0xffffffff810713cf in close_files (files=0xffff880037c24e80)
at kernel/exit.c:495
#13 put_files_struct (files=0xffff880037c24e80) at kernel/exit.c:523
#14 0xffffffff81071493 in exit_files (tsk=0xffff88007c96d500)
at kernel/exit.c:557
---Type <return> to continue, or q <return> to quit---
#15 0xffffffff8107350d in do_exit (code=256) at kernel/exit.c:986
#16 0xffffffff81073c48 in do_group_exit (exit_code=256) at kernel/exit.c:1091
#17 0xffffffff81073cd7 in sys_exit_group (error_code=<optimized out>)
at kernel/exit.c:1102
#18 0xffffffff8100b072 in ?? () at arch/x86/kernel/entry_64.S:488
#19 0x00007fbd9438b028 in ?? ()
#20 0x0000000000000000 in ?? ()

From above you can easily see who are the callee and callers functions.

But there is another problem. If you try “rbreak ext4_*” you will encounter that there is no symbol found. Checking for “ext4” symbol inside the vmlinux file clearly revealed that the function name is also not available inside (“nm vmlinux | grep ext4”). But inside the guest, “grep ext4 /proc/kallsyms” (running as root, as non-root don’t give you the privileges of getting the addresses) clearly revealed that the function addresses and function names are available, not sure how kallsyms get all its names. And by the way, if the addresses in /proc/kallsyms (inside the guest) does not match the addresses in vmlinux file (in the host) then you will not be able to set breakpoint either, or not breaking even after breakpoint is set.

To solve the above problem, copy out the /proc/kallsyms as a file, extract out the addresses needed for breakpoint, and do a “break *0xyyyyyyyy” to explicitly set breakpoint by address. Eg:

Breakpoint 17, 0xffffffffa00a6a70 in ?? ()
(gdb)
#0 0xffffffffa00a6a70 in ?? ()
#1 0xffffffffa00a736d in ?? ()
#2 0x0000000000000099 in ?? ()
#3 0xffff88003790bcc0 in ?? ()
#4 0xffffffffffffff10 in ?? ()
#5 0xffff88003790bde0 in ?? ()
#6 0x0000000000000099 in ?? ()
#7 0xffff88003790bcc0 in ?? ()

#8 0xffff880037ab2438 in ?? ()
#9 0xffff880037ab2588 in ?? ()
#10 0xffff88007c43bc80 in ?? ()
#11 0xffffffff811b3b0f in generic_block_bmap (mapping=<optimized out>,
block=<optimized out>, get_block=<optimized out>) at fs/buffer.c:3038
Backtrace stopped: frame did not save the PC

No names available, quite difficult to read. There is a solution!!!

Inside the guest OS, where CentOS is running, “lsmod | grep ext4” and you can see that ext4.ko is running as a kernel module.

Next, “cat /proc/modules | grep ext4” (running as root) and you can read off the loaded address of ext4.ko (0xffffffffa009e000 for the case below).

cat /proc/modules

i2c_piix4 12608 0 - Live 0xffffffffa0127000
i2c_core 31084 1 i2c_piix4, Live 0xffffffffa0119000
sg 29350 0 - Live 0xffffffffa010c000
ext4 363344 2 - Live 0xffffffffa009e000
jbd2 91554 1 ext4, Live 0xffffffffa007c000
mbcache 8193 1 ext4, Live 0xffffffffa0075000
sd_mod 38976 3 - Live 0xffffffffa0062000

Well, in the host Ubuntu, inside gdb:

(gdb) add-symbol-file /tmp/ext4.ko 0xffffffffa009e000

(where 0xffffffffa009e000 is the address where ext4.ko is shown to be loaded inside /proc/modules of the guest OS). To test out if it works:

(gdb) info breakpoints

Num Type Disp Enb Address What
1 breakpoint keep n 0xffffffffa00a6a70 in ext4_get_blocks
at fs/ext4/inode.c:1277
2 breakpoint keep n 0xffffffff81195cc0 in vfs_readdir
at fs/readdir.c:23
breakpoint already hit 13 times
3 breakpoint keep n 0xffffffffa009f970 in ext4_dir_llseek
at fs/ext4/dir.c:309
4 breakpoint keep n 0xffffffffa00a3270 in ext4_get_branch
at fs/ext4/inode.c:458
5 breakpoint keep n 0xffffffffa00a7180 in ext4_get_block_dio_write
at fs/ext4/inode.c:3694
6 breakpoint keep n 0xffffffffa00ba600 in ext4_get_sb
at fs/ext4/super.c:4519
7 breakpoint keep n 0xffffffffa00d8e00 in ext4_get_acl
at fs/ext4/acl.c:136
breakpoint already hit 9 times
8 breakpoint keep n 0xffffffffa00a5a90 in ext4_get_inode_loc
at fs/ext4/inode.c:5281
breakpoint already hit 18 times
9 breakpoint keep n 0xffffffffa00a2b30 in ext4_get_reserved_space
at fs/ext4/inode.c:1072
breakpoint already hit 2 times
10 breakpoint keep n 0xffffffffa00a3100 in ext4_getattr
at fs/ext4/inode.c:5911
breakpoint already hit 74 times
11 breakpoint keep n 0xffffffffa009e500 in ext4_get_group_desc
at fs/ext4/balloc.c:202
breakpoint already hit 20 times
12 breakpoint keep n 0xffffffffa00a2d60 in ext4_get_inode_flags
at fs/ext4/inode.c:5306
breakpoint already hit 15 times
13 breakpoint keep n 0xffffffffa009e000 in ext4_get_group_no_and_offset at fs/ext4/balloc.c:33
breakpoint already hit 2 times
14 breakpoint keep n 0xffffffffa00a72b0 in ext4_get_block
at fs/ext4/inode.c:1398
breakpoint already hit 15 times
15 breakpoint keep n 0xffffffffa00af1b0 in ext4_get_parent
at fs/ext4/namei.c:1065
16 breakpoint keep n 0xffffffffa00a86a0 in ext4_getblk
at fs/ext4/inode.c:1471
breakpoint already hit 3 times
17 breakpoint keep n 0xffffffffa00a6a70 in ext4_get_blocks
at fs/ext4/inode.c:1277
breakpoint already hit 20 times

Now all the ext4 functions are listed by names and even files lines offset. Doing a backtrace (“bt”):

(gdb) bt
#0 ext4_get_block (inode=0xffff88003790bcc0, iblock=324,
bh_result=0xffff88007c43bbe0, create=0) at fs/ext4/inode.c:1398
#1 0xffffffff811b3b0f in generic_block_bmap (mapping=<optimized out>,
block=<optimized out>, get_block=<optimized out>) at fs/buffer.c:3038
#2 0xffffffffa00a4752 in ext4_bmap (mapping=0xffff88003790bde0, block=324)
at fs/ext4/inode.c:3556
#3 0xffffffff8119b891 in bmap (inode=<optimized out>, block=<optimized out>)
at fs/inode.c:1381
#4 0xffffffffa0085fe3 in ?? ()
#5 0xffff88007c43bd20 in ?? ()
#6 0xffff88007cb00000 in ?? ()
#7 0xffff88007cb00024 in ?? ()
#8 0xffff88007c43bd08 in ?? ()
#9 0xffff88007c43bcf0 in ?? ()
#10 0xffffffffa008628a in ?? ()
#11 0xffff88007c8609c0 in ?? ()
#12 0xffff88007cb00000 in ?? ()
#13 0xffff88007c8609c0 in ?? ()
#14 0xffff88007cb00000 in ?? ()
#15 0xffff88007c43bd20 in ?? ()
#16 0xffffffffa0086e91 in ?? ()
#17 0xffff88007c8609c0 in ?? ()
#18 0xffff88007cb00000 in ?? ()
#19 0x0000000000000000 in ?? ()

Okay, the addresses are partially resolved…..still unresolved for other symbols yet.

Analysing other symbols (eg, vfs_readdir()):

Breakpoint 18, vfs_readdir (file=0xffff88007db05d80,
filler=0xffffffff81195b00 <filldir>, buf=0xffff88007d849f38)
at fs/readdir.c:23
23 {

Since the kernel source files are located in the same directory in the host OS where the kernel images are compiled inside the guest OS, you can clearly list the source via gdb:

(gdb) list
18 #include <linux/unistd.h>
19
20 #include <asm/uaccess.h>
21
22 int vfs_readdir(struct file *file, filldir_t filler, void *buf)
23 {
24 struct inode *inode = file->f_path.dentry->d_inode;
25 int res = -ENOTDIR;
26 if (!file->f_op || !file->f_op->readdir)
27 goto out;

Next, “print *file” clearly can decipher the structure of the pointer values into its structures in details.

(gdb) print *file
$4 = {f_u = {fu_list = {next = 0xffff88007cfe3d80, prev = 0xffff88007cb550e8},
fu_rcuhead = {next = 0xffff88007cfe3d80, func = 0xffff88007cb550e8}},
f_path = {mnt = 0xffff88007cfed9c0, dentry = 0xffff880037bd7240},
f_op = 0xffffffffa00d9c40, f_lock = {raw_lock = {slock = 0}}, f_count = {
counter = 2}, f_flags = 624640, f_mode = 29, f_pos = 0, f_owner = {lock = {
raw_lock = {lock = 16777216}}, pid = 0x0, pid_type = PIDTYPE_PID,
uid = 0, euid = 0, signum = 0}, f_cred = 0xffff880037c01480, f_ra = {
start = 0, size = 0, async_size = 0, ra_pages = 32, mmap_miss = 0,
prev_pos = -1}, f_version = 0, f_security = 0xffff88007d4d1f20,
private_data = 0x0, f_ep_links = {next = 0xffff88007db05e28,
prev = 0xffff88007db05e28}, f_mapping = 0xffff880037bdf5e0}

(gdb) print file->f_path
$7 = {mnt = 0xffff88007cfed9c0, dentry = 0xffff880037bd7240}

(gdb) print file->f_path->dentry
$9 = (struct dentry *) 0xffff880037bd7240

Given the structure is “struct dentry *” you can also dereference via gdb the structure:

(gdb) print *(struct dentry *)file->f_path->dentry
$10 = {d_count = {counter = 1}, d_flags = 8, d_lock = {raw_lock = {
slock = 0}}, d_mounted = 0, d_inode = 0xffff880037bdf4c0, d_hash = {
next = 0xffff88003794c998, pprev = 0xffffc9000002ee48},
d_parent = 0xffff880037bc56c0, d_name = {hash = 3818298688, len = 8,
name = 0xffff880037bd72e0 "incoming"}, d_lru = {next = 0xffff880037bd7340,
prev = 0xffff880037bd71c0}, d_u = {d_child = {next = 0xffff880037bd7350,
prev = 0xffff880037bd71d0}, d_rcu = {next = 0xffff880037bd7350,
func = 0xffff880037bd71d0}}, d_subdirs = {next = 0xffff880037bd72a0,
prev = 0xffff880037bd72a0}, d_alias = {next = 0xffff880037bdf4f0,
prev = 0xffff880037bdf4f0}, d_time = 14757395258967641292, d_op = 0x0,
d_sb = 0xffff88007cb55000, d_fsdata = 0x0,
d_iname = "incoming00\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314\314",
<incomplete sequence \314>}

And gdb can walk the structure and print the details of the field (with names) for you.

Reference:

http://files.meetup.com/1590495/debugging-with-qemu.pdf

http://www.ece.cmu.edu/~ee349/f-2012/lab2/qemu.pdf

http://stackoverflow.com/questions/8662468/remote-g-packet-reply-is-too-long

Have fun!!!

VirtualBox + kgdb: How to analyze ext4 filesystem processing in Linux kernel?

After the “gdb ./vmlinux” at the root of linux kernel source code directory, do a “rbreak ext4_*”:

(gdb) rbreak ext4_*
Breakpoint 1 at 0xffffffff8120ceb9: file fs/ext4/balloc.c, line 640.
int ext4_bg_has_super(struct super_block *, ext4_group_t);
Breakpoint 2 at 0xffffffff8120cf79: file fs/ext4/balloc.c, line 688.
long unsigned int ext4_bg_num_gdb(struct super_block *, ext4_group_t);
Breakpoint 3 at 0xffffffff8120cc65: file fs/ext4/balloc.c, line 478.
int ext4_claim_free_clusters(struct ext4_sb_info *, s64, unsigned int);
Breakpoint 4 at 0xffffffff8120ce50: file fs/ext4/balloc.c, line 565.
ext4_fsblk_t ext4_count_free_clusters(struct super_block *);
Breakpoint 5 at 0xffffffff8120d90e: file fs/ext4/balloc.c, line 222.
unsigned int ext4_free_clusters_after_init(struct super_block *, ext4_group_t,
struct ext4_group_desc *);
Breakpoint 6 at 0xffffffff8120c9fd: file fs/ext4/balloc.c, line 250.
struct ext4_group_desc *ext4_get_group_desc(struct super_block *,
ext4_group_t, struct buffer_head **);
Breakpoint 7 at 0xffffffff8120c9a9: file fs/ext4/balloc.c, line 765.
void ext4_get_group_no_and_offset(struct super_block *, ext4_fsblk_t,
ext4_group_t *, ext4_grpblk_t *);
Breakpoint 8 at 0xffffffff8120d0c6: file fs/ext4/balloc.c, line 167.
void ext4_init_block_bitmap(struct super_block *, struct buffer_head *,
ext4_group_t, struct ext4_group_desc *);
Breakpoint 9 at 0xffffffff8120d969: file fs/ext4/balloc.c, line 765. (2 locations)
ext4_fsblk_t ext4_inode_to_goal_block(struct inode *);
---Type <return> to continue, or q <return> to quit---

And so keep entering “enter” to set the breakpoints automatically. Not all the breakpoints can be set successfully. At about page 7 of the above some error occurred.

/build/buildd/gdb-7.1/gdb/breakpoint.c:6481: internal-error: expand_line_sal_maybe: Assertion `found' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n)
Please answer y or n.
/build/buildd/gdb-7.1/gdb/breakpoint.c:6481: internal-error: expand_line_sal_maybe: Assertion `found' failed.

So I stopped at about page 6 of setting breakpoints, good enough numbers of breakpoints.

(gdb) cont
Continuing.
[New Thread 180]
[Switching to Thread 180]

Breakpoint 6, ext4_get_group_desc (sb=0xffff880030c4f800, block_group=0,
bh=0x0) at fs/ext4/balloc.c:250
250 if (EXT4_HAS_INCOMPAT_FEATURE(sb, EXT4_FEATURE_INCOMPAT_FLEX_BG)) {
(gdb) bt
#0 ext4_get_group_desc (sb=0xffff880030c4f800, block_group=0, bh=0x0)
at fs/ext4/balloc.c:250
#1 0xffffffff812351e5 in ext4_check_descriptors (sb=0xffff880030c4f800,
data=<value optimized out>, silent=6) at fs/ext4/super.c:1965
#2 ext4_fill_super (sb=0xffff880030c4f800, data=<value optimized out>,
silent=6) at fs/ext4/super.c:3401
#3 0xffffffff8117cdd6 in mount_bdev (fs_type=<value optimized out>,
flags=<value optimized out>, dev_name=<value optimized out>, data=0x0,
fill_super=0xffffffff81234100 <ext4_fill_super>) at fs/super.c:1024
#4 0xffffffff812236e5 in ext4_mount (fs_type=<value optimized out>,
flags=<value optimized out>, dev_name=<value optimized out>,
data=<value optimized out>) at fs/ext4/super.c:4779
#5 0xffffffff8117dc13 in mount_fs (type=0xffffffff81c5c8c0, flags=32769,
name=<value optimized out>, data=<value optimized out>) at fs/super.c:1130
#6 0xffffffff81197972 in vfs_kern_mount (type=0xffffffff81c5c8c0,
flags=32769,
name=0xffff88001c6fee00 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=0x0) at fs/namespace.c:707
#7 0xffffffff811981b4 in do_kern_mount (fstype=0xffff88001c69ebf0 "ext4",
flags=32769,
name=0xffff88001c6fee00 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=<value optimized out>) at fs/namespace.c:1804
#8 0xffffffff811999a4 in do_new_mount (
---Type <return> to continue, or q <return> to quit---
dev_name=0xffff88001c6fee00 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c69ebf0 "ext4",
flags=<value optimized out>, data_page=0x0) at fs/namespace.c:1864
#9 do_mount (
dev_name=0xffff88001c6fee00 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c69ebf0 "ext4",
flags=<value optimized out>, data_page=0x0) at fs/namespace.c:2188
#10 0xffffffff8119a190 in sys_mount (
dev_name=0x7fff22af4e30 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type=<value optimized out>, flags=32769,
data=0x0) at fs/namespace.c:2387
#11 <signal handler called>
#12 0x00007f5a220d9cea in __brk_reservation_fn_dmi_alloc__ ()
#13 0xffff880030ceadc0 in __brk_reservation_fn_dmi_alloc__ ()
#14 0xffffffff81c1ece0 in ?? ()
#15 0x0000000000000000 in ?? ()

Analysing the stacktrace, we should know something:

  • These stacktrace are kernel stacks, and so since transitioning from userspace to kernel is always starting with syscall, the lowest part of the stacktrace is always the syscall function.
  • Reading upwards, it shows the calling sequence from syscall functions to other functions.
  • So in the stack trace above, we know the calling sequence start with syscall sys_mount(), and subsequently other functions – reading upwards.

    Some error occurred, so in the starting up of second attempt, the following (different) stacktrace were recorded:

    (gdb) bt
    #0 ext4_get_group_desc (sb=0xffff880030cdc800, block_group=0, bh=0x0)
    at fs/ext4/balloc.c:250
    #1 0xffffffff812351e5 in ext4_check_descriptors (sb=0xffff880030cdc800,
    data=<value optimized out>, silent=6) at fs/ext4/super.c:1965
    #2 ext4_fill_super (sb=0xffff880030cdc800, data=<value optimized out>,
    silent=6) at fs/ext4/super.c:3401
    #3 0xffffffff8117cdd6 in mount_bdev (fs_type=<value optimized out>,
    flags=<value optimized out>, dev_name=<value optimized out>, data=0x0,
    fill_super=0xffffffff81234100 <ext4_fill_super>) at fs/super.c:1024
    #4 0xffffffff812236e5 in ext4_mount (fs_type=<value optimized out>,
    flags=<value optimized out>, dev_name=<value optimized out>,
    data=<value optimized out>) at fs/ext4/super.c:4779
    #5 0xffffffff8117dc13 in mount_fs (type=0xffffffff81c5c8c0, flags=32769,
    name=<value optimized out>, data=<value optimized out>) at fs/super.c:1130
    #6 0xffffffff81197972 in vfs_kern_mount (type=0xffffffff81c5c8c0,
    flags=32769,
    name=0xffff88001c754140 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=0x0) at fs/namespace.c:707
    #7 0xffffffff811981b4 in do_kern_mount (fstype=0xffff88001c683570 "ext4",
    flags=32769,
    name=0xffff88001c754140 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=<value optimized out>) at fs/namespace.c:1804
    #8 0xffffffff811999a4 in do_new_mount (
    ---Type <return> to continue, or q <return> to quit---
    dev_name=0xffff88001c754140 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c683570 "ext4",
    flags=<value optimized out>, data_page=0x0) at fs/namespace.c:1864
    #9 do_mount (
    dev_name=0xffff88001c754140 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c683570 "ext4",
    flags=<value optimized out>, data_page=0x0) at fs/namespace.c:2188
    #10 0xffffffff8119a190 in sys_mount (
    dev_name=0x7fffa93e9e30 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type=<value optimized out>, flags=32769,
    data=0x0) at fs/namespace.c:2387
    #11 <signal handler called>
    #12 0x00007fa7b17decea in __brk_reservation_fn_dmi_alloc__ ()
    #13 0x0000000000000000 in ?? ()
    (gdb) cont
    Continuing.

    And it continue, never seemingly terminate or return back, neither any displayed in VirtualBox.

    And so I restart the whole thing again, now setting only one breakpoint – ext4_get_group_desc:

    Breakpoint 1, ext4_get_group_desc (sb=0xffff880030e8f000, block_group=0,
    bh=0x0) at fs/ext4/balloc.c:250
    250 if (EXT4_HAS_INCOMPAT_FEATURE(sb, EXT4_FEATURE_INCOMPAT_FLEX_BG)) {
    (gdb) bt
    #0 ext4_get_group_desc (sb=0xffff880030e8f000, block_group=0, bh=0x0)
    at fs/ext4/balloc.c:250
    #1 0xffffffff812351e5 in ext4_check_descriptors (sb=0xffff880030e8f000,
    data=<value optimized out>, silent=6) at fs/ext4/super.c:1965
    #2 ext4_fill_super (sb=0xffff880030e8f000, data=<value optimized out>,
    silent=6) at fs/ext4/super.c:3401
    #3 0xffffffff8117cdd6 in mount_bdev (fs_type=<value optimized out>,
    flags=<value optimized out>, dev_name=<value optimized out>, data=0x0,
    fill_super=0xffffffff81234100 <ext4_fill_super>) at fs/super.c:1024
    #4 0xffffffff812236e5 in ext4_mount (fs_type=<value optimized out>,
    flags=<value optimized out>, dev_name=<value optimized out>,
    data=<value optimized out>) at fs/ext4/super.c:4779
    #5 0xffffffff8117dc13 in mount_fs (type=0xffffffff81c5c8c0, flags=32769,
    name=<value optimized out>, data=<value optimized out>) at fs/super.c:1130
    #6 0xffffffff81197972 in vfs_kern_mount (type=0xffffffff81c5c8c0,
    flags=32769,
    name=0xffff88001c679f40 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=0x0) at fs/namespace.c:707
    #7 0xffffffff811981b4 in do_kern_mount (fstype=0xffff88001c6816a8 "ext4",
    flags=32769,
    name=0xffff88001c679f40 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", data=<value optimized out>) at fs/namespace.c:1804
    #8 0xffffffff811999a4 in do_new_mount (
    ---Type <return> to continue, or q <return> to quit---
    dev_name=0xffff88001c679f40 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c6816a8 "ext4",
    flags=<value optimized out>, data_page=0x0) at fs/namespace.c:1864
    #9 do_mount (
    dev_name=0xffff88001c679f40 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type_page=0xffff88001c6816a8 "ext4",
    flags=<value optimized out>, data_page=0x0) at fs/namespace.c:2188
    #10 0xffffffff8119a190 in sys_mount (
    dev_name=0x7fff299bfe30 "/dev/disk/by-uuid/30532f8e-6a1e-4cc1-96d9-e0b01a27e68e", dir_name=<value optimized out>, type=<value optimized out>, flags=32769,
    data=0x0) at fs/namespace.c:2387
    #11 <signal handler called>
    #12 0x00007fa6fa9b4cea in __brk_reservation_fn_dmi_alloc__ ()
    #13 0x000000000000ff4f in __brk_reservation_fn_dmi_alloc__ ()
    #14 0x000000000001eaaf in __brk_reservation_fn_dmi_alloc__ ()
    xxx
    xxx (near 500 lines of __brk_reservation_fn_dmi_alloc__())
    xxx
    #521 0x000000000001fc19 in __brk_reservation_fn_dmi_alloc__ ()
    #522 0x0000001b0000000a in __brk_reservation_fn_dmi_alloc__ ()
    #523 0x000000000000e4a6 in __brk_reservation_fn_dmi_alloc__ ()
    #524 0x000000000001fc1f in __brk_reservation_fn_dmi_alloc__ ()

    #525 0x0000000000000000 in ?? ()
    (gdb) cont
    Continuing.

    What is happening, and why is the __brk_reservation_fn_dmi_alloc__() called recursively so many times?

    An initial walkthrough of an android emulator environment

    After “adb shell” and and then “df”:

    ./adb shell
    # df
    /dev: 47084K total, 0K used, 47084K available (block size 4096)
    /sqlite_stmt_journals: 4096K total, 0K used, 4096K available (block size 4096)
    /system: 73600K total, 73600K used, 0K available (block size 4096)
    /data: 65536K total, 18964K used, 46572K available (block size 4096)
    /cache: 65536K total, 1156K used, 64380K available (block size 4096)
    /sdcard: 2087940K total, 8122K used, 2079818K available (block size 2048)

    then “ps” gives:

    # ps
    USER PID PPID VSIZE RSS WCHAN PC NAME
    root 1 0 296 204 c009a694 0000c93c S /init
    root 2 0 0 0 c004dea0 00000000 S kthreadd
    root 3 2 0 0 c003f778 00000000 S ksoftirqd/0
    root 4 2 0 0 c004aaf4 00000000 S events/0
    root 5 2 0 0 c004aaf4 00000000 S khelper
    root 6 2 0 0 c004aaf4 00000000 S suspend
    root 7 2 0 0 c004aaf4 00000000 S kblockd/0
    root 8 2 0 0 c004aaf4 00000000 S cqueue
    root 9 2 0 0 c017bb3c 00000000 S kseriod
    root 10 2 0 0 c004aaf4 00000000 S kmmcd
    root 11 2 0 0 c006ecac 00000000 S pdflush
    root 12 2 0 0 c006ecac 00000000 S pdflush
    root 13 2 0 0 c007349c 00000000 S kswapd0
    root 14 2 0 0 c004aaf4 00000000 S aio/0
    root 21 2 0 0 c017933c 00000000 S mtdblockd
    root 22 2 0 0 c004aaf4 00000000 S hid_compat
    root 23 2 0 0 c004aaf4 00000000 S rpciod/0
    root 24 2 0 0 c018e530 00000000 S mmcqd
    root 25 1 728 240 c01537e8 afe0c7dc S /system/bin/sh
    system 26 1 796 252 c019a854 afe0ca7c S /system/bin/servicemanager
    root 27 1 832 364 c009a694 afe0cba4 S /system/bin/vold
    root 28 1 656 224 c01a65e8 afe0d40c S /system/bin/debuggerd
    radio 29 1 5420 584 ffffffff afe0d0ec S /system/bin/rild
    root 30 1 82016 15712 c009a694 afe0cba4 S zygote
    media 31 1 20948 1660 ffffffff afe0ca7c S /system/bin/mediaserver
    root 32 1 784 292 c02094ac afe0c7dc S /system/bin/installd
    keystore 33 1 1616 384 c01a65e8 afe0d40c S /system/bin/keystore
    root 34 1 728 244 c003d444 afe0d6ac S /system/bin/sh
    root 35 1 824 320 c00b7dd0 afe0d7fc S /system/bin/qemud
    root 37 1 4468 220 ffffffff 0000eca4 S /sbin/adbd
    root 44 34 780 292 c02094ac afe0c7dc S /system/bin/qemu-props
    system 59 30 149980 26620 ffffffff afe0ca7c S system_server
    app_7 99 30 107840 15960 ffffffff afe0da04 S com.android.inputmethod.pinyin
    radio 102 30 119980 16784 ffffffff afe0da04 S com.android.phone
    app_7 104 30 122600 21592 ffffffff afe0da04 S android.process.acore
    app_3 161 30 103656 16336 ffffffff afe0da04 S android.process.media
    app_14 177 30 112536 16460 ffffffff afe0da04 S com.android.mms
    app_17 200 30 102916 15956 ffffffff afe0da04 S com.android.alarmclock
    app_25 213 30 102060 16600 ffffffff afe0da04 S com.example.android.BluetoothChat
    app_23 219 30 106004 16932 ffffffff afe0da04 S com.android.email
    app_12 230 30 101888 14472 ffffffff afe0da04 S com.svox.pico
    root 237 37 728 328 c003d444 afe0d6ac S /system/bin/sh
    root 241 237 868 332 00000000 afe0c7dc R ps
    #

    and “ls /data/dalvik-cache”:

    -rw-r–r– system app_25 24312 2010-12-26 23:04 data
    -rw-r–r– root root 3721808 2010-12-16 22:15 system
    -rw-r–r– root root 1014960 2010-12-16 22:16 system
    -rw-r–r– root root 6198448 2010-12-16 22:16 system
    -rw-r–r– root root 166520 2010-12-16 22:16 system
    -rw-r–r– root root 1095816 2010-12-16 22:16 system
    -rw-r–r– system system 171288 2010-05-07 00:16 system
    -rw-r–r– system system 55024 2010-05-07 00:16 system
    -rw-r–r– system system 21384 2010-05-07 00:16 system
    -rw-r–r– system system 14936 2010-05-07 00:16 system
    -rw-r–r– system system 68496 2010-05-07 00:16 system
    -rw-r–r– system system 6232 2010-05-07 00:16 system
    -rw-r–r– system system 25456 2010-05-07 00:16 system
    -rw-r–r– system system 7744 2010-05-07 00:16 system
    -rw-r–r– system system 4136 2010-05-07 00:16 system
    -rw-r–r– system system 11448 2010-05-07 00:16 system
    -rw-r–r– system system 42352 2010-05-07 00:16 system
    -rw-r–r– system radio 530496 2010-05-07 00:16 system
    -rw-r–r– system radio 84688 2010-05-07 00:16 system
    -rw-r–r– system app_8 2008 2010-05-07 00:16 system
    -rw-r–r– system app_7 152544 2010-05-07 00:16 system
    -rw-r–r– system app_7 129328 2010-05-07 00:16 system
    -rw-r–r– system app_7 195184 2010-05-07 00:16 system
    -rw-r–r– system app_7 373744 2010-05-07 00:16 system
    -rw-r–r– system app_7 16832 2010-05-07 00:16 system
    -rw-r–r– system app_7 14584 2010-05-07 00:16 system
    -rw-r–r– system app_17 67616 2010-05-07 00:16 system
    -rw-r–r– system app_3 13488 2010-05-07 00:16 system
    -rw-r–r– system app_3 74896 2010-05-07 00:16 system
    -rw-r–r– system app_3 87536 2010-05-07 00:16 system
    -rw-r–r– system app_14 524104 2010-05-07 00:16 system
    -rw-r–r– system app_23 979800 2010-05-07 00:16 system
    -rw-r–r– system app_12 6632 2010-05-07 00:16 system
    -rw-r–r– system system 588928 2010-05-07 00:16 system
    -rw-r–r– system app_24 5104 2010-12-16 22:31 data
    -rw-r–r– system app_0 329624 2010-05-07 00:17 system
    -rw-r–r– system app_7 402048 2010-05-07 00:17 system
    -rw-r–r– system system 40264 2010-05-07 00:16 system

    And using dexdump to dump the framework file:

    Processing ‘system’…
    Opened ‘system’, DEX version ‘035’
    Class #0 –
    Class descriptor : ‘Landroid/Manifest$permission;’
    Access flags : 0x30011 (PUBLIC FINAL VERIFIED OPTIMIZED)
    Superclass : ‘Ljava/lang/Object;’
    Interfaces –
    Static fields –
    #0 : (in Landroid/Manifest$permission;)
    name : ‘ACCESS_CHECKIN_PROPERTIES’
    type : ‘Ljava/lang/String;’
    access : 0x0019 (PUBLIC STATIC FINAL)
    #1 : (in Landroid/Manifest$permission;)
    name : ‘ACCESS_COARSE_LOCATION’
    type : ‘Ljava/lang/String;’
    access : 0x0019 (PUBLIC STATIC FINAL)
    #2 : (in Landroid/Manifest$permission;)
    name : ‘ACCESS_FINE_LOCATION’
    type : ‘Ljava/lang/String;’
    access : 0x0019 (PUBLIC STATIC FINAL)
    #3 : (in Landroid/Manifest$permission;)
    name : ‘ACCESS_LOCATION_EXTRA_COMMANDS’
    type : ‘Ljava/lang/String;’

    wow…..140000 lines long.

    looking at all the tools available in /system/bin:

    # ls -l
    lrwxr-xr-x root shell 2010-05-07 00:13 dd -> toolbox
    -rwxr-xr-x root shell 14288 2010-05-07 00:13 dumpstate
    lrwxr-xr-x root shell 2010-05-07 00:13 ln -> toolbox
    -rwxr-xr-x root shell 9816 2010-05-07 00:13 dumpsys
    lrwxr-xr-x root shell 2010-05-07 00:13 setprop -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 ls -> toolbox
    -rwxr-sr-x root net_raw 26652 2010-05-07 00:13 ping
    -rwxr-xr-x root shell 201 2009-09-19 04:08 input
    lrwxr-xr-x root shell 2010-05-07 00:13 watchprops -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 rm -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 kill -> toolbox
    -rwxr-xr-x root shell 191 2009-09-19 04:08 pm
    lrwxr-xr-x root shell 2010-05-07 00:13 log -> toolbox
    -rwxr-xr-x root shell 5432 2010-05-07 00:14 system_server
    lrwxr-xr-x root shell 2010-05-07 00:13 iftop -> toolbox
    -rwxr-xr-x root shell 5420 2010-05-07 00:13 radiooptions
    -rwxr-xr-x root shell 5488 2010-05-07 00:13 dvz
    lrwxr-xr-x root shell 2010-05-07 00:13 rmdir -> toolbox
    -rwxr-xr-x root shell 9696 2010-05-07 00:13 logwrapper
    lrwxr-xr-x root shell 2010-05-07 00:13 lsmod -> toolbox
    -rwxr-xr-x root shell 90924 2010-05-07 00:13 applypatch
    -rwxr-xr-x root shell 22568 2010-05-07 00:13 fsck_msdos
    lrwxr-xr-x root shell 2010-05-07 00:13 vmstat -> toolbox
    -rwxr-xr-x root shell 5512 2010-05-07 00:13 dalvikvm
    -rwxr-xr-x root shell 5552 2010-02-12 08:54 keypress
    -rwxr-xr-x root shell 199 2009-09-19 04:08 bmgr
    -rwxr-xr-x root shell 251780 2010-05-07 00:13 updater
    lrwxr-xr-x root shell 2010-05-07 00:13 insmod -> toolbox
    -rwxr-xr-x root shell 9676 2010-05-07 00:13 flash_image
    -rwxr-sr-x root inet 5640 2010-05-07 00:13 netcfg
    -rwxr-xr-x root shell 5396 2010-05-07 00:13 schedtest
    -rwxr-xr-x root shell 10716 2010-05-07 00:13 showlease
    lrwxr-xr-x root shell 2010-05-07 00:13 setconsole -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 rmmod -> toolbox
    -rwxr-xr-x root shell 9860 2010-05-07 00:13 service
    -rwxr-xr-x root shell 9740 2010-05-07 00:13 dexopt
    lrwxr-xr-x root shell 2010-05-07 00:13 reboot -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 hd -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 getevent -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 newfs_msdos -> toolbox
    -rwxr-xr-x root shell 301436 2010-05-07 00:13 recovery
    lrwxr-xr-x root shell 2010-05-07 00:13 wipe -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 printenv -> toolbox
    -rwxr-xr-x root shell 5504 2010-05-07 00:13 bugreport
    -rwxr-xr-x root shell 14048 2010-05-07 00:13 installd
    -rwxr-xr-x root shell 205 2009-09-19 04:08 monkey
    lrwxr-xr-x root shell 2010-05-07 00:13 stop -> toolbox
    -rwxr-xr-x root shell 13760 2010-05-07 00:13 qemud
    lrwxr-xr-x root shell 2010-05-07 00:13 smd -> toolbox
    -rwxr-xr-x root shell 10024 2010-05-07 00:13 keystore
    -rwxr-xr-x root shell 194 2009-09-19 04:08 ime
    lrwxr-xr-x root shell 2010-05-07 00:13 chown -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 date -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 ps -> toolbox
    -rwxr-xr-x root shell 5452 2010-05-07 00:14 mediaserver
    lrwxr-xr-x root shell 2010-05-07 00:13 umount -> toolbox
    -rwxr-xr-x root shell 23144 2010-05-07 00:13 bootanimation
    lrwxr-xr-x root shell 2010-05-07 00:13 renice -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 df -> toolbox
    -rwxr-xr-x root shell 143584 2010-05-07 00:13 pppd
    lrwxr-xr-x root shell 2010-05-07 00:13 mount -> toolbox
    -rwxr-xr-x root shell 5524 2010-05-07 00:13 gzip
    lrwxr-xr-x root shell 2010-05-07 00:13 dmesg -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 sync -> toolbox
    -rwxr-xr-x root shell 39052 2010-05-07 00:13 vold
    lrwxr-xr-x root shell 2010-05-07 00:13 mv -> toolbox
    -rwxr-xr-x root shell 9760 2010-05-07 00:13 logcat
    -rwxr-xr-x root shell 214600 2010-05-07 00:13 applypatch_static
    -rwxr-xr-x root shell 6520 2010-05-07 00:13 keystore_cli
    -rwxr-xr-x root shell 18240 2010-05-07 00:13 mtpd
    -rwxr-xr-x root shell 5512 2010-05-07 00:13 qemu-props
    -rwxr-xr-x root shell 192 2009-09-19 04:08 svc
    lrwxr-xr-x root shell 2010-05-07 00:13 cat -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 sendevent -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 notify -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 cmp -> toolbox
    -rwxr-xr-x root shell 5676 2010-05-07 00:13 app_process
    -rwxr-xr-x root shell 159044 2010-05-07 00:13 racoon
    -rwxr-xr-x root shell 18056 2010-05-07 00:13 debuggerd
    -rwxr-xr-x root shell 151868 2009-09-19 04:08 gdbserver
    lrwxr-xr-x root shell 2010-05-07 00:13 dumpcrash -> dumpstate
    -rwxr-xr-x root shell 9864 2010-05-07 00:13 servicemanager
    -rwxr-xr-x root shell 9616 2010-03-02 03:43 fbtool
    lrwxr-xr-x root shell 2010-05-07 00:13 netstat -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 ioctl -> toolbox
    -rwxr-xr-x root shell 66724 2010-05-07 00:13 check_prereq
    lrwxr-xr-x root shell 2010-05-07 00:13 ifconfig -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 sleep -> toolbox
    -rwxr-xr-x root shell 5552 2010-05-07 00:13 sdutil
    -rwxr-xr-x root shell 5628 2010-05-07 00:13 rild
    -rwxr-xr-x root shell 44564 2010-05-07 00:13 dhcpcd
    -rwxr-xr-x root shell 73200 2010-05-07 00:13 toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 start -> toolbox
    -rwxr-xr-x root shell 64208 2010-05-07 00:13 linker
    -rwxr-xr-x root shell 5440 2010-05-07 00:13 surfaceflinger
    lrwxr-xr-x root shell 2010-05-07 00:13 chmod -> toolbox
    -rwxr-xr-x root shell 191 2009-09-19 04:08 am
    lrwxr-xr-x root shell 2010-05-07 00:13 getprop -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 schedtop -> toolbox
    -rwxr-xr-x root shell 22112 2010-02-12 08:54 dmagent
    lrwxr-xr-x root shell 2010-05-07 00:13 mkdir -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 route -> toolbox
    -rwxr-xr-x root shell 86972 2010-05-07 00:13 sh
    lrwxr-xr-x root shell 2010-05-07 00:13 id -> toolbox
    lrwxr-xr-x root shell 2010-05-07 00:13 top -> toolbox

    And “dumpstate” gives:

    ------ MEMORY INFO ------
    MemTotal:          94172 kB
    MemFree:            3424 kB
    Buffers:            4116 kB
    Cached:            35900 kB
    SwapCached:            0 kB
    Active:            32312 kB
    Inactive:          47484 kB
    Active(anon):      16532 kB
    Inactive(anon):    25636 kB
    Active(file):      15780 kB
    Inactive(file):    21848 kB
    Unevictable:         264 kB
    Mlocked:               0 kB
    SwapTotal:             0 kB
    SwapFree:              0 kB
    Dirty:                 0 kB
    Writeback:             0 kB
    AnonPages:         40084 kB
    Mapped:            16400 kB
    Slab:               4756 kB
    SReclaimable:       2336 kB
    SUnreclaim:         2420 kB
    PageTables:         3200 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:       47084 kB
    Committed_AS:     916384 kB
    VmallocTotal:     876544 kB
    VmallocUsed:       11376 kB
    VmallocChunk:     858116 kB

    ------ CPU INFO ------

    User 5%, System 13%, IOW 0%, IRQ 0%
    User 7 + Nice 0 + Sys 16 + Idle 99 + IOW 0 + IRQ 0 + SIRQ 0 = 122

    PID   TID CPU% S     VSS     RSS PCY UID      Thread          Proc
    330   330  17% R    932K    412K  fg shell    top             top
    59   154   0% S 149312K  26320K  fg system   er$SensorThread system_server
    59    68   0% S 149312K  26320K  fg system   er.ServerThread system_server
    59   117   0% S 149312K  26320K  fg system   Binder Thread # system_server
    102   102   0% S 115872K  17192K  fg radio    m.android.phone com.android.phone
    4     4   0% S      0K      0K  fg root     events/0
    102   193   0% S 115872K  17192K  fg radio    Binder Thread # com.android.phone
    102   198   0% S 115872K  17192K  fg radio    Binder Thread # com.android.phone
    104   104   0% S 122600K  21800K  fg app_7    d.process.acore android.process.acore

    User 5%, System 13%, IOW 0%, IRQ 0%
    User 7 + Nice 0 + Sys 16 + Idle 99 + IOW 0 + IRQ 0 + SIRQ 0 = 122

    PID   TID CPU% S     VSS     RSS PCY UID      Thread          Proc
    330   330  17% R    932K    412K  fg shell    top             top
    59   154   0% S 149312K  26320K  fg system   er$SensorThread system_server
    59    68   0% S 149312K  26320K  fg system   er.ServerThread system_server
    59   117   0% S 149312K  26320K  fg system   Binder Thread # system_server
    102   102   0% S 115872K  17192K  fg radio    m.android.phone com.android.phone
    4     4   0% S      0K      0K  fg root     events/0
    102   193   0% S 115872K  17192K  fg radio    Binder Thread # com.android.phone
    102   198   0% S 115872K  17192K  fg radio    Binder Thread # com.android.phone
    104   104   0% S 122600K  21800K  fg app_7    d.process.acore android.process.acore
    104   105   0% S 122600K  21800K  fg app_7    HeapWorker      android.process.acore
    104   107   0% S 122600K  21800K  fg app_7    Signal Catcher  android.process.acore
    104   110   0% S 122600K  21800K  fg app_7    JDWP            android.process.acore
    104   115   0% S 122600K  21800K  fg app_7    Binder Thread # android.process.acore
    104   116   0% S 122600K  21800K  fg app_7    Binder Thread # android.process.acore
    104   138   0% S 122600K  21800K  bg app_7    ApplicationsPro android.process.acore
    104   146   0% S 122600K  21800K  bg app_7    ContactAggregat android.process.acore
    104   196   0% S 122600K  21800K  fg app_7    Binder Thread # android.process.acore
    177   177   0% S 112536K  16908K  bg app_14   com.android.mms com.android.mms
    177   178   0% S 112536K  16908K  bg app_14   HeapWorker      com.android.mms
    177   179   0% S 112536K  16908K  bg app_14   Signal Catcher  com.android.mms
    177   181   0% S 112536K  16908K  bg app_14   JDWP            com.android.mms
    177   184   0% S 112536K  16908K  fg app_14   Binder Thread # com.android.mms
    177   185   0% S 112536K  16908K  fg app_14   Binder Thread # com.android.mms
    200   200   0% S 102916K  16332K  bg app_17   roid.alarmclock com.android.alarmclock
    200   201   0% S 102916K  16332K  bg app_17   HeapWorker      com.android.alarmclock
    200   202   0% S 102916K  16332K  bg app_17   Signal Catcher  com.android.alarmclock
    200   203   0% S 102916K  16332K  bg app_17   JDWP            com.android.alarmclock
    200   205   0% S 102916K  16332K  fg app_17   Binder Thread # com.android.alarmclock
    200   206   0% S 102916K  16332K  fg app_17   Binder Thread # com.android.alarmclock
    213   213   0% S 102060K  17064K  bg app_25   d.BluetoothChat com.example.android.BluetoothChat
    ------ PROCRANK ------

    ------ VIRTUAL MEMORY STATS ------
    nr_free_pages 830
    nr_inactive_anon 6409
    nr_active_anon 4145
    nr_inactive_file 5462
    nr_active_file 3945
    nr_unevictable 66
    nr_mlock 0
    nr_anon_pages 10023
    nr_mapped 4100
    nr_file_pages 10004
    nr_dirty 0
    nr_writeback 0
    nr_slab_reclaimable 586
    nr_slab_unreclaimable 607
    nr_page_table_pages 800
    nr_unstable 0
    nr_bounce 0
    nr_vmscan_write 0
    nr_writeback_temp 0
    pgpgin 12591
    pgpgout 8158
    pswpin 0
    pswpout 0
    pgalloc_normal 76646
    pgalloc_movable 0
    pgfree 77495
    pgactivate 15151
    pgdeactivate 20880
    pgfault 137772
    pgmajfault 1169
    pgrefill_normal 20882
    pgrefill_movable 0
    pgsteal_normal 17491
    pgsteal_movable 0
    pgscan_kswapd_normal 10752
    pgscan_kswapd_movable 0
    pgscan_direct_normal 17696
    pgscan_direct_movable 0
    pginodesteal 0
    slabs_scanned 10752
    kswapd_steal 6561
    kswapd_inodesteal 0
    pageoutrun 137
    allocstall 219
    pgrotated 0
    unevictable_pgs_culled 66
    unevictable_pgs_scanned 0
    unevictable_pgs_rescued 0
    unevictable_pgs_mlocked 0
    unevictable_pgs_munlocked 0
    unevictable_pgs_cleared 0
    unevictable_pgs_stranded 0
    unevictable_pgs_mlockfreed 0

    ------ SLAB INFO ------
    slabinfo - version: 2.1
    # name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <nu
    m_slabs> <sharedavail>
    rpc_buffers            8      8   2048    2    1 : tunables   24   12    0 : slabdata      4      4      0
    rpc_tasks              8     24    160   24    1 : tunables  120   60    0 : slabdata      1      1      0
    rpc_inode_cache        0      0    416    9    1 : tunables   54   27    0 : slabdata      0      0      0
    bridge_fdb_cache       0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
    flow_cache             0      0     80   48    1 : tunables  120   60    0 : slabdata      0      0      0
    cfq_io_context         0      0     96   40    1 : tunables  120   60    0 : slabdata      0      0      0
    cfq_queue              0      0     88   44    1 : tunables  120   60    0 : slabdata      0      0      0
    smb_request            0      0    256   15    1 : tunables  120   60    0 : slabdata      0      0      0
    smb_inode_cache        0      0    320   12    1 : tunables   54   27    0 : slabdata      0      0      0
    fat_inode_cache        5     11    352   11    1 : tunables   54   27    0 : slabdata      1      1      0
    fat_cache              2    145     24  145    1 : tunables  120   60    0 : slabdata      1      1      0
    kioctx                 0      0    160   24    1 : tunables  120   60    0 : slabdata      0      0      0
    kiocb                  0      0    160   24    1 : tunables  120   60    0 : slabdata      0      0      0
    inotify_event_cache      0      0     32  113    1 : tunables  120   60    0 : slabdata      0      0      0
    inotify_watch_cache      6     92     40   92    1 : tunables  120   60    0 : slabdata      1      1      0
    dnotify_cache          0      0     24  145    1 : tunables  120   60    0 : slabdata      0      0      0
    fasync_cache           0      0     16  203    1 : tunables  120   60    0 : slabdata      0      0      0
    ashmem_range_cache      0      0     32  113    1 : tunables  120   60    0 : slabdata      0      0      0
    ashmem_area_cache     26     52    288   13    1 : tunables   54   27    0 : slabdata      4      4      0
    shmem_inode_cache    227    250    392   10    1 : tunables   54   27    0 : slabdata     25     25      0
    nsproxy                0      0     24  145    1 : tunables  120   60    0 : slabdata      0      0      0
    posix_timers_cache      0      0    112   35    1 : tunables  120   60    0 : slabdata      0      0      0
    uid_cache             18     59     64   59    1 : tunables  120   60    0 : slabdata      1      1      0
    UNIX                  82     90    384   10    1 : tunables   54   27    0 : slabdata      9      9      0
    ip_mrt_cache           0      0     96   40    1 : tunables  120   60    0 : slabdata      0      0      0
    UDP-Lite               0      0    480    8    1 : tunables   54   27    0 : slabdata      0      0      0
    tcp_bind_bucket        2    203     16  203    1 : tunables  120   60    0 : slabdata      1      1      0
    inet_peer_cache        0      0     64   59    1 : tunables  120   60    0 : slabdata      0      0      0
    secpath_cache          0      0     32  113    1 : tunables  120   60    0 : slabdata      0      0      0
    xfrm_dst_cache         0      0    288   13    1 : tunables   54   27    0 : slabdata      0      0      0
    ip_fib_alias           0      0     16  203    1 : tunables  120   60    0 : slabdata      0      0      0
    ip_fib_hash            9     92     40   92    1 : tunables  120   60    0 : slabdata      1      1      0

    ------ VMALLOC INFO ------
    0xc684c000-0xc684e000    8192 __arm_ioremap_pfn+0x68/0x2fc ioremap
    0xc6850000-0xc6852000    8192 __arm_ioremap_pfn+0x68/0x2fc ioremap
    0xc6854000-0xc6856000    8192 __arm_ioremap_pfn+0x68/0x2fc ioremap
    0xc6880000-0xc68a1000  135168 binder_mmap+0xb4/0x200 ioremap
    0xc6900000-0xc69ff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc6a00000-0xc6aff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc6b00000-0xc6bff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc6d00000-0xc6dff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc6e00000-0xc6eff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc6f00000-0xc6fff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc7400000-0xc74ff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc7500000-0xc75ff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc7700000-0xc77ff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc7800000-0xc78ff000 1044480 binder_mmap+0xb4/0x200 ioremap
    0xc7900000-0xc79ff000 1044480 binder_mmap+0xb4/0x200 ioremap

    ------ ZONEINFO ------
    Node 0, zone   Normal
    pages free     830
    min      312
    low      390
    high     468
    scanned  0 (aa: 0 ia: 0 af: 0 if: 0)
    spanned  24576
    present  24384
    nr_free_pages 830
    nr_inactive_anon 6409
    nr_active_anon 4159
    nr_inactive_file 5462
    nr_active_file 3945
    nr_unevictable 66
    nr_mlock     0
    nr_anon_pages 10039
    nr_mapped    4102
    nr_file_pages 10004
    nr_dirty     0
    nr_writeback 0
    nr_slab_reclaimable 586
    nr_slab_unreclaimable 607
    nr_page_table_pages 800
    nr_unstable  0
    nr_bounce    0
    nr_vmscan_write 0
    nr_writeback_temp 0
    protection: (0, 0)
    pagesets
    cpu: 0
    count: 3
    high:  18
    batch: 3
    all_unreclaimable: 0
    prev_priority:     12
    start_pfn:         0
    inactive_ratio:    1

    …..yet to continue…..

    %d bloggers like this: