Archive for October, 2010

My gdm kept dying, why?

On my dying laptop:

root 5219 0.0 0.0 3076 1616 tty1 Ss 13:48 0:00 /bin/login —
root 5304 0.0 0.0 4424 1920 tty1 S+ 13:49 0:00 \_ -bash
root 5451 0.0 0.0 15080 1692 ? Ss 13:49 0:00 /usr/sbin/gdm
root 5598 0.0 0.0 15080 1712 ? T 13:50 0:00 \_ /usr/sbin/gdm
root 5645 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [kill] <defunct>

But this was preceded by another attempt to restart (which failed as well):

root 5451 0.0 0.0 15080 1692 ? Ss 13:49 0:00 /usr/sbin/gdm
root 5598 0.0 0.0 15080 1712 ? T 13:50 0:00 \_ /usr/sbin/gdm
root 5645 0.0 0.0 0 0 ? Z 13:50 0:00 \_ [kill] <defunct>
root 5779 0.0 0.0 2956 388 ? S 13:50 0:00 xinit /etc/gdm/failsafeXinit /etc/X11/xorg.conf.failsafe with-gdm — /usr/bin/X :0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt9 -br -once -config /etc/X11/xorg.conf.failsafe -logfile /var/log/Xorg.failsafe.log
root 5781 3.7 0.0 0 0 ? Z<s 13:50 0:00 \_ [Xorg] <defunct>

The above behavior is well described here:

https://bugs.launchpad.net/ubuntu/+source/xorg/+bug/310126

and here:

http://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg1322735.html

Questions:

How does gdm worked? How does it used X?

A good writeup is here:

http://www.jirka.org/gdm-documentation/x84.html

And its security internal is here:

http://projects.gnome.org/gdm/docs/2.14/security.html
GDM uses PAM for username/authentication, though if your machine does not support PAM you can build GDM to work with shadow passwords and crypt.

PAM stands for Pluggable Authentication Module, and is used by most programs that request username/password authentication on your computer. It allows the user to configure different authentication behavior for different programs.

Some GDM features (like turning on automatic login) may require that you update your PAM configuration. PAM has different, but similar, interfaces on different operating systems, so check your pam.d or pam.conf man page for details about how to configure it. Make sure to read the PAM documentation (e.g. pam.d/pam.conf man page) and be comfortable with the security implications of any changes you intend to make to your configuration.

If there is no entry for GDM in your system’s PAM configuration file, then features like tomatic login may not work. Not having an entry will causes GDM to use default behavior, conservative settings are recommended and probably shipped with your distribution.

XDMCP and how it worked?

http://www.faqs.org/docs/Linux-HOWTO/XDMCP-HOWTO.html

Looking up /var/log/gdm/:0.log:

5598 5451
xinit /etc/gdm/failsafeXinit /etc/X11/xorg.conf.failsafe with-gdm — /usr/bin/X :0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt9 -br -once -config /etc/X11/xorg.conf.failsafe -logfile /var/log/Xorg.failsafe.log

X: warning; process set to priority -1 instead of requested priority 0

X.Org X Server 1.6.0
Release Date: 2009-2-25
X Protocol Version 11, Revision 0
Build Operating System: Linux 2.6.24-23-server i686 Ubuntu
Current Operating System: Linux tteikhua-laptop 2.6.34-rc5 #3 SMP Sun Jun 20 23:26:33 SGT 2010 i686
Build Date: 09 April 2009 02:10:02AM
xorg-server 2:1.6.0-0ubuntu14 (buildd)
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (–) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(++) Log file: “/var/log/Xorg.failsafe.log”, Time: Thu Oct 14 13:50:03 2010
(++) Using config file: “/etc/X11/xorg.conf.failsafe”
/usr/bin/X: symbol lookup error: /usr/lib/xorg/modules/extensions//libdri.so: undefined symbol: atiddxAbiDixLookupPrivate
5598 5451
giving up.^M
xinit: Connection refused (errno 111): unable to connect to X server^M
xinit: No such process (errno 3): Server error.

Found solution:

https://bugs.launchpad.net/ubuntu/+source/fglrx-installer/+bug/283836/comments/15

https://bugs.launchpad.net/ubuntu/+source/xserver-xorg-video-ati/+bug/285603

To summarize:

apt-get remove –purge fglrx*
apt-get remove –purge xserver-xorg-video-ati
apt-get remove –purge xserver-xorg-video-radeon
apt-get update
/etc/init.d/gdm restart

And now the error is GONE!!!

Doing a “ps -axf” on the same system I got:

23295 ? Ss 0:00 /usr/sbin/gdm
23299 ? S 0:00 \_ /usr/sbin/gdm
23311 tty7 Ss+ 0:02 \_ /usr/X11R6/bin/X :0 -br -audit 0 -auth /var/lib/gdm/:0.Xauth -nolisten tcp vt7
23352 ? Ssl 0:00 \_ x-session-manager
23476 ? Ss 0:00 \_ /usr/bin/ssh-agent /usr/bin/dbus-launch –exit-with-session /usr/bin/pulse-session /usr/bin/seahorse-agent –execute x
23516 ? Ss 0:00 \_ /usr/bin/seahorse-agent –execute x-session-manager
23539 ? S 0:00 \_ metacity
23540 ? S 0:00 \_ gnome-panel
23541 ? S 0:00 \_ nautilus
23547 ? S 0:00 \_ bluetooth-applet
23550 ? Sl 0:00 \_ /usr/lib/evolution/2.26/evolution-alarm-notify
23553 ? S 0:00 \_ nm-applet –sm-disable
23554 ? S 0:00 \_ update-notifier –startup-delay=60
23557 ? S 0:00 \_ python /usr/share/system-config-printer/applet.py

This is a Intel GM45 chipset, so not sure whether xserver-xorg-video-intel is already inside or not. Anyway now the graphic rendering good – even OpenGL stuff!!

How is gdm/X implementated to talk to the kernel for graphic screen rendering/management?

Doing a “lsof | grep Xorg”:

Lots of the following pattern:

Xorg 23311 root DEL REG 0,4 57566 /drm mm object
Xorg 23311 root DEL REG 0,4 57565 /drm mm object
Xorg 23311 root DEL REG 0,4 57564 /drm mm object
Xorg 23311 root DEL REG 0,4 57563 /drm mm object
Xorg 23311 root DEL REG 0,4 57562 /drm mm object
Xorg 23311 root DEL REG 0,4 57561 /drm mm object
Xorg 23311 root DEL REG 0,4 57560 /drm mm object
Xorg 23311 root DEL REG 0,4 57559 /drm mm object
Xorg 23311 root DEL REG 0,4 57549 /drm mm object
Xorg 23311 root DEL REG 0,4 57548 /drm mm object
Xorg 23311 root DEL REG 0,4 57544 /drm mm object
Xorg 23311 root mem CHR 1,1 4735 /dev/mem
Xorg 23311 root mem REG 8,1 34084 9572 /usr/lib/libdrm_intel.so.1.0.0
Xorg 23311 root mem REG 8,1 554804 16407 /usr/lib/xorg/modules/drivers/intel_drv.so
Xorg 23311 root mem REG 8,1 34164 9570 /usr/lib/libdrm.so.2.4.0
Xorg 23311 root DEL REG 0,4 58058 /drm mm object
Xorg 23311 root DEL REG 0,4 58057 /drm mm object
Xorg 23311 root DEL REG 0,4 58056 /drm mm object
Xorg 23311 root DEL REG 0,4 58055 /drm mm object
Xorg 23311 root DEL REG 0,4 58054 /drm mm object
Xorg 23311 root DEL REG 0,4 58053 /drm mm object

And then:

Xorg 23311 root 1u unix 0xd98ae200 57449 socket
Xorg 23311 root 2u REG 8,1 734 1762 /var/log/gdm/:0.log
Xorg 23311 root 3r unix 0xd98af000 57450 /tmp/.X11-unix/X0
Xorg 23311 root 4w REG 8,1 29902 22833 /usr/lib/xorg/protocol.txt
Xorg 23311 root 5w CHR 4,7 4744 /dev/tty7
Xorg 23311 root 6w REG 0,3 0 4026531909 /proc/mtrr
Xorg 23311 root 7r unix 0xd98af200 57529 socket
Xorg 23311 root 8w REG 0,0 256 9608 /sys/devices/pci0000:00/0000:00:02.0/config
Xorg 23311 root 9u REG 0,3 0 4026531909 /proc/mtrr
Xorg 23311 root 10r CHR 10,175 9029 /dev/agpgart
Xorg 23311 root 11u CHR 226,0 53132 /dev/dri/card0
Xorg 23311 root 12u CHR 226,0 53132 /dev/dri/card0
Xorg 23311 root 13u unix 0xd984f800 58151 socket
Xorg 23311 root 14u CHR 13,72 53188 /dev/input/event8
Xorg 23311 root 15u CHR 13,71 9183 /dev/input/event7
Xorg 23311 root 16u CHR 13,67 5283 /dev/input/event3
Xorg 23311 root 17u CHR 13,70 9118 /dev/input/event6
Xorg 23311 root 18u CHR 13,68 6610 /dev/input/event4

I think the core part will be in /proc/mtrr, /dev/dri/* and /dev/agpgart, as well as /dev/input/event*, /dev/mem – all these are opened by the Xorg process.

Next do a “apt-get source xserver-xorg-core” and analysing the files that open these ports above:

For /proc/mtrr:

./hw/kdrive/src/kmap.c:
mtrr = open (“/proc/mtrr”, 2);
mtrr = open (“/proc/mtrr”, 2);

./hw/xfree86/os-support/linux/lnx_video.c:
/* The file desc for /proc/mtrr. Once opened, left opened, and the mtrr
#define MTRR_FD_UNOPENED (-1) /* We have yet to open /proc/mtrr */
#define MTRR_FD_PROBLEM (-2) /* We tried to open /proc/mtrr, but had
/* Open /proc/mtrr. FALSE on failure. Will always fail on Linux 2.0,
/* Only report absence of /proc/mtrr once. */
mtrr_fd = open(“/proc/mtrr”, O_WRONLY);

For /dev/mem:

./hw/kdrive/src/kmap.c:
fd = open (“/dev/mem”, O_RDWR|O_SYNC);
fd = open (“/dev/mem”, O_RDWR);
FatalError (“KdMapDevice: failed to open /dev/mem (%s)\n”,

./hw/xfree86/common/xf86Configure.c:
ptr->dev_mem_base = DevToConfig[screennum].GDev.MemBase;

./hw/xfree86/common/xf86Config.c:
devicep->MemBase = conf_device->dev_mem_base;

./hw/xfree86/parser/Device.c:
ptr->dev_mem_base = val.num;
if (ptr->dev_mem_base)
fprintf (cf, “\tMemBase 0x%lx\n”, ptr->dev_mem_base);

./hw/xfree86/os-support/bsd/alpha_video.c:
#define DEV_MEM “/dev/mem”
* Check if /dev/mem can be mmap’d. If it can’t print a warning when

./hw/xfree86/os-support/bsd/arm_video.c:
#define DEV_MEM “/dev/mem”
* Check if /dev/mem can be mmap’d. If it can’t print a warning when

./hw/xfree86/os-support/bsd/ppc_video.c:
#define DEV_MEM “/dev/mem”

./hw/xfree86/os-support/bsd/i386_video.c:
#define DEV_MEM “/dev/mem”
* Check if /dev/mem can be mmap’d. If it can’t print a warning when
/* Failed to open /dev/mem, try the aperture driver */
pci_system_init_dev_mem(devMemFd);

./hw/xfree86/os-support/linux/int10/linux.c:
#define DEV_MEM “/dev/mem”

./hw/xfree86/os-support/linux/lnx_video.c:
fd = open(“/dev/mem”, O_RDWR);

./hw/xfree86/os-support/shared/bios_devmem.c:
* Read BIOS via /dev/mem.
# define DEV_MEM “/dev/mem”

./hw/xfree86/parser/xf86Parser.h:
unsigned long dev_mem_base;

./hw/xfree86/os-support/bsd/memrange.h:
* Memory range attribute operations, peformed on /dev/mem

./hw/xfree86/os-support/xf86_OSlib.h:
# define DEV_MEM “/dev/mem”
#define DEV_MEM “/dev/mem”

For /dev/agpgart:

./hw/kdrive/linux/agp.c:
#define AGP_DEVICE “/dev/agpgart”
* Open /dev/agpgart. Keep it open until server exit.

./hw/xfree86/os-support/linux/lnx_agp.c:
#define AGP_DEVICE “/dev/agpgart”
* Close /dev/agpgart. This frees all associated memory allocated during
* Open /dev/agpgart. Keep it open until xf86GARTCloseScreen is called.

./hw/xfree86/os-support/solaris/sun_agp.c:
#define AGP_DEVICE “/dev/agpgart”
* Close /dev/agpgart. This frees all associated memory allocated during
* Open /dev/agpgart. Keep it open until xf86GARTCloseScreen is called.

 

./hw/xfree86/os-support/solaris/agpgart.h:
#define AGP_DEVICE “/dev/agpgart”

Hm…..different ports are for different purposes as documented inline above. Doing a lsmod:

drm_kms_helper 28763 1 i915
drm 165380 5 i915,drm_kms_helper
agpgart 31507 3 drm,intel_agp

So there is no more framebuffer mechanism. It is now managed via DRM:

http://dri.freedesktop.org/wiki/DRM

And the userspace component is DRI:

http://dri.freedesktop.org/wiki/

Ok…..enough for now.

Advertisements

How does “sar” worked at the kernel level?

As noted from below:

http://www.princeton.edu/~unix/Solaris/troubleshoot/sar.html

http://www.princeton.edu/~unix/Solaris/troubleshoot/ram.html

“sar” is used to collect system statistics, eg, runqueue of the CPU in order to determine CPU loading.

How does it worked? Started via “/etc/init.d/sysstat start”, and configured via /etc/sysstat/sysstat, enabled via /etc/default/sysstat (the line ENABLED=”true” must be entered). The cron script “/etc/cron.d/sysstat” will start it (by default) every 10 minutes.

Doing a “apt-get source sysstat” from a Ubuntu box:

And looking into the header files:

grep proc.*stat *.h

common.h:#define STAT “/proc/stat”
common.h:#define DISKSTATS “/proc/diskstats”
common.h:#define NFSMOUNTSTATS “/proc/self/mountstats”
iostat.h: * or the number of “disk_io:” entries in /proc/stat (2.4 kernels),
iostat.h: * for the first four devices in /proc/stat).
mpstat.h: * mpstat: per-processor statistics
pidstat.h: * pidstat: Display per-process statistics.
pidstat.h:#define PID_STAT “/proc/%u/stat”
pidstat.h:#define PID_STATUS “/proc/%u/status”
pidstat.h:#define TASK_STAT “/proc/%u/task/%u/stat”
pidstat.h:#define TASK_STATUS “/proc/%u/task/%u/status”
rd_stats.h:#define FDENTRY_STATE “/proc/sys/fs/dentry-state”
rd_stats.h:#define FINODE_STATE “/proc/sys/fs/inode-state”
rd_stats.h:#define NET_SOCKSTAT “/proc/net/sockstat”
rd_stats.h:#define NET_SOCKSTAT6 “/proc/net/sockstat6”
rd_stats.h:#define VMSTAT “/proc/vmstat”

We can see that it uses the procfs for statistics collection from userspace.

Looking up kernel source, fs/proc/stat.c (just one statistics out of the many possible):

}
sum += arch_irq_stat();

seq_printf(p, “cpu %llu %llu %llu %llu %llu %llu %llu %llu %llu ”
“%llu\n”,
(unsigned long long)cputime64_to_clock_t(user),
(unsigned long long)cputime64_to_clock_t(nice),
(unsigned long long)cputime64_to_clock_t(system),
(unsigned long long)cputime64_to_clock_t(idle),
(unsigned long long)cputime64_to_clock_t(iowait),
(unsigned long long)cputime64_to_clock_t(irq),
(unsigned long long)cputime64_to_clock_t(softirq),
(unsigned long long)cputime64_to_clock_t(steal),
(unsigned long long)cputime64_to_clock_t(guest),
(unsigned long long)cputime64_to_clock_t(guest_nice));
for_each_online_cpu(i) {

/* Copy values here to work around gcc-2.95.3, gcc-2.96 */
user = kstat_cpu(i).cpustat.user;
nice = kstat_cpu(i).cpustat.nice;
system = kstat_cpu(i).cpustat.system;
idle = kstat_cpu(i).cpustat.idle;
idle = cputime64_add(idle, arch_idle_time(i));
iowait = kstat_cpu(i).cpustat.iowait;
irq = kstat_cpu(i).cpustat.irq;
softirq = kstat_cpu(i).cpustat.softirq;
steal = kstat_cpu(i).cpustat.steal;
guest = kstat_cpu(i).cpustat.guest;
guest_nice = kstat_cpu(i).cpustat.guest_nice;
seq_printf(p,
“cpu%d %llu %llu %llu %llu %llu %llu %llu %llu %llu ”
“%llu\n”,
i,
(unsigned long long)cputime64_to_clock_t(user),
(unsigned long long)cputime64_to_clock_t(nice),
(unsigned long long)cputime64_to_clock_t(system),
(unsigned long long)cputime64_to_clock_t(idle),
(unsigned long long)cputime64_to_clock_t(iowait),
(unsigned long long)cputime64_to_clock_t(irq),
(unsigned long long)cputime64_to_clock_t(softirq),
(unsigned long long)cputime64_to_clock_t(steal),
(unsigned long long)cputime64_to_clock_t(guest),
(unsigned long long)cputime64_to_clock_t(guest_nice));
}

Here CPU specific statistics are calculated and updated to the procfs virtual filesystem, where userspace application can periodically collect and analyzed for differences.

Working on my SheevaPlug

First, do a git clone:

git clone git://git.marvell.com/orion.git

Then:

wget http://sheeva.with-linux.com/sheeva/2.6.33.2/sheeva-2.6.33.2.config
cp sheeva-2.6.33.2.config .config
make ARCH=arm oldconfig
make ARCH=arm CROSS_COMPILE=arm-none-eabi-
make ARCH=arm CROSS_COMPILE=arm-none-eabi- uImage
make ARCH=arm CROSS_COMPILE=arm-none-eabi- modules
make ARCH=arm CROSS_COMPILE=arm-none-eabi- INSTALL_MOD_PATH=.. modules_install

where “arm-none-eabi-” is entered here because CodeSourcery’s crosschain tools are used (which are installed in /opt/CodeSourcery/Sourcery_G++_Lite/bin directory):

ls -al arm-none-eabi-*
-rwxr-xr-x 1 root root 547548 2010-04-18 01:05 arm-none-eabi-addr2line
-rwxrwxr-x 2 root root 572276 2010-04-18 01:05 arm-none-eabi-ar
-rwxrwxr-x 2 root root 990396 2010-04-18 01:05 arm-none-eabi-as
-rwxr-xr-x 2 root root 208112 2010-04-18 01:05 arm-none-eabi-c++
-rwxr-xr-x 1 root root 547324 2010-04-18 01:05 arm-none-eabi-c++filt
-rwxr-xr-x 1 root root 207056 2010-04-18 01:05 arm-none-eabi-cpp
-rwxr-xr-x 2 root root 208112 2010-04-18 01:05 arm-none-eabi-g++
-rwxr-xr-x 2 root root 205488 2010-04-18 01:05 arm-none-eabi-gcc
-rwxr-xr-x 2 root root 205488 2010-04-18 01:05 arm-none-eabi-gcc-4.4.1
-rwxr-xr-x 1 root root 26760 2010-04-18 01:05 arm-none-eabi-gcov
-rwxr-xr-x 1 root root 3241976 2010-04-18 01:05 arm-none-eabi-gdb
-rwxr-xr-x 1 root root 3241976 2010-04-18 01:05 arm-none-eabi-gdbtui
-rwxr-xr-x 1 root root 607136 2010-04-18 01:05 arm-none-eabi-gprof
-rwxrwxr-x 2 root root 818784 2010-04-18 01:05 arm-none-eabi-ld
-rwxrwxr-x 2 root root 555612 2010-04-18 01:05 arm-none-eabi-nm
-rwxrwxr-x 2 root root 703284 2010-04-18 01:05 arm-none-eabi-objcopy
-rwxrwxr-x 2 root root 836092 2010-04-18 01:05 arm-none-eabi-objdump
-rwxrwxr-x 2 root root 572308 2010-04-18 01:05 arm-none-eabi-ranlib
-rwxr-xr-x 1 root root 308040 2010-04-18 01:05 arm-none-eabi-readelf
-rwxr-xr-x 1 root root 763280 2010-04-18 01:05 arm-none-eabi-run
-rwxr-xr-x 1 root root 547584 2010-04-18 01:05 arm-none-eabi-size
-rwxr-xr-x 1 root root 346704 2010-04-18 01:05 arm-none-eabi-sprite
-rwxr-xr-x 1 root root 547516 2010-04-18 01:05 arm-none-eabi-strings
-rwxrwxr-x 2 root root 703284 2010-04-18 01:05 arm-none-eabi-strip

Partial results for uImage:

make ARCH=arm CROSS_COMPILE=arm-none-eabi- uImage
CHK include/linux/version.h
CHK include/generated/utsrelease.h
make[1]: `include/generated/mach-types.h' is up to date.
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
Kernel: arch/arm/boot/Image is ready
SHIPPED arch/arm/boot/compressed/lib1funcs.S
AS arch/arm/boot/compressed/lib1funcs.o
LD arch/arm/boot/compressed/vmlinux
OBJCOPY arch/arm/boot/zImage
Kernel: arch/arm/boot/zImage is ready
UIMAGE arch/arm/boot/uImage
Image Name: Linux-2.6.35-rc1-09018-g67a3e12
Created: Wed Oct 13 22:59:54 2010
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 2714736 Bytes = 2651.11 kB = 2.59 MB
Load Address: 0x00008000
Entry Point: 0x00008000
Image arch/arm/boot/uImage is ready

And then the modules:

INSTALL drivers/media/IR/ir-nec-decoder.ko
INSTALL drivers/media/IR/ir-rc5-decoder.ko
INSTALL drivers/media/IR/ir-rc6-decoder.ko
INSTALL drivers/media/IR/ir-sony-decoder.ko
INSTALL drivers/media/IR/keymaps/rc-adstech-dvb-t-pci.ko
INSTALL drivers/media/IR/keymaps/rc-apac-viewcomp.ko
INSTALL drivers/media/IR/keymaps/rc-asus-pc39.ko
xxxxxxxxxxxxx
INSTALL sound/usb/snd-usb-audio.ko
INSTALL sound/usb/snd-usbmidi-lib.ko
MKDIR ../lib/firmware/emi26
INSTALL ../lib/firmware/emi26/loader.fw
INSTALL ../lib/firmware/emi26/firmware.fw
INSTALL ../lib/firmware/emi26/bitstream.fw
MKDIR ../lib/firmware/emi62
INSTALL ../lib/firmware/emi62/loader.fw
INSTALL ../lib/firmware/emi62/bitstream.fw
INSTALL ../lib/firmware/emi62/spdif.fw
INSTALL ../lib/firmware/emi62/midi.fw
MKDIR ../lib/firmware/kaweth
INSTALL ../lib/firmware/kaweth/new_code.bin
INSTALL ../lib/firmware/kaweth/trigger_code.bin
INSTALL ../lib/firmware/kaweth/new_code_fix.bin
INSTALL ../lib/firmware/kaweth/trigger_code_fix.bin
INSTALL ../lib/firmware/ti_3410.fw
INSTALL ../lib/firmware/ti_5052.fw
INSTALL ../lib/firmware/mts_cdma.fw
INSTALL ../lib/firmware/mts_gsm.fw
INSTALL ../lib/firmware/mts_edge.fw
MKDIR ../lib/firmware/edgeport
xxxxxxxx
MKDIR ../lib/firmware/keyspan_pda
INSTALL ../lib/firmware/keyspan_pda/keyspan_pda.fw
INSTALL ../lib/firmware/keyspan_pda/xircom_pgs.fw
DEPMOD 2.6.35-rc1-09018-g67a3e12

Relevant links:

http://www.newit.co.uk/forum/index.php?PHPSESSID=d72a5fe7b3a5e7a3f0bb780b8a45b89f&/topic,285.0.html

And

http://computingplugs.com/index.php/Building_a_custom_kernel

Next is the u-boot…….to be continued.

%d bloggers like this: