Kernel debugging – kmsg

Source: http://bootloader.wikidot.com/linux:android:kmsg

In developing kernel code, it is quite useful to get kernel messages as soon as possible. If the kernel code is loaded early in the boot process, one might not even get any kernel message before the device hangs… Fortunately, these are common challenges developers encounter, and there are already some good solutions for them.

Standard “dmesg”

To invoke the “dmesg” from the control PC, one can simply run

# adb shell dmesg
Continuously “tail” the kernel message?

Linux users may use tail -f to get kernel messages. However, because “syslogd” is possibly not included in Android, there is no such logs, and one may find that the directory “/var” is not even created.

# tail -f /var/log/messages

Fortunately, there is a kernel file /proc/kmsg which does exactly the same thing. One can just run the following command to continuously dump the kernel messages.
# adb shell cat /proc/kmsg
Redirecting kernel messages

If the phone doesn’t even boot up to a stable state where the ADB commands can be used, one might be interested in redirecting the kernel messages to some places where they can be seen. This is to leverage the “console=” command for the kernel.

For different devices, there may be different ports where the messages can be redirected to. If the serial port on the device is enabled, one can use use the serial port. On MSM chips, there is a UART port which may be used. For this, one can use “console=ttyMSM2,115200n8” and the availability of the port can be checked in “/dev” if one wants to use it.

Currently, there is a possibility to use the USB serial ports too. One may want to check if they work on the device. The USB serial port is a software serial port emulated through the USB interface. It often has the names “ttyUSB*” or “ttyHSUSB*”. On Android devices, if one sees “ttyUSB*”, those may be ports provided by the USB gadget drivers (/kernel/drivers/usb/gadget), which could be used as the console (add console=ttyUSB0,9600n8). If one sees “ttyHSUSB*”, they may be a bit different. These may be provided by MSM platform USB drivers, and some might not support the “console” yet. For curious readers, in order to be used as a console, a device driver has to register itself as a console provider by calling register_console in kernel/printk.c, and it has to provide some callbacks for printk to write kernel messages. So the device driver developer would have to explicitly support this for the device driver to be used as a console.

Other methods?

There are also other ways to get kernel messages during boot, one can check out the netconsole, Linux-aware Trace32 (JTAG), or simply dump kernel messages on the LCD — “console=tty0”. The later might not always be the most convenient to read but it provides some quick way to see what has happened without touching the drivers or hardware configuration.

Back to “tail -f” a little bit. Before I realized that “cat /proc/kmsg” would do just what I wanted, the way I did was using a small app to continuously invoke the syscall “klogctl” to read new kernel messages. If one is interested, the source code is here: klog.c. It can be compiled by adding into Android toolbox makefile (system/core/toolbox/Android.mk)


You are reading this post on Joel G Mathew’s tech blog. Joel's personal blog is the Eyrie, hosted here.