Mips Linux: Logging Kernel Panic into mtd partition

Question

We are experiencing kernel panics in the field for our MIPS based embedded devices. How can I log the kernel panic trace in the MTD partition? Do we have to write the trace only into MTD or is it possible to write over the NFS? Can anyone explain how to get useful kernel traces after the panic for the remote boxes.

Answer 1

You can turn on mtdoops module in the kernel and log the kernel panic traces on to mtd partition. I don't think we can write the panic trace over NFS. However, you may want to explore about ramoops.

The following are the steps to configure the kernel to capture the kernel oops on to the mtd flash. Capturing the stack trace after the kernel panic is invaluable to debug the kernel problems particularly the ones happening in the field. During mtdoops module init, the mtd partition is turned into a circular buffer and erased beforehand.

1. The kernel flag, CONFIG_MTD_OOPS, configures the kernel to write the oops stack trace on to the MTD partition. This MTD dev partition information can be hard coded inside mtdoops module or can be specified dynamically. This component can be built as part of kernel or as an separate module. Before building the kernel, you need to make sure that your mtd device has registered a panic_write handler. Remember, normal mtd write handler won't be enough as we have to write to mtd memory after kernel panic. Please run this patch if the mtd device doesn't have its own a panic write handler.

   When built as part of the kernel, CONFIG_MTD_OOPS=y, the mtdoops module needs to be patched with the flash partition information (mtddev).

--- ./drivers/mtd/mtdoops.c.orig 2014-11-17 12:06:59.000000000 +0000
+++ ./drivers/mtd/mtdoops.c 2014-11-17 12:07:36.000000000 +0000
@@ -44,7 +44,7 @@
MODULE_PARM_DESC(record_size,
"record size for MTD OOPS pages in bytes (default 4096)");

-static char mtddev[80];
+static char mtddev[80]="/dev/oops";
module_param_string(mtddev, mtddev, 80, 0400);
MODULE_PARM_DESC(mtddev,
"name or index number of the MTD device to use");

While building it as a module, CONFIG_MTD_OOPS=m, the flash partition information is provided dynamically during module installation (insmod).

insmod mtdoops.ko mtddev=/dev/oops

In addition to enable the MTP OOPS flags, do configure, CONFIG_MAGIC_SYSRQ, to induce the panic and to test this functionality.

2. Now, we need to create a MTD partition (/dev/Oops) to store the panic traces. MTD can be partitioned by modifying the memory layout and partition information defined in the kernel sources at arch///.c. Also, you need to be aware that the partition information passed as part of kernel command line will overrides board.c changes.

{
.name = "loader",
.size = 0x000E0000,
.offset = MTDPART_OFS_APPEND
},
{
.name = "kernel",
.size = 0x002A0000,
.offset = MTDPART_OFS_APPEND
},
{
.name = "oops",
.size = 0x000E0000,
.offset = MTDPART_OFS_APPEND
},
{
.name = "all",
.size = MTDPART_SIZ_FULL,
.offset = 0x00000000
},

3. Build the kernel and mtdoops.ko will get built as part of the root file system. Install the file system and make sure that the partition gets created.

cat /proc/mtd
dev: size erasesize name
mtd0: 000e0000 00020000 "loader"
mtd1: 002a0000 00020000 "kernel"
mtd3: 000e0000 00020000 "Oops"
mtd5: 08000000 00020000 "all"

4. Now, trigger the panic using Magic SysRq keys and observe the kernel panic logs in the Oops partition.

Answer 2

Thanks so much for the detailed steps. Everything seems to be fine, except the device name. When I give the device name as "/dev/oops" mtdoops is not attached to any device in the boot message. After a level of debugging , I found that the device name shall be just "oops" or "Partition number" i.e (if your oops partition is mtd9 just give "9" as partition number). After changing it , it started working. I am able to see the logs using sp-oops-extract.