Linux kernel OOPS dissection notes

What’s this

Every now and then I find myself looking at an Oops or kernel panic, reminding myself how to approach this. So this is where I write down the jots as I go. This isn’t very organized.

Disassembly

• First thing first: Disassemble the relevant parts:

  $ objdump -DS driver.ko > ~/Desktop/driver.asm

• Doing this on the .o or .ko file gives exactly the same result. Like diff-exactly.
• Or if the region of interest belongs to the kernel itself, this can be done (in the kernel tree after a matching compilation):

  $ objdump -DS kernel/locking/spinlock.o > ~/Desktop/spinlock.asm

• Or, directly on the entire kernel image (at the root if the kernel tree of a compiled kernel). This doesn’t just saves looking up where the relevant function is defined (which object file), but the labels used in the function will be correct, even when using -d instead of -DS.

  $ objdump -d vmlinux

  Then search for the function with a colon at the end, so it matches the beginning of the function, and not references to it. E.g.
  <_raw_spin_lock_irqsave>:

• The -DS flag adds inline source in the disassembly when possible. If it fails, go for plain -d instead.
• With the -d flag, usage of labels (in particular calls to functions) outside the disassembled module will appear to be to the address following the command, because the address after the opcode is zero. The disassembly is done on binary that is before linking.

Stack trace

• The offending command is where the RIP part points at. It’s given in the same hex format as the stack trace.
• The stack trace contains the offset points in a function (i.e. after a label) to after the call to the function. It’s label+hex/hex.
• The hex notion is relative to a label as seen in the disassembly (the title row before each segment). The first offset number is the offset relative to the label, and the second is the length of the function (i.e. the offset to the next label).
• <IRQ> and </IRQ> markups show the part that ran as an interrupt (not surprisingly).
• In x86 assembly notation, the first argument is source, the second is destination.

<IRQ>
dump_stack+0x46/0x59
__report_bad_irq+0x40/0xa9
note_interrupt+0x1c9/0x217
handle_irq_event_percpu+0x4c/0x6a
handle_irq_event+0x2e/0x4c
handle_fasteoi_irq+0x9b/0xff
handle_irq+0x19/0x1c
do_IRQ+0x61/0x106
common_interrupt+0xf/0xf
</IRQ>

To be continued, of course…