how to enable and analysis core dump or crash dump in linux ?

Core dump is not enabled by default in embedded systems mainly due to memory limitations. I found this the hard way. I was setting the ulimit to unlimited (ulimit -c unlimited) and also setting the core pattern to directory with read-write permissions(echo /var/tmp/core > /proc/sys/kernel/core_pattern). But core was still not getting dumped when I sent the segmentation fault (kill -11 <pid>) to the the process. Here are some of the reasons why core dump was not generating :

Most of the embedded systems uses busybox. So you need to make sure busybox is configured properly.
You need to have the .init_enable_core for enabling the core dump in the root directory. root directory on most of the embedded systems is read only. So you need to modify your Makefile or post build script to add empty empty file . init_enable_core for the target rootfs.
You need to enable CONFIG_FEATURE_INIT_COREDUMPS for busybox in your .config file.
Problem with custom Signal handlers:
If you have custom signal handlers installed then core will not be dumped for that signal. So you need to make sure that you don’t have custom signal handler installed for the signal for which you are trying to get core dump.
There are lot of information on the web on enabling the core dump. But most of it is for desktop linux distributions and not for embedded linux.


Let’s trigger a crash, and use the dump we obtain to understand the Crash utility. Trigger a crash by trying the following command:
echo c > /proc/sysrq-trigger

This will trigger a panic, and the system boots into the crash kernel, and takes a dump of system memory into the directory /var/crash/<date-time>/. This is named vmcore. Once done, it boots back to the normal kernel.

With the help of the vmcore, vmlinux and system-map files, we will invoke the Crash tool, and view the sample


et’s trigger a crash, and use the dump we obtain to understand the Crash utility. Trigger a crash by trying the following command:
echo c > /proc/sysrq-trigger

This will trigger a panic, and the system boots into the crash kernel, and takes a dump of system memory into the directory /var/crash/<date-time>/. This is named vmcore. Once done, it boots back to the normal kernel.

With the help of the vmcore, vmlinux and system-map files, we will invoke the Crash tool, and view the sample output from it:
[root@DELL-RnD-India linux-2.6]# crash -S System.map vmlinux /var/crash/2011-01-10-12\:23/vmcore

crash 5.1.1
---snip---
crash: overriding /boot/System.map with System.map
GNU gdb (GDB) 7.0
This GDB was configured as "x86_64-unknown-linux-gnu"...
---snip------
  SYSTEM MAP: System.map                
DEBUG KERNEL: vmlinux (2.6.36-rc6-ftrace+)
  DUMPFILE: /var/crash/2011-01-10-12:23/vmcore
        CPUS: 4
        DATE: Mon Jan 10 12:21:33 2011
      UPTIME: 00:06:56
LOAD AVERAGE: 0.80, 0.65, 0.31
       TASKS: 278
    NODENAME: DELL-RnD-India
     RELEASE: 2.6.36-rc6-ftrace+
     VERSION: #2 SMP Wed Sep 29 16:43:59 IST 2010
     MACHINE: x86_64  (2666 Mhz)
      MEMORY: 2 GB
       PANIC: "Oops: 0002 [#1] SMP " (check log for details)
         PID: 7203
     COMMAND: "bash"
        TASK: ffff88007b0d0000  [THREAD_INFO: ffff88007a6ba000]
         CPU: 0
       STATE: TASK_RUNNING (PANIC)

crash>

The above output shows you details about the kernel, the number of processors on the target machine, the command which caused the panic, etc.
Note: Crash can also be invoked on a live system with /dev/mem instead of the vmcore file. For this to work, you need to disable the CONFIG_STRICT_DEVMEM option while compiling the kernel. Stock kernels come with this option enabled, and will not let you use it.
The help command

The most useful command would be the help command, which gives you all the available commands from within the crash tool:
 t             gdb            p              sig            waitq         
btop           help           ps             struct         whatis        
dev            irq            pte            swap           wr            
dis            kmem           ptob           sym            q             
eval           list           ptov           sys           
exit           log            rd             task          
extend         mach           repeat         timer         

crash version: 5.1.1 gdb version: 7.0

To obtain help on any command, run help followed by the command name — for example, help vm.
The bt command

The bt (backtrace) command gives you the stack trace in the current context. And bt -a gives you a stack trace of active tasks on all CPUs. Once the crash tool loads the first context, it sets up information of the panicked process. Here we take a look at the sample output of the command:
crash> bt
PID: 7203 TASK: ffff88007b0d0000  CPU: 0 COMMAND: "bash"
#0 [ffff88007a6bbb00] machine_kexec at ffffffff81027ac7
#1 [ffff88007a6bbb80] crash_kexec at ffffffff810888c9
#2 [ffff88007a6bbc50] oops_end at ffffffff814570c4
#3 [ffff88007a6bbc80] no_context at ffffffff81032ee7
<snipped>
The ps command

This command obtains the status of all the processes, or a selected one. It has an amazing number of options to provide lots of information during dump analysis. Refer to the help section for more details. Here is a sample output:
crash> ps -a 5390
PID: 5390 TASK: ffff8800799ac650  CPU: 2 COMMAND: "httpd"
ARG: /usr/sbin/httpd
ENV: TERM=linux
     PATH=/sbin:/usr/sbin:/bin:/usr/bin
     runlevel=5 \<snipped....>
The set command

You can change the current context using the set command, which takes the PID of the process (which can be obtained from the ps command). It takes various other arguments as well, which can be learnt by running help set. If set is used without arguments, it shows information about the current stack. For example:
crash> set ffff88007d7c0000
    PID: 1
COMMAND: "init"
   TASK: ffff88007d7c0000  [THREAD_INFO: ffff88007d7ba000]

  CPU: 0
  STATE: TASK_INTERRUPTIBLE

Here, the address is the task pointer of the init process.
The files command

This can be used to get all the open files in the current context; it is a context-sensitive command:
crash> set 1
    PID: 1
COMMAND: "init"
   TASK: ffff88007d7c0000  [THREAD_INFO: ffff88007d7ba000]
    CPU: 0
  STATE: TASK_INTERRUPTIBLE
crash> files
PID: 1      TASK: ffff88007d7c0000  CPU: 0   COMMAND: "init"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff880037a58f00 ffff88007cd5be40 ffff88007d090c90 CHR  /dev/null
  1 ffff880037a58f00 ffff88007cd5be40 ffff88007d090c90 CHR  /dev/null
  2 ffff880037a58f00 ffff88007cd5be40 ffff88007d090c90 CHR  /dev/null
  3 ffff880037a58a80 ffff88003747b000 ffff88003750d540 FIFO
  4 ffff880037a586c0 ffff88003747b000 ffff88003750d540 FIFO
  5 ffff880037a58c00 ffff880037493240 ffff88007cdc2ca0 UNKN anon_inode:/inotify
  6 ffff880037a58180 ffff8800374936c0 ffff88007cdc2ca0 UNKN anon_inode:/inotify
  7 ffff880076087a80 ffff8800376d8540 ffff88007ceb87b0 SOCK
  8 ffff880079a25d80 ffff88007a205e40 ffff880079eabc30 SOCK
  9 ffff88007688b6c0 ffff88007a8f0480 ffff88003752e830 SOCK

We have looked into some regularly used commands. For other commands, kindly refer to the help section.