Configure Cisco routers to save core dumps for troubleshooting crashes

David Davis tells you what you need to know about saving core dump information from your Cisco router in the event of a router crash.

———————————————————————————————————–

No one wants it to happen, but sooner or later, you will experience a router crash. When you do, you will want to be prepared to save that critical router memory information in order to find out why the router crashed and prevent future crashes. To do this, you need to configure your router to store core dump information in the event of a crash. Let’s find out what a core dump is, how it can help you, and how you can configure your router to store these important files for analysis.

What is a core dump?

A core dump is simply a full copy of your router’s memory image. When your router has a system crash or unrecoverable error, it cannot continue, so it dumps all its memory and writes the memory contents to a server before it reloads. It is very important that you retain a copy of the dump to help identify possible causes for the crash. While the core dump isn’t readable by you, it is readable and useful to Cisco’s Technical Assistance Center (TAC). When you experience a router crash, you open a TAC case with personnel who are trained to read this output. Keep in mind that the core dump needs to be saved by the router at the time of the crash. A core dump cannot be taken after the crash has happened and the router has rebooted.

Important troubleshooting commands when a router crashes

Before I show you how to configure core dumps, let’s look at a few other commands that are important to troubleshooting router crashes:

• show version: This command shows such things as the configuration of the router hardware, IOS software image version, memory, and interfaces that are available. Perhaps only a piece of hardware failed. This information could also help you see what version of code is running on your router and how much memory and flash is available.
• show stacks: This is another extremely helpful command that is used to monitor the stack usage of processes and interrupt routines. This command will show information such as a bus error or a software-forced crash.
• show context: This command stores information like the reason for the system reboot and stack trace information.

Besides a core dump, the Cisco TAC will likely ask you for the output of the show tech-support command. That command lists every configuration, statistic, and log on your router.

To get additional information on these commands, please see the Cisco documentation “Troubleshooting Router Crashes.”

How do you configure the Cisco IOS to save core dumps?

The Cisco IOS can store or transfer a core dump file using four different methods. They are:

• File Transfer Protocol (FTP)
• Remote Copy Protocol (RCP)
• Trivial Transfer Protocol (TFTP)
• Flash Disk (stored on the router, not transferred over the network)

The recommended method is through File Transfer Protocol (FTP) so that is what I will offer the configuration for. By the way, whether you use FTP, RCP, or any of the other ways mentioned above, be sure each protocol is working correctly before creating the dump. In other words, test a transfer of some kind using that method. For example, you could test FTP by copying the router’s configuration to the same FTP server that the core config will go to:

Router# copy running-config ftp

To manually create a dump without a reload, type the following command in privileged exec mode:

Router# write core

This command is useful in case the router is just malfunctioning but not crashing. Please realize that it dumps the whole memory, not just what is in use, so be sure and have enough memory to accept it. Also - I do not recommend doing this on a production router while it’s in use.

Here are the commands to use for a core dump using FTP:

• ip ftp username username: Configures the username for FTP connections.
• ip ftp password password: Configures the password for the FTP connection.
• exception protocol ftp: Configures the protocol used for core dump FTP.
• exception region-size 65536: Configures the region size.
• exception dump ip-address: Configures the IP address of the server to which the router sends the core dump in case of a crash.

Cisco recommends that you connect the router directly to the FTP server, with no intermediate hops.

The debug sanity command is also a good command to use for debugging memory corruption, especially I/O memory. You will probably use this while working with your Cisco Tech representative. Let’s look at a snippet output of a show version command after a core dump. Please notice the error at the bottom of the output.

Router# show version 

Cisco Internetwork Operating System Software

IOS (tm) RSP Software (RSP-PV-M), Version 12.0(10.6)ST, EARLY DEPLOYMENT

MAINTENANCE INTERIM SOFTWARE

Copyright (c) 1986-2000 by cisco Systems, Inc.

Compiled Fri 23-Jun-00 16:02 by richv

Image text-base: 0x60010908, data-base: 0x60D96000

ROM: System Bootstrap, Version 12.0(19990806:174725), DEVELOPMENT SOFTWARE

BOOTFLASH: RSP Software (RSP-BOOT-M), Version 12.0(9)S, EARLY DEPLOYMENT

RELEASE SOFTWARE (fc1)

Router uptime is 20 hours, 56 minutes

System returned to ROM by error - a Software forced crash, PC 0x60287EE8 

System image file is "slot0:rsp-pv-mz.120-10.6.ST"

Another file that would be useful to you in the event of a crash is the crashinfo file. It’s stored in bootflash or flash memory. For more details on crashinfo, see the Cisco documentation “Retrieving Information from the Crashinfo File.”

Conclusion

It is important that core dump configuration is put on each router so that you can capture the core dump if your router ever crashes. Once the router has been rebooted, if not saved, the crashed core dump cannot be retrieved. By doing this, hopefully you can use that core dump file, with the help of the Cisco TAC, to solve your router crashing issue the first time that it happens.

To learn more about creating core dumps and troubleshooting router crashes, please see the Cisco articles “Creating Core Dumps” and “Troubleshooting Router Crashes.”