Is it possible to set the time for Debug to Undebug?

Routing & Switching
1

HI Sir,

could you please help me out in debug command. is it possible to set particular time to Undebug all or particular debug command .

thanks from Rajamohan Reddy.

2

Hello Rajamohan

There is no specific command that can cause a cisco device to disable all debugging on the system. There is however a feature of IOS that allows you to schedule commands. This is not specific to debugging commands, but any available IOS command. The feature is called the Command Scheduler and uses the kron command.

The Command Scheduler allows admins to schedule fully-qualified EXEC mode CLI commands to run once, at specified intervals, at specified calendar dates and times, or upon system startup. You can schedule an undebug all command whenever you like so you can essentially do what you describe in your question - remove all debugging after a specified amount of time.

More information about the Kron command can be found at this Cisco documentation.

I hope this has been helpful!

Laz

1 Like
3

A long time ago I created an example for Kronā€¦for those that are interested :smile:

1 Like
4

Hello,

I thought about a(n admittedly jerry-rigged) solution against the debug command flooding the router.

Assume I can only telnet or SSH into the device, and want to issue the debug command. First, Iā€™d open a second telnet or SSH session, check which vty line Iā€™m in, and configure that vty line with the command no terminal monitor. Iā€™d then issue the debug command, and if it makes any inputs from my first session impossible, Iā€™d issue the undebug all command in the second window (which wouldnā€™t be flooded by the debug commandā€™s outputs, because earlier I issued the no terminal monitor command on its vty line).

The Kron option is more elegant, but Iā€™m curious: would this work? :smiley: I guess there are some devices that are so old, they donā€™t support the Kron option (Cisco IOS XE Release 2.1 was released around May 4, 2008 if Iā€™m not mistaken: https://www.cisco.com/c/en/us/td/docs/ios/ios_xe/2/xe_2_1_newfeatlist.html , and the other IOS versions are also from around that time period, so I guess devices before 2008 donā€™t have the Kron option).

If the CPU gets completely overwhelmed, maybe both the elegant Kron solution and my makeshift attempt would both fail.

Of course, if the IOS supports it, it would make more sense to use the Kron command, because at least that way, even if you forget to issue the undebug all command, the system will do it for you. But Iā€™d use my makeshift solution along with the Kron command, just in case I make a mistake when issuing the Kron command - provided the makeshift solution actually works. Iā€™d also issue the Kron command to reenable the terminal monitor option on the second vty session an hour or two later, in case I forget to do so manually.

Have a nice day.
Attila

5

Hello Attila

The problem of overwhelming a device with a lot of debugging processes does not arise from the flooding of the CLI with debug/Syslog messages. Even if you have many such messages, you can still type the undebug all command and it will work, even if you canā€™t see the actual text being printed on your screen.

The issue, as you correctly stated later on, has to do with CPU and memory resource usage. In such a case, both the kron and the secondary CLI window you open will be negatively impacted. However, the kron solution has a better chance of succeeding and is definitely preferable. Why? Because the open CLI window requires a response from the device over the network, and also requires that multiple events succeed over time, such as the connection, the processing of each individual packet that is sent with each letter that you type etc. Whereas the kron solution is configured on the device itself and doesnā€™t depend upon network connectivity or response to many packets over time. It is a single command issued all at once internally. This has a better chance of success than the CLI option. Does that make sense?

Using both is even better! In a similar manner, I sometimes use the reload at command to cause a remote device to reload automatically within five or ten minutes while making changes to the configuration. If for any reason I lose contact with the device, it will reload with the last saved startup-config, and Iā€™ll have access to it again. If I make the changes successfully, I cancel the reload and all is well. This logic can also work in the event that any debugs make the device unresponsive.

I hope this has been helpful!

Laz

1 Like
6

Hello Laz,

Thank you very much for the continued correspondence. Sorry about the long post. Iā€™d like to learn as much as I can, I hope itā€™s not an issue if I write so much. I like to understand something as thoroughly as possible.

Iā€™m very interested in learning as many best practices as I can so that I work as efficiently as possible. Iā€™ve written down a couple of things and would read with great interest your comments about them if you have any to make. (Iā€™ve included your excellent advice about timing a restart for completeness.) I like to learn as much as possible from experienced colleagues and Iā€™m curious to hear any feedback they may have.

Some of these are still theoretical considerations, I donā€™t have access to real gear yet, although Iā€™m on my way to do so. (It goes without saying that I would only experiment with my own equipment, never with production devices.)

In Putty, among other things Iā€™ve changed these settings. You can save these settings as defaults and they will get inherited by every new session (although I recommend first saving the actual default settings under a unique name, just in case):

  1. Automatic logging: creates log files in a given folder, with hostname and date and time as the filename, for all CLI activities:
    Session ā†’ Logging ā†’ All session output + Log file name, which I recommend to be C:\Users\EXACTPATH\&H-&Y&M&D-&T.log
  2. Greater line size: Window ā†’ Lines of scrollback; around 2 billion, so that the terminal window never runs out of space. You can just press down a number for a couple of seconds.
  3. Changed the default copy and paste steps of the terminal window: by default, a right-click pastes the contents of the clipboard, and a left click copies it from the terminal window:
    Window ā†’ Selection ā†’ Action of mouse buttons: Windows

I find the default annoying: if my finger slips on the right mouse button when Iā€™ve got a lot of stuff on the clipboard, I have to sit there and wait for it to complete, and it may even change something that I donā€™t want to. This way you can copy and paste with the Ctrl+Shift+C and Ctrl+Shift+V combo, or right mouse button ā†’ Copy/Paste. It also doesnā€™t automatically paste the thing you select, which I like because sometimes, when I click back on the terminal window, I accidentally keep the left mouse button pressed for a millisecond, and that overwrites the clipboard.

  1. At the end of the week, I merge all log files into a single file and send it to myself via mail. That way, the contents are searchable. You can merge them (in Windows) the following way:
    Open the folder that they are in ā†’ type in cmd into the search bar of File Explorer (so that it opens the Command Prompt in the log filesā€™ folder) ā†’ press Enter ā†’ paste the following into the Command Prompt ā†’ copy *.log NAME_OF_MERGED_FILES.txt ā†’ press Enter ā†’ compare the size of the original files with that of the merged file to make sure they were all copied.

Daily best practices:

  • When I log in to a device, I type an exclamation mark (!) and the ticket number of the case that made it necessary to log in to the device, and I press Enter. I then show the running and startup config. This way, in my log files I can later justify why I changed anything (if that happens), and Iā€™ll have a copy of the startup and running configs (note: watch out for the special characters that sneak in when you are prompted to press More).
  • If I have to change something (like adding a new routing entry, etc.), before I change it, I check if that something hasnā€™t already been changed.

Other steps for configuration changes:

  1. If I have to configure a new IP address: first, I ping it. If thereā€™s a response, then itā€™s already configured somewhere. (Of course, there could be a case where it already has been configured somewhere, but that device isnā€™t reachable at the moment when Iā€™m pinging it.)
  2. I issue a show users to check if someone is also logged in and ask them what they are doing (otherwise, you might configure two contradictory settings: e.g. you configure a new interface description, and he gives that same interface another description, etc.).
  3. If I am still sure I have to change the given settings, I follow these steps:

a) I issue the Show archive config differences command to compare the running config with the startup config. If they are the same, then good. If not, I make sure that if the device reloads, the startup config allows connectivity.
b) I issue the command reload in 5 so that the device reloads in 5 minutes. Then, I make the change. If all goes well, I issue a reload cancel command to cancel the reload.

  1. When I want to apply a new config:

a) I open a new Notepad++ tab, and in the first line I type in reload in 5.
b) In the last line I type in reload cancel.
c) Between the first and last lines, I paste the new config.
d) Between a couple of lines of the new config, I paste the reload in 5 command.
e) When Iā€™m pasting the new config into the device, Iā€™m not pasting the entire new config in, but I paste it in sections (so a couple of lines at a time). Each section ends with the reload in 5 command. This way, the reload counter restarts. So if any new configuration changes make me lose my connection, I can get back in 5 minutes (assuming Iā€™ve made sure I can in step a): thatā€™s why itā€™s important to make sure that the startup config allows for connectivity again).
f) When the last line of the new config is good, the last thing to do is to issue the cancel command (which is already there in the Notepad++ document).

  1. If the configuration change is about shutting down an interface, I must make sure that I can reach the device after I shut down that interface.
    The path in this example is:
    R1ā€™s G0/0 ā†’ other devices ā†’ R2ā€™s G0/0
    Assume I telnet to R2 from R1, and I have to shut R2ā€™s G0/0. I must make sure that I can reach R2 through R1ā€™s G0/1 (so not just via G0/0 which is the default). To do that,

a) I first issue a trace from R1 to R2 (to check the path via R1ā€™s G0/0 to R2).
b) Then I configure a static route on R1 to R2 through R1ā€™s G0/1 (meaning the neighbor of R1 which is reachable from R1ā€™s G0/1).
c) Then I issue another trace to R2.

If the two paths are different, and I can reach R2, then I can safely shut the interface on R2.

If I canā€™t reach R2 via R1ā€™s G0/1, I must check these settings on at least one of the routers:

show ip prefix-list 
show route-map
show run interface

interface that the other device will use to telnet or SSH into the target device ā† check for an ACL, and if thereā€™s an ACL, check if any ACE blocks traffic

  1. Before I issue a debug command,
  • I first check which VTY line Iā€™m in (show users, and the one with the * next to it is mine).
  • Then I check if the logging monitor global config mode command is configured.
  • Then I check if on my VTY line the terminal monitor is also configured.
  • Then, I issue an undebug all command with the kron command, 2 minutes into the future. That way, even if I forget to issue the undebug command, or I canā€™t because the debug command overburdens the CPU, the device can hopefully do it for me.
  • Finally, I issue the most specific debug command so that I get as little output as possible, while making sure that it contains the information Iā€™m looking for.

If you have the time, Iā€™d love to read your thoughts on the above. Iā€™m also very happy to read the best practices that others share.

Have a nice week.
Attila

7

Hello Attila

This is a very thorough and useful set of best practices to use in implementing changes and administrating a network.

In general, I have no objection to any of the points you make, I think theyā€™re a result of a lot of thought as well as experience. The only comments I have are general comments that may help in further enhancing this procedure.

First of all, the process you describe is highly personalized, meaning, you put this into practice for yourself. You are also saving the log files locally on your computer for future reference. Depending upon the environment in which you work, it may be beneficial to centralize some of these processes and even to automate them. You can save any and all interactions with the devices using a centralized RADIUS or TACACS solution, where the Accounting part of the AAA is enabled. This will track all changes made by particular users. You can also automate the addition of the ā€œ!ā€ and the ticket number so that you donā€™t have to do it yourself, in case you forget or you input a typo.

The approach is excellent, and if you further centralize the whole process, you are well on your way to developing an operations and administration policy document for the ICT department of a large enterprise, where all of these processes can be defined, and subsequently enforced.

I hope these thoughts have been helpful!

Laz

1 Like