Ping Troubleshooting on Cisco IOS

Lessons Discussion
21

Hello

I have a issue with the ping

For example I tried this ping x…x.x.x and the ping works fine

If I use ping x.x.x.x source any vln

I got loss in the ping packets 3/5 something like this

I saw a lot of drops in the CPU Oliver in the queue ICMP redirect

I tried to put no IP redirect in all the SVI but that did not work

Could help to find an idea to ts this issue?

22

Hello Daniel

It looks like you have packet loss whenever you ping a particular address using an SVI as the source interface. There is no way to troubleshoot this directly with so little information. However, there are a few things you can look at that may be beneficial:

  1. Inspect the interface statistics for the SVI using the show interfaces command, and check for errors, drops, or high utilization.
  2. See if any ICMP rate limiting is enabled on the device, which could result in what you see.
  3. Check for QoS policies that may deprioritize and drop ICMP traffic.
  4. Try testing with different packet sizes
  5. Check the CPU usage. You mentioned something about drops in the queue, ICMP redirect, and CPU. Can you further explain your statement, as it is not entirely clear?

You should also consider your topology and the way it is set up. I have the impression that when you use an SVI as the source, for some reason, there are ICMP redirects taking place. You may need to readjust your configuration somehow, but we can’t know without a detailed view of your topology. Look at this lesson that further discusses ICMP redirects, when they occur and what they’re used for. This may help you to troubleshoot your issue further.

I hope this has been helpful!

Laz

25

Hello,

If host A can’t ping host B, can’t we test host A’s IP addresss if we follow these steps?:

(The first two steps may even just be optional.)

  1. First, on the DHCP server, we add host A’s IP address as an excluded address, so that it doesn’t get leased in the future.
  2. Then, on the switch that host A connects to, we shut the switchport that connects host A to the network.
  3. Next, we configure a loopback IP address on a router that’s close to host A’s default gateway. Let’s call host A’s default gateway RouterA. Which router is the best choice? I think it is the next hop router along the path to host B (host B is the destination that host A can’t reach). Let’s call this next hop router RouterB. On RouterB, we configure as a new loopback IP address (what used to be) host A’s IP address. Let’s call the loopback interface LoopbackA. (We can’t configure host A’s IP address on RouterA, because there would be an error message that says that the newly configured IP address overlaps with the one that is RouterA’s LAN IP address, the one that’s in host A’s subnet.)
  4. Finally, from RouterB, we do an extended ping, and we source the ping from LoopbackA (i.e. using what used to be host A’s IP address).
  5. If it turns out that the ping to host B, which is sourced from the LoopbackA interface of RouterB, is successful, we can conclude that there’s no problem along the path to host B.
  6. If RouterA could ping host A, we know that there must be an ACL on RouterA, or RouterB that’s the problem (if RouterA and RouterB are directly connected via a cable).

So the possible sources of error could be (if RouterA and RouterB are directly connected via a cable):
a) incoming ACL on RouterA (on the interface facing host A),
b) outgoing ACL on RouterA (on the interface facing RouterB),
c) incoming ACL on RouterB (on the interface facing RouterA),
d) outgoing ACL on Router B (on the interface that’s in the direction of host B),
e) a mix of the above.

As we know, the ACL rules on any router don’t apply to packets sourced from that router, so troubleshooting the ACLs by analyzing their statements line-by-line at this point might become inevitable, so the debug command won’t necessarily save us. But if the problem IS with the ACLs on RouterA and RouterB, then this would’ve eventually been our conclusion anyway; and if the issue was somewhere along the path with the specific IP of host A, this way, we know for sure - or not?

Following these steps, couldn’t we replicate exactly what happens when host A pings host B, or any other device? If I’m not mistaken, this would be an exact replication of what happens to the IP packets as they travel through the network (so we could check if for example an ACL is blocking the IP address, or a firewall, or anything else specific to what used to be host A’s IP address).

I don’t have access to real gear and don’t trust emulators or simulators completely, so I’m speculating here (although of course even between real gear, there could be differences due to IOS versions or hardware).

Would this work?

Thanks.
Attila

26

Hello Attila

I understand the logic behind the process you describe. However, there are a couple of things that may be problematic with this:

You wouldn’t be able to do this because the IP address you assign to the looback interface will be within the same subnet as that of the default gateway. A router (or Layer 3 device) will not allow you to assign IP addresses on two different interfaces that are on the same subnet. Alternatively, you could create an SVI on the access switch that host A was connected to with the same IP address that the original host A had, but again, only if another SVI within that subnet has not been previously assigned.

Now having said all of that, I believe that this process is not especially useful.

This is true only to a certain extent. In such a scenario, you would be missing the last link between the gateway and host A, so you are not replicating the communication in full. In my opinion, it would be more practical to use tools like traceroute and ping without making such changes to the network.

Traceroute will give you the router up to which communication is successful. Ping will give you many responses that will tell you the reason behind the failure with a lot of detail. You can see this detail in the types of responses you will receive. You can also see further detail in the ICMP codes and types that are included in the responses.

Remember, a failed ping does not mean that you received no response. You will almost always get a response from intermediary devices telling you in detail why the echo request failed. You can see these ICMP codes and types by doing a debug on the pinging process.

I hope this has been helpful!

Laz

1 Like
27

Hello Laz,

Thank you again for the thorough response.

Sorry about the confusion. After posting my original comment, I’ve edited it a couple of times, and in the newest version (the one I posted before any new comments were published in this thread) I did mention that on hostA’s default gateway, we couldn’t configure hostA’s IP address due to a subnet overlap message, so I’ve recommended configuring it on the router that’s after hostA’s default gateway on the path towards hostA’s destination.

Thank you very much for the links, I’ve read them with great interest. However, it does look like these outputs don’t help point out any ACL issues from hostA towards the final destination, if those ACLs match hostA’s IP address only, or a range that includes hostA’s IP address but not that of hostA’s default gateway.

Let’s say this is the topology:

hostA → RouterA → RouterB → RouterC → hostB

HostA can’t ping hostB. Assume the problem is with an ACL on RouterC. The ping sourced from the LAN interface on RouterA (or from any other interface, on any other router) would succeed if a (mis)configured ACL on RouterC matches packets of hostA’s IP address (or the range is such that it doesn’t include hostA’s default gateway; maybe hostA’s IP address got leased from the higher end of the pool, and the network design always assigns static IPs from the lower end of the pool). If hostA’s IP address were (temporarily) configured as a loopback interface on RouterB, and RouterB would source a ping from that loopback address, couldn’t we see on RouterC’s debug commands that its ACL blocks what is originally hostA’s IP address? And could we see errors on RouterC’s debug if we source the ping from any other IP (that doesn’t fall into the ACL’s range)? I’ve found no indication for the latter in the links.

Another source of error could also be that hostA’s specific IP or the small range that includes it is rerouted by a device (RouterC in this topology). I’m just speculating though, I find these questions very interesting.

I assume a problem like this doesn’t really happen that often, so it would be very rare that the networking team would be forced to replicate a ping like this. But still, couldn’t it work in a scenario like this?

Have a nice day.
Attila

28

Hello Attila

If you take a look at the “Access-list command” section of the following Cisco documentation, you will see that ICMP responds with a “administratively prohibited unreachable” response from ICMP.

The “administratively prohibited unreachable” message that appears in the syslog that comes from an ICMP response, is due to the use of an ACL. Take a look to see if you can modify your approach to incorporate this and detect blockage from ACLs.

In addition, take a look at this list of error indications for ping that is listed in WIkipedia. There you can see other are a few error codes, like A and Z, and X that indicate the “administratively” prohibited idea. I’m not sure under what operating system you would see these, but it doesn’t seem that Cisco has these. Even so, looking at the debugs allows you to see them and use them in your troubleshooting scenarios.

Concerning your other scenario, I am having trouble understanding the troubleshooting process. In any case, you suggest creating a loopback interface on router B with the same IP address as host A. Regardless of what that may achieve, this is problematic because you now have two destinations using the same IP address, and you would have to adjust routing so that any ICMP responses to host A (or the loopback masquerading as host A) need to go back to the loopback and not the real host A. This just adds too many additional changes to the topology that can be the cause of future failures…

Hopefully, the administratively prohibited unreachable message should resolve your issue so you won’t have to go to such lengths for troubleshooting.

I hope this has been helpful!

Laz

1 Like