MTU Troubleshooting on Cisco IOS

Hi Laz,

Thanks for replying.

I intentionally didn’t add the df-bit because I was investigating from a layer 2 perspective.

But I had tested that part before, and the df-bit did cause it to fail because the ip mtu was also set to 1400 on R2’s interface. As a consequence the response did not make it when I pinged above 1400 because it wasn’t allowed to fragment. So that part works as expected.

I just set up the following topology:

R1-------SW1--------R2

SW1 being L2 only, so IP fragmentation doesn’t come into it. I then set SW1’s interface connecting to R1 to an mtu of 1400:

SW1:

SW1#sh int Gi0/0 | i MTU
MTU 1400 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

R1:

R1#sh ip int Gi0/0 | i MTU
MTU is 1500 bytes
R1#sh int Gi0/0 | i MTU
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

R2:

R2#sh ip int Gi0/0 | i MTU
MTU is 1500 bytes
R2#sh int Gi0/0 | i MTU
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

When I ping from R1 to R2:

R1#ping 155.1.12.2 size 1410
Type escape sequence to abort.
Sending 5, 1410-byte ICMP Echos to 155.1.12.2, timeout is 2 seconds:
!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/3/4 ms
R1#ping 155.1.12.2 size 1411
Type escape sequence to abort.
Sending 5, 1411-byte ICMP Echos to 155.1.12.2, timeout is 2 seconds:

Success rate is 0 percent (0/5)

The switch gives the same result the as routers, it allows up to 1424.

I’m wondering if this is because it is allowing for a header size that could potentially be 24 bytes, rather than the usual 14, hence the extra 10.

Thanks,

Sam

Hello Samir

Thanks for the details. This is an interesting one. I recreated the topology you suggest and I have been able to replicate your results. But it’s not immediately perceivable why we get this behavior. So let’s think this through.

When you ping using a size of 1400, what you capture using Wireshark is a frame size on the wire of 1414. So the actual Layer 3 PDU, including IP and ICMP headers is 1400, and the header of the Ethernet frame is 14 for a total of 1414 egressing the host.

Those 1414 bytes of frame reach the interface of the switch, which is set up to have an interface MTU of 1400. But it allows a frame of size up to 1424 on the wire to enter. So the Interface MTU must omit some of the headers of a total size of 24 bytes. Let’s examine what headers are added to an ICMP ping.

Looking at a Wireshark capture of a ping of size 1410, I see the following:

  • ICMP payload: 1382 bytes
  • ICMP header 8 bytes
  • IP header 20 bytes
  • Ethernet header 14 bytes

All of that equals 1424 bytes. So when H1 pings a size of 1410 bytes, it’s really 1382+8+20 = 1410. Plus the 14 bytes of the Ethernet makes it 1424 on the wire.

Now when the switch’s interface with an MTU of 1400 receives this, somehow, it does not consider 10 extra bytes as part of that MTU, and lets them through… To be honest, I don’t know. I’ll have to think about this some more, and share it with Rene to see if we can shed some light on it. Thanks for your patience.

Laz

Hi Laz,

Thanks for investigating. I assumed it was either:

  1. A bug in CML (which you will have disproved if you are using physical H/W)

or

  1. As the MTU configured on the interface is for the frame payload and not its entire length, maybe the switch adds the maximum header size possible and performs the size check on the entire frame (not just the payload) when it is received. This means it allows for headers up to 24 bytes in size, not 14, and I’m taking up the extra 10 bytes with more payload. Just a theory.

I only stumbled across this because I was trying to break OSPF adjacency formation by dropping the MTU to a size that allowed hello packets to pass but nothing else but it wasn’t behaving as I expected.

Thanks.

Sam

Hello Samir

After further investigation, there are a couple of interesting things involved with this scenario. The lab you created, and the one I created as well, were both on CML, so we created the exact same scenario. The first interesting thing is that on physical hardware, it’s not possible to reduce the L2 MTU per interface to a value of less than 1500 bytes. Indeed on most IOS platforms, you can only apply the system MTU globally, and with a minimum of 1500. Even on some physical platforms where you can configure MTU per interface, it’s still a minimum of 1500.

Secondly, Rene tested this out on real hardware (WS-C3850-48T), and with an MTU size of 1500, was able to ping up to a size of 1504 but not 1505:

Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
R1#ping 192.168.12.2 size 1504 repeat 1
Type escape sequence to abort.
Sending 1, 1504-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
!

Success rate is 100 percent (1/1), round-trip min/avg/max = 1/1/1 ms
R1#ping 192.168.12.2 size 1505 repeat 1
Type escape sequence to abort.
Sending 1, 1505-byte ICMP Echos to 192.168.12.2, timeout is 2 seconds:
.

On the wire, the frame length is 1518 bytes. This is expected behavior because it allows for an additional 14 bytes for the Ethernet header and another 4 bytes for a possible VLAN tag. He tried it again with an MTU of 1600, and a ping of 1604 went through but not 1605.

So it seems to be an issue with the emulator. I’m curious to find out what would happen in a similar topology in GNS3 or EVE-NG.

I hope this has been helpful!

Laz

1 Like

Hi Laz,

Actually, I did this using EVE-NG with the ios’s exported from CML.

Thanks.

Sam

Hello Samir

Hmm, that’s interesting. It seems that the emulated topologies and environments share this strange calculation of the MTU. In any case, thanks for sharing that, it’s good to keep in mind.

Laz