MTU Troubleshooting on Cisco IOS

#86

Hello Laz,
Layer 2 switch port and layer 3 router port check the MTU the same way. Correct?

Azm

#87

Hello AZM

If you are referring to the interface MTU then yes.

Laz

1 Like
#88

Wouldnt this all be fixed with Path MTU discovery?

#89

Hello Carlos

Yes, Path MTU discovery would solve these problems, but it cannot always be implemented. PMTUD depends on ICMP packets. Many network security devices block all ICMP messages for security reasons including the errors that are necessary for the proper operation of PMTUD. This can result in connections that complete the TCP three-way handshake correctly, but then hang when data is transferred. This is called a black hole connection.

In these cases, MTU issues must be troubleshot and solved “manually”.

I hope this has been helpful!

Laz

#90

Hi Laz/Rene ,
I have doubt here and i searching for answers , as follows

1.Does Hosts will fragment the packets before sending to gateway routers ?
2.How does the reassembly of fragmented packets takes place and who does this exactly (Router/Host)?

Regards,
Sameer

#91

Hello Sameer.

The quick answer is yes. Fragmentation will occur during encapsulation. If a PC is preparing an email to send, it will take the transport segment and encapsulate it into an IP packet. An IP packet can theoretically be up to 65535 bytes long. So during encapsulation from the Network to the Data Link layer on the PC, if an IP packet is larger than the allowable 1500 bytes (or whatever the interface MTU has been set to), then it will be fragmented into several parts and sent in multiple frames. Now the IPv4 header has some fields called Flags and Fragment Offset.
image
These two fields are used in order to identify fragmented packets and put them back together correctly. The Flags contain three bits:

  • bit 0: Reserved; must be zero
  • bit 1: Don’t Fragment (DF)
  • bit 2: More Fragments (MF)

The first is not currently used. The second is used to indicate during encapsulation that this packet should not be fragmented. The third is the one used for reassembly, which is 1 if there are additional fragments of this IP packet coming in additional frames or if this is the last one. The Fragment offset is used to specify the offset of a particular fragment relative to the beginning of the original unfragmented packet. This way, the reassembly can take place correctly.

Reassembly is performed during the deencapsulation processes when the multiple frames reach the destination. The frames are deencapsulated and the IP headers are examined. Because the MF bit will be set to 1, the receiver keeps the current IP packet in memory and receives the next one and the next until the MF=0. Once all the fragments are collected, the Fragment offset is used to reassemble the packet correctly before deencapsulation is continued upwards towards the Transport and Application Layers.

I hope this has been helpful!

Laz

#92

Hi,

I have set “ip tcp adjust-mss 1430” on a tunnel interface but for some reason the negotiation keeps setting 1460 *mss. resulting in drops has soon as mtu 1500 is reached . MAPI sets the df flag.
Why is it not intercepting the negotiation? what can i do?

thanks

#93

Hello Steven

I suggest you use wireshark to take a look at the value of the MSS filed in both the SYN and the SYN ACK portions of the handshake. Take a look at this thread from the Cisco Support Community, check out the value of the MSS field in the SYN ACK and let us know your progress.

I hope this has been helpful!

Laz

#94

Hi Lapagides,

Thanks for your help.
It is in wireshark I see the SYN packet options having an incorrect MSS value. when the stream reaches 1500 MTU I see icmp fragmentation needed arriving. It is as if the adjust-mss entry does not have any effect.
(EDIT:: removed wrong statement about telnet )

br Steven

#95

What are the prereq. for the adjust-mss to work?

#96

Hi Steven,

Nothing much, you add this command on the interface that connects to your hosts and that’s it. It only works for TCP traffic.

Rene

#97

Does it work in both directions? also traffic “not hitting” the ip of the interface or SVI?
Can it be put on a tunnel interface? Still not clear why it is not working…

#98

Hello Steven

I suggest you do a bit of experimentation by reducing the adjust-mss value to even lower values to see if the SYN MSS value is affected. If not, then there is some issue with the way that the adjust-mss value is being implemented on the tunnel interface. Now there are several options for dealing with IP fragmentation and MTU sizes. These can be found in this Cisco Documentation:


See if any of these solutions fit better with your scenario.

In the meantime, can you send a more detailed description of your setup? Maybe a simple diagram showing the devices and the configuration of the tunnels as well? This way we can help you out in more detail.

I hope this has been helpful!

Laz

#99

Very helpful, both the article and following discussion. I even think I maybe finally understand it :slight_smile:.

Saying that the topic is somewhat confusing. I believe in part due to arbitrary use of terminology in some cases and in part due to actual implementation of the feature (thank you, CISCO almighty for keeping us thoroughly entertained by allowing your developers to implement features that are not systematic, logical or well structured. It saves us from a boredom).

However, if I understood correctly some of the things Lazaros said they seem opposing to what Rene said (and actually proved). Im referring to statement that if packet size exceeds the MTU size on interface its getting fragmented. That seems is not correct - it should be dropped. The whole point of what Rene showed and stated was that packet is only getting fragmented if its size exceeds the IP MTU set size. Quite different.

But I wonder still if I understood the whole thing correctly. So here is the summary of the whole confusing topic as I understood it.

Is it really correct? -

  1. There are multiple values of the MTU pertinent to each individual network layer (or corresponding PDU).

  2. MTU for TCP segments is actually called TCP Max Segment Size. It can be set on the router or switch interface to negotiate with the sending/receiving host the size of the Segment. Its size defines only actual TCP payload and does not account for 20B TCP header. By itself it does not affect traffic except changing size of sent data (which is different from lower layers MTUs).

  3. IP MTU is defined as size of the TCP segment plus TCP header (20B) plus the header for IP packet itself (20B). By default on CISCO routers/switches it is 1500B. It is set on router interface and defines a max size of the packet that can pass through the interfaces unharassed. If packet exceed the IP MTU value it will be fragmented (not clear though whats the rule for fragmenting - how parts are defined).

  4. MTU is defined as the size of L2 frame on interface. It includes the size of the IP MTU and… and nothing else. Since the Frame header now is not counted the MTU by default is equal IP MTU. If IP MTU is set to be smaller then there is no problem, frame just would be ‘underutilized’ since no IP packet would be able to fill up the allowed size for the frame. If IP MTU is larger though then packet can come which is larger than MTU in which case the packet is dropped, frame is not formed. Frames do not support fragmentation as packets do.

  5. It is possible to set MTU globally by defining System MTU - it will provide default MTU for all interfaces.

  6. To avoid packet drops at the very least the IP MTU should be equal or less than frames MTU. Reducing IP MTU will lead to higher probability of fragmentation. It maybe beneficial in certain cases, when there is additional header inserted on top of IP header - in this case IP MTU may allow packet size which together with additional header will exceed frame MTU so the packet gets dropped. Similarly, if IP header is replaced with some other, larger than 20B header it may cause the same problem. In that case reducing IP MTU (and/or increasing frame MTU) may prevent packets drops, though at the price of fragmentation.

  7. Where possible it might be better to leave IP MTU alone (though still OK to increase frame MTU) and instead reduce the TCP Masx Segment Size - that will not fragment the packets, so its better from routers utilization point of view, and will help avoiding packets drops due to exceeding MTU.

I guess that covers the most fundamental part. Would be happy if that is all correct. It still leaves some other questions, like alternate and jumbo MTUs for additional discussion.

Also, to add spice to the topic, Path MTU discovery feature is by default enabled on most contemporary equipment, based on ICMP protocol extensions and consists of detecting the disabled fragmentation for packets exceeding IP MTU (by checking if DF bits in packets set to ‘1’) and requesting the sending equipment to resend the dropped due to too large size packets with smaller TCP payload.

#100

Hello Vadim

Whether a packet will be dropped or fragmented depends on the contents of the Do Not Fragment (DF) bit field in the IP header. If it is set to 1 the packet is dropped. If it is set to 0 the packet is fragmented into two frames.

Your explanation seems to show that you indeed have a solid understanding of the topic as I cannot see any issues with what you have written.

Jumbo MTU is the system MTU that is used for GigabitEthernet ports. This is often set to higher values in order to accommodate specific applications that run over the network.

Alternate MTU, by itself, does nothing until you specifically assign it to a specific port. It defines a third interface MTU that fits between the system MTU and jumbo MTU. The system MTU is the interface MTU for the 10/100 ports. The jumbo MTU is the interface MTU for the GE ports. The alternate MTU can be an interface MTU for any 10/100 or GE port. You apply the Alternate MTU with a global command that maps the alternate MTU to the desired interfaces. A change to the alternate MTU or any of its applied interfaces, like the system and jumbo MTU commands, requires a reload.

Yes, this is another feature that can be very useful, but is often not usable because ICMP packets, which are necessary for its functionality, are often blocked for security reasons from many intermediary devices.

I hope this has been helpful!

Laz

#101

Got a few questions-

  1. You mention that it is no problem when the frame reaches 1514 bytes, but realistically how high can this go before it starts causing an issue on Ethernet?

  2. Why did it cause a problem communicating with the web server when you reduced the MTU to 1400? Should it not have just fragmented it? Is the DF bit used in HTTP? I mean it would slow things down a bit to fragment, but not stop them working altogether?

  3. What happens if one side has an MTU of 1400, and the other 1500? Does the side set to 1400 have an issue receiving packets over this size?

  4. What is the impact of jumbo frames on all of this?

  5. TCP MSS clamping doesn’t seem like a great solution, as surely all of the UDP traffic is going to have issues?

Thanks!

#102

Hello Chris

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

For the purposes of the specific lesson, IP packet size of 1500 was deemed problematic for the MTU of 1400 just to get the point across. The possibility of fragmentation and the settings of the DF bit were not analyzed here in order to maintain the simplicity of the example. Notice in the wireshark capture that DF bit was indeed set, so the communication failed. Now whether or not fragmentation does or should occur when web browsing mainly depends on the configuration of the network in question. HTTP/HTTPS do not have any direct influence nor do they require any specific setting on the DF bit. This is determined by the specific networks that the packets are traversing.

I assume you’re talking about the MTU parameter set on the Ethernet interface. If this is the case, you can connect one switchport with an MTU of 1400 to a switchport with an MTU of 1500. Any frames that attempt to enter the 1400 MTU sized interface that are larger will be dropped. MTU size does not have to be the same on both ends in order for a link to function, unlike some other parameters like speed, duplex, trunk configurations and others.

Jumbo frames are those frames that are larger than the standard 1500 bytes. When a switch is configured to accept jumbo frames, the size of those frames are also specified. Any frame up to that size will be accommodated. However, keep in mind that a jumbo frame is a layer 2 concept. When this frame is then called upon to encapsulate an IP packet which in turn encapsulates a TCP segment, the packet and segment sizes must also be smaller than the maximum jumbo frame MTU.

UDP traffic does not conform to the TCP MSS configuration. UDP does not have a concept of MSS. UDP datagrams are fragmented based on whatever fragmentation the IP protocol can achieve. However, for TCP, MSS can be a solution as it will reduce fragmentation and is initially agreed upon at the three way handshake.

I hope this has been helpful!

Laz

#103

Thanks Laz

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

I guess the question I’m asking here is - how high can you increase the MTU before it’s going to cause an issue? I know the default is 1500, but will 1510 be ok? 1520? 1530? etc

#104

Hello Chris

Problems will only arise if the configured maximum allowed MTU is smaller than the size of the frame that is being sent. The problems come in two forms: Either the DF bit is set and the frame is dropped or the DF bit is not set and the transmission suffers from excessive fragmentation, and thus ultimately slower data transfer rates.

Now the maximum allowed MTU on an interface of a Cisco device can be modified from the default of 1500 to values exceeding 9000 on some platforms. If your allowed MTU is set to 9000 for example, you can easily send frames of up to that size without any problems.

I hope this has been helpful!

Laz

1 Like
#105

Hi Laz

Are you sure there is not also another problem the other way, i.e. the MTU is set too high and then packets are unable to be transmitted if the medium can’t handle it?

Why isn’t the MTU 9000 by default?