MTU Troubleshooting on Cisco IOS

Hi Steven,

Nothing much, you add this command on the interface that connects to your hosts and that’s it. It only works for TCP traffic.

Rene

Does it work in both directions? also traffic “not hitting” the ip of the interface or SVI?
Can it be put on a tunnel interface? Still not clear why it is not working…

Hello Steven

I suggest you do a bit of experimentation by reducing the adjust-mss value to even lower values to see if the SYN MSS value is affected. If not, then there is some issue with the way that the adjust-mss value is being implemented on the tunnel interface. Now there are several options for dealing with IP fragmentation and MTU sizes. These can be found in this Cisco Documentation:


See if any of these solutions fit better with your scenario.

In the meantime, can you send a more detailed description of your setup? Maybe a simple diagram showing the devices and the configuration of the tunnels as well? This way we can help you out in more detail.

I hope this has been helpful!

Laz

Very helpful, both the article and following discussion. I even think I maybe finally understand it :slight_smile:.

Saying that the topic is somewhat confusing. I believe in part due to arbitrary use of terminology in some cases and in part due to actual implementation of the feature (thank you, CISCO almighty for keeping us thoroughly entertained by allowing your developers to implement features that are not systematic, logical or well structured. It saves us from a boredom).

However, if I understood correctly some of the things Lazaros said they seem opposing to what Rene said (and actually proved). Im referring to statement that if packet size exceeds the MTU size on interface its getting fragmented. That seems is not correct - it should be dropped. The whole point of what Rene showed and stated was that packet is only getting fragmented if its size exceeds the IP MTU set size. Quite different.

But I wonder still if I understood the whole thing correctly. So here is the summary of the whole confusing topic as I understood it.

Is it really correct? -

  1. There are multiple values of the MTU pertinent to each individual network layer (or corresponding PDU).

  2. MTU for TCP segments is actually called TCP Max Segment Size. It can be set on the router or switch interface to negotiate with the sending/receiving host the size of the Segment. Its size defines only actual TCP payload and does not account for 20B TCP header. By itself it does not affect traffic except changing size of sent data (which is different from lower layers MTUs).

  3. IP MTU is defined as size of the TCP segment plus TCP header (20B) plus the header for IP packet itself (20B). By default on CISCO routers/switches it is 1500B. It is set on router interface and defines a max size of the packet that can pass through the interfaces unharassed. If packet exceed the IP MTU value it will be fragmented (not clear though whats the rule for fragmenting - how parts are defined).

  4. MTU is defined as the size of L2 frame on interface. It includes the size of the IP MTU and… and nothing else. Since the Frame header now is not counted the MTU by default is equal IP MTU. If IP MTU is set to be smaller then there is no problem, frame just would be ‘underutilized’ since no IP packet would be able to fill up the allowed size for the frame. If IP MTU is larger though then packet can come which is larger than MTU in which case the packet is dropped, frame is not formed. Frames do not support fragmentation as packets do.

  5. It is possible to set MTU globally by defining System MTU - it will provide default MTU for all interfaces.

  6. To avoid packet drops at the very least the IP MTU should be equal or less than frames MTU. Reducing IP MTU will lead to higher probability of fragmentation. It maybe beneficial in certain cases, when there is additional header inserted on top of IP header - in this case IP MTU may allow packet size which together with additional header will exceed frame MTU so the packet gets dropped. Similarly, if IP header is replaced with some other, larger than 20B header it may cause the same problem. In that case reducing IP MTU (and/or increasing frame MTU) may prevent packets drops, though at the price of fragmentation.

  7. Where possible it might be better to leave IP MTU alone (though still OK to increase frame MTU) and instead reduce the TCP Masx Segment Size - that will not fragment the packets, so its better from routers utilization point of view, and will help avoiding packets drops due to exceeding MTU.

I guess that covers the most fundamental part. Would be happy if that is all correct. It still leaves some other questions, like alternate and jumbo MTUs for additional discussion.

Also, to add spice to the topic, Path MTU discovery feature is by default enabled on most contemporary equipment, based on ICMP protocol extensions and consists of detecting the disabled fragmentation for packets exceeding IP MTU (by checking if DF bits in packets set to ‘1’) and requesting the sending equipment to resend the dropped due to too large size packets with smaller TCP payload.

Hello Vadim

Whether a packet will be dropped or fragmented depends on the contents of the Do Not Fragment (DF) bit field in the IP header. If it is set to 1 the packet is dropped. If it is set to 0 the packet is fragmented into two frames.

Your explanation seems to show that you indeed have a solid understanding of the topic as I cannot see any issues with what you have written.

Jumbo MTU is the system MTU that is used for GigabitEthernet ports. This is often set to higher values in order to accommodate specific applications that run over the network.

Alternate MTU, by itself, does nothing until you specifically assign it to a specific port. It defines a third interface MTU that fits between the system MTU and jumbo MTU. The system MTU is the interface MTU for the 10/100 ports. The jumbo MTU is the interface MTU for the GE ports. The alternate MTU can be an interface MTU for any 10/100 or GE port. You apply the Alternate MTU with a global command that maps the alternate MTU to the desired interfaces. A change to the alternate MTU or any of its applied interfaces, like the system and jumbo MTU commands, requires a reload.

Yes, this is another feature that can be very useful, but is often not usable because ICMP packets, which are necessary for its functionality, are often blocked for security reasons from many intermediary devices.

I hope this has been helpful!

Laz

Got a few questions-

  1. You mention that it is no problem when the frame reaches 1514 bytes, but realistically how high can this go before it starts causing an issue on Ethernet?

  2. Why did it cause a problem communicating with the web server when you reduced the MTU to 1400? Should it not have just fragmented it? Is the DF bit used in HTTP? I mean it would slow things down a bit to fragment, but not stop them working altogether?

  3. What happens if one side has an MTU of 1400, and the other 1500? Does the side set to 1400 have an issue receiving packets over this size?

  4. What is the impact of jumbo frames on all of this?

  5. TCP MSS clamping doesn’t seem like a great solution, as surely all of the UDP traffic is going to have issues?

Thanks!

Hello Chris

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

For the purposes of the specific lesson, IP packet size of 1500 was deemed problematic for the MTU of 1400 just to get the point across. The possibility of fragmentation and the settings of the DF bit were not analyzed here in order to maintain the simplicity of the example. Notice in the wireshark capture that DF bit was indeed set, so the communication failed. Now whether or not fragmentation does or should occur when web browsing mainly depends on the configuration of the network in question. HTTP/HTTPS do not have any direct influence nor do they require any specific setting on the DF bit. This is determined by the specific networks that the packets are traversing.

I assume you’re talking about the MTU parameter set on the Ethernet interface. If this is the case, you can connect one switchport with an MTU of 1400 to a switchport with an MTU of 1500. Any frames that attempt to enter the 1400 MTU sized interface that are larger will be dropped. MTU size does not have to be the same on both ends in order for a link to function, unlike some other parameters like speed, duplex, trunk configurations and others.

Jumbo frames are those frames that are larger than the standard 1500 bytes. When a switch is configured to accept jumbo frames, the size of those frames are also specified. Any frame up to that size will be accommodated. However, keep in mind that a jumbo frame is a layer 2 concept. When this frame is then called upon to encapsulate an IP packet which in turn encapsulates a TCP segment, the packet and segment sizes must also be smaller than the maximum jumbo frame MTU.

UDP traffic does not conform to the TCP MSS configuration. UDP does not have a concept of MSS. UDP datagrams are fragmented based on whatever fragmentation the IP protocol can achieve. However, for TCP, MSS can be a solution as it will reduce fragmentation and is initially agreed upon at the three way handshake.

I hope this has been helpful!

Laz

Thanks Laz

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

I guess the question I’m asking here is - how high can you increase the MTU before it’s going to cause an issue? I know the default is 1500, but will 1510 be ok? 1520? 1530? etc

Hello Chris

Problems will only arise if the configured maximum allowed MTU is smaller than the size of the frame that is being sent. The problems come in two forms: Either the DF bit is set and the frame is dropped or the DF bit is not set and the transmission suffers from excessive fragmentation, and thus ultimately slower data transfer rates.

Now the maximum allowed MTU on an interface of a Cisco device can be modified from the default of 1500 to values exceeding 9000 on some platforms. If your allowed MTU is set to 9000 for example, you can easily send frames of up to that size without any problems.

I hope this has been helpful!

Laz

1 Like

Hi Laz

Are you sure there is not also another problem the other way, i.e. the MTU is set too high and then packets are unable to be transmitted if the medium can’t handle it?

Why isn’t the MTU 9000 by default?

Hello Chris

The reason why the MTU is not set to 9000 by default is that 1500 is the default according to the Ethernet standard worldwide. So all vendors, all networks, and ultimately, the Internet, uses this default. If the MTU is set to 9000 on your internal network and you try to send a frame of 9000 out of the pipe that connects you to the Internet, it will be dropped or fragmented drastically. Typically, 1500 is a good number for 100Mbps Ethernet, while other layer 2 technologies have various other MTUs set such as serial, DSL, cable, satellite etc. So if you have 1Gbps on your internal network, you can set your MTU to 9000, but any communication with the outside world will require fragmentation, and this could cause problems for you.

If the Layer 2 technology being used can handle it, such as GigabitEthernet can, then you have no problems. I am not sure however if a 9000 byte MTU will have any negative effects on slower Ethernet standards such as 10Mbps or even 100Mbps. It might be worth an experiment to see how those speeds are affected if at all.

I hope this has been helpful!

Laz

wow this is really good information… this is worthy of being in the lesson!!! this explains a few smaller questions I had but was letting slide.

1 Like

I am interested in this down stream idea. I am curious if we can change the payload size multiple times along the route? I say that we can. I think the ISP have to do this on ISP interfaces I saw all of them was always set at or around 9k. yet they go through mulitple technologies and protocols and then to other customers and other ISP. So it only makes sense that they are changing the payload all along these paths and so are others.

Is it TCP MSS that they are using to do this for all the technologies maybe something else? could you elaborate theory wise about the pros or cons or what you know about changing the payload size? This is pretty much the only real question I have on this I think. I am reading through all these forum post they are priceless. This has to be the single most important forum topic I have read so far I can think of off the top of my head as there is just so much information contained here in the examples given.

Hi Laz,

you had replied to this but I thought TCP MSS was layer 4 as you had posted that earlier in the forum post. For that reason it would not be set on a layer 2 switch correct? It would have to be a switch that could do layer 3 and 4?

Or perhaps I am over complicating it and TCP MSS while it is layer 4 also on receiving side can work within layer 2 functionality just to communicate the smaller payload?

https://tools.ietf.org/html/rfc879#section-3

Hello Brian

Within an ISP’s network, it is usually best practice to set the MTU size to the maximum (that is the L2 MTU) so that this limitation does not become a reason for customer traffic to be dropped. I had a situation where customer traffic was being sent through a MAN where the customer was sending all of his internal VLANs using QinQ which adds an additional 4 bytes of overhead for a second VLAN tag. I had some IP phones registering to a Cisco Call Manager through this link, and one model of IP phone’s worked fine while the other model didn’t This was because the size of the frames sent for registration by one phone were at the maximum (1504 with the additional tag) while the others were smaller. It wasn’t until I increased the MTU of the MAN to let the larger size frames go through that all the phones registered. But I digress…

So it is indeed good practice to have jumbo frames at maximum within the ISP, however, when frames leave the ISP and move into the customer network, or to the Internet at large, you should be aware that MTU sizes will probably change when going in both directions.

If you’re going to be adjusting the TCP MSS, this has to be done at the source of the TCP session, because it is configured at layer 4. Switches and routers along the path would not be involved with this parameter. It is only relevant during encapsulation from layer 4 to layer 3 at the source host of the session. But it can affect whether or not a frame/packet actually does successfully traverse the intervening networks.

There is not much benefit in changing the payload size throughout the path that a packet may take. It may happen of course, because the packets travel through so many different networks that are administered by various entities, but it will be of no benefit to the overall network operation. This is because if you have a restrictive MTU somewhere along the path, the MTU of the communication from end to end must be small enough to accommodate that smallest MTU. If the DF bit is not set, then when a packet reaches a router that goes from a link with a larger MTU to a smaller one, it will be able to fragment the IP packet into two frames of smaller size, however, if the MTU gets larger farther down, routers will not reassemble a fragmented packet. This can occur only at the intended destination.

I hope this has been helpful!

Laz

Hello again Brian

TCP MSS is indeed configured at layer 4, and is only adjustable at the TCP session initiator. It affects the way that TCP segments are encapsulated into IP packets. It cannot be adjusted at all at routers or switches along the path since they will only de-encapsulate up to layer 3. I give a bit more detail in the previous post on this thread.

I hope this has been helpful!

Laz

What effect would the fragmentation have on the network?

Hello Thierno

The quick answer is that the more fragmentation you have on a network, the less efficient, and thus thus the slower the network will operate. If you consider IP packets (layer 3) as pieces of freight and the frames (Layer 2) as trucks carrying that freight, then you can get an idea of what I mean.

Trunks have a maximum capacity of X let’s say, and each piece of freight that comes in and is ready to be loaded on the truck must be of a size of X or less. If it is, it will be loaded and sent away. If it is larger than X, it must be broken in two or more and placed on two or more trucks. The trucks carrying the pieces of this freight must be identified somehow with documentation stating that when the trucks arrive at the next stop, the freight must be removed and reassembled. This whole procedure takes time, additional overhead information and more trucks, and is thus less efficient.

Similarly, if an IP packet is larger than the allowable frame size, during encapsulation, it will be fragmented into two or more pieces, and each piece will be placed in a different frame. Each IP fragment gets its own header (more overhead) and Information about the fragmentation is placed within that header. Each IP fragment is placed in a separate Ethernet frame each with its own header as well (more overhead). CPU resources are required by network devices to decapsualte, reorder and reassemble the IP packet.

Also keep in mind that an efficient network will carry frames that are close to the maximum allowed MTU. This ensures the largest data-to-overhead ratio, allowing for the highest throughput for the lowest amount of overhead. Fragmentation often results in frames much smaller than this maximum.

If you imagine that a network device processes hundreds of thousands or millions of packets per second, you can quickly realize how damaging IP packet fragmentation may get.

I hope this has been helpful!

Laz

Hello Rene

SO what is the difference betwen the l2 mtu and ip mtu? If I adjust or increse the ip mtu should I increse the eth mtu as well?

Hello Rafael

IP MTU is the maximum size that an IP packet can have. This value defines if a segment at the Transport Layer fits into the payload of the IP packet at the Network layer during encapsulation. If it does not, the segment must be fragmented.

The L2 MTU, or interface MTU is a setting on an interface that states the maximum size Ethernet frame (minus the header) that it can forward. If an Ethernet frame that is larger than the configured L2 MTU, the frame is dropped.

So the first has to do with the encapsulation process while the second has to do with the transmission from a particular interface.

I hope this has been helpful!

Laz