MTU Troubleshooting on Cisco IOS

Hello Chris

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

For the purposes of the specific lesson, IP packet size of 1500 was deemed problematic for the MTU of 1400 just to get the point across. The possibility of fragmentation and the settings of the DF bit were not analyzed here in order to maintain the simplicity of the example. Notice in the wireshark capture that DF bit was indeed set, so the communication failed. Now whether or not fragmentation does or should occur when web browsing mainly depends on the configuration of the network in question. HTTP/HTTPS do not have any direct influence nor do they require any specific setting on the DF bit. This is determined by the specific networks that the packets are traversing.

I assume you’re talking about the MTU parameter set on the Ethernet interface. If this is the case, you can connect one switchport with an MTU of 1400 to a switchport with an MTU of 1500. Any frames that attempt to enter the 1400 MTU sized interface that are larger will be dropped. MTU size does not have to be the same on both ends in order for a link to function, unlike some other parameters like speed, duplex, trunk configurations and others.

Jumbo frames are those frames that are larger than the standard 1500 bytes. When a switch is configured to accept jumbo frames, the size of those frames are also specified. Any frame up to that size will be accommodated. However, keep in mind that a jumbo frame is a layer 2 concept. When this frame is then called upon to encapsulate an IP packet which in turn encapsulates a TCP segment, the packet and segment sizes must also be smaller than the maximum jumbo frame MTU.

UDP traffic does not conform to the TCP MSS configuration. UDP does not have a concept of MSS. UDP datagrams are fragmented based on whatever fragmentation the IP protocol can achieve. However, for TCP, MSS can be a solution as it will reduce fragmentation and is initially agreed upon at the three way handshake.

I hope this has been helpful!

Laz

Thanks Laz

The size of the frame will only start causing problems when it is larger than the allowed MTU on an Ethernet port. By default, an Ethernet port will allow an MTU of 1500 (without the Ethernet header) so 1514 with the header is no problem for the default.

I guess the question I’m asking here is - how high can you increase the MTU before it’s going to cause an issue? I know the default is 1500, but will 1510 be ok? 1520? 1530? etc

Hello Chris

Problems will only arise if the configured maximum allowed MTU is smaller than the size of the frame that is being sent. The problems come in two forms: Either the DF bit is set and the frame is dropped or the DF bit is not set and the transmission suffers from excessive fragmentation, and thus ultimately slower data transfer rates.

Now the maximum allowed MTU on an interface of a Cisco device can be modified from the default of 1500 to values exceeding 9000 on some platforms. If your allowed MTU is set to 9000 for example, you can easily send frames of up to that size without any problems.

I hope this has been helpful!

Laz

1 Like

Hi Laz

Are you sure there is not also another problem the other way, i.e. the MTU is set too high and then packets are unable to be transmitted if the medium can’t handle it?

Why isn’t the MTU 9000 by default?

Hello Chris

The reason why the MTU is not set to 9000 by default is that 1500 is the default according to the Ethernet standard worldwide. So all vendors, all networks, and ultimately, the Internet, uses this default. If the MTU is set to 9000 on your internal network and you try to send a frame of 9000 out of the pipe that connects you to the Internet, it will be dropped or fragmented drastically. Typically, 1500 is a good number for 100Mbps Ethernet, while other layer 2 technologies have various other MTUs set such as serial, DSL, cable, satellite etc. So if you have 1Gbps on your internal network, you can set your MTU to 9000, but any communication with the outside world will require fragmentation, and this could cause problems for you.

If the Layer 2 technology being used can handle it, such as GigabitEthernet can, then you have no problems. I am not sure however if a 9000 byte MTU will have any negative effects on slower Ethernet standards such as 10Mbps or even 100Mbps. It might be worth an experiment to see how those speeds are affected if at all.

I hope this has been helpful!

Laz

wow this is really good information… this is worthy of being in the lesson!!! this explains a few smaller questions I had but was letting slide.

1 Like

I am interested in this down stream idea. I am curious if we can change the payload size multiple times along the route? I say that we can. I think the ISP have to do this on ISP interfaces I saw all of them was always set at or around 9k. yet they go through mulitple technologies and protocols and then to other customers and other ISP. So it only makes sense that they are changing the payload all along these paths and so are others.

Is it TCP MSS that they are using to do this for all the technologies maybe something else? could you elaborate theory wise about the pros or cons or what you know about changing the payload size? This is pretty much the only real question I have on this I think. I am reading through all these forum post they are priceless. This has to be the single most important forum topic I have read so far I can think of off the top of my head as there is just so much information contained here in the examples given.

Hi Laz,

you had replied to this but I thought TCP MSS was layer 4 as you had posted that earlier in the forum post. For that reason it would not be set on a layer 2 switch correct? It would have to be a switch that could do layer 3 and 4?

Or perhaps I am over complicating it and TCP MSS while it is layer 4 also on receiving side can work within layer 2 functionality just to communicate the smaller payload?

https://tools.ietf.org/html/rfc879#section-3

Hello Brian

Within an ISP’s network, it is usually best practice to set the MTU size to the maximum (that is the L2 MTU) so that this limitation does not become a reason for customer traffic to be dropped. I had a situation where customer traffic was being sent through a MAN where the customer was sending all of his internal VLANs using QinQ which adds an additional 4 bytes of overhead for a second VLAN tag. I had some IP phones registering to a Cisco Call Manager through this link, and one model of IP phone’s worked fine while the other model didn’t This was because the size of the frames sent for registration by one phone were at the maximum (1504 with the additional tag) while the others were smaller. It wasn’t until I increased the MTU of the MAN to let the larger size frames go through that all the phones registered. But I digress…

So it is indeed good practice to have jumbo frames at maximum within the ISP, however, when frames leave the ISP and move into the customer network, or to the Internet at large, you should be aware that MTU sizes will probably change when going in both directions.

If you’re going to be adjusting the TCP MSS, this has to be done at the source of the TCP session, because it is configured at layer 4. Switches and routers along the path would not be involved with this parameter. It is only relevant during encapsulation from layer 4 to layer 3 at the source host of the session. But it can affect whether or not a frame/packet actually does successfully traverse the intervening networks.

There is not much benefit in changing the payload size throughout the path that a packet may take. It may happen of course, because the packets travel through so many different networks that are administered by various entities, but it will be of no benefit to the overall network operation. This is because if you have a restrictive MTU somewhere along the path, the MTU of the communication from end to end must be small enough to accommodate that smallest MTU. If the DF bit is not set, then when a packet reaches a router that goes from a link with a larger MTU to a smaller one, it will be able to fragment the IP packet into two frames of smaller size, however, if the MTU gets larger farther down, routers will not reassemble a fragmented packet. This can occur only at the intended destination.

I hope this has been helpful!

Laz

Hello again Brian

TCP MSS is indeed configured at layer 4, and is only adjustable at the TCP session initiator. It affects the way that TCP segments are encapsulated into IP packets. It cannot be adjusted at all at routers or switches along the path since they will only de-encapsulate up to layer 3. I give a bit more detail in the previous post on this thread.

I hope this has been helpful!

Laz

What effect would the fragmentation have on the network?

Hello Thierno

The quick answer is that the more fragmentation you have on a network, the less efficient, and thus thus the slower the network will operate. If you consider IP packets (layer 3) as pieces of freight and the frames (Layer 2) as trucks carrying that freight, then you can get an idea of what I mean.

Trunks have a maximum capacity of X let’s say, and each piece of freight that comes in and is ready to be loaded on the truck must be of a size of X or less. If it is, it will be loaded and sent away. If it is larger than X, it must be broken in two or more and placed on two or more trucks. The trucks carrying the pieces of this freight must be identified somehow with documentation stating that when the trucks arrive at the next stop, the freight must be removed and reassembled. This whole procedure takes time, additional overhead information and more trucks, and is thus less efficient.

Similarly, if an IP packet is larger than the allowable frame size, during encapsulation, it will be fragmented into two or more pieces, and each piece will be placed in a different frame. Each IP fragment gets its own header (more overhead) and Information about the fragmentation is placed within that header. Each IP fragment is placed in a separate Ethernet frame each with its own header as well (more overhead). CPU resources are required by network devices to decapsualte, reorder and reassemble the IP packet.

Also keep in mind that an efficient network will carry frames that are close to the maximum allowed MTU. This ensures the largest data-to-overhead ratio, allowing for the highest throughput for the lowest amount of overhead. Fragmentation often results in frames much smaller than this maximum.

If you imagine that a network device processes hundreds of thousands or millions of packets per second, you can quickly realize how damaging IP packet fragmentation may get.

I hope this has been helpful!

Laz

Hello Rene

SO what is the difference betwen the l2 mtu and ip mtu? If I adjust or increse the ip mtu should I increse the eth mtu as well?

Hello Rafael

IP MTU is the maximum size that an IP packet can have. This value defines if a segment at the Transport Layer fits into the payload of the IP packet at the Network layer during encapsulation. If it does not, the segment must be fragmented.

The L2 MTU, or interface MTU is a setting on an interface that states the maximum size Ethernet frame (minus the header) that it can forward. If an Ethernet frame that is larger than the configured L2 MTU, the frame is dropped.

So the first has to do with the encapsulation process while the second has to do with the transmission from a particular interface.

I hope this has been helpful!

Laz

Hi Laz,

So if I for examople change the ip mtu to 9216 bytes, Do I have to change the ethernet mtu also or I should just leave it with its default? And secondly is this going to affect the packets transmision?
Thanks
Rafael

Hello Rafael

When you change the MTU value with the mtu interface configuration command, the IP MTU value will also change to match. However, the opposite is not true, that is, if you change the IP MTU, this does not change the MTU of the interface.

So, if you change the IP MTU to 9216 bytes, and you keep the interface MTU at 1500 bytes, then any IP packet that is larger than 1500 bytes (up to 9216 bytes) will be fragmented in order to be sent in frames of a maximum of 1500 bytes. Remember, the IP MTU of the router allows IP packets of this size, but this assumes that the originating host and all network devices in between allow this size as well without fragmentation.

I hope this has been helpful!

Laz

So if the router cannot alter the TCP mss “negotiation” between a server and host then why do we set it? I get we can set it for traffic generated on the router itself but I was under the impression we cannot alter it in between devices.

Also, if I change the IP MTU or interface MTU will either of these automatically change the MSS on that interface to compensate ?

Thanks :slight_smile:

Hello John

I apologize I was not completely clear in my explanation. You are correct in that you can actually affect the TCP MSS of a session.

Unlike the IP MTU and the Interface MTU which can be configured and affect every frame and packet that traverses an interface of the device, the ip tcp adjust-mss command will intercept only the initial negotiation SYN packet during the three way handshake and is able to adjust or modify the maximum TCP MSS that will be used for that session. All subsequent segments will conform to that adjusted TCP MSS.

Another related command which I had in mind is the ip tcp mss command available in Cisco Nexus platforms that adjust the maximum TCP MSS for all TCP communication that originates or terminates on the network device itself.

Now if you change the IP or interface MTU, these will not automatically change the TCP MSS on that interface to compensate. If segments larger than the IP or interface MTU are sent, fragmentation will take place, fragmenting larger IP packets into multiple frames (if fragmentation is allowed) which will reduce efficiency. Make sure the MSS and MTUs are sufficiently coordinated in order to maximize efficiency.

I hope this has been helpful!

Laz

Thanks for clearing that up Laz !

Last question, is there a scenario were a recieving interface MTU that is configured to say 1200, will refuse to fragment a packet other than just when the DF bit is set?

Basically Is there ever a scenario were it just can’t process a packet?

I sometimes get confused at how an Ethernet interface can fragment a packet to allow it in when it’s cannot actually process that amount of bytes physically. I have no trouble understanding that when the packet is accepted in and the IP MTU is lower that the IP can fragment the packets as to me this is more logical of a logical change than physical.

Hope this makes sense.

Hello John

I think the important thing here to note is that all of the MTU configs affect encapsulation. So if there is a segment with a size larger than the IP MTU, then that segment will be fragmented into multiple IP packets during encapsulation from L4 to L3. This would actually occur at the host that is sending the data. (routers do not de-encapsulate PDUs to layer 4 unless they are functioning as deep inspection firewalls, but I digress).

If a packet comes in to a router and is deencapsulated, its destination IP address is checked, and the exit interface is determined, then the packet will be reencapsulated from L3 to L2. It is at this point that the IP MTU and the Interface MTU are examined. If the interface MTU is too small for the size of the packet, it will be fragmented and encapsulated into two frames unless the DF bit is set. I don’t know of any other scenario in which fragmentation will not occur when the DF bit is not set.

I hope this has been helpful!

Laz

1 Like