CEF (Cisco Express Forwarding)

This lesson demystified some terms I’ve always wondered about!

It looks like the output for sh ip cef changed somewhere along the way. Do you know what equivalent commands would show the type of adjacency in 15.2?

This is all I get:

R1#sh ip cef 3.3.3.0
3.3.3.0/24
  nexthop 10.0.1.2 FastEthernet0/0
R1#

Thanks!

Hi Kay,

You can add some parameters:

R1#show ip cef 2.2.2.0/24 detail 
2.2.2.0/24, epoch 0, flags [rib only nolabel, rib defined all labels]
  recursive via 192.168.12.2
    attached to GigabitEthernet0/1
R1#show ip cef 2.2.2.0/24 internal 
2.2.2.0/24, epoch 0, flags [rnolbl, rlbls], RIB[B], refcnt 5, per-destination sharing
  sources: RIB 
  feature space:
    IPRM: 0x00018000
  ifnums:
    GigabitEthernet0/1(3): 192.168.12.2
  path list 0C4FE384, 3 locks, per-destination, flags 0x269 [shble, rif, rcrsv, hwcn, bgp]
    path 0EFEFE44, share 1/1, type recursive, for IPv4
      recursive via 192.168.12.2[IPv4:Default], fib 0EE77490, 1 terminal fib, v4:Default:192.168.12.2/32
      path list 0C4FE3D4, 2 locks, per-destination, flags 0x49 [shble, rif, hwcn]
          path 0EFEFEAC, share 1/1, type adjacency prefix, for IPv4
            attached to GigabitEthernet0/1, IP adj out of GigabitEthernet0/1, addr 192.168.12.2 0DA2D8B8
  output chain:
    IP adj out of GigabitEthernet0/1, addr 192.168.12.2 0DA2D8B8

Rene

Hi Rene,

Great leasson !!

I have a question… If in the router I have configured ACL … QoS… Etc, all these conditions are evaluated before the CEF, really?

Thx,

Hi Diana,

It depends on a number of factors. One of them is if your traffic is inbound or outbound.

For example, when you have an outbound access-list then first we have to do the routing part to figure out which outgoing interface to use. If that interface has an access-list then we’ll have to check if the packet is permitted or not.

When traffic is inbound, it will first hit the access-list.

It also depends on the platform that you are using. Some routers/switches will have different hardware tables for ACLs, others might store them in software etc.

Rene

Hi Rene,

I couldnt understand What is the relationship cam , tcam and fib? I am trying to complete packet’s travel from source to destination.

Best Regards

CAM: High speed memory that is primarily used for a switch’s layer 2 lookup information. This information allows the switch to decide which port to send a packet to (a known MAC address) or whether to flood it to all ports (unknown MAC address).

TCAM: Not all switches have this. Think of this as an extension of CAM. It is used for very rapid decisions on ACLs and Quality of Service. On high end layer 3 switches, the TCAM can also contain the FIB, again, so specialized hardware can making routing decisions without interrupting the central CPU of the switch.

FIB: When you think of the FIB vs the RIB, or routing table, the difference is where they “live” on the hardware. The RIB lives in the control plane, while FIB lives in the data plane. Any decisions made at the FIB level are fast and do not require an interrupt (and therefore time) from the device itself.

So the CAM/TCAM and FIB aren’t directly related other than the FIB may or may not be held in the TCAM depending on your switch model.

So, in the example of a packet traveling from source to destination, if a packet’s destination is on a remote subnet, the packet’s destination MAC would be set to the gateway’s MAC. The switch would use the CAM to determine in which port the gateway resides, and it would send it there. Depending on the model hardware involved, the layer3 switch or router would use the FIB to decide what the next hop needs to be (and since the FIB might be in the TCAM–again depending on the model–the TCAM might be used in this process). This would continue until the packet arrives at the destination subnet, where the final switch would again use the CAM table to determine the destination’s MAC and corresponding port.

5 Likes

In your Cisco Campus Network Design Basic lesson you outlined the different switches (Access - 2960-X, 3650, 3850, 4500E, Dist/Core - 4500-X, 4500-E, 6807-XL). Could you just elaborate a little on what type of CAM or TCAM would be in these switches? Would a switch (Cisco) have both a CAM table and a TCAM table? Or is it one or the other? Is CAM table identify as a Layer 2 switch and TCAM is a layer 3 switch?

Hi Jason,

On some of the older switches you might find only a CAM for switching. Nowadays, even the L2 switches have some “L3” capabilities like QoS and access-lists so they’ll have a TCAM.

The 2960-X can also do some routing (just static routes with SVI interfaces) so you’ll find a TCAM for sure. I don’t have an older 2950/2960 switch around but if you have one, try the following command to see what kind of TCAM tables it has:

Switch#show platform tcam utilization 

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                       8412/8412        199/199   
 IPv4 IGMP groups + multicast routes:         1120/1120          1/1     
 IPv4 unicast directly-connected routes:      4096/4096          3/3     
 IPv4 unicast indirectly-connected routes:    2048/2048         38/38    
 IPv4 policy based routing aces:               442/442          12/12    
 IPv4 qos aces:                                512/512           6/6     
 IPv4 security aces:                           954/954          42/42

The output above is from a 3560E.

Rene

Hello Rene

Can you explain the difference between “attached” and “received” in SHOW IP CEF command ?

SW1#sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            no route
0.0.0.0/8            drop
0.0.0.0/32           receive              
11.11.11.11/32       receive              Loopback1
11.11.12.0/24        attached             FastEthernet0/24
11.11.12.0/32        receive              FastEthernet0/24

Thanks
Gary

Hi Gary,

You will see a receive entry for the IP addresses that the router is using itself. As you can see this is a /32 entry (single IP address).

The attached entry are for the subnets on the directly connected interfaces.

Rene

Thanks Rene,

I have another question, in the above section where u have mention about ARP request construct, I think the behavior of switches are different

The multilayer switch will check the routing table, notices that 192.168.20 /24 is directly connected and the following will happen:
The destination MAC address changes from FFF (Multilayer switch Fa0/1 ) to BBB (ComputerB).
The source MAC address changes from AAA (ComputerA) to GGG (Multilayer switch Fa0/2).===This seems to be Incorrect

I LAB this up with 3 switches (all real hardware 3550/3560) [ c3560-ipservicesk9-mz.122-55.SE10/ c3550-ipservicesk9-mz.122-44.SE6]

Simple topology- SW is just acting as regular sw with fa 0/23 & 19 as access port, SW2 and SW3 are acting like a end host (sort of PC with IP address on them)

SW2 -----------{fa 0/23 on both end}----------------SW1-----------{fa 0/19 on both end}-----------------SW3

When ARP request is created by SW2, its destination is always SW1 fa 0/19 and NOT fa 0/23 on SW1!, I also took packet capture and its shows the same thing,

Here is the relevant data, I will appreciate if you can review the config and attached screen shot and advise, Thanks Gary

------------------------------------------------------------------------------

SW1#sh int status | inc 0/19|0/23          
Fa0/19                       connected    1          a-full  a-100 10/100BaseTX
Fa0/23                       connected    1          a-full  a-100 10/100BaseTX
SW1#sh cdp nei
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone, 
                  D - Remote, C - CVTA, M - Two-port Mac Relay 

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2              Fas 0/23          146             R S I  WS-C3560- Fas 0/23
SW3              Fas 0/19          128             R S I  WS-C3550- Fas 0/19

!!! Before PING, SW1 MAC table don’t have any dynamic entry!!!

SW1#sh mac address-table 
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
   1    0015.2b95.4c00    DYNAMIC     Fa0/19
   1    0023.05c6.b6c1    DYNAMIC     Fa0/23
Total Mac Addresses for this criterion: 22
SW2#sh run int fa 0/23
Building configuration...

Current configuration : 86 bytes
!
interface FastEthernet0/23
 no switchport
 ip address 10.10.12.2 255.255.255.0
end

!!! Before PING, SW2 MAC table don’t have any dynamic entry!!!

SW2#sh mac address-table 
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
Total Mac Addresses for this criterion: 20
SW3#sh run int fa 0/19
Building configuration...

Current configuration : 86 bytes
!
interface FastEthernet0/19
 no switchport
 ip address 10.10.12.3 255.255.255.0
end

!!! Before PING, SW3 MAC table don’t have any dynamic entry!!!

SW3#sh mac address-table 
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0015.2b95.4c00    STATIC      CPU
 All    0015.2b95.4c01    STATIC      CPU
 All    0015.2b95.4c02    STATIC      CPU
 All    0015.2b95.4c03    STATIC      CPU
 All    0015.2b95.4c04    STATIC      CPU
 All    0015.2b95.4c05    STATIC      CPU
 All    0015.2b95.4c06    STATIC      CPU
 All    0015.2b95.4c07    STATIC      CPU
 All    0015.2b95.4c08    STATIC      CPU
 All    0015.2b95.4c09    STATIC      CPU
 All    0015.2b95.4c0a    STATIC      CPU
 All    0015.2b95.4c0b    STATIC      CPU
 All    0015.2b95.4c0c    STATIC      CPU
 All    0015.2b95.4c0d    STATIC      CPU
 All    0015.2b95.4c0e    STATIC      CPU
 All    0015.2b95.4c0f    STATIC      CPU
 All    0015.2b95.4c10    STATIC      CPU
 All    0015.2b95.4c11    STATIC      CPU
 All    0015.2b95.4c12    STATIC      CPU
 All    0015.2b95.4c13    STATIC      CPU
 All    0015.2b95.4c14    STATIC      CPU
 All    0015.2b95.4c15    STATIC      CPU
 All    0015.2b95.4c16    STATIC      CPU
 All    0015.2b95.4c17    STATIC      CPU
 All    0015.2b95.4c18    STATIC      CPU
 All    0015.2b95.4c19    STATIC      CPU
 All    0015.2b95.4c1a    STATIC      CPU
 All    0100.0c00.0000    STATIC      CPU
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0100.0ccd.cdce    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
Total Mac Addresses for this criterion: 48
SW2#sh int fa 0/23
FastEthernet0/23 is up, line protocol is up (connected) 
  Hardware is Fast Ethernet, address is 0023.05c6.b6c1 (bia 0023.05c6.b6c1)
  Internet address is 10.10.12.2/24
SW1#sh int fa 0/19       
FastEthernet0/19 is up, line protocol is up (connected) 
  Hardware is Fast Ethernet, address is 0023.05cb.2815 (bia 0023.05cb.2815)
SW1#sh int fa 0/23
FastEthernet0/23 is up, line protocol is up (connected) 
  Hardware is Fast Ethernet, address is 0023.05cb.2819 (bia 0023.05cb.2819)
SW3#sh int fa 0/19
FastEthernet0/19 is up, line protocol is up (connected) 
  Hardware is Fast Ethernet, address is 0015.2b95.4c00 (bia 0015.2b95.4c00)
  Internet address is 10.10.12.3/24

!!! AFTER PING, SW1 MAC table now having dynamic entry and notice the MAC address is of SW2 and SW3!!!

SW1#sh mac address-table 
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 All    0100.0ccc.cccc    STATIC      CPU
 All    0100.0ccc.cccd    STATIC      CPU
 All    0180.c200.0000    STATIC      CPU
 All    0180.c200.0001    STATIC      CPU
 All    0180.c200.0002    STATIC      CPU
 All    0180.c200.0003    STATIC      CPU
 All    0180.c200.0004    STATIC      CPU
 All    0180.c200.0005    STATIC      CPU
 All    0180.c200.0006    STATIC      CPU
 All    0180.c200.0007    STATIC      CPU
 All    0180.c200.0008    STATIC      CPU
 All    0180.c200.0009    STATIC      CPU
 All    0180.c200.000a    STATIC      CPU
 All    0180.c200.000b    STATIC      CPU
 All    0180.c200.000c    STATIC      CPU
 All    0180.c200.000d    STATIC      CPU
 All    0180.c200.000e    STATIC      CPU
 All    0180.c200.000f    STATIC      CPU
 All    0180.c200.0010    STATIC      CPU
 All    ffff.ffff.ffff    STATIC      CPU
   1    0015.2b95.4c00    DYNAMIC     Fa0/19 -----!!!!!This is MAC of 0/19 on sw3
   1    0023.05c6.b6c1    DYNAMIC     Fa0/23-----!!!!!This is MAC of 0/23 on sw2
Total Mac Addresses for this criterion: 22

Hi Gaurav,

In your example, SW1 is a multilayer switch but it is only acting as a L2 switch, switching frames within VLAN 1. It’s not doing any routing and won’t alter anything. The only thing it’ll do is learn source MAC addresses and forward frames based on the destination MAC address.

Rene

make sense, thanks Rene

Hi Rene,

Thanks for the nice illustration and explanation.

If we have CEF turn on, is forwarding decision done in the FIB and is base on the longest matched ?

e.g.

router# show ip route
     ....
     D   192.168.32.0/26 [90/25789217] via 10.1.1.1
     R   192.168.32.0/24 [120/4] via 10.1.1.2
     O   192.168.32.0/19 [110/229840] via 10.1.1.3

I would believe the same entries appear in the FIB. So when required to route a packet to e.g. 192.168.32.200, does the forwarding decision take place in the FIB base on the longest match ? or it is still done using the RIB ?

Regards,

Alan

Hi Alan,

The FIB is built based on information in the RIB.

In the RIB we install routes with the most longest match. When two routes are equal, it’s up to the administrative distance to select the route.

In your example, these three routes will also show up in the FIB.

Rene

1 Like

Hi Rene,

Thanks for the response and sorry for the late reply.

router# show ip route

D 192.168.32.0/26 [90/25789217] via 10.1.1.1
R 192.168.32.0/24 [120/4] via 10.1.1.2
O 192.168.32.0/19 [110/229840] via 10.1.1.3

As per mentioned by you, since the 3 routes will exists in the FIB - let’s say i have a packet to 192.168.32.3, with the longest match ( it will take the 192.168.32.0/24 route).

is this decision to take the longest matched route (192.168.32.0/24 via 10.1.1.2) done using the RIB table or the FIB ?

Regards,
Alan

Hi Alan,

In this example, the longest match will be your 192.168.32.0/26 route for 192.168.32.3 :slight_smile:

The decisions are made by the router, the best routes are installed in the routing table (RIB). The information from the RIB is then copied to the FIB so that forwarding can be done as fast as possible.

Rene

1 Like

Hi Rene,

Thanks for your reply.
Yeap i make a mistake, it should be 192.168.32.0/26.

However, i think you have misunderstood me.
What i meant is, since these 3 routes are all installed in the routing table (RIB) and copied to the FIB table; the router must still select 1 path to use (which is the longest match path - 192.168.32.0/26).

But this selection of the routes out of the 3 available, is done by the router using the RIB or the FIB table ?
Meaning

  1. does the router go to the routing table
    (see that there are 3 routes, and choose the longest match route)

  2. or the router go to the FIB table
    (see that there are 3 routes,entries and choose the longest match route)

Regards,
Alan

Hi Alan,
I believe I understand what you are asking. You are asking whether routing rules, such as “always choose the longest match” require processing at the RIB level or whether this can be done by the FIB. The answer is the FIB.

When CEF is functional, the RIB is simply a control plane mechanism that collects all the best routes from all the routing protocols (or static routes) on the router. A mirror image of all of these is copied to the FIB so that there is a one-to-one ratio between what’s in the RIB and the FIB. The goal of the FIB is to avoid interrupting the main CPU of the router as much as possible. Because of this, more simple decisions like “always use longest matching route” is processed by the FIB.

1 Like

Hi Andrew!,

Glad to see you around and yea! thanks for confirming my doubt.

I have a few more doubts that i have no where to turn to and i hope you will enlightened me…

My coreswitch has ip cef turn on my default.

q1) I have been reading up abit on cisco about IP CEF and it seems like to enable/disable IP CEF, you have to do in at the ingress interface as the decision (e.g. load balance is done there). – is it right ?

Assuming i have “no ip cef” and only wish to turn on ip cef on certain interfaces and ->
q2) if i want to do to packet loadbalancing, should “ip load-share per packet” command be issue on the ingress interface as well ?

q3) Reading How to Verify Cisco Express Forwarding -> “http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-releases-120-mainline/47205-cef-whichpath.html

Use the show interface x/x stat command and determine the number of packets and bytes that the router forwarded through "Processor" instead of "Route cache." Note that "Route cache" includes both fast-switched and CEF-switched packets.
router#show interface stats     
FastEthernet0/0           
Switching path Pkts In  Chars In Pkts Out Chars Out 
Processor 95084 26211621 33493 3386174 
Route cache 24581 1132797 24542 13297583 
Distributed cache 0 0 0 0 
Total  119665 27344418 58035 16683757

Should this command be issue on the ingress or egress interface ?
if it is to be issue on the INGRESS interface -> What does the “Pkts Outs” under Route cache means then ?

Really hope to hear from you soon.

Regards,
Alan