Troubleshooting Inter-VLAN Routing

This topic is to discuss the following lesson:

Nice. you are awesome Rene. God bless you.
because of you, i have cleared my first round of interview. Interviewer asked me one question to which i haven’t answered well.
What are steps you will take to troubleshoot a network running ospf protocol?

Could you please help with this?

Hi Rohit,

Sorry for the late reply. In case anyone else reads this, I have two good documents of how to troubleshoot OSPF:

https://networklessons.com/cisco/ccnp-tshoot/troubleshooting-ospf-neighbor-adjacency/

https://networklessons.com/cisco/ccnp-tshoot/troubleshooting-ospf-route-advertisement/

nice units so far. Only three more units to go until finish my first round of studying for Tshoot. I love learning and studying but sometimes when your really hard at it for a few days some of the fun is lost and it becomes work.

1 Like

Hi Renee thabnks very helpful

Hi Laz,

Is this possible to assign ip addresses of different n/w to vlans b/c here ports are switchport not routed. this dragged me in confusion regarding assignment of the ip addresses to the VLAN, kindly clarify.

Hello Pradyumna

You cannot assign IP addresses to switchports because they are Layer 2 ports, but you can assign VLANs to switchports. Each VLAN on a switch can have one or more switchports assigned to it. In addition, each VLAN must have a configured SVI in order for hosts on that VLAN to be able to access other networks. The SVI is configured with an IP address and acts as the default gateway for that subnet.

For this reason, IP addressing must correspond to the VLANs that exist on a switch. What this means is that a single VLAN will typically correspond to a single IP subnet, with a single SVI acting as the gateway for that subnet.

Within a single multipoint switch, you can only have a single SVI per VLAN, and thus only a single IP address for the default gateway. Now if you have two switches connected via a trunk, then you can have multiple SVIs with IP addresses. In this case, if you do use multiple SVIs on multiple switches for the same VLANs, you must be careful how you configure them. They should be on the same subnet, but you may run into some interesting behaviours such as unicast flooding and asymmetric routing. Take a look at this lesson for more information:

I hope this has been helpful!

Laz

Thanks for clarification Laz.I have one more doubt that is What is the SVI and why we need this, is SVI also used to put switch in VLAN as we want ?

Hello Pradyumna

An SVI is the same thing as a VLAN interface. It functions as a (virtual) routed port within the VLAN and acts as the default gateway for all hosts within that specific VLAN, when configured on an L3 switch. On an L2 switch, an SVI is simply an IP address you can use to connect to the switch itself using CLI with protocols such as SSH and telnet.

I hope this has been helpful!

Laz

SVI behavior

This was the closest related topic I found to post this question. My question is about the behavior of SVIs.

When an SVIs VLAN does not contain any interfaces the line protocol will be down until switchports are placed in it’s VLAN. I understand this point.

When an SVI exists but its VLAN does not exist both the status and line protocol will be down. I understand this point.

When an SVI exists and it’s VLAN exists containing switchports the status and line protocol will be up. I got it.

But when the vlan is deleted the status is up and the line protocol is down until I shut down the SVI and no shut the SVI. Now it is down down.

Why does’nt the status change to down without having to manually shutdown the SVI?

Thanks

Hello Donald

This primarily has to do with the internal mechanisms and workings of the switch itself. This reminds me of another issue that involves the creation and deletion of VLANs. If you create a VLAN, and assign certain access ports to that VLAN, and then delete the VLAN, those access ports are still configured on that now-non-existent VLAN that you deleted.

I believe this issue, as well as the one that you are mentioning in your post, has to do with the way VLANs and their related entities are created and maintained in a switch.

The existence of a VLAN is not a prerequisite to assigning a VLAN to an access port. Nor is it a prerequisite to creating an SVI on that VLAN. In other words, the IOS doesn’t check to see if the VLAN exists to allow or deny these actions. It simply allows them, but the device will only operate correctly when the VLAN is created and exists.

So based on this logic of how the Cisco IOS works, I believe that when you delete the VLAN, the IOS doesn’t bother to update the current operational status of the SVI. The result is that the indication of the status up and line protocol down will remain. When you shutdown and bring the SVI back up, only then will it check to see if the VLAN exists to determine the status of the SVI.

Ultimately this doesn’t change anything as far as the operation of the network itself, but it does indicate some of the intricacies of the inner workings of the Cisco IOS.

I hope this has been helpful!

Laz

I though it was an IOS bug. Looks like just how IOS manages VLANs

Thanks LAZ

1 Like

Good morning, Everyone.
I hope all is good with you.
Please, see the diagram of my project :
Please, see also the configurations:
For acc1:

configure terminal 
! Enter configuration commands, one per line.  End with CNTL/Z.
no ip routing
vlan 10
name VLAN10
exit
vlan 20
name VLAN20
exit
interface e0/0
	switchport mode access 
	switchport access vlan 10
	spanning-tree portfast 
	! %Warning: portfast should only be enabled on ports connected to a single
	! host. Connecting hubs, concentrators, switches, bridges, etc... to this
	! interface  when portfast is enabled, can cause temporary bridging loops.
	! Use with CAUTION
	! %Portfast has been configured on Ethernet0/0 but will only
	! have effect when the interface is in a non-trunking mode.
	spanning-tree bpduguard enable 
	exit
interface e1/0
	switchport trunk encapsulation dot1q 
	switchport mode trunk 
	! *Mar 28 05:05:19.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to down
	! *Mar 28 05:05:22.766: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to up
	switchport trunk native vlan 100
	switchport trunk allowed vlan 10,20,100
	exit
do write

For acc2:

configure terminal 
! Enter configuration commands, one per line.  End with CNTL/Z.
no ip routing
vlan 10
name VLAN10
exit
vlan 20
name VLAN20
exit
interface e0/0
	switchport mode access 
	switchport access vlan 20
	spanning-tree portfast 
	! %Warning: portfast should only be enabled on ports connected to a single
	! host. Connecting hubs, concentrators, switches, bridges, etc... to this
	! interface  when portfast is enabled, can cause temporary bridging loops.
	! Use with CAUTION
	! %Portfast has been configured on Ethernet0/0 but will only
	! have effect when the interface is in a non-trunking mode.
	spanning-tree bpduguard enable 
	exit
interface e2/0
	switchport trunk encapsulation dot1q 
	switchport mode trunk 
	! *Mar 28 05:05:19.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to down
	! *Mar 28 05:05:22.766: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet1/0, changed state to up
	switchport trunk native vlan 100
	switchport trunk allowed vlan 10,20,100
	exit
do write

For distrib:

configure terminal 
! Enter configuration commands, one per line.  End with CNTL/Z.
ip routing 
vlan 10
name VLAN10
exit
vlan 20
name VLAN20
exit
interface e1/0
	! *Mar 28 05:17:14.935: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on Ethernet2/0 (1), with acc2 Ethernet2/0 (100).
	switchport trunk encapsulation dot1q 
	switchport mode trunk
	switchport trunk native vlan 100
	switchport trunk allowed vlan 10,20,100
	exit
! *Mar 28 05:19:06.620: %CDP-4-NATIVE_VLAN_MISMATCH: Native VLAN mismatch discovered on Ethernet2/0 (1), with acc2 Ethernet2/0 (100).
interface e2/0
	switchport trunk encapsulation dot1q 
	switchport mode trunk 
	switchport trunk native vlan 100
	switchport trunk allowed vlan 10,20,100
	exit
interface e0/0
	no switchport 
	! *Mar 28 05:23:07.430: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up
	! *Mar 28 05:23:08.434: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
	ip address dhcp
	no shutdown 
	exit 
interface vlan 10
	! *Mar 28 05:25:09.032: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to down
	ip address 192.168.10.10 255.255.255.0
	no shut
	exit
	! *Mar 28 05:25:31.668: %LINK-3-UPDOWN: Interface Vlan10, changed state to up
	! *Mar 28 05:25:32.676: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to up
interface vlan 20
	! *Mar 28 05:25:09.032: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan20, changed state to down
	ip address 192.168.20.20 255.255.255.0
	no shut
	exit
	! *Mar 28 05:25:31.668: %LINK-3-UPDOWN: Interface Vlan20, changed state to up
	! *Mar 28 05:25:32.676: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan20, changed state to up
do show ip interface brief
do show vlan brief
do show interfaces trunk
ip dhcp pool VLAN-10
	network 192.168.10.0 255.255.255.0
	default-router 192.168.10.10
	dns-server 8.8.8.8
	exit
ip dhcp excluded-address 192.168.10.1 192.168.10.199
ip dhcp pool VLAN-20
	network 192.168.20.0 255.255.255.0
	default-router 192.168.20.20
	dns-server 8.8.8.8
	exit
ip dhcp excluded-address 192.168.20.1 192.168.20.199
ip route 0.0.0.0 0.0.0.0 192.168.122.1
ip domain-lookup 
ip name-server 8.8.8.8
interface e0/0
	ip nat outside 
	exit
interface vlan 10
	ip nat inside 
	exit 
interface vlan 20
	ip nat inside 
	exit
access-list 10 permit 192.168.10.0 0.0.0.255
ip nat inside source list 10 interface e0/0 overload 
access-list 20 permit 192.168.20.0 0.0.0.255         
ip nat inside source list 20 interface e0/0 overload 
do write

I am able to ping hosts from different vlans among each other but I am not able to access Internet.
Please, see the error from the hosts :
From the hosts, I am able to ping 192.168.122.9 but not 192.168.122.1.
Also, when I try do show ip nat translations from the “distrib”- switch it shows nothing.
I think, the issue is related with the PAT or the ACls, or both but cannot resolve it for a while now.
Please, give some help for resolving it.
Thank you, and Best Regards,
Asen

Created your lab. I was not able to ping out either. I had to make some changes in the lab to get it work.
Added a router and moved sub-interfaces from distrib to router
router is now doing NAT for sub-interfaces and outside
Moved DHCP to the router
Added routes to distrib after removing the sub-interfaces and access-lists and DHCP

Curious to see how the original configuration is solved.

Hello Asen

Thanks for sharing your details.

I would troubleshoot this problem like so:

  1. First, I would remove all NAT configurations from the distrib switch and see if my routing works correctly. Once NAT is removed, try to ping 192.168.122.9 and 192.168.122.1. If you can ping both you’re OK. If you can’t ping 192.168.122.1, then do some troubleshooting to see where the packet reaches, and where it fails. I have a suspicion that the problem is at the cloud, because it doesn’t know the 192.168.10.0/24 network and can’t return the ping. But you take a look and check it out.
  2. If the pings were successful, then check out the NAT configuration. At first glance, the config looks good, so I think that the problem is most likely routing. But take these steps and let us know your results.

Oh, and one more thing. When you share configs, it is much clearer if you share the relevant output of the show running-config command rather than all of the commands as they are inputted in the CLI. The responses of the CLI and the syslog messages that appear in between make it much more difficult to read.

Let us know how you get along!

I hope this has been helpful!

Laz

Thank you very much for your prompt reply, Laz. I will test what you have advised, and will revert back to you.
Also, I will start sending the show running-config as advised, and will share all additional information as links to the Shared Folders so Everyone who has questions for the particular or similar example to have a solution with a code.
Btw, in VirtualBox, I have created, and setup a pfSense- firewall, MS Server 2016 without AD, and one with AD, DNS, and DHCP, and CentOS 7 with a test web- site (http, and https) on NGINX, (also, I am about to finish Kali with Metasploit, Armitage, and some more installed, and another Cent OS 7 with Metasploitable 2 there). I also have a Mint 21.1 with GNS3 with some Cisco, and other appliances installed (you see it in the image I sent with the post). I am using those, with my students for some security classes. See the screenshot. So if you think, the colleagues here will benefit of this, please let me know where and how to send the *.ova- files.
Best,
Asen

Hi Donald

It looks like you simply moved the routing capabilities from the Layer 3 switch to the router by creating a router-on-a-stick situation. Although not ideal, it is a solution. But it’s not solving the actual problem of getting the Layer 3 switch to operate correctly.

It would be best to keep that functionality in the Layer 3 distrib switch. I’m not sure where the problem is, but hopefully with the troubleshooting steps that I suggested in my previous post, @agrozdanov will come to a solution…

I hope this has been helpful!

Laz

Hello Asen

Great, let us know how you get along.

Yes that will be helpful. Just keep in mind, there is no need to copy and paste the whole config, just the portions of the config that are relevant to the particular question involved. That way posts are easier to read and respond to.

That sounds like a great setup! I will let Rene know and he will get back to you about that.

I hope this has been helpful!

Laz

Good morning Laz, and Donald.

Thank you for your time.

What I have done, was to create new, simplest infrastructure with one 3-layer-switch, and no VLANs (see the screenshot here and also the unsuccessful ping from the host to 192.168.122.1), and to apply what Laz initially has asked me to. Please, see the shortened outcomes below:

hree-l-sw(config)#do sh run      
Building configuration...

Current configuration : 1738 bytes
!
! Last configuration change at 09:39:45 EST Sat Apr 1 2023
!
version 15.2
!
hostname three-l-sw
!
no ip icmp rate-limit unreachable
!
ip dhcp excluded-address 192.168.1.1 192.168.1.199
!
ip dhcp pool MY-DHCP
 network 192.168.1.0 255.255.255.0
 default-router 192.168.1.1 
 dns-server 8.8.8.8 
!
!
ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
!
interface Ethernet0/0
 no switchport
 ip address dhcp
!
interface Ethernet0/1
 no switchport
 ip address 192.168.1.1 255.255.255.0
!
interface Ethernet0/2
!
!
interface Ethernet3/3
!
interface Vlan1
 no ip address
 shutdown
!
ip forward-protocol nd
!
ip tcp synwait-time 5
ip http server
!
ip route 0.0.0.0 0.0.0.0 192.168.122.1
ip ssh server algorithm encryption aes128-ctr aes192-ctr aes256-ctr
ip ssh client algorithm encryption aes128-ctr aes192-ctr aes256-ctr
!


three-l-sw(config)#do sh ip route 
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 192.168.122.1 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 192.168.122.1
      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.1.0/24 is directly connected, Ethernet0/1
L        192.168.1.1/32 is directly connected, Ethernet0/1
      192.168.122.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.122.0/24 is directly connected, Ethernet0/0
L        192.168.122.190/32 is directly connected, Ethernet0/0

So it seems the routing is the issue. When I implement a Router instead of the 3-layer-switch, it is working fine. Would it be something related with the image itself ?

Best,

Asen

Good evening, Laz, and Donald.
Mission completed !
Please, see the screenshot:
I have done successfully the inter-vlan communication, access of all hosts to the Internet, and dhcp- relays where needed.
I will try implementing ASAv instead of the router so the project to become like a basic project for SoHo (or perhaps small SMB)'s with access to the Internet, and internal segmentation.
So far the issue for me is:
I am using a Linux Mint 21.1 VirtualBox Guest, and GNS3 2.2.38 inside. I have no problem to access remotely the IOU- images - Routers, Switches, Docker Containers using either Terminator (with Telnet) or GVncViewer (VNC), but the QEMU images are showing - “Guest has not initialized the display (yet).”- message when I am trying to start it initially with VNC or Spice.
Can you help with this, please - I have researched more than two hours, and I cannot resolve it so far ?
Same problem appears when I try to install an image like Kali, e.g. They even have the OS as QCOW2- image , and still my display is stuck …
Best,
Asen