Cisco SD-WAN Hub and Spoke Topology

Lessons Discussion
21

In addition to my previous comment, I don’t know if you had this in mind when you wrote the quote, but I have noticed that, initially, it is not impossible to build tunnels between MPLS and BIZ-INTERNET TLOCs since BFD sessions are down. This is because, at node Gateway, everything is segmented into 3 VRFs - inet, mpls, and vpn0.

There’s route leaking between vpn0-mpls, and vpn0-inet, but not between mpls-inet, and this is the main culprit. Therefore, you need to perform the route-leaking between these two VRFs, and the BFD sessions, including tunnels will come up instantly.

However, there is a strange thing that I have noticed before performing route-leaking between mpls and inet VRFs - I was able to ping, for example, from Edge2 MPLS interface (Gi2) to Edge 1 INET interface (Gi1) with MPLS default gateway disabled.

How is that possible? If the traffic is sourced from Edge2 Gi2 (172.16.2.2) and destined to Edge 1 Gi1 (172.16.1.1) it needs to go via default gateway (172.16.2.1) but it didn’t according to traceroute, it goes directly to 172.12.1.1.

Even more stranger thing was that, before performing inter-vrf route leaking on the Gateway, TLOCs had the C,I,R status, but the BFD was still down for MPLS-BIZ-INTERNET. I guess in that case, the Edge was using one of two next-hops for default route (172.16.1.254 and 172.16.2.254) but this wasn’t working because route-leaking was not performed yet, so the L3 connection between these two VRFs was failing.

Looking forward to hearing your thoughts.

22

Hello A1A1

I’ve decided to respond to each post separately. For this particular question:

The idea here is that in SD-WAN, your WAN edge routers attempt to build IPsec tunnels between TLOCs. Whether these tunnels can actually form depends on whether the underlying physical networks can reach each other. When you have two Internet circuits (e.g., “biz-internet” and “public-internet” colors), both use the public Internet backbone, and even though they’re different ISPs or different “colors,” they share underlying reachability. This is simply the nature of the Internet itself.

The result is that your SD-WAN routers can successfully build tunnels between different Internet colors (cross-color tunneling).

However, with a private WAN such as an MPLS network, when you have an MPLS circuit and an Internet circuit, this won’t work. They are completely separate, isolated networks, and there is no routing path between them - a packet from your Internet interface cannot reach a private MPLS IP address, and vice versa.

The result is that even though SD-WAN wants to build tunnels between all TLOCs, the underlay network physically prevents tunnel formation between MPLS and Internet colors simply because there is no route.

So with two Internet connections, cross-color tunnel formation happens automatically because public Internet provides universal reachability. However, if you have a private WAN connection (like MPLS) cross-color tunneling between MPLS and Internet cannot happen because these networks are physically isolated from each other at the underlay level.

I hope this has been helpful!

Laz