VXLAN Question Around Loopback reachability across VPC Cluster

Data Center
1

Hi Folks, I hope all is well. I have a question regarding a scenario below & whether there’s any way around it NOT using advertise PIP:

  • Two Switches in a VPC running VXLAN connected to a set of Spines
  • Each switch has a unique loopback configured and is advertised to BGP as EVPN type 5
  • The NVE interfaces are using a secondary IP - in other words an anycast VTEP is configured
  • RD auto & EVPN both applied where both switches have the same ASN and l3vni

The Loopbacks are unable to talk to each other, yet I see the spines reflecting the respective loopbacks to the leaf switches. The leaf switches aren’t accepting the route. Is this because the Secondary IP is locally connected on each switch?

The two options I have are - Advertise PIP command & IBGP over the layer 3 port-channel in my VPC for RT synchronization + communication between switches. Is there another option I’m not considering? I tried to null0 254 the secondary IP from each switch and advertise it out of my BGP but still, the loopbacks in question aren’t being accepted in the leaf switches.

2

Hello Nicolas

This is a very detailed and intricate design, resulting in some interesting routing behavior in your production environment.

I will go over it at a relatively high level, since we’re a more certification-focused forum. Your diagnosis is correct. This is a well-known, expected behavior in Cisco NX-OS VXLAN EVPN vPC designs.

The core problem is that in a vPC pair, both switches share a common Anycast VTEP IP configured as the secondary IP on the NVE source loopback. Without advertise-pip, both vPC peers advertise their EVPN Type-5 routes using this shared Anycast VTEP IP as the BGP next-hop, even for routes that are unique to a specific switch (like individual loopbacks).

When the Spine/Route-Reflector reflects that Type-5 route back to the other vPC peer, the receiving switch inspects the BGP next-hop and recognizes it as one of its own locally configured addresses (the Anycast secondary IP on its own NVE loopback). NX-OS will not install a route whose BGP next-hop resolves to a locally owned address. It is treated as a self-originated or locally connected route and is silently rejected/not installed in the RIB/FIB.

This happens symmetrically on both peers:

  • Peer-1’s loopback Type-5 → advertised with Anycast VTEP as NH → Spine reflects to Peer-2 → Peer-2 sees NH = its own local Anycast IP → route dropped
  • Peer-2’s loopback Type-5 → same thing happens → Peer-1 drops it

The Spines are doing their job correctly. The problem is specifically on the receiving vPC peer.

Now the advertise-pip feature was specifically engineered by Cisco to resolve this exact scenario. When enabled on the NVE interface, it forces NX-OS to use each switch’s unique Primary IP (PIP) as the BGP next-hop for locally-originated EVPN routes (Type-2 and Type-5), rather than the shared Anycast VTEP IP. more info can be found here:

There is no clean, fully supported, scalable workaround that avoids advertise-pip for this specific problem. However, if you HAVE to do it without that feature, then one of these three options may be beneficial depending on your constraints:

  • Option 1: Keep loopbacks in the Underlay (Recommended if avoiding advertise-pip)
  • Option 2: Dedicated L3 Peering Between vPC Peers in the Tenant VRF
  • Option 3: Static Routes (Not Recommended)

For this particular problem, advertise-pip is the cleanest and most correct solution unless you intentionally redesign the reachability so those loopbacks are not dependent on EVPN Type-5 routes using the shared anycast next-hop.

I hope this has been helpful!

Laz