Hello Thomas
Thanks for the suggestion. Yes, clarifications are always helpful to ensure we’re relaying the information as clearly as possible.
I will let Rene know to take a look and consider making changes.
Thanks again!
Laz
Hello, everyone.
A quick question. If I run OSPF internally but BGP with the ISP due to my L3 VPN, I will have to either redistribute or use the network command to put those routes into BGP, so my foreign sites can learn about them.
I randomly thought of this but what if I am either dual or multihomed to my ISP?
Say that R1 is connected to PE1 and R2 is connected to PE2
They both learn about their internal site prefix 10.0.0.0/8 via OSPF but have to redistribute it into BGP so other sites can learn about them.
If R1 advertises this prefix to PE1, PE1 advertises it to PE2, PE2 will advertise it to R2 who will accept it if we are using allowas-in or as-override.. so won’t it technically prefer the MPLS path over the local internal IGP path to reach those prefixes? Since AD 20 < 110 for OSPF.
I feel like allowas-in and as-override can cause a problem here, but they are necessary since we would not accept BGP routes from our foreign sites due to the AS being the same.
What is the best approach in this case?
Thank you
David
Hello David
allowas-in and as-override are not loop-prevention mechanisms. On the contrary, they disable BGP’s AS-path loop check so that the prefix can be accepted, and your feeling that these seemingly necessary commands can cause problems is correct.
Now, whether the MPLS path then gets preferred over the local OSPF path depends on BGP best-path tie-breaking. By default, such a setup is “safe” from loops because of BGP’s default behavior. However, this “safety” is, at best, fragile in a multihomed setup. Let me explain what I mean.
This setup usually appears safe by default because when both R1 and R2 originate 10.0.0.0/8 into BGP locally (via network or redistribute), R2’s own locally-originated entry wins the BGP best-path calculation because Weight (32768 for local origin) is evaluated before AS-path length. The advertisement for the route from PE2 becomes a non-optimal alternative.
Even if it were submitted to the RIB, locally-originated BGP routes (considered iBGP) get AD 200 (not AD 20), which loses to OSPF’s AD of 110. So the OSPF path stays in the RIB and is used.
But this safety can break if certain configurations, such as the following, are applied, and it is not always immediately clear why:
- Manual weight or local-preference changes on the PE for traffic engineering
- Route-reflector or PE-side policy that makes the PE2-advertised route look more attractive
- Asymmetric internal topology where R1 must transit R2 to reach the prefix
Any of these can make the received eBGP route from PE2 win over the OSPF route, sending traffic for locally-reachable prefixes out the MPLS core and back, resulting in suboptimal routing or even a routing loop.
What is the best practice for such scenarios? This is where the BGP Site of Origin (SoO) feature comes in. In the MPLS Layer 3 VPN BGP AS Override lesson, Rene states the following:
In this scenario there are no issues since the CE routers are stubs, they only have one exit path. When your customer sites are multihomed or have a backdoor link then you need to use the BGP SoO (Site of Origin) community to ensure you have a loop free topology.
SoO is an extended community that is specifically designed for this scenario. SoO should be configured on PE1 and PE2’s CE-facing sessions with the same SoO value since R1 and R2 belong to the same site.
SoO is set on a per-neighbor basis and guarantees that PE2 will unconditionally refuse to advertise the site’s own prefixes back to R2, regardless of weight/local-preference manipulation. This is an RFC-based PE-side filter that works independently of as-override or allowas-in.
SoO is defined in RFC4364 where it explicitly states:
We add one more restriction on the distribution of routes from PE to CE: if a route's Site of Origin attribute identifies a particular site, that route must never be redistributed to any CE at that site.
So, you should use as-override on the PE (not allowas-in on the CE) for AS-path handling, and combine it with SoO for loop prevention, resulting in a deterministic, standards-compliant solution that doesn’t depend upon BGP defaults to work properly.
I hope this has been helpful!
Laz
1 Like
Hello Laz,
I have a question. Is it common in the real world to connect customers to a switch instead of the PE router directly?
Like this:
CE routers → SP-switch → PE1
From what I understand, it’s beneficial because it conserves interfaces on PE1. If you have hundreds of customers, it would be easier to connect them to a switch instead of using up the interfaces on PE1.
We can then separate the customers by implementing VLANs on SP-switch and map each VLAN to a VRF (sub)interface on the PE router.
Taking this a step further, I would like to ask about the MPLS Multi-VRF feature.
From what I understand, this allows you to support two or more VPNs for your customers in a way where the VRF is extended to the CE router. So one customer with multiple sites (or multiple customers if they share the CE) can use just one CE router and one physical link exists between CE1 and PE1
However, for this to work, CE1 would need to have each interface connected to a branch office assigned to the respective VRF (as highlighted above). CE1 would also need to configure subinterfaces on the uplink interface and map them to the relevant VRFs. The PE router would have to do the same here.
And from then on, we can treat this as a standard MPLS L3VPN with the only difference being, you provide 2 (or more!) VPNs on the same CE.
The assumption here is, I suppose, that BRANCH-1 and BRANCH-2 should either not talk to eachother directly or have some policies, otherwise they wouldn’t be in separate VRFs and we wouldn’t need to run multi-VRF CE in the first place.
Thank you!
David
Hello David
Let me address your second question first. Multi-VRF (officially “Multi-VRF CE” / “Multi-VRF Support” in Cisco documentation) does exactly what you described. It lets a single CE device host multiple VPNs by maintaining separate VRF tables locally on the CE. Two or more customers (or a single customer’s multiple branch offices) can share one CE device and one physical CE–PE link. Interfaces facing each branch or site are assigned to their respective VRF, and the shared uplink uses VLAN subinterfaces (dot1Q trunking) to separate traffic, with each subinterface bound to its VRF on both the CE and PE.
From the PE’s perspective, Multi-VRF CE is indistinguishable from managing multiple separate CEs. In other words, “there is no difference between using the Multi-VRF Support feature or using multiple CE devices” as stated by Cisco.
So functionally, it behaves as a standard MPLS L3 VPN, just extended one hop further down to the CE.
One helpful clarification worth noting is that the CE–PE link itself typically does not run MPLS or exchange labels. The CE uses standard routing (OSPF, EIGRP, RIP, BGP, or static) to exchange routes with the PE per VRF. The PE alone imposes VPN labels and transport labels for the core. This keeps the CE simpler and is the standard deployment.
Now, concerning your first question, the answer is yes. Placing a switch between the PE and multiple CEs is a common service provider practice in locations with high customer density. It is usually called an Ethernet access or aggregation design, and the switch itself belongs to and is managed by the service provider.
Each customer connection can be placed on a separate VLAN on the switch. The switch then carries those VLANs over a trunk toward the PE. On the PE, a subinterface is created for each customer attachment circuit and assigned to the appropriate VRF. Note that the VLAN-to-VRF association is performed on the PE, not on the switch. The switch identifies and transports the customer’s VLAN, while the PE subinterface terminates that VLAN and places its Layer 3 traffic into the appropriate VRF. Make sense?
I hope this has been helpful!
Laz
1 Like