Abnormal Traceroute with Multiple Interfaces under One VRF

Routing & Switching SD-WAN
1

Hi All,

Good day …

I am facing weird issue for which I would like to open a discussion here and want to take your comments/views/feedback …

I have Topology of type -

CE1–R1–R2–CE2

@R2: I have created following two interfaces 1/2/5.410 with connected network 10.32.181.64/30; 1/2/5.520 with connected network 10.32.177.64/30 …
.65 is configured on R2 and .66 is configured on CE2 at respective sub i/f …
I am running mBGP b/w R1-R2 (OSPF for underlying IPv4 reachability of loopback, LDP for MPLS label distribution) and above mentioned sub-i/f 1/2/5.410 and 1/2/5.520 are part of VRF “aa” @ R2 …
@R2, I have performed Redistribute connected under the VRF “aa” and R1 can successfully learn the corresponding VPNv4 routes as advertised by R2 …

(b/w CE1-R1 , I are running Inter-AS-option-A, eBGP binding per VRF and I am using same VRF “aa” details as configured @R2 i.e. RD=RTi/e @R1 = @R2 )

Now, Issue I see is as below -
A) while doing Trace route from R1 or CE1 for 10.32.181.65 under VRF “aa”, i see as expected response i.e.

traceroute to 10.32.181.65 (10.32.181.65), 255 hops max, 38 byte packets
 1  10.32.181.65  1.252 ms  0.972 ms  1.043 ms

B) However, while doing Trace route from R1 or CE1 for 10.32.181.66 under VRF “aa”, i see an unexpected response i.e.

traceroute to 10.32.181.66 (10.32.181.66), 255 hops max, 38 byte packets
 1  10.32.177.65  1.186 ms  0.948 ms  0.835 ms
 2  10.32.181.66  1.210 ms  1.032 ms  1.130 ms

You see in case-B above, TR is not landing @ 10.32.181.65 at hop1, instead it lands @ 10.32.177.65 …

on the other hand, if I do a TR for 10.32.177.65 or .66, it ends up correctly as expected via 10.32.177.65 @ hop1 …

so, why it is so, and how to fix this …

Best Regards!
Ashish JAIN

2

Hello Ashish

Thanks for the detailed explanation of your topology and configuration. It seems that the traceroute from R1 or CE1 to the IP address of the 1/2/5.410 subinterface on R2 is responding as expected, but when you traceroute to the IP address of the 1/2/5.410 subinterface on CE2, it goes via the other subinterface of R2 1/2/5.520 as an intermediate hop. The interesting thing is that both of these destinations are in the same subnet, and should logically be routed the same way, but they’re not.

The reason for this routing path is not immediately perceivable and will require a bit of troubleshooting. Because you’re involving several different technologies it may need a more trial-and-error approach. Here are a few possible causes for this behavior, and some things you can consider examining in your troubleshooting that may help:

Possible Causes:

  • Route Advertisement: The way routes are advertised in BGP might be causing the issue. It’s possible that the route to 10.32.181.66 is being advertised incorrectly or is being preferred over the direct route due to BGP attributes.
  • Route Redistribution: The redistribution configuration might be causing unexpected routes to be preferred. Check if the redistribution policy is correctly applied.
  • Point-to-point links - Sometimes routing protocols perceive some destinations as /32 (host) routes and may route them differently than the rest of the subnet. This is why you may see the specific IP address being routed differently.

Suggested Troubleshooting Steps:

  • Review BGP Advertisement: Check the BGP advertisements and paths for any anomalies or misconfigurations, particularly for the route to 10.32.181.66.
  • Examine Routing Tables: Look at the routing tables on R1, R2, CE1, and CE2 to see how they are learning and preferring routes. Especially check if there are any /32 host routes that may be causing this different behavior for particular addresses.
  • Check Redistribution Policy: Make sure that the route redistribution policy under VRF “aa” is correctly prioritizing routes.
  • Validate MPLS Configuration: Since LDP is used for MPLS label distribution, ensure that MPLS configurations are correctly set up and there are no issues with label distribution or path selection.

Hopefully, this will give you some insight into how to approach the next steps of your troubleshooting process. Let us know how you get along, and if you need any more help along the way…

I hope this has been helpful!

Laz