Hello Jonas
This is an interesting scenario, and thank you for your thorough description and diagram!
I believe that this situation is due to the violation of a rule described in RFC 2328. Specifically, in section 16.4. Calculating AS external routes, it says:
If the forwarding address is non-zero, look up the
forwarding address in the routing table. The matching
routing table entry must specify an intra-area or inter-area
path; if no such path exists, do nothing with the LSA and
consider the next in the list.
So it all comes down to the forwarding address. What is that? The forwarding address (FA) is a field in a Type 5 External LSAs and Type 7 NSSA External LSAs that instructs routers to send traffic directly to a particular next-hop IP address instead of sending it through the ASBR that originated the LSA. The statement above is saying:
- If the FA is zero (0.0.0.0), then install the route in the LSA, assuming you can reach the advertising router.
- If the FA is non-zero, look up the FA in the local routing table. Install the related route in the LSA only if the FA can be resolved to a normal non-external route.
In R3, the second situation seems to be taking place. When R7, an internal ASBR within the totally NSSA area, sends out a Type 7 LSA, it carries a non-zero forwarding address, and this will typically be R7’s own interface address INSIDE the NSSA. When R5 or R6 translates this Type 7 to a Type 5, they preserve the non-zero FA unchanged. When R3 receives the LSA, to install the external route, it must have a route to the FA, which is the interface of R7.
It seems that the virtual link in combination with this non-zero FA is “breaking” this feature. Either the transit area isn’t propagating the needed summary-LSA for R7’s FA subnet, or R3’s path to that FA ends up depending on the virtual link itself, which can become circular/unresolvable. The result: R3 sees the Type-5 in its LSDB but cannot resolve the FA, so per RFC it refuses to install the route.
You can check this out by looking at the FA field in the LSA that arrives at R3.
How do you fix it? Well, the most direct fix is to suppress the forwarding address when the NSSA ABR translates the Type 7 LSA into a Type 5 LSA. You can do this with this command:
router ospf 1
area 5 nssa translate type7 suppress-fa
More about FA suppresion in Cisco devices can be found here:
This command must be configured on the NSSA ABR performing the Type 7-to-Type 5 translation, that is R6 and R5. This is the most direct solution, not necessarily the best one. The combination of an NSSA area and a virtual link can be cumbersome and have some unexpected results, even if you suppress this feature. In any case, it helps to understand the behavior of OSPF in various circumstances.
I hope this has been helpful!
Laz