I’ve read a lot that OSPF can’t work through IP Sec tunnel but I can’t find any explanation why. Can anybody explain the root cause?
P.S. Under IP sec tunnel I mean pure IP Sec configured in tunnel mode, w/o GRE, etc.
OSPF in its default configuration, will not work through a pure IPsec tunnel (without GRE etc). The reason for this is because OSPF uses multicast traffic to communicate between routers and a pure IPsec tunnel will not support multicast. This is why GRE is often used in conjunction with IPsec for OSPF (and other) implementations, because it is capable of transmitting multicast traffic.
Another solution to implementing OSPF over an IPsec tunnel that doesn’t support multicast is to use the implementation that is used for OSPF over non-broadcast links such as Frame Relay, X.25 and ATM. This will essentially allow OSPF messages to be “unicasted” to neighbours solving the problem of sending multicast traffic over the IPsec tunnel. More info about how to configure OSPF to work over non-broadcast links can be found at the following Cisco Documentation:
Now the interesting thing is that there is a bug in some Cisco IOS versions that incorrectly allow OSPF multicast traffic to be sent over a pure IPsec tunnel even though this should not be the case. For those that have access, this is the bug info:
For those that don’t, the bug text is the following:
This is a documentation bug only. Symptom: A note needs to be added into configuration guide to specify that: As of release 12.4(9)T multicast traffic originated from the box will be encapsulated into IPsec if proxy identities allow this. Further description A typical use case for this is when router is sourcing OSPF packets and traffic selectors for IPsec allows OSPF packets (protocol number 89, group 188.8.131.52 & 184.108.40.206). As of release 12.4(9)T those packets will be put into the tunnel and encrypted. At the same time, please be aware that using "any any" as your proxy identities is HIGHLY discouraged. "any any" proxy identities can be achieved in case of using VTI configuration which is recommended if those proxy identities are desired.
Although this is a bug in this particular IOS version, typical behaviour is to NOT allow multicast traffic over a purely IPsec tunnel.
I hope this has been helpful!