The remote PC is supposed to have access to the two servers through a site-to-site VPN tunnel. Those two servers are behind a firewall in order to restrict access. They’re NAT’d to IP addresses in the 10.46.132.0 subnet. The PC on the local LAN is able to ping the NAT’d IP addresses of 10.46.132.117 and 10.46.132.178 so I guess that implies the NAT firewall is doing what’s called proxy arp. However, when the remote PC tried to ping these pings failed. The VPN tunnel was not the problem. I added route statements on the local switch (in the 10.46.132.0 subnet) for the two NAT’d IP addresses to go to the firewall. Once I did this the remote PC was able to ping the two servers. Here’s my question. When the remote PC pings, it sees that the servers are on a different subnet so it’ll ARP for its gateway. Once the main firewall received that ping packet, it should see that it’s destined for a different subnet and send it to its next hop which is the layer 3 switch. The layer 3 switch has an interface in the same subnet as the NAT firewall so I would think that the NAT firewall would have responded for the ARP requests for the 10.46.132.117 and 10.46.132.178 addresses but obviously did not since I had to add route statements to the layer 3 switch. My question is why did this ARP process not work here? Why did I need to add route statements.
Hello Mike
Actually, Proxy ARP is not taking place. The local PC and the NAT’d IP addresses of the servers are on the same subnet. So when the local PC pings 10.46.132.117 and 10.46.132.178, no gateway is involved and no routing configuration is necessary. If the PC doesn’t know the MAC address of those particular IP addresses, it will simply send an ARP request, to which the NAT firewall will respond with its corresponding MACs. If the NAT firewall wasn’t performing NAT, and the servers were using their real IP addresses, then yes, Proxy ARP would be used.
Even in this case, Proxy ARP is not taking place, but simple ARP from the L3 switch to the NAT firewall should be taking place, but seems to be failing. In order for this to work, routing must be configured in the Main Firewall, the L3 switch, and the NAT firewall. The switch shouldn’t need any routing information on how to reach the 10.46.132.0/24 subnet which is where the NAT’d server IP addresses are, since this is a directly connected network and InterVLAN routing should be taking place. BUT, the Main Firewall does need routing information to know how to reach that 10.46.132.0/24 subnet. Are you sure that it wasn’t the Main Firewall on which you added the route? IN the same way, the NAT firewall needs to know how to reach the Remote PC’s subnet to respond to the pings.
I hope this has been helpful!
Laz
Laz, thank you for the response. It does help clarify the definition of Proxy ARP as I was clearly using the wrong terminology. Thank you for that. The main firewall did have routing in place to reach the L3 switch as its next hop. However, I had to put a couple of static routes on the L3 switch. The routes were telling the L3 switch that to reach 10.46.132.117 and 10.46.132.178 it has to send the traffic to the outside interface of the NAT firewall. Once I did that it started working. I’m just not clear as to why ARP alone wasn’t enough.
The main firewall receives the traffic, checks its routing table and knows to send traffic for 10.46.132.0 to the L3 switch. The L3 switch knows about 10.46.132.0 so I would have thought that ARP would have taken care of this but obviously I was wrong. I’m missing something.
Hello Mike
I’ve been doing a little more digging and I have found a couple of interesting things that may help you in your troubleshooting. First of all, although I stand by what I told you about proxy ARP in my previous post, it seems that the mechanism that the NAT firewall should use to respond to the ARP requests of the Main firewall when routing traffic to the servers, is indeed called proxy ARP. It’s a special type, since the NAT firewall should respond to such ARP requests providing its own outside interface MAC address for the 10.46.132.117 and .178 outside server IP addresses.
I avoided calling this proxy ARP because the ARP request is for an IP address in the same subnet as the requester (main firewall), but the physical location of the devices are actually in another subnet behind NAT.
Now back to your problem. It seems that the NAT firewall is responding differently to the Local PC’s attempt at communication with the servers, and to the communication coming from the remote PC via the Main Firewall. Without knowing more about your configuration, I suggest you look at the following:
- In order to determine if it is indeed an ARP problem, try removing the routing and adding a static ARP entry in the Main Firewall for these two addresses which point to the outside interface MAC address of the NAT firewall.
- Experiment with the
sysopt noproxyarpcommand found at this Cisco command reference link. This may affect the way the routing from traffic from the VPN is dealt with. This will depend upon your ASA version. - Take a look at this information concerning how mapped addresses are routed and how ARP plays a role in this.
- Finally, it has been stated that in some implementations, it is noted that it may be necessary to add an explicit route for static mappings from the outside to the inside address (as you have done) which will resolve the ARP issue.
Take a look at these and let us know how you get along.
I hope this has been helpful!
Laz
Yes, disabling proxy arp on your NAT rule will resolve the issue. Came across this a while back and after taking a packet Capture realized that the ASA was asking for the IP for the server.
Best regards,
DiegoVargas
Laz, thank you for the response. However, I have to make a correction as perhaps it will change your answer. You stated that “I avoided calling this proxy ARP because the ARP request is for an IP address in the same subnet as the requester (main firewall), but the physical location of the devices are actually in another subnet behind NAT”, but the main firewall is on the 192.168.123.0 subnet so it’s on a different subnet than the local PCs and the NAT addresses, Does this change your answer at all?
Hello Mike
Yes you’re absolutely right, I missed that. So then it is the L3 switch between the main and NAT firewalls that must perform the proxy ARP, and the NAT firewall, in turn, must also perform proxy ARP, as described in the previous post. So yes, I stand corrected, thanks for pointing that out.
In the meantime, thanks to @dihegov for sharing his experience. Try it out and let us know…
I hope this has been helpful!
Laz
My apologies i keep butting in but Laz has the right terminology.
Proxy arp can be as disruptive or effective depending on the issue. In the Asa NAT issue i had presented earlier worked by disabling it. But in another scenario i had to enable it because the firewall was advertising a block thru bgp and peerings were in different subnets as well as a switch in between. Which caused an issue on the return traffic… for some reason although traffic was being initiated by the firewall the router on the other side was asking of who had the arp for that public IP and if so to send to the router. By enabling on the firewall the router was able to send the traffic back.
Hello Diego
Thanks for your input here. I agree with you that proxy ARP can be both a boon and a curse depending on the implementation. I believe the best thing to do here is to experiment, and see what configuration resolves the issue. I believe that @mcpp66’s original solution of adding a route to the NAT firewall for the NAT’ed IPs of the servers is the best because it is cleaner, eliminates the need for proxy ARP, and it makes your configuration intentions clear to your future self or to another administrator that may be looking at the configs down the road.
But I believe that the big question here that was originally posed was why does it behave like that? Sometimes small behaviours like this change depending on the version or even the vendor of the firewall being used. Experimentation will verify which behaviour is to be expected under which circumstances.
No problem in you “butting in”, it makes the whole discussion more valuable for all our readers. Sharing your experiences is part of the value of the forum!
I hope this has been helpful!
Laz
Sorry it took so long for me to respond. Unfortunately I won’t be able to make any changes to test your reply above. This is in production now so I have to leave it as is since it’s working. But next time I’ll have something more to go on. Thank you everyone for your input!