DMVPN - Spoke behind ASA FW Can't ping tunnel interface

[stiroman (Roman V)]

Hi Guys,

So finally got time to play with the setup and currently running in to a issue. The quick problem description will be that I can’t ping the far end tunnel how ever the weird stuff is that there is some sort of connection or communication going on based on these outputs bellow. Also I’m able to ping outside IP of the HUB that is directly facing Internet. It was not working at all first but then I configured another interface on ASA with static NAT and that when ipsec start communicating and also traffic switched form being stuck on udp/500 to udp/4500 which is I believe expected with the NAT

So first this is the topology. Mine setup is the one behind FW the other spoke works just fine.

Tunnel 192.168.221.3 (SPOKE) CSRv -----> ASAv ------> INTERNET -------> CSRv (HUB) Tunnel 192.168.221.1

CSR1000v-R1#sh crypto ipsec sa

interface: Tunnel666
    Crypto map tag: Tunnel666-head-0, local addr 10.0.80.2

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (10.0.80.2/255.255.255.255/47/0)
   remote ident (addr/mask/prot/port): (65.x.x.x/255.255.255.255/47/0)
   current_peer 65.x.x.x port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 2556, #pkts encrypt: 2556, #pkts digest: 2556 << --------
    #pkts decaps: 816, #pkts decrypt: 816, #pkts verify: 816 << --------------
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 10.0.80.2, remote crypto endpt.: 65.x.x.x
     plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet1
     current outbound spi: 0xC891BA8B(3364993675)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x4FD5738B(1339388811)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Transport UDP-Encaps, }
        conn id: 2247, flow_id: CSR:247, sibling_flags FFFFFFFF80000008, crypto map: Tunnel666-head-0
        sa timing: remaining key lifetime (k/sec): (4607993/611)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0xC891BA8B(3364993675)
        transform: esp-3des esp-md5-hmac ,
        in use settings ={Transport UDP-Encaps, }
        conn id: 2248, flow_id: CSR:248, sibling_flags FFFFFFFF80000008, crypto map: Tunnel666-head-0
        sa timing: remaining key lifetime (k/sec): (4607987/611)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:
CSR1000v-R1#

CSR1000v-R1#ping 192.168.221.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.221.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
CSR1000v-R1#ping 192.168.221.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.221.2, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
CSR1000v-R1#sh crypto isa
CSR1000v-R1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
10.0.80.2       65.x.x.x  QM_IDLE           1065 ACTIVE

IPv6 Crypto ISAKMP SA

CSR1000v-R1#sh ip nhrp
192.168.221.1/32 via 192.168.221.1
   Tunnel666 created 15:53:05, never expire
   Type: static, Flags:
   NBMA address: 65.x.x.x
192.168.221.2/32
   Tunnel666 created 00:00:46, expire 00:02:18
   Type: incomplete, Flags: negative
   Cache hits: 6
CSR1000v-R1#

And snip from debug...


Aug 20 17:29:53.534: IPSEC(create_sa): sa created,
  (sa) sa_dest= 10.0.80.2, sa_proto= 50,
    sa_spi= 0xE8291530(3895006512),
    sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2231
    sa_lifetime(k/sec)= (4608000/900),
  (identity) local= 10.0.80.2:0, remote= 65.x.x.x:0,
    local_proxy= 10.0.80.2/255.255.255.255/47/0,
    remote_proxy= 65.x.x.x/255.255.255.255/47/0
Aug 20 17:29:53.534: IPSEC(create_sa): sa created,
  (sa) sa_dest= 65.x.x.x, sa_proto= 50,
    sa_spi= 0x84FAF211(2231038481),
    sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2232
    sa_lifetime(k/sec)= (4608000/900),
  (identity) local= 10.0.80.2:0, remote= 65.x.x.x:0,
    local_proxy= 10.0.80.2/255.255.255.255/47/0,
    remote_proxy= 65.x.x.x/255.255.255.255/47/0
Aug 20 17:29:53.556: ISAKMP-ERROR: (0):Failed to find peer index node to update peer_info_list
Aug 20 17:29:53.556: ISAKMP: (1061):Received IPSec Install callback... proceeding with the negotiation
Aug 20 17:29:53.556: ISAKMP: (1061):Successfully installed IPSEC SA (SPI:0xE8291530) on Tunnel666
Aug 20 17:29:53.556: ISAKMP-PAK: (1061):sending packet to 65.x.x.x my_port 4500 peer_port 4500 (R) QM_IDLE
Aug 20 17:29:53.556: ISAKMP: (1061):Sending an IKE IPv4 Packet.
Aug 20 17:29:53.556: ISAKMP: (1061):Node 255717431, Input = IKE_MESG_FROM_IPSEC, IPSEC_INSTALL_DONE
Aug 20 17:29:53.556: ISAKMP: (1061):Old State = IKE_QM_IPSEC_INSTALL_AWAIT  New State = IKE_QM_R_QM2
Aug 20 17:29:53.782: ISAKMP-PAK: (1061):received packet from 65.x.x.x dport 4500 sport 4500 Global (R) QM_IDLE
Aug 20 17:29:53.782: ISAKMP: (1061):deleting node 255717431 error FALSE reason "QM done (await)"
Aug 20 17:29:53.782: ISAKMP: (1061):Node 255717431, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
Aug 20 17:29:53.783: ISAKMP: (1061):Old State = IKE_QM_R_QM2  New State = IKE_QM_PHASE2_COMPLETE
Aug 20 17:29:53.783: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Aug 20 17:29:53.783: IPSEC(key_engine_enable_outbound): rec'd enable notify from ISAKMP
Aug 20 17:29:53.783: IPSEC(update_current_outbound_sa): updated peer 65.x.x.x current outbound sa to SPI 84FAF211
Aug 20 17:30:13.782: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:30:33.783: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:30:40.951: %DUAL-5-NBRCHANGE: EIGRP-IPv4 200: Neighbor 192.168.221.1 (Tunnel666) is down: retry limit exceeded
Aug 20 17:30:42.944: %DUAL-5-NBRCHANGE: EIGRP-IPv4 200: Neighbor 192.168.221.1 (Tunnel666) is up: new adjacency
Aug 20 17:30:43.782: ISAKMP: (1061):purging node 255717431
Aug 20 17:30:53.783: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:30:53.866: ISAKMP-PAK: (1061):received packet from 65.x.x.x dport 4500 sport 4500 Global (R) QM_IDLE
Aug 20 17:30:53.866: ISAKMP: (1061):set new node 3640371719 to QM_IDLE
Aug 20 17:30:53.867: ISAKMP: (1061):processing HASH payload. message ID = 3640371719
Aug 20 17:30:53.867: ISAKMP: (1061):processing DELETE payload. message ID = 3640371719
Aug 20 17:30:53.867: ISAKMP: (1061):peer does not do paranoid keepalives.
Aug 20 17:30:53.867: ISAKMP: (1061):Enqueued KEY_MGR_DELETE_SAS for IPSEC SA (SPI:0x6626A8AB)
Aug 20 17:30:53.867: ISAKMP: (1061):deleting node 3640371719 error FALSE reason "Informational (in) state 1"
Aug 20 17:30:53.867: IPSEC(key_engine): got a queue event with 1 KMI message(s)
Aug 20 17:30:53.867: IDB is NULL : in crypto_ipsec_key_engine_delete_sas (), 5732
Aug 20 17:30:53.867: IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP
Aug 20 17:30:53.867: IPSEC: still in use sa: 0x7F73F3856A50
Aug 20 17:30:53.867: IPSEC(key_engine_delete_sas): delete SA with spi 0x6626A8AB proto 50 for 65.x.x.x
Aug 20 17:30:53.867: IPSEC(delete_sa): deleting SA,
  (sa) sa_dest= 10.0.80.2, sa_proto= 50,
    sa_spi= 0x3865CF33(946196275),
    sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2229
    sa_lifetime(k/sec)= (4608000/900),
  (identity) local= 10.0.80.2:0, remote= 65.x.x.x:0,
    local_proxy= 10.0.80.2/255.255.255.255/47/0,
    remote_proxy= 65.x.x.x/255.255.255.255/47/0
Aug 20 17:30:53.867: IPSEC(delete_sa): deleting SA,
  (sa) sa_dest= 65.x.x.x, sa_proto= 50,
    sa_spi= 0x6626A8AB(1713809579),
    sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2230
    sa_lifetime(k/sec)= (4608000/900),
  (identity) local= 10.0.80.2:0, remote= 65.x.x.x:0,
    local_proxy= 10.0.80.2/255.255.255.255/47/0,
    remote_proxy= 65.x.x.x/255.255.255.255/47/0
Aug 20 17:30:53.868: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
Aug 20 17:30:53.872: IPSEC(ident_delete_notify_kmi): Failed to send KEY_ENG_DELETE_SAS
Aug 20 17:30:53.872: IPSEC(ident_update_final_flow_stats): Collect Final Stats and update MIB
IPSEC get IKMP peer index from peer 0x7F73F3854610 ikmp handle 0x80000006
IPSEC IKMP peer index 0
[ident_update_final_flow_stats] : Flow delete complete event received for flow id 0x240000E5,peer index 0

Aug 20 17:31:13.866: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:31:33.867: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:31:43.867: ISAKMP: (1061):purging node 3640371719
Aug 20 17:31:53.868: ISAKMP: (0):Sending an IKE IPv4 Packet.
Aug 20 17:32:02.453: %DUAL-5-NBRCHANGE: EIGRP-IPv4 200: Neighbor 192.168.221.1 (Tunnel666) is down: retry limit exceeded << ------
Aug 20 17:32:06.366: %DUAL-5-NBRCHANGE: EIGRP-IPv4 200: Neighbor 192.168.221.1 (Tunnel666) is up: new adjacency << ---------

[lagapidis (Lazarus Agapidis)]

Hello Roman

Implementing a DMVPN scenario with an ASA firewall and NATting in the mix can become somewhat complicated. Take a look at this post which deals with the various alternative scenarios, and which also provides a link to relevant Cisco documentation that will shed some light on your issue…

DMVPN Phase 3 Basic Configuration

Hello Roman Typically, when running DMVPN behind an ASA firewall, there are several options, two of which suit your situation. The first involves placing the DMVPN router in the DMZ of your ASA, and assigning it a public address, which means you can filter traffic, but you don’t actually have to open specific ports. The other involves having the DMVPN router behind the firewall, in which case you will need to open/forward some ports. From my understanding, it is the second scenario that you …

I hope this has been helpful!

Laz

[stiroman (Roman V)]

Hi Lazaros,

Correct, the 2nd scenario is what I’m trying to do. This is just something I try to built up with my friends, we already have one spoke up with the HUB, but that spoke is placed at the edge same as the hub. In another word it has public IP assigned to the “outside” interface. In my case the FW is edge device performing NATing and routing and the router sits behind the FW. FW is allowing ports 500, 4500 and GRE and ESP. From packet capture done on FW I can see the traffic leaving and returning between the public IP’s of both Spoke and Hub on port 4500 and on CSR I can see encap/dcaps under “show crypto ipsec sa” However I can’t ping the hub tunnel interface IP.

[lagapidis (Lazarus Agapidis)]

Hello Roman

@stiroman

From packet capture done on FW I can see the traffic leaving and returning between the public IP’s of both Spoke and Hub on port 4500 and on CSR I can see encap/dcaps under “show crypto ipsec sa” However I can’t ping the hub tunnel interface IP.

So from what I understand, you are actually seeing traffic successfully exchanged between HUB and Spoke through the firewall but are simply not able to ping between these “remote” locations? Except for ICMP, are you able to see regular end to end traffic, say Telnet or Web between the “remote” sites? Since you are seeing encap/decaps, it looks like some traffic is getting through. I suggest you try to transmit traffic from some host behind the hub to some other host behind the spoke (not ICMP traffic) and see if you have connectivity. Maybe the ICMP traffic is being blocked by default somewhere along your path.

I hope this helps in your troubleshooting endeavours!

Laz