[strongSwan] SA established while no traffic

[Volodymyr Litovka]

Hello colleagues,

I'm facing a problem of routing between hosts over established SA.

[Ubuntu 18.04/Strongswan 5.6.2] <--> [PAT/Cisco] <--> [OSX 10.12.6]

OSX is client, Ubuntu is server and they're able to establish connection:

Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4> CHILD_SA
pinocchio{5} established with SPIs c61c704d_i 00121be9_o and TS
25.1.1.0/24 === 192.168.1.1/32

# swanctl --list-sas
ikev2-eap-mschapv2: #2, ESTABLISHED, IKEv2, bd64dfef8156901f_i
22426a0bf50a3047_r*
  local  'server' @ X.X.X.252[4500]
  remote 'doka' @ Y.Y.Y.194[4500] [192.168.1.1]
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 217s ago, rekeying in 10083s
  pinocchio: #2, reqid 2, INSTALLED, TUNNEL-in-UDP,
ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 217s ago, rekeying in 3165s, expires in 3743s
    in  c94f7412 (0x00000003),      0 bytes,     0 packets
    out 0f4f18d1 (0x00000004),      0 bytes,     0 packets
    local  25.1.1.0/24
    remote 192.168.1.1/32

Routing table on client side (OXS) looks correct:
25.1.1/24          192.168.1.1        UGSc            0 1  ipsec0
192.168.1.1        192.168.1.1        UH 1        0  ipsec0

Routing configuration on server side (Ubuntu):
***@vpn:~# ip rule show
0:    from all lookup local
220:    from all lookup 220
32766:    from all lookup main
32767:    from all lookup default

***@vpn:~# ip route show table 220
192.168.1.1 via X.X.X.254 dev wan0 proto static src 25.1.1.1

***@vpn:~# ip route show table 0
192.168.1.1 via X.X.X.254 dev wan0 table 220 proto static src 25.1.1.1
default via X.X.X.254 dev wan0 proto static

***@vpn:/etc# ip route
default via X.X.X.254 dev wan0 proto static
25.1.1.0/24 dev wan0 proto kernel scope link src 25.1.1.1

and local address is up and accessible:

***@vpn:/etc# ping 25.1.1.1
PING 25.1.1.1 (25.1.1.1) 56(84) bytes of data.
64 bytes from 25.1.1.1: icmp_seq=1 ttl=64 time=0.024 ms

Debug shows the following:

Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 in (mark 5/0xffffffff) [priority
371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 fwd (mark 5/0xffffffff) [priority
371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 25.1.1.0/24 === 192.168.1.1/32 out (mark 6/0xffffffff) [priority
371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting a
local address in traffic selector 25.1.1.0/24
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
host 25.1.1.1
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting
iface name for index 2
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
X.X.X.254 as nexthop and wan0 as dev to reach Y.Y.Y.194/32
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
installing route: 192.168.1.1/32 via X.X.X.254 src 25.1.1.1 dev wan0
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting
iface index for wan0
Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4> CHILD_SA
pinocchio{5} established with SPIs c61c704d_i 00121be9_o and TS
25.1.1.0/24 === 192.168.1.1/32
Sep 25 15:43:27 vpn strongswan: 04[CHD] <ikev2-eap-mschapv2|4> CHILD_SA
pinocchio{5} state change: INSTALLING => INSTALLED

on server side there is neither NAT nor other iptable rules, it's
accessible from Internet from everywhere. Also, ipv4 forwarding is on
(net.ipv4.ip_forward=1 @ sysctl.conf)

but no traffic (ping -I 25.1.1.1 192.168.1.1) between these addresses

swanctl configuration:

connections {
        ikev2-eap-mschapv2 {
                version = 2
                local_addrs = X.X.X.252
                remote_addrs = %any
                proposals =
aes256gcm16-prfsha256-modp2048,aes128gcm16-prfaescmac-modp2048,aes256-sha256-prfsha256-modp2048
                encap = yes
                fragmentation = yes
                mobike = yes(NOTE: tried NO as well)
                dpd_delay = 300s
                send_cert = always
                unique = keep
                rekey_time = 3h
                pools = mypool
                local-1 {
                        certs = fullchain.pem
                        id = @myfqdn
                }
                remote-1 {
                        id = %any
                        auth = eap-mschapv2
                }
                children {
                        pinocchio {
                                ah_proposals =
                                esp_proposals =
aes256gcm16-modp2048,aes128gcm16-modp2048,aes256-sha256
                                local_ts = 25.1.1.0/24
                                remote_ts = dynamic
                                mode = tunnel
                                dpd_action = clear
                                ipcomp = no
                                mark_in = %unique-dir
                                mark_out = %unique-dir
                                tfc_padding = mtu
                                hw_offload = yes (NOTE: tried NO as well)
                                }
                        }
        }
}
secrets {
        include conf.d/secret-*.conf
}

pools {
     mypool {
        addrs = 192.168.1.0/24
     }
}

Actually, I don't understand where to look into problem. Can anybody
suggest the way I need to follow in order to narrow and fix the problem?

Thanks!

[Volodymyr Litovka]

And one more debug info - when I'm pinging server from client, I see
packets arrived on server's ingress interface:

08:15:24.352861 IP client.4500 > server.4500: UDP-encap:
ESP(spi=0xc5f7be04,seq=0x2e), length 104
08:15:24.456173 IP client.4500 > server.4500: UDP-encap:
ESP(spi=0xc5f7be04,seq=0x2f), length 120
08:15:24.620758 IP client.4500 > server.4500: UDP-encap:
ESP(spi=0xc5f7be04,seq=0x30), length 120
08:15:24.795823 IP client.4500 > server.4500: UDP-encap:
ESP(spi=0xc5f7be04,seq=0x31), length 104

so traffic selectors agreed and at least work on client side, but
nothing in reply from server. I can't even imagine where to look into
problem, so any suggestions are highly welcome.

Thank you.

Post by Volodymyr Litovka
Hello colleagues,
I'm facing a problem of routing between hosts over established SA.
[Ubuntu 18.04/Strongswan 5.6.2] <--> [PAT/Cisco] <--> [OSX 10.12.6]
Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} established with SPIs c61c704d_i 00121be9_o and
TS 25.1.1.0/24 === 192.168.1.1/32
# swanctl --list-sas
ikev2-eap-mschapv2: #2, ESTABLISHED, IKEv2, bd64dfef8156901f_i
22426a0bf50a3047_r*
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 217s ago, rekeying in 10083s
  pinocchio: #2, reqid 2, INSTALLED, TUNNEL-in-UDP,
ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 217s ago, rekeying in 3165s, expires in 3743s
    in  c94f7412 (0x00000003),      0 bytes,     0 packets
    out 0f4f18d1 (0x00000004),      0 bytes,     0 packets
    local  25.1.1.0/24
    remote 192.168.1.1/32
25.1.1/24          192.168.1.1        UGSc            0 1  ipsec0
192.168.1.1        192.168.1.1        UH 1        0  ipsec0
0:    from all lookup local
220:    from all lookup 220
32766:    from all lookup main
32767:    from all lookup default
192.168.1.1 via X.X.X.254 dev wan0 proto static src 25.1.1.1
192.168.1.1 via X.X.X.254 dev wan0 table 220 proto static src 25.1.1.1
default via X.X.X.254 dev wan0 proto static
default via X.X.X.254 dev wan0 proto static
25.1.1.0/24 dev wan0 proto kernel scope link src 25.1.1.1
PING 25.1.1.1 (25.1.1.1) 56(84) bytes of data.
64 bytes from 25.1.1.1: icmp_seq=1 ttl=64 time=0.024 ms
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 in (mark 5/0xffffffff) [priority
371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 fwd (mark 5/0xffffffff)
[priority 371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 25.1.1.0/24 === 192.168.1.1/32 out (mark 6/0xffffffff)
[priority 371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting
a local address in traffic selector 25.1.1.0/24
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
host 25.1.1.1
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting
iface name for index 2
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
X.X.X.254 as nexthop and wan0 as dev to reach Y.Y.Y.194/32
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
installing route: 192.168.1.1/32 via X.X.X.254 src 25.1.1.1 dev wan0
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> getting
iface index for wan0
Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} established with SPIs c61c704d_i 00121be9_o and
TS 25.1.1.0/24 === 192.168.1.1/32
Sep 25 15:43:27 vpn strongswan: 04[CHD] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} state change: INSTALLING => INSTALLED
on server side there is neither NAT nor other iptable rules, it's
accessible from Internet from everywhere. Also, ipv4 forwarding is on
but no traffic (ping -I 25.1.1.1 192.168.1.1) between these addresses
connections {
        ikev2-eap-mschapv2 {
                version = 2
                local_addrs = X.X.X.252
                remote_addrs = %any
                proposals =
aes256gcm16-prfsha256-modp2048,aes128gcm16-prfaescmac-modp2048,aes256-sha256-prfsha256-modp2048
                encap = yes
                fragmentation = yes
                mobike = yes(NOTE: tried NO as well)
                dpd_delay = 300s
                send_cert = always
                unique = keep
                rekey_time = 3h
                pools = mypool
                local-1 {
                        certs = fullchain.pem
                }
                remote-1 {
                        id = %any
                        auth = eap-mschapv2
                }
                children {
                        pinocchio {
                                ah_proposals =
                                esp_proposals =
aes256gcm16-modp2048,aes128gcm16-modp2048,aes256-sha256
                                local_ts = 25.1.1.0/24
                                remote_ts = dynamic
                                mode = tunnel
                                dpd_action = clear
                                ipcomp = no
                                mark_in = %unique-dir
                                mark_out = %unique-dir
                                tfc_padding = mtu
                                hw_offload = yes (NOTE: tried NO as well)
                                }
                        }
        }
}
secrets {
        include conf.d/secret-*.conf
}
pools {
     mypool {
        addrs = 192.168.1.0/24
     }
}
Actually, I don't understand where to look into problem. Can anybody
suggest the way I need to follow in order to narrow and fix the problem?
Thanks!

[Volodymyr Litovka]

The issue was in mark_in/mark_out directives. Since commented,
everything works.

Will spend time on netfilter later.

Thank you.

Post by Volodymyr Litovka
And one more debug info - when I'm pinging server from client, I see
ESP(spi=0xc5f7be04,seq=0x2e), length 104
ESP(spi=0xc5f7be04,seq=0x2f), length 120
ESP(spi=0xc5f7be04,seq=0x30), length 120
ESP(spi=0xc5f7be04,seq=0x31), length 104
so traffic selectors agreed and at least work on client side, but
nothing in reply from server. I can't even imagine where to look into
problem, so any suggestions are highly welcome.
Thank you.

Post by Volodymyr Litovka
Hello colleagues,
I'm facing a problem of routing between hosts over established SA.
[Ubuntu 18.04/Strongswan 5.6.2] <--> [PAT/Cisco] <--> [OSX 10.12.6]
OSX is client, Ubuntu is server and they're able to establish
Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} established with SPIs c61c704d_i 00121be9_o and
TS 25.1.1.0/24 === 192.168.1.1/32
# swanctl --list-sas
ikev2-eap-mschapv2: #2, ESTABLISHED, IKEv2, bd64dfef8156901f_i
22426a0bf50a3047_r*
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 217s ago, rekeying in 10083s
  pinocchio: #2, reqid 2, INSTALLED, TUNNEL-in-UDP,
ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 217s ago, rekeying in 3165s, expires in 3743s
    in  c94f7412 (0x00000003),      0 bytes,     0 packets
    out 0f4f18d1 (0x00000004),      0 bytes,     0 packets
    local  25.1.1.0/24
    remote 192.168.1.1/32
25.1.1/24          192.168.1.1        UGSc            0 1 ipsec0
192.168.1.1        192.168.1.1        UH 1        0  ipsec0
0:    from all lookup local
220:    from all lookup 220
32766:    from all lookup main
32767:    from all lookup default
192.168.1.1 via X.X.X.254 dev wan0 proto static src 25.1.1.1
192.168.1.1 via X.X.X.254 dev wan0 table 220 proto static src 25.1.1.1
default via X.X.X.254 dev wan0 proto static
default via X.X.X.254 dev wan0 proto static
25.1.1.0/24 dev wan0 proto kernel scope link src 25.1.1.1
PING 25.1.1.1 (25.1.1.1) 56(84) bytes of data.
64 bytes from 25.1.1.1: icmp_seq=1 ttl=64 time=0.024 ms
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 in (mark 5/0xffffffff)
[priority 371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 192.168.1.1/32 === 25.1.1.0/24 fwd (mark 5/0xffffffff)
[priority 371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> adding
policy 25.1.1.0/24 === 192.168.1.1/32 out (mark 6/0xffffffff)
[priority 371327, refcount 1]
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
getting a local address in traffic selector 25.1.1.0/24
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
host 25.1.1.1
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
getting iface name for index 2
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4> using
X.X.X.254 as nexthop and wan0 as dev to reach Y.Y.Y.194/32
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
installing route: 192.168.1.1/32 via X.X.X.254 src 25.1.1.1 dev wan0
Sep 25 15:43:27 vpn strongswan: 04[KNL] <ikev2-eap-mschapv2|4>
getting iface index for wan0
Sep 25 15:43:27 vpn strongswan: 04[IKE] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} established with SPIs c61c704d_i 00121be9_o and
TS 25.1.1.0/24 === 192.168.1.1/32
Sep 25 15:43:27 vpn strongswan: 04[CHD] <ikev2-eap-mschapv2|4>
CHILD_SA pinocchio{5} state change: INSTALLING => INSTALLED
on server side there is neither NAT nor other iptable rules, it's
accessible from Internet from everywhere. Also, ipv4 forwarding is on
but no traffic (ping -I 25.1.1.1 192.168.1.1) between these addresses
connections {
        ikev2-eap-mschapv2 {
                version = 2
                local_addrs = X.X.X.252
                remote_addrs = %any
                proposals =
aes256gcm16-prfsha256-modp2048,aes128gcm16-prfaescmac-modp2048,aes256-sha256-prfsha256-modp2048
                encap = yes
                fragmentation = yes
                mobike = yes(NOTE: tried NO as well)
                dpd_delay = 300s
                send_cert = always
                unique = keep
                rekey_time = 3h
                pools = mypool
                local-1 {
                        certs = fullchain.pem
                }
                remote-1 {
                        id = %any
                        auth = eap-mschapv2
                }
                children {
                        pinocchio {
                                ah_proposals =
                                esp_proposals =
aes256gcm16-modp2048,aes128gcm16-modp2048,aes256-sha256
                                local_ts = 25.1.1.0/24
                                remote_ts = dynamic
                                mode = tunnel
                                dpd_action = clear
                                ipcomp = no
                                mark_in = %unique-dir
                                mark_out = %unique-dir
                                tfc_padding = mtu
                                hw_offload = yes (NOTE: tried NO as
well)
                                }
                        }
        }
}
secrets {
        include conf.d/secret-*.conf
}
pools {
     mypool {
        addrs = 192.168.1.0/24
     }
}
Actually, I don't understand where to look into problem. Can anybody
suggest the way I need to follow in order to narrow and fix the problem?
Thanks!