[strongSwan] Problem with MTU in IPSec Transport Mode

[Jan-Philipp Hülshoff]

Hello,
I have the following setup:

A private network "home" with a router "router" doing NAT with one
static public ip. The router establishes a secure IPSec transport from
his public ip to a server "server" on the internet (same as in example [1]).

The following problem can be observed:
Packets sent from inside the home network that are larger than some
value and have the don't fragment bit set get lost somewhere in the
router. No encrypted packets are then sent to the server. The router
does not send ICMP host unreachable fragmentation needed messages back
to the client in the home network. (I suspect that the resulting
encrypted packet will be larger than the mtu of the outgoing interface.)

for example this affects ssh from inside the home network to server.


Is this behaviour intended? Is this use case supported or is it an
unusual way to use ipsec transport mode in combination with NAT/routing?


Here are the tests that demonstrate the behaviour:
client-in-home# ping -s 1450 -M do server
PING server (IP) 1450(1478) bytes of data.
^C
--- server ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

As you can see here, the ping packets get lost. I can confirm that the
router does not send any packets to the server.

client-in-home# ping server -s 1430 -M do
PING server (IP) 1430(1458) bytes of data.
1438 bytes from server (IP): icmp_seq=1 ttl=50 time=17.8 ms
1438 bytes from server (IP): icmp_seq=2 ttl=50 time=16.2 ms
1438 bytes from server (IP): icmp_seq=3 ttl=50 time=16.3 ms
1438 bytes from server (IP): icmp_seq=4 ttl=50 time=18.2 ms
^C
--- server ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 16.205/17.146/18.236/0.912 ms

But sending smaller packets works without problems. On the router the
same ping command works as expected and gives the following output:

router# ping -s 1450 -M do server
PING server (IP) 1450(1478) bytes of data.
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long
^C
--- server ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2005ms

As you can see the router does not send an icmp fragmentation needed
message to client-in-home.
I can confirm this behaviour with another installation with 3 vm's. I
used Debian GNU/Linux jessie (kernel 3.16.0-4-amd64, strongswan
5.2.1-6+deb8).

Kind regards,
Jan-Philipp Hülshoff

[1]:
https://www.strongswan.org/testing/testresults/ikev2/host2host-transport/

[Noel Kuntze]

(I suspect that the resulting encrypted packet will be larger than the mtu of the outgoing interface.)

Yes.

Is this behaviour intended? Is this use case supported or is it an unusual way to use ipsec transport mode in combination with NAT/routing?

Set charon.plugins.kernel-netlink.mtu to 1400 or lower. This causes creation with set MTU sizes, which causes the kernel to send ICMP errors when
packets are supposed to be routed using that route, but they're too large. You need to take the IPsec overhead into account. This setting
DOES NOT work when the sender of the IP packet drops ICMP errors (Instagram and the Twitter image servers drop ICMP errors).

[Jan-Philipp Hülshoff]

Post by Noel Kuntze

(I suspect that the resulting encrypted packet will be larger than

the mtu of the outgoing interface.)

Post by Noel Kuntze
Yes.

Is this behaviour intended? Is this use case supported or is it an

unusual way to use ipsec transport mode in combination with NAT/routing?

Post by Noel Kuntze
Set charon.plugins.kernel-netlink.mtu to 1400 or lower.

done. tested. This does not work.

According to the documentation this will set the mtu for routes _added
by charon_. Charon does not alter the routing table in this scenario.
Any hints?

I would expect that this mtu-foo is done automagically in the linux
kernel. The error that I can see when pinging with a packet size that is
too large directly on the router should be transformed into the correct
icmp error and send back to the sender.

Kind regards
Jan-Philipp Hülshoff

[Noel Kuntze]

Post by Jan-Philipp Hülshoff
Any hints?

Change the route in the main routing table then or manage the route using the updown script.