What is an IPsec VPN? ESP, IKEv2, tunnels and ports explained

IPsec in one paragraph

IPsec (Internet Protocol Security) is a family of IETF standards that protect packets at the IP layer. RFC 4301, the architecture document published in December 2005, says IPsec is "designed to provide security services for traffic at the IP layer." Because it works below TCP and UDP, every application on the machine benefits without being changed. An IPsec VPN is simply IPsec used to carry traffic between two sites, or between your laptop or phone and a gateway, through an encrypted tunnel.

If you are new to the idea of a tunnel, our plain guide on what a VPN does covers the basics. This page goes one level deeper.

The three parts: ESP, AH and IKE

When people say "IPsec" they usually mean three protocols working together.

ESP: the part that encrypts

The Encapsulating Security Payload is defined in RFC 4303. In the RFC's words, "ESP is used to provide confidentiality, data origin authentication, connectionless integrity", along with anti-replay protection. On the wire, ESP is its own IP protocol: the header in front of it carries the value 50. That number matters for firewalls, because ESP is neither TCP nor UDP and needs its own rule.

Which ciphers should ESP use? RFC 8221 (October 2017) sets the implementation requirements. AES-GCM with a 16-byte tag is a MUST, ChaCha20-Poly1305 is a SHOULD, 3DES is a SHOULD NOT and single DES is a MUST NOT. If an old device only offers DES or 3DES, treat that as a reason to replace or reconfigure it.

AH: integrity without encryption

The Authentication Header (RFC 4302, IP protocol 51) authenticates packets and parts of the IP header, but it does not encrypt anything. RFC 4302 points out that ESP offers similar integrity services "and it also provides a confidentiality (encryption) service." RFC 4301 goes further: "most security requirements can be met through the use of ESP by itself." In practice, VPNs use ESP and leave AH unused.

IKEv2: the handshake and key manager

Before ESP can encrypt anything, both ends must prove who they are and agree on keys. That is the job of the Internet Key Exchange. The current version, IKEv2, is specified in RFC 7296 (October 2014), which describes IKE as "a component of IPsec used for performing mutual authentication and establishing and maintaining Security Associations." Authentication can use certificates, a shared secret (pre-shared key) or EAP methods, which is how many company VPNs plug in usernames and passwords.

When a product says it supports "IKEv2", it means IPsec with IKEv2 for key exchange. The older IKEv1 still appears in some equipment for compatibility.

Security associations, in plain terms

A security association (SA) is the agreed set of keys, algorithms and traffic selectors for one direction of traffic. RFC 4301 calls it "a simplex 'connection'", which is why a working tunnel always has at least one pair of SAs: one for each direction. IKEv2 itself runs over an IKE SA, and it then creates the "child" SAs that ESP uses for your data. Keys are renewed on a schedule, so a long-running tunnel quietly replaces its SAs without dropping the connection.

You will see SAs listed when you debug a tunnel. On Linux, for example, strongSwan's swanctl --list-sas shows the IKE SA and its child SAs, with byte counters for each.

Tunnel mode vs transport mode

Each IPsec protocol supports two modes, as RFC 4301 explains.

• Transport mode keeps the original IP header and protects what follows it. It is typically used between two hosts that talk to each other directly.
• Tunnel mode wraps the entire original packet inside a new one with a new IP header. RFC 4301 says that whenever either end of an SA is a security gateway, the SA must be in tunnel mode, apart from two narrow exceptions.

The Canadian Centre for Cyber Security describes the same split in its VPN guidance (ITSAP.80.101, February 2025) and notes that "tunnel mode is commonly used for business VPNs." If you connect a branch office to head office, or your laptop to the corporate network, you are almost certainly using tunnel mode.

IPsec ports and NAT traversal

Most "IPsec doesn't connect" problems are firewall or NAT problems, so it pays to know what goes on the wire.

• UDP 500 for IKE. RFC 7296: "IKE normally listens and sends on UDP port 500."
• UDP 4500 for IKE and for ESP when a NAT is detected. RFC 3948 (January 2005) defines how ESP is wrapped in UDP "for traversing Network Address Translators" and says the UDP port numbers are the same as those used by IKE.
• IP protocol 50 for plain ESP when there is no NAT in the way.

Home routers, hotel networks and phone carriers all use NAT. The two IPsec ends detect it during the IKE exchange (the negotiation was first standardized in RFC 3947 for IKEv1 and is built into IKEv2) and then move everything to UDP 4500. The Cyber Centre's advice to organizations is to "allow only UDP ports 500 and 4500 and encapsulate security payload" for IPsec VPNs and to close everything else on the VPN device.

One more extension matters for phones. MOBIKE (RFC 4555, June 2006) lets the addresses of an IKEv2 tunnel change, so "a mobile Virtual Private Network (VPN) client could use MOBIKE to keep the connection with the VPN gateway active while moving from one address to another." That is what lets a phone move from Wi-Fi to cellular data without rebuilding the tunnel.

What an IPsec VPN connection looks like, step by step

1. Your device sends an IKE_SA_INIT message to the gateway on UDP 500. Both sides propose algorithms and run a Diffie-Hellman exchange.
2. If a NAT is detected, they switch to UDP 4500.
3. In IKE_AUTH, each side proves its identity (certificate, pre-shared key or EAP) and the first child SA is created.
4. Your traffic now flows inside ESP. The gateway decrypts it and forwards it to the internal network.
5. Keys are rekeyed on a timer for as long as the tunnel stays up.

Why the Canadian Cyber Centre recommends IPsec

For organizations choosing a VPN, the Canadian Centre for Cyber Security is direct:

"It is recommended that IPsec be used for VPN access as a primary consideration." (Canadian Centre for Cyber Security, ITSAP.80.101)

Its reasoning is interoperability. The same publication says "IPsec is an open standard, meaning that anyone can build a client or server which will work with other IPsec implementations", while TLS VPNs "often use custom, non-standard features." It also notes that an IPsec client is built into many operating systems, and warns that "some third-party networks restrict or block IPsec traffic", so mobile users may sometimes fail to connect. We compare the two approaches in IPsec vs SSL VPN.

The guidance is written for organizations, not for individuals picking a consumer app. It is still a useful signal of how a Canadian government security agency views the protocol.

When IPsec is a good choice

• Site-to-site links. Connecting two offices, or an office to a cloud network, is the gateway-to-gateway case that tunnel mode was built for.
• Mixed equipment. Because it is an open standard, an IPsec gateway from one vendor can usually talk to a client or gateway from another.
• Built-in clients. If you do not want to install extra software on staff laptops or phones, the operating system's own IKEv2 client may be enough.
• Policy-heavy setups. IPsec has fine-grained traffic selectors and supports certificate hierarchies, which suits larger networks.

When something else may be simpler

IPsec's flexibility comes with more settings to get right. For a personal server or a small group of devices, WireGuard needs far less configuration; see IPsec vs WireGuard. Networks that block UDP 500 and 4500 are another case where a TLS-based VPN on TCP 443 may get through when IPsec cannot. For a broader look at the options, read our guide to VPN protocols.

IPsec software on Linux and routers

On Linux, the kernel does the ESP encryption and a userspace daemon handles IKE. The two best known open source daemons are:

• strongSwan, which describes itself as "a comprehensive implementation of the Internet Key Exchange (IKE) protocols."
• Libreswan, which supports IKE versions 1 and 2 and uses the kernel's built-in XFRM IPsec stack on Linux.

Both descend from the FreeS/WAN project of 1997 to 2004. We cover their history and differences in strongSwan vs Libreswan. Many home and small business routers also include an IPsec server; our page on VPN routers explains what to look for.

Common mistakes

• Opening UDP 500 but forgetting UDP 4500, which breaks any client behind NAT.
• Leaving legacy ciphers such as DES or 3DES enabled for old clients. RFC 8221 rules out the first and discourages the second.
• Using one shared pre-shared key for every remote user instead of certificates or per-user EAP authentication.
• Turning on split tunnelling without a reason. The Cyber Centre tells organizations to "avoid split tunnelling as much as possible."

Common questions