What Is IPsec? How IPsec VPNs Secure Your Traffic
What Is IPsec? How IPsec VPNs Secure Your Traffic
IPsec, short for Internet Protocol Security, is a protocol suite that authenticates and encrypts every IP packet in a communication session. It operates at the network layer, which means it can protect any application traffic without requiring changes to individual programs. That makes it a foundational technology for virtual private networks, site-to-site connections, and secure remote access.
This guide explains how IPsec works, the protocols that make it up, the difference between tunnel and transport mode, and how to verify that your IPsec VPN is actually protecting your traffic.
What Is IPsec?
What Is IPsec? How IPsec VPNs Secure Your Traffic - What Is IPsec?.
IPsec is a group of open standards developed by the Internet Engineering Task Force (IETF). It adds two things that plain IP lacks: authentication and confidentiality. Without IPsec, data sent over the Internet is like a postcard—any router or intermediary between the sender and recipient can read it. IPsec puts that data in an envelope.
The suite provides several security services at the IP layer:
- Peer authentication: Each side verifies the identity of the other before a session begins.
- Data origin authentication: The recipient can confirm that packets came from the claimed sender.
- Data integrity: Any modification to a packet in transit is detectable.
- Confidentiality: Payloads are encrypted so intermediaries cannot read them.
- Replay protection: Old packets cannot be captured and resent to impersonate a legitimate session.
Because IPsec works below the transport layer, it secures all traffic that flows over the IP connection—whether that traffic is web browsing, email, file transfer, or VoIP.
Why IPsec Matters
What Is IPsec? How IPsec VPNs Secure Your Traffic - Why IPsec Matters.
Networking protocols such as TCP/IP were designed for delivery and reliability, not privacy. The original IPv4 specification included almost no security provisions. Anyone with access to a network segment could read or modify traffic passing through it.
IPsec addresses that gap at the most fundamental level. Unlike TLS, which protects specific applications such as HTTPS web traffic, IPsec protects the entire IP conversation. That makes it especially useful for:
- Remote workers connecting to corporate networks over public Wi-Fi
- Branch offices linking to a headquarters over the public Internet
- Cloud connectivity where organizations need private links to cloud providers
- Mobile devices that roam between networks but need a stable secure tunnel
How IPsec Works: The Core Protocols
What Is IPsec? How IPsec VPNs Secure Your Traffic - How IPsec Works: The Core Protocols.
IPsec is not a single protocol. It is a suite built from three main components: Authentication Header (AH), Encapsulating Security Payload (ESP), and the key exchange mechanisms that establish security associations.
Authentication Header (AH)
AH provides data integrity, data origin authentication, and optional replay protection—but no encryption. It works by computing a cryptographic hash over the packet and a shared secret key. If any authenticated field changes in transit, the recipient detects the mismatch and discards the packet.
In IPv4, AH authenticates the IP payload and most header fields, excluding mutable fields such as TTL, checksum, and DSCP that routers legitimately change. In IPv6, it protects a similar set of immutable fields. AH uses IP protocol number 51.
Because AH does not hide data, it is rarely used alone today. It appears most often in legacy deployments or in combination with ESP.
Encapsulating Security Payload (ESP)
ESP is the workhorse of modern IPsec. It provides confidentiality through encryption, plus the same integrity and authentication services as AH. ESP adds its own header before the payload and a trailer after it, then encrypts the protected content.
ESP supports three configurations:
- Encryption and authentication together: The recommended setup.
- Authentication only: Similar to AH but with a different packet structure.
- Encryption only: Strongly discouraged because unauthenticated encryption is vulnerable to active attacks.
ESP uses IP protocol number 50.
Internet Key Exchange (IKE)
Before AH or ESP can protect traffic, both endpoints must agree on algorithms, keys, and parameters. That agreement is called a Security Association (SA). IKE automates the negotiation process.
IKE operates in two phases:
- Phase 1 establishes a secure, authenticated channel between the peers using Diffie-Hellman key exchange.
- Phase 2 negotiates the actual IPsec SAs used to protect data traffic.
The modern version, IKEv2, simplifies the exchange, supports mobility features such as MOBIKE, and handles network changes better than the original IKE. Most current IPsec deployments use IKEv2.
IPsec Tunnel Mode vs. Transport Mode
What Is IPsec? How IPsec VPNs Secure Your Traffic - IPsec Tunnel Mode vs. Transport Mode.
IPsec can operate in two modes, and choosing the right one depends on what you are protecting.
Tunnel Mode
Tunnel mode is the default for VPNs. It encapsulates the entire original IP packet—header and payload—inside a new IP packet. The outer header routes the packet between the two VPN gateways, while the inner header remains encrypted and invisible to intermediaries.
Use cases:
- Site-to-site VPNs between routers or firewalls
- Remote access VPNs where a client connects to a corporate gateway
- Any scenario where the final destination should be hidden from the transit network
Transport Mode
Transport mode encrypts only the payload of the IP packet. The original IP header remains visible, so routers can see the final destination. This reduces overhead but exposes routing information.
Use cases:
- Host-to-host encryption between two servers
- Scenarios where a separate tunneling protocol such as GRE already handles encapsulation
- Situations where packet overhead must be minimized
| Feature | Tunnel Mode | Transport Mode |
|---|---|---|
| Original IP header | Encrypted | Visible |
| Typical use | VPN gateways | Host-to-host |
| Overhead | Higher | Lower |
| Destination privacy | Yes | No |
What Port Does IPsec Use?
IPsec itself does not use TCP or UDP ports in the traditional sense. AH and ESP are IP protocols identified by protocol numbers, not ports. However, the key exchange that sets up IPsec does use ports:
- UDP port 500: Used by IKE for the initial key exchange.
- UDP port 4500: Used for IPsec NAT traversal, which lets IPsec work through network address translation.
Some firewalls block protocol 50 (ESP) and protocol 51 (AH), so NAT traversal over UDP 4500 is essential for many real-world deployments.
IPsec vs. TLS: Which Should You Use?
IPsec and TLS both provide encryption and authentication, but they operate at different layers and solve different problems.
| Aspect | IPsec | TLS |
|---|---|---|
| OSI layer | Network layer (Layer 3) | Transport layer (Layer 4) |
| Scope | Protects all IP traffic | Protects a single application session |
| Configuration | Requires client or gateway setup | Built into browsers and apps |
| Typical use | Site-to-site VPNs, remote access | HTTPS, secure email, web apps |
| NAT traversal | Requires NAT-T (UDP 4500) | Works natively over TCP |
IPsec shines when you need to secure every protocol between two networks. TLS is simpler when you only need to protect specific applications. Many organizations use both: IPsec for network-level connectivity and TLS for application-level security on top of it.
How to Verify Your IPsec VPN Is Working
Setting up an IPsec VPN is only half the job. A misconfigured tunnel can silently leak your real IP address, DNS queries, or WebRTC traffic even when the VPN appears connected. Regular testing should be part of any IPsec deployment.
Check for IP and DNS Leaks
After connecting to your IPsec VPN, verify that your public IP address matches the VPN exit server and not your ISP. A DNS leak test confirms that DNS queries travel through the encrypted tunnel rather than to your local resolver.
BrowserScan offers free web-based tools for exactly this purpose. Its DNS and WebRTC leak tests show whether your VPN is exposing your real IP address or leaking DNS requests outside the tunnel. The IP address lookup also displays geolocation, ISP, and proxy detection data so you can confirm the exit point matches your VPN configuration.
Test Your Browser Fingerprint
Even with a working IPsec tunnel, your browser can still reveal identifying information. BrowserScan's browser fingerprint analysis checks whether your browser profile looks authentic or exposes characteristics that could be used to track you across sessions. For IPsec VPN users, this is a useful complement to leak testing because it verifies that the encrypted tunnel does not undermine your broader privacy posture.
Scan for Open Ports
If you manage the IPsec gateway itself, an Nmap-powered port scanner can identify open services that should not be exposed to the Internet. A properly configured IPsec endpoint should expose only the ports required for IKE and ESP—typically UDP 500 and UDP 4500—and nothing else.
Common IPsec Challenges
MTU and MSS Issues
IPsec adds headers and trailers to every packet, typically 50–60 bytes or more. If the resulting packet exceeds the path MTU, routers must fragment it or drop it. Fragmentation increases latency and can cause connectivity problems, especially with UDP traffic.
The usual fix is to lower the MTU on the tunnel interface or enable MSS clamping so TCP packets fit within the reduced size. Many IPsec implementations handle this automatically, but manual tuning is sometimes required.
NAT Traversal
Network address translation rewrites IP headers, which breaks the integrity checks that AH relies on. ESP can survive NAT only when NAT traversal is enabled, which wraps ESP packets in UDP on port 4500. If you connect through a home router or hotel Wi-Fi, NAT-T is almost always required.
Scaling Site-to-Site Meshes
A full mesh of IPsec tunnels between many sites requires managing a large number of SAs and configurations. Hub-and-spoke topologies reduce complexity but create a single point of failure. Cloud-based IPsec termination points can simplify this by letting each site connect to a single anycast endpoint instead of to every other site.
IPsec in Practice: Deployment Examples
Remote Access VPN
An employee installs a VPN client configured with IKEv2. The client authenticates to the corporate gateway, negotiates an SA, and establishes an ESP tunnel. All traffic from the employee's laptop is encrypted and routed through the corporate network. The employee can access internal file shares, databases, and applications as if sitting in the office.
Site-to-Site VPN
Two office routers establish a persistent IPsec tunnel over the public Internet. Each router encrypts traffic destined for the other office's subnet. Employees at either location can reach resources at the other site without any per-application configuration.
Cloud Connectivity
Organizations connect their on-premises networks to cloud providers using IPsec tunnels terminated at the cloud provider's edge. This creates a private path for hybrid workloads without dedicated physical links.
If you are setting up a personal IPsec VPN, the Algo VPN tutorial walks through deploying a WireGuard and IPsec VPN in the cloud, configuring clients, and verifying the tunnel with leak tests.
Related reading
- TLS Fingerprint Preservation: What It Is and How It Works - Learn what TLS fingerprint preservation is, how JA3, JA4, and JARM work, why preserving fingerprints matters for privacy and automation, and how to test yours.
- Penetration Testing Tools: Types, Top Picks & How to Choose - Explore the best penetration testing tools by category, from Nmap and Burp Suite to Kali Linux. Learn how to choose, combine, and verify your pentest setup.
- Network Security Tools: 15 Essential Types & How to Use Them - Discover the essential network security tools for monitoring, scanning, and defending your systems. Compare free and paid options and learn how to verify your setup.
Frequently Asked Questions
Is IPsec still secure?
Yes. IPsec with modern algorithms such as AES-GCM, SHA-256, and Diffie-Hellman groups 14 or higher remains secure. Older configurations using DES, 3DES, or weak Diffie-Hellman groups should be avoided.
What is the difference between IKEv1 and IKEv2?
IKEv2 simplifies the negotiation process, reduces latency, supports mobility and multihoming through MOBIKE, and handles network changes better than IKEv1. IKEv2 is the recommended choice for new deployments.
Can IPsec work through NAT?
Yes, with NAT traversal enabled. NAT-T encapsulates ESP packets in UDP on port 4500 so they can pass through NAT devices. Most modern IPsec implementations enable NAT-T by default.
Does IPsec hide my IP address?
In tunnel mode, IPsec encrypts the original IP header, so intermediaries cannot see the final destination. However, the outer IP header still shows the VPN gateway addresses. Your real IP address is hidden from the destination but visible to the VPN provider.
How do I know if my IPsec VPN is leaking?
Connect to your VPN and use a leak testing tool to compare your visible IP address against your expected VPN exit IP. Also run DNS and WebRTC leak tests. BrowserScan's free web-based tools let you perform all of these checks without installing software.
Conclusion
IPsec remains one of the most important security protocols on the Internet. Its network-layer design means it can protect any traffic between two endpoints, making it ideal for VPNs, site-to-site links, and hybrid cloud connectivity. Understanding the difference between AH and ESP, tunnel and transport mode, and IKEv1 and IKEv2 helps you choose the right configuration for your needs.
But configuration is only part of the story. A VPN that leaks DNS queries or exposes your real IP address provides false security. After deploying any IPsec tunnel, verify it with independent testing. Use leak tests to confirm your traffic stays inside the tunnel, check your browser fingerprint for unintended exposure, and scan your gateway for open ports. For a deeper look at the broader toolset available, see this guide to essential network security tools and this overview of penetration testing tools that complement IPsec deployments.
For further technical detail, refer to the IPsec specification on Wikipedia, the Cloudflare explanation of IPsec VPNs, and the AWS overview of IPsec.