what is IPSEC VPN how it works

It is a virtual private network that provides a secure tunnel across a public network, so you will learn what is IPSEC VPN and how it works in briefly. There are two types of VPN

(1) Client VPN

(2) site-to-site VPN

  • client VPN connection to a home or roaming user to an office
  • site to site VPN connect remote office to the main office
what is IPSEC VPN how it works
VPN Flow Diagram

what is IPSEC

IPSEC, short for IP security is a suite of protocols, standards, and algorithms to secure traffic over an untrusted network such as the Internet.

IPSEC provides three core services.

  1. confidentiality- prevent data theft of data using encryption.
  2. Integrity – Ensures that data is tempered using a hashing algorithm.
  3. Authentication – confirm the identity of the host sending data using pre-shared keys or a certificate authority.
  4. Anti reply -prevent duplication of encrypted packets by assigning a unique sequence number. The only the private keys can decrypt data.

confidentiality or encryption keys- Keys are generated values used to both encrypt and decrypt data .longer key is more secure.

Symmetric key: It is for both encrypt and decrypts data
Asymmetric key: Require separate key for both process
public key for encryption
public key openly exchanged between the device to encrypt data during transfer private key are never exchanged.

Diffie Hellman (D-H) public key exchange is the most common standard used to create or exchange Keys across insecure mediums.

  • D-H is not used to encrypt data but rather to generate the keys that are used to encrypt and decrypt data.
  • DFS (data encryption standard).
  • 3DES(triple) 168-bit key.
  • AES (Advance)128,192,256 bit key.
  • There are several D-H groups.
  • group 1-786 bit.
  • group 2 –1024 bits. group 5 -2048.

Data Integrity and hushing- Hashes message Authentication code (HMAC) is used to perform this hashing function.

  • two HMAC algorithms are commonly used: HMAC MD5 (message digest 5)128 bit
  • HMAC SHA1 (secure hash algorithm)168 bit

Authentication:- Another concern when sending data across the internet is the source or origin of that data . It’s possible to masquerade or spoof one’s identity or address

For an IPsec VPN tunnel to be able to established both sides of the tunnel must be authenticated and to accomplish this either pre-shared key or RSA digital signature used

IPsec protocols – IPsec uses one of two protocols headers for securing data.
1.Authentication Header ( AH)
2.Encapsulation security payload (ESP)
AH -IP protocols 51 it does not encrypt any date at all.
ESP IP protocols 50 perform confidential,authentication,integrity services.It provides an additional header and trailer to a packet.

Transport vs Tunnel modes:-
Each IPsec protocols (AH or ESP) can operate in one of two modes:
1.Transport mode:-Original IP hacker is the same often security communication from one device to another single device.

2.Tunnel mode:-The entire original packet is hashed and /or encrypted including both.
A temporary IP Header is applied to the packet during transmit.

AH Header and IP packet

All transport mode: –

IP Header   AH Header TCP Header data payload

AH Tunnel mode: –

New IP header Ah header IP header Tcp header Data payload

ESP alters and IP packets

IP header TCP header Data payload

2 ESP Transport Mode:

IP header ESP header TCP Header Data payload ESP trailer


VPN peers establish a (security association (SA))” connection” or “policy” between two endpoints of the VPN tunnel. Thus, for all Full communication to occur two SA must be established one for each direction. The (IKE)protocols control this negotiation process on UDP port 500.

IKE policy sets are crafted to negotiate several parameters including.

  1. Encryption algorithm (such as DES,3DES,AES)
  2. hashing algorithm (such as MD5 ,sha-1)
  3. Authentication method (shared key,RSA)
  4. Diffie Hellman (d-h)group
  5. The SA life time

There are two phases to this negotiating process:

  1. IKE phase 1 establishes the initial tunnel (referred to as the IKE or ISAKMP)
  2. peers authenticated encrypted, encryption and hashing algorithm are negotiated and key are exchange based on the IKE policy sets.

IKE phase 2 used for phase negotiation

1.Main mode: slower but more secure.  

2. Aggressive mode: faster but less secure

IKE phase 2 establishes the (IPSEC SR) which details AH or ESP parameters for securing data.

These parameters are contained in an IPSEC Transform set. IKE phase 1 negotiates parameters for the tunnel (Key ex ) itself, while the IKE phase negotiates parameters for data traveling that tunnel.

The FIVE steps of IPSEC VPN

  1. any traffic that should be secured and sent across the Internet the tunnel is identified as interesting traffic, usually an access list.
  2. IKE phase 1 is initiated peers are authenticated, keys are exchanged, IKE policy sets are negotiated and if successful the IPSEC -SA is established.
  3. IKE phase 2 is initiated IPSEC transform sets are negotiated and if successful the IPSEC SA is established.
  4. DATA is actually transferred using the agreed-upon security policy.
  5. The session is torn down once the SA lifetime expires.