VPNs (virtual private networks) fulfill an essential role in the lives of individuals and businesses. They provide much-needed security and privacy on the internet — but how exactly do they do that?
The mechanism at the heart of a VPN is the VPN protocol.
So, what is a VPN protocol and what does it do?
Table of contents
What is a VPN protocol?
A VPN protocol is a set of rules that dictate how a device (computer, router, smartphone) communicates with a VPN server; i.e., how they encrypt and route data when they establish a secure connection (tunnel).
These rules affect the connection speed, security, and reliability, and determine the protocol’s best use.
What VPN protocols are there?
There are a large number of VPN protocols out there, some with niche uses or proprietary to VPN vendors. The list below shows the most common protocols in use today.
*Note that the assessment given here is general, and the ultimate balance of speed vs security depends on the configuration.
What is VPN encryption?
VPNs use cryptography to secure data during transit. They convert readable information into a scrambled mess (ciphertext) that cannot be read unless decrypted using the correct key.
The complexity of the encryption determines how secure a protocol is — the more complex the encryption, the harder it is to break.
Encryption techniques
Private-key encryption (symmetric cryptography)
Symmetric cryptography means both the sender and receiver use the same encryption/decryption key. This technique is common among VPNs.
Public-key encryption (asymmetric cryptography)
In asymmetric cryptography, there are two keys; public and private. The public key encrypts the data, but decryption can only be done with the private key. That way, anyone can send data, but only an authorized party can read it.
Public-key encryption is used in secure authentication scenarios like digital signatures or non-repudiation systems.
Handshake encryption (RSA)
In handshake encryption the sender and receiver first agree on the algorithms and keys they are going to use before establishing a connection. RSA stands for Rivest-Shamir-Adleman (the last names of the three inventors), and denotes the algorithm used in handshake encryption.
RSA-2048 is a common cipher among VPNs and is considered highly secure, albeit potentially slow because of its high computing power requirements.
Some VPNs use DH encryption (Diffie-Hellman), which is now being succeeded by the more secure Elliptic curve Diffie-Hellman (ECDH).
Secure hash algorithm (SHA)
The secure hash algorithm authenticates SSL/TLS connections and checks the validity of TLS certificates. Besides VPNs it also plays an important role in online communication in HTTPS encryption.
Ciphers
Ciphers are algorithms that carry out encryption and decryption. The key length (e.g. 128-bit, 256-bit) and the algorithm's strength determine how secure a cipher is.
However the longer and more secure the cipher, the longer it will take to encrypt and decrypt the data. VPN vendors may therefore sometimes sacrifice security for performance to keep up the user experience.
Advanced encryption standard (AES)
AES is a NIST-certified and the most widespread cipher among available VPNs. It offers multiple key lengths (AES-128, AES-192, AES-256) and supports block chaining models.
Blowfish
Blowfish also supports a wide range of key lengths, from 32 to 448 bits. It is a fast block cipher, but can be slow when changing keys. However, it also has some vulnerabilities, which are remedied by its successor Twofish.
Camellia
Camellia is a modern cipher developed by Mitsubishi Electric and NTT in Japan, and offers a level of security and performance comparable to AES.
What are the pros and cons of each VPN protocol?
OpenVPN
One of the most widespread protocols in use today, and a staple among VPN offerings. It is an open-source VPN protocol, which means it is usually fast to patch up vulnerabilities and is regularly audited.
OpenVPN is very versatile and can run on most ports as well as both in TCP and UDP mode. It also allows you to choose whether it emulates network layer 2 or 3.
OpenVPN is good at providing online anonymity, as it can bypass filters and firewalls, and runs on all major platforms.
Pros
- Privacy — OpenVPN provides excellent anonymity and is compatible with most firewalls.
- Security — It provides strong encryption and is one of the most secure protocols out there.
- Transparency — OpenVPN is open-source, which means the code is available to the public in full, and anyone with the knowledge can assess it for vulnerabilities.
- Versatility — OpenVPN is the most popular protocol in existence, is optimized for all platforms, and covers a wide spectrum of use cases.
Cons
- Complexity — OpenVPN can be tricky to set up, if you set it up yourself.
- Speed — In some configurations it can be slow, as its complex encryption consumes a lot of computing power.
What is TCP and UDP?
TCP and UDP are communication protocols for data transfer over the internet. For VPNs they represent two modes in which data is sent between users and the VPN server.
TCP (transmission control protocol)
TCP first establishes a connection between the sender and receiver before sending data.
The major advantage of this connection is its reliability. The data packets are sent in a structured and ordered way, and when one is dropped, the receiver requests it be re-sent.
The downside of TCP is that it is slower, though you will only feel this in long-distance connections.
TCP is used for web browsing or working with SaaS apps.
UDP (user datagram protocol)
UDP does not establish a connection. Instead, the sender sends data at the receiver regardless of whether any packets are lost.
The advantage is that UDP is much faster than TCP, especially over long distances, and is also more data-efficient. The downside is that if the receiver is overwhelmed or if there is an outage, the data will simply be lost.
UDP is suited for data transfers where reduced reliability is not such an issue, such as video streaming or real-time broadcasts.
L2TP/IPsec
L2TP/IPsec (layer 2 tunneling protocol / internet protocol security) is an older protocol developed by Microsoft and Cisco. L2TP does not provide any encryption on its own, but relies on IPsec for cryptographic mechanisms.
Pros
- Security — Though lacking any native encryption, L2TP can accept many other encryption protocols, which can provide very high security. It also forms a double-layered tunnel, which increases security further.
Cons
- Privacy — The L2TP/IPsec tunnel is easy to spot, which means the protocol is less useful for bypassing content restrictions on closely surveyed networks.
- Speed — The protocol encapsulates data twice, which means it is slower by default.
- Compatibility — Being a Microsoft product, it has limited support and fewer features in Linux.
IKEv2
IKEv2 (internet key exchange version 2) is a well-established protocol particularly common among remote-access VPNs. It uses IPsec for encryption.
It is a fast protocol that requires low bandwidth and supports NAT traversal, a technique of maintaining connections across gateways that use network address translation (NAT).
It is useful for establishing highly secure connections for mobile devices, thanks to its rapid reconnection ability (MOBIKE) – the user can switch from mobile data to Wi-Fi (or between Wi-Fis) without losing internet access.
Pros
- Speed — An IKEv2 tunnel consumes little bandwidth.
- Reliability — Can quickly reconnect when switching between networks. Useful for stable remote access on mobile devices.
Cons
- Compatibility — Available natively for Windows, but there is no guarantee of cross-platform interoperability.
- Configuration — IKEv2 is harder to configure if you use a Windows client.
- Possible vulnerability — The Diffe-Hellman algorithm’s security was called into question in a 2015 paper by Adrian et al, suggesting a possible vulnerability allowing mass decryption and surveillance by government bodies. However, other investigators have since independently refuted these claims, citing no clear evidence for the above.
Wireguard
A very fast and lightweight VPN protocol, also the youngest from the batch described here. Wireguard is an open-source protocol, originally developed for Linux, though now available for all major platforms as well.
Wireguard is easy to set up, uses strong yet very efficient ciphers, and has a small target silhouette thanks to being low-code. However, this also means that it lacks certain other features, like RADIUS authentication.
Pros
- High speed — Wireguard is very fast and reliable thanks to its low-code architecture.
- Security — Being open-source, anyone in the IT security community can view Wireguard’s code and audit it for vulnerabilities. The minimal amount of code also means a smaller statistical chance that a vulnerability will occur.
- Compatibility — It is available for all major operating systems and suitable for both small devices and large network routers.
- Customizability — Wireguard supports custom extensions to improve its core functionality. Users can add additional cryptographic or authentication mechanisms to tailor the protocol to their needs.
- Simplicity — The protocol is easy to configure and manage.
Cons
- Low adoption — Wireguard is still very young, being officially released in June 2022. Despite its overwhelmingly positive reception, it still isn’t what you’d call a mainstream VPN protocol.
- Features — The protocol lacks some of the advanced features that come built-in with some of the more established protocols.
SSTP
SSTP (secure socket tunneling protocol) is Microsoft’s proprietary VPN protocol introduced with Windows Vista, meant to provide a more secure tunnel than PPTP.
It was designed primarily for remote access, beingTCP-based. This means it struggles with connections over longer distances, and because it is only supported by Windows, it was never widely adopted.
Pros
- Security — SSTP uses strong encryption (AES-256), which puts it up to standard with competitive protocols. It can also bypass firewalls.
- Windows integration — For Windows users this protocol is very easy to set up (on the client side), being fully integrated in their operating systems.
Cons
- Proprietary — SSTP runs on Windows platforms only. And, because it is company-owned software, it isn’t open to an independent audit.
PPTP
PPTP (point-to-point tunneling protocol) is one of the oldest protocols. It was introduced by Microsoft in the 90s and has been in use since then on almost all platforms. It is very fast and easy to set up, and is a popular method of accessing geo-restricted content.
However, the security it provides is no longer up to current standards, which is why it is being deprecated on some operating systems, with macOS being the first to do so.
Pros
- Widely integrated — Many operating systems and devices still support PPTP natively.
- Fast — The high speed makes it useful for streaming geo-restricted content.
Cons
- Unsecure — PPTP contains several vulnerabilities, which make it unsuitable for business-grade use.
- Obsolete — Microsoft no longer maintains PPTP and encourages users to switch to later protocols, like SSTP or L2TP/IPsec.
- Breakable — PPTP ciphers and certificate exchange have both been shown too weak for company remote access.
Which VPN protocol should I use?
The choice of VPN protocol depends primarily on how you want to use your VPN.
- Do you need secure remote access to business systems? Use L2TP/IPsec, IKEv2, SSTP or Wireguard
- Do you need to connect remote branches with an encrypted tunnel? Use OpenVPN, IPsec, or Wireguard
- Do you need privacy while browsing the web or streaming online media? Use OpenVPN, or IKEv2
Closing remarks
Each VPN protocol has strengths and weaknesses that dictate where it is best used. Some protocols are better suited for individual users who want to stream geo-restricted content, while others excel at protecting remote employees and business systems.
If your company is looking to deploy a VPN, but you aren’t sure how to set it up and which VPN protocol to choose, give GoodAccess a try.
We’ve packaged a business VPN that uses several protocols into a stable, secure, and easy-to-use service. Just sign up, connect, and you’re good to go.
