A step-by-step guide on how to implement a Private Endpoint (only) OpenShift cluster on a VPC using WireGuard VPN on IBM Cloud!

9 min readJun 1, 2021

By Philippe THOMAS and Alain Airom (IBM Hybrid Build Cloud Swat Team — EMEA)

TLDR

The purpose of this document is to implement full security so that the OpenShift Cluster can only be accessed thru a WireGuard VPN server in the Virtual Private Cloud (VPC). So, all access for infrastructure/cluster admins, developers, and end-users will go through a VPN tunnel.

Note: Not that the IBM Cloud infrastructure in not secure enough, but in some particular cases, a customer might ask for a private endpoint only (PVE) OpenShift cluster.

WireGuard® is an extremely simple yet fast and modern VPN that utilizes state-of-the-art cryptography. It aims to be faster, simpler, leaner, and more useful than IPsec, while avoiding the massive headache. It intends to be considerably more performant than OpenVPN. WireGuard is designed as a general purpose VPN for running on embedded interfaces and super computers alike, fit for many different circumstances. Initially released for the Linux kernel, it is now cross-platform (Windows, macOS, BSD, iOS, Android) and widely deployable. It is currently under heavy development, but already it might be regarded as the most secure, easiest to use, and simplest VPN solution in the industry.

The solution proposed hereafter would also work on a classic infrastructure, but this article describes what was done on a VPC.

Disclaimer and Important note before continuing the rest of the document: All the domain names, server names, server and local IP addresses provided in this documents are for demonstration purpose only. For the sake of the readers understanding, we kept all the information here, but in case of applying this example to a particular configuration, all values/names/IP addresses should be changed according to the target platform and environment. For example, lots of names were prefixed by the term “nice”, this is only for test and distinction of resources, nothing more!

1-Create a VPC, subnet, OpenShift Cluster …

We suppose that you have already created the following resources:

a VPC
one subnet
one VM inside the VPC
one security group
One OpenShift cluster inside the VPC
all resources are running in one zone — Frankfurt for this example

The target Architecture:

Reminder: the cluster has been implemented with VPE (Virtual Private Endpoint only)

2-Create a VM as a Jumbox for WireGuard

Create a specific VM (2 vCPU x 8 GB) for WireGuard in the VPC:

2 vCPU, 8 GB RAM, 25 GB storage (or the least storage and configuration proposed by IBM Cloud)
Ubuntu 20 minimal
Add a floating IP to this VM for the public IP

From your VPC, On Virtual Server for VPC, click on Create:

Give a name, a resource group, a tag and a location:

Then pick Public VM, Ubuntu 20 Minimal, bx2–2x8 :

Choose ssh key that you have already defined (if you don’t have ssh key for VPC , create one before starting https://www.ssh.com/academy/ssh/keygen):

Pick the VPC and click create virtual server:

Wait a few minutes before the VM has been created and is accessible:

Click on the VM name and go at the bottom of the page:

Click on the pencil on the eth0 row:

Select one of the floating IP (you must have created this IP before):

Click on save.

Check that the IP has been associated to the VM:

Check that you can connect to the Ubuntu VM (the ssh key file should contain the private ssh key):

ssh -i <sshkey-filename> root@161.156.171.206

In case of the following message “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!”

Execute the following:

rm -rf /<user directory>/.ssh/known_hosts

Update and upgrade the Ubuntu system:

apt update 
apt upgrade

Check if reboot is necessary:

cat /var/run/reboot-required

reboot

The VM is now ready to run WireGuard.

ssh -i <sshkey-filename> root@161.156.171.206

3-Install and Configure WireGuard

Install WireGuard with apt:

apt install wireguard

Create Private and Public keys for the WireGuard server:

mkdir -p /etc/wireguard/keys
wg genkey | tee /etc/wireguard/keys/server.key | wg pubkey | tee /etc/wireguard/keys/server.key.pub

Display the WG keys:

cat /etc/wireguard/keys/server.key
cat /etc/wireguard/keys/server.key.pub

Results:

root@nice-wg:~# cat /etc/wireguard/keys/server.key
gCgg2VPH8QbzoUb3IwMtrp2/+d/iRb9y9YaTcGn+J1s=
root@nice-wg:~# cat /etc/wireguard/keys/server.key.pub
8efZqWpWNYWE8dV3bYnuYWstyDXHBTg/9SDR7mECvzs=

Get the ethernet interface:

ip -o -4 route show to default | awk '{print $5}'# ip -o -4 route show to default | awk '{print $5}'
ens3
ens3

Edit the configuration:

nano /etc/wireguard/wg0.conf

The configuration file is empty and you should get something like:

[Interface]
PrivateKey = <contents-of-server-privatekey>
Address = 172.16.0.1/24
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
ListenPort = 51820

[Peer]
PublicKey = <contents-of-client-publickey>
AllowedIPs = 10.0.0.2/32

The definitions use 2 sections (interface and peer). Interface is the WG server and Peer is the WG client(s) list.

Paste this example of server config (just change the PrivateKey with the one you got earlier)

[Interface]
Address = 172.16.0.1/24
ListenPort = 51820
PrivateKey = 4JYiMcICcJLbD1YKsAcn0SUczSgp60B8U3bfaLda4lE=
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE

[Peer]
PublicKey = O3OJ8zmIlHmQGn8GF7QbYCLuR8t2CjWZoTKZ5ia69EA=
AllowedIPs = 9.134.182.146/32

The Peer section is normally concerning the list of WG client. We just create a first one here that we will update later.

Save and exit the file (ctrl+o, ctrl+x)

Change the permissions for different files:

chmod 600 /etc/wireguard/wg0.conf /etc/wireguard/keys/server.key

Start the WireGuard Server:

wg-quick up wg0
wg

you should see both the interface and peer even if the peer (client) has not been created.

Activate permattently wireguard server at boot:

systemctl enable wg-quick@wg0

Change Forwarding in the sysctl.conf file:

nano /etc/sysctl.conf
uncomment : net.ipv4.ip_forward=1

Save and exit the file (ctrl+o, ctrl+x)

sysctl -p
ufw allow 51820/udp
ufw allow 22/tcp
ufw enable
ufw status verbose

Important: the Ubuntu firewall UFW should be started (you mau need to reboot before applying the ufw enable).

3-Create a Security Group for WireGuard

Create a security group nice-sec-vpn (your security group name here)

With 2 rules for ports 22 and 51820 (WireGuard)

Attach this group to the VM:

IMPORTANT : check that the VG VM is only associated to the created rule (here nice-sec-vpn) !!!

Save the security group.

Go back the the security group list Then check the default security group (in my case nice-sec):

Check that all the following rules have been implemented:

Save the security group.

4-Install WireGuard Client on your laptop

Install the WireGuard Client on your laptop:

https://www.wireguard.com/install/

Open the WireGuard Client Interface:

At the bottom of the left pane, create an empty tunnel:

Provide a name (nice-tunnel for example)

Address = 192.168.3.217/32

[Peer]
PublicKey = 8efZqWpWNYWE8dV3bYnuYWstyDXHBTg/9SDR7mECvzs=
AllowedIPs = 166.8.0.0/14, 166.9.0.0/14, 172.16.0.0/24, 10.241.0.0/24, 10.241.64.0/24, 10.241.128.0/24, 10.243.64.13/24
Endpoint = 161.156.171.206:51820

Attention: check the endpoint and port that need to match your WG server IP and Port.

Save the client configuration.

Click on activate:

The light should be green.

Important : After 10 secondes, you should see Data received, Data sent and latest Handshake.

Copy the public key (O3OJ8zmIlHmQGn8GF7QbYCLuR8t2CjWZoTKZ5ia69EA=) to a notepad.

Go back to the server side and open the server configuration:

nano /etc/wireguard/wg0.conf

Add the peer section and replace the public key and provide your own IP of your laptop:

[Interface]
Address = 172.16.0.1/24
ListenPort = 51820
PrivateKey = gA25aKDY2f0Je7vSKLKopLD/sVUytlxBkbkPyKACwEE=
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ens3 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ens3 -j MASQUERADE

[Peer]
PublicKey = O3OJ8zmIlHmQGn8GF7QbYCLuR8t2CjWZoTKZ5ia69EA=
AllowedIPs = 9.134.182.146/32

Save the configuration.

Stop and restart the WireGuard server:

wg-quick down wg0
wg-quick up wg0
wg show wg0

5-Check the VPN

From the Virtual server instances for VPC, take a note of the IP address from the WireGuard VM and ping it:

Go to the OpenShift Cluster in the VPC:

Go to the Networking section:

Copy the Ingress Subdomain:

ping nice-openshift-ba36b2ed0b6b09dbc627b56ceec2f2a4-i000.eu-de.containers.appdomain.cloud

results:

Add the resulting IP : 10.243.13 in the AllowedIPs list on the client:

Save and reactivate the WG client again.

Click on the blue button:

This will open the OpenShift web Console:

You can also try to get the oc login token and try to launch an application and connect to it.

And try the route with or without the WireGuard tunnel:

Without wireguard tunnel (deactivate the tunnel):

Results:

Then reactivate the WG client:

And start the route again: