Hosting services behind a restrictive firewall/CGNAT using DNAT on a VPS hosted wireguard endpoint

This is a short description of how to host services, using STORJ node as an example, on a host behind CG-NAT, or otherwise restrictive firewall, by forwarding packets through WireGuard endpoint on a relatively fast nearby VPS. This is not specific to Storj, and can be adopted to hosting other services.

Here we will use Oracle Cloud instance to host VPN server; the free tier still provides 10TB of monthly traffic that is sufficient for most node operators. Just make sure to create an account in the closest datacenter to minimize extra latency.

This was originally posted as a GitHub gist, and the firewall-cmd configuration as a follow-up forum post.

Notes on configuring the cloud instance

1. Create the new Oracle compute instance (ideally, Ampere, because they are awesome, but if that is not available, any other will do too).

2. Pick any OS you prefer. The examples below cover Ubuntu and Oracle Linux 9.

3. Configure public IP address (this is the default), and upload SSH key to access the instance.

4. Then edit the Ingress Rules in the Default Security List in the VCN associated with the instance and rules to allow:

   • Traffic from anywhere 0.0.0.0/0, any port, to destination port 28967, one for udp, one for tcp. This is for storj.
   • UDP to port 51820, for WireGuard. It does not need to be this specific port, any will do, but adjust the rest accordingly. The source network can also be narrowed down to your ISP’s address range, if desired.

     Stateless	Source	IP Protocol	Source Port Range	Destination Port Range	Type and Code	Allows	Description
     No	0.0.0.0/0	TCP	All	28967		TCP Traffic for port 28967	Storj TCP
     No	0.0.0.0/0	UDP	All	28967		UDP Traffic for port 28967	Storj UDP
     No	0.0.0.0/0	UDP	All	51820		UDP Traffic for port 51820	Wireguard

   That’s all that needs to be done in Oracle console.

5. Optionally, configure the A record to point to the public IP with your DNS provider, to use DNS name and not an ugly IP address in the subsequent configuration and as your storj node external address.

Installing and configuring wireguard tunnel

1. ssh to your new instance, update software, and install wireguard.

   On Ubuntu:

   sudo apt update && sudo apt upgrade
   sudo reboot
   sudo apt install wireguard -y
   

   On Oracle Linux 9:

   sudo dnf update -y
   sudo reboot
   sudo dnf install -y wireguard-tools firewalld
   

   If wireguard-tools is not found, enable Oracle’s EPEL repository and try again:

   sudo dnf install -y oracle-epel-release-el9
   sudo dnf config-manager --enable ol9_developer_EPEL
   sudo dnf install -y wireguard-tools firewalld
   

2. Configure wireguard tunnel between your node and VPS. There are tons of tutorials, here are the steps for reference:

   On the VPS:

   1. Initialize the config file

      (umask 077 && printf "[Interface]\nPrivateKey= " | sudo tee /etc/wireguard/wg0.conf > /dev/null)
      wg genkey | sudo tee -a /etc/wireguard/wg0.conf | wg pubkey | sudo tee /etc/wireguard/publickey
      

   2. Add peer information (public key and address) after configuring it below

   3. Enable ipv4 forwarding: in /etc/sysctl.conf uncomment

      # Uncomment the next line to enable packet forwarding for IPv4
      net.ipv4.ip_forward=1   
      

      and for the change to take effect, load it:

      sudo sysctl -p
      

      Note: it is possible to configure this key in a number of other configuration files, see man sysctl, but in this case either provide path to file to -p argument or simply use sudo sysctl --system, that will parse all configuration files.

   4. Enable and start the wireguard service:

      On Oracle Linux 9, make sure firewalld is also running:

      sudo systemctl enable --now firewalld
      

      Then start WireGuard:

      sudo systemctl enable wg-quick@wg0
      sudo systemctl start wg-quick@wg0
      

   On the client, assuming it’s a TrueNAS, and storj runs in the jail, we would need few things:

   1. In the jail properties tick the allow_tun flag. (e.g. iocage set allow_tun=1 jailname)
   2. On the host under System → Tunables add LOADER variable if_wg_load with the value YES, to load wireguard kernel module.
   3. Initialize the wireguard config file and create keys just like above, noting that in FreeBSD the default configuration file location is /usr/local/etc/wireguard/wg0.conf
   4. In the jail, in the /etc/rc.conf add

      wireguard_enable="YES"
      wireguard_interfaces="wg0"
      

   Generally, the config files shall look like so:

   1. On the server: /etc/wireguard/wg0.conf

      [Interface]
      PrivateKey = <server private key>
      
      ListenPort = 51820
      Address = 10.0.60.1
      
      # Allow WireGuard's own traffic to reach the server.
      PreUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
      PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT
      
      [Peer]
      PublicKey = <client public key>
      AllowedIPs = 10.0.60.2/32
      

   2. On the client: /usr/local/etc/wireguard/wg0.conf

      [Interface]
      PrivateKey = <client private key>
      Address = 10.0.60.2
      
      [Peer]
      PublicKey = <server public key>
      AllowedIPs = 10.0.60.1/32
      Endpoint = sub.example.com:51820
      PersistentKeepalive = 25
      

   3. Start the service on the client:

      service wireguard start
      

   At this point the client shall be able to ping the server, and the server shall be able to ping the client, at 10.0.60.1 and 10.0.60.2 addresses, respectively.

Packet forwarding

using iptables

Now the very last thing, the meat of this tutorial. In the [Interface] section on the server in the /etc/wireguard/wg0.conf add the following PreUp and PostDown rules (PostDown rules are copies of PreUp rules, but with -A or -I options replaced with -D, to delete the rule):

# Allow WireGuard's own traffic to reach the server.
PreUp = iptables -I INPUT -p udp --dport 51820 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 51820 -j ACCEPT

# Allow incoming Storj connections on the public interface BEFORE they are forwarded.
PreUp = iptables -I INPUT -p tcp --dport 28967 -j ACCEPT
PostDown = iptables -D INPUT -p tcp --dport 28967 -j ACCEPT
PreUp = iptables -I INPUT -p udp --dport 28967 -j ACCEPT
PostDown = iptables -D INPUT -p udp --dport 28967 -j ACCEPT

# Port forward incoming Storj traffic to the VPN client.
PreUp = iptables -t nat -I PREROUTING -i ens3 -p tcp --dport 28967 -j DNAT --to-destination 10.0.60.2:28967
PostDown = iptables -t nat -D PREROUTING -i ens3 -p tcp --dport 28967 -j DNAT --to-destination 10.0.60.2:28967
PreUp = iptables -t nat -I PREROUTING -i ens3 -p udp --dport 28967 -j DNAT --to-destination 10.0.60.2:28967
PostDown = iptables -t nat -D PREROUTING -i ens3 -p udp --dport 28967 -j DNAT --to-destination 10.0.60.2:28967

# Allow the now-forwarded traffic to pass from the public interface to the VPN interface.
PreUp = iptables -I FORWARD -i ens3 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PreUp = iptables -I FORWARD -i ens3 -o %i -p tcp -d 10.0.60.2 --dport 28967 -j ACCEPT
PreUp = iptables -I FORWARD -i ens3 -o %i -p udp -d 10.0.60.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i ens3 -o %i -m state --state RELATED,ESTABLISHED -j ACCEPT
PostDown = iptables -D FORWARD -i ens3 -o %i -p tcp -d 10.0.60.2 --dport 28967 -j ACCEPT
PostDown = iptables -D FORWARD -i ens3 -o %i -p udp -d 10.0.60.2 --dport 28967 -j ACCEPT

# Allow outbound traffic from the VPN client out to the internet
PreUp = iptables -I FORWARD -i %i -o ens3 -j ACCEPT
PostDown = iptables -D FORWARD -i %i -o ens3 -j ACCEPT

# Perform NAT for traffic from the VPN client going to the internet
PreUp = iptables -t nat -I POSTROUTING -s 10.0.60.2/32 -o ens3 -j MASQUERADE
PostDown = iptables -t nat -D POSTROUTING -s 10.0.60.2/32 -o ens3 -j MASQUERADE

These achieve few things:

1. Allow traffic to Wireguard port, so that your server can connect to establish the tunnel.
2. Allow new tcp and udp connections to Storj port (We are inserting the rule in the very top of the chain; it’s enough to make sure it’s before rule 6, which on oracle instances is reject; so use -I, not -A).
3. Port forward incoming Storj traffic to the VPN client.
4. Allow the now-forwarded traffic to pass from the public interface to the VPN interface
5. Allow outbound traffic from the VPN client out to the internet.
6. Perform NAT for traffic from the VPN client going to the internet

On the server, restart the wireguard service:

sudo systemctl restart wg-quick@wg0

On the client, restart the wireguard service:

service wireguard restart

using firewall-cmd

Instead of manually manipulating iptables, everything can be accomplished at a higher level of abstraction, and, arguably, more simply, using firewall-cmd.

In the wireguard config we don’t need any PreUp/PostDn commands:

On the server:

# cat /etc/wireguard/wg0.conf
[Interface]
PrivateKey = xxx
ListenPort = 51820
Address = 10.0.60.1/32

[Peer]
PublicKey = yyy
AllowedIPs = 10.0.60.2/32

On the client:

# cat /usr/local/etc/wireguard/oracle.conf
[Interface]
PrivateKey = zzz
Address = 10.0.60.2/32

[Peer]
PublicKey = nnn
AllowedIPs = 0.0.0.0/0
Endpoint = oracle_instance_public_ip:51820
PersistentKeepalive = 25

Note that we are setting AllowedIPs to 0.0.0.0/0. This is to ensure that all traffic originating from the node is routed through the VPN, particularly inadyn traffic that would update DDNS if you were using one. If you do not want this behavior, set only the server’s address in the AllowedIPs list. In this case, there skip enabling masquerading in the steps below.

Edit the configuration at the top, and run the following script.

This keeps the public interface in the public zone, and wg0 in the internal zone. That distinction matters: the public zone has masquerading enabled, while the WireGuard zone must not masquerade traffic that is being forwarded to the node, otherwise the node will see the WireGuard server address instead of the real remote peer address.

WG_CLIENT_IP=10.0.60.2
WG_CLIENT_PORT_EXT=28967
WG_CLIENT_PORT_INT=28967
WG_IFACE="wg0"
# On Oracle instances the default external zone is called "public", not "external"
EXT_ZONE="public"
WG_ZONE="internal"

# Make sure firewalld itself is enabled.
systemctl enable --now firewalld

# Keep SSH allowed before reloading the firewall.
firewall-cmd --permanent --zone="$EXT_ZONE" --add-service=ssh

# Allow WireGuard connections.
firewall-cmd --permanent --zone="$EXT_ZONE" --add-port=51820/udp

# Allow node connections.
firewall-cmd --permanent --zone="$EXT_ZONE" --add-port=$WG_CLIENT_PORT_EXT/udp
firewall-cmd --permanent --zone="$EXT_ZONE" --add-port=$WG_CLIENT_PORT_EXT/tcp

# Enable masquerading on the "$EXT_ZONE" zone oracle has configured. Predefined "external"
# zone would have had that enabled. This is not needed if you don't want to route all traffic
# that originates on the node through the tunnel, see the remark above.
firewall-cmd --permanent --zone="$EXT_ZONE" --add-masquerade

# Add forwarding rules for your node's port.
firewall-cmd --permanent --zone="$EXT_ZONE" --add-forward-port="port=$WG_CLIENT_PORT_EXT:proto=tcp:toport=$WG_CLIENT_PORT_INT:toaddr=$WG_CLIENT_IP"
firewall-cmd --permanent --zone="$EXT_ZONE" --add-forward-port="port=$WG_CLIENT_PORT_EXT:proto=udp:toport=$WG_CLIENT_PORT_INT:toaddr=$WG_CLIENT_IP"

# Keep the public interface in public and the WireGuard interface in internal.
# On Oracle instances the public interface is commonly enp0s6 or enp0s3.
firewall-cmd --permanent --zone="$EXT_ZONE" --remove-interface="$WG_IFACE" 2>/dev/null || true
firewall-cmd --permanent --zone="$WG_ZONE" --add-interface="$WG_IFACE"

# Do not enable intra-zone forwarding on public for this setup.
firewall-cmd --permanent --zone="$EXT_ZONE" --remove-forward 2>/dev/null || true

firewall-cmd --reload

# Review.
firewall-cmd --get-active-zones
firewall-cmd --info-zone="$EXT_ZONE"
firewall-cmd --info-zone="$WG_ZONE"

The last few commands should output something like this:

# firewall-cmd --get-active-zones
internal
  interfaces: wg0
public
  interfaces: enp0s6
# firewall-cmd --info-zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s6
  sources:
  services: dhcpv6-client ssh
  ports: 51820/udp 28967/udp 28967/tcp
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
    port=28967:proto=tcp:toport=28967:toaddr=10.0.60.2
    port=28967:proto=udp:toport=28967:toaddr=10.0.60.2
  source-ports:
  icmp-blocks:
  rich rules:
# firewall-cmd --info-zone=internal
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: wg0
  sources:
  services: cockpit dhcpv6-client mdns samba-client ssh
  ports:
  protocols:
  forward: no
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

At this point the client may already be able to route outbound traffic through the VPS:

# on the client
curl -4 http://ipinfo.io/ip

This should return the public IP of the VPS. The node should also see real remote peer addresses in its logs, not the WireGuard server address.

Note for AlmaLinux, Oracle Linux 9, and similar

Older Oracle Linux 8 systems with firewalld 0.9 can work with only the split-zone configuration above. For example, public owns the external interface with masquerading and forward-ports, internal owns wg0, and no custom policy is needed.

On newer firewalld/nftables systems, including Oracle Linux 9, simply assigning wg0 to the internal zone may not be enough for a full-tunnel client. You may see the tunnel handshake succeed, and the client may be able to ping the WireGuard server address, but outbound traffic through the VPS fails.

If that happens, add an explicit policy from the WireGuard zone to the public zone:

WG_ZONE="internal"
EXT_ZONE="public"
WG_POLICY="wg-to-public"

firewall-cmd --permanent --zone="$WG_ZONE" --add-forward
firewall-cmd --permanent --new-policy "$WG_POLICY" 2>/dev/null || true
firewall-cmd --permanent --policy "$WG_POLICY" --add-ingress-zone "$WG_ZONE"
firewall-cmd --permanent --policy "$WG_POLICY" --add-egress-zone "$EXT_ZONE"
firewall-cmd --permanent --policy "$WG_POLICY" --set-target ACCEPT
firewall-cmd --reload

firewall-cmd --info-zone="$WG_ZONE"
firewall-cmd --info-policy "$WG_POLICY"

The additional output should look like this:

# firewall-cmd --info-zone=internal
internal (active)
  interfaces: wg0
  forward: yes
  masquerade: no
# firewall-cmd --info-policy=wg-to-public
wg-to-public (active)
  priority: -1
  target: ACCEPT
  ingress-zones: internal
  egress-zones: public
  masquerade: no

Do not solve this by putting both the public interface and wg0 in the same public zone when that zone has masquerading enabled. That can make outbound traffic work, but it can also SNAT forwarded inbound connections as they leave toward wg0. In that case the node sees the WireGuard server address, for example 10.0.60.1, instead of the real remote peer address.

Final steps

In the config.yaml of the storage node modify the external address to point to your vps:

# the public address of the node, useful for nodes behind NAT
contact.external-address: sub.example.com:28967

[Re]start the node, and check the status page. It shall be now happily connected.

History