Jails Networking

by Eric Fortis

Uxtely runs on two servers, each one has three jails. The jails for the databases and application servers communicate privately (dotted lines), while the reverse proxies listen on the internet.

Server A Server B Rev. Proxy App Server Database Rev. Proxy App Server Database

Each server has:

  • Two "virtual switches" (if_bridge)
    • ibridge is for replicating the database, and for the application servers to target the peer database.
    • xbridge is for passing incoming traffic to the reverse proxy jail, and outgoing traffic from all the jails.
  • An internal interface, inic, connected to ibridge.
  • An external interface, xnic, not directly connected to xbridge, but redirected through it.
  • Seven virtual cables with a VNIC at either end (epair)
    • Two jail-to-jail (orange lines)
    • Five jail-to-bridge
xnic  xnic Server A Server B  inic  inic xbridge ibridge ibridge nginx_j node_j pg_j .2.20 ngx_b .2.30 node_b .2.40 pg_b .3.20 ngx_node_a .3.30 ngx_node_b .4.30 node_pg_a .4.40 node_pg_b inode_b ipg_b inode_b ipg_b

VNIC Labels

Also, each server has four encrypted tunnels (spiped). The arrows point to the pipe's server-end.

spiped tunnels .56.31 .56.41 .56.30 .56.40 :5432 :5433 :5434 :5435 :5432 :5433 :5434 :5435

Load Balancing

In the DNS provider, there's an "A" record for each server. Therefore, they get load balanced free of charge by a round‑robin algorithm.

Cloudflare Multiple A Records for Load Balancing with DNS

To shut-down a server, remove its "A" record and wait for the TTL to expire. In Cloudflare®, that's 5 minutes by default (Auto TTL).

Lastly, this Health-Checking post explains how to connect to a specific server by its IP.


The configuration files (*.conf) are exactly the same in both servers. For that, they read server-specific settings from other files. For example, Server A's /etc/inic would have

The hardware and company-specific parts will be highlighted in green.


This rc.conf specifies the epairs and bridges to create when host boots up, and how to interconnect them.

About rc.conf's ifconfig syntax

Each of the next two lines rename an interface (bridge0). The first one shows how it's typed it in the shell, which is non-persistent across reboots. While the second one is for the rc.conf (persistent).

ifconfig bridge0 name xbridge

# Rename and Configure the NICs 
ifconfig_xnic="`/bin/cat /etc/xnic`"
ifconfig_inic="`/bin/cat /etc/inic` vlanhwtag 2222"

defaultrouter=`/bin/cat /etc/gateway`

# Create the Virtual Interfaces. e.g., ifconfig bridge0 create
# Rename the Bridges and VNICs ifconfig_bridge0_name=xbridge ifconfig_bridge1_name=ibridge ifconfig_epair0a_name=ngx_a ifconfig_epair0b_name=ngx_b ifconfig_epair1a_name=node_a ifconfig_epair1b_name=node_b ifconfig_epair2a_name=pg_a ifconfig_epair2b_name=pg_b ifconfig_epair3a_name=ngx_node_a ifconfig_epair3b_name=ngx_node_b ifconfig_epair4a_name=node_pg_a ifconfig_epair4b_name=node_pg_b ifconfig_epair5a_name=inode_a ifconfig_epair5b_name=inode_b ifconfig_epair6a_name=ipg_a ifconfig_epair6b_name=ipg_b # Enable the VNICs that we don't assign IPs to ifconfig_ngx_a=up ifconfig_node_a=up ifconfig_pg_a=up ifconfig_ngx_node_a=up ifconfig_node_pg_a=up ifconfig_inode_a=up ifconfig_ipg_a=up # Connect the NICs to the Bridges. e.g., ifconfig ibridge addm inic ifconfig_ibridge="
addm inic
addm inode_a
addm ipg_a
addm ngx_a
addm node_a
addm pg_a
# Services jail_enable=YES jail_reverse_stop=YES gateway_enable=YES pf_enable=YES


The jails boot up in the order they appear in this file. This file also specifies which VNICs the host will hand-over to the jails.

As PostgreSQL requires System V's shared memory, sysvshm provides and namespaces it to the database jail.

path = "/jails/$name";

devfs_ruleset = 4;

exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";

pg_j {
  sysvshm = "new";
  vnet.interface = pg_b, node_pg_b, ipg_b;

node_j {
  vnet.interface = node_b, ngx_node_b, inode_b, node_pg_a;

nginx_j {
  vnet.interface = ngx_b, ngx_node_a;

Jails rc.conf's

This section has the jail's boot-time configurations. These rc.conf files configure the VNICs, gateway, tunnels, and services.

Keep in mind that at this stage we are in the jail, so the file paths are rooted into the jail directory. For example, the hostname for the Nginx jail is at /jails/nginx_j/etc/hostname on the host.


hostname=`/bin/cat /etc/hostname`





hostname=`/bin/cat /etc/hostname`

ifconfig_inode_b=`/bin/cat /etc/ip_inode_b`/24


# Tunnel to Peer Database

spiped_pipe_N2P_source="[`/bin/cat /etc/ip_inode_b`]:5432"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5433"



hostname=`/bin/cat /etc/hostname`

ifconfig_ipg_b=`/bin/cat /etc/ip_ipg_b`/24


# Tunnels
spiped_pipes="N2P PGS PGC"

spiped_pipe_N2P_source="[`/bin/cat /etc/ip_ipg_b`]:5433"
spiped_pipe_N2P_target="[`/bin/cat /etc/ip_ipg_b`]:5432"

spiped_pipe_PGS_source="[`/bin/cat /etc/ip_ipg_b`]:5434"
spiped_pipe_PGS_target="[`/bin/cat /etc/ip_ipg_b`]:5432"

spiped_pipe_PGC_source="[`/bin/cat /etc/ip_ipg_b`]:5435"
spiped_pipe_PGC_target="[`/bin/cat /etc/ip_peer_ipg_b`]:5434"


Firewall (pf)

This firewall configuration denies all traffic by default and then allows what's needed. It assumes xnic is on a public IP, if not, remove its subnet from the <martians> table.

Allowed Incoming Traffic

Only the Nginx jail is open to the internet. SSHing to the host or jails is only possible from IPs listed in the <xpeers> table.

Rate Limits

If an IP connects 100 times within 10 seconds it'll be blocked. Not only from making new connections, but also from using established ones, as flush global terminates them.

Effectively, that IP gets added to the <ratelimit> table. Therefore, you can use a cron to reallow those IPs. For example, to remove IPs that are at least three minutes old from that table:

/sbin/pfctl -t ratelimit -T expire 180

How to exclude a company from rate-limits? Create a table with their IPs, and add a pass in quick rule before the block in quick one (think of quick as an early return). For instance, if Cloudflare® is proxying your traffic, populate a <bypass> table with their IPs.

pass in quick on xnic proto tcp from <bypass> to port 443
block in quick …

Allowed Outgoing Traffic

The Node jail can connect to Stripe® API and Fastmail® . As they need DNS services, Cloudflare® and Google® resolvers are allowed as well.

NTP is allowed via inic to private NTP servers .

How to deploy to these servers? Copy over to them with rsync. The orchestration server IP must be in the <xpeers> table.

How to generate TLS certificates? Like deploying, see my previous post: Isolated Let's Encrypt TLS Certificate Creation.

How to extract backups? rsync from the backup/orchestration server too.

The general answer to those questions is: initiate the connection from the orchestration server.


xbridge_net = "10.0.2/24"
nginx_j     = ""
node_j      = ""
pg_j        = ""

port_nginx  = "{ 443 80 }"
port_email  = "465"

resolvers   = "{ }"
priv_ntp    = "{ }"

table <ratelimit>
table <blocklist> file "/etc/ips_blocklist"  # DShield Daily
table <martians>  file "/etc/ips_martians"   # Bogon →
table <payments>  file "/etc/ips_stripe"     # Stripe API IPs →
table <email>     file "/etc/ips_fastmail"   # 66.111.4/24
table <xpeers>    file "/etc/ips_xnic_peers"
table <ipg_j>     file "/etc/ip_ipg_b"
table <inode_j>   file "/etc/ip_inode_b"
table <pgpeer>    file "/etc/ip_peer_ipg_b"
table <nodepeer>  file "/etc/ip_peer_inode_b"

# Normalization. Prevents IP Fragmentation Attacks and Inspection Evasion
scrub in all fragment reassemble no-df

# Translation
nat on xnic from $xbridge_net -> (xnic:0)
rdr on xnic proto tcp from any      to port $port_nginx -> $nginx_j
rdr on xnic proto tcp from <xpeers> to port 2220 -> $nginx_j port 22
rdr on xnic proto tcp from <xpeers> to port 2230 -> $node_j  port 22
rdr on xnic proto tcp from <xpeers> to port 2240 -> $pg_j    port 22

# Blockers
antispoof quick for xnic
block in quick on xnic \
      from { <blocklist> <ratelimit> <martians> no-route urpf-failed }
block all

# Nginx Incoming Traffic
pass in quick on xnic proto tcp from any \
     to port $port_nginx keep state \
     (max-src-conn-rate 100/10, overload <ratelimit> flush global)
pass out quick on xbridge proto tcp to $nginx_j port $port_nginx

# Tunnels
pass in  quick on ipg_a   proto tcp from <ipg_j>    to <pgpeer> port 5434
pass in  quick on inic    proto tcp from <pgpeer>   to <ipg_j>  port 5434
pass in  quick on inode_a proto tcp from <inode_j>  to <pgpeer> port 5433
pass in  quick on inic    proto tcp from <nodepeer> to <ipg_j>  port 5433
pass out quick on ibridge proto tcp from any        to any      port { 5434 5433 }

# Node.js Outgoing Traffic
pass in  on xbridge proto udp from $node_j to $resolvers port 53
pass in  on xbridge proto tcp from $node_j to <payments> port 443
pass in  on xbridge proto tcp from $node_j to <email>    port $port_email
pass out on xnic    proto udp from any     to $resolvers port 53
pass out on xnic    proto tcp from any     to <payments> port 443
pass out on xnic    proto tcp from any     to <email>    port $port_email

pass in  on xnic    proto tcp from <xpeers> to any          port 22
pass out on xbridge proto tcp from <xpeers> to $xbridge_net port 22

pass out on inic proto udp to $priv_ntp port 123

# Uncomment for updating FreeBSD and packages
# pass in  on xbridge from $xbridge_net
# pass out on xnic


Shawn Webb (2012) Virtually Networked FreeBSD Jails


Cost and Hardware

Each server is rented to Hivelocity® for about $1,400/year.

  • 6 Cores 3.3/4.5GHz E-2136
  • 32GB ECC DDR4
  • 2 × 480GB SSD
  • 20TB/mo at 1Gbps. 5 IPs. DDoS FENS
  • IPMI over VPN

How to simulate the infrastructure?

How to create the jails?

Sponsored by: