FreeBSD 12.0 VNET jail using bridge/epair and PF -

These configuration have been tested only on FreeBSD 12.0-RELEASE.

Description

I’ve been trying for almost half a day to gather informations around about FreeBSD Jails using the VNET/VIMAGE, and since the only informations about this subject are only in /usr/share/examples/jails. It seems there are 2 choices when considering the VNET jail approach:

using netgraph with the ngctl utility, and the jng helper in /usr/share/examples/jails
using bridges and epair interfaces to bridge networks over the public interface, and so to use jib in the same folder

Since I don’t know anything about netgraph, I’ve prefered to take time and put the correct configurations on the host and the jail for bridge/epair.

And the last piece was to use PF, since I think there are no reason my host or jail should be easily accessible over Internet.

Configurations

These configuration files only cover what I’ve modified or added.

The host

Utilities

install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/local/sbin/jib

/etc/sysctl.conf

# Tell PF not to care about the bridge since it will be only use in that VNET
# jail case on this host
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0

/etc/devfs.rules: I’ve also added the tun devices to the jail, later I’d like to install openvpn inside the jail.

[vnet_vpn=11]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add path 'bpf*' unhide
add path 'tun*' unhide
add path pf unhide
add path pflog unhide
add path pfsynv unhide

/etc/jail.conf

exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
vpn {
        host.hostname = "xxx";
        path = "/jails/xxx";

        vnet;
        vnet.interface = "e0b_xxx";
        exec.prestart += "/usr/local/sbin/jib addm xxx em0";
        exec.poststop += "/usr/local/sbin/jib destroy xxx";

        exec.consolelog = "/var/log/jail_xxx_console.log";
        mount.devfs;

        devfs_ruleset = "11";
        allow.set_hostname = 1;
        allow.sysvipc = 1;
}

/etc/rc.conf

pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"

jail_enable="YES"
jail_list="xxx"

/etc/pf.conf

ext_if="em0"

icmp_types="echoreq"

table <whitelist> persist file "/etc/whitelist.pf"
table <blacklist> persist file "/etc/blacklist.pf"

# options
set block-policy return
set loginterface $ext_if

set skip on lo

# scrub
scrub in

# filter rules
block in log
block in log quick on $ext_if from <blacklist> to any

pass out

antispoof quick for { lo }

pass in on $ext_if inet proto tcp from <whitelist> to ($ext_if)
pass in on $ext_if inet proto udp from <whitelist> to ($ext_if)

pass in inet proto icmp all icmp-type $icmp_types

The jail

/etc/rc.conf

hostname="xxx"

ifconfig_e0b_xxx="1.2.3.4 netmask 255.255.255.255 broadcast 1.2.3.4"
static_routes="net1"
route_net1="-net 1.2.5.6/32 -iface e0b_xxx"
defaultrouter="1.2.5.6"

pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"

sshd_enable="YES"

/etc/pf.conf

ext_if="e0b_xxx"

#tcp_services="{ 80, 443 }"
#udp_services="{ 80, 443 }"

icmp_types="echoreq"

table <whitelist> persist file "/etc/whitelist.pf"
table <blacklist> persist file "/etc/blacklist.pf"

# options
set block-policy return
set loginterface $ext_if

set skip on lo

# scrub
scrub in

# filter rules
block in log
block in log quick on $ext_if from <blacklist> to any

pass out

antispoof quick for { lo }

pass in on $ext_if inet proto tcp from <whitelist> to ($ext_if)
pass in on $ext_if inet proto udp from <whitelist> to ($ext_if)

pass in inet proto icmp all icmp-type $icmp_types

#pass in on $ext_if inet proto tcp from any to ($ext_if) port $tcp_services
#pass in on $ext_if inet proto udp from any to ($ext_if) port $udp_services

Configuration FreeBSD