FreeBSD 12.0 VNET jail using bridge/epair and PF


These configuration have been tested only on FreeBSD 12.0-RELEASE.

Description

I’ve been trying for almost half a day to gather informations around about FreeBSD Jails using the VNET/VIMAGE, and since the only informations about this subject are only in /usr/share/examples/jails. It seems there are 2 choices when considering the VNET jail approach:

  • using netgraph with the ngctl utility, and the jng helper in /usr/share/examples/jails
  • using bridges and epair interfaces to bridge networks over the public interface, and so to use jib in the same folder

Since I don’t know anything about netgraph, I’ve prefered to take time and put the correct configurations on the host and the jail for bridge/epair.

And the last piece was to use PF, since I think there are no reason my host or jail should be easily accessible over Internet.

Configurations

These configuration files only cover what I’ve modified or added.

The host

  • Utilities

    install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/local/sbin/jib

  • /etc/sysctl.conf

    # Tell PF not to care about the bridge since it will be only use in that VNET
    # jail case on this host
    net.link.bridge.pfil_bridge=0
    net.link.bridge.pfil_onlyip=0
    net.link.bridge.pfil_member=0

  • /etc/devfs.rules: I’ve also added the tun devices to the jail, later I’d like to install openvpn inside the jail.

    [vnet_vpn=11]
    add include $devfsrules_hide_all
    add include $devfsrules_unhide_basic
    add include $devfsrules_unhide_login
    add include $devfsrules_jail
    add path 'bpf*' unhide
    add path 'tun*' unhide
    add path pf unhide
    add path pflog unhide
    add path pfsynv unhide

  • /etc/jail.conf

    exec.clean;
    exec.system_user = "root";
    exec.jail_user = "root";
    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";
    vpn {
            host.hostname = "xxx";
            path = "/jails/xxx";
    
            vnet;
            vnet.interface = "e0b_xxx";
            exec.prestart += "/usr/local/sbin/jib addm xxx em0";
            exec.poststop += "/usr/local/sbin/jib destroy xxx";
    
            exec.consolelog = "/var/log/jail_xxx_console.log";
            mount.devfs;
    
            devfs_ruleset = "11";
            allow.set_hostname = 1;
            allow.sysvipc = 1;
    }

  • /etc/rc.conf

    pf_enable="YES"
    pf_rules="/etc/pf.conf"
    pflog_enable="YES"
    
    jail_enable="YES"
    jail_list="xxx"

  • /etc/pf.conf

    ext_if="em0"
    
    icmp_types="echoreq"
    
    table <whitelist> persist file "/etc/whitelist.pf"
    table <blacklist> persist file "/etc/blacklist.pf"
    
    # options
    set block-policy return
    set loginterface $ext_if
    
    set skip on lo
    
    # scrub
    scrub in
    
    # filter rules
    block in log
    block in log quick on $ext_if from <blacklist> to any
    
    pass out
    
    antispoof quick for { lo }
    
    pass in on $ext_if inet proto tcp from <whitelist> to ($ext_if)
    pass in on $ext_if inet proto udp from <whitelist> to ($ext_if)
    
    pass in inet proto icmp all icmp-type $icmp_types

The jail

  • /etc/rc.conf

    hostname="xxx"
    
    ifconfig_e0b_xxx="1.2.3.4 netmask 255.255.255.255 broadcast 1.2.3.4"
    static_routes="net1"
    route_net1="-net 1.2.5.6/32 -iface e0b_xxx"
    defaultrouter="1.2.5.6"
    
    pf_enable="YES"
    pf_rules="/etc/pf.conf"
    pflog_enable="YES"
    
    sshd_enable="YES"

  • /etc/pf.conf

    ext_if="e0b_xxx"
    
    #tcp_services="{ 80, 443 }"
    #udp_services="{ 80, 443 }"
    
    icmp_types="echoreq"
    
    table <whitelist> persist file "/etc/whitelist.pf"
    table <blacklist> persist file "/etc/blacklist.pf"
    
    # options
    set block-policy return
    set loginterface $ext_if
    
    set skip on lo
    
    # scrub
    scrub in
    
    # filter rules
    block in log
    block in log quick on $ext_if from <blacklist> to any
    
    pass out
    
    antispoof quick for { lo }
    
    pass in on $ext_if inet proto tcp from <whitelist> to ($ext_if)
    pass in on $ext_if inet proto udp from <whitelist> to ($ext_if)
    
    pass in inet proto icmp all icmp-type $icmp_types
    
    #pass in on $ext_if inet proto tcp from any to ($ext_if) port $tcp_services
    #pass in on $ext_if inet proto udp from any to ($ext_if) port $udp_services