How to use jails with epair with FreeBSD 9.0

Introduction

System used:

FreeBSD testjails 9.0-RELEASE FreeBSD 9.0-RELEASE #1: Thu Jan 19 10:47:03 CET 2012     root@testjails:/usr/obj/usr/src/sys/JAILKERNEL  i386

Build and install a kernel including VIMAGE, epair and briged interfaces

Configure the kernel: (/usr/src/sys/i386/conf/JAILKERNEL)

include GENERIC

cpu             I686_CPU
ident           JAILKERNEL

# Virtual networking for jail
options         VIMAGE
device          epair
device          if_bridge

# The nullFS to mount local directory
options         NULLFS

Make and install the kernel:

cd /usr/src
make buildkernel KERNCONF=JAILKERNEL
make installkernel KERNCONF=JAILKERNEL
reboot

Create the jail's environement

mkdir /jails/somejail/
cd /usr/src
make buildworld
make installworld DESTDIR=/jails/somejail/
make distribution DESTDIR=/jails/somejail/
echo 'hostname="testjail.example.com"' >> /jails/somejail/etc/rc.conf
echo 'nameserver 8.8.8.8"' >> /jails/somejail/etc/resolv.conf
mkdir /jails/somejail/usr/ports

Start and use the jail manually

Start the jail:

# jail -c vnet name=testjail host.hostname=testjail path=/jails/somejail persist
# jls
   JID  IP Address      Hostname                      Path
     1  -               testjail                      /jails/somejail

Create the virtual ethernet cable:

# ifconfig epair0 create
epair0a
# ifconfig epair0b vnet testjail

Create the bridge:

# ifconfig bridge0 create
# ifconfig bridge0 addm em0 addm epair0a
# ifconfig bridge0 up

Configure the network on both sides:

# ifconfig em0 192.168.1.1/24
# ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 14183
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
# jexec testjail ifconfig epair0b 192.168.1.2/24
# jexec testjail ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        inet6 fe80::c0:a4ff:fe00:60b%epair0b prefixlen 64 scopeid 0x2
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

Test the connection:

# jexec testjail ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.3.223): 56 data bytes
64 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=0.795 ms

--- 192.168.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.795/0.795/0.795/0.000 ms

# jexec testjail route add default 192.168.1.250
add net default: gateway 192.168.1.250

# jexec testjail ping -c 1 example.net
PING example.net (192.0.43.10): 56 data bytes
64 bytes from 192.0.43.10: icmp_seq=0 ttl=243 time=92.568 ms

--- example.net ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 92.568/92.568/92.568/0.000 ms

Stop the jail:

# jail -r testjail
enceinte# jls
   JID  IP Address      Hostname                      Path

Clean the network:

# ifconfig epair0a destroy
# ifconfig bridge0 destroy

Start the jail automatically

As of 2011/08/01, /etc/rc.d/jail has problems using the new jail command (v2), so you must patch it if you want to use vnets:
http://www.freebsd.org/cgi/query-pr.cgi?pr=142972
Also, you need this patch to be able to put the epair interface in the jail's vnet after it is created, but before starting the jail's rc.script (via earlypoststart entries in rc.conf).

My original /etc/rc.d/jail script was revision 1.47.

Here is the modified patch that worked for me:
fbsd9_rcdjail147_vnet_patch

Then, in /etc/rc.conf:

#
# Jails networking
#
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0"

#
# Jails configuration
#
jail_enable="YES"
jail_v2_enable="YES"
jail_list="testjail"

jail_testjail_name="testjail"
jail_testjail_hostname="testjail.example.net"
jail_testjail_devfs_enable="YES"
jail_testjail_rootdir="/jails/somejail"
jail_testjail_vnet_enable="YES"
jail_testjail_exec_prestart0="ifconfig epair0 create"
jail_testjail_exec_prestart1="ifconfig bridge0 addm epair0a"
jail_testjail_exec_prestart2="ifconfig epair0a up"
jail_testjail_exec_earlypoststart0="ifconfig epair0b vnet testjail"
jail_testjail_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_testjail_exec_afterstart1="ifconfig epair0b 192.168.1.2/24 up"
jail_testjail_exec_afterstart2="route add default 192.168.1.250"
jail_testjail_exec_afterstart3="/bin/sh /etc/rc"
jail_testjail_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_testjail_exec_poststop1="ifconfig epair0a destroy"

Then:

# /etc/rc.d/jail onestart testjail
Configuring jails:.
Starting jails:epair0a
add net default: gateway 192.168.1.250
Setting hostname: testjail.example.com.
Creating and/or trimming log files.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout
Clearing /tmp (X related).
Updating motd:.
Starting cron.

Thu Jan 19 11:45:24 UTC 2012
 testjail.example.net.
testjails# jls
   JID  IP Address      Hostname                      Path
     1  -               testjail.example.com          /jails/somejail
# jexec testjail ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        inet6 fe80::c0:a4ff:fe00:70b%epair0b prefixlen 64 scopeid 0x2
        inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
# jexec testjail ping -c 1 example.net
PING example.net (192.0.43.10): 56 data bytes
64 bytes from 192.0.43.10: icmp_seq=0 ttl=243 time=92.799 ms

--- example.net ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 92.799/92.799/92.799/0.000 ms

ZEWAREN.NET

Primary links