Index

Build your own FreeBSD ports and make packages out of them using jails, poudriere and portshaker

Nothing beats a good ol' custom pkg repository

December 2017.

Situation

You make custom ports of your applications. You would like to use pkg to install them on your production and development machines.

Solution

Build a poudriere server with portshaker!

Requirements:

Custom ports come in the form of a git port tree
Everything should run in a jail. Git has many dependencies you don't want on your host systems (I'm looking a you bash).
Everything should use ZFS as much as possible

Setting up the base system

Make sure the following kernel modules are loaded:

tmpfs
linux
linux64
nullfs
procfs
fdescfsu
linprocfs

Load them now:


# kldload tmpfs
# kldload linux
# kldload linux64
# kldload nullfs
# kldload procfs
# kldload fdescfsu
# kldload linprocfs

Make sure they're loaded on boot:


# sysrc -f /boot/loader.conf tmpfs_load="YES"
# sysrc -f /boot/loader.conf linux_load="YES"
# sysrc -f /boot/loader.conf nullfs_load="YES"
# sysrc -f /boot/loader.conf procfs_load="YES"
# sysrc -f /boot/loader.conf fdescfsu_load="YES"
# sysrc -f /boot/loader.conf linux64_load="YES"
# sysrc -f /boot/loader.conf linprocfs_load="YES"

Make sure jails can use IPC


# sysctl security.jail.sysvipc_allowed=1
# sysrc jail_sysvipc_allow=YES

Create a ZFS dataset for the poudriere system:


# zfs create zroot/poudriere

Create a jail, and make sure the jail is allowed to create build jails and manage its ZFS dataset. Here the relevant configuration for a ezjail jail:


# cat /usr/local/etc/ezjail/poudriere
[...]
export jail_poudriere_zfs_datasets="zroot/poudriere"
export jail_poudriere_parameters="children.max=20 allow.mount allow.mount.tmpfs allow.mount.devfs allow.mount.procfs allow.mount.zfs allow.mount.nullfs allow.mount.linprocfs allow.raw_sockets allow.socket_af allow.sysvipc allow.chflags enforce_statfs=1 ip6=inherit ip4=inherit"

Setting up poudriere in the jail

Install poudriere and dialog4ports:


# make -C /usr/ports/ports-mgmt/dialog4ports install clean
# make -C /usr/ports/ports-mgmt/poudriere install clean

Set the moundpoint of the ZFS dataset:


# zfs set mountpoint=/poudriere zroot/poudriere

Make sure the ZFS dataset for the ports and the distfiles exists:


# zfs create zroot/poudriere/ports
# zfs create zroot/poudriere/ports/distfiles

Configure poudriere in /usr/local/etc/poudriere.conf:


ZPOOL: zroot
ZROOTFS: /poudriere
FREEBSD_HOST: "ftp.XXX.freebsd.org"
DISTFILES_CACHE: /usr/local/poudriere/ports/distfiles
NO_LINUX: 1

Create a port tree, but don't populate it. It will be done by portshaker later:


# poudriere ports -c -F -f none -M /usr/local/poudriere/ports/main -p main

Create a build jail:


# poudriere jail -c -j 111x64 -v 11.1-RELEASE -a amd64
# poudriere jail -l
JAILNAME VERSION          ARCH  METHOD TIMESTAMP           PATH
111x64   11.1-RELEASE amd64 ftp    2017-12-08 14:58:09 /usr/local/poudriere/jails/111x64

Hehe, we have a jail in a jail.

Setting up portshaker

We're going to build a port tree consisting of the regular FreeBSD main one (using portsnap) merged with our own (using git).

Portskaker is an awesome tool. It will merge different port tree, handling conflicts intelligently, and even merging the UIDs/GIDs files properly.

This example will only build a single port tree, but you can configure as many as you want.

Install portshaker:


# make -C /usr/ports/ports-mgmt/portshaker install clean

Install git:


# Make -C /usr/ports/devel/git install clean

Write /usr/local/etc/portshaker.conf:


mirror_base_dir="/var/cache/portshaker"
ports_trees="main"
main_ports_tree="/usr/local/poudriere/ports/main"
main_merge_from="portsnap custom"

Write the portshaker tree source for portsnap, in /usr/local/etc/portshaker.d/portsnap:


#!/bin/sh
. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
fi
shift
method="portsnap"
run_portshaker_command $*

Write the portshaker tree source for custom, in /usr/local/etc/portshaker.d/custom:


#!/bin/sh
. /usr/local/share/portshaker/portshaker.subr
if [ "$1" != '--' ]; then
  err 1 "Extra arguments"
fi
shift
method="git"
git_clone_uri="git@example.com:SomeGroup/freebsd-ports.git"
git_branch=master
run_portshaker_command $*

Make sure both files are executable.

Fetch both port trees:


# portshaker -U

You should see portsnap and a git clone being executed.

Merge the port trees:


# portshaker -M

You should have a complete port tree in /usr/local/poudriere/ports/main/. Look for your custom ports and check that they're here.

Setting up nginx

We want to use poudriere's web interface.

Install nginx:


# make -C /usr/ports/www/nginx install clean

Add a server in /usr/local/etc/nginx/nginx.conf


server {
    listen 80 default;
    server_name _;
    root /usr/local/share/poudriere/html;

    location /data {
        alias /usr/local/poudriere/data/logs/bulk;
        autoindex on;
    }

    location /packages {
        root /usr/local/poudriere/data;
        autoindex on;
    }
}

That's it! No CGI, no proxy, no whatever. Just plain files to serve.

Test it all

Write a list of package into a pkglist file (/usr/local/etc/poudriere.d/someports-pkglist):


www/nginx
sysutils/tmux

Set the port options:


poudriere options -j 111x64 -p main -f /usr/local/etc/poudriere.d/someports-pkglist

Start the build:


# poudriere bulk -J 8 -j 110x64 -p main -f /usr/local/etc/poudriere.d/someports-pkglist -v -v -v
[00:00:00] ====>> Creating the reference jail... done
[00:00:00] ====>> Mounting system devices for 111x64-main
[00:00:00] ====>> Mounting ports/packages/distfiles
[00:00:00] ====>> Using packages from previously failed build
[00:00:00] ====>> Mounting packages from: /usr/local/poudriere/data/packages/111x64-main
[00:00:00] ====>> Copying /var/db/ports from: /usr/local/etc/poudriere.d/111x64-options
[00:00:00] ====>> Appending to make.conf: /usr/local/etc/poudriere.d/111x64-make.conf
/etc/resolv.conf -> /usr/local/poudriere/data/.m/111x64-main/ref/etc/resolv.conf
[00:00:00] ====>> Starting jail 111x64-main
[00:00:00] ====>> Logs: /usr/local/poudriere/data/logs/bulk/111x64-main/2017-12-08_16h59m32s
[00:00:00] ====>> Loading MOVED
[00:00:01] ====>> Calculating ports order and dependencies
[...]
Creating repository in /tmp/packages: 100%
Packing files for repository: 100%
[00:16:07] ====>> Committing packages to repository
[00:16:07] ====>> Removing old packages
[00:16:07] ====>> Built ports: ports-mgmt/pkg devel/automake-wrapper devel/autoconf-wrapper print/indexinfo devel/pkgconf devel/pcre devel/gettext-runtime www/nginx lang/perl5.24 devel/gettext-tools devel/p5-Locale-gettext devel/gmake misc/help2man print/texinfo devel/m4 devel/libtool devel/autoconf devel/automake devel/libevent sysutils/tmux
[111x64-main] [2017-12-08_16h59m32s] [committing:] Queued: 20 Built: 20 Failed: 0  Skipped: 0  Ignored: 0  Tobuild: 0   Time: 00:16:07
[00:16:07] ====>> Logs: /usr/local/poudriere/data/logs/bulk/111x64-main/2017-12-08_16h59m32s
[00:16:07] ====>> Cleaning up
[00:16:07] ====>> Unmounting file systems

Connect to the web interface:
Look at the build lists:

Look at the build report:

Use your pkg repository on other systems

Now that our packages are built, let's use them on other systems:

Make sure the config directory exists:


# mkdir -p /usr/local/etc/pkg/repos

Write a pkg repository config file in /usr/local/etc/pkg/repos/poudriere.conf:


poudriere: {
    url: "http://pkg.example.com/packages/111x64-someports/",
    mirror_type: "http",
    enabled: yes,
    priority: 100
}

Writing a secure version of this file is left as an exercise for the reader.

Update the local database:


# pkg update

Install your custom ports:


# pkg install custom-port

Enjoy your new custom ports infrastructure!