ZEWAREN.NET

In this series of blog posts, we're going to create a Android build server for continuous integration on FreeBSD.

• Part one will explain how to build Android APKs using Gradle on FreeBSD using the Linux emulation.
• Part two (this post) will explain how to configure Gitlab-CI to be able to run builds automatically for each commit.
• Part three will explain how to configure Jenkins to be able to run builds and email the APKs to people.

Step 1: Install, configure and register the Gitlab-CI runner

Most of the information of that step comes from the original documentation.

Create a nice user for the runner:

# pw group add -n gitlab-runner
# pw user add -n gitlab-runner -g gitlab-runner -s /usr/local/bin/bash
# mkdir /home/gitlab-runner
# chown gitlab-runner:gitlab-runner /home/gitlab-runner

Download the runner executable (or build it yourself (see the end of the post)), and mark it as such:

# fetch -o /usr/local/bin/gitlab-ci-multi-runner https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-ci-multi-runner-freebsd-amd64
# chmod +x /usr/local/bin/gitlab-ci-multi-runner

Create and chmod the log file:

# touch /var/log/gitlab_runner.log && sudo chown gitlab-runner:gitlab-runner /var/log/gitlab_runner.log

Create the rc-script (/usr/local/etc/rc.d/gitlab_runner):

#!/bin/sh
# PROVIDE: gitlab_runner
# REQUIRE: DAEMON NETWORKING
# BEFORE:
# KEYWORD:

. /etc/rc.subr

name="gitlab_runner"
rcvar="gitlab_runner_enable"

load_rc_config $name

user="gitlab-runner"
user_home="/home/gitlab-runner"
command="/usr/local/bin/gitlab-ci-multi-runner run"
pidfile="/var/run/${name}.pid"

start_cmd="gitlab_runner_start"
stop_cmd="gitlab_runner_stop"
status_cmd="gitlab_runner_status"

gitlab_runner_start()
{
    export USER=${user}
    export HOME=${user_home}
    if checkyesno ${rcvar}; then
        cd ${user_home}
        /usr/sbin/daemon -u ${user} -p ${pidfile} ${command} > /var/log/gitlab_runner.log 2>&1
    fi
}

gitlab_runner_stop()
{
    if [ -f ${pidfile} ]; then
        kill `cat ${pidfile}`
    fi
}

gitlab_runner_status()
{
    if [ ! -f ${pidfile} ] || kill -0 `cat ${pidfile}`; then
        echo "Service ${name} is not running."
    else
        echo "${name} appears to be running."
    fi
}

run_rc_command $1

Set the rc script as being executable:

chmod +x /usr/local/etc/rc.d/gitlab_runner

Register the runner with Gitlab-CI:

# sudo -u gitlab-runner -H /usr/local/bin/gitlab-ci-multi-runner register

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/ci):
https://gitlab.example.com/ci

Please enter the gitlab-ci token for this runner:
sqzGW7_PgmarZ3FsSsUa

Please enter the gitlab-ci description for this runner:
[androidbuild]: Example Android build machine

Please enter the gitlab-ci tags for this runner (comma separated):
android
Registering runner... succeeded                     runner=sqzGW7_T

Please enter the executor: docker-ssh+machine, docker, docker-ssh, parallels, shell, ssh, virtualbox, docker+machine:
shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Enable the runner service and start it:

# sysrc -f /etc/rc.conf "gitlab_runner_enable=YES" 
# service gitlab_runner start

Step 2: add the CI configuration to your Android project

Create a file named .gitlab-ci.yml and configure it to your needs.

before_script:
  - export LD_LIBRARY_PATH="/home/androidbuild/android-sdk-linux/build-tools/23.0.2/lib/"
  - export ANDROID_HOME="/home/androidbuild/android-sdk-linux/"

prod:
  only:
    - master
  script:
    - ./gradlew assembleRelease
  artifacts:
    paths:
      - Example/build/outputs/
    expire_in: 1 week

This example file will trigger a build when branch master is updaded, and it will save the artifacts for one week.

Bonus: compiling the runner ourselves

If there's something I don't like, it's downloading executables from the internet when I can build them myself.
Let's build that Gitlab runner ourselves.

Install golang:

# make -C /usr/ports/lang/go install clean

Create a working go directory structure:

% mkdir -p ~/gowork/src/gitlab.com/gitlab-org/
% cd ~/gowork/src/gitlab.com/gitlab-org/
% setenv GOPATH ~/gowork
% setenv PATH "${PATH}:${GOPATH}/bin"

Get the code from Gitlab (checkout the version you want to build):

% git clone https://gitlab.com/gitlab-org/gitlab-ci-multi-runner.git
% cd gitlab-ci-multi-runner/
% git checkout tags/v1.5.2

Build:

% gmake deps docker
% gmake build_simple
% file "out/binaries/gitlab-ci-multi-runner"
out/binaries/gitlab-ci-multi-runner: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, not stripped

Install (as root):

# install -m 755 out/binaries/gitlab-ci-multi-runner /usr/local/bin/gitlab-ci-multi-runner

Done.

Situation

You built some IoT device running a Python compliant OS. You want to be able to plug it anywhere, and be able to find it in the network view of Windows (or the equivalent thing of other systems).

Solution: SSDP/UPnP

I've created some code that implements a SSDP server and a HTTP server, in order to notify the network that a device is here.

The base of this code was the SSDP module of coherence.
I took it, converted it to Python 3, and kept only the interesting parts for this project: the parts that responds to `MSEARCH` queries.

The code is attached to this post, and is available as a mirror on Github.

Technical details

First, a SSDP client will try to search devices on the network by issuing a MSEARCH * query:

    M-SEARCH * HTTP/1.1
    HOST:239.255.255.250:1900
    ST:upnp:rootdevice
    MX:2
    MAN:"ssdp:discover"

lib/ssdp.py will handle it and reply with a 200:

    HTTP/1.1 200 OK
    SERVER: ZeWaren example SSDP Server
    LOCATION: http://192.168.4.207:8088/jambon-3000.xml
    USN: uuid:e427ce1a-3e80-43d0-ad6f-89ec42e46363::upnp:rootdevice
    CACHE-CONTROL: max-age=1800
    EXT: 
    last-seen: 1477147409.432466
    ST: upnp:rootdevice
    DATE: Sat, 22 Oct 2016 14:44:26 GMT

The client will then fetch the device description:

    GET /jambon-3000.xml HTTP/1.1
    Cache-Control: no-cache
    Connection: Keep-Alive
    Pragma: no-cache
    Accept: text/xml, application/xml
    User-Agent: FDSSDP
    Host: 192.168.4.207:8088

And lib/upnp_http_server.py will build and serve that file:

HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.4.3
Date: Sat, 22 Oct 2016 14:44:26 GMT
Content-type: application/xml

<root>
    <specVersion>
        <major>1</major>
        <minor>0</minor>
    </specVersion>
    <device>
        <deviceType>urn:schemas-upnp-org:device:Basic:1</deviceType>
        <friendlyName>Jambon 3000</friendlyName>
        <manufacturer>Boucherie numrique SAS</manufacturer>
        <manufacturerURL>http://www.boucherie.example.com/</manufacturerURL>
        <modelDescription>Jambon Appliance 3000</modelDescription>
        <modelName>Jambon</modelName>
        <modelNumber>3000</modelNumber>
        <modelURL>http://www.boucherie.example.com/en/prducts/jambon-3000/</modelURL>
        <serialNumber>JBN425133</serialNumber>
        <UDN>uuid:e427ce1a-3e80-43d0-ad6f-89ec42e46363</UDN>
        <serviceList>
            <service>
                <URLBase>http://xxx.yyy.zzz.aaaa:5000</URLBase>
                <serviceType>urn:boucherie.example.com:service:Jambon:1</serviceType>
                <serviceId>urn:boucherie.example.com:serviceId:Jambon</serviceId>
                <controlURL>/jambon</controlURL>
                <eventSubURL/>
                <SCPDURL>/boucherie_wsd.xml</SCPDURL>
            </service>
        </serviceList>
        <presentationURL>http://192.168.4.207:5000/</presentationURL>
    </device>
</root>

The client now knows (nearly) everything about the device.

Relevant links

• UPnP™ Device Architecture 1.1 (UPnP forum, October 15, 2008): http://upnp.org/specs/arch/UPnP-arch-DeviceArchitecture-v1.1.pdf
• Coherence - DLNA/UPnP framework for the digital living: https://pypi.python.org/pypi/Coherence

Introduction

We want to be able to use GDB to debug python code efficiently.

Let's say we have the following code:

from threading import Event
import random
from time import sleep

def blocking_function_one():
	while True:
		sleep(1)

def blocking_function_two():
	e = Event()
	e.wait()

if random.random() > 0.5:
	blocking_function_one()
else:
	blocking_function_two()

That code will block, and since it doesn't output anything, we have no way of knowing if we went into blocking_function_one or blocking_function_two. Or do we?

For reference, I'm running a 10.2-RELEASE:

# uname -a
FreeBSD bsdlab 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Step 1: installing a debug version of python

We're going to work in a separate directory, in order not to alter our installation. If we need this in production, we want to be able to let the system as clean as possible when we leave.

# mkdir /usr/local/python-debug

Build python 3.4 from the port collection and set it to be installed in the directory we just created:

# cd /usr/ports/lang/python34
# make install PREFIX=/usr/local/python-debug OPTIONS_FILE_SET+=DEBUG BATCH=1

Normally we would have used NO_PKG_REGISTER=1 to install the package without registering it on the system. Unfortunately, this option is not working anymore (see bug https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=182347).

So, let's copy the files ourselves:

# cp -r work/stage/usr/local/python-debug/* /usr/local/python-debug/

Let's try to run that new python installation:

# setenv LD_LIBRARY_PATH /usr/local/python-debug/lib
# /usr/local/python-debug/bin/python3.4dm
Python 3.4.5 (default, Sep  4 2016, 00:42:59)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>>

The "d" in "dm" means "debug".

Building python also produced an important file that we need to save for later before cleaning the work tree:

# cp work/Python-3.4.5/python-gdb.py ~/

Step 2: building a python aware GDB

Build GDB from the port collection:

• Making sure the python extensions are enabled
• Telling the configure script where to find our python installation
• Telling the configure and build scripts where to find the relevant headers and libraries

# ln -s python3.4 /usr/local/python-debug/bin/python
# cd /usr/ports/devel/gdb
# make PREFIX=/usr/local/python-debug \
OPTIONS_FILE_SET+=PYTHON \
PYTHON_CMD=/usr/local/python-debug/bin \
BATCH=1 \
CFLAGS+="-I/usr/local/python-debug/include -L/usr/local/python-debug/lib" \
CXXFLAGS+="-I/usr/local/python-debug/include -L/usr/local/python-debug/lib"

Also copy that installation manually to our special directory:

# cp -r work/stage/usr/local/python-debug/* /usr/local/python-debug/

Let's check that it's working and has the python extensions:

# /usr/local/python-debug/bin/gdb
GNU gdb (GDB) 7.11.1 [GDB v7.11.1 for FreeBSD]
[...]
(gdb) python
>import gdb
>end
(gdb)

Step 3: wire it all together

Now we have:

• A version of Python that integrates debug information.
• A version of GDB that can run higher level GDB scripts written in Python.
• A python-gdb script to add commands and macros.

Copy the GDB script somewhere where Python can load it:

# mkdir ~/.python_lib
# mv ~/python-gdb.py ~/.python_lib/python34_gdb.py

Let's run our stupid blocking script:

# setenv PATH "/usr/local/python-debug/bin/:${PATH}"
# python where-am-i-blocking.py
[blocked]

In another shell, find the PID of the script, and attach GDB there.

# ps auxw | grep python
root     24226   0.0  0.7  48664  15492  3  I+    3:00AM     0:00.13 python where-am-i-blocking.py (python3.4)
root     24235   0.0  0.1  18824   2004  4  S+    3:00AM     0:00.00 grep python

# setenv PATH "/usr/local/python-debug/bin/:${PATH}"
# gdb python 24226
GNU gdb (GDB) 7.11.1 [GDB v7.11.1 for FreeBSD]
[...]
[Switching to LWP 100160 of process 24226]
0x00000008018e3f18 in _umtx_op () from /lib/libc.so.7
(gdb)

Load the GDB python script:

(gdb) python
>import sys
>sys.path.append('/root/.python_lib')
>import python34_gdb
>end

The python macros are now loaded:

(gdb) py
py-bt               py-down             py-locals           py-up               python-interactive
py-bt-full          py-list             py-print            python

Let's see where we are:

(gdb) py-bt
Traceback (most recent call first):
  <built-in method acquire of _thread.lock object at remote 0x80075c2a8>
  File "/usr/local/python-debug/lib/python3.4/threading.py", line 290, in wait
    waiter.acquire()
  File "/usr/local/python-debug/lib/python3.4/threading.py", line 546, in wait
    signaled = self._cond.wait(timeout)
  File "where-am-i-blocking.py", line 11, in blocking_function_two
    e.wait()
  File "where-am-i-blocking.py", line 16, in <module>
    blocking_function_two()

We're in blocking_function_two.

Let's check the wait's frame local variables:

(gdb) bt
#0  0x00000008018e3f18 in _umtx_op () from /lib/libc.so.7
#1  0x00000008018d3604 in sem_timedwait () from /lib/libc.so.7
#2  0x0000000800eb0421 in PyThread_acquire_lock_timed (lock=0x802417590, microseconds=-1, intr_flag=1) at Python/thread_pthread.h:352
#3  0x0000000800eba84f in acquire_timed (lock=0x802417590, microseconds=-1) at ./Modules/_threadmodule.c:71
#4  0x0000000800ebab82 in lock_PyThread_acquire_lock (self=0x80075c2a8, args=(), kwds=0x0) at ./Modules/_threadmodule.c:139
#5  0x0000000800cfa963 in PyCFunction_Call (func=<built-in method acquire of _thread.lock object at remote 0x80075c2a8>, arg=(), kw=0x0)
    at Objects/methodobject.c:99
#6  0x0000000800e31716 in call_function (pp_stack=0x7fffffff5a00, oparg=0) at Python/ceval.c:4237
#7  0x0000000800e29fc0 in PyEval_EvalFrameEx (
    f=Frame 0x80245d738, for file /usr/local/python-debug/lib/python3.4/threading.py, line 290, in wait (self=<Condition(_lock=<_thread.l
ock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object a
t remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>, timeout=No
ne, waiter=<_thread.lock at remote 0x80075c2a8>, saved_state=None, gotit=False), throwflag=0) at Python/ceval.c:2838
[...]
#25 0x0000000800eb4e86 in run_file (fp=0x801c1e140, filename=0x802418090 L"where-am-i-blocking.py", p_cf=0x7fffffffe978)
    at Modules/main.c:319
#26 0x0000000800eb3ab7 in Py_Main (argc=2, argv=0x802416090) at Modules/main.c:751
#27 0x0000000000400cae in main (argc=2, argv=0x7fffffffeaa8) at ./Modules/python.c:69

(gdb) frame 7
#7  0x0000000800e29fc0 in PyEval_EvalFrameEx (
    f=Frame 0x80245d738, for file /usr/local/python-debug/lib/python3.4/threading.py, line 290, in wait (self=<Condition(_lock=<_thread.lock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object at remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>, timeout=None, waiter=<_thread.lock at remote 0x80075c2a8>, saved_state=None, gotit=False), throwflag=0) at Python/ceval.c:2838
2838                res = call_function(&sp, oparg);

(gdb) py-locals
self = <Condition(_lock=<_thread.lock at remote 0x80075c510>, _waiters=<collections.deque at remote 0x800b0e9d8>, release=<built-in method release of _thread.lock object at remote 0x80075c510>, acquire=<built-in method acquire of _thread.lock object at remote 0x80075c510>) at remote 0x800a6fc88>
timeout = None
waiter = <_thread.lock at remote 0x80075c2a8>
saved_state = None
gotit = False

If you don't want to or can't attach to the running process, you can do the same thing with a core dump:

# gcore 24226
# gdb python core.24226

If you don't want to add the lib directory to the path of python everytime you use GDB, add it to your profile's GDB init script:

cat > ~/.gdbinit <<EOF
python
import sys
sys.path.append('/root/.python_lib')
end
EOF

You'll only need to import the module (python import python34_gdb) and you'll be good to go.

More ressources

• Python documentation -> Python developer's guide -> 22. gdb Support (https://docs.python.org/devguide/gdb.html).
• Python wiki -> DebuggingWithGdb (https://wiki.python.org/moin/DebuggingWithGdb).
• Misspent blog -> Debugging C/C++ and CPython using GDB 7’s new Python extension support (https://misspent.wordpress.com/2012/03/24/debugging-cc-and-cpython-using-gdb-7s-new-python-extension-support/).
• SKGaur's Blog -> Python-Debugging (https://sumitkgaur.wordpress.com/2014/05/13/python-debugging/).

Bonus problem: loading Debian's libc's debug info on a armhf

I've done the exact same thing on a Beagle Bone Black system running Debian.

Unfortunately GDB was complaining that the stack was corrupt.

# gdb /usr/local/opt/python-3.4.4/bin/python3.4dm core.18513
GNU gdb (GDB) 7.11
[...]
Reading symbols from /usr/local/opt/python-3.4.4/bin/python3.4dm...done.
[New LWP 18513]
[New LWP 18531]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
Core was generated by `python3.4dm'.
#0  0xb6f7d7e0 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
[Current thread is 1 (Thread 0xb6fac000 (LWP 18513))]
(gdb) bt
#0  0xb6f7d7e0 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1  0xb6f7d7d4 in recv () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2  0xb64ae6f8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

A search on the internet indicated that it was because I was missing package libc6-dbg, but that package was installed.

# apt-get install libc6-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6-dbg is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

The problem was that using my custom installation directory had GDB look for these files in the wrong place.

(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/local/opt/python-3.4.4/lib/debug".

Setting that variable in the init file solves the problem:

cat >> ~/.gdbinit <<EOF
set debug-file-directory /usr/lib/debug
EOF

[Current thread is 1 (Thread 0xb6fac000 (LWP 18513))]
(gdb) bt
#0  0xb6f7d7e0 in recv () at ../sysdeps/unix/syscall-template.S:82
#1  0xb68fe18e in sock_recv_guts (s=0xb64ae6f8, cbuf=0x50e2e8 '\313' <repeats 199 times>, <incomplete sequence \313>..., len=65536, flags=0)
    at /tmp/Python-3.4.4/Modules/socketmodule.c:2600
[...]
#38 0x00025752 in PyRun_AnyFileExFlags (fp=0x32bcc8, filename=0xb6be5310 "main.py", closeit=1, flags=0xbed1db20) at Python/pythonrun.c:1287
#39 0x0003b1ee in run_file (fp=0x32bcc8, filename=0x2c99f0 L"main.py", p_cf=0xbed1db20) at Modules/main.c:319
#40 0x0003beb8 in Py_Main (argc=2, argv=0x2c9010) at Modules/main.c:751
#41 0x000208d8 in main (argc=2, argv=0xbed1dd14) at ./Modules/python.c:69

It is well known that if you want your Flask apps to support internationalization, you should use The Flask Mega-Tutorial recommends creating a single javascript file per language. While this method is working, I would prefer using a standard gettext style method.

Meet jsgettext. It is basically a boiled down version of the usual gettext tool that is used everywhere in nearly every programming language. It supports the standard gettext functions, including the one using contexts or plural forms.

Unfortunately, I found that the documentation of its usage was quite limited. This is why I uploaded here a sample app that gets its text translated in 3 different parts:

• In the Python code.
• In the Jinga templates.
• In Javascript generated text

Each part include examples of translations including placeholders and/or plural forms.

You can find the code attached to this post, but for convenience I've also uploaded it on GitHub:
https://github.com/ZeWaren/flask-i18n-example