How to use mod_dtrace to use DTrace with Apache on FreeBSD

---

Prefetch Technologies' Apache DTrace module (http://prefetch.net/projects/apache_modtrace/index.html) can be compiled and used under FreeBSD to efficiently monitor an running Apache instance.

System used:

FreeBSD hostmaster1.example.com 9.1-STABLE FreeBSD 9.1-STABLE #0 r248486: Thu Jun 13 00:23:59 CEST 2013     root@hostmaster1.example.com:/usr/obj/usr/src/sys/HOSTMASTER1  amd64

Fetch the module

fetch http://prefetch.net/projects/apache_modtrace/mod_dtrace.c
fetch http://prefetch.net/projects/apache_modtrace/apache.d

Compile and install the module

Make sure you have apache's compiler installed (should be installed with the standard apache22 port).

apxs -c -o mod_dtrace.so mod_dtrace.c && rm .libs/mod_dtrace.so
/usr/sbin/dtrace -G -o apache.o -s apache.d .libs/mod_dtrace.o
gcc -fPIC -shared -lelf -o .libs/mod_dtrace.so .libs/mod_dtrace.o apache.o

Copy the module:

cp .libs/mod_dtrace.so /usr/local/libexec/apache22/

Enable in your usual Apache configuration file (usually /usr/local/etc/apache22/httpd.conf).

LoadModule dtrace_module libexec/apache22/mod_dtrace.so

Restart Apache and check that it is loaded.

httpd -M | grep dtrace
Syntax OK
 dtrace_module (shared)

List probes.

dtrace -l | grep apache
51871 apache61601     mod_dtrace.so          apache_accept_connection accept-connection
51872 apache61601     mod_dtrace.so               apache_check_access check-access
51873 apache61601     mod_dtrace.so        apache_check_authorization check-authorization
51874 apache61601     mod_dtrace.so                 apache_check_user check-user-credentials
51875 apache61601     mod_dtrace.so               apache_create_child create-child
51876 apache61601     mod_dtrace.so                apache_log_request log-request
51877 apache61601     mod_dtrace.so            apache_receive_request receive-request
51878 apache61600     mod_dtrace.so          apache_accept_connection accept-connection
...

Use the module

Unfortunately, the scripts provided by Prefetch Technologies can't be used directly since they were created for Solaris 32bit. If you use a 64bit FreeBSD, you'll have to read the Apache and APR header files to find the location of the relevant fields in the different structures (struct request_rec in httpd.h for example).

Sample script:

#!/usr/sbin/dtrace -s

dtrace:::BEGIN
{
    printf("%42s %-5s", "Request", "Status");
}

::apache_log_request:log-request
{
    this->the_request = copyinstr(*(uintptr_t *)copyin(arg0 + 48,sizeof(uintptr_t)));
    /* this->protocol = copyinstr(*(uintptr_t *)copyin(arg0 + 72,sizeof(uintptr_t))); */
    /* this->status_line = copyinstr(*(uintptr_t *)copyin(arg0 + 104,sizeof(uintptr_t))); */
    this->responsecode = (int)*(uintptr_t *)copyin(arg0 + 112,sizeof(int));
    printf("%42s %5d\n", this->the_request, this->responsecode);
}

Output:

CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN                                    Request Status
  0  51876   apache_log_request:log-request                             GET / HTTP/1.1   200
  0  51876   apache_log_request:log-request                             GET / HTTP/1.1   200
  0  51904   apache_log_request:log-request               GET /does_not_exist HTTP/1.1   404

Problems I got

dtrace: failed to compile script apache.d: "/usr/lib/dtrace/psinfo.d", line 13: operator -> cannot be applied to a forward declaration: no struct devstat definition is available

I was using the dtrace command within a jail. Using it outside solved the problem.

dtrace: failed to compile script apache.d: "/usr/lib/dtrace/regs_x86.d", line 2: type redeclared: struct devinfo

I don't know what the problem was. I just moved /usr/lib/dtrace/io.d away temporarily and it worked.

---