Welcome to zewaren.net. This site presents myself and mostly archives the solutions to some problems I once had.

How to use mod_dtrace to use DTrace with Apache on FreeBSD

Not so frequently asked questions and stuff: 

Prefetch Technologies' Apache DTrace module (http://prefetch.net/projects/apache_modtrace/index.html) can be compiled and used under FreeBSD to efficiently monitor an running Apache instance.

System used:

FreeBSD hostmaster1.example.com 9.1-STABLE FreeBSD 9.1-STABLE #0 r248486: Thu Jun 13 00:23:59 CEST 2013     root@hostmaster1.example.com:/usr/obj/usr/src/sys/HOSTMASTER1  amd64

Fetch the module

fetch http://prefetch.net/projects/apache_modtrace/mod_dtrace.c
fetch http://prefetch.net/projects/apache_modtrace/apache.d

Compile and install the module

Make sure you have apache's compiler installed (should be installed with the standard apache22 port).

apxs -c -o mod_dtrace.so mod_dtrace.c && rm .libs/mod_dtrace.so
/usr/sbin/dtrace -G -o apache.o -s apache.d .libs/mod_dtrace.o
gcc -fPIC -shared -lelf -o .libs/mod_dtrace.so .libs/mod_dtrace.o apache.o

Copy the module:

cp .libs/mod_dtrace.so /usr/local/libexec/apache22/

Enable in your usual Apache configuration file (usually /usr/local/etc/apache22/httpd.conf).

LoadModule dtrace_module libexec/apache22/mod_dtrace.so

Restart Apache and check that it is loaded.

httpd -M | grep dtrace
Syntax OK
 dtrace_module (shared)

List probes.

dtrace -l | grep apache
51871 apache61601     mod_dtrace.so          apache_accept_connection accept-connection
51872 apache61601     mod_dtrace.so               apache_check_access check-access
51873 apache61601     mod_dtrace.so        apache_check_authorization check-authorization
51874 apache61601     mod_dtrace.so                 apache_check_user check-user-credentials
51875 apache61601     mod_dtrace.so               apache_create_child create-child
51876 apache61601     mod_dtrace.so                apache_log_request log-request
51877 apache61601     mod_dtrace.so            apache_receive_request receive-request
51878 apache61600     mod_dtrace.so          apache_accept_connection accept-connection
...

Use the module

Unfortunately, the scripts provided by Prefetch Technologies can't be used directly since they were created for Solaris 32bit. If you use a 64bit FreeBSD, you'll have to read the Apache and APR header files to find the location of the relevant fields in the different structures (struct request_rec in httpd.h for example).

Sample script:

#!/usr/sbin/dtrace -s

dtrace:::BEGIN
{
    printf("%42s %-5s", "Request", "Status");
}

::apache_log_request:log-request
{
    this->the_request = copyinstr(*(uintptr_t *)copyin(arg0 + 48,sizeof(uintptr_t)));
    /* this->protocol = copyinstr(*(uintptr_t *)copyin(arg0 + 72,sizeof(uintptr_t))); */
    /* this->status_line = copyinstr(*(uintptr_t *)copyin(arg0 + 104,sizeof(uintptr_t))); */
    this->responsecode = (int)*(uintptr_t *)copyin(arg0 + 112,sizeof(int));
    printf("%42s %5d\n", this->the_request, this->responsecode);
}

Output:

CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN                                    Request Status
  0  51876   apache_log_request:log-request                             GET / HTTP/1.1   200
  0  51876   apache_log_request:log-request                             GET / HTTP/1.1   200
  0  51904   apache_log_request:log-request               GET /does_not_exist HTTP/1.1   404

Problems I got

dtrace: failed to compile script apache.d: "/usr/lib/dtrace/psinfo.d", line 13: operator -> cannot be applied to a forward declaration: no struct devstat definition is available

I was using the dtrace command within a jail. Using it outside solved the problem.

dtrace: failed to compile script apache.d: "/usr/lib/dtrace/regs_x86.d", line 2: type redeclared: struct devinfo

I don't know what the problem was. I just moved /usr/lib/dtrace/io.d away temporarily and it worked.

How to show the sent TCP packet sizes on FreeBSD using Dtrace

Not so frequently asked questions and stuff: 

How to show the sent TCP packet sizes on FreeBSD using Dtrace, grouped by program name.

As of 2013-06-19, the DTrace TCP provider is not available in FreeBSD, so we need a workaround.

#!/usr/sbin/dtrace -s

fbt::tcp_usr_send:entry
{
  this->mh_len = args[2]->m_hdr.mh_len;
   @[execname] = quantize(this->mh_len);
}

Sample output:

  httpd
           value  ------------- Distribution ------------- count
               2 |                                         0
               4 |@                                        30
               8 |                                         0
              16 |@                                        56
              32 |                                         2
              64 |@@@                                      136
             128 |@@@@@                                    201
             256 |@@@                                      131
             512 |@@@                                      133
            1024 |@                                        23
            2048 |                                         19
            4096 |@@@@@@@@@@@@@@@@@@@@@@                   905
            8192 |                                         0

  mysqld
           value  ------------- Distribution ------------- count
               2 |                                         0
               4 |                                         4
               8 |@@                                       67
              16 |                                         0
              32 |@                                        30
              64 |@@                                       90
             128 |@@                                       84
             256 |@                                        27
             512 |@                                        50
            1024 |@                                        34
            2048 |@                                        35
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@           1330
            8192 |                                         0

  nginx
           value  ------------- Distribution ------------- count
               0 |                                         0
               1 |                                         1
               2 |                                         0
               4 |                                         27
               8 |                                         12
              16 |                                         24
              32 |@                                        61
              64 |@@@                                      191
             128 |@@@@                                     278
             256 |@@                                       158
             512 |@@@@                                     325
            1024 |@                                        92
            2048 |@                                        86
            4096 |@@@@@@@@@@@@@@@@@@@@@@@                  1766
            8192 |                                         0

hostmaster1#

System:

FreeBSD hostmaster1.example.com 9.1-STABLE FreeBSD 9.1-STABLE #0 r248486: Thu Jun 13 00:23:59 CEST 2013     root@hostmaster1.example.com:/usr/obj/usr/src/sys/HOSTMASTER1  amd64

How to show accepted TCP connections on FreeBSD using Dtrace

Not so frequently asked questions and stuff: 

As of 2013-06-19, the DTrace TCP provider is not available in FreeBSD, so we need a workaround.

#!/usr/sbin/dtrace -s

fbt::tcp_usr_accept:entry
{
    this->inp = (struct inpcb*)(args[0]->so_pcb);
   this->inp_lport = this->inp->inp_inc.inc_ie.ie_lport;
   this->inp_fport = this->inp->inp_inc.inc_ie.ie_fport;
   printf("%d.%d.%d.%d:%d -> ", 
       ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependfaddr.ie46_foreign.ia46_addr4.s_addr)[0],
     ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependfaddr.ie46_foreign.ia46_addr4.s_addr)[1],
     ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependfaddr.ie46_foreign.ia46_addr4.s_addr)[2],
     ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependfaddr.ie46_foreign.ia46_addr4.s_addr)[3],
     ntohs(this->inp_fport)
  );

  printf("%d.%d.%d.%d:%d\n", 
        ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependladdr.ie46_local.ia46_addr4.s_addr)[0],
       ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependladdr.ie46_local.ia46_addr4.s_addr)[1],
       ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependladdr.ie46_local.ia46_addr4.s_addr)[2],
       ((unsigned char *)&this->inp->inp_inc.inc_ie.ie_dependladdr.ie46_local.ia46_addr4.s_addr)[3],
       ntohs(this->inp_lport)
  );
}

Sample output:

  2  20578             tcp_usr_accept:entry 10.0.0.13:34529 -> 10.0.0.10:80
  2  20578             tcp_usr_accept:entry 10.0.0.13:34533 -> 10.0.0.10:80
  2  20578             tcp_usr_accept:entry 10.0.0.10:34553 -> 10.0.0.12:3306
  2  20578             tcp_usr_accept:entry 10.0.0.13:34555 -> 10.0.0.10:80
  2  20578             tcp_usr_accept:entry 10.0.0.10:34557 -> 10.0.0.12:3306
  3  20578             tcp_usr_accept:entry 10.0.0.13:34461 -> 10.0.0.10:80
  3  20578             tcp_usr_accept:entry 10.0.0.10:34474 -> 10.0.0.12:3306
  3  20578             tcp_usr_accept:entry 10.0.0.13:34504 -> 10.0.0.10:80
  3  20578             tcp_usr_accept:entry 10.0.0.13:34514 -> 10.0.0.10:80
  3  20578             tcp_usr_accept:entry 10.0.0.13:34518 -> 10.0.0.10:80

System:

FreeBSD hostmaster1.example.com 9.1-STABLE FreeBSD 9.1-STABLE #0 r248486: Thu Jun 13 00:23:59 CEST 2013     root@hostmaster1.example.com:/usr/obj/usr/src/sys/HOSTMASTER1  amd64

How to send emails to an SMTP server using the command line under FreeBSD

Not so frequently asked questions and stuff: 

FreeBSD's native mail command cannot send emails through an SMTP server using a standard TCP connection. I guess it only uses sendmail.

The solution is to install heirloom-mailx, which is an alternate mailx command.

Install:

cd /usr/ports/mail/heirloom-mailx && make install clean

Use:

echo 'Some super relevant information' | /usr/local/bin/mailx -s 'One relevant subject' -S smtp=192.0.2.42 mailx@example.net

Be careful of the fact that the original mailx command is not replaced (/usr/bin/mailx).

How to install pecl-apd with PHP 5.4 on FreeBSD

Not so frequently asked questions and stuff: 

This was written on 2013-02-02.

Fetch source (revision was 297236 at the time).

# svn co http://svn.php.net/repository/pecl/apd/trunk

Apply the following patch:

diff -Naurp php-pecl-apd-1.0.2/apd-1.0.2/php_apd.c php-pecl-apd-1.0.2-patched/apd-1.0.2/php_apd.c
--- php-pecl-apd-1.0.2/apd-1.0.2/php_apd.c  2012-08-04 11:12:58.000000000 -0700
+++ php-pecl-apd-1.0.2-patched/apd-1.0.2/php_apd.c  2012-08-04 12:00:15.000000000 -0700
@@ -337,7 +337,7 @@ char *apd_get_active_function_name(zend_
            }
        } 
        else {
-           switch (execd->opline->op2.u.constant.value.lval) {
+           switch (execd->opline->extended_value) {
            case ZEND_EVAL:
                funcname = estrdup("eval");
                break;
@@ -441,8 +441,8 @@ static void log_time(TSRMLS_D)
        if(utime || stime || rtime) {
             int lineno = 0;
             if(EG(active_op_array)) {
-               if(EG(active_op_array)->start_op) {
-                   lineno = EG(active_op_array)->start_op->lineno;
+               if(EG(start_op)) {
+                   lineno = EG(start_op)->lineno;
                } else if(EG(active_op_array)->opcodes) {
                    lineno = EG(active_op_array)->opcodes->lineno;
                }

Configure and build:

# phpize
# ./configure
# make

Install:

# make install

Load:

zend_extension = "/usr/local/lib/php/20100525/apd.so"
apd.dumpdir="/var/tmp/apd"

How to setup pf to be able to access a natted ftp server inside a jail

Not so frequently asked questions and stuff: 

Situation

You have a proftpd installation inside a FreeBSD jail. Your jails are using an internal IP address and are behind a natted WAN IP address. When entering passive mode, nothing works.

Configuration

  • WAN IP: 198.51.100.42
  • Jail IP: 10.0.0.14

/usr/local/etc/proftpd.conf (in the jail)

MasqueradeAddress       198.51.100.42
PassivePorts            54000 55000

/etc/pf.conf (on the host)

#NAT from jail -> WAN
nat pass on $wan_if from $jail_addr       to any -> $wan_addr

#Redirect good ol' 21
rdr pass on $wan_if proto tcp from any to $wan_addr port 21 -> $jail_addr  port 21
#Redirect passive ports
rdr pass on $wan_if proto tcp from any to $wan_addr port 54000:55000 -> $jail_addr port 54000:*

LDAP server fails to start because of a corrupted database on FreeBSD

Not so frequently asked questions and stuff: 

Situation

Your LDAP server on your FreeBSD box can't start. Nothing shows up in the logs to help you.

# /usr/local/etc/rc.d/slapd start
/usr/local/etc/rc.d/slapd: WARNING: failed to start slapd

Finding the problem

Start the server manually with verbose information and read the output.

# /usr/local/libexec/slapd -h "ldap://10.0.0.XX/ ldap://[your:ipv6:here::]/" -u ldap -g ldap -d -1
[...]
5105220b bdb_db_open: database "dc=test,dc=example,dc=com" cannot be opened, err 22. Restore from backup!
5105220b backend_startup_one (type=bdb, suffix="dc=test,dc=example,dc=com"): bi_db_open failed! (22)
[...]

You now determined that your database is corrupted.

Repairing your database

# ls /usr/local/bin/db_recover*
/usr/local/bin/db_recover-4.6

# /usr/local/bin/db_recover-4.6 -v -h /var/db/openldap-data/
Finding last valid log LSN: file: 1 offset 338791
Recovery starting from [1][338646]
Recovery complete at Sun Jan 27 14:41:12 2013
Maximum transaction ID 80000009 Recovery checkpoint [1][338791]

If all went well, you should be able to start slapd again. Don't forget to backup your database.

How to use apache2's mod_auth_mysql with a drupal 6 user database

Not so frequently asked questions and stuff: 

Situation

You have a nice drupal 6 installation, with users and groups managed by goats or non technical people.

You have another website or system protected by apache and you want your users to be able to log in using their drupal credentials. That way the userbase can be managed through the drupal admin section and not through obscure htaccess and htgroup files.

Solution

Create a role in your drupal installation (here it's utilisateur_git) and add users into it.

Check that you fetch the information manually using a query like this one:


select users.name, users.pass from users LEFT JOIN users_roles ON (users.uid = users_roles.uid) LEFT JOIN role ON (users_roles.rid = role.rid) WHERE role.name = 'utilisateur_git';

Configure apache:

        
          AuthName MyAuthName
          AuthType Basic
          AuthBasicAuthoritative Off
          Auth_MySQL_Host 127.0.0.1
          Auth_MySQL_User mysql_ser
          Auth_MySQL_Password mysql_password
          Auth_MySQL_DB databasename
          Auth_MySQL_Password_Table "users LEFT JOIN users_roles ON (users.uid = users_roles.uid) LEFT JOIN role ON (users_roles.rid = role.rid)"
          Auth_MySQL_Password_Clause " AND role.name = 'utilisateur_git'"
          Auth_MySQL_Username_Field users.name
          Auth_MySQL_Password_Field users.pass
          Auth_MySQL_Encryption_Types PHP_MD5
          require valid-user
        

How to cast a encoded UTF-8 string into a decoded UTF-8 string in PHP

Not so frequently asked questions and stuff: 

The situation

You have some PHP strings that are encoded in utf-8 twice. Using utf8_decode or mb_convert_encoding solves the problem, but you lose the characters that aren't common with the ISO8859-1 character set.

Solution

Create a function to cast the string as binary and convert it into a utf-8 string.

/*
utf8_cast()
Packs a string into binary and convert the result into a utf-8 string.
Useful if you have a utf-8 string in iso8859-1.
ZeWaren / Erwan Martin  September 2012.
Must not be used in a production environment since random behavior can be expected if input data is invalid.
*/
function utf8_cast($str, $ignore_errors=true) {
    $result = '';
 $a = unpack('C*', $str);
  for($i=1; $i> $shift--) & 0x1) {
         $c++;
       }
       if ($c) {
           if ($c == 1) {
              #First byte of a utf-8 character is not supposed to start by 0b10xxxxxx.
                if (!$ignore_errors) {
                  return $result;
             }
               continue;
           }
           #We're dealing with a unicode character. Let's find its value.
            $unicode_value = $achar & (63 >> ($c));
         $cd = $c;
           while(--$c) {
               $unicode_value = $unicode_value 

How to synchronize users and groups from Active Directory (including passwords) to virtually anything

Not so frequently asked questions and stuff: 

The situation

You have an Active Directory server with users and groups. You would like those to be synchronized with something else, be it a sql database, an openldap server, a text file, a samba passdb file, etc.

What you need

To achieve this, you will need:

  • Roles installed on your AD Server: Identity Management for Unix (including password synchronization and administration tools).
    Roles needed to achieve password synchronization
  • A script to fetch the users and the groups from AD using the LDAP protocol.
  • A script that will act as an SSO daemon to be able to synchronize passwords as well. (see http://support.microsoft.com/kb/324542 for more information on the subject). With this, you won't be able to extract the passwords currently stored in the AD server, but you will be notified of any change.

Download, install and configure pSSOd

pSSOd is a collection of perl scripts that provide the synchronization we want.

You can find the scripts on github at: https://github.com/ZeWaren/pSSOd.

In this example, the following hosts are involved:

  • 192.168.42.10: Windows Server 2008 R2.
  • 192.168.42.20: Debian Squeeze.

Configure password synchronization on the Windows host

  • Configure the properties of Password Synchronization
    Image
  • Add an UNIX computer and configure its properties.
    Password Synchronization Host properties
    Image
  • Configure and run perlsync

    Configure and run perlsync.pl to fetch everything you need except the passwords.

    Configure:

use constant LDAP_HOST => "192.168.42.10";
use constant LDAP_USER => "aduser\@grandopen.zwm.fr";
use constant LDAP_PASSWORD => "abcd1234___";
use constant LDAP_BASE => "DC=grandopen,DC=zwm,DC=fr";

Run:

root@debiantest:~# perl perlsync.pl
$VAR1 = {
          'CN=User One,CN=Users,DC=grandopen,DC=zwm,DC=fr' => {
                                                                'name' => 'User One',
                                                                'accountname' => 'uone'
                                                              },
          'CN=Guest,CN=Users,DC=grandopen,DC=zwm,DC=fr' => {
                                                             'name' => 'Guest',
                                                             'accountname' => 'Guest'
                                                           },
          'CN=AD User,CN=Users,DC=grandopen,DC=zwm,DC=fr' => {
                                                               'name' => 'AD User',
                                                               'accountname' => 'aduser'
                                                             },
[...]
        };
$VAR1 = {
          'Even Users' => {
                            'users' => [
                                         'usix',
                                         'ufour',
                                         'utwo'
                                       ],
                            'description' => 'Users that have an even id'
                          },
          'Group two' => {
                           'users' => [
                                        'usix',
                                        'ufive',
                                        'ufour'
                                      ],
                           'description' => 'This is the second group'
                         },
[...]
        };

Configure and start perlssod

Configure and start perlssod.pl in order to be notified of any password change.

Configure:

use constant SSOD_SECRET => "8MRQH_Pa62637f3fG]3T";
use constant SSOD_TCP_HOST => "192.168.42.20";
use constant SSOD_TCP_PORT => 6677;
use constant SSOD_DEBUG_MODE => 0;

Be sure to allow TCP connections between the two hosts, on port 6677.

Watch /var/log/pssod.log to know what is happening.

root@debiantest:/var/log# tail -f pssod.log
2012/09/11 15:52:59 INFO Starting pSSOd.
2012/09/11 15:53:19 INFO Calling callback with user ufour.
2012/09/11 15:53:19 INFO Inside callback with user ufour and password abcd1234$!!.

Complete the scripts

If everything works correctly, you now have a way to fetch the users and groups from the AD server, and a way to be notified of any password change.

You can now complete perlssod.pl and perlsync.pl to store the information where you need it.

Example scripts

Some example scripts are provided in the syncs_and_callbacks folder of pSSOd, to store the information into:

  • A SQL database (MySQL, Postgres, SQLite and whatever DBI supports)
  • Some htpasswd and htgroups files

Also, you can obviously build you own scripts depending on your needs.

Pages

Subscribe to Front page feed