Whilst working on an application that makes use of a 1394a/firewire camera I found a strange bug/feature with libdc1394. The problem was dc1394 was being used to the mode of the camera to 800×600 resolution. However, when querying the frame size after dequeuing the frame, the size was only 640×480. Ie: See the below snippits:

..
dc1394error_t result = dc1394_video_set_mode( this->camera, DC1394_VIDEO_MODE_800x600_RGB8);
if( result != DC1394_SUCCESS )
exit(0);
 
....
dc1394video_frame_t *frame=NULL;
dc1394error_t result;
 
// capture one frame
result = dc1394_capture_dequeue( this->camera, DC1394_CAPTURE_POLICY_WAIT, &frame );
if( result != DC1394_SUCCESS )
exit(0);
 
// Allocate image memory if required
if( this->lastFrame.image == NULL ){
memcpy(this->lastFrame,frame, sizeof(dc1394video_frame_t));
this->lastFrame.image = (uint8_t *)malloc(frame->total_bytes * sizeof(uint8_t));
 
..

At this point total_bytes was 921600 which works out to be 640x480x3 (RGB);

I was stumped. I asked for one mode and got another back. All with no errors what so ever from libdc1394.

After over an hour of debugging, google searching and more,  I eventually found the reason.  The comment that ended up in the code explains the cause:

// Shutdown any isotransports that are setup for the camera.
// It turns out that libDC1394 will quite happily let you set modes,
// framerates, etc but if there is an iso stream running none of them
// will actually take affect. Hence you can set a 800x600 res and get
// back 640x480 video frame - really nice libdc1394. It could at least
// indicate there was a active stream when setting the mode!
// Anyway to deal with this we force the camera to stop capturing
// before we change the settings, if the camera was capturing prior to
// setConfiguration being called, we restart the capture

So the solution is to make sure the iso transmission stream has been shutdown. this can be achieved using:

dc1394_video_set_transmission( this->camera, DC1394_OFF );
dc1394_capture_stop( this->camera );

An easy fix but certainly an annoying setup. I’ve not tested it but I’m guessing many of the other functions:

• dc1394_video_set_framerate
• dc1394_video_set_iso_speed
• dc1394_feature_set_mode
• dc1394_video_set_one_shot

are also affected.

[Thu Jan 20 14:15:16 2011] [error] Oops, no RSA or DSA server certificate found for 'www.somedomain.com:0'?!

Whilst somedomain.com isn’t the official domain name reported (I can’t reveal the client), this error was being printed for every SSL host except the default SSL host. Ironically the default SSL host was simply a redirect to one of the others. A quick check and indeed the problem lied with the SSL hosts – we removed every ssl host and the webserver would start fine – obviously without any ssl.

My colleague and I were  perplexed. He’d tried a quite few things to fix this all without luck. A so call Server Admin told him it was due to not using ip based virtual hosts for ssl, he claimed you can’t use Name Base Virtual hosts with SSL. No doubt this was obtained from a quick google search for the error. The problems is you can run NamedBasedVirtual hosts with SSL on port 443 provided you have a wildcard SSL certificate. A wild is required for NamedBaseVirtual hosts as the SSL connection is established first before the headers are sent. A wildcard will allow any subdomain to use the SSL connection then apache will see the host header and respond with the appropriate vhost. If on only have a single certificate this does not work and you’ll need a separate IP per certificate.

Anyway, we began trying to debug the issue. First we checked the certificate files were at the specified locations – they were. Next we checked the certificates were actually valid. You can use the openssl command below to do this:

openssl x509 -noout -text -in YOURCERTIFICATE.crt

The certificate, key, and certificate authority (CA) were all valid and in date.
Next we tried putting each Vhost in to the config one by one to see if one host had errors over another. Turns out it didn’t matter what order each host was in the config file or which ssl hosts were included, they all had issues – except for the default ssl vhost.

At this point we were a little lost. So we decided to go back to basics and work out what the error really meant. We search to see what apache module the error came from. A simple grep later we’d narrowed down the error to mod_ssl. A search of the mod_ssl source code found the following instance of the message:

# grep Oops *
ssl_engine_init.c:                         "Init: Oops, you want to request client "
ssl_engine_init.c:                "Oops, no RSA or DSA server certificate found "
ssl_engine_init.c:                "Oops, no RSA or DSA server private key found?!");
ssl_engine_io.c:                    (argp != NULL ? "(BIO dump follows)" : "(Oops, no memory buffer?)"));

Looking in ssl_engine_init.c we found the error came from the following function

static void ssl_init_server_certs(server_rec *s,
                                  apr_pool_t *p,
                                  apr_pool_t *ptemp,
                                  modssl_ctx_t *mctx)
{
    const char *rsa_id, *dsa_id;
    const char *vhost_id = mctx->sc->vhost_id;
    int i;
    int have_rsa, have_dsa;
 
    rsa_id = ssl_asn1_table_keyfmt(ptemp, vhost_id, SSL_AIDX_RSA);
    dsa_id = ssl_asn1_table_keyfmt(ptemp, vhost_id, SSL_AIDX_DSA);
 
    <em>have_rsa</em> = ssl_server_import_cert(s, mctx, rsa_id, SSL_AIDX_RSA);
    <em>have_dsa</em> = ssl_server_import_cert(s, mctx, dsa_id, SSL_AIDX_DSA);
 
    if (!(have_rsa || have_dsa)) {
        ap_log_error(APLOG_MARK, APLOG_ERR, 0, s,
                <strong>"Oops, no RSA or DSA server certificate found "
                "for</strong> '%s:%d'?!", s-&gt;server_hostname, s-&gt;port);
        ssl_die();
    }
 
    for (i = 0; i &lt; SSL_AIDX_MAX; i++) {         ssl_check_public_cert(s, ptemp, mctx-&gt;pks-&gt;certs[i], i);
    }
 
    have_rsa = ssl_server_import_key(s, mctx, rsa_id, SSL_AIDX_RSA);
    have_dsa = ssl_server_import_key(s, mctx, dsa_id, SSL_AIDX_DSA);
 
    if (!(have_rsa || have_dsa)) {
        ap_log_error(APLOG_MARK, APLOG_ERR, 0, s,
                "Oops, no RSA or DSA server private key found?!");
        ssl_die();
    }
}

Hence  the error was caused by the certificates not being able to be imported. Once again we checked paths to make sure the certificates/keys were correct. Alas they were. So we began to wonder why the certificates couldn’t be found. We’d specified the correct files, confirmed they were correct. It occurred to me that perhaps the openSSL context had not been setup correctly. But why not? I took a look at the default SSL vhost which did work and noticed a single line that were not in any of the other ssl vhosts.

SSLEngine on

The comment above this line read:

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.

I added “SSLEngine On” to the other ssl vhosts and it worked! So it turns out you can have an vhost setup on port 443 without SSL hence for each vhost you want SSL working in you must add the above line. My colleague was extremely thankful – why it happened in the  first place, we still don’t know. We suspect previously the option may have been enabled globally. However the fix allows apache to run again and works after a clean shutdown and startup.

You see the VNC protocol has the ability to place a client ONHOLD. During this state the clients events are not transmitted and the server sends back no images. This is what was happening to us. Normally the VNCServer will place the client on hold during a connection, asking the user to accept/reject the client. However we had been playing around with configuration files and set the prompt to disabled – we wanted automatic connection, with no prompt. Hence the VNCServer was placing the client on hold, noticed prompts were disabled and because authentication had not been established, sat there twiddling it’s thumbs.

A simple configuration file fix and everything was working again. Ironically at the same time we also discovered that modifying files by had in ~/.gconf/* doesn’t do anything as there’s a daemon that holds the configuration and it periodically writes to ~/.gconf/* overwriting any changes you might have made – hence we used gconftool2.

The Fix?

To allow gnome to prompt the user for authentication. Hence a dialog would show up and the authorisation could take place.

gconftool-2 -s -t bool /desktop/gnome/remote_access/prompt_enabled true

You can also find this same setting in gconf-editor if you want a gui way to update it.

XI2 is the first major rewrite of the XInput specification in years and with it comes a lot of enhanced functionality, but also a new API. The API was deliberately designed not to be backward compatible. Why? To prevent both XI1 and XI2 features to be intermingled. XI2 brings with it a new device model and a new input model. Both of these are incompatible with XI1. The decision to break the API was not a light one by Peter Hutterer, the main XI2 developer. Supporting XI1 would have ment future changes to XI2 would be impossible – there’s only so far you can push an out dated protocol and api. The added complexity of trying to maintain 2 code paths was also determined to be too high. XI2 is more than just a api/abi change. It literally change the entire input subsystem of the Xorg Xserver. XI2 supports:

• Mulitple Independent Master Devices (MPX)
• Subpixel Accuracy for input devices
• Support for 32bit key codes
• .. lots more See: http://fedoraproject.org/wiki/Features/XI2 for full details

With that in mind, the new XI2 API often leaves someone who’s been working with the XI1 wondering what the equivilant XI2 API call is. Most functions have an equivilant in XI2 function though the usage has more often than not, changed dramatically. Below is some examples of how to convert XI1 code to XI2 compatible code and a list of some of the changes which have occurred. If you find something missing, please add to the comments.

Change Overview

The biggest change that any XI1 developer will notice is the XDevice * type has vanished. All XI2 devices make use of the XID type instead. Under the hoods, in the Xserver and XDevice really equates back to an XID anyway now. The next biggest change XI1 developers will notice is the entire XInput2 API has had shifted namespaces. All XInput2 functions, types, etc are now prefixed with XI (ie XIEventMask). This was designed to force segregation of XI1 -> XI2 code.

The header file you include has also changed. Instead of:

#include <X11/extensions/XInput.h>

It’s now:

#include <X11/extensions/XInput2.h>

Notable Changes

• Proximity events are no longer supported, these are now reported as regular events on another axis (ie think of a pressure axis)
• Events are longer use device classes, instead all events are type XIEvent (a type of XEvent). Hence there is no need to pass around an array of integer classes anymore when registering for events, instead a single XIEventMask will do what is needed. The macros XISetMask and XIGetMask help with using the mask.

There’s also been a lot of changes to structures, functions, types and how they are used.

Event Changes

Structure/Type Changes

XIEventMask mask;
mask.deviceid=//some device or XIAllDevices for all devices<
mask.mask_len=2;
mask.mask=calloc(mask.mask_len, sizeof(char));
XISetMask(mask.mask, XI_ButtonPress);
..
XISelectEvents(display, win, &mask, 1)

Function/Macro Changes

Examples

Below are some examples of how to use some of the new XI2 functions.

Event Processing – Registering For Events

A simple indication how to register for events (XI2 greatly simplifies this)

• XInput 1 (Taken from: http://cgit.freedesktop.org/xorg/app/xinput/tree/src/test.c)

static int           motion_type = INVALID_EVENT_TYPE;
static int           button_press_type = INVALID_EVENT_TYPE;
static int           button_release_type = INVALID_EVENT_TYPE;
static int           key_press_type = INVALID_EVENT_TYPE;
static int           key_release_type = INVALID_EVENT_TYPE;
static int           proximity_in_type = INVALID_EVENT_TYPE;
static int           proximity_out_type = INVALID_EVENT_TYPE;
 
static int register_event(Display *dpy, XDeviceInfo *info)
{
    XEventClass         event_list[7];
    int                 i;
    XDevice             *device;
    XInputClassInfo     *ip;
 
    device = XOpenDevice(dpy, info->id);
 
    // Check for open error
    if (device->num_classes > 0) {
        for (ip = device->classes, i=0; i<info->num_classes; ip++, i++) {
            switch (ip->input_class) {
            case ButtonClass:
                DeviceButtonPress(device, button_press_type, event_list[number]); number++;
                DeviceButtonRelease(device, button_release_type, event_list[number]); number++;
                break;
 
            case ValuatorClass:
                DeviceMotionNotify(device, motion_type, event_list[number]); number++;
                if (handle_proximity) {
                    ProximityIn(device, proximity_in_type, event_list[number]); number++;
                    ProximityOut(device, proximity_out_type, event_list[number]); number++;
                }
                break;
 
            default:
                fprintf(stderr, "unknown class\n");
                break;
            }
      }
 
   if (XSelectExtensionEvent(dpy, root_win, event_list, number)) {
            fprintf(stderr, "error selecting extended events\n");
            return 0;
        }
   }

• XInput 2:
  (Taken from: http://who-t.blogspot.com/2009/05/xi2-recipes-part-1.html)
  

XIEventMask eventmask;
unsigned char mask[1] = { 0 }; /* the actual mask */
 
eventmask.deviceid = 2;
eventmask.mask_len = sizeof(mask); /* always in bytes */
eventmask.mask = mask;
/* now set the mask */
XISetMask(mask, XI_ButtonPress);
XISetMask(mask, XI_Motion);
XISetMask(mask, XI_KeyPress);
 
/* select on the window */
XISelectEvents(display, window, &eventmask, 1);

Event Processing – Listening/Processing Events

A simple indication how to get events from XI1 and XI2

• XInput 1:

  //setup via other means
  int motion_type, button_press_type,button_release_type;
   
  void doEvents(Display    *dpy)
  {
   
   XEvent        Event;
   
      while(1) {
          XNextEvent(dpy, &Event);
          if (Event.type == motion_type) {
              XDeviceMotionEvent *motion = (XDeviceMotionEvent *) &Event;
              ...
   
          } else if ((Event.type == button_press_type) ||
                     (Event.type == button_release_type)) {
                        XDeviceButtonEvent *button = (XDeviceButtonEvent *) &Event;
   
              ...
          }else if ((Event.type= .... )){
              ...
       }
  }

• XInput 2

  //Somewhere else...
   
  int xi_opcode;
  if (!XQueryExtension(display, "XInputExtension",&xi_opcode,&event, &error)) {
  printf("X Input extension not available.\n");
  return EXIT_FAILURE;
  }

  void doEvents (Display *dpy )
  {
      while(1) {
          XEvent        ev;
          XGenericEventCookie *cookie = &ev.xcookie;
          XNextEvent(dpy, &ev);
   
          if (XGetEventData(dpy, cookie) && cookie->type == GenericEvent
              && cookie->extension==xi_opcode)
          {
              XIDeviceEvent *event = cookie->data;
   
              printf("EVENT type %d\n", event->evtype);
              switch (event->evtype)
              {
                  //
                  // In all below event>deviceid contains the id of the device
                  //
                  case XI_DeviceChanged:
                      XIDeviceChangedEvent *dc = cookie->data;
                      ...
                      break;
                  case XI_HierarchyChanged:
                      XIHierarchyEvent *he = cookie->data;
                      ...
                      break;
                  case XI_RawEvent:
                      XIRawEvent *re = cookie->data;
                      ...
                      break;
                  case XI_FocusIn:
                  case XI_Enter:
                      XIEnterEvent *ee = cookie->data;
                      ...
                      break;
                  case XI_FocusOut:
                  case XI_Leave:
                      XILeaveEvent *le = cookie->data;
                      ...
                      break;
                  case XI_PropertyEvent:
                      XIPropertyEvent *pe = cookie->data;
                      ...
                      break;
                  case XI_Motion:
                  case XI_ButtonPress:
                  case XI_ButtonRelease:
                  case XI_KeyPress:
                  case XI_KeyRelease:
                      // do something with event (XIDeviceEvent contains data)
                      break;
              }
          }
       }
    }

For quite a while now I’ve been using Apple’s ICal server on FreeBSD and Sunbird/Lightning as a front end to the calendar. However, one thing that has always annoyed me was the lack of a web frontend to my calendars.

Well today after searching on a completely different topic I found a javascript front end to caldav. After only a few minutes setup I had the frontend up and running. Sadly however, It didn’t play happy with ICalServer. A little debugging of the caldav REPORT request and I had it querying correctly.

Now there was valid data populating the table… or was it. Looking a little closer there were bugs with the reoccurring events and also with the lastMonday function. A quick fix (after 20mins finding them) and now the calendar works great!

Best thing about it is I can now have a public caldav calendar which everyone can view whilst being able to update thing directly on my own writable version of the caldav calendar.

# net ads join -U cis-closebs
cis-closebs's password:
Failed to join domain: Improperly formed account name

Now we checked the logs, checked kerberos, samba, but could not get this working. The debug logs showed something but nothing really useful:

# net ads join -U cis-closebs
cis-closebs's password:
Failed to join domain: Improperly formed account name
# net ads join -d 3 -U cis-closebs
[2009/02/02 12:55:26, 3] param/loadparm.c:lp_load(5031)
  lp_load: refreshing parameters
[2009/02/02 12:55:26, 3] param/loadparm.c:init_globals(1430)
  Initialising global parameters
[2009/02/02 12:55:26, 3] param/params.c:pm_process(572)
  params.c:pm_process() - Processing configuration file "/usr/local/etc/smb.conf"
[2009/02/02 12:55:26, 3] param/loadparm.c:do_section(3770)
  Processing section "[global]"
[2009/02/02 12:55:26, 2] lib/interface.c:add_interface(81)
  added interface ip=130.220.236.62 bcast=130.220.237.255 nmask=255.255.254.0
[2009/02/02 12:55:26, 3] libsmb/namequery.c:get_dc_list(1489)
  get_dc_list: preferred server list: "130.220.64.77, uninet.unisa.edu.au, *"
[2009/02/02 12:55:26, 3] libads/ldap.c:ads_connect(394)
  Connected to LDAP server 130.220.64.77
[2009/02/02 12:55:26, 3] libsmb/namequery.c:get_dc_list(1489)
  get_dc_list: preferred server list: "130.220.64.77, uninet.unisa.edu.au, *"
[2009/02/02 12:55:26, 3] libsmb/namequery.c:get_dc_list(1489)
  get_dc_list: preferred server list: "130.220.64.77, uninet.unisa.edu.au, *"
cis-closebs's password:
[2009/02/02 12:55:27, 3] libsmb/namequery.c:get_dc_list(1489)
  get_dc_list: preferred server list: "130.220.64.77, uninet.unisa.edu.au, *"
[2009/02/02 12:55:27, 3] libads/ldap.c:ads_connect(394)
  Connected to LDAP server 130.220.64.77
[2009/02/02 12:55:27, 3] libads/sasl.c:ads_sasl_spnego_bind(213)
  ads_sasl_spnego_bind: got OID=1 2 840 48018 1 2 2
[2009/02/02 12:55:27, 3] libads/sasl.c:ads_sasl_spnego_bind(213)
  ads_sasl_spnego_bind: got OID=1 2 840 113554 1 2 2
[2009/02/02 12:55:27, 3] libads/sasl.c:ads_sasl_spnego_bind(213)
  ads_sasl_spnego_bind: got OID=1 2 840 113554 1 2 2 3
[2009/02/02 12:55:27, 3] libads/sasl.c:ads_sasl_spnego_bind(213)
  ads_sasl_spnego_bind: got OID=1 3 6 1 4 1 311 2 2 10
[2009/02/02 12:55:27, 3] libads/sasl.c:ads_sasl_spnego_bind(222)
  ads_sasl_spnego_bind: got server principal name = not_defined_in_RFC4178@please_ignore
[2009/02/02 12:55:27, 3] libsmb/clikrb5.c:ads_krb5_mk_req(593)
  ads_krb5_mk_req: krb5_cc_get_principal failed (No such file or directory)
[2009/02/02 12:55:27, 1] libsmb/clikrb5.c:ads_krb5_mk_req(602)
  ads_krb5_mk_req: krb5_get_credentials failed for not_defined_in_RFC4178@please_ignore (Server not found in Kerberos database)
[2009/02/02 12:55:27, 1] utils/net_ads.c:net_ads_join(1470)
  error on ads_startup: Server not found in Kerberos database
Failed to join domain: Improperly formed account name
[2009/02/02 12:55:27, 2] utils/net.c:main(1036)
  return code = -1

Turns out that it was the version of samba we were using. Version 3.0.28 had issues with joining a Windows Server 2008 Active Directory domain. This was fixed in Samba 3.0.28a and as can be seen with the FreeBSD ports commit:

Update port to the 3.0.28a revision.

Major changes:

  o Failure to join Windows 2008 domains
  o Windows Vista (including SP1 RC) interop issues

Approved by:	shaun (mentor, implicit)

So if you find yourself hunting around chasing something that surely should work.. consider upgrading samba!

1. It’s light weight
2. It’s cross platform
3. It was already in use by the applications being extended.

I want to try

If you want to give the FLTK changes a shot it’s fairly simple all you have to do is:

• Build a version of Xorg with MPX support (see http://wearables.unisa.edu.au/mpx) for details.
• Build a version of xinput with MPX support
  • This will allow you to create multiple mice via
    xinput create-master mymouse
    xinput list –short       # Take note of one of the mouse id’s, lets call it x and mymouse id y 
    xinput reattach  x y
• Grab the latest version of the 1.3 branch of FLTK from SVN (http://fltk.org/svn.php)
• Apply the patch at http://wcl.ml.unisa.edu.au/~closebs/downloads/fltk-1.3-mpx.patch
  • patch -p0 < fltk-1.3-mpx.patch
• Configure with multiple mice support: ./configure –enable-multiplemice
• Build the toolkit: make
• Test the multiple mice application: test/multiplemice

Future

Win32 support is currently under way with large chunks being already done. Though the lack of native cursor rendering under Win32 is causing greif. Other patches not far around the corner are a custom cursor patch, which allows a custom cursor per mouse as well as the ability to capture input events from the toolkit and inject input events into the toolkit – great for distributed event reconstruction.

Sadly this kept breaking in xorg/lib/libX11 with the error:

xcb_io.c: In function 'require_socket':
xcb_io.c:35: warning: implicit declaration of function 'xcb_take_socket'
xcb_io.c:35: warning: nested extern declaration of 'xcb_take_socket'
xcb_io.c: In function 'process_responses':
xcb_io.c:188: error: 'GenericEvent' undeclared (first use in this function)
xcb_io.c:188: error: (Each undeclared identifier is reported only once
xcb_io.c:188: error: for each function it appears in.)
xcb_io.c:189: error: 'xcb_ge_event_t' undeclared (first use in this function)
xcb_io.c:189: error: expected expression before ')' token
xcb_io.c:193: error: expected expression before ')' token
xcb_io.c: In function '_XSend':
xcb_io.c:332: warning: implicit declaration of function 'xcb_writev'
xcb_io.c:332: warning: nested extern declaration of 'xcb_writev'

For the life of me I couldn’t make sense of it. GenericEvent is defined in xcb/xcb.h and checking the preprocessor output xcb.h was being included. Likewise the lack of xcb_ge_event_t was baffling me as it was certain in xcb/xcb.h. It wasn’t till I looked a little closer that I realised that the wrong xcb.h was being used. Turns out that /usr/local/include/xcb/xcb.h was being used however I wanted /usr/local/MPX/include/xcb/xcb.h to be used. Hence I tried adding -I/usr/local/MPX to CFLAGS with no success and also tried setting CC=”gcc -I/usr/local/MPX” also with no success. It just didn’t make sense!

Looking at the gcc line, /usr/local/MPX should have been used first. It wasn’t until I finally check the preprocessor what it thought the system paths were did it all make sense.

The output from cpp is below:

Using built-in specs.
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
Thread model: posix
gcc version 4.2.1 20070719  [FreeBSD]
 /usr/libexec/cc1 -E -quiet -v -I. -I../include/X11 -I../include -I../include/X11
-I../include -I../include/X11 -I../src/xcms -I../src/xkb -I../src/xlibi18n -I/usr/local/include 
-D_LONGLONG -DHAVE_CONFIG_H -D_BSD_SOURCE -DHAS_FCHOWN -DHAS_STICKY_DIR_BIT -D_THREAD_SAFE -D_BSD_SOURCE
-DX11_t -DTRANS_CLIENT -isystem /usr/local/MPX/include xcb_io.c -o xcb_io.lo -Wall -Wpointer-arith
-Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -fno-strict-aliasing -fworking-directory -O0

ignoring duplicate directory "/usr/local/MPX/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "../include/X11"
ignoring duplicate directory "../include"
ignoring duplicate directory "../include/X11"
#include "..." search starts here:
#include <...> search starts here:
 .
 ../include/X11
 ../include
 ../src/xcms
 ../src/xkb
 ../src/xlibi18n
 /usr/local/include
 /usr/local/MPX/include
 /usr/include
End of search list.

It turns out that jhbuild sets the environment variable C_INCLUDE_PATH, in this case to /usr/local/MPX/include. This variable adds a library to the system search path. Whilst it prepends to the system search path,  -I options prepend the system search path, hence with one -I option being /usr/local/include, we get the ordering incorrect and the wrong header is used.  

So now it was up to working out why /usr/local/include was needed. Searching config.log it turns out the bigfontproto was needed by libX11, pkg_config was searching for it and finding it not in /usr/local/MPX/lib/share/pkg_config but in /usr/local/lib/share/pkg_config. The reason for this is bigfontproto wasn’t listed as a dependancy in the xorg.modules file used by jhbuild (avaliable at:http://cgit.freedesktop.org/xorg/util/modular/tree/xorg.modules). Because of this bigfont proto wasn’t being built by jhbuild. Hence the only bigfontproto that existed on the box was the default system bigfont proto which WAS located under /usr/local/include. 

The fix was simple. List xf86bigfontproto as a dependency in xorg.modules and all was good. bigfontproto is now pulled in as a dependency and hence gets built before libX11. pkg_config now correctly finds bigfontproto in /usr/local/MPX/lib/share/pkg_config hence doesn’t need to fallback to the system version so -I/usr/local/include doesn’t get added to the gcc line. Hence the compiler uses /usr/local/MPX/include/xcb/xcb.h rather than the one in /usr/local/include/xcb/xcb.h and the compiler errors vanish.

So now things are building and I’m off for a coffee!
(xorg.modules has also been correctly updated so some other poor person doesn’t have the same issues).

The first step in setting up TLS/Auth support is to install the required additional libraries. For SSL support (required by TLS) the following ports must be installed:

• security/cyrus-sasl
• security/cyrus-sasl-saslauthd//

These must be installed before sendmail is recompiled.

Sendmail in FreeBSD by default is not compiled with TLS/Auth support. In order to allow it to work with these features it must be recompiled. This is actually quite simple as long as the FreeBSD machine has the FreeBSD source collection (aka /usr/src). First you need to edit ///etc/make.conf// (//cp /etc/defaults/make.conf// if it doesn’t already exist. Edit the file so the following lines exist/are uncommented.

  # with SASLv2:
  SENDMAIL_CFLAGS=-I/usr/local/include -DSASL=2
  SENDMAIL_LDFLAGS=-L/usr/local/lib
  SENDMAIL_LDADD=-lsasl2

Next you need to recompile sendmail. Due to the way sendmail exists in the ports collections you must compile some of the libraries first. The following lines show the procedure for recompiling the libraries and sendmail.

1. Compile the smutil library

  cd /usr/src/lib/libsmutil
  make clean
  make obj
  make

1. Compile the smlibrary

  cd /usr/src/lib/libsm
  make clean
  make obj
  make

1. Compile Sendmail

  cd /usr/src/usr.sbin/sendmail
  make clean
  make obj
  make
  make install

At this point sendmail with TLS/Auth support is installed. Now sendmail must be configured to work with these features. Setting Up TLS Support

TLS stands for Transport Layer Security. It’s a bit like SSL (Socket Layer Security) in that it provides encryption between two points. The difference is that TLS provides it only in the data, SSL provides encryption of the headers as well.

Using TLS is a good idea. It provides encyrption for authentication purposes and also Trusted mail headers. Ie A Mail server can create a TLS connection between itself and another TLS server and this is reported in the mail headers and the mail headers are thus deemed ‘accurate’.

To setup TLS support you need to first generate a public/private key pair for use with the mail server. This is outside the scope of this document. Then you need to setup sendmail with the following options:

  define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')
  define(`confCACERT_PATH', `CERT_DIR')
  define(`confCACERT', `CERT_DIR/CAcert.pem')
  define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')
  define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')
  define(`confCLIENT_CERT', `CERT_DIR/cert.pem')
  define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')

You must set both the client and the server key so that incomming mail can be encrypted and so that outgoing mail can be encrypted.

Once you have set this up, restart sendmail and test it (See below) Setting Up Auth Support

In order to use authentication support you must first add a few options to the sendmail configuration file. Authentication is used to allow relaying from domains that are not listed as relay domains provided authentication exists. Ie: as long as the authentication is successful, then the mail server is affectivly an open relay on that connection. This is great for roaming laptop users who want to send mail but are often in a different domain or on dialup.

To set this up add the following to your //sendmail.mc// file:

  define(`confAUTH_MECHANISMS',`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')
  TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN');

This tells sendmail to support the listed authentication methods. There is also a ‘PLAIN’ option but this should be avoided as the password is sent in plain text. Even of TLS/SSL this is not a good idea.

With that option in place you now have to tell sendmail what it is allowed to relay based on authentication. To allow relays to anywhere provided authentication works add the following to the access file.

  From: * OK
  To: * RELAY

Make sure that you rebuild the access database (run make) then restart sendmail and authentication should be enabled so test it! Extra Sendmail Configuration Options

It is possible and highly recommended that sendmail be setup to force TLS for authentication purposes. Otherwise sendmail will quite happily accept plain text passwords across an unencrypted data stream. This is just a big security problem. Hence adding the following line to sendmail forces the use to use TLS/SSL before authentication is possible:

  define(`confAUTH_OPTIONS',`p,y')

Testing it all

After everything has been setup, it is extreemly important to test that the results are as expected. There are a number of tests you will want to run. These include the following: Test Test Implementation TLS Test

The following shows a telnet to the mail server, type the text in bold and look for the result in italics/bold

  $ telnet localhost 25
  220 mail.example.net ESMTP Sendmail 8.11.1/8.11.1; Sat, 19 May 2001 08:04:04 -0400
  EHLO localhost
  250-mail.example.net Hello IDENT:jose@[127.0.0.1], pleased to meet you
  250-ENHANCEDSTATUSCODES
  250-EXPN
  250-VERB
  250-8BITMIME
  250-SIZE
  250-DSN
  250-ONEX
  250-ETRN
  250-XUSR
  250-STARTTLS
  250-DELIVERBY
  250 HELP
  Auth Test

The following shows a telnet to the mail server, type the text in bold and look for the result in italics/bold

  $ telnet localhost 25
  220 mail.example.net ESMTP Sendmail 8.11.1/8.11.1; Sat, 19 May 2001 08:04:04 -0400
  EHLO localhost
  250-mail.example.net Hello IDENT:jose@[127.0.0.1], pleased to meet you
  250-ENHANCEDSTATUSCODES
  250-EXPN
  250-VERB
  250-8BITMIME
  250-SIZE
  250-DSN
  250-ONEX
  250-ETRN
  250-AUTH DIGEST-MD5 CRAM-MD5
  250-XUSR
  250-STARTTLS
  250-DELIVERBY
  250 HELP

You might find that the authentication methods supported don’t match what you selected. This is due to various libraries not supporting those methods of authentication.

Open Relay Test

http://www.relaycheck.com telnet relay-test.mail-abuse.org Authentication Test On the system it should work Off the system it should require authentication

  telnet server 25
  helo server
  mail from: user@server
  rcpt to: someotheruser@someotherserver

This should complain about relaying being denied without authorisation if not on the local machine

Authentication Test

With SSL & Authentications Enabled You should be able to email anywhere.

• Mozilla Thunderbird
• Windows Address Book
• OS X Address Book
• Squirrel Mail

In fact any client that supports LDAP.

By determining this schema, ClearChain can finally have a ‘go anywhere’ addressbook.

Conceptual Layout

Stage 1

The idea is that everything is stored in a database. Databases are great for storing and manipulating information. Currently we use a MySQL database. Data is extracted from the database in xml format. This is actually ridiculously easy. MySQL (on the command line at least) supports output formats of xml. On the command line when connecting to the database you just throw in a –xml flag, for example:

mysql --xml -u frank -p

Then when you get any text back from any queries you run the results will be wrapped in xml. Even commands such as:

mysql> show databases;

will return results marked up with xml, for example:

<?xml version="1.0"?>

<resultset statement="show databases;">
  <row>
        <field name="Database">information_schema</field>
  </row>

  <row>
        <field name="Database">addressbook</field>
  </row>

  <row>
        <field name="Database">budget</field>
  </row>

  <row>
        <field name="Database">dinner</field>
  </row>

  <row>
        <field name="Database">mysql</field>
  </row>

  <row>
        <field name="Database">wiki_thesis</field>
  </row>
</resultset>

This is awesome because it means that we can easily convert the xml into any format that we want in particular an LDAP schema that Thunderbird understands, an LDAP schema that Outlook understands, but also just as easily HTML, XHTML, plain text… you name it the xml can be converted to it simply with an xml transformation.

In the case of an LDAP server once the XML has been converted to the appropriate format existing LDAP data is blown away and then replaced by the new schema.

NOTE: Through php the mysql calls do not support fetch results in xml format, except via an extension. For this reason I wrote a simple method that wraps the mysql like xml around the data set returned via the usual query call.

Stage 2

The second stage of this project is to provide the various email clients and addressbooks with the ability to add and modify contacts in the database. This is more of a pipe dream at the moment, but should be relatively straight forward with an email client such as Thunderbird.

LDAP Schema

This section details the various fields used by different applications for the address book. I’ve tried to map the fields to the official ldap schemas that exist. A full list that are supported by thunderbird is available at: http://www.mozilla.org/projects/thunderbird/specs/ldap.html

There is however a lot of undocumented mozilla/thunderbird specific attributes. http://www.pro-linux.de/t_office/mozilla_op20.schema

Documents a complete mozilla schema but is yet to be tested

OpenLdap

In order to use create a schema for openldap that works for at least Thunderbird, download the schema linked above then make sure you have the following lines in your slapd.conf

include         /usr/local/etc/openldap/schema/core.schema

# Add additional schema
include                 /usr/local/etc/openldap/schema/cosine.schema
include                 /usr/local/etc/openldap/schema/inetorgperson.schema
include                 /usr/local/etc/openldap/schema/mozilla_op20.schema

Contributors

Aaron Stafford has written a substantial amount of code for this project, including:

• HTML front end which communicates with the the database over PHP
• XSL transformation that converts XML extracted from the database into the openldap ready text

Code Access

Access to the code is currently restricted, but hopefully this will become a public upon request. If you are interested in seeing this project become public please drop us a line.

git clone username@git.clearchain.com:/export/git/projects/ldapab

On the mac you have to specify where you want it checked out to so use:

git clone username@git.clearchain.com:/export/git/projects/ldapab ldapab

If you are on a mac. It might be the case that you need to do this on any platform. I haven’t tested it anywhere else though.

External Links

http://www.plaxo.com/ – Plaxo is an interesting application. It lets you sync a whole bunch of different address books and calendar information from different application and online accounts. I have not used this but it appears to be a near perfect solution if you are prepared to hand over your usernames and passwords to another company. ldapab is designed to be managed by the person looking after their own contacts, not having someone else manage it for you.