Snappy was something setup so the lab could see how construction was going and also so we could have some time lapse photography about the building being built.

Ironically, it appears that Snappy has grown. The VC of the Uni checks it, the architechs in Canberra are using it to monitor progress and a lot of the people involved are using it!

You see snappy consist of the camera and a web frontend to the photos snappy has taken. The frontend is a bunch of PHP scripts created by myself (Benjamin Close), Aaron Stafford, Ross Smith and Micheal Marner. Each one of us has worked on a part of either the scripts, the hardware or getting things working. Robert Speedie has been a big help in making this work as well. He has the contacts and funding to help it happen.

So if your interested in seeing Snappy, visit the url:

• http://wcl.ml.unisa.edu.au/~snappybackup/

and have a look. One of the photos he’s taken is below –  a great sunrise.

• Mozilla Thunderbird
• Windows Address Book
• OS X Address Book
• Squirrel Mail

In fact any client that supports LDAP.

By determining this schema, ClearChain can finally have a ‘go anywhere’ addressbook.

Conceptual Layout

Stage 1

The idea is that everything is stored in a database. Databases are great for storing and manipulating information. Currently we use a MySQL database. Data is extracted from the database in xml format. This is actually ridiculously easy. MySQL (on the command line at least) supports output formats of xml. On the command line when connecting to the database you just throw in a –xml flag, for example:

mysql --xml -u frank -p

Then when you get any text back from any queries you run the results will be wrapped in xml. Even commands such as:

mysql> show databases;

will return results marked up with xml, for example:

<?xml version="1.0"?>

<resultset statement="show databases;">
  <row>
        <field name="Database">information_schema</field>
  </row>

  <row>
        <field name="Database">addressbook</field>
  </row>

  <row>
        <field name="Database">budget</field>
  </row>

  <row>
        <field name="Database">dinner</field>
  </row>

  <row>
        <field name="Database">mysql</field>
  </row>

  <row>
        <field name="Database">wiki_thesis</field>
  </row>
</resultset>

This is awesome because it means that we can easily convert the xml into any format that we want in particular an LDAP schema that Thunderbird understands, an LDAP schema that Outlook understands, but also just as easily HTML, XHTML, plain text… you name it the xml can be converted to it simply with an xml transformation.

In the case of an LDAP server once the XML has been converted to the appropriate format existing LDAP data is blown away and then replaced by the new schema.

NOTE: Through php the mysql calls do not support fetch results in xml format, except via an extension. For this reason I wrote a simple method that wraps the mysql like xml around the data set returned via the usual query call.

Stage 2

The second stage of this project is to provide the various email clients and addressbooks with the ability to add and modify contacts in the database. This is more of a pipe dream at the moment, but should be relatively straight forward with an email client such as Thunderbird.

LDAP Schema

This section details the various fields used by different applications for the address book. I’ve tried to map the fields to the official ldap schemas that exist. A full list that are supported by thunderbird is available at: http://www.mozilla.org/projects/thunderbird/specs/ldap.html

There is however a lot of undocumented mozilla/thunderbird specific attributes. http://www.pro-linux.de/t_office/mozilla_op20.schema

Documents a complete mozilla schema but is yet to be tested

OpenLdap

In order to use create a schema for openldap that works for at least Thunderbird, download the schema linked above then make sure you have the following lines in your slapd.conf

include         /usr/local/etc/openldap/schema/core.schema

# Add additional schema
include                 /usr/local/etc/openldap/schema/cosine.schema
include                 /usr/local/etc/openldap/schema/inetorgperson.schema
include                 /usr/local/etc/openldap/schema/mozilla_op20.schema

Contributors

Aaron Stafford has written a substantial amount of code for this project, including:

• HTML front end which communicates with the the database over PHP
• XSL transformation that converts XML extracted from the database into the openldap ready text

Code Access

Access to the code is currently restricted, but hopefully this will become a public upon request. If you are interested in seeing this project become public please drop us a line.

git clone username@git.clearchain.com:/export/git/projects/ldapab

On the mac you have to specify where you want it checked out to so use:

git clone username@git.clearchain.com:/export/git/projects/ldapab ldapab

If you are on a mac. It might be the case that you need to do this on any platform. I haven’t tested it anywhere else though.

External Links

http://www.plaxo.com/ – Plaxo is an interesting application. It lets you sync a whole bunch of different address books and calendar information from different application and online accounts. I have not used this but it appears to be a near perfect solution if you are prepared to hand over your usernames and passwords to another company. ldapab is designed to be managed by the person looking after their own contacts, not having someone else manage it for you.

Background

With the growing amount of personal data every person is starting to accumulate, one of the biggest problems lies in how to back the data up. The introduction of cheap broadband has made network bandwidth relatively quick, and also affordable. However, despite this, backups require disk space in alternative locations. With Google ^[1] now offering 3GB gmail^[2] email accounts, one possibility is to backup your data in your email.

However emailing files to yourself in not only cumberson but also very error prone. Also with a maximum size limit ^[3] smaller than a number of files you may want to backup, things become even more complex. As a user you would have to send multiple emails, all containing sections of a file

Introducing the Backup Gmail FS

Early on after the creation of gmail based filesystem^[4] using a fuse ^[5], I got to thinking gmail could be used to backup data via a file system.

The problem with the existing GMail FS is doesn’t deal with redundancy, has issues with large files, puts all the data content in one place and is limited to the size of 1 gmail account.

Backup Gmail FS (BGMailFS for short) is the next step up. The aim of it is to feature:

• data redundancy
• data security
• self sizing filesystem
• no limit on file size

Each of these are detailed below.

Data Redundancy

Considering this is aimed at storing backups, Data Redundancy is of prime consideration. As such taking some sort of approach where CRC’s of data is stored is important. Ideally spreading the data across multiple accounts in some time of mirror, or striped set.

Data Security

The one problem to backup up to Google, is how much do you trust them? Whilst I know there motto is: Don’t Be Evil ^[6] I can’t help thinking that a giant company might take a sneak peak at my data. Since the data is backups, it could very well be IP sensitive.

Hence to combat this, using a method like the one describe in the Information Dispersal Algorithm (IDA) ^{[7] [8]} proposed by Micheal Rabin in 1989, would enable us to distribute the data and also keep it secure. Though it does rely on the fact that various bits of data stored in different accounts are never pieced together (though encryption could help out there).

Self Sizing Filesystem

One thing that would be great is to have the filesystem automatically grow when it’s size reaches a certain point. This is quite possible using gmail accounts. All you have to do is register another account and add it to the file system.

No Limit on File Size

The one thing you don’t want is to only be able to have 20M max files in the file system. Hence the implementation must accommodate files of any size.

Possible Implementation

With the idea embedded, it now comes down to how would you go about implementing something like this.

With the advent of ZFS^[9] under FreeBSD things suddenly became much more possible.

ZFS, FreeBSD & Geom

ZFS is a great filesystem / logical volume manager. It supports resizing of the file system on the fly, software based parity & double parity, read checksums, caching and compression. This makes it perfect for use as the frontend to the BGmailFS.

FreeBSD has had the ZFS fs ported to it by pjd. It’ll be an experimental feature in FreeBSD 7.0 and hopefully a fully fledged FS in 7.1. ZFS on FreeBSD to make use of the Geom storage system.

Geom is abstraction of the block devices available to the operating system. It consists of consumers and producers. A consumer makes use of a geom device. Ie a filesystem consumes a geom device. A producer provides the infrastructure for a consumer to write blocks.

Now under FreeBSD a zpool (the lvm size of zfs) is both a geom producer and a geom consumer. It consumes disks. It produces a storage pool.

Putting Things together

So the question becomes how can we make use of zfs to provide the file system front end and somehow provide the backend which talks to gmail. The diagram below gives some insight to how this could fit together.

The key behind the design is the the use of geom to map a gmail account to a geom producer. This can then be used with a ZFS ZPool to let zfs do most of the work.

The concept is simple. Someone implements

• The BGFSDeamon
• The BGFSDriver
• The BGFSCommand

These all work together to end up giving the user a redunant file system.

BGFSDeamon

This Daemon is responsible for mapping low level disk requests in to the relevant gmail account commands. For instance, when a write is done to a ‘disk’ the relevant block is considered an email in a gmail account. It can be written either via SMTP or now gmail supports imap ^[10], via imap. For security of data. This block may be spread between multiple gmail account using the IDA algorithm. Like wise a read obtains all the relevant pieces of the data block from the known accounts and presents the disk block back to the BGFSDriver.

Communication between the driver and the Daemon happen vio ioctls, preferably creating a mmaped region that both can read for efficiency.

BGFS Driver

The driver is actually fairly simple. It works similar to how the fuse driver works but at a block level rather than a file system level. Ie every block request the driver gets to read/write a block, it passes via the device file back to the BGmailDeamon. The driver is really just a translation layer!

BGFSCommand

This BGFSCommand is a command line tool that can be used for adding more space (ie more accounts), checking network communications or retiring accounts. It talks to the BGFSDeamon which actually does the work. Consider it like what apachectl is to apache.

After thoughts / Conclusion

It’s ironic that after writing this page, there is many other possibilities that exist for BGmailFS. With most of the work done in the daemon, it’s entirely plausable that this same system could work with hotmail, yahoo or for that matter a separate daemon that listens on another machine somewhere. This may make even more sense as many people now run servers 24×7 behind their broadband connection. Many of them have the same problem of how to backup their data securely.

Perhaps they could run one daemon, one kernel module and then it just works!

References

1. ↑ http://google.com
2. ↑ http://www.gmail.com
3. ↑ http://mail.google.com/support/bin/answer.py?answer=6589
4. ↑ http://richard.jones.name/google-hacks/gmail-filesystem/gmail-filesystem.html
5. ↑ http://fuse.sourceforge.net/
6. ↑ http://en.wikipedia.org/wiki/Don’t_Be_Evil
7. ↑ http://www.answers.com/topic/information-dispersal-algorithm
8. ↑ http://bryanmills.net/archives/2007/09/information-dispersal-algorithms/
9. ↑ http://www.opensolaris.org/os/community/zfs/
10. ↑ http://mail.google.com/support/bin/topic.py?topic=12760