The Shed, Septic & Frames Take Shape

Posted by Benjamin Close on August 18, 2009 under Country Living | Be the First to Comment

After a couple weeks with little happening, it seems things are finally starting to pick up. Last week saw the shed start to go up, and the house being rubbled and the septic tank going in. It’s nice to finally have the rubble in place. We’ve been told quite a few times it’s too wet for anything to happen on the house. Now there is no excuse! Granted it’s still a little wet at the front of the house – the clay takes a while to dry out. With rubble over the top of the clay it’s at least workable.

So starting this week things really started happening. Monday had half the steel for the house walls going up, Tuesday found the rest of the walls going up and the shed getting it’s capping and floor poured.

It was interesting talking to the guy’s putting up the house walls. Apparently due to the wind speed in our area (51m/s) the amount of dynabolts that get put in almost triple. Every stud gets bolted as apposed to one dynabolt every 1200mm. The house is starting to look great. Walking through it Tuesday night, it’s nice to see what was designed on paper really working out the way that was imagined.

In regards to the shed – We have a shed! Apart from the tanks it’s the first main structure on the block – at least that is complete. Roof, walls, floor, doors! After the weekend there is even some temporary plumbing in place to collect some water. (We even collected 5 mm’s worth on Sunday night). Enough to at least make it to the tap level in the tank.

So for now, check out the photos below about how things are looking.

VMWare woes with DISK IO & Possible Solution

Posted by Benjamin Close on August 12, 2009 under ClearChain, Computers | Read the First Comment

Recently the server hosting clearchain.com (aka Leo) has been having disk io errors. This has had me quite perplexed. You see Leo is a virtual machine running on redundant hardware as part of an VMWare ESX cluster. Hence whilst I can understand slow performance and delayed access at some times, disk IO’s don’t make sense.

You see Leo has a virtual disk. This virtual disk is provided by software running on the ESX host. The only way for disk IO errors are if the physical media that it runs on has errors or something inbetween the physical media and the machine providing the virtual disk has errors.  I’ve been assure by our hosting provider (Hmon – A great little company) that neither of these situations have happened.

Despite that disk errors (Virtual in this case) have been occurring:

Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=103662968832 size=16384 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=103662968832 size=8192 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=103662977024 size=8192 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141542912 size=98304 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141542912 size=1024 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141543936 size=1024 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141544960 size=1024 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141545984 size=1024 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141547008 size=1024 error=5
Aug 11 02:00:49 leo root: ZFS: vdev I/O failure, zpool=tank path=/dev/da0s1e offset=169141548032 size=1024 error=5

A little perplexed I started to try and work out why. Then I hit the article: http://virtualgeek.typepad.com/virtual_geek/2009/06/vmware-io-queues-micro-bursting-and-multipathing.html
and it all made sense. The problem is Unix machines are much harder on disks sending many more requests to them than a standard windows machine. Looking at the logs it’s clear there is lots of requests happening very rapidly. Whilst the disk IO (iscsi backplanes in this case) can keep up, the burst rate of requests (IO’s per second) appears to be too low. Hence vmware panics as it can’t service the requests via the vscsi queues quick enough and hence the virtual machine (Leo in this case) gets told there is a disk error. All the while the physical disks in the virtual cluster haven’t even been stressing.

Whilst this is currently only a theory I have, I’ve asked Hmon to look into increasing the vscsi queues to help deal with the burst rate IO better.

VNC Connects Then Hangs

Posted by Benjamin Close on August 6, 2009 under Computers, OpenSource, UniSA | Be the First to Comment

The other day I was working with a few colleges using VNC. We came across an issue with our VNC setup. We could connect a VNC client (be it tightvnc, vncviewer, vinagre or x4vnclient) to our vnc server but never see anything on the screen. It was if the server was freezing or hanging.  This left us wondering why. Initially we thought we had broken something in the VNC server – after all we had made code modifications. However the solution turned out to be a very simple fix.

You see the VNC protocol has the ability to place a client ONHOLD. During this state the clients events are not transmitted and the server sends back no images. This is what was happening to us. Normally the VNCServer will place the client on hold during a connection, asking the user to accept/reject the client. However we had been playing around with configuration files and set the prompt to disabled – we wanted automatic connection, with no prompt. Hence the VNCServer was placing the client on hold, noticed prompts were disabled and because authentication had not been established, sat there twiddling it’s thumbs.

A simple configuration file fix and everything was working again. Ironically at the same time we also discovered that modifying files by had in ~/.gconf/* doesn’t do anything as there’s a daemon that holds the configuration and it periodically writes to ~/.gconf/* overwriting any changes you might have made – hence we used gconftool2.

Progress On The Block

Posted by Benjamin Close on August 3, 2009 under Country Living | Read the First Comment

With winter well underway, it’s time for another update. Since the last blog post, quite a lots has happened. Format has poured the slab and hence our house is taking shape. I really do feel for the folks who poured the slab. I went up there and saw it taking place. The day was an absolute shocker! 8 degrees, 33 mm of rain fell, and it hailed! The poor buggers!. Having said that they did a good job. The slab, looks great, it’s pretty level – the best they could do in the weather. There’s a little bit of pitting due to the rain but nothing to cause concern.

Format’s next step was delivery of the steel frame. Sadly the weather has not played well. Between the slab being poured and the steel being delivered there’s been the wettest July in 13 years. Over 80mm! This has played havoc with both our driveway and also with Format’s delivery of the steel. The problem came down to the ability of a semi to turn at the end of the access track. Sadly the water made it impossible for a truck to turn. Added to Format changing the delivery date from one where there had been 5 days of fine weather to one where there had been 20mm of rain. Hence a crane was needed – At our cost At least now the steel frame is ready to go up. Sadly it’s still too wet around the edges of the house for a bobcat to clean up and lay some more rubble (even though the rubble has been delivered).

On the non format side of things we have quite a few other things on the go. Our rain water tanks have been installed! 3 tanks at 23650 litres each! The base for these took quite a while to prepare. 12 tonnes of dolamite sand was used to level up the area and also to provide a solid base for the tanks. After all it’s 1tonne per 1000 litres of water. Hence that’s a total of 72 tonnes we have in an area of 14 metres by 4 meters. We certainly don’t want them to move!

With the tanks delivered (Many thanks to John, Phil, Ross, Drew and Andrew for their help), it was now time for the shed to be put up. We need the shed up to both help catch water and also to give us somewhere to store things on the block – something that has proved very difficult – It’s really annoying when you have lots of tools but can’t use them as they are all packed.

So the base of the shed has now been prepared (18 tonnes of rubble – all shifted and leveled by hand). The holes for the frame have been dug and the shed is set to go up on Thursday. So it’s all underway. We can’t wait to get in to the house and every little delay is really annoying there, but it’s getting there and we know it’s worth the wait.