jump to navigation

The case for Haskell 2011 January 25 8:36

Posted by diamond in : Tech , 2 comments

Over the last couple of weeks, i’ve seen a couple of interesting discussions about the suitability of python for large-scale development (for the sake of argument, lets define that as >1000 LOC). One thing that came out of both discussions is that while duck typing is very attractive for small projects, where development speed is often king, once you get to a large mature program, it becomes a heavy burden to bear. This kicked off a train of thought about what i actually want in a programming language, and the answer appears to be ‘Haskell’. Here’s the two big reasons:

Strong static typing: the compiler will never allow you to treat an int as a float, etc. This eliminates a large source of bugs in python, where you have to spend a lot of time writing checks to make sure you’ve actually been passed the type you expect. Code which seems to work fine now can have many hidden type traps that only become apparent when you change something somewhere else. Unit testing for this is exhausting. Yes it means it’s harder to get something running, but once it compiles, you’ve already done a lot of the debugging.

Purely functional: A function’s output is dependant solely on the arguments it is passed. This sounds rather simple, but it’s actually huge. Again, it vastly cuts down on the amount of unit testing required. The function will always return the same result for a given set of arguments. So unit tests only have to be designed to pass in edge cases. There’s no complex setting up of program state, and trying to make sure that the test stays correctly isolated when changes are made to the program. The other big advantage that appeals to me is that it means that code is inherently parallelisable. The biggest complications in making something multi-threaded is making sure via locking and seralising that the threads don’t stomp all over each other’s state. When your functions can’t affect state, well, that’s gone :) A nice example is when the ganeti htools codebase was made multi-threaded by a one-line change, improving performance many-fold.

And finally, Haskell is pretty well known and mature at this point. It runs on lots of platforms, there are multiple compilers for it, there are lots of libs and support. Choosing the best programming language in the world is of little use if you end up having to create an entire ecosystem just to get basic stuff done. A good example of this is the reddit rewrite from lisp to python. While the language was great for them, “one of the biggest issues was the lack of widely used and tested libraries“. The fact that it is compiled means the startup time for Haskell is going to be a lot faster than python, and faster again than java.

So, i’m going back to learn Haskell. I played around with it about 2 years ago for a while, and quite liked it, but never got around to doing anything serious in it. I’m looking forward to getting stuck back into it :) For a better overview of why functional programming is great, have a look at Functional Programming For The Rest of Us. And if you decide you want to learn Haskell, Real World Haskell is the place to start. Enjoy!

mounting usb drives at boot using udev on debian 2010 March 27 11:31

Posted by diamond in : Tech , 5 comments

I have an external 1.5TB usb hard disk that i use for backups. If i add it to /etc/fstab as normal, it will fail to mount on boot, as the usb subsystem won’t be initialized that early. This is unsatisfactory. So, here’s what i did to fix this.

  1. I added the drive to /etc/fstab with this line:/dev/disk/by-label/bmopbackup1 /mnt/backup ext3 defaults,noauto 0 0
    Note the noauto option (don’t mount this drive on boot), and the fsck pass number set to zero (don’t check this filesystem for errors on boot). As you can see, i labelled the external hard drive as bmopbackup1, using e2label. It just makes it a little easier to work with.
  2. Next, i needed to check how udev can recognise my drive:
    # udevadm info -q env -n /dev/disk/by-label/bmopbackup1
    ID_VENDOR=WD
    ID_MODEL=15EADS_External
    ID_REVISION=1.75
    ID_SERIAL=WD_15EADS_External_57442D574341565530323131333934-0:0
    ID_SERIAL_SHORT=57442D574341565530323131333934
    ID_TYPE=disk
    ID_INSTANCE=0:0
    ID_BUS=usb
    ID_PATH=pci-0000:00:10.4-usb-0:4:1.0-scsi-0:0:0:0
    ID_FS_USAGE=filesystem
    ID_FS_TYPE=ext3
    ID_FS_VERSION=1.0
    ID_FS_UUID=d519f829-eef8-4651-9a21-70a0552ad933
    ID_FS_UUID_ENC=d519f829-eef8-4651-9a21-70a0552ad933
    ID_FS_LABEL=bmopbackup1
    ID_FS_LABEL_ENC=bmopbackup1
    ID_FS_LABEL_SAFE=bmopbackup1

    As you can see, udev sets ID_FS_LABEL to bmopbackup1, so i can use that to uniquely identify the device to udev. If you don’t have a label on the drive, you could just as easily use ID_FS_UUID.
  3. So, here’s the magic part. I created a udev rules file /etc/udev/rules.d/99-bmopbackup.rules with the following line:
    SUBSYSTEMS=="block", ENV{ID_FS_LABEL}=="bmopbackup1", ACTION=="add", RUN+="/etc/scripts/bmopbackup-connected"
    This tells udev that when it adds a block device with the filesystem label bmopbackup1, run the specified script.
  4. I then created the /etc/scripts/bmopbackup-connected script:
    #!/bin/bash
    {
    echo "Starting @ $(date)"
    fsck -aC /dev/disk/by-label/bmopbackup1
    ret=$?
    echo "Fsck finished @ $(date) with return $ret"
    if [ $ret -eq 0 ]; then
    echo "Fsck succeeded, mounting"
    mount /dev/disk/by-label/bmopbackup1
    else
    echo "Fsck failed, not mounting"
    fi
    } &>> "/tmp/$(basename "$0").log" &
    # Has to run in background, because it blocks udev

    If i didn’t want to run fsck on the drive, i could just tell udev to run mount /mnt/backup directly. However, given that this drive is used for backups, i definitely want the drive checked every boot (if i was really paranoid, i’d add the -f to fsck, to force it to run a full filesystem check if it’s marked clean).
  5. All of the above works fine for devices which are hotplugged. But, if a device is plugged in when the system boots (a.k.a. coldplugged), then udev finds it during boot, when it’s only using the tiny number of rules in the initramfs. Udev will see your device, not have any special rules for it, and ignore it. To fix this, I added the following to /etc/rc.local:

    udevadm trigger --verbose --action=add --property-match ID_FS_LABEL=bmopbackup1

    This simulates a hotplug event for the device once the system has finished booting.

If you try something similar to the above and it doesn’t seem to be working, a useful check is to see what udev thinks it should run when a given device is connected:
# udevadm test /sys/block/sdb/sdb1
This program is for debugging only, it does not run any program,
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.
...
udevtest: run: '/etc/scripts/bmopbackup-connected'

To run this test, you have to supply the sysfs path to where your device is currently connected. /dev/disk/by-label/bmopbackup1 is a symlink to /dev/sdb1, so /sys/block/sdb/sdb1 is the equivalent sysfs path.

Updated (2012-08-06): Added ACTION==”add” parameter to the udev rules file, to prevent the script from being run when the device change‘s (due to the fsck), or is removed (via physical disconnection). Also added the rc.local trick to cover coldbooting.