ubuntu cachefs experiments

15 jun 2011

motivation

the main purpose is a ‘high availability’ file storage with with two servers: main and failover. the main server has two raid systems:

  • one with fast disks and moderate disk sizes and
  • another raid with slow but very big disks

the idea is to find a way to use the fast raid for** file-caching similar to ZFS or BTRFS’.**

the goal of this experimental setup is to experiment with fscache [1]. so far it seems that fscache is either bound to NFS, AFS or ISOFS (or a similar network fs technology).

experiment using:

  • virtualbox 3.x on gentoo linux
  • ubuntu 10.10 (guest)
  • currently only main is in the experimental setup and DRBD is not used yet
  • not shown in the diagram is the PV/VG/LV stuff (LVM)

there are two interesting documentations:

  • [3] how to use fscache
  • [4] how fscache is implemented (and how to debug it)

setup using nfs

this is a small setup to test if the mount and fscache are working on the target platform. in this case it was working great with ubuntu 10.10 desktop. setup steps:

  1. using [2] we install the needed tools on ubuntu linux:

    apt-get install nfs-kernel-server nfs-common

  2. cat /etc/exports

    /media/share 127.0.0.1(async,no_subtree_check,rw,insecure,all_squash)

  3. nfs-kernel-server neustarten:

    /etc/init.d/nfs-kernel-server restart

  4. change the permissions on /media/share

    chmod 0777 /media/share

  5. only to test if the nfs export works we do:

    mount -t nfs 127.0.0.1:/media/share /mnt/mounted-Share

  6. after a successful test we unmount it:

    umount /mnt/mounted-Share

  7. create the cacheDir disk:

    dd if=/dev/zero bs=1M count=512 of=cacheLoopDevice.dd mkfs.ext4 /mnt/cacheLoopDevice.dd

    according to [3]:

    tune2fs -o user_xattr /mnt/cacheLoopDevice.dd mount -o loop,user_xattr /mnt/cacheLoopDevice.dd /mnt/cacheDir

  8. enable the cachefilesd daemon:

    vi /etc/default/cachefilesd # remove the # before RUN=yes change the dir=… in /etc/cachefilesd.conf: dir /mnt/cacheDir /etc/init.d/cachefilesd restart

  9. finally let’s mount it again but this time using ‘fscache’:

    mount 127.0.0.1:/media/share /mnt/cached-Share/ -o fsc

  10. let’s do a functionality tests:

    touch /mnt/cached-Share/a ls -la /media/share

setup using nfs - no networking filesystem

as shown in the two following examples, it does not work with ext2 or ext4. it seems as if fscache forces the use of NFS/AFS/ISOFS.

ext2

mkfs.ext2 /dev/sdb1 
mount /dev/sdb1 /mnt/cached-Share/ 
umount /mnt/cached-Share/
mount /dev/sdb1 /mnt/cached-Share/ -o fsc

the error message (response to the shell command):

mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail  or so

ext4

mkfs.ext4 /dev/sdb1 
mount /dev/sdb1 /mnt/cached-Share/ 
umount /mnt/cached-Share/
mount /dev/sdb1 /mnt/cached-Share/ -o fsc

the error message (response to the shell command):

mount: wrong fs type, bad option, bad superblock on /dev/sdb1,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail  or so

the error message in (/var/log/messages):

[  846.008443] EXT4-fs (sdb1): Unrecognized mount option "fsc" or missing value

some performance experiments

to answer the question if all this setup is worth the effort, lets’ make some experiments:

  • time cp /usr/src/linux /mnt/cached-Share/
  • grep “__” /usr/src/linux -R
  • time rm -Rf /mnt/cached-Share/usr

please note: i’ve changed the described setup a little bit: the virtual machine guest was given a virtual disk which was stored on a SSD driver.

performance tests inside a virtualbox guests seem to be worthless. i have to repeat similar tests on the target machine. but using a local NFS just to get ‘fscache’ working seems to be a bad design IMHO. we’ll see if it is worth it.

conclusion

seems hot-data can be used with the NFS workaround but not without it. in contrast: zfs/btrfs have direct support for adding ‘hot’ data cache:

  • btrfs: ‘hot data relocation’ functionality [5]

  • zfs: this is referred to as ‘Separate Cache Devices’ [6] or ‘ssd caching’ [7]

it would be nice to have a similar feature to fscache but for any filesystem (not limited to NFS/AFS/ISOFS) with the ability to be added dynamically (maybe using remount, so one does not have to unmount/mount the filesystem).

Edit: bcache [8] is what i actually wanted.

article source