Our setup consists of 18 computers, all of those are used for students doing exercises with. At the end of each day most computers need to be reseted but because we don't know if a change has taken place we have to reset all. We're doing this manually because we don't want to (altough could be done easily) reinstall only computers when it's needed (those which were used).

A reset lasts about 25-35 minutes and covers about 3-6gb/day disk io using xfs_restore.

So why not use the real '/' mounted only read only? Doing so requires a very special setup which is usually used for live CDs like Knoppix, Gentoo or SuSE. Just to name some.

This page will cover most needed setup steps for UNIONFS SUPPORT using gentoo linux and a read only '/' (root filesystem)

The 'mount' command will show something like this, using unionfs: 

pc2a ~ # mount
/dev/hda3 on / type xfs (rw,noatime)
/dev/hda3 on /mnt/readonlyfs type xfs (ro,noatime)
/dev/hda4 on /mnt/difffs type ext2 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec)
udev on /dev type tmpfs (rw,nosuid)
devpts on /dev/pts type devpts (rw,nosuid,noexec)
none on /tmp type tmpfs (rw,mode=1777,size=500M)
none on /dev/shm type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw,noexec,nosuid,devmode=0664,devgid=85)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev)

Contents

Requirements

In our setup 'pxeboot was used which will invoke a chain of actions: 

pxe->get kernel+linuxrc->dhcp->exec linuxrc->pivot_root->exec chroot->init

So obviously UNIONFS had to be integrated into this chain.

Also you need:

  • sys-fs/unionfs installed on the image and copy the unionfs.ko to your pxe-image.

linuxrc (pxe boot)

Most important parts are marked bold but also have a look at the function create_union() which is most important. 

#!/bin/bash 
export PATH="/bin:/sbin:/usr/bin:/usr/sbin"
#. /functions
mount -t proc proc /proc 
CMDLINE=`cat /proc/cmdline` 
sysctl -w kernel.panic=10
#menu.lst panic=10 is besser
 
srvipaddr="192.168.0.1"
srvaddr="http://${srvipaddr}:8008"
 
function init_disk {
  test "$(sfdisk -s /dev/hda)" -gt 20000000 || exec bash
  hdparm -c 1 -d 1 /dev/hda
  sfdisk /dev/hda -uM << EOT
,1000,83
,2000,82
,9000,83
,11000,83
EOT
    mkfs.ext2   /dev/hda1
    mkswap      /dev/hda2
    mkfs.xfs -f /dev/hda3
    mkfs.ext2   /dev/hda4
}
 
function mount_newroot {
  mount -t xfs -o noatime /dev/hda3 /mnt/new 
}
 
function restore_system {
  init_disk || exec bash
  mount_newroot || exec bash
  curl ${srvaddr}/images/default.xfsdump | xfsrestore -J - /mnt/new
  curl ${srvaddr}/images/default.xfsdump.info > /mnt/new/root/Desktop/image-info.txt
  echo `uname -a`  >> /mnt/new/root/Desktop/image-info.txt
}
 
function post_restore {
  wget -O - ${srvaddr}/root.tar | tar -xf - -C /mnt/new
  wget ${srvaddr}/shutdownandreinstall -O /mnt/new/root/.shutdownandreinstall
  chmod 0755 /mnt/new/root/.shutdownandreinstall
  wget -O - ${srvaddr}/ssh_config.cgi  > /mnt/new/etc/ssh/ssh_config
}
 
function create_union {
          #originally we had
          #mount -t xfs -o noatime /dev/hda3 /mnt/new 
 
          # proc unmounten
          sync
 
          # our union is called new
          insmod /unionfs.ko
 
          #umount /mnt/new || exec bash
          mkdir -p /mnt/readonlyfs
          if [ $COMMAND -eq 7 ]; then
            # strange: terminal logins don't work anymore using xfs as rw overlay (instead of ext3)
            #mkfs.xfs -f /dev/hda4
            mkfs.ext2 /dev/hda4
          fi
          mount -t xfs -o ro,noatime /dev/hda3 /mnt/readonlyfs
          mkdir -p /mnt/difffs
          mount /dev/hda4 /mnt/difffs
 
          mount -t unionfs -o dirs=/mnt/difffs=rw:/mnt/readonlyfs=ro none /mnt/new 
}
 
MAC="$(echo "$BOOTIF" | sed -e 's/^[^-]*-//' -e 's/-/:/g')"
nameif lab "${MAC}"
udhcpc -f -q -s /udhcpc.script -i lab
  
if [ -f /etc/ ]; then
  echo "error /etc does not exist"
  exit
fi
 
curl ${srvaddr}/mac_config.cgi | sed 's/eth/nix_eth/' > /etc/mactab
nameif -c /etc/mactab
 
curl ${srvaddr}/mac_config.cgi > /etc/mactab
nameif -c /etc/mactab
 
# steht ev. auch in der /etc/conf.d/net
#ip link show lab > /dev/null 2>&1 || (
#        ip link set eth0 name lab
#        ip link set eth1 name eth0
#        ip link set eth2 name eth1
#        ip link set eth3 name eth2 
#)
 
BOOTDEFAULT="$(wget -O - ${srvaddr}/bootmode.cgi)"
 
cat << EOM
================================================================================
  please use this from now on
 
  6: USE THIS - (NORMAL BOOT) 
  7: USE THIS - (NORMAL BOOT) + (RESET of changes)
                             ..................... using new UNIONFS technology
 
================================================================================
 
 
 below - DEPRECATED - DEPRECATED - DEPRECATED - DEPRECATED - below
 
  1: DO NOT USE ANYMORE - boot from hard disk
  2: DO NOT USE ANYMORE - boot from hard disk after pre-boot reconfiguration
  3: DO NOT USE ANYMORE - install new system from server
  4: DO NOT USE ANYMORE - install new system from server and SHUT DOWN
  5: install new system from server WITHOUT post-boot reconfiguration
 
  8: spwan a shell
 
 
 
  >>>>> $BOOTDEFAULT in 30s <<<<<
EOM
 
read -n1 -t30 COMMAND || COMMAND=$BOOTDEFAULT
echo
case $COMMAND in
        1)
          echo "STARTING LOCAL SYSTEM, NO INSTALLATION is performed"
          mount_newroot || exec bash
        ;;
        2)
          echo "STARTING LOCAL SYSTEM WITH PRE-BOOT RECONFIGURATION"
          mount_newroot && post_restore || exec bash
          echo 1 | nc ${srvipaddr} 9999 > /dev/null 2>&1
        ;;
        3)
          echo "STARTING INSTALLATION"
          restore_system && \
          post_restore && \
          echo 1 | nc ${srvipaddr} 9999 > /dev/null 2>&1
        ;;
        4)
          echo "STARTING INSTALLATION (shutdown)"
          restore_system && \
          post_restore && \
          echo 1 | nc ${srvipaddr} 9999 > /dev/null 2>&1
          #touch /mnt/new/SHUTDOWN
          sync
          umount /mnt/new || exec bash
          sync
          exec poweroff -f
        ;;
        5)
          echo "STARTING INSTALLATION without reconfiguration"
          restore_system && \
          echo 1 | ${srvipaddr} 9999 > /dev/null 2>&1
        ;;
        6)
          echo "UNIONFS extension (working, not stress-tested yet)"
          create_union && \
          echo 1 | ${srvipaddr} 9999 > /dev/null 2>&1
        ;;
        7)
          echo "UNIONFS+RESET extension (working, not stress-tested yet)"
          create_union && \
          echo 1 | ${srvipaddr} 9999 > /dev/null 2>&1
        ;;
        *)
          exec bash
        ;;
esac
 
sync
sync
# deconfigure network
ip addr flush dev lab
ip link set lab down
 
cd /mnt/new 
mkdir initrd 
umount /proc 
 
pivot_root . initrd 
 
if [ $COMMAND -gt 5 ]; then
  echo "Doing funny stuff"
  mkdir -p /mnt/difffs
  mkdir -p /mnt/readonlyfs
  mount --move /initrd/mnt/readonlyfs /mnt/readonlyfs
  mount --move /initrd/mnt/difffs /mnt/difffs
  mount -t proc none /proc
  cat /proc/mounts > /etc/mtab
  umount /proc
fi
 
#/bin/bash
exec chroot . /bin/sh <<- EOF >dev/console 2>&1 
umount /initrd 
rm -rf initrd 
blockdev --flushbufs /dev/ram0 
exec /sbin/init ${CMDLINE} 
EOF

what other files needed to be changed?

I had two problems which are quite strange at first but after some hours....

halt.sh script

/etc/init.d/halt.sh to shutdown the / filesystem cleanly which was a real mess to figure out but basically look for the line with: 

/bin/fuser -s -k -9 -m "${x}"

which I changed to: 

/bin/fuser -s -m "${x}"

Now the root filesystem can be unmounted also using unionfs. WARNING: THIS IS A HACK AND OBVIOUSLY NOT DONE CORRECTLY SO PLEASE TRY TO FIGURE THIS OUT YOURSELF!

xfs vs ext3

We were using this setup first: 

* xfs on / (hda3) read only realroot
* xfs on / (hda4) read/write diffroot (here the changes are written to)

But for some unknown reason some scripts didn't work as expected as for example we were not able to login on a real terminal (that means not using X's virtual terminals). We solved this problem using ext3 on /dev/hda4 instead of xfs.

/etc/fstab

We didn't change anyting in /etc/fstab 

# <fs>                  <mountpoint>    <type>          <opts>                  <dump/pass>

# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
/dev/hda1               /boot           ext2            noauto,noatime          1 2
/dev/hda3               /               xfs             noatime                 0 1
/dev/hda2               none            swap            sw                      0 0
# oh, do we need this entry without a CDROM_DRIVE?
#/dev/cdrom             /mnt/cdrom      auto            noauto,ro,user          0 0
/dev/hdb7               /home           xfs             noatime,noauto
#/dev/cdroms/cdrom0     /mnt/cdrom      iso9660         noauto,ro               0 0
#/dev/fd0               /mnt/floppy     auto            noauto                  0 0
none                    /tmp            tmpfs           mode=1777,size=500M     0 0

# NOTE: The next line is critical for boot!
none                    /proc           proc            defaults                0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
# Adding the following line to /etc/fstab should take care of this:

none                    /dev/shm        tmpfs           defaults                0 0

problems using this setup

  • none so far altough unionfs is not recommended to use on production machines right now (read this somewhere)

advantages of this setup

It's not only much faster in regard of system reset but also gives you the benefit of being able to track changes as they happen because of 'Copy on write' (aka COW). All files which changed can be found in /initrd/mnt/difffs in my setup and all files which are on / can be found in /initrd/mnt/readonlyfs.

Another interesting thing is to use diff on those files like: 

pc2a ~ # diff /mnt/readonlyfs/etc/fstab /mnt/difffs/etc/fstab
7c7,8
< /dev/cdrom            /mnt/cdrom      auto            noauto,ro,user          0 0
---
> # oh, do we need this entry without a CDROM_DRIVE?
> #/dev/cdrom           /mnt/cdrom      auto            noauto,ro,user          0 0

links & futher reading

thanks

  • Many thanks to irc.kde.org #gentoo.de for very detailed help in some gentoo aspects I had no idea of before
  • Many thanks also to Andreas Korsten who helped me to get the pxe-boot installed.
  • Also thanks! to all who I forgot to mention and not to forget the linux folks who made all this possible!
Retrieved from "http://lastlog.de/wiki/index.php/Unionfs"
Powered by MediaWiki