Auto backup a server to a hotswap USB disk

This BASH command-line script allows you to connect a USB-attached hard disk, leave it overnight to get backed-up to, unplug it the next day and take it away. It is intended to be run on a server. It backs up /home, /etc, /usr and /var.

Download and use version 0.3.1 for free: backup-to-hotswap-usb-disk.sh.

You need to do the following manual steps to set it up:

Configuration On Each Backup Disk

  1. Connect the disk to the computer you'll be using it on. Use a combination of the following commands to give you an understanding of which device reference you need to use - you might need to install those commands you don't already have:
    # hwinfo --block --short
    # lsblk
    # blkid
    There will likely be multiple disks, your internal hard disk(s) and your removable disk. The removable disk will likely already be partitioned with a (potentially a FAT32 filesystem) single partition.

    Be completely sure you have the correct disk device and disk partition reference else you could destroy your server!
  2. Preferably do this on a different computer running Linux, but on the server if you have to. Become superuser. Change the partition's filesystem type to Linux:
    # cfdisk /dev/<disk device reference i.e. sdc>
    Select the partition → Type → 83 → Write → yes [Enter] → Quit
  3. (If the disk has now been automatically mounted by something like GNOME or KDE, you won't be able to complete the next step without unmounting it by either, right-clicking on it and choosing to unmount; or, from the command-line umount /media/<name of disk>)
  4. Format (and verify) the disk with the Linux ext4 filesystem:
    # mkfs.ext4 -c -c /dev/<disk partition reference i.e. sdc1>
    You should see the progress of the formatting. This can take a very long time; the two -c's cause it to do a read-write bad block test, you can speed it up by using just one -c so it does just a read test, or remove both -c's to not test for bad blocks at all.
  5. Name the disk 'backup':
    # e2label /dev/<disk partition reference i.e. sdc1> backup
  6. Put a file on the disk to signify which disk in particular it is, for example 'backup-disk-1', so you can differentiate between disks when logged in remotely:

Configuration On The Server

  1. Download the latest version of the backup script to root's home directory:
    # wget http://thegoldenear.org/toolbox/unices/backup-to-hotswap-usb-disk.sh -O /root/backup-to-hotswap-usb-disk.sh
  2. Make it executable:
    # chmod +x /root/backup-to-hotswap-usb-disk.sh
  3. Create a /media/backup directory:
    # mkdir /media/backup
  4. Schedule this backup script to run daily at 02:00:
    # echo "00 2 * * * root /root/backup-to-hotswap-usb-disk.sh" >> /etc/crontab
  5. Create a /etc/fstab entry, for the disk labeled 'backup' to mount to /media/backup:
    # echo "LABEL=backup /media/backup ext4 defaults,noauto 0 0" >> /etc/fstab
    (If you happen to be using a desktop environment on your server with an auto disk mounting feature, with the disk named 'backup' it should auto mount to /media/backup, when connected, so you shouldn't need this, but you'd also possibly need to take the mount command out of the backup script)

Notes

If you want to test the backup is going to work you can run it manually by logging in as root and running ~/backup-to-hotswap-usb-disk.sh. You should see files and directories being added in /media/backup.

To start the backup manually, from a remote terminal, and be able to logout, leaving the backup running, use this command when logged in as root: ~/backup-to-hotswap-usb-disk.sh &. Note however that when you logout it will fail to logout (this may be fixable).

When the backup is running you should see processes such as these when using top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3237 root 18 0 1864 592 1704 R 1.0 0.1 0:00.18 cp
702 root 10 -5 0 0 0 D 0.3 0.0 4:30.13 usb-storage

If you've amended your backup script to instead use scp rather than cp then when the backup is running you should see processes such as these when using top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12265 root 15 0 5204 3004 1928 S 10.7 0.3 1:37.82 ssh
12264 root 18 0 4228 1236 956 S 2.0 0.1 0:14.08 scp
702 root 10 -5 0 0 0 D 0.3 0.0 4:30.13 usb-storage

If a file is busy during the backup then it won't be backed up. A record of such files will be kept in the log, with the comment 'Text file busy'.

Beware of USB extension cables, they can have sloppy connectors that cause the disk drive to fall off. Those that come with the Freecom Toughdrive for example are unusable. If this is happening you'll see messages like this in dmesg:

sdb:<6>usb 6-7: reset high speed USB device using ehci_hcd and address 2
usb 6-7: reset high speed USB device using ehci_hcd and address 2
usb 6-7: device descriptor read/64, error -71
sd 2:0:0:0: scsi: Device offlined - not ready after error recovery
sd 2:0:0:0: SCSI error: return code = 0x00050000
end_request: I/O error, dev sdb, sector 24
Buffer I/O error on device sdb, logical block 3
usb 6-7: USB disconnect, address 2
sd 2:0:0:0: rejecting I/O to offline device
Buffer I/O error on device sdb, logical block 3
 unable to read partition table

To read the backup disk from a machine running Microsoft Windows you can install Ext2Fsd from www.ext2fsd.com.

Occasionally you should perform a filesystem check on the backup disk by doing the following:

If you add or remove a fixed SCSI hard disk in the machine (or an ATA disk for some Linux distributions) then the 'disk device reference i.e. sdb' and 'disk partition reference i.e. sdc1' will change accordingly. You should not have the backup disk drive attached to the machine whilst you attach such a new fixed disk (in the case of hotswap fixed disks) and whilst you power up the machine afterward (in the case of non-hotswap fixed disks) so that the fixed disk can take the backup disk's more stable device reference and the backup disk then take another.

If instead of backing up the same server as the backup script is running on, you want to backup a remote server, replace in the script 'cp' with 'scp -p -r root@server:'; and remove the '--archive --update' (which has the downside that all files will be copied regardless, so the backup will take considerably longer to run). So this command isn't prompted for a password, create a certificate on the remote server and put it on the local server (details to be added later on how to do this).

The log files pile up over time. You can delete all those older than 30 days with:
find /var/log/hotswap-backup -type f -mtime +30 -exec rm {} \;

Changes To This Page

2.0 - 9 March 2016

1.6.1 - 6 January 2011

When doing a filesystem check on the backup disk, use fsck -p LABEL=backup rather than e2fsck -p /dev/<disk partition reference i.e. sdb1> so you don't need to keep track of the device reference.

1.5.0 - 10 June 2009

This guide now specifies an inode size of 128 bytes when formatting the disk.

The fix for existing disks is to format them again using mkfs.ext3 -I 128 /dev/<disk partition reference i.e. sdb1> (which will delete everything on the disk) and name them again using e2label /dev/<disk partition reference i.e. sdb1> backup.

You can find the inode size of an existing partition using tune2fs -l /dev/<disk partition reference i.e. sdb1> as root, amongst the information will be something like Inode size: 256 or Inode size: 128.

1.4.1 - 27 April 2009

1.4.0 - 15 April 2009

Changes To The Script

0.3.1 - 2 February 2010

No changes are required in order to deploy this new version over version 0.3.0 other than to replace the file backup-to-hotswap-usb-disk.sh.

0.3.0 - 12 November 2008

No changes are required in order to deploy this new version over version 0.2.1 other than to replace the file backup-to-hotswap-usb-disk.sh.