zfsbackup

#!/bin/bash -e
# zfsbackup – Jonathan Chan <jonmchan@gmail.com>
#
# ZFSBackup is a remote backup management utility that utilizes the power of
# zfs send/receive along with other utils to provide a complete remote backup
# solution.
#
# Feature List:
#   * Simple intuitive interface – zfsbackup backup, zfsbackup restore rpool; done.
#   * Supports backup retention expiration dates – automatically delete snapshots
#     and set full backups to be run after a certain interval
#   * GPG Encryption of all remote data, allowing backups to be stored anywhere,
#     even on insecure internet locations.
#   * Account for transport errors and provide for some means of repairing
#     the remote backup system (Partial support – I am sure I will find more errors when in use)
#   * Monitoring and validation of backup sets (utilizing md5). Works quickly and efficiently
#     without transferring any data on the network utilizing the remote computer to check
#     checksum!
#   * Email status reports (still need to make this more robust and clear).
#
# Requirements:
# This script was built on a solaris system, but any system with the following
# software should work:
#
# * ZFS File System with the standard zfs utility
# * GNU coreutils – cat, tee, md5sum, etc.
# * gnupg 1.x branch – on solaris, you can get this from the CSW repository
# * sqlite3
# * sendmail – make sure it works
#
# You will also need a remote system that supports ssh remote commands (cat and md5sum) and that
# has plenty of space. Minimally, you must have n+1 * Size of File System where n is the number
# of full backups you want to keep. The reason this is true is that the system creates a new full
# backup first and makes sure that works before it deletes the old full backup.
#
# Wishlist:
#   * CODE CLEANUP! It is very messy after several late night code sprints.
#   * Check if executables exist for all programs used and check if remote directory exists and
#     make it if it doesn’t
#   * Allow pre and post scripts to be run
#   * Compression of remote data with gzip
#   * External conf file with multiple conf file support
#   * Account for manual edits from the zfs commandline
#   * Change md5sum tmpfile to pipes or something more intelligent
#
# ChangeLog:
# Sat, 28 Aug 2010 23:22:51 -0400
# * First public release
# * Added a help
# * Added cronjob (same as backup, but emails all output and adds status report)
# * Finished repair code
#
# Thu, 26 Aug 2010 13:32:40 -0400
# * Changed system db layout to key -> value
# * Fixed db layout
# * Added different list views
# * Implemented incremental backup
# * Implemented some of the repair code and error/incomplete checking
# * Implemented retention expiration dates
#
# Wed, 25 Aug 2010 19:19:37 -0400
# * Initial development.
# * skeletoning system
# * Implemented full backup, and restore – still need to create incremental backup
General usage:
  1. Install zfsbackup ~/zfsbackup/ or somewhere.
  2. Edit the configuration section in zfsbackup.
  3. run ./zfsbackup init to build keys
  4. run ./zfsbackup backup when you want to backup.
  5. When you are ready to automate this, run ./zfsbackup test-email to check if your email is working. (if it isn’t, you can follow this useful smarthost tutorial to get your mail working with a mail relay – assuming you have access to one.)
  6. Add ./zfsbackup to a cronjob, say daily? 0 3 * * * /home/bashusr/zfsbackup/zfsbackup
  7. Let it do it’s thing for the next year…
  8. uh oh… I am getting an email that one of my backup files have been corrupted? run ./zfsbackup repair to restore from local snapshot.
  9. uh oh… local snapshot doesn’t exist anymore, use ./zfsbackup repair destructive to delete the corrupted backup and all incremental backups after it that has been rendered unrestorable.
  10. back up two steps, if the remote data is not actually corrupt, but just not verifying correctly because the internet connection was down or something like this, use ./zfsbackup verify to check the hashes again and unmark the corrupt flag from the backup set.
  11. big uh oh! An earthquake destroyed by RAID-Z or mirrored backup set, how do I restore? Use ./zfsbackup restore rpool/restore. /rpool/restore must exist before running this command (pfexec zfs create rpool/restore). You can restore a specific snapshot by doing ./zfsbackup list, finding the date that you want to restore and then typing ./zfsbackup restore rpool/restore $DATE.

Please try it out and give me feedback if it works (or fails) for you. Thanks.

Download Links:

zfsbackup20100829.sh

Leave a Reply