Adventures in Tapeland ====================== Christian Reis 2003-11-17 - Backing up to DAT over a LAN So after enough trouble with the DAT tape on our main server, I decided to move it to a separate box. I actually avoided doing this for a while because I was actually scared of how much work and how bad the performance would be. I also was a bit worried about how heavy the network traffic during the backup would turn out to be. - Approaches I for some reason had it more or less set in my head that the NBD would be an alternative. Then it dawned on me that a tape is actually a character device. So there went the NBD alternative. Apparently the "correct" way to do this is using cpio (or tar) output piped over a network tube. It may sound obvious now -- I guess almost everything that's basic Unix is going to be trivial -- but I was more than a bit surprised. cpio and tar actually have inbuilt support for running over rsh (and ssh) connections -- just specify a hostname prefixed to the device and it works fine. They use a small program called "rmt" to perform the actual tape writing at the destination. Often I have pondered using Amanda as a backup system. I never actually do, relying instead on a set of scripts that Jon Nelson provided me (and that I hacked up pretty much endlessly). In the end, I suspect that Amanda will end up using cpio or something over the network -- it would probably help recovering from a backup, but recovering is something that the scripts also make pretty trivial. - mt-st and GNU mt Some of the commands you need to issue during a backup session -- retensioning, ejecting, setting block size and compression -- are configurable using the mt tool. There are two popular versions of these tools on Linux: GNU mt and mt-st. The former is older, doesn't contain some of the commands the latter does, but it *does* support sending commands to remote tape drives. The latter has the compression/defcompression commands, but lacks remote tape support. - The "Buffer" issue Even though cpio and tar use rmt to handle writing to remote tapes, one alternative that is suggested in many newsgroup messages is using Lee McLoughlin's buffer program, which is actually a pair of programs that communicate using a SHM buffer. Buffer is said to really smooth out writes to a tape drive. - RSH, no buffer cpio --rsh-command=/usr/bin/rsh -H crc --block-size=64 --create -O gasolinux:/dev/st0 477664K, 3m57.964s (2015 K/s) -- CPU 25% busy (all of it doing rmt) - RSH, using buffer cpio -H crc --block-size=64 --create | rsh gasolinux "/usr/bin/buffer -p 85 -z 32768 -o /dev/st0" 477664K, 3m39.763s (2182K/s) -- CPU 25% busy (10% for each buffer) So it seems to give a nice boost, though running a larger test load would be a better test. - SSH vs RSH The computer which hosts my DAT tape drive is a Pentium-MMX that runs at 233 Mhz. This presents an odd problem to the setup: it's too slow to decrypt the ssh connection at a reasonable speed. The most it can do is 1.2MB/s, which is only a yarn over half the speed my SDT-9000 tape drive runs, which is about 2MB/s. I ended up setting up rsh-server and rsh-client to allow the MMX box to run as fast as possible. This involved hacking root's .rhosts file (root doesn't use hosts.equiv, a trivial gotcha) initially, though I changed permissions to do backups using the nobody user instead. - SSH, compression off cpio -H crc --block-size=64 --create | ssh -o "Compression no" root@gasolinux "/usr/bin/buffer -S 1024 -p 85 -s 32768 -o /dev/st0" 477664K, 5m43.581s (1472K/s) -- CPU maxed out - SSH, compression on cpio -H crc --block-size=64 --create | ssh -o "Compression yes" root@gasolinux "/usr/bin/buffer -S 1024 -p 85 -s 32768 -o /dev/st0" 477664K, 5m58.948s (1408K/s) -- CPU maxed out - Local from a 4x CD-ROM (on the same -- slow -- SCSI bus) cpio -H crc --block-size=64 --create | "/usr/bin/buffer -p 85 -z 32768 -o /dev/st0" 293792K, 7m39.118s (666K/s) -- CPU 10% busy This speed difference means 4 hour versus 2.5 hour backups! - DAT Hardware Compression My DAT tape drive does hardware compression, which improves its capacity to store data and throughput simultaneously. It seems that mt-st's defcompression/compression flags control this behaviour (and mt-gnu doesn't seem to include a replacement). So I ran a test to see how much difference it made. - defcompression 0, compression 0, using the fastest RSH setup above 477664K, 7m8.526s (1115K/s) -- CPU 15% busy The fastest RSH test above ran, of course, with both at 1. The defcompression flag indicates the default compression, which the tape drive remembers over tape insertions. The compression flag can override this default for the current tape, but will be reset when the tape is changed. I ejected and loaded a tape changing defcompression to make sure, and it seems to work correctly for my drive. One interesting aspect of this is that the backup speed actually varies over your backup load, since parts of it will be better compressible than others. - Fixed or variable block sizes My DAT tape allows setting its hardware block size to 0, which means variable block size. I am curious to discover if this affects backup performance itself. My tests, however, were inconclusive -- if it *does* make a difference, I can't measure it with a 500MB backup. Setting the hardware block size, actually, doesn't seem to do anything for me (at least when using buffer) -- I set the block size to 16384 and buffered blocks at 32768, and it worked without a hitch. I do seem to have some st0: Write not multiple of tape block size. messages in some old logs, so it may work in some mysterious way I'm not understanding right now. - The actual backup run The 15G (16524064K) -- production -- backup, faired quite well over rsh: 516377 blocks in 160m0.993s (1721K/s) I forgot, however, to tell cpio extract to use rsh instead of ssh, which gives me the time we'd probably take if I was using ssh for the backup: 516377 blocks in 632m13.178s I decided that it was unnecessary to run the actual backup as root (the collection step needs to run as root, because it needs access to all files on the system), so I set up the backup user to access the tape drive and set my scripts up to rsh -l backup.