Back up of large filesystem with a very large number of files

Backing up large filesystems with very large numbers of files and directories is challenging. I routinely deal with 20-40TB filesystems with millions, sometimes tens of millions, of files. Filesystem with Unix home directories in a place with large number of users is an example of such a filesystem. The problem is more of backup speed than capacity.

Some commercial backup packages can do a better job here, but if you rely on free backup solutions, the situation is a bit more challenging. I have tried quite a few open-source backup programs, and performance of most of them is not acceptable in this type of environment. However, I have identified a few approaches which performance comes as acceptable, barely acceptable but acceptable from my perspective. Read the rest of this entry »

Tape changer script

One of great features of GNU tar is that it can create multi-volume archives. This translates to ability of using multiple tapes when tar is used with a tape changer/library device. The only trick is to automate process of changing tapes. Such automation can be achieved by providing a script changing tape via “-F” argument to tar. Here, there is the script that I use with my changer (16x LTO5 tapes). Read the rest of this entry »

Backup each Samba share to a separate archive

The backup script that I use on my Samba file servers, stores each Samba share in a separate cpio archive (i.e. in a separate backup file). I find such share-based backup arrangements very convenient.

My script – see below – extracts share names and paths from Samba configuration file. When a new share is configured in Samba, it is automatically added to backups.

The script runs full backup followed by 30 incremental backups. After 30 incrementals, the cycle repeats itself – full + 30 incrementals. The script keeps track of the count for each share (their counts do not have to be in sync).
Read the rest of this entry »