STOREBACKUP(1)

NAME
storeBackup.pl - fancy compressing managing checksumming
                 hard-linking deduplicating 'cp -ua'

DESCRIPTION
This program copies trees to another location. Every file copied is potentially compressed (see --exceptSuffix). The backups after the first backup will compare the files with an md5 checksum with the last stored version. If they are equal, it will only make an hard link to it. It will also check mtime, ctime and size to recognize idential files in older backups very fast. It can also backup big image files fast and efficiently on a per block basis (data deduplication).

You can overwrite options in the configuration file on the command line.

SYNOPSIS
        storeBackup.pl --help
or
        storeBackup.pl -g configFile
or
        storeBackup.pl [-f configFile] [-s sourceDir]
              [-b backupDirectory] [-S series] [--checkCompr] [--print]
              [-T tmpdir] [-L lockFile] [--unlockBeforeDel]
              [--exceptDirs dir1] [--contExceptDirsErr]
              [--includeDirs dir1]
              [--exceptRule rule] [--includeRule rule]
              [--exceptTypes types]
              [--specialTypeArchiver archiver [--archiveTypes types]]
              [--cpIsGnu] [--linkSymlinks]
              [--precommand job] [--postcommand job]
              [--followLinks depth] [--stayInFileSystem] [--highLatency]
              [--ignorePerms] [--lateLinks [--lateCompress]] [--autorepair]
              [--checkBlocksSuffix suffix] [--checkBlocksMinSize size]
              [--checkBlocksBS] [--checkBlocksCompr check|yes|no]
              [--checkBlocksParallel] [--queueBlock]
              [--checkBlocksRule0 rule [--checkBlocksBS0 size]
               [--checkBlocksCompr0 key] [--checkBlocksRead0 filter]
               [--checkBlocksParallel0]]
              [--checkBlocksRule1 rule [--checkBlocksBS1 size]
               [--checkBlocksCompr1 key] [--checkBlocksRead1 filter]
               [--checkBlocksParallel1]]
              [--checkBlocksRule2 rule [--checkBlocksBS2 size]
               [--checkBlocksCompr2 kdey] [--checkBlocksRead2 filter]
               [--checkBlocksParallel2]]
              [--checkBlocksRule3 rule [--checkBlocksBS3 size]
               [--checkBlocksCompr3 key] [--checkBlocksRead3 filter]
               [--checkBlocksParallel3]]
              [--checkBlocksRule4 rule [--checkBlocksBS4 size]
               [--checkBlocksCompr4 key] [--checkBlocksRead4 filter]
               [--checkBlocksParallel4]]
              [--checkDevices0 list [--checkDevicesDir0]
               [--checkDevicesBS0] [checkDevicesCompr0 key]
               [--checkDevicesParallel0]]
              [--checkDevices1 list [--checkDevicesDir1]
               [--checkDevicesBS1] [checkDevicesCompr1 key]
               [--checkDevicesParallel1]]
              [--checkDevices2 list [--checkDevicesDir2]
               [--checkDevicesBS2] [checkDevicesCompr2 key]
               [--checkDevicesParallel2]]
              [--checkDevices3 list [--checkDevicesDir3]
               [--checkDevicesBS3] [checkDevicesCompr3 key]
               [--checkDevicesParallel3]]
              [--checkDevices4 list [--checkDevicesDir4]
               [--checkDevicesBS4] [checkDevicesCompr4 key]
               [--checkDevicesParallel1]]
              [--saveRAM] [-c compress] [-u uncompress] [-p postfix]
              [--noCompress number] [--queueCompress number]
              [--noCopy number] [--queueCopy number]
              [--withUserGroupStat] [--userGroupStatFile filename]
              [--exceptSuffix suffixes] [--addExceptSuffix suffixes]
              [--compressSuffix] [--minCompressSize size] [--comprRule]
              [--doNotCompressMD5File] [--chmodMD5File] [-v]
              [-d level] [--progressReport number[,timeframe]]
              [--ignoreReadError] [--suppressInfo]
              [--suppressWarning key] [--linkToRecent name]
              [--doNotDelete] [--deleteNotFinishedDirs] [--maxHardLinks]
              [--resetAtime] [--keepAll timePeriod] [--keepWeekday entry]
              [[--keepFirstOfYear] [--keepLastOfYear]
               [--keepFirstOfMonth] [--keepLastOfMonth]
               [--firstDayOfWeek day] [--keepFirstOfWeek]
               [--keepLastOfWeek] [--keepDuplicate] [--keepMinNumber]
               [--keepMaxNumber]
                | [--keepRelative] ]
              [-l logFile
               [--plusLogStdout] [--suppressTime] [-m maxFilelen]
               [[-n noOfOldFiles] | [--saveLogs]]
               [--compressWith compressprog]]
              [--logInBackupDir [--compressLogInBackupDir]
               [--logInBackupDirFileName logFile]]
              [otherBackupSeries ...]

OPTIONS

--help
        show this help

--generate, -g
        generate a template of the configuration file

--checkCompr, -C
        check compression for all files bigger than 1k to check if
        it makes sense to compress them
        overwrites options
            exceptSuffix, addExceptSuffix, minCompressSize, comprRule

--print
        print configuration read from configuration file
        or command line and stop

--file, -f
        configuration file (instead of or additionally to options
        on command line)

--sourceDir, -s
        source directory (must exist)

--backupDir, -b
        top level directory of all backups (must exist)

--series, -S
        series directory, default is 'default'
        relative path from backupDir

--tmpdir, -T
        directory for temporary files, default is </tmp>

--lockFile, -L
        lock file, if exists, new instances will finish if an old
        is already running, default is $lockFile
        this type of lock files does not work across multiple servers
        and is not designed to separate storeBackup.pl and
        storeBackupUpdateBackup.pl or any other storeBackup
        process in a separate PID space.
        Default lock file for root is in /var/lock, for others in \$HOME

--unlockBeforeDel
        remove the lock file before deleting old backups
        default is to delete the lock file after removing old
        backups

--exceptDirs, -e
        directories to except from backing up (relative path),
        wildcards are possible and should be quoted to avoid
        replacements by the shell
        use this parameter multiple times for multiple
        directories

--contExceptDirsErr
        continue if one or more of the exceptional directories
        do not exist (default is to stop processing)

--includeDirs, -i
        directories to include in the backup (relative path),
        wildcards are possible and have to be quoted
        use this parameter multiple times for multiple directories

--exceptRule
        Files to exclude from backing up.
        see README: 'including / excluding files and directories'

--includeRule
        Files to include in the backug up - like exceptRule
        see README: 'including / excluding files and directories'

--writeExcludeLog
        write a file name .storeBackup.notSaved.bz2 with the names
        of all skipped files

--exceptTypes
        do not save the specified type of files, allowed: Sbcfpl
            S - file is a socket
            b - file is a block special file
            c - file is a character special file
            f - file is a plain file
            p - file is a named pipe
            l - file is a symbolic link
            Sbc can only be saved when using option [cpIsGnu]

--archiveTypes
        save the specified type of files in an archive instead saving
        them directly in the file system
        use this if you want to backup those file types but your target
        file or transport (eg. sshfs or non gnu-cp) system does not support
        those types of files
            S - file is a socket
            b - file is a block special file
            c - file is a character special file
            p - file is a named pipe
        you also have to set --specialTypeArchiver when using this option

--specialTypeArchiver
        possible values are 'cpio' or 'tar'. default is 'cpio'
        tar is not able to archive sockets
        cpio is not part of the actual posix standard any more

--cpIsGnu
        Activate this option if your systems cp is a full-featured
        GNU version. In this case you will be able to also backup
        several special file types like sockets.

--linkSymlinks
        hard link identical symlinks

--precommand
        exec job before starting the backup, checks lockFile (-L)
        before starting (e.g. can be used for rsync)
        stops execution if job returns exit status != 0
        This parameter is parsed like a line in the configuration
        file and normally has to be quoted.

--postcommand
        exec job after finishing the backup, but before erasing of
        old backups  reports if job returns exit status != 0
        This parameter is parsed like a line in the configuration
        file and normally has to be quoted.

--followLinks
        follow symbolic links like directories up to depth
        default = 0 -> do not follow links

--stayInFileSystem
        only store the contents of file systems named by
        --sourceDir and symlinked via --followLinks

--highLatency
        use this for a very high latency line (eg. vpn over
        the internet) for better parallelization

--ignorePerms
        If this option is selected, files will not necessarily have
        the same permissions and owner as the originals. This
        speeds up backups on network drives a lot. Recovery with
        storeBackupRecover.pl will restore them correctly.

--lateLinks
        do *not* write hard links to existing files in the backup
        during the backup
        you have to call the program storeBackupWriteLateLink.pl
        later on your server if you set this flag to 'yes'
        you have to run storeBackupUpdateBackup.pl later - see
        description for that program

--lateCompress
        only in combination with --lateLinks
        compression from files >= minCompressSize will be done
        later, the file is (temporarily) copied into the backup

--autorepair, -a
        repair simple inconsistencies (from lateLinks) automatically
        without requesting the action

--checkBlocksSuffix
        Files with suffix for which storeBackup will make an md5
        check on blocks of that file. Executed after
        --checkBlocksRule(n)
        This option can be repeated multiple times

--checkBlocksMinSize
        Only check files specified in --checkBlocksSuffix if there
        file size is at least this value, default is 100M

--checkBlocksBS
        Block size for files specified with --checkBlocksSuffix
        Default is $checkBlocksBSdefault (1 megabyte)

--checkBlocksCompr
        if set, the blocks generated due to checkBlocksSuffix
        are compressed, default is 'no'
        if set to 'check', tries to estimate if compression helps

--checkBlocksParallel
        Read files specified here in parallel to "normal" ones.
        This only makes sense if they are on a different disk.
        Default value is 'no'

--queueBlock
        length of queue to store files before block checking,
        default = $queueBlock

--checkBlocksRule0
        Files for which storeBackup will make an md5 check
        depending on blocks of that file.

--checkBlocksBS0
        Block size for option checkBlocksRule
        Default is $checkBlocksBSdefault (1 megabyte)

--checkBlocksCompr0
        if set, the blocks generated due to this rule are
        compressed

--checkBlocksRead0
        Filter for reading the file to treat as a blocked file
        eg. 'gzip -d' if the file is compressed. Default is no
        read filter.
        This parameter is parsed like the line in the
        configuration file and normally has to be quoted,
        eg. 'gzip -9'

--checkBlocksParallel0
        Read files specified here in parallel to "normal" ones.
        This only makes sense if they are on a different disk.
        Default value is 'no'

--checkBlocksRule1
--checkBlocksBS1
--checkBlocksCompr1
--checkBlocksRead1
--checkBlocksParallel1
--checkBlocksRule2
--checkBlocksBS2
--checkBlocksCompr2
--checkBlocksRead2
--checkBlocksParallel2
--checkBlocksRule3
--checkBlocksBS3
--checkBlocksCompr3
--checkBlocksRead3
--checkBlocksParallel3
--checkBlocksRule4
--checkBlocksBS4
--checkBlocksCompr4
--checkBlocksRead4
--checkBlocksParallel4
--checkDevices0
        List of devices for md5 ckeck depending on blocks of these
        devices (eg. /dev/sdb or /dev/sdb1)

--checkDevicesDir0
        Directory where to store the backup of the device

--checkDevicesBS0
        Block size of option checkDevices0,
        default is 1M (1 megabyte)

--checkDevicesCompr0
        Compress blocks resulting from option checkDevices0
        possible values are 'check', 'yes' or 'no', default is 'no'

--checkDevicesParallel0
        Read devices specified in parallel to the rest of the
        backup. This only makes sense if they are on a different
        disk. Default value is 'no'

--checkDevices1
--checkDevicesDir1
--checkDevicesBS1
--checkDevicesCompr1
--checkDevicesParallel1
--checkDevices2
--checkDevicesDir2
--checkDevicesBS2
--checkDevicesCompr2
--checkDevicesParallel2
--checkDevices3
--checkDevicesDir3
--checkDevicesBS3
--checkDevicesCompr3
--checkDevicesParallel3
--checkDevices4
--checkDevicesDir4
--checkDevicesBS4
--checkDevicesCompr4
--checkDevicesParallel4
--saveRAM
        write temporary dbm files in --tmpdir
        use this if you do not have enough RAM

--compress, -c
        compress command (with options), default is <bzip2>
        This parameter is parsed like the line in the
        configuration file and normally has to be quoted,
        eg. 'gzip -9'

--uncompress, -u
        uncompress command (with options), default is  <bzip2 -d>
        This parameter is parsed like the line in the
        configuration file and normally has to be quoted, eg.
        'gzip -d'

--postfix, -p
        postfix to add after compression, default is <.bz2>

--exceptSuffix
        do not compress files with the following
        suffix (uppercase included):
        '\.zip', '\.bz2', '\.gz', '\.tgz', '\.jpg', '\.gif',
        '\.tiff?', '\.mpeg', '\.mpe?g', '\.mpe?[34]', '\.ogg',
        '\.gpg', '\.png', '\.lzma', '\.xz', '\.mov'
        This option can be repeated multiple times
        If you do not want any compression, set this option
        to '.*'

--addExceptSuffix
        like --exceptSuffix, but do not replace defaults, add

--compressSuffix
        Like --exceptSuffix, but mentioned files will be
        compressed. If you chose this option, then files not
        affected be execptSuffix, addExceptSuffix or this Suffixes
        will be rated by the rule function COMPRESSION_CHECK wether
        to compress or not

--minCompressSize
        Files smaller than this size will never be compressed
        but copied

--comprRule
        alternative to --exceptSuffix, compressSuffix and minCompressSize:
        definition of a rule which files will be compressed

--noCompress
        maximal number of parallel compress operations,
        default = chosen automatically

--queueCompress
        length of queue to store files before compression,
        default = 1000

--noCopy
        maximal number of parallel copy operations,
        default = 1

--queueCopy
        length of queue to store files before copying,
        default = 1000

--withUserGroupStat
        write statistics about used space in log file

--userGroupStatFile
        write statistics about used space in name file
        will be overridden each time

--doNotCompressMD5File
        do not compress .md5CheckSumFile

--chmodMD5File
        permissions of .md5CheckSumFile and corresponding
        .storeBackupLinks directory, default is 0600

--verbose, -v
        verbose messages

--debug, -d
        generate debug messages, levels are 0 (none, default),
        1 (some), 2 (many) messages, especially in
        --exceptRule and --includeRule

--resetAtime
        reset access time in the source directory - but this will
        change ctime (time of last modification of file status
        information)

--doNotDelete
        check only, do not delete any backup

--deleteNotFinishedDirs
        delete old backups which have not been finished
        this will only happen if doNotDelete is set

--maxHardLinks
        maximum number of hard links to use
        zero (default) means limit depends on file system used only

--keepAll
        keep backups which are not older than the specified amount
        of time. This is like a default value for all days in
        --keepWeekday. Begins deleting at the end of the script
        the time range has to be specified in format 'dhms', e.g.
          10d4h means 10 days and 4 hours
          default = 20d

--keepWeekday
        keep backups for the specified days for the specified
        amount of time. Overwrites the default values chosen in
        --keepAll. 'Mon,Wed:40d Sat:60d10m' means:
          keep backups from Mon and Wed 40days + 5mins
          keep backups from Sat 60days + 10mins
          keep backups from the rest of the days like spcified in
          --keepAll (default $keepAll)
        if you also use the 'archive flag' it means to not
        delete the affected directories via --keepMaxNumber:
          a10d4h means 10 days and 4 hours and 'archive flag'
        e.g. 'Mon,Wed:a40d5m Sat:60d10m' means:
          keep backups from Mon and Wed 40days + 5mins + 'archive'
          keep backups from Sat 60days + 10mins
          keep backups from the rest of the days like specified in
          --keepAll (default 30d)

--keepFirstOfYear
        do not delete the first backup of a year
        format is timePeriod with possible 'archive flag'

--keepLastOfYear
        do not delete the last backup of a year
        format is timePeriod with possible 'archive flag'

--keepFirstOfMonth
        do not delete the first backup of a month
        format is timePeriod with possible 'archive flag'

--keepLastOfMonth
        do not delete the last backup of a month
        format is timePeriod with possible 'archive flag'

--firstDayOfWeek
        default: 'Sun'. This value is used for calculating
        --keepFirstOfWeek and --keepLastOfWeek

--keepFirstOfWeek
        do not delete the first backup of a week
        format is timePeriod with possible 'archive flag'

--keepLastOfWeek
        do not delete the last backup of a week
        format is timePeriod with possible 'archive flag'

--keepDuplicate
        keep multiple backups of one day up to timePeriod
        format is timePeriod, 'archive flag' is not possible
        default = 7d

--keepMinNumber
        Keep that miminum of backups. Multiple backups of one
        day are counted as one backup. Default is 10.

--keepMaxNumber
        Try to keep only that maximum of backups. If you have more
        backups, the following sequence of deleting will happen:
        - delete all duplicates of a day, beginning with the old
          once, except the last of every day
        - if this is not enough, delete the rest of the backups
          beginning with the oldest, but *never* a backup with
          the 'archive flag' or the last backup

--keepRelative, -R
        Alternative deletion scheme. If you use this option, all
        other keep options are ignored. Preserves backups depending
        on their *relative* age. Example:
        -R '1d 7d 61d 92d'
        will (try to) ensure that there is always
        - One backup between 1 day and 7 days old
        - One backup between 5 days and ~2 months old
        - One backup between ~2 months and ~3 months old
        If there is no backup for a specified timespan
        (e.g. because the last backup was done more than 2 weeks
        ago) the next older backup will be used for this timespan.

--progressReport, -P
        print progress report after each 'number' files
        additional you may add a time frame after which a message is
        printed
        if you want to print a report each 1000 files and after
        one minute and 10 seconds, use: -P 1000,1m10s

--printDepth, -D
        print depth of actual read directory during backup

--ignoreReadError
        ignore read errors in source directory; not readable
        directories do not cause storeBackup.pl to stop processing

--suppressWarning
        suppress (unwanted) warnings in the log files;
        to suppress warnings, the following keys can be used:
          excDir (suppresses the warning that excluded directories
                 do not exist)
          fileChange (suppresses the warning that a file has changed
                     during the backup)
          crSeries (suppresses the warning that storeBackup had to
                   create the 'default' series)
          hashCollision (suppresses the warning if a possible
                        hash collision is detected)
         fileNameWithLineFeed (suppresses the warning if a filename
                              contains a line feed)
         use_DB_File (suppresses the warning that you should install
                      perl module DB_File for better perforamnce)
         use_MLDBM (suppresses the warning that you should install
                    perl module MLDBM if you want to use rule functions
                    MARK_DIR or MARK_DIR_REC together with option saveRAM)
         use_IOCompressBzip2 (suppresses the warning that you should
                              instal perl module IO::Compress::Bzip2
                              for better performance)
         noBackupForPeriod (suppresses warning that there are
                            no backups for certain periods when using
                            option keepRelative)
        This option can be repeated multiple times on the command line.

--suppressInfo
        suppress (unwanted) infos in the log files;
        to suppress infos, the following keys can be used:
          readCheckSums (suppress the information that checksums
                        of blocked files are read)
        This option can be repeated multiple times on the command line.

--linkToRecent
        after a successful backup, set a symbolic link to
        that backup and delete existing older links with the
        same name

--logFile, -l
        log file (default is STDOUT)

--plusLogStdout
        if you specify a log file with --logFile you can
        additionally print the output to STDOUT with this flag

--suppressTime
        suppress output of time in logfile

--maxFilelen, -m
        maximal length of log file, default = 1e6

--noOfOldFiles, -n
        number of old log files, default = 5

--saveLogs
        save log files with date and time instead of deleting the
        old (with [-noOfOldFiles])

--compressWith
        compress saved log files (e.g. with 'gzip -9')
        default is 'bzip2'
        This parameter is parsed like a line in the configuration
        file and normally has to be quoted.

--logInBackupDir
        write log file (also) in the backup directory
        Be aware that this log does not contain all error
        messages of the one specified with --logFile!

--compressLogInBackupDir
        compress the log file in the backup directory

--logInBackupDirFileName
        filename to use for writing the above log file,
        default is .storeBackup.log

otherBackupSeries
        List of other backup series to consider for
        hard linking. Relative path from backupDir!
        Format (examples):
        backupSeries/2002.08.29_08.25.28 -> consider this backup
        or
        0:backupSeries ->last (youngest) in <backupDir>/backupSeries
        1:backupSeries ->one before last in <backupDir>/backupSeries
        n:backupSeries ->
          n'th before last in <backupDir>/backupSeries
        3-5:backupSeries ->
          3rd, 4th and 5th in <backupDir>/backupSeries
        all:backupSeries -> all in <backupDir>/backupSeries
        You can also use wildcards in series names. See documentation,
        section 'Using Wildcards for Replication' for details.
        Default is to link to the last backup in every series

COPYRIGHT
Copyright (c) 2000-2022 by Heinz-Josef Claes (see README). Published under the GNU General Public License, either version 3 of the License, or (at your option) any later version.