# FTPMIRROR

## Introduction

Ftpmirror is an utility to copy directory hierarchy (this is called ``mirror'') with FTP. A transfer is triggered only when modified, it can be efficient transferring directory trees.

It is userful for ftp-servers maintenance, contentns sync with web server, system backupping and so on.

A similar perl script exists, whose name is `mirror', this implementation is fully comparing directory lists of sources and distination, it take a heavy memmory load. If use on a large directory trees, so eat a lot of memory(or swap) spaces. When I use ftp-server with 16MB memory, 'mirror' software is not usable :-p, so I wrote software from scratch, that is frpmirror. For example, full mirroring of FreeBSD distribution (ftp://ftp.FreeBSD.ORG/pub/FreeBSD/*) needs 8~9 MB. *(Notice by translator) Original document wrote at old(1998). May be this results has obsolete.*

You need perl-5.003 or later to use ftpmirror. Please ensure that perl-5.003 or later was installed in your environment, at first. But, perl 5.004 has a bug of memory leak. It eat a lot of memory spaces when mirroring directory tree. I test with perl5.005, it has small memory leak too, it is smaller than perl 5.004, it is not critical for real usage. I suggest you using Perl 5.005.

## How to use ftpmirror

1. Install ftpmirror first. The newest version of ftpmirror is available from ftp.intec.co.jp.

   Please use these URLs.

   - ftp://ftp.intec.co.jp/pub/utils/ftpmirror/ : release
   - ftp://ftp.intec.co.jp/pub/utils/ftpmirror/beta/ : beta

   Since ftpmirror is a archive of instal files, you must only extract the archive. Do one of following:

         % cd /usr/local/src
       % gzip -cd < /tmp/ftpmirror-x.y.tar.gz | tar xf -

   So create a directory named ftpmirror-x.y. Change directory and execute configure:
   Notice: If you need define specific perl verions, please set perls path for enviroment variable "PERL".

       % cd ftpmirror-x.y
       % ./configure

   Or, when you need specific perl version:

       % cd ftpmirror-x.y
       % env PERL=/usr/local/bin/perl5.005 ./configure

   Execute make:

       % make

   When finished make with no errors. So install:

       % su
       # make install

   In finished this procedure, so installed files are below paths:

       /usr/local/bin/ftpmirror
       /usr/local/bin/rotate
       /usr/local/etc/ftpmirror.cf-sample
       /usr/local/lib/perl5/site_perl/Fan.pm
       /usr/local/lib/perl5/site_perl/Fan/...   (and some dirs)
       /usr/local/lib/perl5/site_perl/auto/Fan/...  (and some dirs)

       * Notice: libraries install directory can changes
           with perl's installed settings.

   And copying and editing a configuration files, please add difinition what you needs.

       # cd /usr/local/etc
       # cp ftpmirror.cf-sample ftpmirror.cf
       # vi ftpmirror.cf

   ## Configuration

   You can define parameters for ftpmirror with the default configuration file, package specific configuration file, and run-time options (commands arguments). In addition, configuration files are set /usr/local/etc/ftpmirror.cf by defauls, it can change by command argument "--load-config=/hoge/local.cf" if you need. And argument "--load-config=+/hogehoge/local.cf" is used, this is append reading after adove defaults reading.

   Configuration parameters have follwoing types.

       Default parametes:
       Configuration files settings. They are written before
       Serverss and Packages parameters.

       Server parameters:
       Difinition of server specific(timeouts, gateways and
       so on..). They are provided at configuration files,
       with 'server' parameters.

       Package parameters:
       Difinition of packages specific(direcotry, transfer
       modes and so on). They are provided at configuration files,
       with 'package' parameters.

       Option parameters:
       It provide by command args.

   Their parameters are settings by adove order(when exists). Option parameters are most prefferd.

   How to settings parameters at configuration files:
   That has two syntaxr,

       param-name = value
       or
       param-name += value

       =  is set param-name by value.
       += is adding value(strings) for param-name.

   How to settings parameters at commands args:
   When using by commands args,any parameters must be prepended by "--" in command line options. That use --(two minus) before options. For example, following syntax is same as adove configuration settings.

       % ftpmirror --param-name=value ...
       % ftpmirror --param-name+=value ...

   ## Parameters

   There are many acceptable parameters as follows:

   - Parameters for system and behavior
     - todo:(string, default: full-mirror)
       Settings of command behavior. Currently ftpmirror supports these behavior.
       - scan-remote : listup remote sides files.
       - scan-local : listup local sides files.
       - update-master : update local sides master index.
       - synch-remote : synchronize index files.
       - step-mirror : mirroring as slave mode.
       - full-mirror : execute basic mirroring(default).
       In defaults, it set basic mirroring behavior "full-mirror". *-regexp filters are enabled at first, any behavior works with these filters.
     - verbose: (boolean, default: no)
       When set yes, transfer logs ouput verbosey. If set yes, logs output about non-transfererd/non-modified files.
     - log-mask: (comma separated list, default: none)
       Settings about output logs. These difinition separates by perl libraries. For example, log-mask = Fan=6,Fan::FTP=7 are set, a output log include filelist of non transfererd/ non modified, and ftp's verbosey log.
     - test-mode (boolean, default: no)
       dry-run. When set yes, only show filelist of transfer targets without real transfer.
   - Parameters for FTP session
     - ftp-server: (string, default: none)
       The server name to connect FTP session. This must be specified for mirroring.
     - ftp-gateway: (string, default: none)
       Specify this parameter if you use FTP proxy gateway such as TIS Internet FireWall ToolKit. For example, "ftp-user = anonymous", "ftp-server = ring.etl.go.jp", "ftp-gateway = proxy.intec.co.jp" are set, real-connection goes to proxy.intec.co.jp, and username is anynymou@ring.etl.go.jp (ftp-pass is through simple). You may define this with server parameter.
     - ftp-port: (service, default: ftp) ftp port definition. If you need change a port number, please set. When nomarlly usage, it is not necessary.
     - ftp-bindaddr: (ip address, default: none) ftp local sides IP address difinitition. This can set a specific difinition of local ip address. When using multi-homed hosts(The hosts have multiple ip addresses), this settings provides a clear log files. When nomarlly usage, it is not necessary.
     - ftp-user: (string, default: anonymous)
       Login name for FTP session. If you want to use anonymous FTP, please specify "anonymous" as this parameter.
     - ftp-pass: (string, default: `whoami`@`hostname`)
       Password for FTP session.
     - ftp-group: (string, default: none)
       If you want change group id after logged to the server, specify the group name. This must be specified with `ftp-gpass`.
     - ftp-gpass: (string, default: none)
       If you want change group id after logged to the server, specify a password to change group. This must be specified with `ftp-group`.
     - ftp-passive: (boolean, default: no)
       Set this parameter to yes if passive connection(PASV command in FTP) is required. This helps you if you use ftpmirror over a SYN/ACK style firewall.
     - ftp-idle: (numeric, default: 0)
       If this parameter is not zero, ftpmirror tries to set idle timer of FTP session to this value. FTP server must support SITE IDLE command.
     - ftp-max-idle: (boolean, default: 0)
       If this parameter is not zero, ftpmirror tries to set idle timer of FTP session to maximum value. FTP server must support SITE IDLE command.
     - ftp-list-method: (LIST or STAT or STAT-A or STAT-A or STAT-AT default: STAT)
       You can specify which of STAT or LIST command should be used to get server's directory entries. Since STAT command returns server's directory entries over control connection, it is little faster than LIST command. Some ftp server software does not returns directory entry which is required by ftpmirror, in that case you must speficy as ftp-list-method = list.
     - ftp-timeout: (numeric, default: 300)
       Timeout of connection. Unit is second.
     - ftp-login-retry: (numeric, default: none)
       *This function is not implemented yet.*
     - ftp-login-delay: (numeric, default: 60)
     - ftp-stats: (boolean, default: no)
       If it set yes, when mirroring is finished, total transaction staistics is shown. It appear "control connection in/out", and "data connection in/out" by octet units.
     - http-Proxy: (string, default: none) http-proxy settings when you need accessing ftp server with http. But, http cannot get direcroty information, so use ls-LR file, or using slave mode.
       For example,
       - package = FreeBSD
       - ftp-server = ftp.tokyonet.ad.jp
       - http-proxy = proxy.my.company.com
       - remote-directory = /pub/FreeBSD
       - local-directory = ~ftp/pub/FreeBSD
       - lslR-file = /pub/FreeBSD/ls-lR.Z
       Notice: HTTP is usgin per file transfer TCP connection. So cause large session logs, and heavy latency.
   - Parameters for a package
     - server: (boolean, default none)
       This is used to define FTP server specific parameters. After this parameter, any parameter are treated as the server specific parameter, until package or (next)server parameter will be found. Especially, it's reasonable to specify
   - Parameters for a mirroring
     - unlink:(boolean, default: no)
       When set yes, local directory trees are copied to remote hosts. For example,
       - package = webcopy
       - ftp-server = www.intec.co.jp
       - ftp-user = webadmin
       - ftp-pass = nan-jara-hoi
       - ftp-passive = yes
       - put-mode = yes
       - remote-directory = /usr/local/etc/httpd/htdocs
       - local-directory = ~ikuo/web/htdocs
         - unlink: (boolean or 'rename' default: yes)
           Set this parameter "yes" or 1 if you want to unlink a file which is removed from servers directory. You may set this parameter as "rename", in this case, ftpmirror renames a file with a trailing Tilda('~').
         - unlink-limit: (size value, default: 0)
           ** !!! This is not support current version. This is obsolete.**
           If this parameter is defined, ftpmirror will not remove large files or large directories. This may help you when server side directory hierachy is dramatically changed. For example, if `unlink-limit` is defined as 30M, any file or directory who is larger than 30Mbytes will not be removed. You may also specify 500K or 3G as the maximum size. When `unlink-limit` is defined as 100 (e.g. no size suffix), ftpmirror will not remote the directory who has more than 100 entries.
         - ftp-force-mtime:(boolean, default: no)
           When it set yes, time comparing is using MDTM commands.It set no, MDTM commands does not call using only file transfering.
         - ignore-mtime: (boolean, default: no)
         - check-mtime: (default: 1)
           By default, ftpmirror try to check modified times of local and remote files. If this parameter is zero, ftpmirror does NOT check modified time. In this case, if local and remote files have same size ftpmirror will only change local file timestamp to match remote one.
         - temp-directory: (pathname, default: /tmp)
           Ftpmirror stores temporary files to the directory specified by this parameter. Currently, ls-lR or dirinfo files may be stored in this directory.
         - lock-directory: (pathname, default: /tmp)
           Ftpmirror use lock mechanism to avoid invoking multiple ftpmirrors for a package. lock-directory parameter means the directory to create lock file. local-db
         - create-directory : (boolean, default: yes) When local-directory, temp-directory, lock-directory is set, but target directories dont exists, case of set yes to mkdir, set no to not and error aborting.
         - remote-directory: (boolean, default: none)
           You must specify the directory on the FTP server where you want to mirror as this parameter. Any mirror is recursively performed from this directory.
         - local-directory: (pathname, default: none)
           You must set this parameter to the directory in local-side host, e.g. the directory paired with remote-directory. Next example shows that ftpmirror will mirror "/pub/FreeBSD" in the FTP server to "/ftp/pub/FreeBSD" in local.

               remote-directory = /pub/FreeBSD
               local-directory = /ftp/pub/FreeBSD

           If home-directory is defined (or ftp user's home directory exists), you can specify as follows:

               remote-directory = /pub/FreeBSD
               local-directory = ~/pub/FreeBSD

         - master-db-directory: (pathname, default: none)
           ftpmirror can store localhosts filelist and last mirroring datas, if use this features, set master-db-directory and create storing directory. For example, following esettings will create filelist of ~ftp/pub/FreeBSD files. In this case, "n" is index files revision. And cycle a mirroring, ~ftp/db/FreeBSD/step.m is created too. "m" is step files revisions. step.m is diff of index.m => index.m+1.

               	package = FreeBSD
               	 ftp-server = ftp.FreeBSD.ORG
               	 remote-directory = /pub/%s
               	 local-directory = ~ftp/pub/%s
               	 master-db-directory = ~ftp/db/%s

           Addition, when dont use mirroring, master-db-direcoty can manage directory tree. Example is below.

               	package = original-tools
               	 local-directory = ~ftp/pub/%s
               	 master-db-directory = ~ftp/db/%s

           For update database only please execute:

               	% ftpmirror --todo=update-master original-tools

           If remote server has a master-db-directory, it can use "slave mode" mirroring. Please set remote-db-directory and local-db-cirectory parameters.
         - remote-db-directory: (pathname, default: none)
         - local-db-directory: (pathname, default: none)
           When remote server has a remote-db-directory, it can efficiency mirroring. This called "slave mode". "slave mode" can be use setting remote-db-directory and local-db-directory. If you use it, please create directory that local-db-directory specified. A parameters of master-db-directory is must set remote sides master-db-direcotry.
           These parameters can %s conversion(same local-directry parameters) And, local-db-directory supports tilda(~) directory conversion. When remote-db-directory and local-db-directory set, ftpmirror's behavior is below. At first, starts sync master-db-directory, second, start mirroring with db information. In this case, all file informations are provide by db, so without LIST, STAT commands. And, database syncronization is not fully copy, this synthethis a old index file and step files. Full copy is only apper at two case one is first copy, one of db files are broken. For example, using "slave mode" with adove(master-db-directory's example) server, sample configuration is below.

               	package = FreeBSD
                	 ftp-server = ftp.master.server.host
               	 remote-directory = /pub/%s
               	 local-directory = ~ftp/pub/%s
               	 remote-db-directory = /db/%s
               	 local-db-directory = ~ftp/db/%s

           For advence, you can cumulative mirroring with step files. Configuration is same adove, use:

               	 % ftpmirror --todo=step-mirror FreeBSD

         - lslR-file: (pathname, default: none)
           You can use ls-lR(or similar) on the FTP server rather than directory scan with STAT/LIST. Specify server side filename of ls-lR(or similar) as this parameter. If `lslR-file` has an extension of .gz or .Z, gunzip or uncompress will be automatically invoked. For example, if you mirror ftp.freebsd.org:/pub/FreeBSD with ls-lR.gz, you may specify as next:

               ftp-server = ftp.freebsd.org
               remote-directory = /pub/FreeBSD
               local-directory = ~/pub/FreeBSD
               lslR-file = /pub/FreeBSD/ls-lR.gz

         - lslR-copy: (pathname, default: none)
           When you need archive copying lslR-file on localhost, set this parameter. If you set a relative path, a path leading from local-directory. And, relative path set override-file parameters automatically. So lslR-copy is prevent from overwrite with mirroring. This setting is set, when lslR-file is transfered, and chacheing to lslR-copy file, then prevent needless transfer. The check of size/last modified are non-transferring with lslR-file.
           Addition, when you set this parameters, you can use efficient mirroring with ls-lR file without lslR-file parameters.
         - transfer-file-regexp: (regexp, default: matches all)
           Regular expressions(perl's regular expression) to match filenames to transfer. Ftpmirror tries to transfer only files on the server which matches to this expression. Any filename on the FTP server is represented as relative path (from remote-directory) beginning with "./". That is, if remote-directory is configured as "/pub/FreeBSD", the file "/pub/FreeBSD/2.1.5-RELEASE/README.TXT" on the server is represented as "./2.1.5-RELEASE/README.TXT".

           Some examples follow:

                 transfer-file-regexp += !/~$/
                 transfer-file-regexp += !/\/\.in/
                 transfer-file-regexp += !/\/\.nfs/

           With this configuration, ftpmirror does NOT transfer files end with "~" and begin with ".in" or ".nfs".

           (Notes: "=" means replacing a parameter value, and "+=" means appending a parameter string)

                 transfer-file-regexp += /\/bash-/
                 transfer-file-regexp += /\/gcc-/
                 transfer-file-regexp += !

           With adobe configuration, ftpmirror tries to transfer bash-* and gcc-* files, but not others. The last line in this example means ftpmirror does NOT transfer by default.
         - transfer-directory-regexp: (default: matches all)
           Regular expressions(perl's regular expression) to match directory names to scan in a FTP server. Note that any directory in a FTP server is represented with trailing slash ('/'). For example, next shows configuration to avoid ftpmirror NOT to scan(off course, this means NOT to transfer) directories whose names are "lost+found", or directories ended with '~'.

                 transfer-directory-regexp += !/\/lost+found\/$/
                 transfer-directory-regexp += !/~\/$/

         - override-file-regexp: (default: matches all)
           Regular expressions(perl's regular expression) to match files you can override in local directories. In actual mirror session, ftpmirror tries to transfer files which match `transfer-file-regexp`, and also match `override-file-regexp`. Any file which does not match `override-file-regexp` will not be modified by ftpmirror. In the next example, ftpmirror tries to transfer bash-* files, but not modified any other files. This is useful when you want to get files from multiple FTP servers to a single local-side directory.

                 transfer-file-regexp += /\/bash-/
                 transfer-file-regexp += !
                 override-file-regexp += /\/bash-/
                 override-file-regexp += !

         - override-directory-regexp: (default: matches all)
           Regular expressions(perl's regular expression) to match directories ftpmirror will modify. You can transfer directories from multiple FTP servers to a single local-side directory with this parameter. Following example shows that you can transfer the FreeBSD-nonUS into the FreeBSD directory.

                 package = FreeBSD
                  ftp-server = ftp.tokyonet.ad.jp
                  remote-directory = /pub/FreeBSD
                  local-directory = ~/pub/FreeBSD
                  override-directory-regexp += !/\.\/FreeBSD-nonUS\//

                 package = FreeBSD-nonUS
                  ftp-server = ftp.waseda.ac.jp
                  remote-directory = /pub/FreeBSD-nonUS
                  local-directory = ~/pub/FreeBSD/FreeBSD-nonUS

         - load-local-dirinfo: (default: 0)
           Set this parameter to 1 if you want to load local ".dirinfo" files. A ".dirinfo" file contains directory information described below.
         - store-local-dirinfo: (default: 0)
           Set this parameter to 1 if you want to generate ".dirinfo" files in local-side directories after each mirror session. A ".dirinfo" file contains directory information described below.
         - load-remote-dirinfo: (default: 0) Set this parameter to 1 if you want to load and use directory information stored in the FTP server. Using directory information, you can mirror files without STAT/LIST/SIZE/MDTM, and ftpmirror can check MD5 checksum of files.
         - override-file-uid: (default: 0)
           You can specify owner of files/directories in local. Any files and directories which match `override-*-regexp`, will be owned by this user id. If ftpmirror is invoked by non privileged user, this parameter is ignored.
         - override-file-gid: (default: 0)
           Same as `override-file-uid`, but this parameter is for group id. Note that if one of `override-file-uid` and `override-file-gid` is defined, any file which matches `override-*-regexp` (even if it is not transfered) will be modified with chown.
         - override-file-mode: (default: root -> 0444, other -> 0644)
           You can specify local-side file mode. Ftpmirror applies chmod to any file which matches `override-*-regexp` if this parameter is specified.
         - override-directory-mode: (default: 0755)
           Same as `override-file-mode`, but this parameter is for directories.
         - default-file-uid: (default: 0)
           Undef `override-file-uid`, and set this parameter to specify default owner of local-side files. Ftpmirror decides file owner in order: `override-file-uid`, owner in FTP server, `default-file-uid`.
         - default-file-gid: (default: 0)
           Same as `default-file-uid`, but this parameter is for group id.
         - default-file-mode: (default: root -> 0444, other -> 0644)
           Undef `override-file-mode` and you can specify default file mode. Ftpmirror tries to change file mode in order: `override-file-mode`, file mode in FTP server, `default-file-mode`. Note that since file mode is masked with 0777, you can not set setuid/setgid file modes.
         - default-directory-mode: (default: 0755)
           Same as `default-file-mode`, but this parameter is for a directory mode.
         Ftpmirror tries to compare files in local and remote with these entries. For example, ftpmirror compares as:
         1. file file-type
            Are sizes same? and are md5checksums same?
         2. symlink file-type
            Do they point same file?
         3. directory file-type
            Is the begin-time of last update of local directory is before the end-time of last update of remote directory?
         Especially, comparing directory update times are effective, if no file is changed in the remote directory, because ftpmirror do nothing about the directory (even directory scan is skipped).

         Ftpmirror generate ".dirinfo" file if `store-local-dirinfo` parameter is 1. You can also generate this file with mkdirinfo:

              # mkdirinfo /ftp/pub/utils

         ## Mailing list for ftpmirror

         Technical questions, suggestions, bug reports are welcome. If you are interesting to ftpmirror, there is a mailing-list to discuss about ftpmirror. To subscribe, send a message

              subscribe ftpmirror

         to majordomo@ftp.intec.co.jp.

         ## Credits

         I would like to thank to anyone who helped me by testing, bug-reports, suggestions, sending patches. Especially, members in RingProject helped me by heavy testing `:-)` and many suggestions.

         ## Miscellaneous...

         Please read COPYRIGHT for copyright notice. Any comments, bug fixes, suggestions are welcome. Please contact to Ikuo Nakagawa <ikuo@intec.co.jp>.

         Last updated: 1997/01/27 by Ikuo Nakagawa
         This is community translated file. This is not original document. Translated on 2007-07-07 by Fumihito YOSHIDA (hito@kugutsu.org). Thank you.