                                   FTPMIRROR

Introduction

   Ftpmirror is an utility to copy directory hierarchy (this is called
   ``mirror'') with FTP. A transfer is triggered only when modified, it can
   be efficient transferring directory trees.

   It is userful for ftp-servers maintenance, contentns sync with web server,
   system backupping and so on.

   A similar perl script exists, whose name is `mirror', this implementation
   is fully comparing directory lists of sources and distination, it take a
   heavy memmory load. If use on a large directory trees, so eat a lot of
   memory(or swap) spaces. When I use ftp-server with 16MB memory, 'mirror'
   software is not usable :-p, so I wrote software from scratch, that is
   frpmirror. For example, full mirroring of FreeBSD distribution
   (ftp://ftp.FreeBSD.ORG/pub/FreeBSD/*) needs 8~9 MB. *(Notice by
   translator) Original document wrote at old(1998). May be this results has
   obsolete.*

   You need perl-5.003 or later to use ftpmirror. Please ensure that
   perl-5.003 or later was installed in your environment, at first. But, perl
   5.004 has a bug of memory leak. It eat a lot of memory spaces when
   mirroring directory tree. I test with perl5.005, it has small memory leak
   too, it is smaller than perl 5.004, it is not critical for real usage. I
   suggest you using Perl 5.005.

How to use ftpmirror

    1. Install ftpmirror first. The newest version of ftpmirror is available
       from ftp.intec.co.jp.

       Please use these URLs.
          * ftp://ftp.intec.co.jp/pub/utils/ftpmirror/ : release
          * ftp://ftp.intec.co.jp/pub/utils/ftpmirror/beta/ : beta

       Since ftpmirror is a archive of instal files, you must only extract
       the archive. Do one of following:

   % cd /usr/local/src
 % gzip -cd < /tmp/ftpmirror-x.y.tar.gz | tar xf -

       So create a directory named ftpmirror-x.y. Change directory and
       execute configure:
       Notice: If you need define specific perl verions, please set perls
       path for enviroment variable "PERL".

 % cd ftpmirror-x.y
 % ./configure

       Or, when you need specific perl version:

 % cd ftpmirror-x.y
 % env PERL=/usr/local/bin/perl5.005 ./configure

       Execute make:

 % make

       When finished make with no errors. So install:

 % su
 # make install

       In finished this procedure, so installed files are below paths:

 /usr/local/bin/ftpmirror
 /usr/local/bin/rotate
 /usr/local/etc/ftpmirror.cf-sample
 /usr/local/lib/perl5/site_perl/Fan.pm
 /usr/local/lib/perl5/site_perl/Fan/...   (and some dirs)
 /usr/local/lib/perl5/site_perl/auto/Fan/...  (and some dirs)

 * Notice: libraries install directory can changes
     with perl's installed settings.

       And copying and editing a configuration files, please add difinition
       what you needs.

 # cd /usr/local/etc
 # cp ftpmirror.cf-sample ftpmirror.cf
 # vi ftpmirror.cf

Configuration

       You can define parameters for ftpmirror with the default configuration
       file, package specific configuration file, and run-time options
       (commands arguments). In addition, configuration files are set
       /usr/local/etc/ftpmirror.cf by defauls, it can change by command
       argument "--load-config=/hoge/local.cf" if you need. And argument
       "--load-config=+/hogehoge/local.cf" is used, this is append reading
       after adove defaults reading.

       Configuration parameters have follwoing types.

 Default parametes:
 Configuration files settings. They are written before
 Serverss and Packages parameters.

 Server parameters:
 Difinition of server specific(timeouts, gateways and
 so on..). They are provided at configuration files,
 with 'server' parameters.

 Package parameters:
 Difinition of packages specific(direcotry, transfer
 modes and so on). They are provided at configuration files,
 with 'package' parameters.

 Option parameters:
 It provide by command args.

       Their parameters are settings by adove order(when exists). Option
       parameters are most prefferd.

       How to settings parameters at configuration files:
       That has two syntaxr,

 param-name = value
 or
 param-name += value

 =  is set param-name by value.
 += is adding value(strings) for param-name.

       How to settings parameters at commands args:
       When using by commands args,any parameters must be prepended by "--"
       in command line options. That use --(two minus) before options. For
       example, following syntax is same as adove configuration settings.

 % ftpmirror --param-name=value ...
 % ftpmirror --param-name+=value ...

Parameters

       There are many acceptable parameters as follows:
          * Parameters for system and behavior
               * todo:(string, default: full-mirror)
                 Settings of command behavior. Currently ftpmirror supports
                 these behavior.
                    * scan-remote : listup remote sides files.
                    * scan-local : listup local sides files.
                    * update-master : update local sides master index.
                    * synch-remote : synchronize index files.
                    * step-mirror : mirroring as slave mode.
                    * full-mirror : execute basic mirroring(default).
                 In defaults, it set basic mirroring behavior "full-mirror".
                 *-regexp filters are enabled at first, any behavior works
                 with these filters.
               * verbose: (boolean, default: no)
                 When set yes, transfer logs ouput verbosey. If set yes, logs
                 output about non-transfererd/non-modified files.
               * log-mask: (comma separated list, default: none)
                 Settings about output logs. These difinition separates by
                 perl libraries. For example, log-mask = Fan=6,Fan::FTP=7 are
                 set, a output log include filelist of non transfererd/ non
                 modified, and ftp's verbosey log.
               * test-mode (boolean, default: no)
                 dry-run. When set yes, only show filelist of transfer
                 targets without real transfer.
          * Parameters for FTP session
               * ftp-server: (string, default: none)
                 The server name to connect FTP session. This must be
                 specified for mirroring.
               * ftp-gateway: (string, default: none)
                 Specify this parameter if you use FTP proxy gateway such as
                 TIS Internet FireWall ToolKit. For example, "ftp-user =
                 anonymous", "ftp-server = ring.etl.go.jp", "ftp-gateway =
                 proxy.intec.co.jp" are set, real-connection goes to
                 proxy.intec.co.jp, and username is anynymou@ring.etl.go.jp
                 (ftp-pass is through simple). You may define this with
                 server parameter.
               * ftp-port: (service, default: ftp) ftp port definition. If
                 you need change a port number, please set. When nomarlly
                 usage, it is not necessary.
               * ftp-bindaddr: (ip address, default: none) ftp local sides IP
                 address difinitition. This can set a specific difinition of
                 local ip address. When using multi-homed hosts(The hosts
                 have multiple ip addresses), this settings provides a clear
                 log files. When nomarlly usage, it is not necessary.
               * ftp-user: (string, default: anonymous)
                 Login name for FTP session. If you want to use anonymous
                 FTP, please specify "anonymous" as this parameter.
               * ftp-pass: (string, default: `whoami`@`hostname`)
                 Password for FTP session.
               * ftp-group: (string, default: none)
                 If you want change group id after logged to the server,
                 specify the group name. This must be specified with
                 ftp-gpass.
               * ftp-gpass: (string, default: none)
                 If you want change group id after logged to the server,
                 specify a password to change group. This must be specified
                 with ftp-group.
               * ftp-passive: (boolean, default: no)
                 Set this parameter to yes if passive connection(PASV command
                 in FTP) is required. This helps you if you use ftpmirror
                 over a SYN/ACK style firewall.
               * ftp-idle: (numeric, default: 0)
                 If this parameter is not zero, ftpmirror tries to set idle
                 timer of FTP session to this value. FTP server must support
                 SITE IDLE command.
               * ftp-max-idle: (boolean, default: 0)
                 If this parameter is not zero, ftpmirror tries to set idle
                 timer of FTP session to maximum value. FTP server must
                 support SITE IDLE command.
               * ftp-list-method: (LIST or STAT or STAT-A or STAT-A or
                 STAT-AT default: STAT)
                 You can specify which of STAT or LIST command should be used
                 to get server's directory entries. Since STAT command
                 returns server's directory entries over control connection,
                 it is little faster than LIST command. Some ftp server
                 software does not returns directory entry which is required
                 by ftpmirror, in that case you must speficy as
                 ftp-list-method = list.
               * ftp-timeout: (numeric, default: 300)
                 Timeout of connection. Unit is second.
               * ftp-login-retry: (numeric, default: none)
                 *This function is not implemented yet.*
               * ftp-login-delay: (numeric, default: 60)
               * ftp-stats: (boolean, default: no)
                 If it set yes, when mirroring is finished, total transaction
                 staistics is shown. It appear "control connection in/out",
                 and "data connection in/out" by octet units.
               * http-Proxy: (string, default: none) http-proxy settings when
                 you need accessing ftp server with http. But, http cannot
                 get direcroty information, so use ls-LR file, or using slave
                 mode.
                 For example,
                    * package = FreeBSD
                    * ftp-server = ftp.tokyonet.ad.jp
                    * http-proxy = proxy.my.company.com
                    * remote-directory = /pub/FreeBSD
                    * local-directory = ~ftp/pub/FreeBSD
                    * lslR-file = /pub/FreeBSD/ls-lR.Z
                 Notice: HTTP is usgin per file transfer TCP connection. So
                 cause large session logs, and heavy latency.
          * Parameters for a package
               * server: (boolean, default none)
                 This is used to define FTP server specific parameters. After
                 this parameter, any parameter are treated as the server
                 specific parameter, until package or (next)server parameter
                 will be found. Especially, it's reasonable to specify
          * Parameters for a mirroring
               * unlink:(boolean, default: no)
                 When set yes, local directory trees are copied to remote
                 hosts. For example,
                    * package = webcopy
                    * ftp-server = www.intec.co.jp
                    * ftp-user = webadmin
                    * ftp-pass = nan-jara-hoi
                    * ftp-passive = yes
                    * put-mode = yes
                    * remote-directory = /usr/local/etc/httpd/htdocs
                    * local-directory = ~ikuo/web/htdocs
                         * unlink: (boolean or 'rename' default: yes)
                           Set this parameter "yes" or 1 if you want to
                           unlink a file which is removed from servers
                           directory. You may set this parameter as "rename",
                           in this case, ftpmirror renames a file with a
                           trailing Tilda('~').
                         * unlink-limit: (size value, default: 0)
                           ** !!! This is not support current version. This
                           is obsolete.**
                           If this parameter is defined, ftpmirror will not
                           remove large files or large directories. This may
                           help you when server side directory hierachy is
                           dramatically changed. For example, if unlink-limit
                           is defined as 30M, any file or directory who is
                           larger than 30Mbytes will not be removed. You may
                           also specify 500K or 3G as the maximum size. When
                           unlink-limit is defined as 100 (e.g. no size
                           suffix), ftpmirror will not remote the directory
                           who has more than 100 entries.
                         * ftp-force-mtime:(boolean, default: no)
                           When it set yes, time comparing is using MDTM
                           commands.It set no, MDTM commands does not call
                           using only file transfering.
                         * ignore-mtime: (boolean, default: no)
                         * check-mtime: (default: 1)
                           By default, ftpmirror try to check modified times
                           of local and remote files. If this parameter is
                           zero, ftpmirror does NOT check modified time. In
                           this case, if local and remote files have same
                           size ftpmirror will only change local file
                           timestamp to match remote one.
                         * temp-directory: (pathname, default: /tmp)
                           Ftpmirror stores temporary files to the directory
                           specified by this parameter. Currently, ls-lR or
                           dirinfo files may be stored in this directory.
                         * lock-directory: (pathname, default: /tmp)
                           Ftpmirror use lock mechanism to avoid invoking
                           multiple ftpmirrors for a package. lock-directory
                           parameter means the directory to create lock file.
                           local-db
                         * create-directory : (boolean, default: yes) When
                           local-directory, temp-directory, lock-directory is
                           set, but target directories dont exists, case of
                           set yes to mkdir, set no to not and error
                           aborting.
                         * remote-directory: (boolean, default: none)
                           You must specify the directory on the FTP server
                           where you want to mirror as this parameter. Any
                           mirror is recursively performed from this
                           directory.
                         * local-directory: (pathname, default: none)
                           You must set this parameter to the directory in
                           local-side host, e.g. the directory paired with
                           remote-directory. Next example shows that
                           ftpmirror will mirror "/pub/FreeBSD" in the FTP
                           server to "/ftp/pub/FreeBSD" in local.

 remote-directory = /pub/FreeBSD
 local-directory = /ftp/pub/FreeBSD

                           If home-directory is defined (or ftp user's home
                           directory exists), you can specify as follows:

 remote-directory = /pub/FreeBSD
 local-directory = ~/pub/FreeBSD

                         * master-db-directory: (pathname, default: none)
                           ftpmirror can store localhosts filelist and last
                           mirroring datas, if use this features, set
                           master-db-directory and create storing directory.
                           For example, following esettings will create
                           filelist of ~ftp/pub/FreeBSD files. In this case,
                           "n" is index files revision. And cycle a
                           mirroring, ~ftp/db/FreeBSD/step.m is created too.
                           "m" is step files revisions. step.m is diff of
                           index.m => index.m+1.

         package = FreeBSD
          ftp-server = ftp.FreeBSD.ORG
          remote-directory = /pub/%s
          local-directory = ~ftp/pub/%s
          master-db-directory = ~ftp/db/%s

                           Addition, when dont use mirroring,
                           master-db-direcoty can manage directory tree.
                           Example is below.

         package = original-tools
          local-directory = ~ftp/pub/%s
          master-db-directory = ~ftp/db/%s

                           For update database only please execute:

         % ftpmirror --todo=update-master original-tools

                           If remote server has a master-db-directory, it can
                           use "slave mode" mirroring. Please set
                           remote-db-directory and local-db-cirectory
                           parameters.
                         * remote-db-directory: (pathname, default: none)
                         * local-db-directory: (pathname, default: none)
                           When remote server has a remote-db-directory, it
                           can efficiency mirroring. This called "slave
                           mode". "slave mode" can be use setting
                           remote-db-directory and local-db-directory. If you
                           use it, please create directory that
                           local-db-directory specified. A parameters of
                           master-db-directory is must set remote sides
                           master-db-direcotry.
                           These parameters can %s conversion(same
                           local-directry parameters) And, local-db-directory
                           supports tilda(~) directory conversion. When
                           remote-db-directory and local-db-directory set,
                           ftpmirror's behavior is below. At first, starts
                           sync master-db-directory, second, start mirroring
                           with db information. In this case, all file
                           informations are provide by db, so without LIST,
                           STAT commands. And, database syncronization is not
                           fully copy, this synthethis a old index file and
                           step files. Full copy is only apper at two case
                           one is first copy, one of db files are broken. For
                           example, using "slave mode" with
                           adove(master-db-directory's example) server,
                           sample configuration is below.

         package = FreeBSD
          ftp-server = ftp.master.server.host
          remote-directory = /pub/%s
          local-directory = ~ftp/pub/%s
          remote-db-directory = /db/%s
          local-db-directory = ~ftp/db/%s

                           For advence, you can cumulative mirroring with
                           step files. Configuration is same adove, use:

          % ftpmirror --todo=step-mirror FreeBSD

                         * lslR-file: (pathname, default: none)
                           You can use ls-lR(or similar) on the FTP server
                           rather than directory scan with STAT/LIST. Specify
                           server side filename of ls-lR(or similar) as this
                           parameter. If lslR-file has an extension of .gz or
                           .Z, gunzip or uncompress will be automatically
                           invoked. For example, if you mirror
                           ftp.freebsd.org:/pub/FreeBSD with ls-lR.gz, you
                           may specify as next:

 ftp-server = ftp.freebsd.org
 remote-directory = /pub/FreeBSD
 local-directory = ~/pub/FreeBSD
 lslR-file = /pub/FreeBSD/ls-lR.gz

                         * lslR-copy: (pathname, default: none)
                           When you need archive copying lslR-file on
                           localhost, set this parameter. If you set a
                           relative path, a path leading from
                           local-directory. And, relative path set
                           override-file parameters automatically. So
                           lslR-copy is prevent from overwrite with
                           mirroring. This setting is set, when lslR-file is
                           transfered, and chacheing to lslR-copy file, then
                           prevent needless transfer. The check of size/last
                           modified are non-transferring with lslR-file.
                           Addition, when you set this parameters, you can
                           use efficient mirroring with ls-lR file without
                           lslR-file parameters.
                         * transfer-file-regexp: (regexp, default: matches
                           all)
                           Regular expressions(perl's regular expression) to
                           match filenames to transfer. Ftpmirror tries to
                           transfer only files on the server which matches to
                           this expression. Any filename on the FTP server is
                           represented as relative path (from
                           remote-directory) beginning with "./". That is, if
                           remote-directory is configured as "/pub/FreeBSD",
                           the file "/pub/FreeBSD/2.1.5-RELEASE/README.TXT"
                           on the server is represented as
                           "./2.1.5-RELEASE/README.TXT".

                           Some examples follow:

   transfer-file-regexp += !/~$/
   transfer-file-regexp += !/\/\.in/
   transfer-file-regexp += !/\/\.nfs/

                           With this configuration, ftpmirror does NOT
                           transfer files end with "~" and begin with ".in"
                           or ".nfs".

                           (Notes: "=" means replacing a parameter value, and
                           "+=" means appending a parameter string)

   transfer-file-regexp += /\/bash-/
   transfer-file-regexp += /\/gcc-/
   transfer-file-regexp += !

                           With adobe configuration, ftpmirror tries to
                           transfer bash-* and gcc-* files, but not others.
                           The last line in this example means ftpmirror does
                           NOT transfer by default.
                         * transfer-directory-regexp: (default: matches all)
                           Regular expressions(perl's regular expression) to
                           match directory names to scan in a FTP server.
                           Note that any directory in a FTP server is
                           represented with trailing slash ('/'). For
                           example, next shows configuration to avoid
                           ftpmirror NOT to scan(off course, this means NOT
                           to transfer) directories whose names are
                           "lost+found", or directories ended with '~'.

   transfer-directory-regexp += !/\/lost+found\/$/
   transfer-directory-regexp += !/~\/$/

                         * override-file-regexp: (default: matches all)
                           Regular expressions(perl's regular expression) to
                           match files you can override in local directories.
                           In actual mirror session, ftpmirror tries to
                           transfer files which match transfer-file-regexp,
                           and also match override-file-regexp. Any file
                           which does not match override-file-regexp will not
                           be modified by ftpmirror. In the next example,
                           ftpmirror tries to transfer bash-* files, but not
                           modified any other files. This is useful when you
                           want to get files from multiple FTP servers to a
                           single local-side directory.

   transfer-file-regexp += /\/bash-/
   transfer-file-regexp += !
   override-file-regexp += /\/bash-/
   override-file-regexp += !

                         * override-directory-regexp: (default: matches all)
                           Regular expressions(perl's regular expression) to
                           match directories ftpmirror will modify. You can
                           transfer directories from multiple FTP servers to
                           a single local-side directory with this parameter.
                           Following example shows that you can transfer the
                           FreeBSD-nonUS into the FreeBSD directory.

   package = FreeBSD
    ftp-server = ftp.tokyonet.ad.jp
    remote-directory = /pub/FreeBSD
    local-directory = ~/pub/FreeBSD
    override-directory-regexp += !/\.\/FreeBSD-nonUS\//

   package = FreeBSD-nonUS
    ftp-server = ftp.waseda.ac.jp
    remote-directory = /pub/FreeBSD-nonUS
    local-directory = ~/pub/FreeBSD/FreeBSD-nonUS

                         * load-local-dirinfo: (default: 0)
                           Set this parameter to 1 if you want to load local
                           ".dirinfo" files. A ".dirinfo" file contains
                           directory information described below.
                         * store-local-dirinfo: (default: 0)
                           Set this parameter to 1 if you want to generate
                           ".dirinfo" files in local-side directories after
                           each mirror session. A ".dirinfo" file contains
                           directory information described below.
                         * load-remote-dirinfo: (default: 0) Set this
                           parameter to 1 if you want to load and use
                           directory information stored in the FTP server.
                           Using directory information, you can mirror files
                           without STAT/LIST/SIZE/MDTM, and ftpmirror can
                           check MD5 checksum of files.
                         * override-file-uid: (default: 0)
                           You can specify owner of files/directories in
                           local. Any files and directories which match
                           override-*-regexp, will be owned by this user id.
                           If ftpmirror is invoked by non privileged user,
                           this parameter is ignored.
                         * override-file-gid: (default: 0)
                           Same as override-file-uid, but this parameter is
                           for group id. Note that if one of
                           override-file-uid and override-file-gid is
                           defined, any file which matches override-*-regexp
                           (even if it is not transfered) will be modified
                           with chown.
                         * override-file-mode: (default: root -> 0444, other
                           -> 0644)
                           You can specify local-side file mode. Ftpmirror
                           applies chmod to any file which matches
                           override-*-regexp if this parameter is specified.
                         * override-directory-mode: (default: 0755)
                           Same as override-file-mode, but this parameter is
                           for directories.
                         * default-file-uid: (default: 0)
                           Undef override-file-uid, and set this parameter to
                           specify default owner of local-side files.
                           Ftpmirror decides file owner in order:
                           override-file-uid, owner in FTP server,
                           default-file-uid.
                         * default-file-gid: (default: 0)
                           Same as default-file-uid, but this parameter is
                           for group id.
                         * default-file-mode: (default: root -> 0444, other
                           -> 0644)
                           Undef override-file-mode and you can specify
                           default file mode. Ftpmirror tries to change file
                           mode in order: override-file-mode, file mode in
                           FTP server, default-file-mode. Note that since
                           file mode is masked with 0777, you can not set
                           setuid/setgid file modes.
                         * default-directory-mode: (default: 0755)
                           Same as default-file-mode, but this parameter is
                           for a directory mode.
                      Ftpmirror tries to compare files in local and remote
                      with these entries. For example, ftpmirror compares as:
                        1. file file-type
                           Are sizes same? and are md5checksums same?
                        2. symlink file-type
                           Do they point same file?
                        3. directory file-type
                           Is the begin-time of last update of local
                           directory is before the end-time of last update of
                           remote directory?
                      Especially, comparing directory update times are
                      effective, if no file is changed in the remote
                      directory, because ftpmirror do nothing about the
                      directory (even directory scan is skipped).

                      Ftpmirror generate ".dirinfo" file if
                      store-local-dirinfo parameter is 1. You can also
                      generate this file with mkdirinfo:

  # mkdirinfo /ftp/pub/utils

Mailing list for ftpmirror

                      Technical questions, suggestions, bug reports are
                      welcome. If you are interesting to ftpmirror, there is
                      a mailing-list to discuss about ftpmirror. To
                      subscribe, send a message

  subscribe ftpmirror

                      to majordomo@ftp.intec.co.jp.

Credits

                      I would like to thank to anyone who helped me by
                      testing, bug-reports, suggestions, sending patches.
                      Especially, members in RingProject helped me by heavy
                      testing :-) and many suggestions.

Miscellaneous...

                      Please read COPYRIGHT for copyright notice. Any
                      comments, bug fixes, suggestions are welcome. Please
                      contact to Ikuo Nakagawa <ikuo@intec.co.jp>.

--------------------------------------------------------------------------------

      Last updated: 1997/01/27 by Ikuo Nakagawa
      This is community translated file. This is not original document.

--------------------------------------------------------------------------------

 Translated into English on 2007-07-07 by Fumihito YOSHIDA <hito@kugutsu.org>.

--------------------------------------------------------------------------------
