PARSYNCFP2(1)

NAME
parsyncfp2 - a MultiHost parallel rsync wrapper

SYNOPSIS

parsyncfp2 [OPTION]... SRC... DEST

DESCRIPTION

parsyncfp2 wraps rsync to provide load balancing and parallel operation across network connections to substantially increase the amount of data it can send simultaneously. parsyncfp2 exploits parallel operation to decrease the impact of the TCP Round Trip Time(rtt) to significantly increase the total bandwidth of data across networks. Even on low-latency networks, it can speed large transfers by 4-10x. However, it is not effective for small transfers, since the startup overhead will slow the effective throughput.

parsyncfp2 uses fpart to split the source data into chunks of a specified size (default: 10 GB), and then starts parallel instances of rsync to transfer those chunks. parsyncfp2 only needs to be installed on the source machine. It can optionally monitor the system’s load average and limit the impact of the data transfer. A multi-host mode is also available (several senders and receivers).

MAIN OPTIONS

--np|NP=processes
    Number of parallel rsync processes to start. Default: sqrt(#CPUs)

--chunksize|cs=size
    Aggregate size of files allocated to one rsync process. Can be specified using human-readable suffixes (100M, 1G). Default: 10G

--ro=options
    Options to pass to rsync. Default: -asl

--help
    Show more detailed information about parsyncfp2 usage

IMPORTANT NOTES

rsync --delete options will not work with --ro because the multiple parallel rsyncs that parsyncfp launches are independent and therefore don’t know about each other (and so cannot exchange info about what should be deleted or not. Use a final, separate rsync --delete to clean up the transfer if that’s your need.

parsyncfp2 only works on dirs and files that originate from the current dir (or specified via "--startdir"). You cannot include dirs and files from discontinuous or higher-level dirs.

EXAMPLE

Use 4 rsync processes and a chunksize of 1G, and change rsync options to enable compression
    parsyncfp2 --np=4 --chunksize=1G --ro="-az" source_dir host:target_dir

ADDITIONAL DOCUMENTATION

The parsyncfp2 manual