git-annex/ special remotes/ rsync

This special remote type rsyncs file contents to somewhere else.

Setup example:

    # git annex initremote myrsync type=rsync rsyncurl=rsync://rsync.example.com/myrsync encryption=shared
    # git annex describe myrsync "rsync server"

Or for using rsync over SSH

    # git annex initremote myrsync type=rsync rsyncurl=ssh.example.com:/myrsync encryption=shared
    # git annex describe myrsync "rsync server"

## configuration

These parameters can be passed to `git annex initremote` to configure rsync:

- `rsyncurl` - Required. This is the url or `hostname:/directory` to pass to rsync to tell it where to store content.

- `encryption` - One of "none", "hybrid", "shared", or "pubkey". See encryption.

- `keyid` - Specifies the gpg key to use for encryption.

- `exporttree` - Set to "yes" to make this special remote usable by git-annex-export. It will not be usable as a general-purpose special remote.

- `annexobjects` - When set to "yes" along with "exporttree=yes", this allows storing other objects in the remote along with the exported tree. They will be stored under .git/annex/objects/ in the remote.

- `shellescape` - Optional. This has no effect when using rsync 3.2.4 or newer. Set to "no" to avoid shell escaping normally done when using older versions of rsync over ssh. That escaping is needed with typical setups, but not with some hosting providers that do not expose rsynced filenames to the shell. You'll know you need this option if `git annex get` from the special remote fails with an error message containing a single quote (`'`) character. If that happens, you can run enableremote setting shellescape=no.

- `chunk` - Enables chunking when storing large files.
  This is typically not a win for rsync, so no need to enable it. But, it makes this interoperate with the directory special remote.

The `remote.name.annex-rsync-options` git configuration setting can be used to pass parameters to rsync. To pass parameters to rsync only when it's downloading and uploading, use `remote.name.annex-rsync-download-options` and `remote.name.annex-rsync-upload-options`

## annex-rsync-transport

You can use the `remote.name.annex-rsync-transport` git configuration setting to choose whether we run rsync over ssh or rsh. This setting is also used to specify parameters that git annex will pass to ssh/rsh.

ssh is the default transport; if you'd like to run rsync over rsh:

    git config remote.name.annex-rsync-transport rsh

To pass parameters to ssh/rsh, include the parameters after "rsh" or "ssh". For example, to configure ssh to use the private key at `/path/to/private/key`:

    git config remote.name.annex-rsync-transport "ssh -i /path/to/private/key"

Added use cases to "special remotes"
Thanks @joeyh. I've taken the liberty to add your use case description to special remotes. Hope this helps.
Comment by http://olivier.berger.myopenid.com/ — Tue Jul 22 18:11:47 2025
comment 8

There are many use cases for a rsync special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could `git annex move --to rsyncremote` large files when your local drive is getting full, and then `git annex move` the files back when free space is again available. You could have one repository copy files to a rsync remote, and then `git annex get` them on another repository, to transfer the files between computers that do not communicate directly. The git-annex assistant makes it easy to set up rsync remotes using this last scenario, which is referred to as a transfer repository, and arranges to drop files from the transfer repository once they have been transferred to all known clients.

None of these use cases are particular to rsync remotes. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
rsync remote is basically more intended for backups ?

If I get it correctly, it is mainly useable as a backup, which will accumulate contents of the objects managed by git-annex over time.

It would be great to have a use case illustrating its use in concrete matters. Thanks in advance.

Comment by http://olivier.berger.myopenid.com/ — Tue Jul 22 18:11:47 2025
comment 6
The first example on this page shows using rsync:// to store files on a system using the rsync daemon.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
rsync daemon mode

rsync has a --daemon mode with a simple challenge-response authentication but no encryption. This offers a nice lightweight alternative to ssh, especially when we store/transfer encrypted content anyways. Is this already supported in git-annex, if yes how to set it up?

Comment by cehteh — Tue Jul 22 18:11:47 2025
rsync.net support git and rsync, not git-annex

how would i use rsync.net as a full repository ? (annex files and git repo)

It support's rsync, and I discovered now git as well.

I had a look at the webapp and can't figure out how to set it up, so I can have multiple remote's sync to rsync.net for annex files and git sync.

Link: http://www.rsync.net/resources/howto/github_integration.html

Comment by diepes — Tue Jul 22 18:11:47 2025
sshfs ? and no sync from special remote to 2nd git-annex ?

- It sound to me it would be 1st prize if the cloud provider supported the git-annex functionality over ssh.
- Then it could be a full git-annex repo, and used for recovery if my laptop with the git info gets lost.
- From special remotes "These can be used just like any normal remote by git-annex"

- Your comment "No, special remotes do not contain a copy of the git repository"

- so a special remote is

- "A. just a remote filesystem, that contains the objects with sha1 names ? "

- "B. there is no git info and thus no file name & directory details or am i missing something ?"
- "C. you can't use the remote as a cloud drive to sync changes e.g. file moves, renames between two other git-annex repositories ? (no meta data)"

- "D. the data on the cloud/rsync storage can not be used directly it has to moved into a git-annex capable storage location. "

- Would it not be better to mount the remote storage over ssh(sshfs) and then use full git-annex on mounted directory ?

Comment by diepes — Tue Jul 22 18:11:47 2025
comment 2
No, special remotes do not contain a copy of the git repository, and no, git-annex does not notice when files are added to remote rsync repositories. Suggest you read special remotes.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 16

turns out that if you get into the habit of writing rsync remotes like rclone remotes, git annex breaks

I had a SSH config entry with the name `annex` and set the rsync URL to `annex:`. The works for moving content in, but fsck and get will fail and undo the content tracking.

Comment by aurelia — Tue Jul 22 18:11:47 2025
comment 15

@davidriod you can do things like this with special remotes, as long as the special remotes are not encrypted.

I don't really recommend it. With such a shared special remote R and two disconnected git repos -- call them A and B, some confusing situations can occur. For example, the only copies of some files may be on special remote R and git repo B. A knows about the copy in R, so git-annex is satisfied there is one copy of the file. But now, B can drop the content from R, which is allowed as the content is in B. A is then left unable to recover the content of the files at all, since they have been removed from R.

Better to connect the two repositories A and B, even if you do work in two separate branches. Then if a file ends up located only on B, A will be able to say where it is, and could even get it from B (if B was set up as a remote).

Comment by joey — Tue Jul 22 18:11:47 2025
Sharing rsync special remote between repository
I was wondering if it is possible to share a rsync special remote between repository which are not parented in any way. The use case would be that even if these repositories are not related at all they still may contains the same binary file. It would be useful to have a single rsync remote in order to reduce space usage. I think it could work as the object names are based on their checksum, but I wonder if anyone has already try that ?
Comment by davidriod — Tue Jul 22 18:11:47 2025
comment 13

@davidriod if you're using a rsyncurl that uses ssh, then yes, the transmission goes over a secure connection. If the rsyncurl uses the rsync protocol, there would be no encryption.

Of course, encryption=none does not keep the data encrypted at rest, so the admin of the rsync server can see it etc.

Comment by joey — Tue Jul 22 18:11:47 2025
ssh and encryption
I was wondering if the transmission of data was safe over ssh with encryption=none ?
Comment by davidriod — Tue Jul 22 18:11:47 2025
comment 11
@Chris yes in that case you still need, a central git repository (which need not be on a host that supports git-annex), or the assistant can use xmpp to sync the git data.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 10
@joey I don't understand you last comment where you state that special remotes can act as transfer repositories "to transfer the files between computers that do not communicate directly". If there's no communication, ie git pushes or pulls, between the computers then they don't know what file names the files on the special remote map to. They need to somehow communicate the git repo too, don't they?
Comment by http://cstork.org/ — Tue Jul 22 18:11:47 2025
rsync description and example

Hi, I would like to see a example of setting up / using e.g. rsync.net as a repo.

Please also provide a highlevel description of how the rsync repo fits in with git-annex.

Q1. can you adds files to the remote rsync repo, and will they be detected and synced back ? Q2. is all the git history rsync'd to remote ? how do i recover if i loose all data except the remote rsync repo ?

Comment by diepes — Tue Jul 22 18:11:47 2025
Comments on this page are closed.