git-annex/ chunking

Most special remotes have support for breaking large files up into chunks that are stored on the remote.

This can be useful to work around limitations on the size of files on the remote.

Chunking also allows for resuming interrupted downloads and uploads.

Note that git-annex has to buffer chunks in memory before they are sent to a remote. So, using a large chunk size will make it use more memory.

To enable chunking, pass a `chunk=nnMiB` parameter to `git annex initremote`, specifying the chunk size.

Good chunk sizes will depend on the remote, but a good starting place is probably `1MiB`. Very large chunks are problematic, both because git-annex needs to buffer one chunk in memory when uploading, and because a larger chunk will make resuming interrupted transfers less efficient. On the other hand, when a file is split into a great many chunks, there can be increased overhead of making many requests to the remote.

To disable chunking of a remote that was using chunking, pass `chunk=0` to `git annex enableremote`. Any content already stored on the remote using chunks will continue to be accessed via chunks, this just prevents using chunks when storing new content.

To change the chunk size, pass a `chunk=nnMiB` parameter to `git annex enableremote`. This only affects the chunk sized used when storing new content.

# old-style chunking

Note that older versions of git-annex used a different chunk method, which was configured by passing `chunksize=nnMib` when initializing a remote.

The old-style chunking had a number of problems, including being less efficient, and not allowing resumes of encrypted uploads.

It's not possible to change a remote using that old chunking method to the new one, but git-annex continues to support the old-style chunking to support such remotes.

See also: design document

comment 4

@mike, you can use `git-annex info $remotename`

The output will include a "chunking" line which ends in "(old style)" when applicable.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 3

I always use this

    git show git-annex:remote.log

to print the current configurations of all remotes. It should have all the arguments you passed to `initremote` or `enableremote`. If there is a better way, I would also like to know.

(the git command prints the content of the file `remote.log` from branch `git-annex`)

Comment by MatusGoljer1 — Tue Jul 22 18:11:47 2025
comment 2
Is there a way to show the chunking parameters used for a remote? In particular I would like to know if I had used the old-style chunking for a remote.
Comment by mike — Tue Jul 22 18:11:47 2025
buffering chunks in memory when uploading to remote
"git-annex has to buffer chunks in memory before they are sent to a remote" -- would it be hard to remove this restriction? I want to have a non-chunked S3 remote, so that files in it can be accessed through a URL. But if some files are 50GB, then git-annex running on a 16GB machine would fail when talking to this repo?
Comment by Ilya_Shlyakhter — Tue Jul 22 18:11:47 2025
Comments on this page are closed.