git-annex/ design/ assistant/ chunks

To avoid leaking even the size of your encrypted files to cloud storage providers, add a mode that stores fixed size chunks.

May be a useful starting point for deltas.

May also allow for downloading different chunks of a file concurrently from multiple remotes.

Also, can allow resuming of interrupted uploads and downloads.

# legacy chunking

Supported by only the webdav and directory special remotes.

Filenames are used for the chunks that make it easy to see which chunks belong together, even when encryption is used. There is also a chunkcount file, that similarly leaks information.

It is not possible to enable chunking on a non-chunked remote.

Problem: Two uploads of the same key from repos with different chunk sizes could lead to data loss. For example, suppose A is 10 mb chunksize, and B is 20 mb, and the upload speed is the same. If B starts first, when A will overwrite the file it is uploading for the 1st chunk. Then A uploads the second chunk, and once A is done, B finishes the 1st chunk and uploads its second. We now have [chunk 1(from A), chunk 2(from B)].

# new requirements

Every special remote should support chunking. (It does not make sense to support it for git remotes, but gcrypt remotes should support it.)

S3 remotes should chunk by default, because the current S3 backend fails for files past a certian size. See Truncated file transferred via S3.

The size of chunks, as well as whether any chunking is done, should be configurable on the fly without invalidating data already stored in the remote. This seems important for usability (eg, so users can turn chunking on in the webapp when configuring an existing remote).

Two concurrent uploaders of the same object to a remote should be safe, even if they're using different chunk sizes.

The legacy chunk method needs to be supported for back-compat, so keep the chunksize= setting to enable that mode, and add a new chunk= setting for the new mode.

# obscuring file sizes

To hide from a remote any information about the sizes of files could be another goal of chunking. At least two things are needed for this:

1. The filenames used on the remote don't indicate which chunks belong together.

2. The final short chunk needs to be padded with random data, so that a remote sees only encrypted files with uniform sizes and cannot make guesses about the kinds of data being stored.

Note that padding cannot completely hide all information from an attacker who is logging puts or gets. An attacker could, for example, look at the times of puts, and guess at when git-annex has moved on to encrypting/decrypting the next object, and so guess at the approximate sizes of objects. (Concurrent uploads/downloads or random delays could be added to prevent these kinds of attacks.)

And, obviously, if someone stores 10 tb of data in a remote, they probably have around 10 tb of files, so it's probably not a collection of recipes..

Given its inneficiencies and lack of fully obscuring file sizes, padding may not be worth adding, but is considered in the designs below.

# design 1

Add an optional chunk field to Key. It is only present for chunk 2 and above. Ie, SHA256-s12345--xxxxxxx is the first chunk (or whole object), while SHA256-s12345-c2--xxxxxxx is the second chunk.

On an encrypted remote, Keys are generated with the chunk field, and then HMAC enrypted.

Note that only using it for chunks 2+ means that git-annex can start by requesting the regular key, so an observer sees the same request whether chunked or not, and does not see eg, a pattern of failed requests for a non-chunked key, followed by successful requests for the chunked keys. (Both more efficient and perhaps more secure.)

Problem: This makes putting chunks easy. But there is a problem when getting an object that has been chunked. If the key size is not known, we cannot tell when we've gotten the last chunk. (Also, we cannot strip padding.) Note that `addurl` sometimes generates keys w/o size info (particularly, it does so by design when using quvi).

Problem: Also, this makes `checkPresent` hard to implement: How can it know if all the chunks are present, if the key size is not known?

Problem: Also, this makes it difficult to download encrypted keys, because we only know the decrypted size, not the encrypted size, so we can't be sure how many chunks to get, and all chunks need to be downloaded before we can decrypt any of them. (Assuming we encrypt first; chunking first avoids this problem.)

Problem: Does not solve concurrent uploads with different chunk sizes.

# design 2

When chunking is enabled, always put a chunk number in the Key, along with the chunk size. So, SHA256-1048576-c1--xxxxxxx for the first chunk of 1 megabyte.

Before any chunks are stored, write a chunkcount file, eg SHA256-s12345-c0--xxxxxxx. Note that this key is the same as the original object's key, except with chunk number set to 0. This file contains both the number of chunks, and also the chunk size used. `checkPresent` downloads this file, and then verifies that each chunk is present, looking for keys with the expected chunk numbers and chunk size.

This avoids problems with multiple writers using different chunk sizes, since they will be uploading to different files.

Problem: In such a situation, some duplicate data might be stored, not referenced by the last chunkcount file to be written. It would not be dropped when the key was removed from the remote.

Note: This design lets an attacker with logs tell the (appoximate) size of objects, by finding the small files that contain a chunk count, and correlating when that is written/read and when other files are written/read. That could be solved by padding the chunkcount key up to the size of the rest of the keys, but that's very innefficient; `checkPresent` is not designed to need to download large files.

# design 3

Like design 1, but add an encrypted chunk count prefix to the first object. This needs to be done in a way that does not let an attacker tell if the object has an encrypted chunk count prefix or not.

This seems difficult; attacker could probably tell where the first encrypted part stops and the next encrypted part starts by looking for gpg headers, and so tell which files are the first chunks.

Also, `checkPresent` would need to download some or all of the first file. If all, that's a lot more expensive. If only some is downloaded, an attacker can guess that the file that was partially downloaded is the first chunk in a series, and wait for a time when it's fully downloaded to determine which are the other chunks.

Problem: Two uploads of the same key from repos with different chunk sizes could lead to data loss. (Same as in design 2.)

# design 4

Use key SHA256-s12345-S1048576-C1--xxxxxxx for the first chunk of 1 megabyte.

Note that keeping the 's'ize field unchanged is necessary because it disambiguates eg, WORM keys. So a 'S'ize field is used to hold the chunk size.

Instead of storing the chunk count in the special remote, store it in the git-annex branch.

The location log does not record locations of individual chunk keys (too space-inneficient). Instead, look at a chunk log in the git-annex branch to get the chunk count and size for a key.

`checkPresent` would check if any of the logged sets of chunks is present on the remote. It would also check if the non-chunked key is present, as a fallback.

When dropping a key from the remote, drop all logged chunk sizes. (Also drop any non-chunked key.)

As long as the location log and the chunk log are committed atomically, this guarantees that no orphaned chunks end up on a remote (except any that might be left by interrupted uploads).

This has the best security of the designs so far, because the special remote doesn't know anything about chunk sizes. It uses a little more data in the git-annex branch, although with care (using the same timestamp as the location log), it can compress pretty well.

## chunk log

Stored in the git-annex branch, this provides a mapping `Key -> <span class="createlink">Key</span>`.

Note that a given remote uuid might have multiple sets of chunks (with different sizes) logged, if a key was stored on it twice using different chunk sizes. Also note that even when the log indicates a key is chunked, the object may be stored non-chunked on the remote too.

For fixed size chunks, there's no need to store the list of chunk keys, instead the log only records the number of chunks (needed because the size of the parent Key may not be known), and the chunk size.

Example:

    1287290776.765152s e605dca6-446a-11e0-8b2a-002170d25c55:10240 9

Later, might want to support other kinds of chunks, for example ones made using a rsync-style rolling checksum. It would probably not make sense to store the full [Key] list for such chunks in the log. Instead, it might be stored in a file on the remote.

To support such future developments, when updating the chunk log, git-annex should preserve unparsable values (the part after the colon).

## chunk then encrypt

Rather than encrypting the whole object 1st and then chunking, chunk and then encrypt.

Reasons:

1. If 2 repos are uploading the same key to a remote concurrently, this allows some chunks to come from one and some from another, and be reassembled without problems.

2. Also allows chunks of the same object to be downloaded from different remotes, perhaps concurrently, and again be reassembled without problems.

3. Prevents an attacker from re-assembling the chunked file using details of the gpg output. Which would expose approximate file size even if padding is being used to obscure it.

Note that this means that the chunks won't exactly match the configured chunk size. gpg does compression, which might make them a lot smaller. Or gpg overhead could make them slightly larger. So `checkPresent` cannot check exact file sizes.

If padding is enabled, gpg compression should be disabled, to not leak clues about how well the files compress and so what kind of file it is.

## chunk key hashing

A chunk key should hash into the same directory structure as its parent key. This will avoid lots of extra hash directories when using chunking with non-encrypted keys.

Won't happen when the key is encrypted, but that is good; hashing to the same bucket then would allow statistical correlation.

## resuming interupted transfers

Resuming interrupted downloads, and uploads are both possible.

Downloads: If the tmp file for a key exists, round it to the chunk size, and skip forward to the next needed chunk. Easy.

Uploads: Check if the 1st chunk is present. If so, check the second chunk, etc. Once the first missing chunk is found, start uploading from there.

That adds one extra checkPresent call per upload. Probably a win in most cases. Can be improved by making special remotes open a persistent connection that is used for transferring all chunks, as well as for checking checkPresent.

Note that this is safe to do only as long as the Key being transferred cannot possibly have 2 different contents in different repos. Notably not necessarily the case for the URL keys generated for quvi.

Both done.

## parallel

If 2 remotes both support chunking, uploading could upload different chunks to them in parallel. However, the chunk log does not currently allow representing the state where some chunks are on one remote and others on another remote.

Parallel downloading of chunks from different remotes is a bit more doable.

comment 2

git-annex does not pad chunks, although the data it stores is designed to allow adding padding later if there's a good reason to.

Comment by joey — Wed Oct 29 14:05:27 2025
comment 1
I am unclear about something: does git-annex in fact pad chunks? And does that mean if I annex a 500 KB file, it will actually use 1MB of space (or whatever the chunk size is?)
Comment by pat — Wed Oct 29 14:05:27 2025
Comments on this page are closed.