git-annex/ special remotes/ httpalso

This special remote allows downloading annexed objects from other special remotes that also publish their content by http. Not to be confused with the web special remote, or with accessing git-annex repositories over http, this is only useful in combination with some other special remote.

Suppose, for example, that you have a directory special remote. And the directory happens to be published by a web server. (Or it could be a rsync special remote, or many other kinds.) To let git-annex know that the content of this special remote can also be accessed over http, set up a httpalso special remote.

    git annex initremote --sameas=foo foo-http type=httpalso url=http://example.com/foo

The --sameas parameter tells git-annex what other special remote this httpalso remote is accessing. (See multiple remotes accessing the same data store.) Since the httpalso remote is read-only, it can only be used to download content that is stored in that other remote.

This special remote is compatible with many, but not all, other special remotes. If the special remote does something unusual with the name a file is stored under, or with how the data is stored, it might not work. See multiple remotes accessing the same data store for a list of known working combinations.

## configuration

- `url` - The http or https url to where the content is stored by the other special remote.
- `exporttree` - If the other special remote has `exporttree=yes` set, it also needs to be set when initializing the httpalso remote. Setting this does not allow trees to be exported to the httpalso remote, because it's read-only. But it does let exported files be downloaded from it.
- `annexobjects` - If the other special remote has `annexobjects=yes` set (along with `exporttree=yes`), it also needs to be set when initializing the httpalso remote.

Configuration of encryption and chunking is inherited from the other special remote, and does not need to be specified when initializing the httpalso remote.

Setup dirhash parameter for httpalso to avoid hammering server

Hi @Joey, I was wondering if it would be possible to setup the dirhash format for the special remote.

We push data to a datalad ria-store through ssh that gets RO accessed through a httpalso remote. However, I observe that get operations try different directory hashing schema before hitting the right one that datalad ORA remote uses. This is also the case when doing repo-wide `fsck` in recurring tests of the dataset. All of that is doubling the number of http request, and could in some setups trigger fail2ban-type mechanisms at network or server levels. If there was a parameter that could be set on the httpalso to directly use the right URLs, that would be great.

Thanks!

Comment by Basile.Pinsard — Tue Jul 22 18:11:47 2025
comment 8

@kdm some of the hourly autobuilds are updated with it already, eg the x86-64 linux one. Others should update today. I do hope to make a release soon.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 7
Phenomenal, many many thanks Joey. Any chance of a new release/build that includes this change in the coming couple of weeks? And apologies for the misuse of comments -- I'll use todo in future.
Comment by kdm9 — Tue Jul 22 18:11:47 2025
comment 6

I've made it support chunking.

(Please in the future post to todo rather than making a comment when you have a new feature to suggest.)

Comment by joey — Tue Jul 22 18:11:47 2025
chunking

Hi Joey,

Another question: could the httpalso special remote be made to support mirroring remotes that enable chunking (with chunk=xxMiB)? This would in general allow anonymous access to e.g. webdav shares also published over https.

Best, KDM

Comment by kdm9 — Tue Jul 22 18:11:47 2025
comment 4

@tomdhunt I think it could probably be used with a git remote that exported its .git files over http, but since git-annex supports using such a remote as a normmal git remote, there does not seem to be any point in using it that way.

http authentication is not currently supported, I suppose it could be added if useful, file a todo if you have a use case.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 3
Actually, no need to use httpalso at all. According to this, it will just work.
Comment by Lukey — Tue Jul 22 18:11:47 2025
comment 2
I guess it could work by pointing the httpalso remote directly at the git-annex object store at `.git/annex/objects`.
Comment by Lukey — Tue Jul 22 18:11:47 2025
Re: Setup dirhash parameter for httpalso to avoid hammering server

httpalso learns the layout of the remote, so one a process has found one file it will use that same layout going forward. So you should see only a few 404's per process.

Please open a todo with details about the problem if that's not the case, or if that's somehow insufficient.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 1

Is it possible to use httpalso with a standard git remote that also runs git-annex?

Also, does it support HTTP authentication?

Comment by tomdhunt — Tue Jul 22 18:11:47 2025
Comments on this page are closed.