git-annex/ special remotes/ web

git-annex can use the web as a special remote, associating an url with an annexed file, and downloading the file content from the web. See using the web as a special remote for usage examples.

The web special remote is always enabled, without any manual setup being needed. Its name is "web".

This special remote can only be used for downloading content, not uploading content, or removing content from the web.

This special remote uses urls on the web as the source for content. There are several other ways http can be used to download annexed objects, including a git remote accessible by http, S3 with a `publicurl` configured, and the httpalso special remote.

## configuration

These parameters can be passed to `git annex initremote` or `git-annex enableremote` to configure a web remote:

- `urlinclude` - Only use urls that match the specified glob.
  For example, `urlinclude="https://s3.amazonaws.com/*"`
- `urlexclude` - Don't use urls that match the specified glob.
  For example, to prohibit http urls, but allow https, use `urlexclude="http:*"`

Globs are matched case-insensitively.

When there are multiple special remotes of type web, and some are not configured with `urlinclude` and/or `urlexclude`, those will avoid using urls that are matched by the configuration of other web remotes.

For example, this creates a second web special remote named "slowweb" that is only used for urls on one host, and that has a higher cost than the "web" special remote. With this configuration, `git-annex get` will first try to get the file from the "web" special remote, which will avoid using any urls that match slowweb's urlinclude. Only if the content can't be downloaded from "web" (or some other remote) will it fall back to downloading from slowweb.

    git-annex initremote --sameas=web slowweb type=web urlinclude='*//slowhost.com/*'
    git config remote.slowweb.annex-cost 300

comment 5

@spiderbit, to support recursively adding content, it would need to parse html, and that's simply too complex and too unlikely to be useful to many people.

Comment by joey — Tue Jul 22 18:11:47 2025
update

Sorry, should have read the man page.

of course I have to use %url and %file

So it works with "rsync %url %file" but it seems to not work recursive also it renames the files instead of adding them with their normal name. So not useful for what I want to do.

I want to access a normal unmodified directory on my server and add the files to my local directory. That would be a minimal setup, everything else means just extremly big setups with assistant runnig and a cronjob to delete unused files, and lots of cpu load for indexing this files on the server.

I think such minimal setup would be great to get startet without having very complex setups, you dont want to commit to such a tool and hours of setup to get somethnig useful, just to look if its useful for you.

there are 2 aproaches either have a normal repos on the server, with again cronjob and flat setup. which is quite a setup to get.

or use only a real repos on the client, both have big disadvantages normal repos complex setup, to complex to just test it. web links seem to be simple enough but is not recursive therefor only good for youtube links or stuff like that.

Is there really no simple solution for what I want to do?

Comment by spiderbit — Tue Jul 22 18:11:47 2025
comment 3 e6ce9bb92c973350852c9498b7ffb50f
[[!comment Error: unsupported page format sh]]
— Tue Jul 22 18:11:47 2025
comment 2

When it says "arbitrary urls", it means it. The only requirement is that the url be well formed and that wget or whatever command you have it configured to use via annex.web-download-command knows how to download it.

Update 2018: That used to be the case, but it's now limited by default to http and https urls.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Which URL prefix are supported ?
It is not clear whether only http:// URLs are supported. Can you list others ?
Comment by http://olivier.berger.myopenid.com/ — Tue Jul 22 18:11:47 2025
Comments on this page are closed.