git-annex/ git-annex-addurl

# NAME

git-annex addurl - add urls to annex

# SYNOPSIS

git annex addurl `[url ...]`

# DESCRIPTION

Downloads each url to its own file, which is added to the annex.

When `yt-dlp` is installed, it can be used to check for a video embedded in a web page at the url, and that is added to the annex instead. (However, this is disabled by default as it can be a security risk. See the documentation of annex.security.allowed-ip-addresses in git-annex(1) for details.)

Special remotes can add other special handling of particular urls. For example, the bittorrent special remotes makes urls to torrent files (including magnet links) download the content of the torrent, using `aria2c`.

Normally the filename is based on the full url, so will look like "www.example.com_dir_subdir_bigfile". In some cases, addurl is able to come up with a better filename based on other information. Options can also be used to get better filenames.

# OPTIONS

- `--fast`

  Avoid immediately downloading the url. The url is still checked (via HEAD) to verify that it exists, and to get its size if possible.

- `--relaxed`

  Don't immediately download the url, and avoid storing the size of the url's content. This makes git-annex accept whatever content is there at a future point.

  This is the fastest option, but it still has to access the network to check if the url contains embedded media. When adding large numbers of urls, using `--relaxed --raw` is much faster.

- `--verifiable` `-V`

  This can be used with the `--fast` or `--relaxed` option. It improves the safety of the resulting annexed file, by letting its content be verified with a checksum when it is transferred between git-annex repositories, as well as by things like `git-annex fsck`.

  When used with --relaxed, content from the web special remote will always be accepted, even if it has changed, and the checksum recorded for later verification.

  When used with --fast, the checksum is recorded the first time the content is downloaded from the web special remote. Once a checksum has been recorded, subsequent downloads from the web special remote must have the same checksum.

  When addurl was used without this option before, the file it added can be converted to be verifiable by migrating it to the VURL backend. For example: `git-annex migrate foo --backend=VURL`

- `--raw`

  Prevent special handling of urls by yt-dlp, and by bittorrent and other special remotes. This will for example, make addurl download the .torrent file and not the contents it points to.

- `--no-raw`

  Require content pointed to by the url to be downloaded using yt-dlp or a special remote, rather than the raw content of the url. if that cannot be done, the add will fail.

- `--raw-except=remote`

  Prevent special handling of urls by all special remotes except for the specified one. To allow special handling only by yt-dlp, use `--raw-except=web`.

- `--file=name`

  Use with a filename that does not yet exist to add a new file with the specified name and the content downloaded from the url.

  If the file already exists, addurl will record that it can be downloaded from the specified url(s).

- `--preserve-filename`

  When the web server (or torrent, etc) provides a filename, use it as-is, avoiding sanitizing unusual characters, or truncating it to length, or any other modifications.

  git-annex will still check the filename for safety, and if the filename has a security problem such as path traversal or a control character, it will refuse to add it.

- `--pathdepth=N`

  Rather than basing the filename on the whole url, this causes a path to be constructed, starting at the specified depth within the path of the url.

  For example, adding the url http://www.example.com/dir/subdir/bigfile with `--pathdepth=1` will use "dir/subdir/bigfile", while `--pathdepth=3` will use "bigfile".

  It can also be negative; `--pathdepth=-2` will use the last two parts of the url.

- `--prefix=foo` `--suffix=bar`

  Use to adjust the filenames that are created by addurl. For example, `--suffix=.mp3` can be used to add an extension to the file.

- `--no-check-gitignore`

  By default, gitignores are honored and it will refuse to download an url to a file that would be ignored. This makes such files be added despite any ignores.

- `--jobs=N` `-JN`

  Enables parallel downloads when multiple urls are being added. For example: `-J4`

  Setting this to "cpus" will run one job per CPU core.

- `--batch`

  Enables batch mode, in which lines containing urls to add are read from stdin.

- `-z`

  Makes the `--batch` input be delimited by nulls instead of the usual newlines.

- `--with-files`

  When batch mode is enabled, makes it parse lines of the form: "$url $file"

  That adds the specified url to the specified file, downloading its content if the file does not yet exist; the same as `git annex addurl $url --file $file`

- `--json`

  Enable JSON output. This is intended to be parsed by programs that use git-annex. Each line of output is a JSON object.

- `--json-progress`

  Include progress objects in JSON output.

- `--json-error-messages`

  Messages that would normally be output to standard error are included in the JSON instead.

- `--backend`

  Specifies which key-value backend to use.

- Also the git-annex-common-options(1) can be used.

# CAVEATS

If annex.largefiles is configured, and does not match a file, `git annex addurl` will add the non-large file directly to the git repository, instead of to the annex. However, this is not done when --fast or --relaxed is used.

# SEE ALSO

git-annex(1)

git-annex-rmurl(1)

git-annex-registerurl(1)

git-annex-importfeed(1)

# AUTHOR

Joey Hess id@joeyh.name

Warning: Automatically converted into a man page by mdwn2man. Edit with care.

bug report

    ENV:
    macOS 10.14.6, installed by 'brew install git-annex'
    git annex version
    git-annex version: 8.20201129
    build flags: Assistant Webapp Pairing FsEvents TorrentParser MagicMime Feeds Testsuite S3 WebDAV
    dependency versions: aws-0.22 bloomfilter-2.0.1.0 cryptonite-0.27 DAV-1.3.4 feed-1.3.0.1 ghc-8.10.3 http-client-0.7.3 persistent-sqlite-2.11.0.0 torrent-10000.1.1 uuid-1.3.13 yesod-1.6.1.0
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 BLAKE2B256E BLAKE2B256 BLAKE2B512E BLAKE2B512 BLAKE2B160E BLAKE2B160 BLAKE2B224E BLAKE2B224 BLAKE2B384E BLAKE2B384 BLAKE2BP512E BLAKE2BP512 BLAKE2S256E BLAKE2S256 BLAKE2S160E BLAKE2S160 BLAKE2S224E BLAKE2S224 BLAKE2SP256E BLAKE2SP256 BLAKE2SP224E BLAKE2SP224 SHA1E SHA1 MD5E MD5 WORM URL X*
    remote types: git gcrypt p2p S3 bup directory rsync web bittorrent webdav adb tahoe glacier ddar git-lfs httpalso borg hook external
    operating system: darwin x86_64
    supported repository versions: 8
    upgrade supported from repository versions: 0 1 2 3 4 5 6 7

    STEPS:
    File copied to git (git-annex) repo's dir  (did not 'git add' 'git annex add')
    name: 'f.mp4'
    Now run 'git annex addurl' (via Python, see below)

    RESULT: (same if run in bash)
    File "/opt/anaconda3/lib/python3.7/subprocess.py", line 512, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['git', 'annex', 'addurl', '--file=f.mp4', '--raw', '--relaxed', 'https://www.youtube.com/watch?v=U33dsEcKgeQ']' returned non-zero exit status 1.

    The command works after doing 'git annex add f.mp4' first
      but it results in a backend not 'URL backend for youtube'
      I'd like to use 'URL backend for youtube' cause I worry about youtube-video binay-change, in which case all future download will fail backend verification.

    NOTE:
    command line taken from https://git-annex.branchable.com/tips/using_the_web_as_a_special_remote/

Comment by m15 — Tue Jul 22 18:11:47 2025
Re: provide flags to youtube-dl

@gan, there's not much point in providing flags that are only used in the initial download; the main point in adding the url to git-annex is so you can download the same content from it again later.

Comment by joey — Tue Jul 22 18:11:47 2025
Re: Inconsistent idiom

Hi @joey,

Thanks for your continued work on git-annex and for responding to my last comment. I agree that consistency is not everything, but I do think that it's also important to balance functionality against the amount of cognitive load that an interface places on an end-user. My perspective is no doubt influenced by my cognitive science background, but whenever I find an interface where it's easy to confuse two similar operations that use different syntaxes, I'm reminded of Don Norman's rant about the early Unix UI. I'm also reminded of common phenomena described in memory research such as retroactive interference, wherein a more recently learned memory interferes with something that was learned previously. In this case, if I were to learn the syntax for addurl first, and then learned the syntax for rmurl much later, my internal representation of rmurl would to some degree "overwrite" my previous knowledge of addurl and compete with it. Making the two syntaxes consistent with each other in this case would eliminate any competition between internal mental representations of how the two commands are structured.

I'm also not entirely sure why a positional argument can't be optional. If there's a good reason for this not to be so then I won't argue my point anymore, but something like the following syntax would make the most sense from my view:

`git annex rmurl [url] [file]`

`git annex addurl [url] [file; optional positional argument]`

Comment by pellman.john — Tue Jul 22 18:11:47 2025
Clarification
So, to clarify - I read your first answer. But if this coulud be done during get perhaps then it's OK because it is an explicit request for the potentially unsafe operation?
Comment by gan — Tue Jul 22 18:11:47 2025
Provide flags to youtube-dl?
Is there already a way to specify flags to youtube-dl on a per-file basis. I think it would be OK to do it during either during addurl (modifying the resulting reference that is stored in the annex somehow), or during git-annex get. This is so that the preferred format can be specified. Primarily this would enable to download audio-only formats for some files. ) Apologies if I missed some documentation on how to achieve this)
Comment by gan — Tue Jul 22 18:11:47 2025
comment 4

@john, the difference is that while addurl can make up a filename to use if you do not provide one, rmurl needs you to specifiy a filename.

So, yes, "git annex rmurl --file=whatever url" would be more consistent, but it requires typing more my making something that is not actually optional into an option. And "git annex addurl file url" would make the command more consistent with rmurl, but harder to use.

Consistency is not everything.

(Also, the rmurl batch interface would then be less consistent to its command-line interface.)

Comment by joey — Tue Jul 22 18:11:47 2025
Inconsistent idiom

In using git-annex in the past, I've always found it counterintuitive that rmurl uses the following form to remove a URL from a file:

    git annex rmurl [file] [url]

While, in contrast, addurl uses a flag to designate the file that a URL should be added to the list of URLs a file points to:

    git annex addurl [url] --file=[file]

It would make sense (at least to me) to make the syntax for these more congruous so that both commands use either two positional arguments or one positional argument and one keyword argument / flag.

Comment by pellman.john — Tue Jul 22 18:11:47 2025
comment 2

There's not currently a way to do per-file youtube-dl options. The difficulty is that we don't know what youtube-dl options might be unsafe, and which such a feature could make eg `git annex get` use when run by a different user.

I feel that this needs some support in youtube-dl to avoid git-annex needing to know about all its safe options. Especially since which options are available, or safe, could vary between versions of youtube-dl.

Comment by joey — Tue Jul 22 18:11:47 2025
Re: securehashesonly conflicts with addurl
Opened a bug report: securehashesonly conflicts with addurl
Comment by joey — Tue Jul 22 18:11:47 2025
securehashesonly conflicts with addurl

Turning on `securehashesonly` seems to disable the `addurl` command:

    % git config --get annex.securehashesonly
    true
    % git annex addurl https://www.gutenberg.org/cache/epub/2591/pg2591-images.html
    addurl https://www.gutenberg.org/cache/epub/2591/pg2591-images.html
      annex.securehashesonly blocked transfer of URL key
    failed
    addurl: 1 failed
    % git annex addurl https://www.gutenberg.org/cache/epub/2591/pg2591-images.html --relaxed
    addurl https://www.gutenberg.org/cache/epub/2591/pg2591-images.html (to www.gutenberg.org_cache_epub_2591_pg2591-images.html) ok
    (recording state in git...)
    % ls -l www.gutenberg.org_cache_epub_2591_pg2591-images.html
    www.gutenberg.org_cache_epub_2591_pg2591-images.html -> .git/annex/objects/gg/kG/URL--https&c%%www.gutenberg.org%cache%epub%2591%pg2591-images.html/URL--https&c%%www.gutenberg.org%cache%epub%2591%pg2591-images.html

Does this have something to do with the URL prefix that the annex object has?

Comment by jt — Tue Jul 22 18:11:47 2025
comment 14

@matthias.risze length is not an issue. You should avoid characters that are not usually in urls, particularly whitespace and newline.

It seems to me though that your special remote would perhaps be better served by using the SETSTATE and GETSTATE commands (see external special remote protocol)

Comment by joey — Tue Jul 22 18:11:47 2025
Limitations on allowed characters in and length of URLs
Are there any limitations on which characters are allowed in an URL, and on how long an URL can be? I am working on a special remote which kind of abuses this feature by saving information about how to get the file in the URL with a specific scheme. For now it seems like I can use URLs of the form `scheme:<arbitrary json>`, but I am not sure if this might become an issue later. I could also encode the data with base64 or something similar, in which case size limitations would still be relevant; if there are any. Although, the json variant has the added benefit of being much more easily readable in whereis output.
Comment by matthias.risze — Tue Jul 22 18:11:47 2025
checksums and addurl --fast

> Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?

Hash is not recorded, but file size is. You can disable the size check with `--relaxed`. See using the web as a special remote. After `git-annex-get`ting the file, you can use `git-annex-migrate` to record it under a new checksum-based hash, then use `git-annex-unused` to find and remove the old key.

Sometimes you can get the hash without downloading the file, e.g. if the hash is stored next to the file at `http://my/file.md5`, or if the file is stored in the Google Cloud. Then you can use the plumbing commands `git-annex-registerurl` to associate the checksum-based key with the URL, and `git-annex-setpresentkey` to record the key's presence in the (web) remote.

Related discussion: alternate keys for same content

Comment by Ilya_Shlyakhter — Tue Jul 22 18:11:47 2025
Hashes for files added via addurl

If you add a file to your repo first via `addurl --fast`, it writes the filename as a symlink to a file that incorporates the URL, rather than the file hash. This is expected, since git-annex can't know the file hash until it's actually downloaded the file.

If you then `git annex get` that file, it downloads the file to the path that uses the URL. Is the hash ever recorded for these files? If you were to drop and re-download the file, would git-annex accept a different file?

Comment by tomdhunt — Tue Jul 22 18:11:47 2025
comment 10

@m15 this page is not a bug tracking system. File bug reports over at bugs.

Comment by joey — Tue Jul 22 18:11:47 2025
Using youtube-dl commandline options with git-annex-addurl

I have been trying to figure out how to use addurl to get this video. I have this in my mscourtstuff annex as a large binary, but I would really like to use the web as a remote for this.

Hughes v Hosemann 2010-CA-01949-SCT-43112001.mp4 youtube-dl --referer 'http://judicial.mc.edu/case.php?id=24206' http://player.vimeo.com/video/43112001

Comment by joseph.rawson.works — Tue Jul 22 18:11:47 2025
Comments on this page are closed.