git-annex/ special remotes

git-annex can transfer data to and from configured git remotes. Normally those remotes are normal git repositories (bare and non-bare; local and remote), that store the file contents in their own git-annex directory.

But, git-annex also extends git's concept of remotes, with these special types of remotes. These can be used by git-annex to store and retrieve the content of files.

- adb (for Android devices)
- Amazon Glacier
- bittorrent
- borg
- bup
- compute
- ddar
- directory
- gcrypt (encrypted git repositories!)
- git-lfs
- git
- hook
- httpalso
- rclone
- rsync
- S3 (Amazon S3, and other compatible services)
- tahoe
- tor
- web
- webdav

The above special remotes are built into git-annex, and can be used to tie git-annex into many cloud services.

Here are specific instructions for using git-annex with various services:

- Amazon Glacier
- Amazon S3
- Backblaze B2
- Box.com
- Ceph
- chef-vault
- Dropbox
- FTP
- Flickr
- Freenet and Siacoin Skynet
- Google Cloud Storage
- Google Drive
- hubiC
- IMAP
- Internet Archive via S3
- ipfs
- Jottacloud
- Mega
- Microsoft Azure Blob Storage
- Microsoft OneDrive
- NNCP
- OpenDrive
- Openstack Swift / Rackspace cloud files / Memset Memstore
- OwnCloud
- pCloud
- QingStor
- SFTP
- SkyDrive
- smb / sftp
- Usenet
- Yandex Disk

If a service is not mentioned above, it's worth checking if rclone supports it, then you can use the rclone special remote.

Want to add support for something else? Write your own!

## Ways to use special remotes

There are many use cases for a special remote. You could use it as a backup. You could use it to archive files offline in a drive with encryption enabled so if the drive is stolen your data is not. You could git annex move --to specialremote large files when your local drive is getting full, and then git annex move the files back when free space is again available. You could have one repository copy files to a special remote, and then git annex get them on another repository, to transfer the files between computers that do not communicate directly.

None of these use cases are particular to particular special remote types. Most special remotes can all be used in these and other ways. It largely doesn't matter for your use what underlying transport the special remote uses.

## Setting up a special remote

To initialize a new special remote, use git-annex initremote. See the documentation for the special remote you want to use for details about configuration and examples of how to initremote it.

Once a special remote has been initialized, other clones of the repository can also enable it, by using git-annex enableremote with the same name that was used to initialize it. (Run the command without any name to get a list of available special remotes.)

Initializing or enabling a special remote adds it as a remote of your git repository.

## Storing a git repository in a special remote

Most special remotes do not include a clone of the git repository by default, so you can't use commands like `git push` and `git pull` with them. (There are some exceptions like git-lfs.)

But it is possible to store a git repository in many special remotes, using the git-remote-annex command. This involves configuring the remote with an "annex::" url. It's even possible to `git clone` from a special remote using such an url. See the documentation of git-remote-annex for details.

## Unused content on special remotes

Over time, special remotes can accumulate file content that is no longer referred to by files in git. Normally, unused content in the current repository is found by running `git annex unused`. To detect unused content on special remotes, instead use `git annex unused --from`. Example:

    $ git annex unused --from mys3
    unused mys3 (checking for unused data...)
      Some annexed data on mys3 is not used by any files in this repository.
        NUMBER  KEY
        1       WORM-s3-m1301674316--foo
      (To see where data was previously used, try: git log --stat -S'KEY')
      (To remove unwanted data: git-annex dropunused --from mys3 NUMBER)
    $ git annex dropunused --from mys3 1
    dropunused 12948 (from mys3...) ok

## Removing special remotes

Like git remotes, a special remote can be removed from your repository by using `git remote remove`. Note that does not delete the special remote, or prevent other repositories from enabling or using it.

## Testing special remotes

To make sure that a special remote is working correctly, you can use the git annex testremote command. This expects you to have set up the remote as usual, and it then runs a lot of tests, using random data. It's particularly useful to test new implementations of special remotes.

comment 9
A ridiculously cool possibility would be to allow them to match against URLs and then handle those (youtube-dl for youtube video URLs, for instance), but that would be additional work on your end and isn't really necessary.
Comment by https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY — Tue Jul 22 18:11:47 2025
'webhook' special remote?

Is there any chance a special remote that functions like a hybrid of 'web' and 'hook'? At least in theory, it should be relatively simple, since it would only support 'get' and the only meaningful parameters to pass would be the URL and the output file name.

Maybe make it something like git config annex.myprogram-webhook 'myprogram $ANNEX_URL $ANNEX_FILE', and fetching could work by adding a --handler or --type parameter to addurl.

The use case here is anywhere that simple 'fetch the file over HTTP/FTP/etc' isn't workable - maybe it's on rapidshare and you need to use plowshare to download it; maybe it's a youtube video and you want to use youtube-dl, maybe it's a chapter of a manga and you want to turn it into a CBZ file when you fetch it.

Comment by https://www.google.com/accounts/o8/id?id=AItOawlRThEwuPnr8_bcuuCTQ0rQd3w6AfeMiLY — Tue Jul 22 18:11:47 2025
Rapidshare

Would it be possible to support Rapidshare as a new special remote? They offer unlimited storage for 6-10€ per month. It would be great for larger backups. Their API can be found here: http://images.rapidshare.com/apidoc.txt

Comment by https://www.google.com/accounts/o8/id?id=AItOawn-UoTjMBsVh6q4HNViGwJi-5FNaCVQB7E — Tue Jul 22 18:11:47 2025
Rackspace US/UK
It'd be awesome to be able to use Rackspace as remote storage as an alternative to S3, I would submit a patch, but know 0 Haskell :D
Comment by https://www.google.com/accounts/o8/id?id=AItOawlBia1J9-PoXgZYj2LASf7Bs__IqK3T8qQ — Tue Jul 22 18:11:47 2025
comment 52

@gaknuyardi that is expected, they are hash directories. You can see the same effect in .git/annex/object/ hash directories when there are enough objects.

Comment by joey — Tue Jul 22 18:11:47 2025
Understanding encrypted special remote folder structure

@joey While inspecting my special remote (rsync, encryption=hybrid) I noticed that a couple files ended up in the same folder tree, is that normal or is something wrong? Obfuscated tree output below.

    ├── [   0]  xxx
    │   └── [   0]  yyy
    │       ├── [   0]  GPGHMACSHA512--1234
    │       │   └── [29M]  GPGHMACSHA512--1234
    │       └── [   0]  GPGHMACSHA512--5678
    │           └── [ 2G]  GPGHMACSHA512--5678

Comment by gaknuyardi — Tue Jul 22 18:11:47 2025
comment 50

It would be possible for initremote to allow reusing a name that is used for an old remote that has been marked as dead. The reason that is not allowed, I think, is that there is some footgun potential: If the old remote somehow was lost and then was found, it would either be hard to unmark it as dead, or if that was done, there would be 2 remotes with the same name, which would complicate using enableremote.

So it seems best to rename the old one before creating the new one. That way it's got a unique name still.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 5
The web special remote is recently able to use file:// URL's, so you can just point to files on some arbitrary storage if you want to.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Re: "Any way to delete a remote?"
There is also `git-annex-renameremote` . (And a old post on the topic, but that's before `git-annex-renameremote` was added.)
Comment by Ilya_Shlyakhter — Tue Jul 22 18:11:47 2025
comment 48

I've tried git-annex-dead, it removes the remotes from command outputs but the name is still taken. This isn't great if I want to delete and reconfigure the same remote.

I can always just call it remote2, but that's unaesthetic.

Comment by tomdhunt — Tue Jul 22 18:11:47 2025
comment 47
Try `git-annex-dead` remotename
Comment by Ilya_Shlyakhter — Tue Jul 22 18:11:47 2025
Any way to delete a remote?
I can't find any way to delete a special remote. I created one that was explicitly temporary (to transfer between some other systems), and now I can't get rid of it. I've already run "git remote rm" on it, but it seems git-annex didn't pick this up.
Comment by tomdhunt — Tue Jul 22 18:11:47 2025
comment 45

I mean, I made it do so already, but since this is not a bug tracking system, but an increasingly too long set of disjoint comments, I didn't want to follow up here to say that.

Please, if you have some question about special remotes, post it in the forum. If you have an improvement, post a todo item. If you have a problem, file a bug report. If you're commenting on this page, please think about all the subsequent visitors who will see a huge comment thread and how they will react.

Comment by joey — Tue Jul 22 18:11:47 2025
preventing fetching from special remotes
"set remote.name.skipFetchAll to true" -- is there a reason git-annex itself doesn't do that?
Comment by Ilya_Shlyakhter — Tue Jul 22 18:11:47 2025
Re: Avoid fetching from special remotes

@Dan, set remote.name.skipFetchAll to true. Or make a remote group containing the remotes you do want to fetch from.

Comment by joey — Tue Jul 22 18:11:47 2025
Avoid fetching from special remotes

I have a few special remotes configured in my git annex-ed repo. As far as `git` is concerned, these are just ordinary remotes (they have entries in the `.git/config` file). As a result, when I do something like `git fetch --all` it still tries to fetch from them. This of course fails (since they aren't actually git repositories). It isn't the end of the world, but causes some extra noise in my output (I use `magit` in emacs for most of my `git` workflow, so I have to drill down a bit to see what actually failed and then dismiss it as an expected failure).

It'd be nice if I could just prevent `git` from fetching from these special remotes. Is there a clever way I can set `remote.<name>.fetch` in the local config so as to leave these remotes alone when fetching?

Comment by Dan — Tue Jul 22 18:11:47 2025
comment 41

@giuly.ippoliti this is not the best place to ask.. external special remote protocol is a better place to discuss special remote implementation.

Anyway, git-annex's --quiet option will shut up the INFO. DEBUG is only ever displayed when you use the --debug option. ERROR should only be used if you have a problem that the user is going to care about seeing.

Comment by joey — Tue Jul 22 18:11:47 2025
Disable git annex logs

Hi,

So while writing the globus special remote (git-annex-remote-globus) I often redirect my logs to annex. Nevertheless these logs are always logged out in the console, them being INFO, ERROR, DEBUG and I would like to control that. Is there a way to disable console logging of logs sent back to git annex?

Something like ANNEX_LOG_LEVEL=self.annex.ERROR

Thanks !!

Regards Giulia

Comment by giuly.ippoliti — Tue Jul 22 18:11:47 2025
JABOF special remote

Similar to a JABOD, this would be Just A Bunch Of Files. I already have a NAS with a file structure conducive to serving media to my TV. However, it's not capable (currently) of running git-annex locally. It would be great to be able to tell annex the path to a file there as a remote much like a web remote from "git annex addurl". That way I can safely drop all the files I took with me on my trip, while annex still verifies and counts the file on the NAS as a location.

There are some interesting things to figure out for this to be efficient. For example, SHAs of the files. Maybe store that in a metadata file in the directory of the files? Or perhaps use the WORM backend by default?

Comment by https://www.google.com/accounts/o8/id?id=AItOawmRFKwny4rArBaz-36xTcsJYqKIgdDaw5Q — Tue Jul 22 18:11:47 2025
Re: Not connect to remote as part of initremote?

@andrew while it might be possible in theory to set up a rsync special remote that's usable without connecting to the rsync server yet, there are other types of special remotes that do need to connect to the server. S3 comes to mind; it needs to either create a bucket or check if an existing bucket is already being used as a git-annex remote.

So, I don't think this can be supported generally across all special remote implementations. We could try to support it for specific ones like rsync. I don't actually see anywhere in the code for the rsync special remote where initremote will connect to the remote server though.

Comment by joey — Tue Jul 22 18:11:47 2025
Not connect to remote as part of initremote?

Hey,

I'm setting up some git repos as part of a FAI [0] build, and part of this process is running "git annex initremote" to add an rsync remote.

git annex is trying to connect to the remote, but as this an automated build there are no ssh keys available, and warnings etc are spat out. Is it possible to instruct initremote to not connect to the remote as part of this process?

Cheers, Andrew

[0] http://fai-project.org

Comment by andrew — Tue Jul 22 18:11:47 2025
comment 37

@bec.watson, better to open a bug report for this kind of problem.

Seems that the AWS library that git-annex is using does not use V4 authorization yet. Work is in progress: https://github.com/aristidb/aws/pull/199

As to the endpoint hostname, that must be a special case for China, so I've made git-annex aware of that special case.

Comment by joey — Tue Jul 22 18:11:47 2025
Special remote's annex cost

Looks like it's not possible to set the annex cost of the "web" and "bittorrent" special remotes annex cost.

This doesn't seem to work:

[remote "web"] annex-cost = 500

Perhaps, because these remotes don't actually exist.

Setting annex cost for a webdav remote works but is incremented by 50 for some reason.

Comment by Dilyin — Tue Jul 22 18:11:47 2025
comment 35

@grawity yes, enableremote is the way to change configuration of an existing special remote.

The special remote names are a bit funky; to keep the user from needing to enter in a long uuid when enabling a particular special remote, a name has to be recorded for the remote, and that becomes shared across clones of that repository, in a way that the names of git remotes are not normally. (Normally, my "origin" might be your "upstream" etc.)

While it could ignore dead remotes when initializing a new remote with an existing name, then if the old remote got brought back from the dead, there would be a naming conflict. So, I think it's best to not go down that path, to avoid the undead horrors lurking there. ;-)

Comment by joey — Tue Jul 22 18:11:47 2025
comment 34

Hmm, I just found that `annex enableremote` accepts parameters to be modified; that should work for now.

Though I still wonder about situations where one wants to add a new empty remote with new UUID, but reuse the old name...

Comment by grawity — Tue Jul 22 18:11:47 2025
comment 33

I'm trying to write a remote (for smb:// support via GNOME's Gvfs), and I can't seem to find a way to change an existing special remote's parameters?

Even when marked as "dead" (the closest to deleting a remote that I could find), it still blocks subsequent `annex initremote` calls with the same name.

Also, ideally I'd want to reuse the same name and keep the same UUID (e.g. when the backend is moved/renamed). Though of course there are situations where a new UUID would be wanted as well... (I guess that could depend on whether the remote is currently "dead" or not?)

Comment by grawity — Tue Jul 22 18:11:47 2025
comment 32

@openmedi git-annex doesn't currently keep track of how much space it's using on a special remote. It's actually quite a difficult problem to do that in general, since multiple distributed clones of a repository can be uploading to the same special remote at the same time.

If it runs out of space and transfers fail, git-annex will handle the failures semi-gracefully, which is to say nothing will stop it from trying again or trying to send other data, but it will certianly be aware that files are not reaching the special remote.

If a particular storage service has a way to check free space, it would not be hard to make git-annex's special remote implementation check it and avoid trying transfers when it's full.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 31
How does git-annex handle space issues with special remotes? For example my Owncloud instance has 100 GB space. What happens if I run out of space on that remote? Does git-annex handle that gracefully? Do I have to do something? Can I set a sort of "quota"?
Comment by openmedi — Tue Jul 22 18:11:47 2025
anyone saw/worked on backend for watchdox service? (not free one but needed :-/)
subject
Comment by https://me.yahoo.com/a/EbvxpTI_xP9Aod7Mg4cwGhgjrCrdM5s-#7c0f4 — Tue Jul 22 18:11:47 2025
MediaFire REST API
Wouldn't this be enough? http://developers.mediafire.com/index.php/REST_API
Comment by https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8 — Tue Jul 22 18:11:47 2025
git-annex-remote-b2
I wrote an external remote for Backblaze's B2 object storage service: git-annex-remote-b2. I can't be arsed to write a full page on it, but the README covers everything an experienced git-annex user needs to know to use it.
Comment by encryptio — Tue Jul 22 18:11:47 2025
Thanks
Sorry, didn't post a follow up, but git annex enableremote did exactly what I expected. Thanks.
Comment by craig — Tue Jul 22 18:11:47 2025
backblaze support

Backblaze B2 with unlimited storage at $.005/GB/mo seems to be a great option for a special remote. Is it feasible to add support for it? I'd love to contribute financially.

https://www.backblaze.com/b2

Comment by alex — Tue Jul 22 18:11:47 2025
comment 26

@craig, this can be slightly confusing, since `git-annex enableremote` uses the same name that you used when creating the remote in the first place, with `git-annex initremote`... which might be different than the name used for that remote in some repository or other, and from the description shown in `git annex into`.

Since every remote listed by `git annex info` is apparently a regular git repo, not a special remote, with the exception of the glacier one, process of deduction suggests that the "gitannexpics" special remote is the same as the glacier one.

I've made some changes now, so `git annex enableremote` will list the uuid and description, along with the name used by enableremote, and will accept any one of those things to specify which remote to enable.

Comment by joey — Tue Jul 22 18:11:47 2025
Replicating my key for encrypted special remotes

Cool, thanks. I see the gpg key in remote.log in the git-annex branch, so it's saved, which is the thing I care about most. I'm now sure I could recover my data in a DR scenario. However, I seem to be missing something with enableremote and how this is all supposed to work.

My main repo is ~/local/pics and here's the result of git annex info glacier:

    11:45:24 [24623]; git annex info glacier
    remote: glacier
    description: glacier backup [glacier]
    uuid: e34c5f10-2a97-4477-a248-b96e050557dc
    trust: semitrusted
    cost: 1050.0
    type: glacier
    creds: stored locally
    glacier vault: ...snip...
    encryption: encrypted (encryption key stored in git repository)
    chunking: none
    remote annex keys: 27519
    remote annex size: 47.95 gigabytes

I used git annex assistant to create a manual mode remote on my usb key. This created a annex-pics directory on the usb key with a bare repo.

    11:50:50 [24626]; pwd
    /media/craig/KINGSTON/annex-pics
    11:50:51 [24627]; ls
    annex  branches  config  description  HEAD  hooks  info  objects  refs

I then did a git clone from the bare repo into a tmp dir:

    11:52:48 [24670]; git clone /media/craig/KINGSTON/annex-pics annex-pics
    Cloning into 'annex-pics'...
    done.

But when I enable the glacier remote, which I'd have to do in a DR scenario, I get an error:

    11:53:29 [24672]; git annex enableremote glacier
    (merging origin/git-annex into git-annex...)
    (recording state in git...)
    git-annex: Unknown special remote name.
    Known special remotes: gitannexpics

It knows about the remote, but hasn't assigned a name to it:

    11:53:36 [24673]; git annex info glacier
    git-annex: glacier is not a directory or an annexed file or a remote or a uuid
    11:53:49 [24674]; git annex info
    repository mode: indirect
    trusted repositories: 0
    semitrusted repositories: 7
            00000000-0000-0000-0000-000000000001 -- web
            00000000-0000-0000-0000-000000000002 -- bittorrent
            082b6805-3264-4f1b-8e15-a5cd0cef3e7f -- craig@jester:~/local/pics
            71132283-1a3b-47f8-a548-1d5fb2f645d4 -- craig@jester:~/tmp/annex-pics [here]
            a4598ad6-390c-40cf-a71d-edfe7511b20c -- craig@storage:~/local/pics
            d4ee044d-e0e0-4012-b6cd-353da37b9867 -- KINGSTON [origin]
            e34c5f10-2a97-4477-a248-b96e050557dc -- glacier backup
    untrusted repositories: 1
            3796b34c-61af-4e45-a276-07097c1ac6f9 -- craig@desktop:~/local/pics
    transfers in progress: none
    available local disk space: 414.4 gigabytes (+1 megabyte reserved)
    local annex keys: 0
    local annex size: 0 bytes
    annexed files in working tree: 33204
    size of annexed files in working tree: 48.38 gigabytes (+ 148 unknown size)
    bloom filter size: 32 mebibytes (0% full)
    backend usage:
            SHA1: 148
            SHA256E: 33056

Doing a git annex info on the uuid does something, but I'm not clear what it does:

    11:53:56 [24675]; git annex info e34c5f10-2a97-4477-a248-b96e050557dc
    remote annex keys: 27519
    remote annex size: 47.95 gigabytes

An enable remote on the uuid doesn't work either:

    11:57:33 [24678]; git annex enableremote e34c5f10-2a97-4477-a248-b96e050557dc
    git-annex: Unknown special remote name.
    Known special remotes: gitannexpics

I feel like I'm missing a step. What am I missing?

Thanks, Craig

Comment by craig — Tue Jul 22 18:11:47 2025
comment 24

@craig, all of git-annex's information about a special remote is stored in the git-annex branch in git, so any clone of the git repository is sufficient to back that up. You can run `git annex enableremote` in an clone to enable an existing special remote.

The only catch is that, if you have chosen to initremote a special remote using a gpg key, with `keyid=whatever`, you'll of course also need that gpg key to to use it. If you run `git annex info $myremote` it will tell you amoung other things, any gpg keys that are used by that remote.

Comment by joey — Tue Jul 22 18:11:47 2025
Replicating my key for encrypted special remotes

I'm using git annex assistant to auto backup my pictures off-site to glacier. The files in glacier are encrypted. However, if I lose my main machine, I've also lost the encryption key, which makes my off-site backup useless. I figured I could fix this by creating a manual mode remote on a usb drive that I keep on my keychain. I figured this would replicate the encryption key (as I might want to pull down files from the full backup glacier remote), but would not replicate the files themselves, as I have more pictures than space on the usb drive.

However, it seems the that the new remote is configured to only talk to my main machine and not glacier; the encryption key is not in the .git/ directory. How do I ensure that I've got an off-site copy of the glacier encryption key?

Thanks, Craig

Comment by craig — Tue Jul 22 18:11:47 2025
comment 23

@donkeyicydragon one way to accomplish that would be to just tar up `.git` -- excluding `.git/annex/objects` and add that to git-annex like any other file. You could make a git post-commit hook that does that, but that seems overboard.

Or, you could just make a git clone of your repo to a local removable drive, and use that as a local backup.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Enhancing special remotes as backup spaces

Hi Joey,

I am thinking about using google drive as an encrypted backup for my important files. However, I fear that if all my git annex repositories are unrecoverable that the encrypted data on the special remote will not help me much. Assuming I have backed up my gpg key I still get a bunch of decrypted files but the folder structure is lost. Would it be possible to implement something like a safety feature that also uploads an (encrypted) tar of all symlinks (pointing to the respective encrypted files) of the (current/or master-branch) git working tree?

I am almost sure this is already implementable using hooks however I could not find information on which types of hooks are available. I am looking for one that is triggered once after all copy/move operations to a special remote are finished. Can you point me in the right direction?

Marek

Comment by https://www.google.com/accounts/o8/id?id=AItOawmp1ThsNNAbSn46ju-gwFELfStlhl8usJo — Tue Jul 22 18:11:47 2025
Testing a special remote

Is there a unit test or integration test to check for the behavior of a special remote implementation and/or validity?

I don't speak Haskell, so maybe there are some in the source but maybe I wouldn't recognize, so I haven't checked. If there are any tests how should I use it?

Thank you, Bence

Comment by https://www.google.com/accounts/o8/id?id=AItOawnWvnTWY6LrcPB4BzYEBn5mRTpNhg5EtEg — Tue Jul 22 18:11:47 2025
comment 2
Mediafire does not appear to offer any kind of API for its storage.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
compressed storage/transfer -- gzip Content-Type

FWIW -- eh -- unfortunately it seems not that transparent. wget seems to not support decompression at all, curl can do with explicit --compressed, BUT it doesn't distinguish url to a "natively" .gz file and pre-compressed content. And I am not sure if it is possible to anyhow reliably distinguish the two urls. In the case of obtaining pre-compressed file from my sample apache server the only difference in the http response header is that it gets "compound" ETag: compare ETag: "3acb0e-17b38-4dd5343744660" (for directly asking zeros100.gz) vs "3acb0e-17b38-4dd5343744660;4dd5344e1537e" (requesting zeros100) where portion past ";" I guess signals the caching tag for gzipping, but not exactly sure on that since it seems to be not a part of standard. Also for zeros100 I am getting "TCN: choice"... once again not sure if that is any how reliably indicative for my purpose. So I guess there is no good way ATM via Content-Type request.

Comment by http://yarikoptic.myopenid.com/ — Tue Jul 22 18:11:47 2025
comment 18

All special remotes store files compressed when you enable encryption. Not otherwise, though.

As far as the web special remote and pre-compressed files, files are downloaded from the web using `wget` or (of wget is not available) `curl`. So if you can make it work with those commands, it should work.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Re: compression -- storage and transfer

ha -- apparently it is trivial to configure apache to serve pre-compressed files (e.g. see http://stackoverflow.com/questions/75482/how-can-i-pre-compress-files-with-mod-deflate-in-apache-2-x) and they arrive compressed to client with

Content-Encoding: gzip

but unfortunately git-annex doesn't like those (fails to "verify") -- do you think it could be implemented for web "special remotes"? that would be really nice -- then I could store such load on another website, and addurl links to the compressed content

Comment by http://yarikoptic.myopenid.com/ — Tue Jul 22 18:11:47 2025
compression -- storage and transfer

Is there any remote which would not only compress during transfer (I believe rsync does that, right?) but also store objects compressed?

I thought bup would do both -- but it seems that git annex receives data uncompressed from a bup remote, and bup remote requires ssh access.

In my case I want to make publicly available files which are binary blobs which could be compressed very well. It would be a pity if I waste storage on my end and also incur significant traffic, which could be avoided if data load was transferred compressed. May be HTTP compression (http://en.wikipedia.org/wiki/HTTP_compression) could somehow be used efficiently for this purpose (not sure if load then originally could already reside in a compressed form to avoid server time to re-compress it)?

Comment by http://yarikoptic.myopenid.com/ — Tue Jul 22 18:11:47 2025
remotes costs

Thank you -- that is nice!

Could costs be presented in 'whereis' and 'status' commands? e.g. like we know APT repositories priorities from apt-cache policy -- now I do not see them (at least in 4.20130501... updating to sid's 0521 now)

Comment by http://yarikoptic.myopenid.com/ — Tue Jul 22 18:11:47 2025
comment 14
You do not need to use the assistant to configure the costs of remotes. Just set `remote.<name>.annex-cost` to appropriate values. See also the documentation for the `remote.<name>.annex-cost-command` which allows your own code to calculate costs.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Remote costs

This should be implemented with costs

I refer you too: http://git-annex.branchable.com/design/assistant/blog/day_213__costs/

This has been implemented in the assistant, so if you use that, changing priority should be as simple as changing the order of the remotes on the web interface. Whichever remote is highest on the list, is the one your client will fetch from.

Comment by https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8 — Tue Jul 22 18:11:47 2025
How to establish local preference for (special) remotes

Sorry if it is RTFM... If I have multiple original (reachable) remotes, how could I establish my preference for which one to be used in any given location?

usecase: if I clone a repository within amazon cloud instance -- I would now prefer if this (or all -- user-wide configuration somehow?) repository 'get's load from URLs originating in the cloud of this zone (e.g. having us-east-1.s3.amazonaws.com/ in their URLs).

Comment by http://yarikoptic.myopenid.com/ — Tue Jul 22 18:11:47 2025
Re: Webhook special remote
@Alex: You might see if the newly-added wishlist: allow configuration of downloader for addurl could be made to do what you need... I've not played around with it yet, but perhaps you could set the downloader to be something that can sort out the various URLs and send them to the correct downloading tool?
Comment by andy — Tue Jul 22 18:11:47 2025
Rackspace Cloud Files support
It'd be really cool to have Rackspace cloud files support. Like the guy above me said, I would submit a patch but not if I have to learn Haskell first :)
Comment by https://www.google.com/accounts/o8/id?id=AItOawkQqKSVY98PVGDIaYZdK9CodJdbh7cFfhY — Tue Jul 22 18:11:47 2025
MediaFire
MediaFire offers 50GB of free storage (max size 200MB). It would be great to support it as a new special remote.
Comment by https://www.google.com/accounts/o8/id?id=AItOawk9nck8WX8-ADF3Fdh5vFo4Qrw1I_bJcR8 — Tue Jul 22 18:11:47 2025
Comments on this page are closed.