git-annex/ special remotes/ ipfs

This special remote stores file contents in ipfs.

Warning: ipfs is written in Go, a language controlled by Google, which collects user's data with its compiler. If installing ipfs from source you should consider investigating how to disable Go's data collection. Or install precompiled ipfs from a software distributor you trust.

## prerequisites

- Install git-annex-remote-ipfs somewhere in PATH and `chmod +x` the script.
- Install go-ipfs somewhere in PATH.
- Run `ipfs init` and start the `ipfs daemon`

(Note that this special remote does not use ipfs's FUSE support; it communicates with ipfs using the `ipfs` command-line utility.)

## configuration

These parameters can be passed to `git annex initremote` to configure the remote:

- `encryption` - One of "none", "hybrid", "shared", or "pubkey". See encryption. Note that this is git-annex's encryption, not ipfs's encryption.

- `keyid` - Specifies the gpg key to use for encryption.

Setup example:

    # git annex initremote ipfs type=external externaltype=ipfs encryption=none

## content distribution

After `git annex copy --to ipfs`, a file will typically only have been copied to your computer's local ipfs object store. It will not reach other ipfs nodes on the network until they request the content.

If you set up a clone of your repository on another computer, and install ipfs and enable the ipfs remote there, you can proceed with using it to get files that have been stored in ipfs:

    # git annex sync
    # git annex enableremote ipfs
    # git annex copy --from ipfs

## content removal

Removing content from ipfs requires all nodes that have a copy to decide to delete it. This is not something git-annex can arrange to happen, or reliably tell has happened, so `git annex drop --from ipfs` will always fail.

## using ipfs addresses

Once a file has been copied to ipfs, you can use `git annex whereis` to look up the ipfs address of the file:

    # git annex whereis somefile
    whereis somefile
        ed1c811d-fe42-4436-aa75-56566c990aa8 -- ipfs

    ipfs: ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W

In the example above, the ipfs address for the file is `QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W`. You can give this address to any other ipfs user and they can use it to download the file!

You can also use ipfs addresses with `git annex addurl`. For example:

    # git annex addurl ipfs:QmYgXEfjsLbPvVKrrD4Hf6QvXYRPRjH5XFGajDqtxBnD4W --file somefile

That's a real file; try it!

## future directions

While perhaps useful, this is just a proof of concept. It's particularly lacking in that it doesn't integrate well git-annex's location tracking with ipfs.

Tracking which ipfs nodes have a copy of an annexed object would make this special remote work better. In particular, git-annex does not currently trust ipfs to contain a copy of an object, since it has no way of keeping track of which which ipfs nodes might contain it. So, eg, `git annex drop` will refuse to trust ipfs.

comment 9
There's now a service that pins your IPFS files for a low price: https://www.eternum.io/, I was looking for something like this to start using IPFS with git-annex. Maybe I'll do try now.
Comment by fiatjaf — Wed Oct 29 14:05:27 2025
git remote
there is now a git remote for IPFS. I believe it suffers from the same problem as all files within ipfs, namely that the contents are duplicated inside the repo. However, it certainly make for an interesting decentralised way of sharing metadata. --anarcat
Comment by anarcat — Wed Oct 29 14:05:27 2025
git-annex-remote-ipfs
Silly me - I messed up my bashrc. Fixed now, thanks!
Comment by rob.syme — Wed Oct 29 14:05:27 2025
comment 6

@rob.syme I think that error message is pretty clear: git-annex-remote-ipfs is not present in your PATH or perhaps is not executable.

(Still, the error message could be improved a bit, doing that now.)

Comment by joey — Wed Oct 29 14:05:27 2025
Finding IPFS hash

The use case is only very silly - I just wanted to look directly at the data stored at that IPFS hash - it was not important and certainly not worth changing the way git-annex stores its keys.

On a new machine with a new version of git-annex, I can no longer initialize a new IFPS remote:

    ⟫ ipfs version
    ipfs version 0.3.10-dev

    ⟫ git annex version
    git-annex version: 5.20151116-gbe86081
    build flags: Assistant Webapp Webapp-secure Pairing Testsuite S3 WebDAV Inotify DBus DesktopNotify XMPP ConcurrentOutput DNS Feeds Quvi TDFA TorrentParser Database
    key/value backends: SHA256E SHA256 SHA512E SHA512 SHA224E SHA224 SHA384E SHA384 SHA3_256E SHA3_256 SHA3_512E SHA3_512 SHA3_224E SHA3_224 SHA3_384E SHA3_384 SKEIN256E SKEIN256 SKEIN512E SKEIN512 SHA1E SHA1 MD5E MD5 WORM URL
    remote types: git gcrypt S3 bup directory rsync web bittorrent webdav tahoe glacier ddar hook external
    local repository version: 5
    supported repository version: 5
    upgrade supported from repository versions: 0 1 2 4

    ⟫ git annex initremote ipfs type=external externaltype=ipfs encryption=none
    initremote ipfs
    git-annex: git-annex-remote-ipfs: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    failed
    git-annex: initremote: 1 failed

It looks like I'm missing a binary somewhere, but I'm not sure which one is missing.

Comment by rob.syme — Wed Oct 29 14:05:27 2025
comment 4

@rob.syme, the reason that does not work when using encryption is that git-annex internally generates an encrypted key, and stores that on IPFS. The ipfs special remote records the IPFS address that can be used for a key, but it's recorded on the encrypted key, which is not the one you're querying with whereis.

This is a general problem with git-annex; ipfs is probably the first special remote that exposes the problem.

Anyway, I'm not sure what benefit knowing the IPFS address of a file encrypted by git-annex is. You need to use git-annex to decrypt it, in general. You might be able to use your gpg key to decrypt it, if the special remote was set up using pubkey encryption, but that's not going to help anyone else you give the IPFS address to access the encrypted data..

Does your use case involve pinning the IPFS address, or something like that?

Comment by joey — Wed Oct 29 14:05:27 2025
Finding IPFS hash
I see now that git-annex only reports the ipfs hash when the remote is unencrypted. It the ipfs hash not reported for encrypted ipfs remotes for security reasons?
Comment by rob.syme — Wed Oct 29 14:05:27 2025
Finding IPFS hash

I'm using git-annex version: 5.20150522-gb199d65

The example above gives the ipfs hash for the file, but when I run whereis, the hash is not reported:

    $ git annex whereis test.txt
    whereis test.txt (2 copies)
            1bd0f840-2247-4888-9237-596515788671 -- rob@rob-G60JX:/tmp/gitannextest [here]
            adf361b3-7b1a-43e1-8360-937f7e436e90 -- [ipfs]
    ok

Is there a way for git-annex to report the file's ipfs hash ID?

Comment by rob.syme — Wed Oct 29 14:05:27 2025
comment 16

i had written a hacky special remote for web3.storage at https://github.com/xloem/gitlakepy/blob/master/scripts/git-annex-remote-w3-subprocess but it was kind of flaky, wrapped the cli tool.

i guess arweave isn't relevant here for ipfs. the support for ipfs is nice with w3.storage because you can download things with the ipfs url without requiring the special remote installed.

i have cognitive decline, haven't found recovery from political trafficking, so depending on how my life goes i may make more and more spammy expressions. feel free to clean them up, and thank you for commenting on this one.

Comment by xloem — Wed Oct 29 14:05:27 2025
comment 15

sorry for the comments that could come across as marketing or spam

i am often in a situation where i have nowhere to back files up, so simply appreciated the free services and thought others might need something similar as well

their terms may have changed by now, not sure. they helped me a lot at the time. i'm now using arweave for doing similar free things, which has a custom protocol but is intended to never kick data out. node2.bundlr.network provides free "bundlr" arweave storage for objects 100,000 bytes or smaller.

Comment by xloem — Wed Oct 29 14:05:27 2025
comment 14

https://nft.storage/terms/ claims this is limited to NFTs and not other data. (Whatever that means; why can't I mint a NFT for a nice gpg encrypted blob?)

I also doubt it actually will store anything for longer than the current NFT mania; the terms have a loophole that allow them to provide 90 days notice before ending the service.

It does have an API if someone wants to build a special remote and use it for general data storage. But that's out of scope for this IPFS special remote AFAICS.

Comment by joey — Wed Oct 29 14:05:27 2025
comment 13

@xloem I'm not sure how services that have CLI interfaces to upload files to them, and then presumably pin them in IPFS are relevant to this IPFS special remote?

Your history of good comments is keeping me from deleting the above 2 as spam, but...

Comment by joey — Wed Oct 29 14:05:27 2025
free dweb storage services

ipfs appears to be running some free storage services for large files on ipfs backed by filecoin. https://nft.storage/ and htps://web3.storage/ are both presently ostensibly free and permanent and have cli interfaces to upload to them.

Comment by xloem — Wed Oct 29 14:05:27 2025
comment 12
addendum: note that filecoin itself is not permanent, the server needs to keep recreating deals for the data. a permanent solution would be something like arweave, a permastorage blockchain with tools, which is one of the solutions i found in the ethereum storage docs.
Comment by xloem — Wed Oct 29 14:05:27 2025
IPFS Deduplication

IPFS now has an experimental feature for not duplicating file content in the ipfs store, which could be very helpful for this use. See description at https://github.com/ipfs/go-ipfs/issues/3397#issuecomment-284337564 .

To use this, git-annex needs to pass the `--nocopy` flag when running `ipfs add`.

Comment by xloem — Wed Oct 29 14:05:27 2025
about copying to the local store
there's a discussion happening upstream about how copying to the local datastore could be avoided.
Comment by https://id.koumbit.net/anarcat — Wed Oct 29 14:05:27 2025
Comments on this page are closed.