git-annex/ special remotes/ glacier

This special remote type stores file contents in Amazon Glacier.

To use it, you need to have glacier-cli installed.

The unusual thing about Amazon Glacier is the multiple-hour delay it takes to retrieve information out of Glacier. To deal with this, commands like "git-annex get" request Glacier start the retrieval process, and will fail due to the data not yet being available. You can then wait approximately four hours, re-run the same command, and this time, it will actually download the data.

## configuration

The standard environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are used to supply login credentials for Amazon. You need to set these only when running `git annex initremote` (or `enableremote`), as they will be cached in a file only you can read inside the local git repository.

A number of parameters can be passed to `git annex initremote` to configure the Glacier remote.

- `encryption` - One of "none", "hybrid", "shared", or "pubkey". See encryption.

- `keyid` - Specifies the gpg key to use for encryption.

- `embedcreds` - Optional. Set to "yes" embed the login credentials inside the git repository, which allows other clones to also access them. This is the default when gpg encryption is enabled; the credentials are stored encrypted and only those with the repository's keys can access them.

  It is not the default when using shared encryption, or no encryption. Think carefully about who can access your repository before using embedcreds without gpg encryption.

- `datacenter` - Defaults to "us-east-1".

- `vault` - By default, a vault name is chosen based on the remote name and UUID. This can be specified to pick a vault name.

- `fileprefix` - By default, git-annex places files in a tree rooted at the top of the Glacier vault. When this is set, it's prefixed to the filenames used. For example, you could set it to "foo/" in one special remote, and to "bar/" in another special remote, and both special remotes could then use the same vault.

that is an ancient version of git-annex, Shane
The first git-annex release to support glacier without using hooks was 3.20121126.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
Annex can't find Glacier remote

I'm setting up git-annex with glacier-cli for the first time. I have installed git-annex via Yum and glacier-cli according to the instructions on Github. The `glacier` command is in my path. I did not set up hooks with Git annex as it appears that using hooks for glacier is no longer required.

Here is my version information for git-annex:

    $ git-annex version
    git-annex version: 3.20120523
    local repository version: 3
    default repository version: 3
    supported repository versions: 3
    upgrade supported from repository versions: 0 1 2

When I attempt to add my Glacier remote, here is what I see:

    $ git annex initremote glacier type=glacier keyid=ABCDEFG
    git-annex: Unknown remote type glacier

Is there something else I need to do in order to correctly install Glacier integration with git-annex? I'm having trouble finding up-to-date information that describes the installation process.

Comment by https://www.google.com/accounts/o8/id?id=AItOawlhaXbLZQ5uQ6fM2kV0n8bj6IwZjx2CLeM — Tue Jul 22 18:11:47 2025
Expected cost for a repository

Is there a way to estimate the cost of storing a repo on glacier?

I'm especially worried because of the cost of STORE and RETRIEVE requests; there are hundreds of thousands of small files in my annex repo, so that request cost could easily dominate storage cost. Does the glacier remote do anything to minimize the number of objects stored in glacier?

Comment by https://www.google.com/accounts/o8/id?id=AItOawlq4ClC5EMN1Vq1DpWXAqP5TiDnCK1mSfk — Tue Jul 22 18:11:47 2025
I know this thread is old but...
It's working well for me, though I seem to sometimes still hit the duplicate bug listed above.
Comment by https://www.google.com/accounts/o8/id?id=AItOawmZgZuUhZlHpd_AbbcixY0QQiutb2I7GWY — Tue Jul 22 18:11:47 2025
comment 5
You are not the only one, Justin. I am just getting into git-annex and I am setting up a glacier remote as I write this.
Comment by http://id.clacke.se/ — Tue Jul 22 18:11:47 2025
comment 4
Let's discuss this in a bug. I've created http://git-annex.branchable.com/bugs/Glacier_remote_uploads_duplicates/
Comment by basak — Tue Jul 22 18:11:47 2025
comment 3

Hi! :)

The main issue I'm hitting is the "Multiple rows were found for one()" error. I think I get this when git-annex tries to upload the same file twice (which may be a bug in git-annex, which could apply de-duplication earlier), but I think I also get it when trying to upload a file whose upload I've canceled in the past.

I don't quite understand what git-annex needs here, and I totally understand that you're writing a general-purpose tool. But there does seem to be an issue that git-annex needs fixed one way or another.

I'm happy to try fixing it myself if you can help me understand what's going on (I didn't quite understand your review in the PR), but if I'm the only person in the world using git-annex to back up to glacier, that scares me a little!

    copy foo/bar/baz (checking glacier...) Traceback (most recent call last):
      File "/home/jlebar/code/glacier-cli/glacier", line 694, in <module>
        App().main()
      File "/home/jlebar/code/glacier-cli/glacier", line 680, in main
        args.func(args)
      File "/home/jlebar/code/glacier-cli/glacier", line 579, in archive_checkpresent
        last_seen = self.cache.get_archive_last_seen(args.vault, args.name)
      File "/home/jlebar/code/glacier-cli/glacier", line 157, in get_archive_last_seen
        result = self._get_archive_query_by_ref(vault, ref).one()
      File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2182, in one
        "Multiple rows were found for one()")
    sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one()

Comment by https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c — Tue Jul 22 18:11:47 2025
comment 2

I'm the glacier-cli author. It is not abandoned!

glacier-cli is supposed to map to Glacier exactly, so that it is compatible with all other tools. Most of the outstanding PRs break this essential behaviour, so I have not merged them. Many of the feature requests and bugs related to the upstream boto library, which is just about the best maintained client library that exists for AWS on any platform (and Amazon have adopted it now, IIRC). I have written appropriate reviews on all the PRs.

If there is specific behaviour that git-annex needs, them I am happy to accept PRs for this, provided that they do not break the ability (and default) for glacier-cli to talk to Glacier natively without an extra layer of interpretation. If an extra layer of interpretation is needed (eg. forbidding duplicate "keys"), then this needs to be an option, or wrapped in a separate tool, or written into git-annex's Glacier special remote.

Comment by basak — Tue Jul 22 18:11:47 2025
Re: Archive not found

@Tim

Sounds like you need https://github.com/basak/glacier-cli/#user-content-cache-reconstruction

Comment by basak — Tue Jul 22 18:11:47 2025
Archive not found

For a couple of years already I've been regularly uploading my pictures to glacier through git-annex. After losing my local disk I'm trying to restore my files. However I get these errors:

    $ git annex get . --from glacier
    get DSC05172.JPG (from glacier...)
    glacier: archive 'GPGHMACSHA1--a0c0b35b3c2967e78a012ec95af2a77abf78465a' not found
    failed
    get DSC05173.JPG (from glacier...)
    glacier: archive 'GPGHMACSHA1--272c741c5dc61a3a45246b55d9777c451425b2bc' not found
    failed
    get DSC05173.variant-611a.JPG (from glacier...)
    glacier: archive 'GPGHMACSHA1--272c741c5dc61a3a45246b55d9777c451425b2bc' not found
    failed

No files have been deleted, the glacier bucket still exists and has all its data. Has anything changed in the naming scheme or anything? Or is it possible that this error is masking a different issue?

Comment by tim — Tue Jul 22 18:11:47 2025
RequestTimeoutException when uploading to glacier

I sometimes receive the following error when trying to upload files to glacier:

    Traceback (most recent call last):
     File "/home/victor/bin/glacier", line 736, in <module>
        main()
      File "/home/victor/bin/glacier", line 732, in main
        App().main()
      File "/home/victor/bin/glacier", line 718, in main
        self.args.func()
      File "/home/victor/bin/glacier", line 500, in archive_upload
        file_obj=self.args.file, description=name)
      File "/usr/lib/python2.7/site-packages/boto/glacier/vault.py", line 178, in create_archive_from_file
        writer.close()
      File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 228, in close
        self.partitioner.flush()
      File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 79, in flush
        self._send_part()
      File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 75, in _send_part
        self.send_fn(part)
      File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 222, in _upload_part
        self.uploader.upload_part(self.next_part_index, part_data)
      File "/usr/lib/python2.7/site-packages/boto/glacier/writer.py", line 129, in upload_part
        content_range, part_data)
      File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 1279, in upload_part
        response_headers=response_headers)
      File "/usr/lib/python2.7/site-packages/boto/glacier/layer1.py", line 119, in make_request
        raise UnexpectedHTTPResponseError(ok_responses, response)
    boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)
    gpg: [stdout]: write error: Broken pipe
    gpg: DBG: deflate: iobuf_write failed
    gpg: [stdout]: write error: Broken pipe
    gpg: filter_flush failed on close: Broken pipe
    gpg: [stdout]: write error: Broken pipe
    gpg: filter_flush failed on close: Broken pipe
    git-annex: fd:17: hPutBuf: resource vanished (Broken pipe)

It happens only sometimes. glacier-cli can upload files without problems. The progress of the file upload is also erratic, it jumps to ~90% and then gets stuck. Can I do something to resolve this?

    git-annex version: 5.20140717
    build flags: Assistant Inotify DBus TDFA
    key/value backends: SHA256E SHA1E SHA512E SHA224E SHA384E SHA256 SHA1 SHA512 SHA224 SHA384 WORM URL
    remote types: git gcrypt bup directory rsync web glacier ddar hook external

I am using glacier-cli from git master.

Comment by victorsavu3 — Tue Jul 22 18:11:47 2025
re: UnicodeDecodeError

This part:

      File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
        msg += message_body
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 0: ordinal not in range(128)

Tells me that you need to read glacier-cli problem report #61.

There is a one-line code change in a library named boto (glacier-cli depends on it) which will fix this. (And probably that change will get merged in sometime, so you won't have to do this anymore.)

Comment by dave — Tue Jul 22 18:11:47 2025
Glacier-cli works, thanks, some encode / decode error now...

Hi Joey,

Thanks for the hand, it started uploading once I had manually created the vault but then borked with:

    fozz@cobol:~/Store/family_pictures $ git annex --verbose copy 201/2011/05/07/P1010004.RW2 --to familypictures-glacier
    copy 201/2011/05/07/P1010004.RW2 (gpg) (checking familypictures-glacier...) (to familypictures-glacier...)
    81%           4.0MB/s 0sTraceback (most recent call last):
      File "/home/fozz/.vendor/bin/glacier", line 730, in <module>
        App().main()
      File "/home/fozz/.vendor/bin/glacier", line 716, in main
        self.args.func()
      File "/home/fozz/.vendor/bin/glacier", line 498, in archive_upload
        file_obj=self.args.file, description=name)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/vault.py", line 177, in create_archive_from_file
        writer.write(data)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 219, in write
        self.partitioner.write(data)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 61, in write
        self._send_part()
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 75, in _send_part
        self.send_fn(part)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 222, in _upload_part
        self.uploader.upload_part(self.next_part_index, part_data)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/writer.py", line 129, in upload_part
        content_range, part_data)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 1279, in upload_part
        response_headers=response_headers)
      File "/usr/local/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 114, in make_request
        data=data)
      File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
        retry_handler=retry_handler)
      File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 943, in _mexe
        request.body, request.headers)
      File "/usr/lib/python2.7/httplib.py", line 979, in request
        self._send_request(method, url, body, headers)
      File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
        self.endheaders(body)
      File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
        self._send_output(message_body)
      File "/usr/lib/python2.7/httplib.py", line 833, in _send_output
        msg += message_body
    UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 0: ordinal not in range(128)
    gpg: [stdout]: write error: Broken pipe
    gpg: DBG: deflate: iobuf_write failed
    gpg: build_packet failed: file write error
    gpg: [stdout]: write error: Broken pipe
    gpg: iobuf_flush failed on close: file write error
    gpg: symmetric encryption of `[stdin]' failed: file write error
    git-annex: fd:17: hPutBuf: resource vanished (Broken pipe)
    failed
    git-annex: copy: 1 failed

Comment by git-annex_branchable.com — Tue Jul 22 18:11:47 2025
comment 11

@forbesmyester, I think that the "glacier" program you have installed is the one from boto, not the one from glacier-cli. git-annex only supports the glacier-cli one.

Note that, since version 5.20150219, git-annex probes to see if the "glacier" program in PATH is the one from boto, and fails with a nicer error message.

Comment by joey — Tue Jul 22 18:11:47 2025
Not sure what is wrong... looks like `glacier` might be called wrongly.

Have done a full setup as much as I can, with all the GPG / AWS stuff but it keeps doing non stop...

    copy 201/2011/05/20/P1010044.RW2 (checking familypictures-glacier...) [2015-08-18 08:37:05 BST] read: glacier ["--region=us-east-1","archive","checkpresent","familypictures","--quiet","GPGHMACSHA1--[SOME_HASH]"]
    (to familypictures-glacier...)
    [2015-08-18 08:37:05 BST] feed: glacier ["--region=us-east-1","archive","upload","--name","GPGHMACSHA1--[SOME_HASH]","familypictures","-"]
    [2015-08-18 08:37:05 BST] chat: gpg ["--quiet","--trust-model","always","--batch","--passphrase-fd","40","--symmetric","--force-mdc","--no-textmode"]
    0%            0.0 B/s 0s
    glacier <command> [args]

        Commands
            vaults    - Operations with vaults
            jobs      - Operations with jobs
            upload    - Upload files to a vault. If the vault doesn't exits, it is
                        created

        Common args:
            --access_key - Your AWS Access Key ID.  If not supplied, boto will
                           use the value of the environment variable
                           AWS_ACCESS_KEY_ID
            --secret_key - Your AWS Secret Access Key.  If not supplied, boto
                           will use the value of the environment variable
                           AWS_SECRET_ACCESS_KEY
            --region     - AWS region to use. Possible values: us-east-1, us-west-1,
                           us-west-2, ap-northeast-1, eu-west-1.
                           Default: us-east-1

        Vaults operations:

            List vaults:
                glacier vaults

        Jobs operations:

            List jobs:
                glacier jobs <vault name>

        Uploading files:

            glacier upload <vault name> <files>

            Examples :
                glacier upload pics *.jpg
                glacier upload pics a.jpg b.jpg

    gpg: [stdout]: write error: Broken pipe
    gpg: DBG: deflate: iobuf_write failed
    gpg: build_packet failed: file write error
    gpg: [stdout]: write error: Broken pipe
    gpg: iobuf_flush failed on close: file write error
    gpg: [stdout]: write error: Broken pipe
    gpg: iobuf_flush failed on close: file write error
    gpg: symmetric encryption of `[stdin]' failed: file write error
    git-annex: fd:46: hPutBuf: resource vanished (Broken pipe)
    failed

Comment by forbesmyester — Tue Jul 22 18:11:47 2025
comment 1

The glacier-cli tool seems to have been abandoned, and there are a number of outstanding issues with it. boto has a `glacier` tool, but it doesn't seem to include caching, which seems to be something git annex needs.

Looking through the PRs, it seems like we should build a tool specifically tailored to git annex's needs. It seems that there are at least three of us willing to hack on this if it's in Python. I'm not sure any of us knows haskell, though...

Comment by https://www.google.com/accounts/o8/id?id=AItOawmUJBh1lYmvfCCiGr3yrdx-QhuLCSRnU5c — Tue Jul 22 18:11:47 2025
Comments on this page are closed.