git-annex/ tips/ using Amazon Glacier

Amazon Glacier provides low-cost storage, well suited for archiving and backup. But it takes around 4 hours to get content out of Glacier.

Recent versions of git-annex support Glacier. To use it, you need to have glacier-cli installed.

First, export your Amazon AWS credentials:

        # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
        # export AWS_SECRET_ACCESS_KEY="s3kr1t"

Now, create a gpg key, if you don't already have one. This will be used to encrypt everything stored in Glacier, for your privacy. Once you have a gpg key, run `gpg --list-secret-keys` to look up its key id, something like "2512E3C7"

Next, create the Glacier remote.

    # git annex initremote glacier type=glacier keyid=2512E3C7
    initremote glacier (encryption setup with gpg key C910D9222512E3C7) (gpg) ok

The configuration for the Glacier remote is stored in git. So to make another repository use the same Glacier remote is easy:

        # cd /media/usb/annex
        # git pull laptop
        # git annex enableremote glacier
        initremote glacier (gpg) ok

Now the remote can be used like any other remote.

        # git annex move my_cool_big_file --to glacier
        copy my_cool_big_file (gpg) (checking glacier...) (to glacier...) ok

But, when you try to get a file out of Glacier, it'll queue a retrieval job:

    # git annex get my_cool_big_file
    get my_cool_big_file (from glacier...) (gpg)
    glacier: queued retrieval job for archive 'GPGHMACSHA1--862afd4e67e3946587a9ef7fa5beb4e8f1aeb6b8'
      Recommend you wait up to 4 hours, and then run this command again.
    failed

Like it says, you'll need to run the command again later. Let's remember to do that:

    # at now + 4 hours
    at> git annex get my_cool_big_file

Another oddity of Glacier is that git-annex is never entirely sure if a file is still in Glacier. Glacier inventories take hours to retrieve, and even when retrieved do not necessarily represent the current state.

So, git-annex plays it safe, and avoids trusting the inventory:

    # git annex copy important_file --to glacier
    copy important_file (gpg) (checking glacier...) (to glacier...) ok
    # git annex drop important_file
    drop important_file (gpg) (checking glacier...)
      Glacier's inventory says it has a copy.
      However, the inventory could be out of date, if it was recently removed.

    (unsafe)
      Could only verify the existence of 0 out of 1 necessary copies

To avoid this problem, you can either use `git annex move` to move content to Glacier, or you can set the remote to be trusted.

A final potential gotcha with Glacier is that glacier-cli keeps a local mapping of file names to Glacier archives. If this cache is lost, or you want to retrieve files on a different box than the one that put them in glacier, you'll need to use `glacier vault sync` to rebuild this cache.

See Glacier for details.

comment 8

@ben, it's generating the encryption key, and is blocked waiting on enropy. You can pass --fast to use lower-quality randomness.

Comment by joey — Tue Jul 22 18:11:47 2025
Problems initializing glacier remote

Hi, when I try to create a glacier remote, the command freezes without further output:

    $ git init
    $ git annex init
    $ git annex initremote glacier type=glacier keyid=xxxxxxxx
    initremote glacier (encryption setup)

I can see the following processes in sleep state:

    11438 pts/0    S+     0:00 git --git-dir=/home/b/Documents/annex/.git --work-tree=/home/b/Documents/annex cat-file --batch
    11440 pts/0    SL+    0:00 gpg2 --batch --no-tty --use-agent --quiet --trust-model always --gen-random --armor 2 512

I'm on fedora 22, git-annex version: 5.20140717. Any suggestions appreciated, thanks!

Comment by ben — Tue Jul 22 18:11:47 2025
confirmed
Yeah, I choose no encryption for this one for worst case scenario reasons (I still want photos of my kid even if I loss my gpg key and my house burns down). Now about setting up http://git.kitenet.net/?p=gpg.git;a=blob;f=README.sss;hb=HEAD ......
Comment by http://grossmeier.net/ — Tue Jul 22 18:11:47 2025
good
If you mean the creds are not remembered, that's controlled by the embedcreds= option to initremote, and it only defaults to embedding them for gacier when using strong encryption (not encryption=shared).
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 4

Along with stupid python problems which are now fixed (all my fault, and hopefully didn't cause more noise here than needed), the only thing that didn't go as stated was:

[[!format sh """ greg@x200s:~/Photos$ git-annex enableremote glacier enableremote glacier Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use glacier git-annex: Failed creating glacier vault. greg@x200s:~/Photos$ AWS_ACCESS_KEY_ID=lolno AWS_SECRET_ACCESS_KEY=lolno git-annex enableremote glacier enableremote glacier ok (Recording state in git...) greg@x200s:~/Photos$ """]]

The guide says that info is sync'd.

Comment by http://grossmeier.net/ — Tue Jul 22 18:11:47 2025
comment 3
I've changed it to avoid running glacier value create when enabling an existing glacier remote. Hopefully that fixes it.
Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 2

@greg, the only thing you might have missed is the need to use `glacier vault sync` to build a cache if enabling the glacier remote in another place. And that whole issue with it needing a local cache may mean few people are using glacier with more than one repository accessing the remote.

However, this sounds like a bug. There is a comment in the source code that "glacier vault create will succeed even if the vault already exists." .. perhaps it has changed since that was written. Or perhaps the command failed for some other reason, I don't know.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
missing steps?

I setup a glacier remote on one machine and it successfully created the vault and is syncing files to it.

One another machine, after git-annex sync'ing, I did:

[[!format sh """ greg@x200s:~/Photos$ git-annex enableremote glacier enableremote glacier Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use glacier git-annex: Failed creating glacier vault. """]]

So then I try: [[!format sh """ greg@x200s:~/Photos$ AWS_ACCESS_KEY_ID=HAHA AWS_SECRET_ACCESS_KEY=NOPE git-annex --debug enableremote glacier [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","show-ref","git-annex"] [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","show-ref","--hash","refs/heads/git-annex"] [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","log","refs/heads/git-annex..8108714116d08f93aa427b9ddced48cd5f2b4b72","--oneline","-n1"] [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","log","refs/heads/git-annex..742ba908f791e440a6cc85073ef505a96dd66aa4","--oneline","-n1"] [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","log","refs/heads/git-annex..071487394544a20253a70ada4ea71fcc28f9fc13","--oneline","-n1"] [2014-02-18 15:43:56 PST] read: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","log","refs/heads/git-annex..c8aecc22da7b84bbb82f083ce783cc699cef1c67","--oneline","-n1"] [2014-02-18 15:43:56 PST] chat: git ["--git-dir=/home/greg/Photos/.git","--work-tree=/home/greg/Photos","cat-file","--batch"] enableremote glacier [2014-02-18 15:43:56 PST] call: glacier ["--region=us-west-2","vault","create","glacier-7e5c0010-2634-4a5e-bc7b-6fea84b8b947"] git-annex: Failed creating glacier vault. """]]

What am I missing?

Also, why is it trying to create the valut? It's already there with content in it!

Comment by http://grossmeier.net/ — Tue Jul 22 18:11:47 2025
Comments on this page are closed.