git-annex/ tips/ using Google Cloud Storage

For using Google Cloud Storage as a special remote, there are currently three choices:

- Google Cloud Storage supports supports the same API as Amazon S3, so git-annex's built-in S3 special remote can be used with it. You may need to configure Google Cloud Storage to allow "Interoperable Access". Here is how to set up the special remote:

  git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80

- Alternatively, rclone supports Google Cloud Storage's native API, so the rclone special remote can be used.

- Alternatively, there is a dedicated special remote, https://github.com/bgilbert/gcsannex
  (Last updated 2016)

At this time it's not clear which is better, so if you find one works best, please comment below.

Coldline
Wanted to add that "storageclass=COLDLINE" appears to work seamlessly, both from my mac and arm NAS. As far as I can tell, this appears to be a no-brainer vs glacier - builtin git annex client, simpler/cheaper billing, and no 4 hour delay!
Comment by scottgorlin — Tue Jul 22 18:11:47 2025
comment 7

@Dosenpfand at a guess, perhaps you have missed enabling "Interoperable Access" in google cloud storage?

Or, maybe Google managed to break their S3 compatability..

gcsannex is probably a better way to tie git-annex and GCS together than using the S3 interface.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 6

When trying to use google cloud storage i get the following error:

    $ git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80 storageclass=NEARLINE
    initremote cloud (checking bucket...) (creating bucket in US...) git-annex: S3Error {s3StatusCode = Status {statusCode = 400, statusMessage = "Bad Request"}, s3ErrorCode = "MalformedHeaderValue", s3ErrorMessage = "An HTTP header value was malformed.", s3ErrorResource = Nothing, s3ErrorHostId = Nothing, s3ErrorAccessKeyId = Nothing, s3ErrorStringToSign = Nothing, s3ErrorBucket = Nothing, s3ErrorEndpointRaw = Nothing, s3ErrorEndpoint = Nothing}

Does anybody know what is causing this? I am pretty sure that the credentials are correctly set and ommiting storageclass=NEARLINE does not help.

Comment by Dosenpfand — Tue Jul 22 18:11:47 2025
comment 5

Nice job bgilbert on the external special remote.

git-annex's S3 special remote also supports nearline now:

    git annex initremote cloud type=S3 encryption=none host=storage.googleapis.com port=80 storageclass=NEARLINE

For the nearline support to work, git-annex has to be built with version 0.13.0 of the haskell aws library.

But, it seems like bgilbert's external special remote is easier to set up, due to not needing to configure Google Cloud Storage for interoperable access.

Comment by joey — Tue Jul 22 18:11:47 2025
External special remote
I've written an external special remote for Google Cloud Storage using Google's Python bindings to the native JSON API. The special remote supports all storage classes, including Nearline, and doesn't require Interoperable Access to be enabled. Feedback welcome.
Comment by bgilbert — Tue Jul 22 18:11:47 2025
comment 3

I just found the "Interoperable access" thing. The instructions from Google (linked in previous comments) are not perfect.

After step 4 click "Google Cloud Storage", then at the bottom you have a heading called "Interoperable Access" where you have to activate it for this project.

Then it appears at the bottom of the left hand menu.

Comment by https://www.google.com/accounts/o8/id?id=AItOawlPfB0_GcGJ827kZqRE1PF4D23rxpfOeg4 — Tue Jul 22 18:11:47 2025
Can't find developer API key and secret

According to the last comment, and this page: * https://developers.google.com/storage/docs/migrating#keys

there's supposed to be a "Interoperable Access" menu option, something I can't find. Note that both the previous comment and this page refers to Googles old Developer Console.

I can find a few things under "APIs and Auth" => "Credentials" under the heading "Public API access", but no matter what kind of key I try to create I can't get anything other than an API key. No secrets here, and no "Show"-button for newly created keys.

Another thing worth mentioning is that these API keys I'm seeing do not like described in that first link in this comment. It's much longer, contains both upper- and lowercase and also contains dashes.

Comment by https://www.google.com/accounts/o8/id?id=AItOawlPfB0_GcGJ827kZqRE1PF4D23rxpfOeg4 — Tue Jul 22 18:11:47 2025
AWS credentials

This looks very valuable - Google are offering a free 5GB up until 2013 June 30th

Sign in here[1] get your credentials here[2] under the section “Interoperable Access” (Source[3])

    # Set up AWS credentials
    $ export AWS_ACCESS_KEY_ID="YOUR-KEY"
    $ export AWS_SECRET_ACCESS_KEY="YOUR-SECRET"

1. http://gs-signup-redirect.appspot.com/ -or- https://developers.google.com/storage/docs/signup
2. https://storage.cloud.google.com/m
3. http://fog.io/storage/

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Tue Jul 22 18:11:47 2025
Comments on this page are closed.