git-annex/ tips/ public Amazon S3 remote

Here's how to create a Amazon S3 special remote that can be read by anyone who gets a clone of your git-annex repository, without them needing Amazon AWS credentials.

If you want to publish files to S3 so they can be accessed without using git-annex, see publishing your files to the public.

Note: Bear in mind that Amazon will charge the owner of the bucket for public downloads from that bucket.

## create public remote

First, export your Amazon AWS credentials:

    # export AWS_ACCESS_KEY_ID="08TJMT99S3511WOZEP91"
    # export AWS_SECRET_ACCESS_KEY="s3kr1t"

Now, create the remote:

    # git annex initremote pubs3 type=S3 encryption=none public=yes
    initremote pubs3 (checking bucket) (creating bucket in US) ok

The public=yes makes git-annex set a S3 ACL so the files in the bucket are publically readable. For git-annex to be able to download files from that bucket without needing your AWS credentials, you then need to tell it the url of the bucket. Find that url, and run:

    # git annex enableremote pubs3 publicurl=...

In the above example, no encryption was used, but it will also work if you enable encryption=shared. Then files will be encrypted on S3, and anyone with a clone of the git repository will be able to download and decrypt them.

It's also ok to enable chunking when setting up the remote.

Now, copy some files to the remote, in the usual way, and push your git repository to someplace where someone else can access it.

## use public remote

Once the S3 remote is set up, anyone who can clone the git repositry can get files from the remote, without needing any Amazon AWS credentials.

Start by checking out the git repository.

In the checkout, enable the S3 remote:

    # git annex enableremote pubs3
    enableremote pubs3 ok

Now, git-annex can be used as usual to download files from that remote.

## sharing urls

You can also share urls to files stored in a public S3 remote to people who are not using git-annex. To find the url, use `git annex whereis`.

See S3 for details about configuring S3 remotes.

comment 2

It seems to me that you need to tell git-annex the publicurl to use to download files from this S3 remote. That's the url to the top of the S3 bucket. Run `git annex enableremote publics3 publicurl=...' and push, and it should then become available in every clone.

The documentation for publicurl says it's enabled by default when public=yes, but AFAICS it's never actually done that. And I'm not sure I want to hardcode those urls into git-annex. I've updated the documentation instead.

Comment by joey — Tue Jul 22 18:11:47 2025
Unable to access public s3 remote without S3 credentials.

I'm setting up a repository using the following commands:

    git annex init
    export AWS_ACCESS_KEY_ID="[MY KEY]"
    export AWS_SECRET_ACCESS_KEY="[MY SECRET]"
    git annex initremote publics3 type=S3 encryption=none bucket=[BUCKET] exporttree=yes public=yes encryption=none
    git annex export --tracking master --to publics3

I then add something and export it to s3 using:

    git annex sync --content

I then go on to a different computer and clone the repository and run:

    git annex init
    git annex enableremote publics3
    git annex get .

and receive this message:

    # git annex get .
    get IMG_2714.MOV (from publics3...)
    Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use S3

    No S3 credentials configured

    Unable to access these remotes: publics3

    Try making some of these repositories available:
         6bc91baa-5250-41e2-8195-d3c1ef3c86e3 -- .
         7f84dcff-5a74-42b0-b422-5b74490e4aa7 -- [publics3]

     (Note that these git remotes have annex-ignore set: origin)

failed git-annex: get: 1 failed

It is my understanding that setting the remote to "public=yes" should allow me to download the files from the remote repository without entering the credentials again. Am I missing something?

Thanks!

Comment by jared — Tue Jul 22 18:11:47 2025
Comments on this page are closed.