git-annex/ tips/ publishing your files to the public

You have a git-annex repository, and you want to publish the files in it to the public. One way is to setup a public repository on a web site, but perhaps you don't have a web server that can run git-annex, and you just want to publish the current files, not the whole git-annex repository.

The git-annex export command is the solution. It lets a tree of files from your git-annex repository be published to Amazon S3, as well as other types of special remotes like webdav and directory.

# Publishing to Amazon S3

Let's create a bucket in Amazon S3 named $BUCKET and a special remote named public-s3. Exporting has to be enabled when setting up a special remote for the first time.

Set up your special S3 remote with (at least) these options:

    git annex initremote public-s3 type=S3 encryption=none bucket=$BUCKET exporttree=yes public=yes

Be sure to replace $BUCKET with something like "public-bucket-joey" when you follow along in your shell.

Want to only export files in a subdirectory of the master branch? Use `master:subdir`.

Any git treeish can be used with the export command, so you can also export tags, etc.

Then export the files in the master branch to the remote:

    git annex export master --to public-s3

Each exported file will be available to the public at `http://$BUCKET.s3.amazonaws.com/$FILE`

Note: Bear in mind that Amazon will charge the owner of the bucket for public downloads from that bucket.

# Using `git-annex sync --content`

So far, the current contents of the master branch have been exported to public-s3, and to update the export when the branch changes, you have to remember to run `git annex export` again.

If you use a `git annex sync` workflow, it's useful to configure it to also export changes to the remote. This is done by setting the remote's `annex-tracking-branch` configuration:

    git config remote.public-s3.annex-tracking-branch master

That tells git-annex that the export should track changes to master. When you run `git annex sync --content`, it will update all tracking exports. The git-annex assistant also automatically updates tracking exports.

Want to only export files in a subdirectory of the master branch?

    git config remote.public-s3.annex-tracking-branch master:subdir

# Amazon S3 indexes

By default, there is no index.html file exported, so if you open `http://$BUCKET.s3.amazonaws.com/` in a web browser, you'll see an XML document listing the files.

For a nicer list of files, you can make an index.html file, check it into git, and export it to the bucket. You'll need to configure the bucket to use index.html as its index document, as explained here.

# Old method

To use `git annex export`, you need git-annex version 6.20170909 or newer. Before we had `git annex export` an old method was used instead.

Re: Publishing subdir is publishing files but not subdir?

The reason exporting a subdir puts the files at the root of the folder is that it's exporting a git tree, and files in the tree of a subdir are relative to that subdir.

It's certianly possible to make git construct a tree object that contains a subdirectory containing the files you want to export, and then exporting that tree object would replicate that tree structure, including the subdir. One way is to check out a branch, delete the files you don't want to export and once things are arranged as you like, commit to that branch and then export it.

(Or you could just make the webdav special remote's url include the subdir where you wanrt it to export stuff.)

Comment by joey — Wed Oct 29 14:05:27 2025
comment 6

You should be able to run `git annex export` in the bare git repository on your server. It just needs the branch you want to export to be known to git, and the annexed file contents to be available there.

Comment by joey — Wed Oct 29 14:05:27 2025
Re: Problem with publishing subdir. Subdir does not exist, but it does.

I reproduced export of master:subdir not working. It's a bug in git-annex, now fixed!

Comment by joey — Wed Oct 29 14:05:27 2025
Publishing subdir is publishing files but not subdir?

When using the uuid the files get exported to the root of the specified webdav folder. Is there a possibility to keep the directory structure intact, so that the special remote has the same folder structure as has the repository?

Thanks in advance

Comment by jajpater — Wed Oct 29 14:05:27 2025
Works when first using git annex get

I dropped the files from the local directory. Publishing does work when I first do a

    git annex get *.*

But now my question has changed:

I have a git repository on a server. Is there a possibility to publish/export directly from the server to a special remote? So without checking out the files locally?

Comment by jajpater — Wed Oct 29 14:05:27 2025
Also fails when using UUID but differently

I tried to use the dir UUID - found by using

    git ls-tree origin/master

This however also fails:

    git annex export fd94c0f44ff40b07bcbe58d4c3921c6fb35c7074 -t box.com
    export box.com file1 (not available) failed
    export box.com file2 (not available) failed
    ...
    export box.com file10 (not available) failed
    (recording state in git...)
    git-annex: export: 10 failed

Does anybody know what's wrong?

Comment by jajpater — Wed Oct 29 14:05:27 2025
Problem with publishing subdir. Subdir does not exist, but it does.

When trying this, I get the following error.

    git annex export master:subdir -t box.com
    fatal: Path 'subdir:' does not exist in 'master'
    git-annex: unknown tree

I really don't get it, since as far as I can see the subdir actually does exist in master

Comment by jajpater — Wed Oct 29 14:05:27 2025
Comments on this page are closed.