git-annex/ tips/ using gitolite with git-annex

Gitolite is a git repository manager. Here's how to add git-annex support to gitolite, so you can `git annex copy` files to a gitolite repository, and `git annex get` files from it.

A nice feature of using gitolite with git-annex is that users can be given read-only access to a repository, and this allows them to `git annex get` file contents, but not change anything.

First, you need new enough versions:

- the current `master` branch of gitolite works with git-annex (tested 2014-04-19), but v3.5.3 and earlier v3.x require use of the `git-annex` branch.
- gitolite 2.2 also works -- this version contains a git-annex-shell ADC and supports "ua" ADCs.
- git-annex 3.20111016 or newer needs to be installed on the gitolite server. Don't install an older version, it wouldn't be secure!

### Instructions for gitolite `master` branch

To setup gitolite to work with git-annex, you can follow the instructions on the gitolite website, and just add `'git-annex-shell ua',` to the ENABLE list in `~/.gitolite.rc`.

Here are more detailed instructions:

1: Create a `git` user

    sudo adduser \
       --system \
       --shell /bin/bash \
       --gecos 'git version control' \
       --group \
       --disabled-password \
       --home /home/git git

2: Copy a public SSH key for the user you want to be the gitolite administrator. In the instructions below, I placed the key in a file named `/home/git/me.pub`.

3: Clone and install gitolite

First switch to the `git` user (e.g. `sudo su - git`) and then run:

    cd
    git clone https://github.com/sitaramc/gitolite.git
    mkdir -p bin
    ./gitolite/install -ln

4: Add `~/bin` to `PATH`

Make sure that `~/bin` is in the `PATH`, since that's where gitolite installed its binary. Do something like this:

    echo 'export PATH=/home/git/bin:$PATH' >> .profile
    export PATH=/home/git/bin:$PATH

5: Configure gitolite

Edit `~/.gitolite.rc` to enable the git-annex-shell command. Find the `ENABLE` list and add this line in there somewhere:

    'git-annex-shell ua',

Now run gitolite's setup:

    gitolite setup -pk me.pub
    rm me.pub

### Instructions for gitolite 2.2

And here's how to set it up. The examples are for gitolite as installed on Debian with apt-get, but the changes described can be made to any gitolite installation, just with different paths.

Set `$GL_ADC_PATH` in `.gitolite.rc`, if you have not already done so.

    echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/";' >>~gitolite/.gitolite.rc

Make the ADC directory, and a "ua" subdirectory.

    mkdir -p /usr/local/lib/gitolite/adc/ua

Install the git-annex-shell ADC into the "ua" subdirectory from the gitolite repository.

    cd /usr/local/lib/gitolite/adc/ua/
    cp gitolite/contrib/adc/git-annex-shell .

Now all gitolite repositories can be used with git-annex just as any ssh remote normally would be used. For example:

    # git clone gitolite@localhost:testing
    Cloning into testing...
    Receiving objects: 100% (18/18), done.
    # cd testing
    # git annex init
    init  ok
    # cp /etc/passwd my-cool-big-file
    # git annex add my-cool-big-file
    add my-cool-big-file ok
    (Recording state in git...)
    # git commit -m added
    [master d36c8b4] added
     1 files changed, 1 insertions(+), 0 deletions(-)
     create mode 120000 my-cool-big-file
    # git push --all
    Counting objects: 17, done.
    Delta compression using up to 2 threads.
    Compressing objects: 100% (12/12), done.
    Writing objects: 100% (14/14), 1.39 KiB, done.
    Total 14 (delta 0), reused 1 (delta 0)
    To gitolite@localhost:testing
       c552a38..db4653e  git-annex -> git-annex
       29cd204..d36c8b4  master -> master
    # git annex copy --to origin
    copy my-cool-big-file (checking origin...) (to origin...)
    WORM-s2502-m1318875140--my-cool-big-file
            2502 100%    0.00kB/s    0:00:00 (xfer#1, to-check=0/1)

    sent 2606 bytes  received 31 bytes  1758.00 bytes/sec
    total size is 2502  speedup is 0.95
    ok

### Troubleshooting

I got an error like this when setting up gitolite after setting up a local git repo and git annex:

    git-annex-shell: First run: git-annex init
    Command ssh ["git@git.example.com","git-annex-shell 'configlist' '/~/myrepo.git'"] failed; exit code 1

because I forgot to "git push --all" after adding the new gitolite remote.

comment 9
Ah right. git-annex normalizes all git ssh style user@host:dir to valid uris, which is where the `/~/` comes from. I don't anticipate this changing on the git-annex side.
Comment by http://joey.kitenet.net/ — Tue Jul 22 18:11:47 2025
afaict git annex normalizes urls on the client side.

After some debugging printing, here is my current understanding.

- urls of the form git@host:~repo or ssh://git@host

  - git sends commands like "git-receive-pack '~/repo'
  - gitolite converts these to $REPO_BASE/~/repo which fails. ~/repo would also fail fwiw.
  - git-annex sends seems /~/repo, which works

- urls of the form git@host:/repo or ssh://git@host/repo

  - git sends "git-receive-pack '/db/cs3383'"
  - gitolite converts this to $REPO_BASE/repo which works
  - git annex sends "git-annex-shell 'inannex' '/repo' ..." which works, but only with the patch above.

- urls of the form git@host:repo

  - git sends "git-receive-pack 'repo'
  - gitolite converts this to $REPO_BASE/repo, which works
  - git-annex sends "git-annex-shell 'inannex' '/~/db/cs3383'...", which also works for git-annex-shell.

So the weird case is the last one where git and git-annex are sending different things over the wire. I don't know if you have other motivations for doing the url normalization on the client side, but it isn't needed for gitolite, and in some sense complicates things a little. On the other hand, now that I see what is going on, it isn't a big deal to just strip the leading /~ off in the adc. It does lead to the odd situation of some URLs working for git-annex but not git.

Comment by bremner — Tue Jul 22 18:11:47 2025
comment 7

That patch seems ok, it doesn't seem to allow through any repo locations that were blocked before.

So, it has my blessing.. but the ADC is in gitolite and will need to be patched there.

Comment by http://joey.kitenet.net/ — Tue Jul 22 18:11:47 2025
ssh://gitolite-host/repo-name is supposed to work

I confirmed with Sitaram that this is intentional, if probably under-documented. Since the ADC strips the leading /~/ in assigning $start anyway, I guess something like the following will work

    diff --git a/contrib/adc/git-annex-shell b/contrib/adc/git-annex-shell
    index 7f9f5b8..523dfed 100755
    --- a/contrib/adc/git-annex-shell
    +++ b/contrib/adc/git-annex-shell
    @@ -28,7 +28,7 @@ my $cmd=$ENV{SSH_ORIGINAL_COMMAND};
     # the second parameter.
     # Further parameters are not validated here (see below).
     die "bad git-annex-shell command: $cmd"
    -    unless $cmd =~ m#^(git-annex-shell '\w+' ')/\~/([0-9a-zA-Z][0-9a-zA-Z._\@/+-
    +    unless $cmd =~ m#^(git-annex-shell '\w+' ')/(?:\~\/)?([0-9a-zA-Z][0-9a-zA-Z.
     my $start = $1;
     my $repo = $2;
     my $end = $3;

Comment by bremner — Tue Jul 22 18:11:47 2025
gitolite gets different paths for different urls

I guess there is some path rewriting going in in gitolite proper because if try a url of the form ssh://git@localhost/testing, then it still works with gitolite, but fails with the ADC because the repo is passed as /testing:

    Running: ssh ["git@host","git-annex-shell 'configlist' '/recommend'"]
    Running: ssh ["git@host","git-annex-shell 'configlist' '/recommend'"]

What I have to ask Sitaram and or find in the docs is if this is a bug or a feature in gitolite. I can see how the leading slash would get swallowed up by this line

    $repo = "'$REPO_BASE/$repo.git'"

in gl-auth-command, but I guess that isn't the whole story.

Comment by bremner — Tue Jul 22 18:11:47 2025
comment 4

Well a repo url like `gitolite@localhost:testing` puts it in the gitolite user's /~/testing

This worked when I added the gitolite stuff, anyway.. Let's see if it still does:

    joey@gnu:~/tmp>mkdir g
    joey@gnu:~/tmp>cd g
    joey@gnu:~/tmp/g>git init
    Initialized empty Git repository in /home/joey/tmp/g/.git/
    joey@gnu:~/tmp/g>git annex init
    init  ok
    joey@gnu:~/tmp/g>git remote add test 'gitolite@localhost:testing'
    joey@gnu:~/tmp/g>touch foo
    joey@gnu:~/tmp/g>git annex add foo
    add foo (checksum...) ok
    (Recording state in git...)
    joey@gnu:~/tmp/g>git annex copy foo --to test --debug
    git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","ls-files","--cached","-z","--","foo"]
    git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","check-attr","annex.numcopies","-z","--stdin"]
    git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","show-ref","--hash","refs/heads/git-annex"]
    git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","show-ref","git-annex"]
    git ["--git-dir=/home/joey/tmp/g/.git","--work-tree=/home/joey/tmp/g","cat-file","--batch"]
    Running: ssh ["-4","gitolite@localhost","git-annex-shell 'configlist' '/~/testing'"]

Still seems right, the ADC's regexp will match this the git-annex shell command.

Comment by http://joey.kitenet.net/ — Tue Jul 22 18:11:47 2025
repo name conventions?
I'm confused by the fact that the git-annex-shell adc rejects any repo names that don't start with /~/ since none of my repos start that way. It seems work ok if I just delete /\~ from the front of the regex, but I feel like I must be missing something.
Comment by bremner — Tue Jul 22 18:11:47 2025
Where does gitolite stores the annex objects contents?
Hi. Does anyone knows where does gitolite stores the annex objects' contents? (I got it to work, and git annex is working fine, but I want to "see the annex files" in the central gitolite server. And it puzzles me I can't find them! It definitively is not in the repo.git bare repository under the gitolite admin repositores folder)
Comment by davicastro — Tue Jul 22 18:11:47 2025
Using gitolite 3.6.6 mirror not working with annex

I find out that I need to add the following lines to the gitolite.rc in the server side.

'git-annex-shell ua'

The signal repository works with gitolite as expected. However, the mirroring feature is not working for slaves. When I do

git annex copy --to origin

The master server store the annex file correctly. The file managed by annex is not syncing to the slave mirrors at all.

Comment by git-annex — Tue Jul 22 18:11:47 2025
comment 23 70dcb7e7ffdd14351adaf4c40ee7fdd0
[[!comment Error: unsupported page format hs]]
— Tue Jul 22 18:11:47 2025
Using Gitolite 3.6.1

Hi!

I've made a fresh install of Gitolite 3.6.1 and got this error when trying to sync :

FATAL: bad git-annex-shell command: git-annex-shell 'configlist' '/nicolas.git' at /home/git/bin/commands/git-annex-shell line 25, line 1.

Comment by https://www.google.com/accounts/o8/id?id=AItOawk3gWkz5Qgy5qEa-myg8KWtiTrHSKW1x7Q — Tue Jul 22 18:11:47 2025
Worked for me, updated wiki
I tried the `master` branch of gitolite today on my server, and it works fine! Updated the wiki with the commands I used to setup gitolite.
Comment by https://www.google.com/accounts/o8/id?id=AItOawn26WQjIP5fnMgQF_L_k3Q3UrR5v8mjRTY — Tue Jul 22 18:11:47 2025
comment 20

@Adrian, that's good news. It would be helpful if someone could update the top of this page to document how to use git-annex with the new version of gitolite. This is a wiki..

You're not supposed to need to use initremote when dealing with normal git remotes. It seems that something got lost that automatically initialize the remote repository in this situation. I've fixed it so that it will be set up when `git-annex-shell configlist` is run.

(@wayne, this fixes the problem you reported too..)

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 2
I've fixed the typo (anyone can edit pages in this wiki FWIW.)
Comment by http://joey.kitenet.net/ — Tue Jul 22 18:11:47 2025
How can the git annex init command be called on the server?

The latest commit in the gitolite repository "git-annex support, finally in master!" looks really promissing. I'm currently using ubuntu trusty with updated gitolite3 package and the configuration provided by Khaije. One line needs to be changed: 'git-annex-shell' => 'ua', instead of 'git-annex-shell' =>1,

However, one little detail is still open for me. I need to call the server side "git annex init" on the server (sudo su gitolite3; cd ~/repositories/testing.git && git annex init). I cannot find a way to initialize the server from client side. E.g. git annex init && git push --all is not enough. The man page describes git annex initremote for other server types but not for gitolite remotes. Wouldn't we need something similar for gitolite as well? Or is there a better solution which I do not recognize?

Comment by https://www.google.com/accounts/o8/id?id=AItOawnftLWVOF0DUdXr2HSW4IKzuqtW8V7X3YU — Tue Jul 22 18:11:47 2025
Using Gitolite 3.2

Adding:

'git-annex-shell' =>1,

To the .gitolite.rc file resulted in the "FATAL: suspicous characters loitering about 'git-annex-shell 'configlist' '/~/testing''...

Gitolite source code (https://github.com/sitaramc/gitolite/commit/b1d3c0571409b7c6279fc6a77253c3bc262ab425#diff-79a3701e9e2cee0ea1316451c21a3fec) requires this entry:

'git-annex-shell ua'

Comment by https://www.google.com/accounts/o8/id?id=AItOawll4Kgp7nMuOKdB0FfbcYZ3KRq7HCS0Slc — Tue Jul 22 18:11:47 2025
comment 17

my gitolite.rc is available at https://gist.github.com/khaije1/7609848

For whatever reason I've found this to be very simple to get working so I'd guess there's a missing ingredient somewhere. The combination of gitolite and git-annex is valuable to me so I'll add documents to the url above in hopes it will assist some people with getting the same value.

Comment by https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M — Tue Jul 22 18:11:47 2025
Any further info regarding gitolite support?

We have reached the same point as the previous poster from 25 days ago. $ git annex copy --to origin FATAL: suspicious characters loitering about 'git-annex-shell 'configlist' '/~/testing''

Remote origin does not have git-annex installed; setting remote.origin.annex-ignore git-annex: cannot determine uuid for origin

Anyone actually have this working?

Comment by https://www.google.com/accounts/o8/id?id=AItOawmbo-yMzW_lkBrS4ICNn5XMr8saYtI1_WY — Tue Jul 22 18:11:47 2025
git-annex with gitolite-rc

@François

The proper array in .gitolite.rc seems to be the "ENABLE" array, which it appears is parsed into the COMMANDS array in src/lib/Gitolite/Rc.pm

@Khaije

Using 'git-annex-shell' => 'ua' doesn't seem to work for me. The program still fails in src/gitolite-shell around line 163 (gitolite repo version b1d3c05):

    _die "suspicious characters loitering about '$soc'"
      if $rc{COMMANDS}{ words[0] } ne 'ua' and $soc !~ $REMOTE_COMMAND_PATT;

When I insert $rc{COMMANDS}{ words[0] } into the _die message, it shows up as "1" instead of "ua" as I was expecting.

When I manually set $rc{COMMANDS}{ words[0] } to 'ua' slightly earlier in the script, the git-annex-shell command gets run but it seems to fail to parse the result of the configlist command properly because then I get

    Failed to get annex.uuid configuration of repository origin

    Instead, got: "annex.uuid=\ncore.gcrypt=\n"

I am actually about to give up on the notion of using git-annex and gitolite together. Maybe. I am interested to know if anyone else is having similar problems.

Comment by https://www.google.com/accounts/o8/id?id=AItOawnRaueN1AcM8pIMofH5-wQ1Kr4T0GBe8sA — Tue Jul 22 18:11:47 2025
comment 14

@khaije

Could you paste your config file? Here is mine: http://paste.debian.net/44856/ I don't have any COMMANDS array. Could you elaborate your modifications please?

Thanks.

Comment by https://www.google.com/accounts/o8/id?id=AItOawkSbvo_NbY-ev1VKtzwo7nEqUmvRO6rXGA — Tue Jul 22 18:11:47 2025
git-annex with gitolite FTW

The steps to activate git-annex integration have changed/simplified for v3.

1) during install, be sure to use the 'git-annex' branch, rather than master[fn:1].

2) to enable git-annex-shell, open ~/.gitolite.rc and insert 'git-annex-shell' => 'ua' into the hash list in the COMMANDS array.[fn:2]

# 'git-annex-shell' => 'ua',

[fn:1] We'd like to have this feature-branch merged to master, so please send Sitaram feedback, positive and negative, based on your experiences. [fn:2] There is no GL_ADC_PATH and no "ua" subdirectory here, and nothing to "install"; the command now comes with gitolite.

Comment by https://www.google.com/accounts/o8/id?id=AItOawm9tgeFE5v-arAYYftSv3yUTI5Q4qB2C9M — Tue Jul 22 18:11:47 2025
comment 12
latest code of gitolite does not support git-annex ? I could not find a way to make it work ?
Comment by https://www.google.com/accounts/o8/id?id=AItOawn5RcmefXjrl1vmbHIiOWQyXGXVKxlm3rg — Tue Jul 22 18:11:47 2025
Problems with URL ending with ".git"

Hi,

I noticed using the git-annex branch of gitolite v3 that the same URL with ".git" at the end would not work in git-annex. For example my test repository was `git@git2.mildred.fr:u/mildred/Annex.git` but it didn't work until I converted it to `git@git2.mildred.fr:u/mildred/Annex`

On the server, the repository is in `repositories/u/mildred/Annex.git`

If I try a copy with git-annex for example, I would get:

    $ git annex copy titi --to test
    copy titi (checking test...) FATAL: u/mildred/Annex.git mildred DENIED

    (unable to check test) failed
    git-annex: copy: 1 failed

(test is the name of my remote and titi is my file)

Note, in my gitolite conf, I have:

    repo u/CREATOR/[a-zA-Z0-9].*
      C    = @all
      RW+D = CREATOR
      RW   = WRITERS
      R    = READERS

Comment by http://mildred.fr/ — Tue Jul 22 18:11:47 2025
git-annex no longer supported by gitolite g3
See http://gitolite.com/gitolite/dev-status.html for some details.
Comment by http://ertai.myopenid.com/ — Tue Jul 22 18:11:47 2025
comment 1

Looks like you are missing a closing double quote on the line:

echo '$GL_ADC_PATH = "/usr/local/lib/gitolite/adc/;' >>~gitolite/.gitolite.rc

right after /;

I got this working by the way - great stuff.

Comment by http://www.openid.albertlash.com/openid/ — Tue Jul 22 18:11:47 2025
Comments on this page are closed.