git-annex/ tips/ centralized git repository tutorial/ on your own server

This tutorial shows how to set up a centralized git repository hosted on your own git server, which can be any unix system with ssh and git and git-annex installed. A VPS, a home server, etc.

This sets up a very simple git server. More complex setups are possible. See for example using gitolite with git-annex.

## set up the server

On the server, you'll want to install git, and git-annex, if you haven't already. If possible, install it using your distribution's package manager:

    server# sudo apt-get install git git-annex

Note that git-annex-shell needs to be located somewhere in the PATH, so that a client can successfully run "ssh yourserver git-annex-shell". Installing git-annex using a package manager will take care of this for you. But if you're not root or otherwise can't install git-annex that way, you may need to do more work; see get git-annex-shell into PATH.

Decide where to put the repository on the server, and create a bare git repo there. In your home directory is a simple choice:

    server# cd
    server# git init annex.git --bare --shared
    server# cd annex.git
    server# git annex init
    init ok

That's the server setup done!

## make a checkout

Now on your laptop, clone the git repository from the server:

    laptop# git clone ssh://example.com/~/annex.git
    Cloning into 'annex'...
    warning: You appear to have cloned an empty repository.
    Checking connectivity... done.

Tell git-annex to use the repository:

    laptop# cd annex
    laptop# git annex init
    init ok

## add files to the repository

Add some files, obtained however.

    # git annex add *.mp4
    add Haskell_Amuse_Bouche-b9OVqxmI.mp4 (checksum) ok
    (Recording state in git...)
    # git commit -m "added a video. I have not watched it yet but it sounds interesting"

Feel free to rename the files, etc, using normal git commands:

    # git mv Haskell_Amuse_Bouche-b9OVqxmI.mp4 Haskell_Amuse_Bouche.mp4
    # git commit -m 'better filenames'

Now push your changes back to the central repository on your server. As well as pushing the master branch, remember to push the git-annex branch, which is used to track the file contents.

    # git push origin master git-annex
    To git@github.com:joeyh/techtalks.git
     * [new branch]      master -> master
     * [new branch]      git-annex -> git-annex

That push went fast, because it didn't upload large videos to the server.

So, to finish up, tell git-annex to sync all the data in the repository to your server:

    # git annex sync origin --content
    ...

## make more checkouts

So far you have a central repository on your server, and a checkout on a laptop. Let's make another checkout elsewhere. Clone the central repository as before.

    elsewhere# git clone ssh://example.com/~/annex.git
    elsewhere# cd annex

Notice that your clone does not have the contents of any of the files yet. If you run `ls`, you'll see broken symlinks. It's easy to download them from your server either by running `git annex sync --content`, or by asking git-annex to download individual files:

    # git annex get Haskell_Amuse_Bouche.mp4
    get Haskell_Amuse_Bouche.mp4 (from origin...)
    12877824   2%  255.11kB/s    00:00
    ok

centralized repository workflow

Hi, what would be a recommended setup and the working procedures for the following scenario:

- using git-annex version: 8.20210223, which is the one in ubuntu-22.04 (can't upgrade easily)
- a central server as a mutable central archive, many users (over ssh)
- users are all trusted
- the server shall keep all annexed files, but only the HEAD version is relevant, that is: if the file is removed by the user, it shall eventually be permanently removed from the central server too, to save space.
- users would tipically not need all the files, but only some, so `git annex get files...` would do
- users would also add or remove annexed files (and push them to the central repository)
- a user might remove his/her local repository at any time, so the central server shall not keep track about clones or at least shall not care if any or all clones get removed

I have created central repository like this (please correct me):

    git init test --bare
    cd test
    git annex init
    git annex required . "include=*"

On the user site

    git clone ssh://some.server/repo/test test
    cd test
    dd if=/dev/random of=./bigfile bs=1M count=10
    git annex add bigfile
    # how to sync (push only)?
    # how to permanently remove big file?
    cd ..
    # done with the task
    chmod -R 777 test
    rm -rf test

What I am looking for is the sequence of commands for the users, to:

- sync to the latest state (without fetching the content)
- add new annexed file to the repository and push it
- permanently delete annexed file

There are several issues I am facing at the moment. I was expecting to push the new file with `git annex sync --content --no-pull`, but this command also pulls the contents of all annexd files, which I don't want. The server does not want to remove the old content. It looks like I am doing something wrong. Appreciate your suggestions about this scenario.

Comment by zoran.bosnjak — Wed Oct 29 14:05:27 2025
re: problems with init after clone

@jrollins Well, it looks like you have maybe not installed git-annex on your server. If it is installed and you still have this problem, you may need to consult get git-annex-shell into PATH.

Comment by joey — Wed Oct 29 14:05:27 2025
problems with init after clone

I'm trying to following the instructions here, but keep running into the following problem whenever I try to clone the remote repo:

    0$ git clone ssh://remote.example.com/~/tmp/archive/git
    Cloning into 'git'... warning: You appear to have cloned an empty repository.
    0$ cd git/
    0$ git annex init local
    init servo (scanning for unlocked files...)

      Unable to parse git config from origin

      Remote origin does not have git-annex installed; setting annex-ignore

      This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git annex enableremote origin
    ok
    (recording state in git...)
    0$

What am I doing wrong? The remote repo is setup bare, exactly as suggested.

Comment by jrollins — Wed Oct 29 14:05:27 2025
comment 6

I must add that after the previous commands finished, content of the repository was not shown on the server (it was all in .git). I made

    git checkout synced/master

and files appeared. Unfortunately it seems all timestamps were new, which I didn't like, and it was already asked here (http://git-annex.branchable.com/todo/does_not_preserve_timestamps/).

Comment by metst13 — Wed Oct 29 14:05:27 2025
clone to the server

As usually, create a directory on the server, `git init`, then `git-annex init` there.

Add that locally: `git remote add my-server-name my-server:~/my-repo`

`git-annex sync` locally seems to work fine and pushes data to the server.

I needed to have this workaround before, because I could not get data from my laptop while on the server (I wasn't sure I had an open IP address for my laptop). This is mostly a basic thing in git, but I had errors with git-annex earlier and I try to be cautious now.

Comment by metst13 — Wed Oct 29 14:05:27 2025
tutorial up to date?

When I follow the tutorial and run

    git push origin master git-annex

it replies

    error: src refspec master does not match any.
    error: failed to push some refs to 'vds:~/test.git'

However one can use `git-annex sync` instead and that works fine.

Comment by ynikitenko — Wed Oct 29 14:05:27 2025
Is it essential to clone from the server?

    Now on your laptop, clone the git repository from the server:

Is it possible to init git-annex repo on your local machine, add git-remote there and push local data to the server? Won't there be any problems with this 'non-official' approach?

Comment by ynikitenko — Wed Oct 29 14:05:27 2025
comment 2

git-annex sets up its own ssh connection caching because this makes it a lot faster.

To disable this feature, you can set annex.sshcaching=false, or set remote.origin.annex-ssh-options as you have.

git-annex has no way to know if you have another ssh socket to use, so it seems fine for you to need to configure it if you want it to use one.

Comment by joey — Wed Oct 29 14:05:27 2025
git annex using the "wrong" ssh socket

To avoid frequent typing of pin + RSA passcode + password, we typically establish an ssh control master just once. This works fine with regular git commands, but the git-annex command apparently try to create a different socket. Even that would be ok, except that apparently it is a new socket each time we enter a command.

With sufficient "-vvvv" we see things like:

    ...
    debug1: Executing proxy command: exec ssh -l fred proxy.xxx.yyy direct host
    ...

(Note I have eliminated references to the actual machine names and userid's.)

If the command had instead been:

     exec ssh -l fred proxy.xxx.yyy direct /home/fred/.ssh/master_host:22

everything would have worked fine. In fact, we are now using:

     git config remote.origin.annex-ssh-options '-S /home/fred/.ssh/master_host:22'

and this eliminates the issue. But it would be nice if git annex could somehow automatically use the pre-existing connection. Is there a better way to achieve this?

Comment by tom_clune — Wed Oct 29 14:05:27 2025
Comments on this page are closed.