git-annex/ design/ external special remote protocol/ async appendix

This is an appendix to the external special remote protocol.

## introduction

Normally, an external special remote can only be used to do one thing at a time, and when git-annex has concurrency enabled, it will start up multiple processes for the same external special remote.

The `ASYNC` extension lets a single external special remote process handle multiple concurrent jobs, which can be useful if multiple processes would use too many resources, or if it can be better coordinated using a single process.

## protocol overview

As usual, the protocol starts by the external special remote sending the version of the protocol it's using.

    VERSION 1

This extension is negotiated by git-annex sending an `EXTENSIONS` message that includes `ASYNC`, and the external special remote responding in kind.

    EXTENSIONS INFO ASYNC
    EXTENSIONS ASYNC

(Older versions of git-annex will not include `ASYNC` in their extensions list. To support them, it's a good idea for the external special remote to fall back to using the regular protocol.)

Once the extension is negotiated, messages in the protocol are tagged with a job number, by prefixing them with "J n".

As usual, the first message git-annex sends is generally PREPARE, which gets tagged with a job number:

    J 1 PREPARE

Rather than just responding PREPARE-SUCCESS, the job number has to be included in the reply:

    J 1 PREPARE-SUCCESS

Suppose git-annex wants to make some transfers. It can request several at the same time, using different job numbers:

    J 1 TRANSFER RETRIEVE Key1 file1
    J 2 TRANSFER RETRIEVE Key2 file2

The special remote can now perform both transfers at the same time. If it sends PROGRESS messages for these transfers, they have to be tagged with the job numbers:

    J 1 PROGRESS 10
    J 2 PROGRESS 500
    J 1 PROGRESS 20

The special remote can also send messages that query git-annex for some information. These messages and the reply will also be tagged with a job number.

    J 1 GETCONFIG url
    J 3 RETRIEVE Key3 file3
    J 1 VALUE http://example.com/

One transfers are done, the special remote sends `TRANSFER-SUCCESS` tagged with the job number.

    J 2 TRANSFER-SUCCESS RETRIEVE Key2
    J 1 PROGRESS 100
    J 1 TRANSFER-SUCCESS RETRIEVE Key1

Lots of different jobs can be requested at the same time.

    J 4 CHECKPRESENT Key3
    J 5 CHECKPRESENT Key4
    J 6 REMOVE Key5
    J 4 CHECKPRESENT-SUCCESS Key3
    J 6 REMOVE-SUCCESS Key5
    J 5 CHECKPRESENT-FAILURE Key4

## notes

There will be one job number for each thread that git-annex runs concurrently, so around the same number as the -J value, although in some cases git-annex does more concurrent operations than the -J value.

`PREPARE` is sent only once per run of a special remote program, and despite being tagged with a job number, it should prepare the special remote to run that and any other jobs.

`ERROR` should not be tagged with a job number if either git-annex or the special remote needs to send it.

`VERSION`, `EXTENSIONS` and `ERROR` are the only protocol messages that do not get tagged with a job number.

comment 2

The first process that negotiates support for ASYNC will be the last process that git-annex starts for that external special remote. So typically it will start one process, that negotiates support for ASYNC. Then that single process will do all the work.

Comment by joey — Tue Jul 22 18:11:47 2025
is -J shared between workers and ASYNC or "multipliticative"?
question: is ASYNC and parallel workers in annex share the `-J` somehow? I.e. if I start `annex get -J3 .` it would create three workers and then possibly talk to special remote(s). Special remotes then might announce support of ASYNC and accept multiple jobs. Would it be 3 active jobs across all workers+async or it would be up to 3*3=9 active jobs if every worker supports ASYNC?
Comment by yarikoptic — Tue Jul 22 18:11:47 2025
Comments on this page are closed.