git-annex/ tips/ flickrannex

# flickrannex

Hook program for gitannex to use flickr as backend

# Requirements:

    python2

Credit for the flickr api interface goes to: http://stuvel.eu/flickrapi Credit for the png library goes to: https://github.com/drj11/pypng Credit for the png tEXt patch goes to: https://code.google.com/p/pypng/issues/detail?id=65

# Install

Clone the git repository in your home folder.

    git clone git://github.com/TobiasTheViking/flickrannex.git

This should make a ~/flickrannex folder

# Setup

Make the file executable, and link it into PATH

    cd ~/flickrannex; chmod +x git-annex-remote-flickr; sudo ln -sf `pwd`/git-annex-remote-flickr /usr/local/bin/git-annex-remote-flickr

# Commands for gitannex:

    USERNAME="username@provider.com" git annex initremote flickr type=external externaltype=flickr encryption=shared folder=gitannex

An oauth authentication link should now be launched in the default browser. The hook will wait for 30s for you to login and authenticate.

    git annex describe dropbox "the flickr library"

# Notes

## Unencrypted mode

The photo name on flickr is currently the GPGHMACSHA1 version.

## Encrypted mode

The current version base64 encodes all the data, which results in ~35% larger filesize.

## Including directories as tags

This feature is currently disabled, if it gets implemented again it will most likely not require user action to enable it.

In this case the image: /home/me/annex-photos/holidays/2013/Greenland/img001.jpg would get the following tags: "holidays" "2013" "Greenland" (assuming "/home/me/annex-photos" is the top level in the annex...)

Caveat Emptor - Tags will always be NULL for indirect repos - we don't (easily) know the human-readable file name.

chunksize

Hi! Does this backend support chunksize option? If yes, is it possible to set it after the remote has been added to the repository? Thanks, Milan.

Comment by https://www.google.com/accounts/o8/id?id=AItOawleVyKk2kQsB_HgEdS7w1s0BmgRGy1aay0 — Wed Oct 29 14:05:27 2025
comment 8
I'll make it so, in the next version i push.
Comment by https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA — Wed Oct 29 14:05:27 2025
Unencrypted flickr can only accept picture and video files

Thanks and sorry to trouble you, it is my error, I picked unencrypted option (thinking it would be less of an issue) and am using a text file for test, gave an error line:

    10:53:07 [flickrannex-0.1.5] main : 'Unencrypted flickr can only accept picture and video files'

I've not looked through your code yet, but could that message be printed when not in debug mode?

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
comment 6

That's weird...

You could try adding "--dbglevel 1 --stderr" arguments to the hook command and give me the output. But the way i read the log it seems like it doesn't even launch the python intrepreter. I might be wrong though.

Comment by https://www.google.com/accounts/o8/id?id=AItOawmWg4VvDTer9f49Y3z-R0AH16P4d1ygotA — Wed Oct 29 14:05:27 2025
comment 5

Thanks, but on my machine I get:

    $ which python2
    /usr/bin/python2

I have scripted all my walkthrough commands, blowing away the test repositories and flickr settings first each time. This re-runs the flickr scripts and git config annex.flickr-hook etc.

I can't spot anything here.

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
missing configuration for flickr-checkpresent-hook

Many thanks.

I used gitannex-install and was left with a slight anomaly:

    Installing...........done
    git-annex version 4.20130601 has been installed
    $ git-annex version
    git-annex version: 4.20130531-g5df09b5

But I guess this includes the new hook format. I get a bit further:

    $ git annex copy . --to flickr
    copy walkthrough.sh (checking flickr...) (user error (sh ["-c","/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py"] exited 1)) failed
    copy walkthrough.sh~ (checking flickr...) (user error (sh ["-c","/usr/bin/python2 /home/nrb/repos/gits/flickrannex/flickrannex.py"] exited 1)) failed
    git-annex: copy: 2 failed

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
comment 4

The path for the binary "/usr/bin/python2" is wrong.

It could be any of /usr/bin/python /usr/bin/python2.6 /usr/bin/python2.7

Or maybe in /usr/local/bin

you can try running "which python" or "which python2" to get the real path.

Comment by https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8 — Wed Oct 29 14:05:27 2025
missing configuration for flickr-checkpresent-hook

https://github.com/TobiasTheViking/flickrannex/issues/3

9 days ago: [the annex] "hook format a few versions ago, and this is using the new hook format".

Looks very handy. I am just starting with this, but can't seem to get it working as a remote after following the simple walkthrough. All goes well until:

    $ git annex copy . --to flickr
    copy walkthrough.sh (checking flickr...)
      missing configuration for flickr-checkpresent-hook
    git-annex: checkpresent hook misconfigured

my Ubuntu 12.04:

    $ git annex version
    git-annex version: 4.20130516.1
    build flags: Assistant Webapp Pairing Testsuite S3 WebDAV Inotify DBus XMPP
    local repository version: 3
    default repository version: 3
    supported repository versions: 3 4
    upgrade supported from repository versions: 0 1 2

I guess my "git-annex version is still too old"? Any idea what version is needed? Even better if I can figure out which Linux distribution/release has the most up to date version of annex.

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
comment 2

Get the statically linked version from here http://git-annex.branchable.com/install/Linux_standalone/

I believe the new hook format was introduced in version 4.20130521

Comment by https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8 — Wed Oct 29 14:05:27 2025
Dead project, or just working smoothly? Also, which hat color?

Q1: No commit on https://github.com/TobiasTheViking/flickrannex/commits/master since Jan 13, 2014. Is it still working?

Q2: I'm not sure I get it correctly. Is it to use Flickr photo storage so that you can store git-annex content that is actually photos to visible Flickr collections (on web site) and vice versa (a nice white-hat hack)? Or is it just a cuckoo's hack to stuff arbitrary data on Flickr's servers without any content (a grey-zone hack)?

Comment by https://launchpad.net/~stephane-gourichon-lpad — Wed Oct 29 14:05:27 2025
How I got it working

I'm using pipsi to install packages without polluting the system, Michael's fork from magthe's fork from TobiasTheViking repo (I don't know if the other branches are better or worse, but this is working).

## Instructions

First thing: go to https://www.flickr.com/services/apps/create/noncommercial/? and create an app (if you use the keys provided in the flickrannex repository you'll be giving access to your flickr to everybody).

Now follow the steps:

    wget https://github.com/mikemccracken/flickrannex/archive/master.zip
    unzip master.zip
    cd flickrannex-master/
    edit setup.py # replace 'flickrapi' with 'flickrapi==1.4.5'
                  # otherwise it will install a wrong version of the dependency.
    edit flickrannex # on lines 46 and 47, replace the api_key and api_secret
                     # with the values from the application you created in the
                     # first step.
    pipsi install ./
    cd
    flickrannex # go through the oauth setup, will open a window in the browser
                # some commands will appear after you're successful. ignore them.
    cd ~/annex # your git-annex folder
    git config annex.flickr-hook $HOME/.local/bin/flickrannex
    git annex initremote flickr type=hook hooktype=flickr encryption=none

Comment by fiatjaf — Wed Oct 29 14:05:27 2025
a more recent fork
Just a note that I poked around on github and saw a more recent cleaned up version in this fork: https://github.com/magthe/flickrannex/tree/devo
Comment by https://www.google.com/accounts/o8/id?id=AItOawmvzzyDA8uXFz8yokeCrepbh8PwWe_WrjE — Wed Oct 29 14:05:27 2025
current status?

Hi, just wondering what the current status of this plugin is. The repo at TobiasTheViking looks a little odd - it has a few large commits from January with an unknown author, and the last one appears to completely remove the main script.

What's going on? does it need a fork?

Comment by https://www.google.com/accounts/o8/id?id=AItOawmvzzyDA8uXFz8yokeCrepbh8PwWe_WrjE — Wed Oct 29 14:05:27 2025
great job on that plugin!

it's pretty awesome to have 1TB of free storage like that out there... but for storing photos, it could be improved - I filed a few bugs on the github repo here:

https://github.com/TobiasTheViking/flickrannex/issues/created_by/anarcat?state=open

thanks!

Comment by https://id.koumbit.net/anarcat — Wed Oct 29 14:05:27 2025
Version 0.1.10 pushed

Since the initial release of this hook a lot of issues have been fixed, and a few features added.

I would highly suggest that everyone who is using this hook update to the latest version as i would consider one of the bugs to be fairly major.

Comment by https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8 — Wed Oct 29 14:05:27 2025
re: git annex get failed

Another try - this time a slightly simpler setup using my version of the walkthrough commands

[[!format bash """

nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex drop walkthrough.sh --from usbdrive drop usbdrive walkthrough.sh ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex move walkthrough.sh --to flickr move walkthrough.sh (gpg) (checking flickr...) (to flickr...) /home/nrb/repos/gits/flickrannex/flickrannex.py:92: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: /home/nrb/repos/gits/flickrannex/flickrannex.py:100: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead. if res: ok (Recording state in git...) nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex whereis whereis walkthrough.sh (1 copy) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) ok whereis walkthrough.sh~ (3 copies) 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) 7803d853-d231-4bb4-b696-f12a950fb96b -- here (my laptop) d60d75f9-d878-4214-af20-fa055134ae77 -- usbdrive (portable USB drive) ok nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex get walkthrough.sh get walkthrough.sh (from flickr...) (gpg) git-annex: /home/nrb/repos/annex/laptop-annex/.git/annex/tmp/GPGHMACSHA1--02f600d7e8b071d2945270fd5e7fc26dd066ff31: openBinaryFile: does not exist (No such file or directory) gpg: decrypt_message failed: eof

Unable to access these remotes: flickr

Try making some of these repositories available: 161b7af0-2075-4314-9767-308a49b86018 -- flickr (the flickr library) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$ git annex fsck --from flickr fsck walkthrough.sh (gpg) (checking flickr...) (fixing location log) Based on the location log, walkthrough.sh was expected to be present, but its content is missing.

No known copies exist of walkthrough.sh failed fsck walkthrough.sh~ (checking flickr...) (fixing location log) Based on the location log, walkthrough.sh~ ** was expected to be present, but its content is missing. failed (Recording state in git...) git-annex: fsck: 2 failed nrb@nrb-ThinkPad-T61:~/repos/annex/laptop-annex$

""" ]]

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
git annex get failed

Hi, I am coming back to this and testing Flickr as a repository for moving files about and have run into what may be my very basic misunderstanding with vanilla annex.

I copied one file to Flickr and dropped it elsewhere (--force). I assumed that the file was on Flickr ok but that the numcopies setting required the force because of the semi-trust level of the Flickr remote.

Then I find I can't get the file back, even though there is a record of it from whereis.

Can you help enlighten me as to what am I missing? I assumed whereis would only report files that exist and can be copied back. If not my error, I can raise bug or search for logs. Thanks in advance for any help.

[[!format perl """

nrb@nrb-ThinkPad-T61:~/tmp$ git annex whereis whereis libpeerconnection.log (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis test.cgi (1 copy) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) ok whereis walkthrough.sh (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok whereis walkthrough.sh~ (3 copies) 31124688-0792-4214-9e00-7ed115aa6b8e -- flickr (the flickr library) 3e3d40d7-de8f-4591-a4ab-747d74a3b278 -- origin (my laptop) ec2d64fc-30d6-48b4-99bf-7b1bc22d420d -- portable USB drive ok nrb@nrb-ThinkPad-T61:~/tmp$ git annex get test.cgi get test.cgi (from flickr...)

git-annex: /home/nrb/tmp/.git/annex/tmp/SHA256E-s48--a01eedbee949120aeda41e566f9ae8faef1c2bacaa6d7bb8e45050fb8df6d09d.cgi: rename: does not exist (No such file or directory) failed git-annex: get: 1 failed nrb@nrb-ThinkPad-T61:~/tmp$

"""]]

Comment by https://www.google.com/accounts/o8/id?id=AItOawnaH44G3QbxBAYyDwy0PbvL0ls60XoaR3Y — Wed Oct 29 14:05:27 2025
comment 10

I'm not even sure if chunksize is exposed to the hooks at all.

As it is, the hook will check the filesize, and if the filesize is more than 30mbyte it will exit 1.

Chunking may be implemented down the road. I do believe joeyh might have some plans that will touch this issue, so I'd rather wait. Than re-invent the wheel yet again.

Comment by https://www.google.com/accounts/o8/id?id=AItOawmkBwMWvNKZZCge_YqobCSILPMeK6xbFw8 — Wed Oct 29 14:05:27 2025
Comments on this page are closed.