git-annex/ tips/ automatically adding metadata

git-annex's metadata works best when files have a lot of useful metadata attached to them.

To make git-annex automatically set the year and month when adding files, run: `git config annex.genmetadata true`

## git commit hook

A git commit hook can be set up to extract lots of metadata from files like photos, mp3s, etc. Whenever annexed files are committed, their metadata will be extracted and stored.

Download pre-commit-annex and install it in your git-annex repository as `.git/hooks/pre-commit-annex`
Remember to make the script executable! `chmod +x .git/hooks/pre-commit-annex`

### using extract

The git commit hook can use extract to get metadata.

Install it from http://www.gnu.org/software/libextractor/
`apt-get install extract`

Configure which metadata fields to ask extract for: `git config metadata.extract "artist album title camera_make video_dimensions"`

To get a list of all possible fields, run: `extract -L | sed 's/ /_/g'`

### using exiftool

The git commit hook can also use exiftool to get metadata.

Install it from http://owl.phy.queensu.ca/~phil/exiftool/
`apt-get install libimage-exiftool-perl`

Configure which metadata fields to ask exiftool for: `git config metadata.exiftool "Model ImageSize FocusRange GPSAltitude GPSCoordinates"`

To get a list of all possible fields, run: `exiftool -list`

### using both extract and exiftool

If you want some metadata that extract knows about, and other metadata that exiftool knows about, just install them both, and set both `metadata.extract` and `metadata.exiftool`.

### overwriting existing metadata

By default, if a git-annex already has a metadata field for a file, its value will not be overwritten with metadata taken from files. To allow overwriting, run: `git config metadata.overwrite true`

comment 9

@woffs hmm, the hook script seems to quote every use of the filename, which should avoid such problems. I tested it, using both extract and exiftool, and a file named "foo'bar.jpg", and it worked fine.

If you have a case where it does not work, suggest you file a bug report with enough information to reproduce the problem.

Comment by joey — Tue Jul 22 18:11:47 2025
hook and quoting

Hi,

just detected that the provided pre-commit-annex script is broken for filenames containing a '

Comment by woffs — Tue Jul 22 18:11:47 2025
pre-commit is OK on windows now - auto adding last mod datetime

@Joey just tested a nightly build and now pre-commit-annex is called, and with my modifications it autoadds last modified times for content. Trivially it's just the matter of adding:

    field="datemod"
    value=$(stat -c %Y $f)
    addmeta "$f" "$field" "$value"

to the body of the process() function to the supplied pre-commit-annex script. thanks

Comment by https://www.google.com/accounts/o8/id?id=AItOawnW_CrhP9p50n9UUhTg_a9glyKWSvnrjRQ — Tue Jul 22 18:11:47 2025
comment 6

@Michele after testing, git-annex on Windows seems to not see a file that does have the executable bit set as executable. I have opened a bug report windows isExecutable fail, and worked around the problem now.

Comment by joey — Tue Jul 22 18:11:47 2025
comment 5

@Michele git annex sync in a direct mode repository does bypass the pre-commit hook. However, it will try to run the pre-commit-annex hook.

Most likely, the hook script does not appear executable on Windows, so git-annex cannot run it.

Comment by joey — Tue Jul 22 18:11:47 2025
direct mode pre-commit hooks [on windows]

seemingly pre-commit hooks are not being called on windows, it could have to do with git annex sync bypassing them when doing commits ?

on the other side genmetadata works. although that is not enough for me since I'd want to preserve complete last modification date/time and I was in the process of modifying the supplied pre-commit script to call for "stat %Y" (which btw is working fine on windows, while the last binaries for extract are failing there).

am I correct in assuming that direct mode [on windows at least] bypasses hooks [namely pre-commit as well as pre-commit-annex] ?

Comment by https://www.google.com/accounts/o8/id?id=AItOawnW_CrhP9p50n9UUhTg_a9glyKWSvnrjRQ — Tue Jul 22 18:11:47 2025
comment 3

@anarcat, I have modified pre-commit-annex so if it's passed already annexed files, it'll extract their metadata.

So this can be used to add metadata to files added before you installed the hook, or if you've configured more fields to be extracted.

Comment by http://joeyh.name/ — Tue Jul 22 18:11:47 2025
comment 2

is there a way for this to be done globally, without having to install and configure the hook for each repository? it seems like a fairly useful feature that could be factored in git-annex itself (as opposed to be shipped as a shell script)...

also, is there a way to retroactively parse the tags from existing files (as opposed to only new files added to the repo).

thanks

Comment by https://id.koumbit.net/anarcat — Tue Jul 22 18:11:47 2025
Re: Automatically adding metadata can be very slow

You could add a config to the script that skips over files larger than a certian size.

Or for that matter, the script could be adapted to filter the files to only include images/videos, using eg:

    git annex find --mimetype='image/*' --or --mimetype='video/*'

Should be a fairly easy change, patches accepted.

Comment by joey — Tue Jul 22 18:11:47 2025
Automatically adding metadata can be very slow

Running extract on very large files (system backups) can be too long (killed it after running several hours). In general `extract` seem slow on tar.gz archives.

I added `timeout 100s` before the tool is called in the pre commit script:

`LC_ALL=C timeout 100s $tool_exec "./$f" | ...`

This allows to have the commit to complete in reasonable time, probably loosing some metadata.

Comment by aurelf — Tue Jul 22 18:11:47 2025
re: adding support for additional metadata tools?

@max, notice that the hook script contains special case handling for exiftool, including a config option for it. That was contributed by Klaus Ethgen. I'd be inclined to merge patches that add handling for other tools.

I imagine you could add config settings for the format string etc.

Comment by joey — Tue Jul 22 18:11:47 2025
adding support for additional metadata tools?

hi there, sincere gratitude for your work on git-annex.

is it worth considering a more modular setup (maybe a python script :) or even haskell) for working with metadata extractors?

currently it really only supports field-based filtering extractors. i am particularly interested in using (minimally):

mp3info - offers a formatting string for printing but doesn't fit super nicely into the "want" fields in shell script. could probably be hacked in.

mp4info - does not seem to offer any native parameters for filtering; would likely need some engineering/thought about how to take in all possible fields then filtering off of those.

alternatively i could just write some personal scripts here, but just thought others would find it useful for auto extracting content from mp3/m4a files. extract doesn't seem to perform as well on these as, say, .flac files.

thanks again!

Comment by max — Tue Jul 22 18:11:47 2025
comment 1
http://projects.iq.harvard.edu/fits might be an even better choice than libextractor. We use it in work and its not too bad, but it can be slow to startup due to the JVM.
Comment by https://www.google.com/accounts/o8/id?id=AItOawkSq2FDpK2n66QRUxtqqdbyDuwgbQmUWus — Tue Jul 22 18:11:47 2025
Comments on this page are closed.