Find duplicate files in a git-annex repository

git-annex

Find a file

Mike Gerber 77589ebda4 📝 README: Contributing		2026-09-03 09:48:38 +02:00
.gitignore	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
.python-version	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
default.nix	⬆️ Nix: Bump rev	2026-09-02 18:09:43 +02:00
git_annex_duplicates.py	🎨 Rename main file to git_annex_duplicates.py	2026-09-02 18:08:10 +02:00
LICENSE.txt	📝 Add a license	2026-09-03 09:42:10 +02:00
pyproject.toml	📝 Add a license	2026-09-03 09:42:10 +02:00
README.md	📝 README: Contributing	2026-09-03 09:48:38 +02:00
uv.lock	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00

README.md

git-annex-duplicates

Find duplicate files in a git-annex repository.

git-annex-duplicates groups files by their annex key and prints every group that contains more than one path. It is read-only: it reports duplicates but does not remove or otherwise modify any files.

Requirements

git-annex
Python 3.11 or newer
uv (recommended for installation)

Installation

Install the command directly from the Git repository:

uv tool install "git+https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git"

Alternatively, run it from a checkout:

git clone https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git
cd /path/to/your-git-annex-repo
uv run --project /path/to/git-annex-duplicates git-annex-duplicates

Usage

Run the command from inside a git-annex repository:

git-annex-duplicates

This will list any duplicate files below the current working directory. The output contains one group of duplicate paths at a time, with groups separated by a blank line:

photos/holiday.jpg
backup/holiday.jpg

documents/report.pdf
archive/report-final.pdf

Exclude one or more path patterns with --exclude:

git-annex-duplicates --exclude '*.iso' --exclude 'archive/**'

Additional arguments are passed to git annex find. Place -- before options that belong to git annex find, so they are not interpreted as options to git-annex-duplicates:

# Only report duplicates whose content is present in this repository
git-annex-duplicates -- --in=here

# Combine a local exclusion with a git-annex find option
git-annex-duplicates --exclude 'archive/**' -- --in=here

See git annex find --help for the available filters.

How it works

The command runs git annex find, requests each file's annex key, groups the returned paths by that key, and prints groups containing at least two paths. Files with the same annex key refer to the same content as far as git-annex is concerned.

Only annexed files are considered. Regular Git files and untracked files are not included, and file content does not need to be downloaded to produce the report.

Contributing

This program is almost trivial, so I don't expect many contributions. However, if you wish to implement a new feature, please open an issue at GitHub first, so we can discuss it.