Find duplicate files in a git-annex repository

git-annex

Find a file

Mike Gerber 1977d87965 📝 Add a README		2026-09-02 19:41:38 +02:00
.gitignore	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
.python-version	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
default.nix	⬆️ Nix: Bump rev	2026-09-02 18:09:43 +02:00
git_annex_duplicates.py	🎨 Rename main file to git_annex_duplicates.py	2026-09-02 18:08:10 +02:00
pyproject.toml	🐛 pyproject: Correct description	2026-09-02 19:26:54 +02:00
README.md	📝 Add a README	2026-09-02 19:41:38 +02:00
uv.lock	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00

README.md

git-annex-duplicates

Find duplicate file paths in a git-annex repository.

git-annex-duplicates groups files by their annex key and prints every group that contains more than one path. It is read-only: it reports duplicates but does not remove or otherwise modify any files.

Requirements

git-annex
Python 3.11 or newer
uv (recommended for installation)

Installation

Install the command directly from the Git repository:

uv tool install "git+https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git"

Alternatively, run it from a checkout:

git clone https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git
cd git-annex-duplicates
uv run git-annex-duplicates --help

Usage

Run the command from inside a git-annex repository:

git-annex-duplicates

The output contains one group of duplicate paths at a time, with groups separated by a blank line:

photos/holiday.jpg
backup/holiday.jpg

documents/report.pdf
archive/report-final.pdf

Exclude one or more path patterns with --exclude:

git-annex-duplicates --exclude '*.iso' --exclude 'archive/**'

Additional arguments are passed to git annex find. Place -- before options that belong to git annex find, so they are not interpreted as options to git-annex-duplicates:

# Only report duplicates whose content is present in this repository
git-annex-duplicates -- --in=here

# Combine a local exclusion with a git-annex find option
git-annex-duplicates --exclude 'archive/**' -- --in=here

See git annex find --help for the available filters.

How it works

The command runs git annex find, requests each file's annex key, groups the returned paths by that key, and prints groups containing at least two paths. Files with the same annex key refer to the same content as far as git-annex is concerned.

Only annexed files are considered. Regular Git files and untracked files are not included, and file content does not need to be downloaded to produce the report.