Find duplicate files in a git-annex repository

git-annex

Find a file

Mike Gerber 8cb0fe4b19 🐛 Check result of git annex find		2026-09-04 14:20:39 +02:00
.gitignore	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
.python-version	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00
default.nix	📦 Nix: fetchGit → cleanSource ./.	2026-09-03 14:36:34 +02:00
git_annex_duplicates.py	🐛 Check result of git annex find	2026-09-04 14:20:39 +02:00
LICENSE.txt	📝 Add a license	2026-09-03 09:42:10 +02:00
pyproject.toml	📝 Add a license	2026-09-03 09:42:10 +02:00
README.md	✨ Find duplicates across directories.	2026-09-03 13:38:31 +02:00
uv.lock	🛠️ Add pyproject.toml and use uv	2026-09-02 16:42:53 +02:00

README.md

git-annex-duplicates

Find duplicate files in a git-annex repository.

git-annex-duplicates groups files by their annex key and prints every group that contains more than one path. It is read-only: it reports duplicates but does not remove or otherwise modify any files.

Requirements

git-annex
Python 3.11 or newer
uv (recommended for installation)

Installation

Install the command directly from the Git repository:

uv tool install "git+https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git"

Alternatively, run it from a checkout:

git clone https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git
cd /path/to/your-git-annex-repo
uv run --project /path/to/git-annex-duplicates git-annex-duplicates

Usage

Run the command from inside a git-annex repository:

git-annex-duplicates

This will list any duplicate files below the current working directory. The output contains one group of duplicate paths at a time, with groups separated by a blank line:

photos/holiday.jpg
backup/holiday.jpg

documents/report.pdf
archive/report-final.pdf

Exclude one or more path patterns with --exclude:

git-annex-duplicates --exclude '*.iso' --exclude 'archive/**'

Find duplicates in the given directories:

git-annex-duplicates dir_a/ dir_b/

Finding duplicates across directories

To find duplicates that are duplicated across directories, use the --across flag:

git-annex-duplicates --across data/ new_data/

This will only list duplicate files that are found in more than one of the given directories.

How it works

The command runs git annex find, requests each file's annex key, groups the returned paths by that key, and prints groups containing at least two paths. Files with the same annex key refer to the same content as far as git-annex is concerned.

Only annexed files are considered. Regular Git files and untracked files are not included, and file content does not need to be downloaded to produce the report.

Contributing

This program is almost trivial, so I don't expect many contributions. However, if you wish to implement a new feature, please open an issue at GitHub first, so we can discuss it.