mikegerber/git-annex-duplicates
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
Find duplicate files in a git-annex repository
12 commits 1 branch 0 tags 58 KiB
Find a file
Mike Gerber 7e80e38918 ✨ Find duplicates across directories. 
The new CLI flag --across now lets users list duplicates only if they
are found in multiple of the given directories.

Closes gh-5.

This also removes support for arbitrary arguments to `git annex find`,
as we need to know which of the CLI arguments are directories to search
in. This might be re-introduced later.

Closes gh-4.
2026-09-03 13:38:31 +02:00
.gitignore 🛠️ Add pyproject.toml and use uv 2026-09-02 16:42:53 +02:00
.python-version 🛠️ Add pyproject.toml and use uv 2026-09-02 16:42:53 +02:00
default.nix ⬆️ Nix: Bump rev 2026-09-02 18:09:43 +02:00
git_annex_duplicates.py ✨ Find duplicates across directories. 2026-09-03 13:38:31 +02:00
LICENSE.txt 📝 Add a license 2026-09-03 09:42:10 +02:00
pyproject.toml 📝 Add a license 2026-09-03 09:42:10 +02:00
README.md ✨ Find duplicates across directories. 2026-09-03 13:38:31 +02:00
uv.lock 🛠️ Add pyproject.toml and use uv 2026-09-02 16:42:53 +02:00

README.md

git-annex-duplicates

Find duplicate files in a git-annex repository.

git-annex-duplicates groups files by their annex key and prints every group that contains more than one path. It is read-only: it reports duplicates but does not remove or otherwise modify any files.

Requirements

  • git-annex
  • Python 3.11 or newer
  • uv (recommended for installation)

Installation

Install the command directly from the Git repository:

uv tool install "git+https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git"

Alternatively, run it from a checkout:

git clone https://cvs.moegen-wir.net/mikegerber/git-annex-duplicates.git
cd /path/to/your-git-annex-repo
uv run --project /path/to/git-annex-duplicates git-annex-duplicates

Usage

Run the command from inside a git-annex repository:

git-annex-duplicates

This will list any duplicate files below the current working directory. The output contains one group of duplicate paths at a time, with groups separated by a blank line:

photos/holiday.jpg
backup/holiday.jpg

documents/report.pdf
archive/report-final.pdf

Exclude one or more path patterns with --exclude:

git-annex-duplicates --exclude '*.iso' --exclude 'archive/**'

Find duplicates in the given directories:

git-annex-duplicates dir_a/ dir_b/

Finding duplicates across directories

To find duplicates that are duplicated across directories, use the --across flag:

git-annex-duplicates --across data/ new_data/

This will only list duplicate files that are found in more than one of the given directories.

How it works

The command runs git annex find, requests each file's annex key, groups the returned paths by that key, and prints groups containing at least two paths. Files with the same annex key refer to the same content as far as git-annex is concerned.

Only annexed files are considered. Regular Git files and untracked files are not included, and file content does not need to be downloaded to produce the report.

Contributing

This program is almost trivial, so I don't expect many contributions. However, if you wish to implement a new feature, please open an issue at GitHub first, so we can discuss it.