What you end up with
Identical (5000 bytes, 563d1dc1b55c):
a.png
2025\a-copy.png
Identical (300000 bytes, 58c9402bf416):
IMG_2041 (1).jpg
Phone\IMG_2041.jpg
2025\summer\IMG_2041.jpg
Duplicate groups: 2
A group is one and the same file in different places. Which to keep is your call: delete the extras by hand.
Step by step
Write the script
The file
duplicates.py:import hashlib from pathlib import Path FOLDER = Path.home() / "Pictures" def file_hash(path): """SHA-256 of the content: identical files give an identical hash; name and date don't matter.""" with open(path, "rb") as f: return hashlib.file_digest(f, "sha256").hexdigest() # Step 1: group by size — files of different sizes are definitely different, so don't read them by_size = {} for path in FOLDER.rglob("*"): if path.is_file(): size = path.stat().st_size by_size.setdefault(size, []).append(path) # Step 2: within a same-size group, compare hashes by_hash = {} for size, paths in by_size.items(): if len(paths) < 2: continue for path in paths: by_hash.setdefault(file_hash(path), []).append(path) groups = 0 for digest, paths in by_hash.items(): if len(paths) < 2: continue groups = groups + 1 print(f"Identical ({paths[0].stat().st_size} bytes, {digest[:12]}):") for path in paths: print(" ", path.relative_to(FOLDER)) print("Duplicate groups:", groups)Run it
python duplicates.pyFor another folder, change
FOLDER, for example toPath("D:/Photos").Go through the groups
Open the files in a group and delete the extra copies in Explorer — to the Recycle Bin, not permanently. Run the script again: those groups are gone.
What’s new here
- Reading a file:
with open.open(path, "rb")opens a file for reading (r) as bytes (b) — an image is read as bytes, not text.withcloses the file when the block ends, even if an error happened inside. - A file hash. SHA-256 turns content of any size into 64 hexadecimal characters. Matching hashes mean matching content: two different files with the same SHA-256 by chance don’t occur in practice. The output shows the first 12 characters, enough to tell groups apart.
- Grouping:
setdefault. A dictionary serves as the memory of “what we’ve already seen”: the key is a size or a hash, the value a list of paths.by_size.setdefault(size, [])returns the list for that key, and if the key isn’t there yet, puts an empty one in first..append(path)adds the path..items()walks through the “key, value” pairs.
Why size first
A hash needs the whole file read, while the size is known without reading — the file system stores it. In a photo folder nearly every size is different, and only a handful of files ever reach hashing. Without the first step the script would read every byte in the folder.
If it doesn’t work
Two photos that look the same didn’t show up as duplicates
- Cause
- The content differs: a phone or messenger recompressed the photo, changed its size or metadata. The hash changes with any byte.
- Fix
- The script finds only exact copies. Similar images are found differently — by comparing the images, which is a separate task.
AttributeError: module 'hashlib' has no attribute 'file_digest'
- Cause
- Python older than 3.11:
hashlib.file_digestarrived in 3.11. - Fix
- Install a current Python (
python --versionshould say 3.11 or newer).
The script is silent for a long time on a video folder
- Cause
- Many files share a size, and each is read whole: hashing a 10 GB video means reading 10 GB from disk.
- Fix
- Wait, or run it on a smaller folder. Files of different sizes aren’t read at all — that’s why it’s fast on photos.
How to check
Copy any photo into a neighbouring folder and rename the copy. The script shows them as one group. Change one pixel in the copy in an editor and save — the group is gone.