basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Duplicate photos by content: a hash and a dictionary as memory

    The script finds identical files even when their names and folders differ: IMG_2041.jpg, IMG_2041 (1).jpg and a copy from the phone. It compares content by SHA-256 hash and reads only files whose sizes match. It deletes nothing — it prints the groups, and you decide.

    New hereReading a file: with openFile hashesGrouping: setdefault

    What you end up with

    Identical (5000 bytes, 563d1dc1b55c):
        a.png
        2025\a-copy.png
    Identical (300000 bytes, 58c9402bf416):
        IMG_2041 (1).jpg
        Phone\IMG_2041.jpg
        2025\summer\IMG_2041.jpg
    Duplicate groups: 2

    A group is one and the same file in different places. Which to keep is your call: delete the extras by hand.

    Step by step

    1. Write the script

      The file duplicates.py:

      import hashlib
      from pathlib import Path
      
      FOLDER = Path.home() / "Pictures"
      
      
      def file_hash(path):
          """SHA-256 of the content: identical files give an identical hash; name and date don't matter."""
          with open(path, "rb") as f:
              return hashlib.file_digest(f, "sha256").hexdigest()
      
      
      # Step 1: group by size — files of different sizes are definitely different, so don't read them
      by_size = {}
      for path in FOLDER.rglob("*"):
          if path.is_file():
              size = path.stat().st_size
              by_size.setdefault(size, []).append(path)
      
      # Step 2: within a same-size group, compare hashes
      by_hash = {}
      for size, paths in by_size.items():
          if len(paths) < 2:
              continue
          for path in paths:
              by_hash.setdefault(file_hash(path), []).append(path)
      
      groups = 0
      for digest, paths in by_hash.items():
          if len(paths) < 2:
              continue
          groups = groups + 1
          print(f"Identical ({paths[0].stat().st_size} bytes, {digest[:12]}):")
          for path in paths:
              print("   ", path.relative_to(FOLDER))
      print("Duplicate groups:", groups)
    2. Run it

      python duplicates.py

      For another folder, change FOLDER, for example to Path("D:/Photos").

    3. Go through the groups

      Open the files in a group and delete the extra copies in Explorer — to the Recycle Bin, not permanently. Run the script again: those groups are gone.

    What’s new here

    • Reading a file: with open. open(path, "rb") opens a file for reading (r) as bytes (b) — an image is read as bytes, not text. with closes the file when the block ends, even if an error happened inside.
    • A file hash. SHA-256 turns content of any size into 64 hexadecimal characters. Matching hashes mean matching content: two different files with the same SHA-256 by chance don’t occur in practice. The output shows the first 12 characters, enough to tell groups apart.
    • Grouping: setdefault. A dictionary serves as the memory of “what we’ve already seen”: the key is a size or a hash, the value a list of paths. by_size.setdefault(size, []) returns the list for that key, and if the key isn’t there yet, puts an empty one in first. .append(path) adds the path. .items() walks through the “key, value” pairs.

    Why size first

    A hash needs the whole file read, while the size is known without reading — the file system stores it. In a photo folder nearly every size is different, and only a handful of files ever reach hashing. Without the first step the script would read every byte in the folder.

    If it doesn’t work

    Two photos that look the same didn’t show up as duplicates

    Cause
    The content differs: a phone or messenger recompressed the photo, changed its size or metadata. The hash changes with any byte.
    Fix
    The script finds only exact copies. Similar images are found differently — by comparing the images, which is a separate task.

    AttributeError: module 'hashlib' has no attribute 'file_digest'

    Cause
    Python older than 3.11: hashlib.file_digest arrived in 3.11.
    Fix
    Install a current Python (python --version should say 3.11 or newer).

    The script is silent for a long time on a video folder

    Cause
    Many files share a size, and each is read whole: hashing a 10 GB video means reading 10 GB from disk.
    Fix
    Wait, or run it on a smaller folder. Files of different sizes aren’t read at all — that’s why it’s fast on photos.

    How to check

    Copy any photo into a neighbouring folder and rename the copy. The script shows them as one group. Change one pixel in the copy in an editor and save — the group is gone.