Managing a photographic archive

Finding duplicates in a photo archive: hashes and variants

Pale houses with tiled roofs stand on a hillside planted with olive trees.
Houses among olive groves.Photograph by Robert Harper. Robert Harper Photograph Library, Archive.London. From the archive, shown for illustration.

Three different things get called duplicates: identical files, the same picture saved differently, and neighbouring frames from one shoot. Each needs a different test, and only the first can be deleted by rule.

The Robert Harper Photograph Library's inventory gives two totals: 177,143 frames, and 95,487 unique photographs. The first includes duplicates and variants; the second does not. The distance between those numbers, more than 80,000 frames, is not waste. It is the normal condition of a working photographer's archive, and understanding what fills it is the difference between tidying a collection and damaging one.

"Duplicate" is used for three quite different things. Sorting them out is mostly a matter of choosing the right test for each.

What counts as a duplicate?

Kind What it is Example Right action
Exact duplicate The same file, byte for byte A folder copied to a second drive Keep the copies you need as backups; remove strays by rule
Near-duplicate The same picture in a different file A TIFF master and its JPEG; a resized or recompressed copy Keep the best original; record the rest as derivatives
Variant A different photograph that looks similar The frame before and after; a bracketed exposure; a retouched version Keep all; link as a group

Only the first kind is a duplicate in the strict sense. The second is a family of derivatives with one parent. The third is not duplication at all: a studio photographer who exposed many frames of one set-up made many photographs, and the choice among them is part of the record.

On film the distinction is physical. Every frame on a roll is a separate original, even when ten of them show the same shoe. Duplicate transparencies, copied from an original for sending to clients, are a genuine fourth case: a physical derivative, usually identifiable only by its mount or by notes on the sleeve.

How do you find exact duplicates?

With a cryptographic hash. A hash function reads every byte of a file and produces a short fixed-length string. The Digital Preservation Coalition's Handbook calls the result a digital fingerprint: identical files give identical values, and any change at all gives a completely different one.

That makes exact duplicates easy. Compute SHA-256 for every file in the archive, sort the list, and every value that appears twice marks two copies of one file. Names, dates and folders are irrelevant, which is the point. DSC_0421.NEF on a backup drive and the same file renamed under an archive scheme are shown to be one and the same.

Three cautions apply.

  • Do not use size, name or date as the test. Different pictures can share a name, since cameras reuse their numbering. Identical pictures can have different dates if they were copied carelessly.
  • A changed caption makes a changed file. If metadata has been written into one copy of a JPEG and not another, their hashes differ although the image data is the same. They are near-duplicates.
  • Some duplicates are deliberate. Backups are exact duplicates by design. De-duplication applies within a copy of the archive, never across the copies that make up its 3-2-1 backup scheme.

The same hash values serve a second purpose. Recorded once and checked on a schedule, they prove that files have not altered, which is why the Handbook recommends keeping them alongside the files and in a separate system as well.

How do perceptual hashes find near-duplicates?

A cryptographic hash is designed so that similar inputs give unrelated outputs. A perceptual hash is designed to do the opposite: similar-looking pictures give similar fingerprints.

The simplest kinds work like this. The image is shrunk to a tiny greyscale thumbnail, a few dozen pixels in all, which throws away size, fine detail and colour. An average hash (aHash) then records, for each pixel, whether it is lighter or darker than the average. A difference hash (dHash) records whether each pixel is lighter or darker than its neighbour. Either way the result is a string of bits, commonly 64, that describes the picture's broad structure of light and dark.

Two fingerprints are compared by counting the positions at which they differ, a figure known as the Hamming distance. A resized or recompressed copy of a picture changes very few bits. An unrelated picture changes about half.

Cryptographic hash (SHA-256) Perceptual hash (aHash, dHash)
Calculated from The file's bytes The picture's appearance
Same picture, renamed Identical Identical
Same picture, resized or re-saved Completely different Identical or close
Same picture, lightly retouched Completely different Close
Different frame, same set-up Completely different Often close
Answer it gives Yes or no A degree of similarity
Safe to automate deletion Yes, after verification No

The last two rows carry the warning. A perceptual hash cannot tell a derivative from a variant. Two frames of the same still life on the same background, taken a moment apart, may sit closer together than a master and a heavily cropped copy of it. There is no universal threshold: the right cut-off depends on the material, and has to be set by looking at what a given distance actually brings together in your own collection. Crops, rotations and mirror images defeat simple perceptual hashes altogether.

To see both kinds of fingerprint for a real file, the image hash and fingerprint tool calculates the SHA-256 and a perceptual hash in the browser and compares two images for similarity. When a match needs a human eye, the image comparison tool puts two pictures under a slider and reports the percentage of pixels that differ.

What should you do with the matches?

Group first. Delete last, if at all.

  1. Inventory before touching anything. Produce a list of every file with its path, size and SHA-256.
  2. Resolve exact duplicates within the working copy. Choose the copy in the right place, confirm that its hash matches the others, and remove the strays. Keep the list of what was removed and where it was.
  3. Cluster near-duplicates and choose a parent. The parent is the least processed, highest-quality version: the raw file before the TIFF, the TIFF before the JPEG. The guide to preserving raw files explains why the camera original outranks everything made from it.
  4. Record relationships in the catalogue. "Derived from", "variant of" and "same sitting as" are worth more than a tidy folder. A consistent file naming scheme helps by giving a master and its derivatives the same base name.
  5. Keep variants. Mark the frame that was selected or published, if that is known, and leave the others in place.
  6. Count twice. Report the number of files and the number of unique photographs, and write down the rule that separates them.

For film, the equivalent of clustering is the contact sheet: laying a whole sleeve or folder out at once, so that sequences and repeats are visible. Scans can be laid out the same way with the contact sheet maker.

Why do variants matter to buyers and researchers?

To a picture researcher, variants are the reason to visit an archive. The published frame is already known; the frame beside it is new. Reading a run of near-identical frames to see what the photographer was working towards is the skill described in the guide to reading a contact sheet.

To a licensee, variants are a question of scope. An exclusive licence for one frame means little if a nearly identical one can be licensed to a competitor, so careful agreements deal with similars explicitly.

To a buyer of image data, the two totals answer different questions. The number of files indicates volume; the number of distinct photographs indicates variety. A buyer will usually want to know both, how they were counted, and whether near-duplicates have been grouped. That is the kind of information a dataset card exists to state, and Archive.London publishes a dataset card for its own holdings.

The archive's own numbers

The figures quoted at the top come from the published inventory of 503 folders. It records 95,487 unique photographs, of which 52,684 are analogue and 42,803 digital, and 177,143 frames in all including duplicates and variants, 134,340 of them analogue. The digital files are Nikon NEF raw files, TIFFs and JPEGs. The film, mainly 6 x 7 colour transparency and described in the analogue collection, is largely unscanned.

Two caveats belong with those numbers. They are the inventory's own counts: its 503 lines add up exactly to both totals, but they have not been checked frame by frame against the physical material, and collections may overlap. For the unscanned film there are no files to hash yet, so the distinction between frames and unique photographs rests on the inventory's count and not on file-level fingerprints. Checking the counts against file-level fingerprints is one of the jobs digitisation makes possible.

Data buyers and researchers who want to understand how the totals were arrived at, or to discuss a sample, can start with the page on AI training data licensing, where licences are negotiated directly with the single rights holder, subject to rights review.

Frequently asked questions

How do I find exact duplicate photos?

Calculate a cryptographic hash, such as SHA-256, for every file and look for values that occur more than once. Identical hashes mean identical contents, byte for byte, even if the names, dates or folders differ. This is quick, objective and safe to automate. It will not find copies that have been resized, re-saved or edited, because any change alters the hash completely.

What is a perceptual hash?

A perceptual hash is a short fingerprint calculated from the appearance of an image, not from its bytes. The picture is reduced to a tiny greyscale version and its pattern of light and dark is turned into a string of bits. Pictures that look alike give fingerprints that differ in only a few bits, so resized or recompressed copies can be matched.

Can I delete near-duplicate photos automatically?

It is unwise. Perceptual matching finds candidates, not answers. Two similar fingerprints may be a master and its web copy, or two different frames from one sitting with a changed expression. Use the match to bring images side by side, keep the highest-quality original, and record the others as derivatives or variants. Deleting should be a human decision, and a logged one.

Are bracketed exposures and alternate frames duplicates?

No. Frames taken seconds apart, at different exposures or with small changes of pose, are separate photographs, and choosing between them was part of the photographer's work. An archive should keep them and link them as a group. They matter to researchers, who often want the frame that was not published, and to any licence that promises exclusivity.

Do duplicates matter when licensing images as data?

Yes. A buyer of images for machine learning will want to know how many distinct pictures a set contains, since repeated or nearly identical images add little and can distort results. Reporting both the total number of files and the number of unique photographs, and explaining how the second was arrived at, is part of describing a dataset honestly.

Sources

Published by Archive.London. Corrections are welcome under our editorial policy.

Help digitise the archive

52,684 analogue originals, mostly medium-format transparencies, are waiting to be scanned and catalogued. We are speaking to scanning, cataloguing and tagging partners.

See the digitisation briefWrite to the archive