AI and images

Evaluation sets for image models: the case for unscanned photographs

Gloved hands reach through yellow vine leaves to harvest pale grapes.
Grapes gathered by hand.Photograph by Robert Harper. Robert Harper Photograph Library, Archive.London. From the archive, shown for illustration.

A benchmark is a test only until it is published. After that it is on the web, and the web is what image models are trained on. Photographs that were never there have a narrow but real use.

The most valuable property a test image can have is one no amount of engineering can add afterwards: that the model has not met it before.

Every published benchmark has a shelf life, and it starts running on the day of publication. The images go online, are mirrored, thumbnailed and written about, and the next web-scale training set sweeps them up. Scores then rise for reasons that are partly ability and partly memory. The angle of this article is narrow and, we think, underused: for evaluation, the property that matters most in a photograph is not quality or subject but absence. An image with no digital copy anywhere cannot have been trained on. A film archive that is largely unscanned has that property by accident, and the task is to state it without overstating it.

Nothing below is a test result. It is a description of fit.

What is contamination, and why does it matter for image models?

A test measures generalisation only if the test items are new to the model. When they are not, the measurement is inflated by an amount nobody can easily estimate from outside.

The classic demonstration predates today's generators. In 2019 Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt and Vaishaal Shankar rebuilt the test sets for two long-standing benchmarks, following the original recipes as closely as they could. Models that had been tuned for years against the old test sets dropped 3% to 15% in accuracy on the new CIFAR-10 set and 11% to 14% on the new ImageNet set. Nothing was leaked in the crude sense. A field had simply spent a decade adapting to particular images.

Web-scale training makes the problem literal. LAION-5B, released in October 2022, holds 5.85 billion image and text pairs, 2.32 billion of them with English text. At that size the question is no longer whether a public benchmark's images are in the training data but how many, in how many resized and cropped versions.

How do benchmarks leak into training data?

Three routes, in rising order of subtlety.

Exact copies. The benchmark's files are hosted publicly and scraped along with everything else.

Near duplicates. The same photograph exists on its original site, on a stock library with a watermark, in a news article at a different crop and in a hundred reposts. A test image may be absent as a file and present as a picture.

Similarity. Even without duplicates, the training set may contain many images very like the test images: the same landmarks from the same viewpoints, the same product shots.

The third route has been studied directly. Prasanna Mayilvahanan and colleagues asked whether the strong out-of-distribution performance of CLIP models could be explained by LAION simply containing lookalikes of the test sets. They pruned LAION to remove the similar images and retrained. Performance dropped on some benchmarks but stayed high overall, so similarity is part of the story and not the whole of it. Their result cuts both ways, and it should: contamination is real and measurable, and it is not an all-purpose excuse for dismissing every score.

Ali Shirali and Moritz Hardt came at the question from the other side, rebuilding an ImageNet-like set from LAION using captions alone. Models trained on ImageNet did markedly worse on it. Test sets carry the fingerprints of how they were collected, and a model can learn the fingerprints.

Do generative models remember specific images?

Some, sometimes. In January 2023 Nicholas Carlini and co-authors reported that diffusion models memorise individual training images and can emit them at generation time; with a generate-and-filter pipeline they extracted over a thousand training examples from state-of-the-art models. Memorised images tended to be those duplicated many times in the training data.

For evaluation this matters in a specific way. If you test a generator by asking it to reproduce the manner of a well-known photograph, and that photograph was in training many times over, you have tested retrieval. The only clean version of the experiment uses a reference the model cannot have seen.

What makes a good held-out test set?

ObjectNet is the model to study. Its authors collected 50,000 new photographs, the size of the ImageNet test set, across 313 object classes of which 113 overlap with ImageNet, with backgrounds, rotations and viewpoints deliberately controlled. It ships with no training set by design. They report that object detectors lose 40 to 45% of their performance on it compared with other benchmarks.

From that and the work above, the requirements can be listed.

Requirement Why How it is shown
No prior digital copy Rules out exact and near-duplicate leakage Publication history; search of public indexes by perceptual hash
Documented origin Lets users reason about what is being tested Per-image source, date range, medium
Fixed identity Proves the files tested are the files supplied SHA-256 hash register kept by the supplier
Known labels Scores need ground truth Catalogue fields with stated confidence
Controlled release The set stays held-out only while it stays private Licence terms: evaluation only, no redistribution
Stated biases Every set over-represents something A dataset card that says what

The first row deserves a caution of its own. A search for near duplicates can only cover indexes that are open to search. The training sets behind many commercial models are not published, so a supplier can show that an image is absent from public collections and cannot show that it is absent from a private one. That is why the absence of any digital copy, anywhere, is a stronger guarantee than any search result: what was never a file cannot have been collected.

The fifth row is the uncomfortable one. A held-out set is consumed by use. Each image shown in a paper, a demo or a leaderboard page is on the web the next day. Sensible practice is to treat the set as a reserve, releasing small audited portions and retiring them afterwards.

Why would unscanned photographs be useful, and where does the argument stop?

Film that has never been digitised has no pixels to scrape. That is a clean starting position, and it is unusual: most professional photographs of the past twenty years were born digital, and a great many went online straight away.

Four qualifications keep the claim honest.

  1. Print is not nothing. A photograph made for a magazine may have run in that magazine, and somebody may have scanned the issue. The transparency in the sleeve is unscanned; a reproduction of the picture may still exist online. Only a search can tell.
  2. Subjects are not unseen. A perfume bottle or a London street has been photographed by many people. Held-out status belongs to the image, not to what it shows.
  3. Preview images are public. Anything an archive puts on its own website, watermarked or not, has left the reserve.
  4. One photographer is one distribution. A single-photographer archive tests a model against one eye, one set of clients and one run of decades. That is a sharp probe, and a poor census.

What such material is suited to, on its face: testing recognition and captioning on period product photography; testing whether a model handles film formats and their artefacts; providing dated reference images for the kind of anachronism described in why image models get the past wrong; and supplying references for film-look work, as argued in what a Kodachrome prompt gets wrong. Whether it is actually good at any of these is an empirical question for whoever runs the test.

How should an evaluation set be documented and licensed?

Documentation first. A dataset card should state counts, media, date range, how labels were made and what is unknown, and we have written separately about what AI developers look for in a licensed image dataset. For evaluation, add two things: a hash register, and a contamination statement describing what was checked and when. The image hash and fingerprint tool computes a SHA-256 and a perceptual hash locally, which is enough to begin both; the wider question of which identifiers to keep is covered in giving every photograph a permanent identifier.

Then the licence. Evaluation use differs from training use and should be written as such: testing permitted, training on the material excluded, redistribution limited, publication of examples capped and agreed. Those terms protect the buyer's experiment as much as the owner's rights, because a set that leaks stops being a test.

What the Robert Harper archive can and cannot claim

By its inventory count the Robert Harper Photograph Library holds 95,487 unique photographs: 52,684 analogue and 42,803 digital. More than half of the archive is still on film and has yet to be scanned, and the great majority of the archive has never been digitised or published online. One folder of 842 transparencies has been scanned. The archive inventory lists all 503 folders with shot and frame counts.

The qualifications above apply in full. Robert Harper was a regular contributor to Elle and The Telegraph Magazine for more than fifteen years and worked for commercial clients, so some photographs will have been reproduced in print, and we have not surveyed which. The counts come from the inventory and are not yet independently reconciled. Frame-level labels largely await cataloguing. No evaluation has been run on this material, and we report no results.

What we can offer is the starting condition and the paperwork: film originals with one rights holder, a public inventory, and a willingness to scan a defined selection under controlled terms and hash it at the moment of creation. Research teams who want to discuss an evaluation-only arrangement can begin with AI and data licensing, which is available by negotiation and subject to rights review.

Frequently asked questions

What is dataset contamination in image model evaluation?

Contamination is when images, or near copies of images, used to test a model were also present in its training data. The model's score then reflects recall as well as ability. With training sets scraped from billions of web pages, any benchmark published online is at risk, and the overlap is hard to measure from outside.

Why do benchmarks leak into training data?

Because benchmarks are built from public images and then published, mirrored and discussed. Web-scale training sets such as LAION-5B, with 5.85 billion image and text pairs, are gathered from the same web. Exact files, resized copies, crops and thumbnails of test images can all end up in training, with captions that describe them.

How do you build a held-out evaluation set for images?

Use images with no public digital copy, keep them private, document where each came from, and record a cryptographic and a perceptual hash for every file. Before use, search public indexes for near duplicates. Release only what must be released, and treat any image that has been published as retired from held-out use.

Are unpublished photographs automatically uncontaminated?

No. A photograph that was never scanned may still have appeared in print, and the printed page may have been scanned by someone else. Similar pictures of the same product or place may be online. Held-out status has to be checked image by image, and it concerns the specific file, not the subject.

Can an archive license images for evaluation only?

Yes, in principle. An evaluation licence can limit use to testing, forbid training on the material and restrict redistribution, which also preserves the set's held-out value. Terms depend on the rights in each group of images, so such licences are negotiated directly with the rights holder and are subject to rights review.

Sources

Published by Archive.London. Corrections are welcome under our editorial policy.

Licensing the archive for AI and data projects

95,487 photographs from one photographer and one rights holder. Read the dataset card, search the 503-folder inventory, discuss a sample set under NDA, or arrange a first look as scanning proceeds.

See AI and data licensingWrite to the archive