AI and images

Searching an uncatalogued archive by picture: embeddings explained

Crew in colourful waterproof clothing aboard the heeling yacht Grey Goose.
Grey Goose under sail.Photograph by Robert Harper. Robert Harper Photograph Library, Archive.London. From the archive, shown for illustration.

A folder system can tell you where the slingbacks are. It cannot tell you where the red ones are. Similarity search can, before a single keyword exists, and it will also tell you some confident nonsense along the way.

Similarity search finds what looks alike, and an archive's whole difficulty is that what looks alike is so often not what is alike.

The Robert Harper inventory has forty-three folders of shoes. They are filed the way a studio thinks: clogs, slingbacks, heeled mules, wellington boots, and then by designer. Nothing is filed by colour. So a picture researcher who wants every red shoe has a question the archive cannot answer, and ordinarily would not be able to answer until someone had looked at 1,891 catalogued shots and typed "red" beside some of them. Similarity search answers it first. Our angle is that this does not replace cataloguing; it reverses its order, letting you find before you describe, and that its mistakes are all of one kind. It retrieves what looks alike. An archive's difficulty is that what looks alike is often not what is alike.

What is an embedding, in plain terms?

An embedding is a description of a picture written for a machine: a list of a few hundred numbers produced by a neural network. No single number means anything a person could name. Together they act as coordinates. Pictures the network regards as similar get coordinates that are close, and pictures it regards as different are far apart.

That alone gives search by example: choose one image, ask for its nearest neighbours, and receive more like it. The step that made the idea useful to archives came in 2021. OpenAI researchers published CLIP on 26 February of that year, a model trained on 400 million image-text pairs gathered from the internet. Its training task was simple to state: given a batch of images and a batch of captions, work out which caption belongs to which image. To do that it had to learn two encoders, one for pictures and one for text, that place matching pairs close together in the same space.

The consequence is that a typed phrase and a photograph can be compared directly. Type "red high-heeled shoe on a white background", embed the phrase, and the nearest image vectors are, with luck, red high-heeled shoes on white backgrounds. Nobody tagged them. The paper showed that this approach, with no task-specific training, matched an established image classifier on a standard benchmark and transferred across more than 30 other data sets.

How does similarity search work on a box of scans?

The pipeline is short, and most of the labour is the scanning that precedes it.

  1. Make working copies. The model looks at small versions, a few hundred pixels across. The master scans are not touched.
  2. Embed every image once. Each working copy goes through the image encoder and comes out as a vector. This is the slow step, and it is done once per model.
  3. Build an index. The vectors are loaded into a library built for nearest-neighbour search. Faiss, developed mainly at Meta's research group and released under the MIT licence, is the best known; it describes itself as a library for efficient similarity search and clustering of dense vectors.
  4. Query. By text, by example image, or both.
  5. Confirm and record. A person looks at the results, accepts or rejects, and writes approved terms into the catalogue.

The arithmetic is kinder than people expect. A common CLIP variant produces 512 numbers per image. Stored as ordinary 32-bit values, that is about 2 KB each, so 100,000 photographs need roughly 200 MB. At that size a computer can simply compare the query against every vector, which is exact. Faiss's documentation is clear that the cleverer compressed indexes, which scale to billions of vectors, generally come at the cost of a less precise search. An archive of this size does not need them and should not pay that cost.

For scale: this archive's inventory counts 95,487 unique photographs and 177,143 frames including duplicates and variants.

What can vector search do that keywords cannot?

It answers questions nobody anticipated when the folders were labelled.

Task With folders and keywords With similarity search
All red shoes Open 43 folders and look One text query, then confirm
More like this frame Remember where you saw it Query by example
Same set-up across years Rely on job numbers, if any Cluster by composition and lighting
Misfiled items Found by accident Outliers within a folder stand out
Variants of one shot Compare sleeves by eye Near neighbours group together
Uncatalogued scans Invisible until described Searchable on the day they are scanned

The last row matters most to a collection in mid-digitisation. Scans normally sit in a queue until a cataloguer reaches them, and during that time they cannot be found. With an index they are findable at once, in a rough way, which lets the cataloguer work in batches: retrieve two hundred probable fragrance bottles, deselect the six that are decanters, apply the term to the rest. Our guide to AI image tagging for archives covers the related business of machine-suggested keywords, and controlled vocabularies the terms a person should apply once the pictures are found.

Picture researchers gain something as well. "More like this" is how a human researcher thinks, and a folder list cannot do it.

Where does similarity search mislead?

In consistent ways, each documented in the research.

It does not read sentences. Mert Yuksekgonul and colleagues built a benchmark called ARO, with more than 50,000 test cases on attributes, relations and word order, and posted their results on 4 October 2022. Vision-language models showed weak relational understanding, sometimes failed to attach attributes to the right objects, and were largely insensitive to word order. The authors' phrase is that the models behave like bags of words. Ask for "a red shoe beside a black bag" and expect black shoes beside red bags among the results.

It reads writing in the picture. In an article in Distill on 4 March 2021, OpenAI researchers reported that CLIP was using text within images to classify them, and confirmed that typographic attacks, in which a written word overrides what the object is, were real. An archive is full of writing: handwritten transparency mounts, captions on Polaroids, magazine pages. A search for "Chanel" will favour images in which the word appears, whether or not the product is the subject.

It knows the internet, not your archive. A model trained on web captions recognises what the web captions often. It is poor at the specific: which designer made a shoe, which year a telephone dates from, who a sitter is. None of those is reliably visible, and a similarity score is not evidence of any of them.

Near-duplicates crowd the results. Studio work is shot in sequences. With 177,143 frames for 95,487 unique photographs, the nearest neighbours of any frame will mostly be its own siblings. Grouping variants first makes results far more useful; our guide to finding duplicates with hashes explains the method, and the image hash generator shows it on single files.

The score is not a probability. A similarity of 0.31 against 0.29 means only that one vector is slightly closer. Where to draw the line between match and non-match has to be set by experiment for each kind of query.

Scan condition may leak in. Faded dyes, colour casts and film rebate edges are visual features too. Whether a model groups pictures by subject or by the state of the film is worth testing before anyone trusts the clusters.

How should results be checked?

By measurement on a sample, never by impression. Take a few hundred scans across every format, have a cataloguer label them for a dozen test queries, and count: of the top twenty results for each query, how many are right, and of the known right answers, how many appear at all. Repeat when the model changes. Record which model and version produced the vectors, because embeddings from different models cannot be mixed in one index.

Then keep the two kinds of statement apart in the catalogue. "Retrieved as visually similar to red shoes" is a machine suggestion. A line such as "court shoe, Manolo Blahnik, folder Shoes 10" is a record a person has verified against the sleeve. Only the second should reach a licensee as fact.

Does this need the images to leave the building?

No. Open models and open index software run on a single workstation, which matters for unpublished material. If a hosted service is used, the questions in the tagging guide apply unchanged: can the provider retain the images, and can it train on them.

What this means for the Robert Harper archive

The published inventory lists 503 folders with shot and frame counts, and it is a folder-level description: it says what each box holds and does not yet describe individual frames. More than half of the archive is still on film and has yet to be scanned. That is the situation in which similarity search earns its keep, because an index can be built scan by scan as digitisation proceeds, and the first useful searches do not have to wait for the last caption.

It will not date a frame or name a sitter; we take up the dating problem in can a machine date a photograph. Digitisation and cataloguing vendors who build embedding indexes into their workflow, and want a varied analogue collection to work on, will find the brief on the digitisation partners page.

Frequently asked questions

It is search that uses a model such as CLIP to turn every image into a vector, a list of numbers, and turns a typed query into a vector in the same space. Images whose vectors lie closest to the query are returned. Because the model learned from 400 million captioned images, it can match pictures to words without any tags.

Can you search a photo archive that has no keywords?

Yes, once it is scanned. Each scan is embedded once, the vectors are stored in an index, and queries by text or by example image return the nearest matches. The results are visual resemblances, not verified facts, so they are best used to find candidates quickly and then confirmed and described by a person.

It is strong on generic visible content such as object type, colour, setting and composition, and weak on anything not visible: names, dates, places, clients. Research also shows weak handling of relationships and word order, and sensitivity to text inside the picture. Accuracy on a particular archive has to be measured on a hand-checked sample.

Do I need to upload my images to use embeddings?

No. Open models in the CLIP family and index libraries such as Faiss, which is MIT licensed, can run on local hardware, so unpublished scans need not leave the building. If a cloud service is used instead, check in writing whether the provider may keep the images or use them to train its own models.

Does similarity search replace cataloguing?

No. It changes the order of work. Instead of describing every frame before anything can be found, a cataloguer can retrieve groups of similar frames, confirm them and apply terms in batches. The facts that make an archive usable, such as who, when, where and for whom, still come from sleeves, job sheets and research.

Sources

Published by Archive.London. Corrections are welcome under our editorial policy.

Licensing the archive for AI and data projects

95,487 photographs from one photographer and one rights holder. Read the dataset card, search the 503-folder inventory, discuss a sample set under NDA, or arrange a first look as scanning proceeds.

See AI and data licensingWrite to the archive