The text attached to a photograph is half of what a multimodal model learns from. Most captions were written to sell, rank or amuse. A useful one simply says what is there, and says how it knows.
When OpenAI's researchers built DALL-E 3, they did not train it mainly on the captions that came with their images. They trained a captioning model, rewrote the dataset and used a mix of 95% machine-written captions and 5% original ones. Their paper, "Improving Image Generation with Better Captions", gives the reason plainly: captions found on the internet are often incomplete and "oftentimes simply incorrect".
That one decision says a great deal about what a caption is worth to a model developer, and what an archive should aim for when it describes its own pictures.
This article is general information, not legal advice. Checked on 17 November 2026.
Why do captions matter for training image models?
Text-to-image and vision-language models learn from pairs: a picture and some words about it. The words are the only route by which the model connects the word "decanter" to the look of a decanter, or "backlit" to a rim of light. If the words are vague, the connection is vague. If they are wrong, the model learns the wrong thing with complete confidence.
The DALL-E 3 authors found that training on detailed, accurate captions improved how faithfully the model followed a prompt. Better text made a better model from the same images. For a supplier, that turns description from an afterthought into part of the product, a point buyers raise early, as noted in what AI developers look for in a licensed image dataset.
What is wrong with most captions found online?
They were written for another purpose. Alt text and surrounding copy exist to sell a product, rank a page or raise a smile. The DALL-E 3 paper lists what such captions usually leave out:
- the presence of objects other than the main subject;
- where things are in the frame, and how many there are;
- colours and sizes;
- text that appears in the image.
It adds that web captions are frequently advertisements or memes with little relation to the picture. Agency and archive captions have a different weakness. They are strong on who, where and when, and often silent on what the picture looks like, because a picture editor could see that for themselves.
What makes a caption useful to a model?
A working standard, drawn from those findings and from ordinary cataloguing practice:
- Describe what is visible first. Subject, secondary objects, setting, background.
- Be specific with nouns. "A faceted glass bottle with a gold cap", not "a product".
- Give counts and positions. "Three bottles in a row, the tallest at left."
- Name colours, materials and surfaces. These are what a model of products or interiors most needs.
- Describe light and viewpoint. Backlit, overhead, shallow focus, on a white ground.
- Transcribe visible text exactly, in quotation marks.
- State the medium where known. Colour transparency, black-and-white negative, Polaroid print.
- Add context only from a record. Date, place and names come from sleeve notes, job sheets or published credits, not from a guess.
- Mark uncertainty. "Circa", "unidentified", and square brackets for anything the writer supplied.
- Leave out judgement. "Stunning" teaches a model nothing.
- Keep a consistent order across the collection, so the text can be parsed.
A short illustration, which is invented for the purpose and is not a record from any archive:
| Weak | Useful |
|---|---|
| "Luxury fragrance, studio shot" | "A rectangular clear glass perfume bottle with a black cap, standing upright at centre on a sheet of black acrylic. Its reflection is visible below. Lit from behind, with a bright rim along both edges. The label reads 'EAU DE PARFUM'. Plain dark grey background. Studio still life, colour transparency." |
The second version contains nothing a viewer could not confirm by looking, apart from the medium, which comes from the physical original.
Should captions be written by people or by machines?
By both, for different things, and never without saying which.
A captioning model is quick and consistent at the visible layer: objects, colours, arrangement. It cannot know anything that is not in the pixels. It has no access to who the sitter was, which year the job was shot or which magazine it was for, and when asked it may supply a confident invention. The strengths and failures are set out in can AI tag an archive?.
The contextual layer comes from evidence: the photographer's sleeve annotations, contact sheets, job numbers and published pages. The method for reading that evidence is covered in writing captions from contact sheets and sleeve notes.
The two layers should be stored separately and each should record its source. A minimal provenance note for a caption has three parts: who or what wrote it, from what evidence, and when. "Machine description, model and version, date" is a perfectly respectable entry. An unlabelled mixture is not, because a buyer cannot tell which statements to trust, and a developer filling in a training content summary under the EU AI Act has no way to describe it.
How should captions be structured and stored?
Offer layers and let the developer choose:
| Layer | Length | Content |
|---|---|---|
| Alt text | One sentence | What the image shows, for a reader who cannot see it |
| Caption | One to three sentences | Visible content plus recorded context |
| Extended description | A paragraph | Full visual detail: arrangement, light, materials, text |
| Keywords | A list | Terms from a controlled vocabulary |
| Technical | Fields | Medium, format, date or date range, identifier |
| Caption provenance | Fields | Author or model, evidence, date written |
The IPTC Photo Metadata Standard has homes for most of these: Description, Headline, Keywords, and separate accessibility fields for alt text and extended description. Embedding them keeps the text with the file. The captions and keywords tool writes title, description and keywords into a JPEG's XMP in the browser, and the metadata viewer shows what a file already holds. How the embedded standards relate is explained in IPTC, XMP and EXIF compared.
For delivery at scale, the same fields go into a manifest, one row per image, keyed to a stable identifier. Controlled keyword lists, discussed in keywording a photographic archive, stop "trainer", "sneaker" and "plimsoll" from becoming three unrelated subjects.
Names, people and rights in captions
A caption can create a problem the photograph alone did not. A name attached to a face is personal data about a living person, and a dataset that pairs the two systematically is far more sensitive than one that describes "a man in a grey suit, seated". What is decided is that UK and EU data protection law applies to such processing. How regulators and courts will treat particular training uses is still being worked out, as the article on people in the picture describes.
Two practical rules follow. Hold identifications in a separate field that can be withheld from a delivery, so a descriptive caption stands on its own. And treat brand names in the same deliberate way: a transcribed label is a fact about the image, while a claim about a commission or endorsement is not something a caption should assert without a record.
Captions also have authors. Text written by a cataloguer is original work in its own right, which is worth remembering when a licence defines what is being supplied.
Captioning at the Robert Harper archive
The Robert Harper Photograph Library is described at present at collection and folder level: 330 collections online across 24 principal categories, and a public inventory of 503 folders. Image-level captioning of the kind set out above largely lies ahead, alongside scanning, since more than half of the archive is still on film. The subjects are ones where exact description matters, from fragrance bottles and jewellery to early mobile telephones.
Archive.London is looking for cataloguing and metadata partners for that work, described on the digitisation partners page. Developers interested in the collection as licensed data, with captions whose origin is recorded, will find the terms of engagement on the AI training data licensing page; licences are negotiated directly with the single rights holder, subject to rights review.
Frequently asked questions
What makes a good image caption for AI training?
A good training caption is literal, specific and complete. It names the main subject and the secondary objects, gives counts, positions, colours and materials, describes the setting and lighting, and transcribes any visible text. It avoids opinion and marketing language, and it marks anything the writer inferred instead of saw.
How long should a caption be for training data?
There is no single correct length. Many datasets pair each image with a one-sentence caption and a longer description of several sentences, so a developer can choose. The DALL-E 3 research found that long, descriptive captions improved how closely a model followed prompts. Offer both layers and keep them in separate fields.
Can I use AI to write captions for my archive?
Yes, for what is visible. Captioning models describe objects, colours and composition quickly. They cannot know who a person is, where or when a picture was taken or why, and they sometimes state things that are not in the frame. Label machine captions as such and keep them separate from captions based on records.
What is the difference between alt text and a caption?
Alt text is a short description written so that someone who cannot see an image understands its content, usually a sentence. A caption in the archival sense adds context: who, what, where and when. For training data both are useful. The IPTC standard has separate fields for them, so neither needs to be squeezed into the other.
Should captions include people's names?
Only where the identification is reliable and you have considered the consequences. A name attached to a photograph of a living person is personal data, and pairing names with faces is exactly what makes an image dataset sensitive. Many suppliers deliver a descriptive caption without names and hold identifications separately. This is general information, not legal advice.
Sources
- Betker et al., Improving Image Generation with Better Captions (OpenAI)
- IPTC Photo Metadata Standard specification
- European Commission: Q&A on the training content summary template
- EDPB Opinion 28/2024 on data protection aspects of AI models (17 December 2024)
This article is general information, not legal advice. It reflects the position as we understood it on the date shown and may be out of date. Published by Archive.London. Corrections are welcome under our editorial policy.

