AI and images

Model collapse: when image models learn from generated pictures

Cloudy grape juice pours from an orange press into a white bucket.
Freshly pressed grape juice.Photograph by Robert Harper. Robert Harper Photograph Library, Archive.London. From the archive, shown for illustration.

The research does not say image generators will eat themselves by Christmas. It says something narrower and more useful: the unusual pictures go first, and the cure is knowing where every image came from.

Model collapse is not a machine forgetting how to draw. It is a machine forgetting that anything unusual ever happened.

Model collapse is what researchers call the slow damage done to a generative model when it is trained on the output of earlier models instead of on the world. The popular version of the story has image generators eating their own tails and dissolving into mush. The research says something narrower. The first casualties are the rare cases, the collapse depends heavily on whether real data is replaced or merely diluted, and the practical remedy is not a better algorithm but a record of where each image came from. That last point is our angle: read carefully, the model collapse papers are an argument about provenance, and a dated photograph on film is the kind of evidence they ask for.

What is model collapse, in plain language?

Photocopy a photograph, then photocopy the copy, and repeat. Each generation is recognisably the same picture and slightly worse, and the losses compound. That is the intuition, but the mechanism in a learning system is more specific, and more interesting.

A generative model learns an estimate of how its training data is distributed: what is common, what is uncommon, what never happens. It then produces samples from that estimate. Common things are sampled often. Uncommon things are sampled seldom or not at all, and the estimate of them was shakier to begin with. Train the next model on those samples and the uncommon things are rarer still. After enough rounds they are gone, and the model has no way of knowing they were ever there.

Ilia Shumailov and colleagues set this out in a preprint in May 2023 titled The Curse of Recursion, and in final form in Nature in July 2024 (volume 631, pages 755 to 759). They describe a degenerative process in which models trained on earlier models' output lose information about the original distribution, and they report that it causes "irreversible defects" in the resulting models. The tails of the distribution, the statisticians' word for unusual cases, disappear first. Later the model's output narrows towards a bland average with little variety.

Translate that into pictures. An image model trained on photographs has seen millions of ordinary sunsets and a few thousand photographs of, say, a diamond lit through a tank of water, or a 1970s shop front with hand-painted lettering. Under recursive training the sunsets survive. The diamond and the shop front are tails.

What did the experiments on image models find?

Three studies carry most of the weight for images, and each adds a qualification the headlines tend to drop.

Study Date What was tested Finding
Alemohammad et al., Self-Consuming Generative Models Go MAD arXiv, 4 July 2023 Image generators trained in loops, with and without fresh real data Without enough fresh real data each generation, quality or diversity progressively falls
Bohacek and Farid, Nepotistically Trained Generative-AI Models Collapse arXiv, 20 November 2023 A text-to-image model retrained on its own output Even a small share of self-generated images produced heavily distorted output; recovery on real images was incomplete
Gerstgrasser et al., Is Model Collapse Inevitable? arXiv, 1 April 2024 Replacing real data versus accumulating real and synthetic data Replacement tends to collapse; accumulation keeps error bounded

The first paper, from a group at Rice University and collaborators, coined the phrase Model Autophagy Disorder, or MAD, autophagy meaning self-consumption. Its authors examined three kinds of loop, which differ in whether fresh real data is available at each generation and in whether the earlier models' samples were biased towards quality or towards diversity. That second variable matters in practice. People do not publish a random sample of what a generator makes. They publish the best-looking results, and the paper finds that selecting for quality in this way costs diversity down the line.

Matyas Bohacek and Hany Farid worked with a widely used text-to-image model. They report that retraining it on even a small amount of its own output led to heavily distorted images, that the distortion spread beyond the prompts used in retraining, and that once affected the model struggled to recover fully even when retrained on real images only.

Matthias Gerstgrasser and colleagues supply the corrective. They asked what happens when generated data piles up alongside the real data instead of displacing it, which is closer to how the internet actually changes. In that setting, across several model sizes and architectures including diffusion models, collapse did not occur, and for a simplified linear model they prove that test error stays bounded however many generations pass.

Is model collapse actually happening to image generators?

Not in the dramatic sense, and it would be wrong to claim otherwise. The collapse experiments are deliberately closed loops. Commercial developers do not train exclusively on their previous model's output; they mix sources, filter, and keep their original data.

What is established is that generated images are entering training sets through ordinary channels. Bloomberg reported in April 2024 that about 5% of the images used to train Adobe Firefly were AI-generated, having been submitted to Adobe's stock library by contributors. That is a small share, labelled at source, in a curated collection. On the open web the share is unknown and the labels are frequently missing, which is the situation the accumulation result does not fully reassure about: accumulation protects a developer who keeps the real data and knows which part of the pile is which.

Synthetic data also has legitimate uses. A 2023 Google Research paper reported better image classification when generated samples were added to a real training set, as we discuss in synthetic data versus licensed photographs. The dispute in the literature is not whether generated data is poison. It is how much real data a training process needs in each round, and how a developer can tell the two apart.

Why do researchers talk about clean data?

Because every remedy in these papers assumes the developer can identify real data. Shumailov and colleagues make the point directly: in a web increasingly filled with model output, data collected from genuine human activity becomes more valuable, and its provenance has to be preserved if models are to keep learning the rare cases.

The comparison commentators reach for is low-background steel. Steel made before the first atmospheric nuclear tests carries no trace of fallout, so it has been sought for instruments that must measure faint radiation. The analogy is imperfect, as analogies are, but it captures the logic. Material with a verifiable date before a contaminant existed has a property that later material cannot acquire, whatever its other merits.

For photographs the property is easy to state. An image made before text-to-image generators were in public use cannot be a generated image. The difficulty is the word verifiable. A JPEG with a file date of 2009 proves little, since dates are editable and files are copied. Evidence improves in steps:

  • a file with an embedded capture date and camera data;
  • a raw file from the camera, which is harder to fabricate than a JPEG;
  • a published appearance with a date, such as a magazine page;
  • a physical original: a negative or transparency that exists as an object and can be inspected.

Our article on training data provenance and audit trails covers how such evidence is documented for a dataset.

Why does film count as evidence?

A colour transparency is a piece of film that was in a camera when light from the subject reached it. Whatever else may be uncertain about it, such as who the sitter was or which month it was shot, it is not the output of a model. A scan of it has an external reference that a born-digital file lacks: anyone with access can put the film on a lightbox and compare.

The Robert Harper Photograph Library is mostly this kind of material. Its inventory counts 95,487 unique photographs, of which 52,684 are analogue, principally 6 x 7 colour transparencies, with material running from the 1960s and 1970s into the digital era. The 42,803 digital photographs are held as Nikon NEF raw files, TIFFs and JPEGs. The counts come from the archive's own inventory and have not yet been independently reconciled. The archive's dataset card and published inventory set out what is held and in what form.

Two cautions keep this honest. First, human-made does not mean rights-free or suitable for any use: licensing of this archive is available by negotiation, subject to rights review, and we make no claim beyond that. Second, clean film can be made into unclean files. Generative dust removal, AI upscaling and generative fill all insert synthetic pixels. An archive that wants its scans treated as real data has to keep an unretouched master, record every processing step, and say plainly which derivatives have been through which tools. Conventional restoration is a different matter; our guide to dust removal on film scans covers the infrared and manual methods.

What should a data team do about model collapse?

The papers point to a short list, none of it exotic.

  1. Keep the original real data. The accumulation result depends on it. Do not let a later generation's training set displace the first.
  2. Know the origin of every image. Record source, date and licence at ingestion. Treat unknown-origin images scraped after about 2022 as possibly synthetic.
  3. Label synthetic data as synthetic. The EU's template for summarising training content lists synthetic data as its own source category, so the record is needed for compliance as well as quality.
  4. Do not rely on detection alone. Watermarks and credentials help where present, and our comparison of SynthID, C2PA and visible marks explains what each survives. A file can be checked for a manifest with the Content Credentials checker, but absence of a label proves nothing.
  5. Watch the tails. Evaluate on rare subjects, unusual lighting and period detail, not only on averages, because that is where collapse shows first.

The fifth point is where specialised collections earn their place. Studio still life of reflective and transparent objects, dated street scenes, obsolete technology: these are the uncommon cases an averaged model loses.

Where the archive fits

We do not claim that licensed photographs prevent model collapse, and no paper says so. The research supports a more modest statement: real, dated, documented images are what the remedies require, and film originals are unusually strong evidence of being real. AI and data teams who want to assess the Robert Harper Photograph Library as a source can start with the page on AI training data licensing, where terms are negotiated directly with the single rights holder, subject to rights review.

Frequently asked questions

What is model collapse?

Model collapse is the name given by Shumailov and colleagues to a degenerative process in which generative models trained on the output of earlier models lose information about the original data. Rare cases disappear first, then variety narrows. Their paper in Nature in July 2024 described the resulting defects as irreversible in the settings they studied.

Does training on AI-generated images always cause model collapse?

No. The research separates replacing real data from adding to it. Gerstgrasser and colleagues found in April 2024 that when generated data accumulates alongside the original real data, error stays bounded, across several model types. Alemohammad and colleagues likewise found the damage depends on how much fresh real data enters each generation.

What is Model Autophagy Disorder?

It is the term coined in a July 2023 paper, Self-Consuming Generative Models Go MAD, for what happens when generative models are trained in a loop on their own output. Autophagy means self-consumption. The authors found that without enough fresh real data in each generation, later models lose quality or diversity.

Why are photographs made before generative AI called clean data?

Because they cannot contain machine-generated content. The comparison often drawn is with low-background steel, made before atmospheric nuclear testing and valued for sensitive instruments. A photograph with a verifiable date before image generators existed, especially one with a physical film original, offers the same kind of assurance about what is not in it.

Can an archive prove its images are not AI-generated?

For film, largely yes: the transparency or negative is physical evidence, and the scan can be compared with it. For born-digital files the evidence is weaker and rests on raw files, metadata, dates and custody records. In both cases any generative retouching or upscaling applied later has to be disclosed, since it adds synthetic content to an otherwise human-made image.

Sources

Published by Archive.London. Corrections are welcome under our editorial policy.

Licensing the archive for AI and data projects

95,487 photographs from one photographer and one rights holder. Read the dataset card, search the 503-folder inventory, discuss a sample set under NDA, or arrange a first look as scanning proceeds.

See AI and data licensingWrite to the archive