How Accurate Are AI Summaries? What Studies Show

AI summary accuracy varies by model and by task, and no summary is guaranteed correct. On Vectara’s benchmark (October 2026), models added unsupported content to between 1.8% and 24.2% of summaries, and peer-reviewed studies also find summaries that over-generalize or leave things out. Use a summary as a first read, then check numbers, names and key claims against the source.

What are the two ways an AI summary goes wrong?

An AI summary can fail in two directions, and each needs a different check.

It adds something. The summary states a fact, figure, name or conclusion that the source never gave. Researchers call this a hallucination, or unfaithful content. It is the easier error to catch: you look for the claim in the source and it is not there.

It leaves out or stretches something. The summary drops a caveat, a condition or a counter-argument, or it turns a narrow statement into a broad one. Suppose a speaker says that most of the 200 customers in one survey preferred the new design, and the summary says customers prefer the new design. Nothing was invented, but the claim is now bigger than the evidence. This error is harder to notice, because every word of the summary sounds like the source.

Most public numbers measure only the first kind.

How often do AI summaries add unsupported content?

As of November 2026, one public, regularly updated measurement is the Vectara Hallucination Leaderboard. Models are asked to summarize documents using only the facts in them, and an automated model then scores how often a summary contains content the document does not support. On the table updated 22 October 2026, which lists 111 models, rates run from 1.8% to 24.2%, with a median of about 9.6%.

Read those numbers with four caveats:

• A vendor runs it. Vectara is an AI company, the judge is its own commercial model, and the leaderboard is not peer-reviewed.
• The judge is automated. That model (HHEM-2.3) does the scoring, not human readers.
• The setup is specific. The test set is private, the articles run from about 50 to 24,000 words, and summaries are capped at 20% of the source length. That is not the same job as summarizing a two-hour transcript.
• Older figures are not comparable. Older articles quote rates below 1% for the best models. Those came from the leaderboard’s previous dataset, last updated in October 2025. The current table uses a different one, so the two sets of numbers cannot be placed side by side.

Automated judging has limits of its own. FaithBench, a peer-reviewed benchmark presented at NAACL 2025, had human experts label summaries from 10 language models in 8 model families, concentrating on cases where automated detectors disagreed with each other. On that deliberately hard set, even the best hallucination detectors were only about 50% accurate. GPT-4o and GPT-3.5-Turbo hallucinated least among the models tested, which were the ones available in 2024. The lasting lesson is about measurement: catching unsupported content automatically is hard, so a leaderboard figure is an estimate.

Do AI summaries leave things out or over-generalize?

Two peer-reviewed studies look at omission and over-generalization, the second kind of error.

Peters and Chin-Yee (2025), writing in Royal Society Open Science, analyzed 4,900 summaries of scientific texts produced by 10 language models. Most models stated findings more broadly than the original texts did. For DeepSeek, ChatGPT-4o and LLaMA 3.3 70B, that happened in 26% to 73% of cases. Model-written summaries were nearly five times more likely than human-written ones to contain broad generalizations (odds ratio 4.85, 95% confidence interval 3.06 to 7.70), and newer models tended to do worse, not better. The study covers scientific writing, where hedges carry much of the meaning. It does not give a rate for lectures or podcasts.

In a clinical study, Asgari and colleagues (2025), published in npj Digital Medicine, had clinicians annotate 12,999 sentences from AI-written medical summaries across 18 experimental configurations. They report a 1.47% hallucination rate and a 3.45% omission rate, so in that setting leaving something out was more than twice as common as making something up. One disclosure matters: all of the authors were affiliated with Tortus AI, the vendor behind the work. Treat it as a careful vendor evaluation of a tuned clinical system, not as an independent rating of consumer apps.

Is the EBU and BBC finding about summaries?

No, the EBU and BBC finding is about news question answering. The News Integrity in AI Assistants report was published by the European Broadcasting Union in October 2025 and built on an earlier BBC study. Journalists at 22 public-service media organizations in 18 countries, working in 14 languages, assessed more than 3,000 answers that ChatGPT, Copilot, Gemini and Perplexity gave to questions about news and current affairs. Almost half of the answers had at least one significant issue, about a third had serious sourcing problems, and about a fifth had major accuracy issues.

That is a different task from summarizing one document. The assistant has to find sources, choose among them and attribute them correctly before it writes anything. The report is not peer-reviewed. It is strong evidence that AI answers about the news need checking, and weak evidence about a summary of a video or PDF that you supply yourself.

Why is there no single accuracy number for AI summaries?

The benchmarks and studies on AI summaries measure different things. The tasks differ (document summaries, scientific texts, clinical notes, news questions), and so do the models, the judges (an automated model, human experts, clinicians, journalists) and the definition of an error. Benchmarks also change, as the leaderboard’s dataset switch shows. We did not find a peer-reviewed 2026 study of long-document summarization, which is the closest match to an hour-long transcript.

Videos add one more variable: the captions. A tool that works from the transcript can only summarize what the captions say, and YouTube’s Help Center notes that automatic captions vary in quality with accents, dialects, mispronunciations and background noise. A misheard name or number can pass into the summary. Captions also decide whether a summary is possible at all, as explained in why AI can’t summarize some YouTube videos.

So nobody can honestly state one percentage that holds for every tool and every source. That includes us.

How do you check an AI summary quickly?

Checking an AI summary does not mean watching the whole video again. It means verifying the parts you will use.

1. Pick what matters. Mark the two or three claims you would repeat, cite, study or act on. The rest can stay unverified.
2. Go to the moment. For a video, use the timestamp next to the point and listen to the original for a few seconds. For a document, search for a distinctive phrase.
3. Look in the transcript. Names, dates and figures are quick to find in a full transcript. If the number is not there, do not use it.
4. Ask where a claim comes from. In a tool with source-based chat, ask which part of the source supports the statement, then look at that part. If the tool cannot point to a place, treat the claim as unconfirmed.
5. Slow down on four things: numbers and units, names, negations (not, never, except) and hedges (may, in some cases, early results). Additions and over-generalizations hide there.

For a medical, legal or financial decision, read or watch the source itself. A summary can show you where to look. It should not make the decision.

What does SumAI say about its own accuracy?

SumAI is our app, so weigh our view accordingly. SumAI does not publish an accuracy figure, and we will not invent one. Its summaries are written by AI from a video’s transcript or a document’s text, and like any AI summary they can miss nuance or get a detail wrong.

What SumAI offers is a short path back to the source:

• Tappable timestamps. Key moments and chapters carry times. Tap one to jump to that part of the video.
• The full transcript, with timestamps, ready to copy.
• Answers tied to the source. Ask AI answers from the video or document itself, and for videos the answers include timestamps you can tap.

Two limits apply. Videos need captions or auto-generated subtitles, and PDFs need selectable text. And because SumAI reads the transcript, a chart shown on screen but never described out loud is not part of what it summarizes.

When is an AI summary enough, and when is it not?

• Deciding whether to watch or read something: a summary is enough. A wrong detail costs little.
• Catching up on a talk, an interview or an article: read the summary, then check the one or two points you will repeat to someone else.
• Studying: use the summary as a map, and verify definitions, formulas and figures before they go into your notes. TL;DR vs Detailed explains how much depth each style keeps.
• Quoting, citing, or deciding about health, money or legal matters: go to the source, and use the summary only to find the right passage.