RAG vs AI Agents: 10,000 Clinical Queries, One Missing Comparison

TL;DR: Microsoft Research's 10,000-query BenchmarkQED study reported that LazyGraphRAG won all 96 retrieval comparisons. The study tested no AI-agent arm and reported no human escalation rate. The defensible path is a controlled benchmark with frozen inputs and separate quality and operational scoreboards.

Key Takeaways: - A retrieval-method win is not an agent-versus-RAG verdict. - Escalation can change an operational ranking, but only with arm-level rates and predefined triggers. - Choose architectures by query class and workflow, then validate them through controlled shadow-mode testing.

A claim that human escalation reversed the clinical AI winner would be powerful. It is also unsupported by the cited BenchmarkQED research.

Measured: Microsoft Research's BenchmarkQED evaluated 10,000 clinical queries with GPT-4o held constant across retrieval and RAG conditions.

>

Not measured: No AI-agent arm and no reported human escalation rate.

>

Study result: LazyGraphRAG won all 96 comparisons. All but one met the study's threshold for statistical confidence.

The win is real, but its endpoint is narrow. It ranks retrieval conditions under one generator, not complete clinical workflows.

Agents add planning, tool calls, state, permissions, retries, and a route to human review. Any of those can change the endpoint, so treating this as an architecture verdict is a category error.

For healthcare technology teams, the practical trap is choosing an architecture before defining success. - Retrieval endpoint: Can the system find and use the right evidence? - Workflow endpoint: Can it complete the task safely under real tool, permission, and escalation conditions?

The CTO takeaway is blunt: define the evaluation endpoint before selecting the architecture. Otherwise, the benchmark rewards whichever system is easiest to score, not the system that is safest to operate.

The missing agent arm matters for the interpretation because it changes the unit of work, not just the model. Comparing the architectures requires separating what each one actually does.

Traditional RAG follows a retrieve-then-generate path. It retrieves evidence and passes it to the generator.

An AI agent can rewrite the query, choose tools, call APIs, inspect results, and continue until a stopping rule is met. Agentic RAG places retrieval inside that loop. Retrieval becomes one tool among several, governed by the agent's plan and permissions. - Fixed RAG: One retrieval path feeding one generation step. - Agent workflow: Dynamic queries, tool selection, actions, retries, and termination logic. - Agentic RAG: Retrieval controlled by a broader workflow rather than a fixed pipeline.

Suppose one arm answers more cases without review, but its unsupported claims are severe. Another arm escalates uncertain cases, receives correction, and reaches a safe disposition.

Under quality-only scoring, the first can look better. Under safe-resolution scoring, the second can win.

The cited research contains no human escalation rates, so it cannot establish that escalation flipped the ranking.

Escalation is not automatically failure. Missing evidence, contradictory guidance, tool failure, or irreversible action can make review the correct outcome.

A low escalation rate is not automatically better. You must score severity, final disposition, and whether the human accepted, corrected, or rejected the recommendation.

Clinical review changes the endpoint by correcting unsafe outputs before use. That is a workflow property, not a model defect.

Holding GPT-4o constant isolates retrieval behavior. An agent test must also freeze prompts, tool schemas, permissions, retry rules, and the work completed before review.

Otherwise, a stronger agent receives more evidence or a better path, which breaks the comparison. - RAG arm: Fixed retrieval, generation, and refusal path. - Agent arm: Documented tools, permissions, retries, and stopping conditions. - Controlled difference: Only the architecture under evaluation should change.

The key is to treat escalation as a measured control, not a footnote. That distinction becomes concrete in the retrieval configurations that won the cited benchmark.

In the 10,000-query clinical AI benchmark, LazyGraphRAG won all 96 comparisons. GPT-4o stayed constant. All but one comparison met the study's threshold for statistical confidence.

For AI/ML training teams, the lesson is about method: isolate the layer you claim to improve.

`LGR_b200_c200` delivered the best overall performance. It paired a larger retrieval budget with smaller chunks.

More retrieved material can improve recall, while smaller chunks can reduce irrelevant context. The result shows their pairing worked for the aggregate workload.

For DataLocal queries, `LGR_b50_c200` performed slightly better. That result points to a smaller retrieval budget while retaining the smaller chunk size.

For ActivityLocal queries, `LGR_b50_c600` had a slight edge, showing that chunk size still mattered within that local class.

These are exceptions, not contradictions. They show the aggregate winner can hide workload-specific behavior.

GraphRAG Global had a relative lead on global queries. The pattern matches the broader finding that a method can lead on the class it was designed to serve.

Results snapshot: - Aggregate winner: `LGR_b200_c200`, LazyGraphRAG, best overall. - DataLocal exception: `LGR_b50_c200`, LazyGraphRAG, slight lead over the aggregate winner. - ActivityLocal exception: `LGR_b50_c600`, LazyGraphRAG, slight edge within that query class. - Global lane: GraphRAG Global, relative lead for global queries.

A reproducible routing candidate would look like this:

Retrieval budget and chunk granularity pull against each other. A broader search can surface more candidates, but larger chunks can dilute each candidate. A tighter budget can work when only a small evidence set matters.

The benchmark supports a query-specific setup, not one universal setting for every clinical workload. A CTO should reproduce the best configuration per query class before standardizing it.

Query-class routing adds maintenance burden, but it also prevents a strong aggregate score from hiding a weak local case.

The benchmark therefore points to a testing method for CTOs, not a universal architecture verdict. The next question is how to run that test without changing the task midway.

An agent benchmark can fail before the model is judged. The team changes the prompt, gives one arm better tools, and calls the result architecture.

Hold the generative model, prompt versions, corpus snapshot, evaluator, and query set constant during a test round. BenchmarkQED used GPT-4o across retrieval conditions, which isolates retrieval behavior.

Set and record tuning budgets before the run. If one arm gets prompt revisions after seeing failures while the other stays frozen, the comparison moves the goalposts.

The failure mode resembles Why Your LLM Evals Approve Models That Fail: a test can pass while its real endpoint remains untested.

Compare a fixed RAG baseline with an agentic workflow where retrieval is one tool. Keep task, source material, success rubric, and available evidence aligned.

Document every extra tool, action, permission, and source. An agent with broader access is doing a different task, even if its prompt looks similar.

For AI agents, assign each tool a schema, timeout, retry rule, and permission boundary. Apply the same experimental discipline used in AI/ML training: freeze the split, lock the metric, and publish the exceptions.

Publish quality and evidence separately. Track factual correctness, evidence completeness, unsupported claims, and clinical review findings.

Then publish operational results. Track safe completion, human escalation rate, latency, cost per resolved case, tool failures, retry counts, and final disposition.

An aggregate rank cannot hide the trade-off. A system can lead on answer quality while lagging on safe completion.

Define escalation triggers before the run. Include missing evidence, contradictory guidance, tool failure, high-risk interpretation, and irreversible action.

Lock the denominator, severity scale, reviewer instructions, and adjudication rules. Log retrieved evidence, routing decisions, tool calls, final response, escalation reason, and human disposition for every case.

Report DataLocal, ActivityLocal, and global slices. Include confidence intervals and statistical tests, not only one aggregate champion.

A local exception can determine routing. An aggregate win can hide which clinical question class still fails.

Once the variables are isolated, architecture selection becomes a workflow decision. The answer depends on how the work is actually structured.

Use RAG when the answer lives in approved documents, guidelines, or records and the task is mainly read-only. Use an AI agent when the workflow requires decisions, tool calls, API actions, or multi-step execution across systems.

Use agentic RAG when the system must plan dynamically but still needs auditable evidence retrieval. This combines iterative search, source inspection, and a constrained recommendation.

Architecture labels matter less than task fit, evidence traceability, escalation behavior, and maintainability. Calling RAG dead, as RAG Is Dead for Healthcare AI argues, misses this workload split.

A RAG failure in one workflow does not make RAG useless across all workflows.

Route missing or contradictory evidence, tool failures, high-risk interpretations, and irreversible actions to explicit human approval. Use least-privilege access, bounded tools, approval gates, and complete audit logs.

Your AI Agent Is Your Engineer in the Audit Log explains why attribution matters once agents act. A healthcare technology system should not treat an answer without evidence as a harmless failure.

Start in shadow mode. Compare recommendations with expert decisions without letting the system affect care. Introduce sampled human review, then expand automation only where quality and escalation behavior justify it.

Deployments of AI agents should be earned through measured control, not vendor labeling. Keep the decision record small. It should name workflow, evidence, escalation rule, and owner.

The remaining question is whether that decision can survive clinical review, audit, and production reality.

The result is not "RAG wins" or "agents win." The CTO gains a workload-specific architecture, a measurable escalation policy, and a routing rule based on evidence.

That is more useful than a leaderboard. It can be reviewed against a fixed query set, fixed generator, and fixed endpoint.

Evidence about deployment longevity, compliance posture, retention, and delivery speed can help assess operational partner risk. It cannot substitute for arm-level answer quality or escalation data. A stable system can still produce weak clinical answers.

For healthcare technology leaders, this distinction prevents procurement evidence from replacing clinical evidence. A partner such as Levitation should be judged against the same evidence standard as any internal platform.

Claims about deployment volume, customer retention, compliance, production longevity, or delivery speed need supporting records. None can replace arm-level clinical results.

When reviewing top AI companies for healthcare in India, ask for the same query set and generator where applicable. Request explicit escalation denominators and a clinically reviewed adjudication process.

Also request evidence for production longevity, compliance controls, customer retention, and delivery timelines. The main decision should fit in a one-page architecture decision record.

The record contains the workload mix, winning configuration per query class, escalation rules, human approval gates, service objectives, and operating-cost model. Keep unresolved assumptions visible. A decision record should expose what remains uncertain, not hide it behind a vendor label.

These distinctions raise a practical question for CTOs: what evidence would be sufficient to approve the architecture? The sharpest one is whether the benchmark measured the system they actually plan to operate.

Neither, in the RAG-versus-agents sense. BenchmarkQED compared retrieval and RAG conditions with GPT-4o held constant, and LazyGraphRAG won all 96 comparisons. It included no AI-agent arm, so it cannot establish an architecture winner.

The cited research reports no human escalation rate. A flip claim needs arm-level rates, a clear denominator, and a predefined escalation rule.

`LGR_b200_c200`, the larger-budget and smaller-chunk setup, performed best overall. `LGR_b50_c200` edged it for DataLocal queries, while `LGR_b50_c600` had a slight edge for ActivityLocal queries.

Use RAG for approved-document retrieval and explanation. Use AI agents for dynamic retrieval, tool calls, or cross-system actions.

Agentic RAG combines both, with human approval for high-risk steps. See AI/ML training for the related evaluation discipline.

No. In a separate model-level test on clinical guideline questions, Mistral Small 3.2 had the lowest hallucination rate among several models, at 1.3%.

It does not prove superiority over a GPT-4o retrieval benchmark, workflow controls, or escalation policies.

The first benchmark question should be narrow: did the benchmark measure the system you plan to operate?