Anthropic — How we enable self-service data analytics with Claude

engineering blog · source date 2026-06-03 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog

Self-service analytics fails in two directions: give an agent broad data access and metric definitions diverge across teams; lock access down and it cannot answer the business questions people actually have.
Three specific failure modes drive wrong answers: concept-to-entity ambiguity (mapping a question to the right tables), data staleness (schemas and definitions drifting out of date), and retrieval failure (the agent cannot find the relevant context).
Analytics agents fail silently — a wrong number looks exactly like a right one.

A layered stack: governed data foundations with dimensional modeling and colocated, version-controlled metadata; a compiled semantic layer of metric definitions that is the mandatory first query path, so every surface returns "one number"; and Skills as markdown knowledge folders.
Skills use a pairwise architecture: a knowledge skill narrows the search space to roughly 30 curated reference files, and runbook skills encode analyst workflows and reusable analysis patterns.
Sources of truth are lineage graphs, business-context knowledge graphs, and structured query reference documentation — deliberately not raw SQL retrieval.
CI enforces that a modeling change updates its downstream documentation, which is what keeps the whole thing from going stale.

Reported outcomes: about 95% of Anthropic's business analytics queries automated through Claude at roughly 95% aggregate accuracy, versus 21% accuracy without skills.
Validation is layered too: offline evals on snapshot-pinned question sets, ablations at PR granularity run hourly, and online adversarial review agents plus provenance footers. Adversarial review buys about 6 points of accuracy for 32% more tokens and 72% higher latency — an explicit, quantified quality/cost trade.
The blunt lesson is that structured skills, not raw information access, are what move accuracy: adding the full query corpus moved it by under one point. Stale documentation is the dominant failure pattern at every layer.
Caveat: a first-party account with self-reported numbers, and the approach's portability depends on how complex and well-governed your data model already is.

Comments

No comments yet.