engineering blog · source date 2026-06-03 · added 2026-08-07 15:52:43 · updated 2026-08-07 15:52:43 · Open original blog
Deep dive — Anthropic: How we enable self-service data analytics with Claude
Source: Anthropic (Claude blog), published 2026-06-03. Authors: Chen Chang, Clement Peng, Justin Leder, Johanne Jiao, Josh Cherry — Anthropic Data Science and Engineering. Original: https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude
Why this matters
Most "chat with your data" demos work on a clean schema and fall apart on a real warehouse. This post is interesting because it is not a demo. It is an internal deployment where roughly 95% of business analytics queries are handled by Claude, and the write-up spends most of its length on the unglamorous parts: data modeling, documentation freshness, skill design, and evaluation.
The most useful claim in it is also the most deflating one for anyone hoping for a model-level fix: accuracy went from 21% to consistently above 95% because of structured skills, and giving the agent access to the full historical query corpus moved accuracy by less than one point. The bottleneck was not information. It was structure.
1. The two-sided failure of self-service analytics
The framing is a dilemma. Grant broad data access and different teams compute "active users" three different ways, so the org loses a shared definition of its own metrics. Restrict access tightly and the agent simply cannot answer most real questions, which pushes people back to filing tickets with the data team.
Anthropic's answer is not to pick a side but to make the correct path the easy path: a governed semantic layer that the agent must try first, with raw table access as the fallback rather than the default.
2. Three failure modes worth naming
The stack is organized around three specific ways analytics agents go wrong:
- concept-to-entity ambiguity: the question says "revenue," and the warehouse has eleven columns that could plausibly mean revenue
- data staleness: the schema changed, the documentation did not, and the agent confidently uses a column that no longer means what the docs say
- retrieval failure: the right context exists somewhere in the repo, and the agent never finds it
These map cleanly onto the three layers of the fix: semantic layer, CI-enforced documentation, and skills-based retrieval narrowing.
3. The semantic layer as a mandatory first path
Metric definitions are compiled into a semantic layer that functions as the required first query path. The goal is stated as "one number" consistency: the same metric resolves identically whether it is asked for in a dashboard, a notebook, or a Claude conversation.
This is the part most teams skip, and it is the part that makes agent answers governable. A semantic layer converts a question of model accuracy ("did Claude write the right SQL?") into a question of resolution ("did Claude pick the right defined metric?"), which is a much smaller and much more checkable problem.
4. Skills: pairwise, and deliberately narrow
Skills are markdown knowledge folders, used in a pairwise architecture:
- a knowledge skill narrows the search space to roughly 30 curated reference files
- runbook skills encode analyst workflows, including reusable analysis patterns
The design decision worth copying is the narrowing. Thirty curated files is a small number, chosen on purpose. The sources of truth are lineage graphs, business-context knowledge graphs, and structured query reference documentation — explicitly not a pile of raw historical SQL.
That choice is validated by their own ablation: adding the full query corpus moved accuracy by under a point. Raw examples look like they should help. Curated structure is what actually does.
5. Documentation freshness is a CI problem
The stated dominant failure pattern across every layer is poor or stale documentation. Their response is mechanical rather than cultural: metadata is colocated with the data models in version-controlled repos, and CI checks enforce that a modeling change updates downstream documentation.
Claude drafts documentation well, but humans still own curation. The engineering claim here is that skill and doc maintenance has to be treated as first-class work with CI enforcement, not as a wiki someone updates when they remember.
6. Evaluation: offline sets, hourly ablations, online adversaries
The validation story has three parts, and it is the part most relevant to eval practice.
Offline, there are two eval types: dashboard-based sets (auto-generated, then human-validated) and long-tail domain evals for the questions dashboards do not cover. Results are stored as warehouse telemetry tagged with the git SHA and the model, so a regression can be attributed to a specific change. Domains gate their own rollout at roughly a 90% threshold.
Ablations run at PR granularity, hourly, testing single-component changes against a fixed eval set. This is what makes claims like "the query corpus contributed under one point" possible at all — without per-component ablation you can only observe that the whole system got better or worse.
Online, adversarial review agents check answers, and provenance footers show where a number came from. The cost is quantified: about 6 points of accuracy for 32% more tokens and 72% higher latency.
7. The trade that is actually stated
That adversarial-review number is the most honest thing in the post. Most vendor write-ups report the accuracy gain and omit the latency cost. Publishing "+6 points for +72% latency" makes it a decision rather than a recommendation: worth it for a board-bound analysis, probably not for an interactive question someone asks in Slack.
What I would reuse
1. Make the governed path the default path — a semantic layer the agent must try first. 2. Curate the retrieval surface aggressively; measure whether more raw context actually helps before adding it. 3. Put documentation freshness under CI, colocated with the models it describes. 4. Pin eval question sets to snapshots and tag results with git SHA and model version. 5. Run per-component ablations, not just end-to-end scores, so you know which layer earned the gain. 6. Gate rollout per domain on an accuracy threshold rather than shipping globally. 7. Price your quality mechanisms in latency and tokens, then decide per surface.
Main caveats
This is a first-party account and all numbers are self-reported. Silent failures — plausible-looking wrong answers — are acknowledged as still hard to catch, which is the honest limit of a 95%-accuracy system: the remaining 5% does not announce itself. Portability also depends on starting conditions; an organization without dimensional modeling or governed metric definitions is not one semantic layer away from this result. And the post recommends human sign-off for leadership-bound outputs, which is a reasonable admission that the loop is not fully closed.
Bottom line
The interesting result is not that an agent can query a warehouse. It is that the accuracy came from data engineering and skill curation rather than from the model, and that they measured which layer contributed what. The transferable artifact here is the evaluation setup — snapshot-pinned question sets, hourly per-PR ablations, SHA-tagged telemetry, and per-domain rollout gates — more than the analytics stack itself.