The Deterministic Core

These are our notes from building and running these systems, and how the work relates to Anthropic’s article on self-service analytics with Claude.

The short version

The hard part of putting an LLM on your data is not access to more tables. It is grounding: making a question resolve to one approved definition before the model writes anything. The shape that gets you there is a deterministic core wrapped in a probabilistic shell, kept honest by evals.

Six components:

• Probable and determined. The model proposes, the code decides.
• Constrain the vocabulary to approved company language.
• Resolve to data entities, not strings, and keep each value’s source.
• Calibrate, then escalate. Resolve when sure, ask when unsure, decline when lost.
• Gate with offline evals before shipping.
• Watch online and harvest every correction into the next cycle.

If you only have two minutes, that is the article. The rest is each piece in practice, and where it came from.

The real bottleneck

When an LLM returns wrong answers about your data, the common reflex is to give it more access. More tables, more context, more retrieval. That rarely fixes it.

Anthropic shared a useful data point in their writeup on building self-service analytics with Claude. They gave the agent direct access to every dashboard, transformation, and analyst notebook in the company, and accuracy barely moved. The hard part was not finding information. It was mapping a question to the right entity. When someone asks for “revenue,” they mean one specific definition out of many.

So the goal of a governed workflow is not a smarter or better-read model. It is to settle that ambiguity before the model generates anything, so “revenue” points to one approved dataset instead of a dozen candidates.

This piece walks through the components that get you there, drawn from systems we build and run. The examples use a semantic layer, a Salesforce agent, and dbt, but the pattern holds whatever your stack: Claude or Gemini, dbt or LookML, Salesforce or anything else. One idea runs through all of it:

An LLM is only as trustworthy as the layer you make it resolve against.

1. Probable and determined

A governed workflow has two layers, and most of the work is keeping each one to its job.

The first layer is probable. It is where the language model works: reading a request, routing it, judging relevance, and working out what someone most likely meant. It is not deterministic. Ask it the same thing twice and you can get two phrasings or two paths.

The second layer is determined. It is code. It resolves a value to an approved entity, checks it against the schema and the rules, and runs. It returns one number, the same number every other system in the company returns.

The rule is simple to state and harder to hold to. The model proposes, and the code decides. The model can suggest, interpret, and route. It does not get to produce the final fact on its own. Each of its guesses is handed to code that either resolves it cleanly or stops.

The model proposes. The code decides.

Anthropic describes the same split. Their semantic-layer queries compile to SQL and run deterministically, while the model handles disambiguation and routing. They warn against letting the probable layer leak into the determined one. When they had a model auto-generate metric definitions, it produced plausible definitions that carried the same ambiguity they were trying to remove. The boundary is the point.

2. Constrain the vocabulary to approved company language

The first thing the code does is keep the model from writing a raw value into the system. Whatever text the model produces is matched to an approved vocabulary, a fixed list of values the company already uses.

In a Salesforce agent we built that creates opportunities from natural language, a salesperson might describe a deal as “solar.” The model does not get to write “solar” into the record. It matches that text to the real picklist in three steps:

• Exact match against the approved list, ignoring case.
• Known alias. A maintained map, so “solar” resolves to “Solar Power World,” for the shorthand people use.
• Fuzzy match with a confidence score. The score decides what happens next, not the model.

This is the same idea as a semantic layer being the default path for every query. If “revenue” resolves to one approved dataset, the ambiguity is gone before the agent ever searches. A controlled vocabulary applies that to every field the model touches: metrics, dimensions, picklist values, entity names.

It only holds if the system enforces it. The agent is routed to the approved list first, and there is no path where it writes a value that is not on the list. That is what separates a workflow that is governed from one that is only documented.

3. Enforce resolution to data entities, not strings

Matching a value to a vocabulary is half of grounding. The other half is resolving a real thing to a real identifier, and recording where that came from.

When the agent hears “the Acme deal,” it does not store “Acme.” It searches live Salesforce, resolves to an account ID, and if more than one Acme comes back, it stops and asks instead of choosing. The warehouse behind it works the same way. One advertiser shows up under five names across five ad platforms, and a conformed dimension joins all five into one identity. The model works with resolved entities, not raw strings it might have invented.

Each resolved value also keeps its provenance, a record of how it got there. We track it as a small ledger: every field carries its value, its source (typed by a person, looked up, defaulted, or inferred), and its status. A required field is written only once it has both a value and a known source.

Provenance is the precondition for writing, not an afterthought.

Anthropic frames this as provenance over confidence. Instead of the model stating a confident answer, show which tier it came from (semantic layer, approved reference, or raw table), how fresh it is, and who owns it. Trust becomes something you can check field by field.

4. Calibrate, then escalate

Look again at the fuzzy-match step, because it holds the most important behavior in the system: what the agent does when it is not sure.

The resolver does not return a best guess and move on. It returns a guess and a confidence, and a threshold sets the path:

1. Above the threshold: resolve and continue.
2. In between: stop and ask a person, showing the top candidates.
3. Below the threshold: decline, and show the valid values rather than invent one.

The middle path is what keeps the system from being confidently wrong. A model will always return something. Calibration makes “I am not sure, which did you mean?” a normal outcome instead of a failure.

This goes beyond picklists. The same shape, resolve when sure, ask when unsure, decline when lost, keeps an analytics agent from picking the wrong revenue definition, or a support agent from citing a policy that does not exist. The threshold is where you set how much risk you trade for autonomy.

5. Gate with offline evals

Grounding lets the system be right. Evaluations are how we know it still is. There are two kinds, and they are easy to confuse. The difference is the request path, not the environment.

Offline evals run off the live path, on inputs you choose, against a known answer. You pick the cases and you have a reference, so you can run them as often as you want before shipping. They answer one question: on the cases we care about, is the new version correct and at least as good as the last one?

Take our email reporting. A few patterns earn their place:

• Golden tests. We lock a known-good result from a past, frozen window, such as a quarter’s email totals, and fail the build if the new number moves outside a set band. If the change is intended, we update the locked number in the same pull request, so the change is visible and reviewed.

• Reconciliation tests. We compare the source against the reporting layer and flag anything that did not make it through. For example, a campaign that appears in the raw click table but never lands in the reporting model gets flagged for review.

• LLM-as-judge, with guardrails. When the thing being checked is itself a model’s output, we check it two ways. The answer has to match the approved list exactly, and it has to clear a confidence score. Below that, a person reviews it, and it is never applied automatically. We run the judge at temperature zero so it repeats.

• CI gates. On every pull request, we compare row counts and column values of the changed models against production before the merge is allowed, so a regression is caught before anyone sees it.

Anthropic sets a clear bar here: offline accuracy should be close to 100%, and a domain is not announced to stakeholders until it clears about 90%. Offline evals are the gate. Nothing ships that has not passed them.

6. Watch online, harvest corrections

Online evals run on live traffic, on inputs you did not choose, usually with no known answer at the moment. Real users ask real questions, and you rarely know the right answer in the moment, so you rely on signals. They answer a different question: in production, on inputs we did not plan for, is it working?

Thumbs up and thumbs down is the obvious one, and it is real. In the Salesforce agent, a rating and an optional note is saved and lands in the warehouse. Explicit feedback is one of three signals:

1. Explicit. The user tells you, with a rating, a “that’s wrong,” or a correction.
2. Implicit. They re-ask, rephrase, copy the result, or give up. No click needed.
3. Inline guardrails. Checks on live output, such as provenance and freshness gating, a confidence floor, or a second model reviewing the answer. One such reviewer raised accuracy about 6% for 32% more tokens and 72% more latency, a questionable benefit for the added complexity.

The point to hold onto is that logs and telemetry are the raw record, not the evaluation. Traces tell you what the agent did: the tools it called, the tokens, the latency, the values it resolved. The evaluation is the judgment you put on top. The work is joining the two, so you can study real sessions and improve the agent’s code, prompts, and resources.

Then the loop that matters. Every online failure becomes an evaluation requirement for the next development cycle, and feeds back into the offline gate. A thumbs-down or a correction becomes a new offline test, so the same failure cannot return. Anthropic does the same, turning each correction into an eval and storing it like telemetry. That is the difference between a demo and something you can rely on.

The operating model

Set the tools aside and the same components remain. None of them are about a smarter model. They are about the structure around it.

1. Probable and determined. A determined core inside a probable shell. The model proposes, the code decides.
2. Constrain the vocabulary to approved company language. The model never writes a raw value.
3. Enforce resolution to data entities, not strings. Names become IDs and conformed dimensions, each with its source.
4. Calibrate, then escalate. Resolve when sure, ask when unsure, decline when lost.
5. Gate with offline evals. Golden tests, reconciliation, and a guarded judge, all green before shipping.
6. Watch online, harvest corrections. Feedback and provenance in production, and every correction feeds the next cycle.

None of this needs a frontier model or a specific vendor. It comes down to treating the layer under the model, the definitions, the conformed entities, the evals, and the provenance, as the product, and treating the model as one capable part on top of it.

The capability race gets the attention. The grounding is what ships.