The Three Levels of AI Adoption

In February, a QA report on my own platform ended with a deployment gate in capital letters: DO NOT DEPLOY TO PRODUCTION. Three synthetic personas had spent two rounds interrogating our restaurant-analytics agents, and every claim the agents made was checked against the database. 118 assertions. 88 passed. 20 failed outright. In the worst one, the agent reported 221 orders for a period where the database held 835. It was missing nearly three quarters of a merchant's business and reporting the remainder with total confidence.

I run AI platform work at a large payments company, and that report crystallized something I had been circling for a year. AI adoption comes in three distinct levels, and each level has a different blocker. At Level 1, AI is an information engine, and the blocker is trust. At Level 2, it is a collaborative partner, and the blocker is task allocation. At Level 3, it is an autonomous workforce layer, and the blocker is control. Programs stall when they solve the wrong blocker for the rung they are actually on.

The uncomfortable part, and the reason for this piece: you cannot fix the Level 1 blocker with Level 1 tools. The thing that caught the 221-orders lie was Level 3 machinery. The ladder is an architecture before it is a curriculum.

Level 1 is where everyone starts: summarize, explain, draft, answer. The value is immediate, and so is the failure mode. One confident wrong answer and people write the whole system off, and they are right to. Our confident wrong answers were spectacular. In the same QA run, an agent claimed 5,121 guests for a window where the database held 1,320, a 288% inflation delivered in the same calm tone as every correct answer.

The standard response is training: teach people to verify, warn them about hallucination. We did that, and it moved very little, because verification advice puts the audit burden on the least equipped person in the loop. So we moved the burden into the product. First, the model never writes free-form SQL. Every metric flows through a semantic layer with nine fixed domains (revenue, labor, guest, product, operations, payments, tax compliance, forecasting, server performance), so an answer can be wrong but it cannot be unaccountable. Second, every AI-generated insight carries a provenance record, a frozen Pydantic model that rejects unknown fields:

class QueryProvenance(BaseModel):
    """Provenance for the queries that generated this insight."""
    model_config = ConfigDict(extra="forbid", frozen=True)

    date_from: str          # YYYY-MM-DD
    date_to: str            # YYYY-MM-DD
    skill_name: str         # which analysis produced this insight
    location_ids: list[str] # exactly which locations were queried

Third, progressive disclosure with a floor of raw evidence. Tier 1 is the headline. Tier 2 is the breakdown with charts. Tier 3 is the recommended action. Tier 4 is the raw rows, the record count, the data freshness, and the actual SQL behind a Show Query button. Fourth, the confidence label is mechanical, not vibes: an insight built on 1,000 or more data points is labeled high confidence, 100 or more is medium, below that low. And every insight card carries a trace ID into our observability stack, so when a number looks wrong we replay exactly what the agent did instead of arguing about it.

None of that makes the system right. It makes the system checkable, which is the actual unit of trust.

Level 2 begins when people stop asking for finished answers and start co-producing work: critique this plan, structure this review, turn these notes into a decision brief. The blocker shifts from trust to allocation. Half the users treat the system like a search box and get search-box value. The other half quietly delegate the decision itself, which is worse.

We made the allocation structural instead of cultural. Our insights agent runs four modes (explore, investigate, plan, review), and when it wants to change anything real, it does not act. It emits a proposal artifact. The interesting part is where that rule lives: in the frontend artifact registry, a quick-action artifact declares required handlers, onApproveQuickAction and onDeclineQuickAction, and a required-handler check enforced at compile time keeps any surface from rendering proposals without wiring both. The agent owns retrieval, drafting, and the proposal. The human owns the verb.

That one pattern did more for Level 2 adoption than any training deck we produced, because it answers the allocation question inside the interface. Nobody has to remember who decides. The button knows.

Level 3 is different in kind. You are no longer handing employees an assistant; you are running agents with tools, data access, and bounded autonomy. Most teams get here by accretion: one bespoke agent, then five, each with hand-wired permissions and its own chat page. We did that too, briefly. Then we rebuilt around a rule that now sits at the top of the repo's contributor doc: never build bespoke agents.

Every agent on the platform is a declarative manifest (16 of them today; the analytics product this pattern came from runs 15) compiled through the same ten-step pipeline. Three gates sit in that pipeline, and all of them fail closed. Gate 1 checks whether this user may reach this agent at all, and raises an access error if not. Gate 2 filters the tool catalog by the user's permissions and logs the arithmetic, N of M tools authorized. Gate 3 drops tools whose business domain has not crossed its activation threshold, and here is the detail I care most about: if a manifest explicitly overrides a tool that Gate 3 dropped, the compiler does not quietly honor the override. It refuses to compile the agent. An inconsistent agent does not run in a degraded state. It does not run.

INSIGHTS_AGENT_MANIFEST = AgentManifest(
    agent_type="insights",
    graph_strategy="composed",
    graph_config=GraphStrategyConfig(max_iterations=8),
    model_use_case="REASONING",
    tool_groups=["semantic_query", "insights", "knowledge_base"],
    capabilities=["insight_artifact", "action_plan_artifact", "table_artifact"],
    access=AgentAccess(allowed_realms=["tenant", "platform"]),  # Gate 1
)

Behavior gets the same treatment. We keep 76 golden tests against the chat agent: 27 semantic-layer queries, 14 analysis cases, 9 forecasts, 6 simulations, 6 chart renders, 4 dashboard operations, 10 edge cases. The assertions are not "sounds right." A revenue test pins the number, 102,583.60 with a tolerance of 100. A no-hallucination assertion requires that every figure in the response came out of a tool call rather than the model's imagination. A greeting has to come back under a fixed millisecond budget. And observability is instrumented by hand, trace and span per phase, because the off-the-shelf callback handler could not track streaming latency accurately, and a latency number you cannot trust is a latency number you do not have.

Put the three rungs side by side and the pattern is hard to unsee. Each level fails for a different reason, so each level needs a different piece of infrastructure, and almost none of it is model work.

Read the right-hand column top to bottom and the point makes itself. It is one system. The provenance record that earns Level 1 trust is generated by agents that only exist because a Level 3 compiler agreed to build them.

Back to that February report, because the anatomy matters more than the headline. Iteration 2 of the QA run passed 39 of 55 assertions, 71%. We fixed things. Iteration 3 passed 49 of 63, 78%. All 20 hard failures across both rounds were the same category, data issues: the agent's claimed metric disagreed with the database. Order counts missing joined orders. Guest counts missing deduplication. And a subtler one that generated 10 warnings: asked about a multi-period range, the agent would answer for the most recent period only, 84,800 against a true full-range 159,808, off by 46.9% while being locally correct.

The gate held. Every requirement box on the report sat unchecked: order variance under 2%, guest variance under 2%, zero revenue warnings, an iteration-4 pass rate above 85%. The recommendation attached a four-day remediation timeline, fix, regress, stage, then production, and until those boxes were checked, nothing shipped. That is the cost, and I will pay it every time, because the alternative is board math: a merchant who catches your product misstating their order count by 73% does not file a bug ticket. They call their rep, and the conversation is about churn, not software.

Notice what did the catching. The user-facing provenance makes any single answer checkable, but no merchant is going to diff 118 assertions against a database. The persona harness, the variance thresholds (3 to 5% warns, above 10% fails), the deployment gate: that is Level 3 infrastructure doing Level 1's job. The ladder is load-bearing in both directions.

Most adoption programs report seats and sessions. Usage can be sky-high while every interaction stays at Level 1, and plenty of executives are currently presenting exactly that as transformation. The measure that matters is ladder position: what fraction of the work moved from drafting, to governed collaboration, to bounded autonomy with gates that can say no.

So here is the claim I will defend: if your platform cannot refuse to build an agent, you are not at Level 3. You are at Level 1 with more steps. The refusal is the level. A compiler that fails closed, tests that pin numbers to a tolerance, a gate that freezes a release for four days while people ask why: that is what an autonomous workforce layer actually costs. Teams unwilling to pay it should stay at Level 2 on purpose, and I mean that as respect, not concession. The worst position on the ladder is claiming a rung your architecture cannot hold.