What Memory Actually Changed

The hardest acceptance criterion in my memory system has nothing to do with remembering. It reads, roughly: start a session as agent Y, share a fact from agent X to Y, revoke the share in the same session, and the very next message must not see the fact. Same session. No restart. The system prompt has already been compiled and cached with the shared fact baked in, so passing this test means refreshing the memory slot, rebuilding the prompt, and prepending a fresh system message on every turn, per agent.

Every vendor demo of agent memory shows the remember path. A user mentions their kid's peanut allergy in March and the agent recalls it in June, applause. Nobody demos the revoke path, because the revoke path is where the architecture actually lives. Recall is a retrieval query. Revocation on the next message is a statement about how your entire runtime handles caching, scope, and prompt assembly. If you want to know whether a memory feature is real, ask the vendor to run the revoke test in front of you.

My original memory plan tried to ship three things in one lifecycle: topic-scoped policy, cross-agent sharing, and a knowledge graph. I put it through adversarial external code review. Seven rounds closed roughly 27 findings, and then the cross-agent routing collided with runtime reality and surfaced 4 more high-severity findings on top. The conclusion I wrote into the plan afterward is the most useful sentence I produced that quarter: cross-agent memory sharing is a runtime-plumbing refactor with a feature on top.

That failure cost me the mega-plan. I split it into separate lifecycles and wrote the expensive lessons directly into the spec with an instruction to future builders: do not strip these out for concision. The review trail did not stop at seven rounds either. The plan documents corrections from rounds 12 and 13, and code comments in the retrieval module reference findings as late as round 17. The feature itself, two tables and a UNION query, was a week of work. The plumbing to make it safe consumed everything else.

Concretely, the review found seven integration gaps that had to be fixed before any feature code landed, and they are all embarrassingly mundane. The runtime context did not carry the agent's type or the household member's identity through the tool-invocation chain. The system prompt baked memory into a frozen string at compile time and cached it across turns. The main tool-context path and the fallback path had diverged, so fixing one missed the other. Headless runs deliberately set the user ID to the agent's own ID, which broke the audit log's assumption that a human UUID sits behind every read. None of these are memory problems. All of them block memory.

So the shipped plan has a foundational work package, WP0, with 7 substrate acceptance criteria that are pure context plumbing, and a hard rule: the feature work packages (WP8 through WP13) do not start until all 7 are green. The WP0 criteria include stamping agent_type onto the runtime context at manifest dispatch, carrying member_id through both tool-context paths, rebuilding the prompt on the session-cache hit path, propagating both fields through agent-to-agent delegation, and adding an explicit execution_mode flag so headless runs identify themselves structurally instead of by string-matching user IDs.

That last one is worth a beat. The first design detected headless runs by checking whether the user ID equaled the agent type. Review killed it, correctly, because the orchestration scheduler spawns instances with IDs like coordinator-abc123 that would slip past the string check and write garbage into the audit log. The fix is a boring literal field, interactive or headless, set at the two entry points. Boring fields that make invalid states unrepresentable are most of what production memory is.

The sharing layer is two tables. The first, memory_fact_shares, materializes grants: fact F is visible to agent Y via grant type G, where G is constrained at the database level.

grant_type IN ('explicit', 'agent_group', 'household', 'organization')
-- private is intentionally absent: private means zero rows
UNIQUE (fact_id, shared_with_agent_id, grant_type)

Explicit and agent-group grants materialize as rows at write time. Household and organization scope resolve at retrieval, to avoid row explosion. Retrieval itself goes through a single unified path that runs a 4-branch UNION: the agent's own facts, plus explicitly shared, plus household-scope, plus org-scope, deduplicated and re-ranked. Both the context loader and the retrieval tools route through this one helper, because two retrieval paths means two security models and eventually one of them is wrong.

The second table, memory_access_log, records every retrieval that returned at least one fact the reader did not own: one row per distinct grant type in the result set, with the reading agent, the human user on whose behalf it read, and a fact count. Retrievals of purely your own facts write zero rows, so the log stays signal-dense. The fact_id column deliberately has no foreign key, so audit rows survive fact deletion. You can delete a memory; you cannot delete the record that it was read. One more scope rule came out of review: the search tool used to accept a member_id argument, which meant the model could ask for another member's scope. That parameter is gone. Scope now comes from the runtime context only, and the model cannot override it. All of it is covered by 11 dedicated test files for this phase alone, including the revoke, delegation, headless, and upsert paths.

Two review findings deserve their own paragraph because they generalize to anyone building this. First, the nightly consolidation job, which merges duplicate facts, grouped only by user and organization. It would happily merge facts belonging to different household members, drop the member scoping on the merged fact, and never touch the share rows, quietly widening who could see what while everyone slept. Consolidation now groups on member and topic too, carries both onto the merged fact, and re-reconciles the share rows afterward. Second, a legacy visibility path in the old memory service could swap to a shared proxy user and bypass the share tables entirely. Rather than refactor a legacy service mid-flight, the migration pins that legacy visibility to private for every agent on the new path, closing the bypass with one field. Every background job that touches memory is a policy engine whether you designed it as one or not.

Preference memory is half the story. The half nobody markets as memory is knowledge capture, and in our analytics product it did more work. Every insight the system generates flows through one generation service regardless of origin, and origin is a constrained field with exactly four values: chat_pin (a user pinned an agent answer), user_explore (an analyze button on a dashboard), automated_run (a scheduled analysis), and manual_trigger (a human asked the hub for it). Four values, enforced by a database check constraint, and the model cannot invent a fifth.

Saved insights get scored for priority with a Financial Urgency Score, 0 to 100: severity contributes up to 30 points, extracted dollar impact up to 35, metric velocity up to 20, and domain time-sensitivity up to 15. A labor finding scores higher urgency than a menu finding at the same dollar value because labor decisions expire at the next schedule. When an insight is saved, the service also writes baseline metric snapshots of the findings, so three weeks later the system can answer whether the thing you acted on actually moved.

And those outcomes feed back. The learning layer hashes each recommended action (SHA-256 over the normalized text) and matches it against actions with recorded prior outcomes. A proven action with a boost factor above 1.5 gets promoted to high priority with its evidence appended to the rationale. An action with poor history gets demoted to low priority with a caution note, and an action marked blocked is removed from the recommendation list entirely before the user sees it. That is memory changing behavior, not memory as a parlor trick.

The cost side, since memory is a recurring tax rather than a one-time build. Fact extraction runs after conversations, so it runs constantly, and it therefore runs on the cheapest capable model tier, gated hard: no extraction under 2 messages, analyze at most the last 10, default confidence 0.8, dedupe before insert. Injection is capped at 20 facts so the memory context cannot silently eat the prompt budget. The expensive part was the one-time plumbing; the part that compounds forever is per-conversation extraction, and that is the line item to engineer down.

Hygiene is also scheduled, because unmaintained memory converges on noise. Facts decay 1% in confidence per day unaccessed and get a 0.1 boost when actually used. A monthly recalibration pass detects drift, proposes archiving facts unused for 90 days or more with confidence under 0.3, merges near-duplicates, and flags contradictions for the user to resolve. Users can browse every stored fact, delete any of them, export the lot, or turn the whole system off. A memory the user cannot inspect and cannot kill is surveillance with a friendlier name.

What memory did not change: answer quality on any single question. A model with the right context in the prompt answers the same whether that context came from a memory system or from the user retyping it. It did not reduce cost; extraction and retrieval add spend on every conversation. And it did not make agents smarter in any measurable way I would defend in front of a board.

What it did change: who has to repeat themselves (nobody), what happens when trust is withdrawn (revocation lands on the next message), whether cross-agent access is a policy or a promise (it is a table with a CHECK constraint and an audit log), and whether recommendations improve from history (proven actions rise, failed ones are blocked). Notice that three of those four are governance properties. That is the honest shape of the feature.

So here is the stance: if your memory feature has no revoke test and no access log, you built a cache with good PR. Most of what shipped as agent memory in 2026 is vector recall over chat history, which is fine, useful even, and is to a memory system what a sticky note is to a ledger. The market will not distinguish them until the first incident where a shared fact showed up somewhere it should not have, and then the only vendors left standing will be the ones who can produce the access log.