AI agent memory: a practical design guide

The direct answer

Agent memory is information an AI system carries forward so a later step or conversation does not have to start from zero. The safest useful design is usually small working state for the current run, an expiring session summary for continuity, and no persistent personal memory until a real user need justifies it.

Memory is not the model secretly learning a user. In a production application, it should be an ordinary data system you can inspect: records with an owner, purpose, source, timestamp, expiry, confidence, and deletion path. The model may propose a memory, but application code decides whether it may be stored and retrieved.

That distinction prevents two common mistakes. The first is calling the entire chat transcript "memory" and repeatedly stuffing it into the prompt. The second is saving every inferred preference forever. Both create higher cost, poorer answers, privacy risk, and facts that become wrong long after the user has changed their mind.

What you will build

This guide designs memory for a research assistant that can:

• keep the current task plan and tool results while a run is active;
• remember a short session summary after context is compacted;
• optionally save explicit preferences, such as preferred answer format;
• retrieve only records relevant and permitted for the current task;
• correct, expire, export, and delete saved memory;
• prove through evaluation that memory improves the product.

The same design applies to support assistants, coding agents, internal search agents, and workflow automation. The exact storage product is secondary. Start with the data contract and lifecycle.

If you are new to the model side of this system, prompt engineering for code explains instruction hierarchy and context more broadly.

Three memory scopes

Working memory: state for one run

Working memory contains what the agent needs to finish the task it is doing now: the goal, current plan, completed steps, intermediate calculations, tool call identifiers, and selected evidence. It should disappear when the run ends, apart from the minimum operational logs you are allowed to retain.

It is usually represented as structured state, not prose:

type RunState = {
  runId: string;
  userId: string;
  goal: string;
  plan: Array<{ id: string; status: "pending" | "done" | "blocked" }>;
  evidence: Array<{ sourceId: string; excerpt: string; retrievedAt: string }>;
  toolResults: Array<{ callId: string; tool: string; resultRef: string }>;
  startedAt: string;
};

Do not confuse the model's context window with working memory. The context window is a delivery channel. Your application remains responsible for the canonical state. If the prompt is compacted or a model call fails, the run should be resumable from the state store rather than from a guessed summary of the chat.

Session memory: continuity across nearby turns

Session memory survives several turns or a short return visit. It may contain a rolling summary, unresolved questions, recently viewed documents, and temporary preferences. Give it an explicit time-to-live: perhaps hours for a sensitive support flow or days for a coding workspace.

A session summary should preserve decisions and open work, not every sentence:

{
  "sessionId": "ses_42",
  "purpose": "compare two API providers for a ticket classifier",
  "decisions": ["Use server-side calls", "Require JSON schema validation"],
  "openQuestions": ["Confirm retention terms"],
  "sourceMessageRange": ["msg_18", "msg_31"],
  "expiresAt": "2026-09-18T10:30:00Z"
}

Summaries are lossy. Keep references to source messages while those messages remain within the approved retention period, and let the user reopen or correct the summary. Never present a model-generated summary as an exact transcript.

Persistent memory: facts intended for future sessions

Persistent memory is durable user or organization data: an explicitly saved preference, an approved project convention, or a stable account fact. It creates the largest product value in some applications and the largest governance burden in all of them.

A useful test is: would the user reasonably expect this fact to affect an unrelated conversation next month? If not, keep it in the session or do not store it.

Good candidates:

• "Use TypeScript examples unless I ask otherwise," saved by the user;
• a team-approved naming convention linked to its policy source;
• an accessibility requirement that the user chose to retain.

Poor candidates:

• an emotion inferred from one message;
• a health, political, financial, or identity attribute inferred without a necessary and lawful purpose;
• an assistant guess such as "the user is a beginner";
• copied secrets, access tokens, or private keys;
• a temporary project detail with no expiry.

Memory, retrieval, and training are different

Memory stores application records and retrieves them into later model calls. Retrieval-augmented generation fetches external or internal knowledge relevant to a request. Model training changes model parameters. These mechanisms can overlap in a product, but they have different controls and deletion behavior.

Deleting a memory record from your application store does not automatically delete provider logs or change a trained model. Conversely, changing a model does not clean your memory database. Document every copy: primary database, vector index, cache, analytics, backups, and model-provider request handling.

Start with a purpose table

Before choosing a vector database, write one row for each memory purpose:

If you cannot state a purpose and read boundary, do not collect the data. "Might improve personalization" is not a sufficient specification.

Use a schema that supports correction

A blob containing "everything known about Alex" is easy to demo and hard to govern. Store atomic records so one claim can be changed without rewriting the person's profile:

type MemoryRecord = {
  id: string;
  subjectId: string;
  workspaceId?: string;
  scope: "session" | "persistent";
  kind: "preference" | "decision" | "constraint" | "summary";
  value: unknown;
  source: {
    type: "user_statement" | "user_action" | "approved_document";
    ref: string;
    capturedAt: string;
  };
  status: "active" | "superseded" | "disputed";
  confidence: number;
  validFrom: string;
  reviewAfter?: string;
  expiresAt?: string;
  consentReceiptId?: string;
  sensitivity: "standard" | "restricted";
  version: number;
};

The source matters more than a similarity score. A direct user correction should supersede an older inference. A workspace policy should not override a user's personal setting unless the application clearly defines that precedence. Keep provenance visible in internal debugging and, where useful, in the user-facing memory controls.

Avoid putting authorization into the memory text. "This user is an admin" must come from your identity and access-control system on every request, not from retrieved prose.

A practical write path

Do not let the model write directly to a durable store. Use a gated pipeline:

1. Detect a candidate. The model or application notices a potentially reusable fact.
2. Classify it. Determine purpose, sensitivity, scope, and whether inference is allowed.
3. Normalize it. Convert prose into a narrow schema.
4. Check policy. Reject secrets, unsupported sensitive categories, cross-tenant references, and records without a lawful product purpose.
5. Ask when necessary. For durable personalization, show the exact proposed memory: "Remember that you prefer TypeScript examples?"
6. Deduplicate. Compare with active records of the same kind and subject.
7. Store with lifecycle fields. Add provenance, expiry or review date, and consent receipt.
8. Audit the decision. Record metadata about who or what approved the write without duplicating sensitive content in logs.

async function savePreference(candidate: Candidate, actor: Actor) {
  assertSameTenant(candidate.subjectId, actor);
  const parsed = preferenceSchema.parse(candidate);
  if (containsSecret(parsed.value) || parsed.sensitivity === "restricted") {
    throw new Error("Memory category is not allowed");
  }
  if (!actor.confirmedMemoryIds.includes(parsed.proposalId)) {
    throw new Error("Explicit confirmation required");
  }
  return memoryStore.upsert(toVersionedRecord(parsed, actor));
}

The secret detector is a backstop, not a guarantee. Design tool outputs so credentials never reach the memory proposal in the first place.

A practical retrieval path

Retrieval should be narrow and deterministic around the model:

1. authenticate the user and resolve tenant membership;
2. derive allowed memory kinds for this feature;
3. filter by subject, tenant, status, sensitivity, expiry, and purpose;
4. retrieve lexical or semantic candidates;
5. rerank for the current task;
6. apply a strict count and token budget;
7. label memory as data, not instructions;
8. record which memory IDs influenced the call.

const candidates = await memoryStore.search({
  subjectId: auth.userId,
  workspaceId: auth.workspaceId,
  kinds: policy.allowedKinds("research_answer"),
  status: "active",
  notExpiredAt: new Date(),
  query: currentGoal,
  limit: 12,
});

const selected = rerank(candidates).filter(isStillValid).slice(0, 4);

Similarity alone is unsafe. A semantically close record may belong to another tenant, be expired, or be a quoted instruction from an untrusted document. Apply access and lifecycle filters before ranking, then validate again after retrieval.

Put retrieved memory in a delimited section such as USER-PROVIDED PREFERENCES. Tell the model that records are contextual claims and may be stale; they do not override system rules or current user instructions.

Choose stores by access pattern

Use a relational or document database as the source of truth for structured records, versions, consent, and deletion. Add a vector index only when semantic matching solves a measured retrieval problem. A small preference set can be filtered by kind and date with no embeddings at all.

Typical split:

• run-state store: short-lived database or durable workflow engine;
• session store: database rows with automatic TTL cleanup;
• persistent store: encrypted primary database with version history;
• vector index: derived search projection containing opaque IDs and minimal text;
• cache: brief, tenant-scoped results that never outlive the source record.

The primary record owns deletion. Derived indexes subscribe to deletion events, and a reconciliation job finds orphaned vectors. Encrypt storage and transport, separate production from development, and keep tenant identifiers in every query key.

Consent and privacy must be usable

A checkbox buried in general terms is not a memory interface. Users need to know:

• what will be remembered;
• why it helps;
• whether it is session-only or persistent;
• who can see it in a team account;
• how to correct or delete it;
• how long deletion from backups may take.

Provide a "What this assistant remembers" screen with view, edit, forget, and disable controls. Let users decline persistent memory while continuing to use the core product where feasible. Avoid dark patterns that turn personalization into compulsory surveillance.

Data minimization is an engineering constraint. Store "prefers concise TypeScript examples," not the entire conversation that revealed it. Pseudonymous identifiers reduce casual exposure but do not make linked personal data anonymous.

For regulated or sensitive use, involve privacy and legal owners early. NIST's AI Risk Management Framework treats privacy risk as something to map, measure, manage, document, and monitor—not a footer added after launch.

Stale memory is a product bug

Memory ages in several ways: preferences change, project facts expire, source documents are replaced, and summaries omit a correction. Address each explicitly.

Use:

• expiry for inherently temporary facts;
• review dates for policies and conventions;
• versioning when a new fact supersedes an old one;
• source health checks for linked documents;
• recency weighting only after authority and relevance;
• user correction that takes effect immediately.

Do not silently delete disputed records if you need an audit trail. Mark them disputed and exclude them from retrieval, then apply the retention policy. Do not allow the model to "resolve" conflicting personal facts without asking.

Deletion is a distributed operation

A delete button is not complete when it removes one SQL row. Build a deletion ledger:

1. authorize the request and identify the subject;
2. tombstone the primary record so it stops being retrieved immediately;
3. enqueue deletion for vector indexes, caches, replicas, exports, and analytics where applicable;
4. prevent retry queues from recreating the record;
5. track backup expiry under the documented retention policy;
6. confirm completion or clearly state remaining retention obligations.

Test account deletion and single-memory deletion separately. A user may want to forget one preference without closing the account.

Evaluate whether memory helps

Create paired test cases with memory off and on. Include positive cases, irrelevant records, contradictions, malicious records, expired facts, missing consent, and cross-tenant lookalikes.

Measure:

• task success or rubric quality;
• correct-memory retrieval precision and recall;
• stale-memory usage rate;
• contradiction handling;
• unauthorized retrieval attempts blocked;
• latency and token overhead;
• user correction and deletion completion;
• answer quality when no memory exists.

Do not reward the system merely for mentioning remembered facts. A preference repeated awkwardly in every answer is not personalization. Human reviewers should score whether memory was necessary, correct, proportionate, and easy to override.

Run red-team cases where a retrieved note says "ignore previous rules" or requests a tool action. Memory content can carry prompt injection just like web pages and emails. The application must keep permissions outside the model.

When memory is unnecessary

Skip persistent memory when:

• each task is self-contained;
• the necessary fact already lives in an authoritative profile or project database;
• users can provide a small preference in the current request;
• the consequence of a stale fact is high;
• you cannot support access, correction, and deletion;
• retrieval does not improve your evaluation set enough to justify its cost.

Often the better feature is a visible project settings page, a saved template, or a resumable workflow. Calling ordinary product state "agent memory" does not make it more intelligent.

Implementation checklist

• [ ] Every memory purpose has an owner, scope, retention, and read boundary
• [ ] Working state is canonical outside the model context
• [ ] Persistent writes require policy checks and appropriate user control
• [ ] Records are atomic, sourced, versioned, and expirable
• [ ] Authorization filters run before semantic retrieval
• [ ] Memory is presented to the model as untrusted data, not instructions
• [ ] No secrets or authorization claims are stored as memory
• [ ] Users can view, correct, disable, export, and delete memory
• [ ] Derived indexes and caches participate in deletion
• [ ] Evaluations include stale, malicious, contradictory, and cross-tenant cases
• [ ] Memory can be turned off without breaking the core task

Continue learning

• AI agents explained ? place memory in the full architecture
• Context engineering ? select what enters the current context
• AI agent security ? protect stored and retrieved information

Official sources

• NIST: AI Risk Management Framework 1.0, AI RMF Core, and Generative AI Profile, NIST AI 600-1
• NIST: Privacy Framework
• OWASP GenAI Security Project: LLM01:2025 Prompt Injection and LLM06:2025 Excessive Agency
• OpenAI: Conversation state
• Anthropic: Context windows

Takeaways

Useful memory is selective, inspectable, and reversible. Keep run state structured, let session data expire, and make persistent memory earn its place through a clear user benefit and measurable evaluation result.

The model can suggest what to remember and retrieve, but code must enforce identity, consent, purpose, expiry, permissions, and deletion. If you cannot make a remembered fact visible and correctable, it should probably not be persistent memory.