How to Build an AI That Actually Remembers: A Memory-Retrieval Blueprint You Can Copy

Large language models are stateless. Every message, your model wakes up with no idea who it’s talking to, what happened yesterday, or what the person actually cares about. You can’t fix that by cramming more into the prompt — context windows have a hard limit, and past a certain point, dumping everything makes the model worse, not better: it drowns in noise and burns tokens.

The answer is a memory-retrieval system: a small, always-on core of what matters, plus a smarter on-demand layer that finds the right memories for the right moment.

We are using a similar design for our product, and I will demonstrate it with an assistant example; the architecture turned out so clean and reusable that I think anyone building an AI product should steal it. 

This article is the blueprint — stripped of any product-specific naming, reworked around a use case you can map to almost anything.

Let’s use a concrete example throughout: an assistant for a freelance consultant that remembers her clients, projects, skills, and meeting notes.

The use case: an assistant that knows the consultant

Imagine “Carol,” a freelance brand consultant. She talks to her assistant about clients, runs workshops, juggles five projects, and slowly reveals what works and what doesn’t.

The assistant needs to remember:

• Facts — “Dana is the point of contact at Alpine Health.”

• Skills — “Carol is strong at workshop facilitation.”

• Insights — “Carol loses energy when a project drags past six weeks.”

• Conversations — “Last month she and Dana discussed a rebrand for the wellness portal.”

The hard part isn’t storing these. It’s retrieving them at exactly the right moment — and doing it cheaply enough to run on every single message.

Here’s the architecture that makes it work.

The mental model: memory is a two-tier cache

Don’t think of memory as one big bucket. Think of it the way CPUs think of RAM: two tiers, each with a different job.

Tier 1 — Always on. A small, budgeted set of the most important memories gets baked into the system prompt on every turn. For Carol: her top 15 insights and top 20 skills, ranked by importance. It’s small, stable, and cheap — but it guarantees the model never starts from zero.

Tier 2 — On demand. On every message, the assistant embeds the query and does a semantic search across everything it has ever stored. This is where deep, specific memories surface — the ones too niche to earn a permanent slot in Tier 1.

This two-tier split is the single most important idea in the whole system. Tier 1 gives you baseline relevance; Tier 2 gives you targeted recall. Together they let you serve rich context without blowing your token budget.

Building block 1: embeddings

The engine underneath both tiers is embeddings — turning text into a fixed-length vector of numbers that captures meaning.

// Your embedding function — an API call, a local model, any embedding you have.
async function embed(text: string): Promise<number[]> {
  const res = await fetch('/api/embed', {
    method: 'POST',
    body: JSON.stringify({ text }),
  });
  return res.json(); // e.g. a 384- or 1536-dimensional vector
}

The only things that matter are the contract: use the same model for storing and querying, and keep vectors a fixed length. Pick whatever embedding you already use — text-embedding-3-small, all-MiniLM-L6-v2, anything. The retrieval logic that follows is identical no matter where the numbers come from.

Where those vectors get stored — a database, a search index, an object store — is a storage detail, and it doesn’t change the retrieval structure at all. I’ll leave storage for a separate post; here we only care about how memories are organized and pulled back.

Building block 2: semantic search over a vector store

Once you can embed text, “retrieval” is just math: embed the query, compare it to every stored vector, and keep the closest ones.

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, na = 0, nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na  += a[i] * a[i];
    nb  += b[i] * b[i];
  }
  const mag = Math.sqrt(na) * Math.sqrt(nb);
  return mag === 0 ? 0 : dot / mag;
}

async function remember(id: string, text: string) {
  const vector = await embed(text);
  await db.put('memories', id, { id, text, vector, createdAt: Date.now() });
}

async function recall(query: string, topK = 5, threshold = 0.5) {
  const q = await embed(query);
  const all = await db.getAll('memories');
  return all
    .map(m => ({ ...m, score: cosineSimilarity(q, m.vector) }))
    .filter(m => m.score >= threshold)
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Two knobs matter more than the model choice:

• threshold — a floor that filters out weak matches. Setting it too low floods the context with noise; too high and you recall nothing.

• topK — a cap on how many memories you're willing to spend context on.

This is why semantic search beats keyword search here: Carol can ask “how should I follow up with the health client?” and retrieve a memory that never contained the words “follow up” but means the same thing.

Building block 3: the forgetting curve

Raw similarity isn’t enough. Two memories can score the same cosine similarity, but one is three days old and the other is nine months old. The recent one is almost always more relevant.

So every memory is ranked by a score, not just a similarity:

function recencyBoost(updatedAt: Date): number {
  const ageDays = (Date.now() - updatedAt.getTime()) / 86_400_000;
  if (ageDays < 7)  return 2.0;   // this week
  if (ageDays < 30) return 1.0;   // this month
  if (ageDays < 90) return 0.5;   // this quarter
  return 0;                       // older than that, forget it
}
function score(memory: Memory): number {
  return memory.confidence * 2          // how sure we are this is true
       + salience(memory)               // how much it matters to the person
       + recencyBoost(memory.updatedAt) // how recent it is
       + Math.min(memory.evidenceCount, 2); // how often it came up
}

Notice what this is really doing: it’s a tiny, hand-tuned model of what a human remembers. We remember things that are (a) recent, (b) emotionally significant, and (c) repeated. The score just codifies that.

For Carol, this means a client preference she stated yesterday outranks an equally similar insight from last year — exactly what you’d want.

Building block 4: relatedness, not just similarity

Semantic search finds memories that look like the query. But often what you need is a memory that’s connected to the match, even if it doesn’t resemble the query at all.

That’s where a graph comes in. Store explicit edges between memories — a skill links to the projects where it was used; an insight links to the client it’s about. When a search finds one node, pull in its neighbors.

async function enrich(matches: Memory[]) {
  const seen = new Set(matches.map(m => m.id));
  for (const m of [...matches]) {
    for (const n of await graph.neighbors(m.id, { minWeight: 0.5 })) {
      if (seen.has(n.id)) continue;
      seen.add(n.id);
      matches.push(await load(n));
    }
  }
  return matches;
}

The minWeight floor is the guardrail: only strong links get followed, so enrichment doesn't spiral into dumping the whole graph.

For Carol: she asks about “the wellness portal.” Similarity finds the project insight, and the graph pulls in the skill (facilitation) and the person (Dana) attached to it. Now the assistant can answer with people and skills it never would have found by similarity alone.

Building block 5: two-stage recall for conversations

Raw conversations are the hardest thing to retrieve. They’re long, meandering, and full of filler. You can’t embed whole transcripts and hope for the best.

The solution is a two-stage funnel: summaries route, messages select.

async function recallConversations(query: string) {
  // Stage 1: route to a few candidate conversations via their summaries
  const candidates = await search(query, {
    type: 'conversation', topK: 4, threshold: 0.35,
  });
// Stage 2: search individual messages inside those conversations
  const ids = candidates.flatMap(c => c.messageIds);
  const passages = await search(query, {
    type: 'message', ids, topK: 10, threshold: 0.45,
  });

 return formatExcerpts(passages, { maxConversations: 2, maxChars: 1400 });
}

Stage 1 is cheap and coarse: each conversation is compressed into a short “routing text” (its title, summary, topic tags, and signal labels), embedded once. This narrows thousands of messages down to a handful of conversations.

Stage 2 is fine-grained: embed the actual messages inside those candidates and pick the exact passage that matches. The result is a bounded excerpt — a window around the matching turn — rather than a whole transcript.

This two-stage pattern shows up everywhere once you look for it: coarse first, precise second. It’s the same reason search engines use an index before scanning documents.

Putting it together: the full pipeline

Now assemble all five building blocks into a single flow that runs on every message:

async function buildContext(query: string) {
  // Tier 1 — always-on core, computed once and cached
  const core = await topRankedMemories({ insights: 15, skills: 20 });
  // Tier 2 - on-demand, computed for this specific query
  const semantic = await recall(query, { topK: 5, threshold: 0.5 });
  const enriched = await enrich(semantic);
  const conversations = await recallConversations(query);
  return {
    systemPrompt: core,                     // baked into the system prompt
    dynamicContext: [...enriched, conversations], // appended to this turn
  };
}

The result is a context window that’s deliberately assembled, not dumped:

• A stable core the model always knows.

• Query-matched memories for this exact message.

• Graph neighbors for related context.

• Conversation excerpts for what was said before.

Every layer is bounded — capped counts, thresholds, character budgets — so the context stays lean and focused no matter how much memory accumulates.

Trade-offs and production notes

This system works, but it comes with real decisions worth making deliberately.

Storage is a separate decision. The retrieval logic doesn’t care whether the vectors live in a vector database, a search index, or somewhere else entirely. Keep it decoupled so you can swap the storage layer later without touching the retrieval structure.

Thresholds are everything. The whole quality of retrieval lives in the threshold and topK values. 0.5 is a reasonable default for generic memories, but conversation routing wants something looser (0.35) because summaries are short and less distinctive. Tune per content type, not globally.

Score, don’t just filter. Cosine similarity alone ranks by “looks like the query.” The forgetting-curve score ranks by “matters to the person.” Layer them: filter by similarity, then order by score.

Cap everything. Every stage — Tier 1 counts, Tier 2 topK, enrichment neighbor counts, excerpt character budgets — needs a hard limit. The moment you remove a cap, the context grows unbounded, and the model’s attention degrades.

Invalidate on change. When a memory is edited or deleted, its embedding must be updated or removed too. Wire this into the storage layer so it’s impossible to forget: every save/update/delete touches the embedding store in the same transaction.

Where this applies — “anywhere”

The consultant example is just one shape. The same architecture drops into:

• Customer support — Tier 1 holds the account’s standing facts; Tier 2 recalls the relevant past tickets.

• Coaching & therapy apps — insights and recurring themes replace facts and skills; the forgetting curve is literally how people work.

• Writing assistants — remember the author’s voice, past decisions, and unfinished threads.

• Sales & CRM — the graph becomes the relationship map (person ↔ company ↔ deal).

• Personal “second brain” — your own notes, with two-stage recall over your reading history.

The nouns change. The verbs don’t: store, embed, retrieve, rank, enrich, budget.

The takeaway

You don’t need a vector database, a cloud pipeline, or a custom model to give your AI a memory. You need five ideas:

1. Two tiers — a small always-on core plus a large on-demand layer.

2. Embeddings + cosine similarity — meaning, not keywords.

3. A forgetting curve — rank by recency and salience, not just similarity.

4. A graph — retrieve related memories, not just similar ones.

5. Two-stage recall — coarse routing, then precise selection.

None of it is exotic. All of it is copy-paste-able. And it’s the difference between an assistant that feels like a stranger every morning — and one that feels like it’s been paying attention all along.