Skip to content

Compounding Memory: Pipeline

This doc is a walking tour of how raw memories become wiki pages — the harness’s distillation layer that turns scattered observations into structured, queryable pages. It’s written to be read top to bottom; each step covers one thing the pipeline does, why it does it, and what can go wrong. The pipeline is the load-bearing implementation of compounding memory’s product promise: reproducible page outputs from the same input set (Reliability) and a complete provenance trail from every page back to the source memories that wrote it (Traceability).

The first half is plain-language narrative. The second half (“Under the hood”) has the code paths, SQL, and prompt contracts for anyone who needs to modify or debug the pipeline itself.

There are two compile passes that run in the same job:

  1. Leaf pass. Reads new memories, decides which concrete entity pages should be created or updated. Produces pages like Momofuku Daishō, Franklin Barbecue, Biscuit (dog). This is where most wiki content originates.
  2. Aggregation pass. Runs after the leaf pass completes. Looks across the agent’s recently-changed pages and decides which hub topics deserve rollup sections (e.g. Toronto with a Restaurants section) and which sections have become dense enough to promote into their own page (e.g. a Restaurants section on Austin promotes into a standalone Austin Restaurants topic page).

Both passes are driven by one compile job. The job is always scoped to one (tenant, owner) pair. If anything fails, the cursor stays where it was and the next job safely replays.

A job row lands in wiki_compile_jobs in one of four ways:

  • After a retain. The memory-retain Lambda fires maybeEnqueuePostTurnCompile after every successful retainTurn. Jobs dedupe on a 5-minute bucket, so a burst of turns collapses into one queued compile.
  • Admin trigger. The compileWikiNow(tenantId, ownerId) mutation enqueues a job directly.
  • Lint sweep. The nightly lint pass enqueues promotion jobs for unresolved mentions that have crossed the ≥ 3 mentions in 30 days threshold.
  • CLI. thinkwork wiki compile wraps the admin mutation for operator use.

wiki-compile Lambda claims the pending job (FOR UPDATE SKIP LOCKED) and reads the cursor for that (tenant, owner) — a bookmark of the last memory the worker processed.

Then it reads the next 50 memories from Hindsight where updated_at > cursor. Hindsight is the canonical memory substrate; the wiki is a derived store.

Step 1.3 — The planner reads memories in context

Section titled “Step 1.3 — The planner reads memories in context”

The batch of memories plus the scope’s existing pages plus any open unresolved mentions get handed to the leaf planner — Bedrock via the Converse API, defaulting to Claude Haiku 4.5 but swappable via the modelId event parameter.

The planner sees:

  • The new memories, in full.
  • Titles, slugs, and summaries of every active page in scope.
  • Every open unresolved mention with its accumulated context.

The planner is explicitly biased toward caution. Holding an alias as “unresolved” is almost free; creating a spurious page is expensive to undo. When in doubt, it holds.

The plan is a small JSON document with six arrays:

Page updates. “This existing page should have its Visits section rewritten given this new memory.”

New pages. “There’s enough evidence for a new page about Taberna dos Mercadores. Here’s a title, sections, aliases.”

Unresolved mentions. “Someone named ‘Chef João’ showed up but doesn’t justify a page yet — hold the alias.”

Promotions. “Chef João has been held for months and three new memories push him over the threshold. Promote to a real page, here are the sections.”

Page links. “Taberna dos Mercadores should link to Lisbon. Both pages exist or are being created in this plan.”

Parent section updates and section promotions are also emitted by the planner contract but the leaf planner rarely fills them — the aggregation pass does that work.

The planner never writes to the database. It produces a plan; the compiler applies it.

Step 1.5 — The section writer patches one section at a time

Section titled “Step 1.5 — The section writer patches one section at a time”

For every section the plan wants to update, the compiler calls a second Bedrock model — the section writer. It’s narrow: take the existing section markdown, the planner’s proposed body, and the specific memories cited — produce final markdown for this one section only.

Before calling, the compiler checks isMeaningfulChange — a cheap edit-distance filter that skips sections when the proposal is effectively identical to the stored body. No point burning tokens polishing prose that already says the same thing.

Section titled “Step 1.6 — Hallucinated IDs and stray wikilinks get filtered”

Before the model output is trusted:

  • isValidUuid() gates every page-id and mention-id lookup. Smaller models sometimes emit truncated ids like "814e6b70"; Postgres would reject those as invalid UUIDs and kill the job. The guard silently skips invalid rows instead.
  • stripWikilinks() removes [[Obsidian-style]] syntax from section bodies at the repo boundary. Models were told not to emit it, but they slip; the strip runs inside upsertSections so every write path is covered.
  • linkifyKnownEntities() rewrites **Entity Name** bolded mentions into real [**Entity Name**](/wiki/<type>/<slug>) markdown links when the name matches a page in scope. Longest-title-first match prevents shadowing.
  • Invalid pageLinks (bad page types, blank slugs) are dropped with a CloudWatch warning rather than failing the whole plan.

All five filters exist because some model has produced the exact failure mode we’re guarding against. Prompts are the first line of defense; these repo-level guards are the second.

Step 1.7 — Provenance gets written with deliberate care

Section titled “Step 1.7 — Provenance gets written with deliberate care”

Every section carries a list of which memories were used as sources. The rule is strict: only the specific memories the planner cited for this section. Not “all the memories in the batch.” Not “every memory that looks related.”

An earlier version of the compiler fell back to “use the whole batch as sources” when the planner omitted citations. The result was nonsense: a memory about Austin weather got written as a source for unrelated CRM pages that happened to be in the same batch. The reverse-lookup (MemoryRecord.wikiPages) then returned those CRM pages when you tapped the weather memory. Everything downstream broke.

The fix: zero cited memories means zero source rows. Blank provenance is always preferable to wrong provenance. resolveCitedRecords encodes this rule and no path bypasses it.

Before creating any page the leaf planner proposed in newPages[], the compiler tries to merge the proposal into an existing page. The match runs in two passes:

  1. Exact alias match. The proposed title + aliases are normalized and looked up against wiki_page_aliases in the same scope. A hit turns the proposal into a pageUpdate against the existing page and bumps alias_dedup_merged.
  2. Trigram fuzzy fallback. If no exact alias matches, the compiler queries wiki_page_aliases.alias and wiki_pages.title with Postgres pg_trgm similarity. Matches at similarity() >= 0.85 are accepted only when the matched page has the same type as the proposal — entity into entity, topic into topic. Cross-type fuzzy matches are always rejected. Successful fuzzy merges bump fuzzy_dedupe_merges, tracked separately from exact because fuzzy is the over-collapse risk.

The repo helper is findAliasMatchesFuzzy in packages/api/src/lib/wiki/repository.ts; it sits on the pg_trgm GIN indexes added by migration 0015_pg_trgm_alias_title_indexes.sql. The helper’s internal try/catch makes a missing pg_trgm extension a graceful no-match rather than a job failure.

Without this step, planner-emitted "Paris, France" creates a second page when "Paris" already exists. With it, the proposal collapses into the existing topic/paris page and its new sections land there.

After the leaf plan applies, two deterministic emitters write additional reference links. Both are LLM-free, both gate on WIKI_DETERMINISTIC_LINKING_ENABLED (default true, pinned in terraform next to WIKI_AGGREGATION_PASS_ENABLED).

  • emitDeterministicParentLinks — reads parent-expander candidates (Step 2.1’s deriveParentCandidates*). For each city / journal candidate with an exact-title match in scope, it writes a reference edge from every leaf entity page sourced by that record into the parent topic. Context tag: deterministic:city:<slug> or deterministic:journal:<slug>.
  • emitCoMentionLinks — reads wiki_section_sources (not the leaf planner’s pageLinks) to find entity pages that share a memory_unit source. Emits reciprocal reference edges between those pages, capped at 10 directed edges per memory (slug-ascending order). Context tag: co_mention:<memory_unit_id>.

Both emitters persist through upsertPageLink’s onConflictDoNothing on the (from_page_id, to_page_id, kind) unique index, so a replay is a no-op. The context-tag format exists so targeted rollback can drop only the deterministic rows without touching planner-emitted references.

When the flag is off, the job records deterministic_linking_flag_suppressed: true in metrics and both links_written_* counters stay at 0. The rollback SQL lives in docs/metrics/wiki-link-density.md in the repo.

Every job records duplicate_candidates_count — the count of (owner_id, title) groups with more than one active row, scoped to the tenant. A rising value means the dedupe step is losing ground (fuzzy gate too strict, new aliases slipping through). It’s the first signal operators should watch after flipping WIKI_DETERMINISTIC_LINKING_ENABLED.

Once the plan applies cleanly, the worker moves its bookmark to the last memory it processed. If anything failed mid-batch, the bookmark stays and the next worker replays. Safe to retry.

Keep fetching 50-memory batches until: cursor drains, hard cap (500 records / 25 new pages / 100 section rewrites), or Bedrock rate-limits. Then the job finishes and emits metrics.

Large backfills naturally spread across multiple jobs — cursor-preserving means no replay hazard.

The leaf pass ends with fresh entity pages in the scope, but they’re not yet organized. Marco now has 90 restaurants, events, and places — but there’s no “Austin” topic page tying his Austin ones together, no “Restaurants” hub, no rollups you can scroll through. That’s what the aggregation pass builds.

Two deterministic expanders scan the current job’s records + the scope’s recently-changed pages to suggest hubs that obviously need to exist:

  • deriveParentCandidates(records) — pulls city, journal, and tag clusters out of raw record metadata. Records with place.city = "Austin" group into an Austin candidate. Shared journal_ids group into a trip topic. Repeated tags group into a tag-collection hub.
  • deriveParentCandidatesFromPageSummaries(pages) — scans each scope page’s summary text for city names. Catches “Korean restaurant in Toronto.” style summaries where raw metadata isn’t in the current batch. Walks comma-separated postal addresses from the end to pull Toronto out of ", Toronto, ON M6J 1G1, Canada.".

Both expanders are cheap — no LLM, no extra DB hits beyond the already-hydrated pages. They exist so the LLM gets grounded seed hubs instead of inventing them.

A second Bedrock call, with its own system prompt, receives:

  • Every recently-changed page in scope (with summary, sections, existing aggregation metadata).
  • Parent candidates from Step 2.1.
  • Link neighborhoods (inbound link counts).

The prompt is strict about membership: every page listed under a hub must have visible evidence in its summary. An earlier version hallucinated an Austin Restaurants hub containing Toronto, Tokyo, and Bogotá restaurants — the prompt now requires geographic proof per linked page and refuses to create a hub when fewer than 3 pages clearly match.

Output: parentSectionUpdates, sectionPromotions, hub newPages, hierarchy pageLinks.

Step 2.3 — Rollup bodies are rendered deterministically

Section titled “Step 2.3 — Rollup bodies are rendered deterministically”

When a parentSectionUpdate carries linked_page_slugs, the compiler skips the section writer and renders the body with renderRollupList:

- [**Franklin Barbecue**](/wiki/entity/franklin-barbecue) — BBQ joint in Austin, TX.
- [**Momofuku Daishō**](/wiki/entity/momofuku-daisho) — Korean-inspired restaurant in Toronto.

Why deterministic: the LLM was previously writing prose (“Franklin Barbecue is a BBQ joint…”) that mentioned entity names as plain text with no hyperlinks. A structured list with real markdown links is cheaper, consistent, and clickable. Summaries are trimmed to the first sentence — the full summary lives on the child page.

The markdown URL /wiki/<type>/<slug> matches the mobile router path. Mobile’s wiki detail view hooks onLinkPress to intercept these and route internally rather than handing off to the OS Linking API.

Step 2.4 — Dedupe across sections on the same parent

Section titled “Step 2.4 — Dedupe across sections on the same parent”

A single parent page like Austin Activities previously listed Lady Bird Lake Hike & Bike Trail under both Overview AND Outdoor & Family Attractions. The aggregation planner was emitting the same child under multiple sections.

The compiler now tracks claimedChildrenByParent per job — first section to claim a child wins; later sections on the same parent drop the duplicate before rendering. The aggregation prompt was also updated to forbid emitting the same child twice.

Every section with aggregation metadata gets scored by scoreSectionAggregation. A pure function, no LLM:

score = 0.25·linked_page_count_signal // saturates at 20 child pages
+ 0.25·supporting_record_signal // saturates at 30 records
+ 0.20·temporal_spread_signal // saturates at 30 days
+ 0.15·coherence_signal // tag overlap ratio
+ 0.15·readability_pressure_signal // body length >1200 chars

Thresholds: candidate >= 0.4, promote_ready >= 0.55. When a section crosses promote_ready, its aggregation.promotion_status flips to candidate and it’s eligible for promotion on the next aggregation pass. Stickiness: once promotion_status = "promoted", never flaps back — operator override only.

When the aggregation planner emits a sectionPromotion, the compiler:

  1. Creates the new topic page from newPage.seed.
  2. Calls setParentPage({ pageId: newPage.id, parentPageId: parent.id }) — writes parent_page_id AND mirrors into wiki_page_links with kind='parent_of'/'child_of'.
  3. Rewrites the parent’s section body down to the parentSummary + topHighlights the planner wrote, followed by See: [Title](/wiki/...) as a real markdown link.
  4. Marks the parent section promotion_status = "promoted" with the child’s page id stored in aggregation.promoted_page_id.

This is the compounding loop — a section outgrows its parent, spins off its own page, and that new page can itself grow sections that eventually promote. Austin → Austin Activities → (eventually) Austin Kid-Friendly Activities, on and on as data accumulates.

Every page the aggregation pass touched gets its hubness_score recomputed:

hubness = inbound_reference_links
+ 2 · promoted_child_count
+ floor(avg(section.supporting_record_count) / 10)

The score is coarse and monotonic — it’s used for ordering, not as a precise ranking number. A page that gains a promoted child jumps ahead of one with only inbound references.

Bedrock throttling. ThrottlingException comes back when Claude’s per-minute quota is exhausted. The job fails, cursor stays put, next invoke retries clean.

JSON truncation. Model output that exceeds maxTokens comes back malformed. maxTokens is clamped per-model via MODEL_MAX_OUTPUT_TOKENS so Nova Micro (10k hard limit) doesn’t reject outright; Haiku/gpt-oss get the full 24k planner budget.

Hallucinated IDs. Covered in Step 1.6 — caught at repo boundary, not the prompt.

Over-grouping in aggregation. The Austin Restaurants = Toronto restaurants bug. Fixed by adding evidence requirements to the prompt and showing the planner per-page summaries (previously only titles were visible to the LLM).

Env vars reset on deploy. Terraform apply on an unrelated merge re-applies Lambda env config, wiping WIKI_AGGREGATION_PASS_ENABLED. Known paper cut; pin in terraform is the fix.

Cursor precision drift. Postgres timestamps are microsecond-precision but JS Date only carries milliseconds. The cursor query date_trunc('milliseconds', …) on both sides so a record at xx:yy:zz.689019 isn’t seen as “after” a stored cursor of xx:yy:zz.689 forever.

Typical full recompile of an agent with ~700 memory units runs $0.50 – $1.00 in Bedrock costs. Per-batch cost: $0.04 – $0.12. Aggregation pass adds ~$0.01 – $0.05 per job on top of leaf cost.

Every job’s metrics JSON records input_tokens, output_tokens, cost_usd, and aggregation-specific counters (aggregation_planner_calls, deterministic_parents_derived, parent_sections_updated, sections_promoted). Watch those when tuning prompts or thresholds.

Link-densification + alias-dedupe counters (added by the deterministic-linking and fuzzy-dedupe passes):

  • links_written_deterministicreference edges written by emitDeterministicParentLinks.
  • links_written_co_mentionreference edges written by emitCoMentionLinks.
  • duplicate_candidates_count — R5 precision canary (active (owner, title) groups with more than one row).
  • alias_dedup_mergednewPages[] entries collapsed into an existing page by exact alias match.
  • fuzzy_dedupe_merges — same, via pg_trgm similarity ≥ 0.85 (same-type gate).
  • deterministic_linking_flag_suppressedtrue when WIKI_DETERMINISTIC_LINKING_ENABLED=false skipped both emitters.

  • Post-retainmaybeEnqueuePostTurnCompile in packages/api/src/lib/wiki/enqueue.ts. Checks tenants.wiki_compile_enabled, confirms adapter is Hindsight, inserts a job row with dedupe key ${tenantId}:${ownerId}:${floor(epoch_s/300)}, fires Lambda.invoke with InvocationType: "Event" and {jobId} payload.
  • Continuation (cap hit) — when a job exits on any cap_hit without draining its cursor, runCompileJob enqueues a follow-up into the next dedupe bucket via (parseCompileDedupeBucket(parent.dedupe_key) + 1) * DEDUPE_BUCKET_SECONDS. The chained job inherits the parent’s trigger, so bootstrap_import continuations keep the higher MAX_RECORDS_PER_BOOTSTRAP_JOB = 1000 cap and self-complete without manual re-kick. Parsing the parent’s dedupe bucket (not Date.now() or created_at) is what keeps the chain monotonic — see docs/solutions/logic-errors/compile-continuation-dedupe-bucket-2026-04-20.md.
  • Admin mutationcompileWikiNow(tenantId, ownerId, modelId?) resolver.
  • Nightly lintpackages/api/src/handlers/wiki-lint.ts.
  • CLIthinkwork wiki compile --tenant … --owner ….
packages/api/src/lib/wiki/compiler.ts
const RECORD_PAGE_SIZE = 50;
const MAX_RECORDS_PER_JOB = 500;
const MAX_RECORDS_PER_BOOTSTRAP_JOB = 1000; // applied only when trigger='bootstrap_import'
const MAX_NEW_PAGES_PER_JOB = 25;
const MAX_SECTIONS_REWRITTEN_PER_JOB = 100;
const MAX_AGGREGATION_PAGES = 60; // top-N recently-changed pages fed to the aggregation planner
packages/api/src/lib/memory/adapters/hindsight-adapter.ts
WHERE bank_id = ${bankId}
AND (
date_trunc('milliseconds', COALESCE(updated_at, created_at)) > ${sinceTs.toISOString()}::timestamptz
OR (
date_trunc('milliseconds', COALESCE(updated_at, created_at)) = ${sinceTs.toISOString()}::timestamptz
AND id::text > ${sinceId}
)
)
{
"pageUpdates": [
{ "pageId": "<uuid>", "sections": [{ "slug": "overview", "proposed_body_md": "...", "source_refs": ["<memory_unit_id>"] }], "aliases": ["..."] }
],
"newPages": [
{ "type": "entity|topic|decision", "slug": "...", "title": "...", "summary": "...", "aliases": ["..."], "source_refs": ["<memory_unit_id>"],
"sections": [{ "slug": "overview", "heading": "Overview", "body_md": "...", "source_refs": ["<memory_unit_id>"] }] }
],
"unresolvedMentions": [
{ "alias": "Chef João", "suggestedType": "entity", "context": "mentioned when describing…", "source_ref": "<memory_unit_id>" }
],
"promotions": [
{ "mentionId": "<uuid>", "type": "entity", "title": "...", "slug": "...", "sections": [/* same shape */] }
],
"pageLinks": [
{ "fromType": "entity", "fromSlug": "...", "toType": "topic", "toSlug": "...", "context": "why this link exists" }
],
"parentSectionUpdates": [], // reserved for leaf planner, rarely populated
"sectionPromotions": [] // same
}
{
"parentSectionUpdates": [
{
"pageId": "<parent hub page id>",
"sectionSlug": "restaurants",
"heading": "Restaurants",
"proposed_body_md": "...", // ignored when linked_page_slugs is populated
"linked_page_slugs": [{ "type": "entity", "slug": "franklin-barbecue" }],
"source_refs": ["<memory_unit_id>"],
"observed_tags": ["bbq", "food"],
"rationale": ""
}
],
"sectionPromotions": [
{
"pageId": "<parent page id>",
"sectionSlug": "restaurants",
"reason": "Section has 20+ linked entities and 30+ days of provenance.",
"parentSummary": "Short paragraph to leave on the parent section.",
"topHighlights": ["- Franklin Barbecue", "- Uchi", "- Suerte"],
"newPage": { /* same shape as leaf newPages[] */ }
}
],
"newPages": [], // hub topic pages only — banned from creating leaf entities
"pageLinks": []
// all other fields forced to [] — aggregation planner is prevented from re-entering the leaf planner's job
}
packages/api/src/lib/wiki/repository.ts
export function isValidUuid(raw: unknown): raw is string { … }
export function stripWikilinks(md: string | null | undefined): string { … }
// and inside compiler.ts:
export function linkifyKnownEntities(body, refs): string { … }
packages/api/src/lib/wiki/promotion-scorer.ts
export const DEFAULT_PROMOTION_THRESHOLDS = { candidate: 0.4, promoteReady: 0.55 };
const WEIGHTS = { linked: 0.25, supporting: 0.25, temporal: 0.2, coherence: 0.15, readability: 0.15 };
const SATURATION = { linkedPageCount: 20, supportingRecordCount: 30, temporalSpreadDays: 30, bodyLength: 1800 };
packages/api/src/lib/wiki/parent-expander.ts
export function deriveParentCandidates(records): DerivedParentCandidate[]; // record metadata
export function deriveParentCandidatesFromPageSummaries(pages): DerivedParentCandidate[]; // page summaries
export function mergeParentCandidates(...lists): DerivedParentCandidate[];
packages/api/src/lib/wiki/deterministic-linker.ts
export async function emitDeterministicParentLinks(args): Promise<number>; // city/journal → parent topic
export async function emitCoMentionLinks(args): Promise<number>; // entity ↔ entity via shared memory_unit

Gated on WIKI_DETERMINISTIC_LINKING_ENABLED (default true). Context tags deterministic:city:<slug>, deterministic:journal:<slug>, co_mention:<memory_unit_id> make the rows droppable via a single DELETE ... WHERE context LIKE 'deterministic:%' OR ... query — see docs/metrics/wiki-link-density.md for the rollback SQL.

packages/api/src/lib/wiki/repository.ts
export async function findAliasMatches(args): Promise<AliasMatch[]>; // exact match (existing)
export async function findAliasMatchesFuzzy(args): Promise<AliasMatch[]>; // pg_trgm similarity ≥ 0.85

Backed by migration packages/database-pg/drizzle/0015_pg_trgm_alias_title_indexes.sql (CREATE EXTENSION IF NOT EXISTS pg_trgm + GIN trigram indexes on wiki_page_aliases.alias and wiki_pages.title). The compiler’s maybeMergeIntoExistingPage tries exact first, falls through to fuzzy with a strict same-type gate, and returns "exact" | "fuzzy" | null so the caller bumps the right metric.

  • packages/api/scripts/wiki-link-density-baseline.ts — per-agent density snapshot; appends a timestamped markdown file under docs/metrics/ so before/after flag flips can be diffed.
  • packages/api/scripts/wiki-link-backfill.ts — one-off --tenant --owner [--dry-run] that applies both deterministic emitters to the existing corpus. Idempotent via onConflictDoNothing on the (from, to, kind) unique index.
  • WIKI_COMPILE_ENABLED — tenant-scoped, column on tenants.
  • WIKI_AGGREGATION_PASS_ENABLED — Lambda env var, pinned in terraform.
  • WIKI_DETERMINISTIC_LINKING_ENABLED — Lambda env var, pinned in terraform next to aggregation. Default true. false ⇒ both deterministic emitters skip and metrics record deterministic_linking_flag_suppressed: true.
  • Default: openai.gpt-oss-120b-1:0 (env: BEDROCK_MODEL_ID).
  • Per-job override: { modelId: "…" } in the compile Lambda event payload.
  • Known-model output caps (Nova Micro’s 10k limit, etc.) live in MODEL_MAX_OUTPUT_TOKENS inside bedrock.ts so large maxTokens requests get clamped, not rejected.
packages/api/src/lib/wiki/compiler.ts
function resolveCitedRecords(
byId: Map<string, ThinkWorkMemoryRecord>,
ids: string[] | undefined,
): ThinkWorkMemoryRecord[] {
if (!ids || ids.length === 0) return [];
const out: ThinkWorkMemoryRecord[] = [];
for (const id of ids) {
const r = byId.get(id);
if (r) out.push(r);
}
return out;
}

No fallback to batch-wide sources. Single most important correctness invariant in the pipeline.

Against a full recompile of one agent (~700 source memory units) running gpt-oss-120b with aggregation on:

  • ~90 pages created (mix of entity and topic, ~5–10% topics).
  • ~115 page links across the graph (reference + parent_of/child_of).
  • 1–3 sections promoted per full rebuild depending on density.
  • ~$0.50–$1.00 total compile cost.
  • 4–6 aggregation rollup sections populated on hub pages.
  • Per-batch cost $0.04–$0.12; aggregation pass adds $0.01–$0.05.