Compounding Memory: Pipeline
This doc is a walking tour of how raw memories become wiki pages — the harness’s distillation layer that turns scattered observations into structured, queryable pages. It’s written to be read top to bottom; each step covers one thing the pipeline does, why it does it, and what can go wrong. The pipeline is the load-bearing implementation of compounding memory’s product promise: reproducible page outputs from the same input set (Reliability) and a complete provenance trail from every page back to the source memories that wrote it (Traceability).
The first half is plain-language narrative. The second half (“Under the hood”) has the code paths, SQL, and prompt contracts for anyone who needs to modify or debug the pipeline itself.
High-level shape
Section titled “High-level shape”There are two compile passes that run in the same job:
- Leaf pass. Reads new memories, decides which concrete entity pages should be created or updated. Produces pages like
Momofuku Daishō,Franklin Barbecue,Biscuit (dog). This is where most wiki content originates. - Aggregation pass. Runs after the leaf pass completes. Looks across the agent’s recently-changed pages and decides which hub topics deserve rollup sections (e.g.
Torontowith aRestaurantssection) and which sections have become dense enough to promote into their own page (e.g. aRestaurantssection onAustinpromotes into a standaloneAustin Restaurantstopic page).
Both passes are driven by one compile job. The job is always scoped to one (tenant, owner) pair. If anything fails, the cursor stays where it was and the next job safely replays.
Pass 1: Leaf compile
Section titled “Pass 1: Leaf compile”Step 1.1 — A compile job gets queued
Section titled “Step 1.1 — A compile job gets queued”A job row lands in wiki_compile_jobs in one of four ways:
- After a retain. The
memory-retainLambda firesmaybeEnqueuePostTurnCompileafter every successfulretainTurn. Jobs dedupe on a 5-minute bucket, so a burst of turns collapses into one queued compile. - Admin trigger. The
compileWikiNow(tenantId, ownerId)mutation enqueues a job directly. - Lint sweep. The nightly lint pass enqueues promotion jobs for unresolved mentions that have crossed the
≥ 3 mentions in 30 daysthreshold. - CLI.
thinkwork wiki compilewraps the admin mutation for operator use.
Step 1.2 — The compile worker wakes up
Section titled “Step 1.2 — The compile worker wakes up”wiki-compile Lambda claims the pending job (FOR UPDATE SKIP LOCKED) and reads the cursor for that (tenant, owner) — a bookmark of the last memory the worker processed.
Then it reads the next 50 memories from Hindsight where updated_at > cursor. Hindsight is the canonical memory substrate; the wiki is a derived store.
Step 1.3 — The planner reads memories in context
Section titled “Step 1.3 — The planner reads memories in context”The batch of memories plus the scope’s existing pages plus any open unresolved mentions get handed to the leaf planner — Bedrock via the Converse API, defaulting to Claude Haiku 4.5 but swappable via the modelId event parameter.
The planner sees:
- The new memories, in full.
- Titles, slugs, and summaries of every active page in scope.
- Every open unresolved mention with its accumulated context.
The planner is explicitly biased toward caution. Holding an alias as “unresolved” is almost free; creating a spurious page is expensive to undo. When in doubt, it holds.
Step 1.4 — The planner produces a plan
Section titled “Step 1.4 — The planner produces a plan”The plan is a small JSON document with six arrays:
Page updates. “This existing page should have its Visits section rewritten given this new memory.”
New pages. “There’s enough evidence for a new page about Taberna dos Mercadores. Here’s a title, sections, aliases.”
Unresolved mentions. “Someone named ‘Chef João’ showed up but doesn’t justify a page yet — hold the alias.”
Promotions. “Chef João has been held for months and three new memories push him over the threshold. Promote to a real page, here are the sections.”
Page links. “Taberna dos Mercadores should link to Lisbon. Both pages exist or are being created in this plan.”
Parent section updates and section promotions are also emitted by the planner contract but the leaf planner rarely fills them — the aggregation pass does that work.
The planner never writes to the database. It produces a plan; the compiler applies it.
Step 1.5 — The section writer patches one section at a time
Section titled “Step 1.5 — The section writer patches one section at a time”For every section the plan wants to update, the compiler calls a second Bedrock model — the section writer. It’s narrow: take the existing section markdown, the planner’s proposed body, and the specific memories cited — produce final markdown for this one section only.
Before calling, the compiler checks isMeaningfulChange — a cheap edit-distance filter that skips sections when the proposal is effectively identical to the stored body. No point burning tokens polishing prose that already says the same thing.
Step 1.6 — Hallucinated IDs and stray wikilinks get filtered
Section titled “Step 1.6 — Hallucinated IDs and stray wikilinks get filtered”Before the model output is trusted:
isValidUuid()gates every page-id and mention-id lookup. Smaller models sometimes emit truncated ids like"814e6b70"; Postgres would reject those as invalid UUIDs and kill the job. The guard silently skips invalid rows instead.stripWikilinks()removes[[Obsidian-style]]syntax from section bodies at the repo boundary. Models were told not to emit it, but they slip; the strip runs insideupsertSectionsso every write path is covered.linkifyKnownEntities()rewrites**Entity Name**bolded mentions into real[**Entity Name**](/wiki/<type>/<slug>)markdown links when the name matches a page in scope. Longest-title-first match prevents shadowing.- Invalid
pageLinks(bad page types, blank slugs) are dropped with a CloudWatch warning rather than failing the whole plan.
All five filters exist because some model has produced the exact failure mode we’re guarding against. Prompts are the first line of defense; these repo-level guards are the second.
Step 1.7 — Provenance gets written with deliberate care
Section titled “Step 1.7 — Provenance gets written with deliberate care”Every section carries a list of which memories were used as sources. The rule is strict: only the specific memories the planner cited for this section. Not “all the memories in the batch.” Not “every memory that looks related.”
An earlier version of the compiler fell back to “use the whole batch as sources” when the planner omitted citations. The result was nonsense: a memory about Austin weather got written as a source for unrelated CRM pages that happened to be in the same batch. The reverse-lookup (MemoryRecord.wikiPages) then returned those CRM pages when you tapped the weather memory. Everything downstream broke.
The fix: zero cited memories means zero source rows. Blank provenance is always preferable to wrong provenance. resolveCitedRecords encodes this rule and no path bypasses it.
Step 1.7a — Alias dedupe on newPages
Section titled “Step 1.7a — Alias dedupe on newPages”Before creating any page the leaf planner proposed in newPages[], the compiler tries to merge the proposal into an existing page. The match runs in two passes:
- Exact alias match. The proposed title + aliases are normalized and looked up against
wiki_page_aliasesin the same scope. A hit turns the proposal into apageUpdateagainst the existing page and bumpsalias_dedup_merged. - Trigram fuzzy fallback. If no exact alias matches, the compiler queries
wiki_page_aliases.aliasandwiki_pages.titlewith Postgrespg_trgmsimilarity. Matches atsimilarity() >= 0.85are accepted only when the matched page has the sametypeas the proposal — entity into entity, topic into topic. Cross-type fuzzy matches are always rejected. Successful fuzzy merges bumpfuzzy_dedupe_merges, tracked separately from exact because fuzzy is the over-collapse risk.
The repo helper is findAliasMatchesFuzzy in packages/api/src/lib/wiki/repository.ts; it sits on the pg_trgm GIN indexes added by migration 0015_pg_trgm_alias_title_indexes.sql. The helper’s internal try/catch makes a missing pg_trgm extension a graceful no-match rather than a job failure.
Without this step, planner-emitted "Paris, France" creates a second page when "Paris" already exists. With it, the proposal collapses into the existing topic/paris page and its new sections land there.
Step 1.7b — Deterministic link emission
Section titled “Step 1.7b — Deterministic link emission”After the leaf plan applies, two deterministic emitters write additional reference links. Both are LLM-free, both gate on WIKI_DETERMINISTIC_LINKING_ENABLED (default true, pinned in terraform next to WIKI_AGGREGATION_PASS_ENABLED).
emitDeterministicParentLinks— reads parent-expander candidates (Step 2.1’sderiveParentCandidates*). For eachcity/journalcandidate with an exact-title match in scope, it writes areferenceedge from every leaf entity page sourced by that record into the parent topic. Context tag:deterministic:city:<slug>ordeterministic:journal:<slug>.emitCoMentionLinks— readswiki_section_sources(not the leaf planner’spageLinks) to find entity pages that share amemory_unitsource. Emits reciprocalreferenceedges between those pages, capped at 10 directed edges per memory (slug-ascending order). Context tag:co_mention:<memory_unit_id>.
Both emitters persist through upsertPageLink’s onConflictDoNothing on the (from_page_id, to_page_id, kind) unique index, so a replay is a no-op. The context-tag format exists so targeted rollback can drop only the deterministic rows without touching planner-emitted references.
When the flag is off, the job records deterministic_linking_flag_suppressed: true in metrics and both links_written_* counters stay at 0. The rollback SQL lives in docs/metrics/wiki-link-density.md in the repo.
Step 1.7c — R5 precision canary
Section titled “Step 1.7c — R5 precision canary”Every job records duplicate_candidates_count — the count of (owner_id, title) groups with more than one active row, scoped to the tenant. A rising value means the dedupe step is losing ground (fuzzy gate too strict, new aliases slipping through). It’s the first signal operators should watch after flipping WIKI_DETERMINISTIC_LINKING_ENABLED.
Step 1.8 — The cursor advances
Section titled “Step 1.8 — The cursor advances”Once the plan applies cleanly, the worker moves its bookmark to the last memory it processed. If anything failed mid-batch, the bookmark stays and the next worker replays. Safe to retry.
Step 1.9 — The worker loops
Section titled “Step 1.9 — The worker loops”Keep fetching 50-memory batches until: cursor drains, hard cap (500 records / 25 new pages / 100 section rewrites), or Bedrock rate-limits. Then the job finishes and emits metrics.
Large backfills naturally spread across multiple jobs — cursor-preserving means no replay hazard.
Pass 2: Aggregation
Section titled “Pass 2: Aggregation”The leaf pass ends with fresh entity pages in the scope, but they’re not yet organized. Marco now has 90 restaurants, events, and places — but there’s no “Austin” topic page tying his Austin ones together, no “Restaurants” hub, no rollups you can scroll through. That’s what the aggregation pass builds.
Step 2.1 — Derive parent candidates
Section titled “Step 2.1 — Derive parent candidates”Two deterministic expanders scan the current job’s records + the scope’s recently-changed pages to suggest hubs that obviously need to exist:
deriveParentCandidates(records)— pulls city, journal, and tag clusters out of raw record metadata. Records withplace.city = "Austin"group into an Austin candidate. Sharedjournal_ids group into a trip topic. Repeated tags group into a tag-collection hub.deriveParentCandidatesFromPageSummaries(pages)— scans each scope page’s summary text for city names. Catches “Korean restaurant in Toronto.” style summaries where raw metadata isn’t in the current batch. Walks comma-separated postal addresses from the end to pullTorontoout of", Toronto, ON M6J 1G1, Canada.".
Both expanders are cheap — no LLM, no extra DB hits beyond the already-hydrated pages. They exist so the LLM gets grounded seed hubs instead of inventing them.
Step 2.2 — The aggregation planner
Section titled “Step 2.2 — The aggregation planner”A second Bedrock call, with its own system prompt, receives:
- Every recently-changed page in scope (with summary, sections, existing aggregation metadata).
- Parent candidates from Step 2.1.
- Link neighborhoods (inbound link counts).
The prompt is strict about membership: every page listed under a hub must have visible evidence in its summary. An earlier version hallucinated an Austin Restaurants hub containing Toronto, Tokyo, and Bogotá restaurants — the prompt now requires geographic proof per linked page and refuses to create a hub when fewer than 3 pages clearly match.
Output: parentSectionUpdates, sectionPromotions, hub newPages, hierarchy pageLinks.
Step 2.3 — Rollup bodies are rendered deterministically
Section titled “Step 2.3 — Rollup bodies are rendered deterministically”When a parentSectionUpdate carries linked_page_slugs, the compiler skips the section writer and renders the body with renderRollupList:
- [**Franklin Barbecue**](/wiki/entity/franklin-barbecue) — BBQ joint in Austin, TX.- [**Momofuku Daishō**](/wiki/entity/momofuku-daisho) — Korean-inspired restaurant in Toronto.Why deterministic: the LLM was previously writing prose (“Franklin Barbecue is a BBQ joint…”) that mentioned entity names as plain text with no hyperlinks. A structured list with real markdown links is cheaper, consistent, and clickable. Summaries are trimmed to the first sentence — the full summary lives on the child page.
The markdown URL /wiki/<type>/<slug> matches the mobile router path. Mobile’s wiki detail view hooks onLinkPress to intercept these and route internally rather than handing off to the OS Linking API.
Step 2.4 — Dedupe across sections on the same parent
Section titled “Step 2.4 — Dedupe across sections on the same parent”A single parent page like Austin Activities previously listed Lady Bird Lake Hike & Bike Trail under both Overview AND Outdoor & Family Attractions. The aggregation planner was emitting the same child under multiple sections.
The compiler now tracks claimedChildrenByParent per job — first section to claim a child wins; later sections on the same parent drop the duplicate before rendering. The aggregation prompt was also updated to forbid emitting the same child twice.
Step 2.5 — Section promotion scoring
Section titled “Step 2.5 — Section promotion scoring”Every section with aggregation metadata gets scored by scoreSectionAggregation. A pure function, no LLM:
score = 0.25·linked_page_count_signal // saturates at 20 child pages + 0.25·supporting_record_signal // saturates at 30 records + 0.20·temporal_spread_signal // saturates at 30 days + 0.15·coherence_signal // tag overlap ratio + 0.15·readability_pressure_signal // body length >1200 charsThresholds: candidate >= 0.4, promote_ready >= 0.55. When a section crosses promote_ready, its aggregation.promotion_status flips to candidate and it’s eligible for promotion on the next aggregation pass. Stickiness: once promotion_status = "promoted", never flaps back — operator override only.
Step 2.6 — Section promotion
Section titled “Step 2.6 — Section promotion”When the aggregation planner emits a sectionPromotion, the compiler:
- Creates the new topic page from
newPage.seed. - Calls
setParentPage({ pageId: newPage.id, parentPageId: parent.id })— writesparent_page_idAND mirrors intowiki_page_linkswithkind='parent_of'/'child_of'. - Rewrites the parent’s section body down to the
parentSummary+topHighlightsthe planner wrote, followed bySee: [Title](/wiki/...)as a real markdown link. - Marks the parent section
promotion_status = "promoted"with the child’s page id stored inaggregation.promoted_page_id.
This is the compounding loop — a section outgrows its parent, spins off its own page, and that new page can itself grow sections that eventually promote. Austin → Austin Activities → (eventually) Austin Kid-Friendly Activities, on and on as data accumulates.
Step 2.7 — Hubness recompute
Section titled “Step 2.7 — Hubness recompute”Every page the aggregation pass touched gets its hubness_score recomputed:
hubness = inbound_reference_links + 2 · promoted_child_count + floor(avg(section.supporting_record_count) / 10)The score is coarse and monotonic — it’s used for ordering, not as a precise ranking number. A page that gains a promoted child jumps ahead of one with only inbound references.
What can go wrong
Section titled “What can go wrong”Bedrock throttling. ThrottlingException comes back when Claude’s per-minute quota is exhausted. The job fails, cursor stays put, next invoke retries clean.
JSON truncation. Model output that exceeds maxTokens comes back malformed. maxTokens is clamped per-model via MODEL_MAX_OUTPUT_TOKENS so Nova Micro (10k hard limit) doesn’t reject outright; Haiku/gpt-oss get the full 24k planner budget.
Hallucinated IDs. Covered in Step 1.6 — caught at repo boundary, not the prompt.
Over-grouping in aggregation. The Austin Restaurants = Toronto restaurants bug. Fixed by adding evidence requirements to the prompt and showing the planner per-page summaries (previously only titles were visible to the LLM).
Env vars reset on deploy. Terraform apply on an unrelated merge re-applies Lambda env config, wiping WIKI_AGGREGATION_PASS_ENABLED. Known paper cut; pin in terraform is the fix.
Cursor precision drift. Postgres timestamps are microsecond-precision but JS Date only carries milliseconds. The cursor query date_trunc('milliseconds', …) on both sides so a record at xx:yy:zz.689019 isn’t seen as “after” a stored cursor of xx:yy:zz.689 forever.
Typical full recompile of an agent with ~700 memory units runs $0.50 – $1.00 in Bedrock costs. Per-batch cost: $0.04 – $0.12. Aggregation pass adds ~$0.01 – $0.05 per job on top of leaf cost.
Every job’s metrics JSON records input_tokens, output_tokens, cost_usd, and aggregation-specific counters (aggregation_planner_calls, deterministic_parents_derived, parent_sections_updated, sections_promoted). Watch those when tuning prompts or thresholds.
Link-densification + alias-dedupe counters (added by the deterministic-linking and fuzzy-dedupe passes):
links_written_deterministic—referenceedges written byemitDeterministicParentLinks.links_written_co_mention—referenceedges written byemitCoMentionLinks.duplicate_candidates_count— R5 precision canary (active(owner, title)groups with more than one row).alias_dedup_merged—newPages[]entries collapsed into an existing page by exact alias match.fuzzy_dedupe_merges— same, viapg_trgmsimilarity ≥ 0.85 (same-type gate).deterministic_linking_flag_suppressed—truewhenWIKI_DETERMINISTIC_LINKING_ENABLED=falseskipped both emitters.
Related pages
Section titled “Related pages”- Compounding Memory — the overview
- Compounding Memory: Pages — the data model
- Operating Compounding Memory — hands-on recipes
- Compounding Memory (API) — GraphQL reference
Under the hood
Section titled “Under the hood”Triggers
Section titled “Triggers”- Post-retain —
maybeEnqueuePostTurnCompileinpackages/api/src/lib/wiki/enqueue.ts. Checkstenants.wiki_compile_enabled, confirms adapter is Hindsight, inserts a job row with dedupe key${tenantId}:${ownerId}:${floor(epoch_s/300)}, firesLambda.invokewithInvocationType: "Event"and{jobId}payload. - Continuation (cap hit) — when a job exits on any
cap_hitwithout draining its cursor,runCompileJobenqueues a follow-up into the next dedupe bucket via(parseCompileDedupeBucket(parent.dedupe_key) + 1) * DEDUPE_BUCKET_SECONDS. The chained job inherits the parent’strigger, sobootstrap_importcontinuations keep the higherMAX_RECORDS_PER_BOOTSTRAP_JOB = 1000cap and self-complete without manual re-kick. Parsing the parent’s dedupe bucket (notDate.now()orcreated_at) is what keeps the chain monotonic — seedocs/solutions/logic-errors/compile-continuation-dedupe-bucket-2026-04-20.md. - Admin mutation —
compileWikiNow(tenantId, ownerId, modelId?)resolver. - Nightly lint —
packages/api/src/handlers/wiki-lint.ts. - CLI —
thinkwork wiki compile --tenant … --owner ….
Caps (leaf pass)
Section titled “Caps (leaf pass)”const RECORD_PAGE_SIZE = 50;const MAX_RECORDS_PER_JOB = 500;const MAX_RECORDS_PER_BOOTSTRAP_JOB = 1000; // applied only when trigger='bootstrap_import'const MAX_NEW_PAGES_PER_JOB = 25;const MAX_SECTIONS_REWRITTEN_PER_JOB = 100;Caps (aggregation pass)
Section titled “Caps (aggregation pass)”const MAX_AGGREGATION_PAGES = 60; // top-N recently-changed pages fed to the aggregation plannerCursor SQL
Section titled “Cursor SQL”WHERE bank_id = ${bankId} AND ( date_trunc('milliseconds', COALESCE(updated_at, created_at)) > ${sinceTs.toISOString()}::timestamptz OR ( date_trunc('milliseconds', COALESCE(updated_at, created_at)) = ${sinceTs.toISOString()}::timestamptz AND id::text > ${sinceId} ) )Leaf planner output contract
Section titled “Leaf planner output contract”{ "pageUpdates": [ { "pageId": "<uuid>", "sections": [{ "slug": "overview", "proposed_body_md": "...", "source_refs": ["<memory_unit_id>"] }], "aliases": ["..."] } ], "newPages": [ { "type": "entity|topic|decision", "slug": "...", "title": "...", "summary": "...", "aliases": ["..."], "source_refs": ["<memory_unit_id>"], "sections": [{ "slug": "overview", "heading": "Overview", "body_md": "...", "source_refs": ["<memory_unit_id>"] }] } ], "unresolvedMentions": [ { "alias": "Chef João", "suggestedType": "entity", "context": "mentioned when describing…", "source_ref": "<memory_unit_id>" } ], "promotions": [ { "mentionId": "<uuid>", "type": "entity", "title": "...", "slug": "...", "sections": [/* same shape */] } ], "pageLinks": [ { "fromType": "entity", "fromSlug": "...", "toType": "topic", "toSlug": "...", "context": "why this link exists" } ], "parentSectionUpdates": [], // reserved for leaf planner, rarely populated "sectionPromotions": [] // same}Aggregation planner output contract
Section titled “Aggregation planner output contract”{ "parentSectionUpdates": [ { "pageId": "<parent hub page id>", "sectionSlug": "restaurants", "heading": "Restaurants", "proposed_body_md": "...", // ignored when linked_page_slugs is populated "linked_page_slugs": [{ "type": "entity", "slug": "franklin-barbecue" }], "source_refs": ["<memory_unit_id>"], "observed_tags": ["bbq", "food"], "rationale": "…" } ], "sectionPromotions": [ { "pageId": "<parent page id>", "sectionSlug": "restaurants", "reason": "Section has 20+ linked entities and 30+ days of provenance.", "parentSummary": "Short paragraph to leave on the parent section.", "topHighlights": ["- Franklin Barbecue", "- Uchi", "- Suerte"], "newPage": { /* same shape as leaf newPages[] */ } } ], "newPages": [], // hub topic pages only — banned from creating leaf entities "pageLinks": [] // all other fields forced to [] — aggregation planner is prevented from re-entering the leaf planner's job}Repo-boundary guards
Section titled “Repo-boundary guards”export function isValidUuid(raw: unknown): raw is string { … }export function stripWikilinks(md: string | null | undefined): string { … }// and inside compiler.ts:export function linkifyKnownEntities(body, refs): string { … }Promotion scorer
Section titled “Promotion scorer”export const DEFAULT_PROMOTION_THRESHOLDS = { candidate: 0.4, promoteReady: 0.55 };const WEIGHTS = { linked: 0.25, supporting: 0.25, temporal: 0.2, coherence: 0.15, readability: 0.15 };const SATURATION = { linkedPageCount: 20, supportingRecordCount: 30, temporalSpreadDays: 30, bodyLength: 1800 };Parent expanders
Section titled “Parent expanders”export function deriveParentCandidates(records): DerivedParentCandidate[]; // record metadataexport function deriveParentCandidatesFromPageSummaries(pages): DerivedParentCandidate[]; // page summariesexport function mergeParentCandidates(...lists): DerivedParentCandidate[];Deterministic link emitters
Section titled “Deterministic link emitters”export async function emitDeterministicParentLinks(args): Promise<number>; // city/journal → parent topicexport async function emitCoMentionLinks(args): Promise<number>; // entity ↔ entity via shared memory_unitGated on WIKI_DETERMINISTIC_LINKING_ENABLED (default true). Context tags deterministic:city:<slug>, deterministic:journal:<slug>, co_mention:<memory_unit_id> make the rows droppable via a single DELETE ... WHERE context LIKE 'deterministic:%' OR ... query — see docs/metrics/wiki-link-density.md for the rollback SQL.
Fuzzy alias dedupe
Section titled “Fuzzy alias dedupe”export async function findAliasMatches(args): Promise<AliasMatch[]>; // exact match (existing)export async function findAliasMatchesFuzzy(args): Promise<AliasMatch[]>; // pg_trgm similarity ≥ 0.85Backed by migration packages/database-pg/drizzle/0015_pg_trgm_alias_title_indexes.sql (CREATE EXTENSION IF NOT EXISTS pg_trgm + GIN trigram indexes on wiki_page_aliases.alias and wiki_pages.title). The compiler’s maybeMergeIntoExistingPage tries exact first, falls through to fuzzy with a strict same-type gate, and returns "exact" | "fuzzy" | null so the caller bumps the right metric.
Operator scripts
Section titled “Operator scripts”packages/api/scripts/wiki-link-density-baseline.ts— per-agent density snapshot; appends a timestamped markdown file underdocs/metrics/so before/after flag flips can be diffed.packages/api/scripts/wiki-link-backfill.ts— one-off--tenant --owner [--dry-run]that applies both deterministic emitters to the existing corpus. Idempotent viaonConflictDoNothingon the(from, to, kind)unique index.
Feature flags (wiki)
Section titled “Feature flags (wiki)”WIKI_COMPILE_ENABLED— tenant-scoped, column ontenants.WIKI_AGGREGATION_PASS_ENABLED— Lambda env var, pinned in terraform.WIKI_DETERMINISTIC_LINKING_ENABLED— Lambda env var, pinned in terraform next to aggregation. Defaulttrue.false⇒ both deterministic emitters skip and metrics recorddeterministic_linking_flag_suppressed: true.
Model selection
Section titled “Model selection”- Default:
openai.gpt-oss-120b-1:0(env:BEDROCK_MODEL_ID). - Per-job override:
{ modelId: "…" }in the compile Lambda event payload. - Known-model output caps (Nova Micro’s 10k limit, etc.) live in
MODEL_MAX_OUTPUT_TOKENSinsidebedrock.tsso largemaxTokensrequests get clamped, not rejected.
Provenance rule
Section titled “Provenance rule”function resolveCitedRecords( byId: Map<string, ThinkWorkMemoryRecord>, ids: string[] | undefined,): ThinkWorkMemoryRecord[] { if (!ids || ids.length === 0) return []; const out: ThinkWorkMemoryRecord[] = []; for (const id of ids) { const r = byId.get(id); if (r) out.push(r); } return out;}No fallback to batch-wide sources. Single most important correctness invariant in the pipeline.
Measured cost + shape
Section titled “Measured cost + shape”Against a full recompile of one agent (~700 source memory units) running gpt-oss-120b with aggregation on:
- ~90 pages created (mix of entity and topic, ~5–10% topics).
- ~115 page links across the graph (
reference+parent_of/child_of). - 1–3 sections promoted per full rebuild depending on density.
- ~$0.50–$1.00 total compile cost.
- 4–6 aggregation rollup sections populated on hub pages.
- Per-batch cost $0.04–$0.12; aggregation pass adds $0.01–$0.05.