Phase 3 PR Specs — Retrieval + Ingestion Reliability + Cost

Last Updated: 2026-02-28
Content-Type: How-to (implementation-ready PR specs)
Audience: Engineering

---

Goal of Phase 3

Make retrieval and memory ingestion:

• cheaper (avoid duplicate embeddings)
• more reliable (atomic chunk writes, bounded background work)
• more deterministic (consistent scoping, fewer silent failure modes)

---

PR3.1 — Exactly one embedding per user turn

Why (problem)

Current flows can generate embeddings multiple times per turn:

• asoContextRunnable generates embedding for memory context and again for CSO “system-wide knowledge”.
• Chunk retrieval (retrieveMemoryChunks) always embeds inside the retriever, even when upstream already computed an embedding.

Standardization target

1. A single turn produces ≤ 1 embedding call for retrieval.
2. Retriever APIs accept a precomputed embedding and never re-embed when provided.

Implementation approach (recommended)

1. Update chunk retrieval API to accept optional precomputed embedding:

• retrieveMemoryChunks({ queryText, queryEmbedding?: number[] , ... })

2. Update getThreeLayeredContext() chunk path to call chunk retrieval directly (not via adapter) when an embedding is already available:

• Today: createMemoryChunksRetriever(...).invoke(userInput) → embeds internally.
• Target: retrieveMemoryChunks({ queryText: userInput, queryEmbedding, ... }) → no embed.

3. Update asoContextRunnable to compute embedding once and reuse:

• Compute queryEmbedding once per run and pass it to:
  • getThreeLayeredContext(...)
  • getSystemWideKnowledge(queryEmbedding) (CSO only)

Acceptance criteria

• For a standard turn with chunk retrieval enabled: exactly one embedding call (verify metric/log).
• Context results (tiers + filtering) remain the same as before, modulo expected improvements.

Primary files

• backend/src/services/langchain/retrievers/pgvectorMemoryChunksRetriever.ts
• backend/src/services/aso/context.service.ts
• backend/src/services/langchain/chains/contextRunnable.ts

---

PR3.2 — Atomic chunk ingestion (no delete-then-fail data loss)

Why (problem)

upsertMemoryChunksFromContent() currently:

1. deletes all chunks for a memory
2. inserts new rows

If insert fails, all chunks are lost.

Standardization target

Chunk replacement must be atomic.

Implementation options

Option A (recommended): transaction wrap

• Wrap destroy + bulkCreate in a transaction.
• Ensure failure rolls back to previous chunk set.

Option B: UPSERT by (memoryId, chunkIndex)

• Upsert new rows and delete stale rows.
• More complex but avoids full replacement windows.

Acceptance criteria

• Simulated failure during insert does not lose previously existing chunks.
• Ingestion produces a consistent (memoryId, chunkIndex) set.

Primary files

• backend/src/services/aso/ingestion.service.ts
• backend/src/models/memoryChunk.model.ts (constraints/indexing, if needed)

---

PR3.3 — Bounded background jobs + per-key dedupe

Why (problem)

Background jobs are enqueued with setImmediate() and have:

• no global concurrency cap
• no dedupe by key (e.g. repeated ingestion for the same memory)
• no backpressure

This can overload:

• the DB
• embedding provider
• CPU (chunking + embedding in parallel)

Standardization target

1. Background jobs are bounded (global concurrency).
2. Jobs dedupe by key for high-frequency work (memoryId-based ingestion/embedding).

Implementation approach (minimal, in-memory)

• Replace setImmediate() runner with a small queue:
  • global concurrency limit (e.g. 2–5)
  • in-flight key map (e.g. embedding:memoryId, ingest:memoryId)
  • drop/merge duplicate jobs (choose semantics; for ingestion, latest wins is acceptable)

Longer-term migration can move the queue to a durable worker system, but Phase 3 should deliver immediate protection.

Acceptance criteria

• Burst traffic does not create unbounded concurrent ingestion.
• For repeated turns, ingestion jobs for the same memoryId do not stack indefinitely.

Primary files

• backend/src/services/backgroundJob.service.ts
• backend/src/services/postResponsePipeline.service.ts

---

PR3.4 — Explicit timeouts for embedding HTTP requests

Why (problem)

backend/src/services/embedding.service.ts uses axios without a timeout, risking hung requests.

Standardization target

• All external calls have explicit timeouts and retry policy is consistent.

Implementation steps

• Add timeout to the axios client used for embeddings.
• Ensure timeout errors are logged with provider + model + retry count.

Acceptance criteria

• Hanging network calls terminate within configured timeout.
• Retrieval and ingestion degrade gracefully (no stuck background jobs).

Primary files

• backend/src/services/embedding.service.ts

---

Phase 3 “quality” follow-ups (can be folded into PR3.1+)

Hybrid keyword search should not require embeddings

In retrieveMemoryChunks(), keyword queries include embedding IS NOT NULL even though keyword search doesn’t need embeddings. This blocks recall during backfill or partial ingestion.

Conversation duplication controls in LangChain chunk injection

When LangChain injects retrieved chunks into the system prompt, it should pass:

• conversationId
• excludeConversationPersonal / excludeConversationAso when structured history is provided
• allowedAsoSources when protocol requires it