System Audit and Standardization Roadmap

Last Updated: 2026-02-28
Content-Type: Explanation + Roadmap (PR-ready)
Audience: Engineering (Backend + Frontend + DevOps)
Goal: Remove overlapping execution paths, close policy gaps, reduce duplicated work, and standardize system boundaries until the platform behaves deterministically.

---

Quick-wins guardrail owner policy

To avoid duplicate fallback execution, guardrail ownership is now single-owner:

• Canonical owner is orchestration (aso.service.ts).
• langchainToolTurn.service.ts no longer owns fallback forcing.
• GUARDRAIL_OWNER_MODE is kept as a compatibility flag, but runtime behavior is coerced to orchestration-only.

Diagnostic logs include:

• guardrail_owner
• guardrail_source
• guardrail_reason

---

1) Executive summary (what is broken / why it matters)

Critical correctness + security gaps

1. Tool access policy is not fully enforced: catalog policies like explicit_user_selected exist but are not enforced in the execution path. Enforcement currently checks protocol allowlist and self_only only.

   • Primary: backend/src/services/toolExecutor.service.ts
   • Related: backend/src/config/toolCatalog.ts, tool controllers, portal tool permissions UI

2. A LangChain-only “hidden tool” bypasses the Tool System: extract_and_save_user_facts is injected into LangChain tools directly. This bypasses tool catalog policy/enablement/audit expectations and is invisible to prompt “AVAILABLE TOOLS”.

   • Primary: backend/src/services/langchain/langchainToolTurn.service.ts

3. Frontend group chat is partially wired as single chat: suggested prompts trigger single send; group payload drops preferences; optimistic UI is missing for group; group tool progress/status read from single state.

   • Primary: aso-portal/src/components/chat/ChatWindow.tsx, aso-portal/src/stores/utils/messageSendUtils.ts, aso-portal/src/services/groupSocketServices.ts

Performance + reliability gaps (cost + latency)

4. Duplicate query embeddings: chunk retrieval re-embeds even when embedding already computed; some flows can embed 2–3 times per turn.

   • Primary: backend/src/services/aso/context.service.ts, backend/src/services/langchain/retrievers/pgvectorMemoryChunksRetriever.ts, backend/src/services/langchain/chains/contextRunnable.ts

5. Non-atomic chunk ingestion: delete → insert without a transaction can lose chunks if the insert fails.

   • Primary: backend/src/services/aso/ingestion.service.ts

6. Unbounded background job concurrency: setImmediate() jobs have no global cap/dedupe; burst traffic can overload DB and embedding providers.

   • Primary: backend/src/services/backgroundJob.service.ts, backend/src/services/postResponsePipeline.service.ts

7. Embedding HTTP requests have no explicit timeout: axios client lacks timeout, increasing hung-request risk.

   • Primary: backend/src/services/embedding.service.ts

Standardization gaps (duplication + drift)

8. Multi-path tool calling implementations drift (legacy JSON path vs LangChain): different schema sources, alias normalization, and guardrails exist in parallel.

   • Primary: backend/src/services/llmToolGateway.service.ts, backend/src/services/langchain/langchainToolTurn.service.ts, backend/src/services/tools/toolSchemas.ts, backend/src/services/tools/toolCompilation.service.ts

9. Brittle prompt “OUTPUT FORMAT” stripping: LangChain prompt runnable does string surgery based on fences; template changes can break it.

   • Primary: backend/src/services/langchain/chains/promptRunnable.ts

10. Env/config is not single-source-of-truth: .env templates diverge from validation and from direct process.env reads; validation runs after some env-dependent imports; some modules call dotenv.config() locally.

• Primary: backend/src/config/env.validation.ts, backend/src/config/database.ts, backend/src/services/embedding.service.ts

---

2) System map (current call graphs)

2.1 Backend chat runtime (conceptual)

Standardization target: One runtime path (LangChain default), and one tool platform.

2.2 Frontend message send paths (conceptual)

Standardization target: One canonical sendChat() orchestrator with consistent payload, error handling, and optimistic UI across single/group/streaming.

---

3) “Single source of truth” standards (target end-state)

3.1 Tool Platform (backend)

Canonical sources:

• Tool catalog + metadata (id, description, defaults): backend/src/config/toolCatalog.ts
• Tool schemas registry: backend/src/services/tools/toolSchemas.ts
• Tool execution + access enforcement: backend/src/services/toolExecutor.service.ts
• Tool availability compilation (protocol + enablement + overrides): backend/src/services/tools/toolCompilation.service.ts

Rules:

1. All tool calls (LangChain, REST, prompt debug) must route through the same schema validation, alias mapping, enablement, and access enforcement.
2. No “hidden” tools: if a tool is callable, it must be present in the canonical catalog (even if marked internal/admin-only).
3. Access policy is enforced centrally (including explicit_user_selected) and is test-covered.

3.2 Prompt Contract (backend)

Split responsibilities:

• Composition: base template + protocol overlays + tool definitions injection (no runtime-only content).
• Runtime context: current session/user directives, retrieval snippets, budgets, feature flags.

Rules:

1. Runtime must not rely on string-stripping template sections (no brittle “OUTPUT FORMAT” surgery).
2. Structured runtime context (if used) must be explicit, schema-validated, and parse-safe.

3.3 Retrieval Contract (backend)

Rules:

1. Exactly one query embedding per turn unless explicitly required (instrumented).
2. Retriever APIs accept precomputed embedding to avoid recomputation.
3. Chunk injection respects conversationId and “exclude conversation chunks” flags when structured history already exists.

3.4 Env/config (backend + portal)

Rules:

1. Env is loaded once, validated once, and exported as a typed object (env). No ad-hoc dotenv.config() inside services.
2. .env.example files are the only committed templates; no “env copy” files are committed.
3. env.example matches validation and actual usage (including DB pool flags and feature flags).

---

4) Issue register (duplication / overlap / logic flaws)

Legend: S=Severity (P0 critical → P3 low), R=Risk (Low/Med/High), Fix=planned PR.

ID	Area	Symptom	Impact	S	R	Primary files	Fix
1	Tool Policy	explicit_user_selected not enforced	observer/admin-only tools can be callable unintentionally	P0	Med	backend/src/services/toolExecutor.service.ts	PR1.1
2	Tool System	Hidden tool bypasses catalog	policy/enablement/audit drift	P0	Med	backend/src/services/langchain/langchainToolTurn.service.ts	PR1.2
3	Portal Chat	Group flow wired like single in places	broken UX + inconsistent state	P0	Low	aso-portal/src/components/chat/ChatWindow.tsx	PR1.3
4	Retrieval	Duplicate embeddings per turn	cost/latency + failure amplification	P1	Med	backend/src/services/aso/context.service.ts	PR3.1
5	Ingestion	delete-then-insert non-atomic	intermittent chunk loss	P1	Med	backend/src/services/aso/ingestion.service.ts	PR3.2
6	Background Jobs	no global concurrency/dedupe	DB/provider overload	P1	Med	backend/src/services/backgroundJob.service.ts	PR3.3
7	Embeddings	no HTTP timeout	hung requests → timeouts elsewhere	P1	Low	backend/src/services/embedding.service.ts	PR3.4
8	Tool Schema	3 schema sources	drift across runtimes	P1	High	toolSchemas.ts + llmToolGateway.service.ts	PR2.1/PR2.2
9	Prompt	brittle OUTPUT FORMAT stripping	prompt regressions on template edits	P2	Med	backend/src/services/langchain/chains/promptRunnable.ts	PR5.1
10	RuntimeCtx	JSON extraction via braces	silent failure on braces/formatting	P2	Med	backend/src/services/langchain/structuredRuntimeContext.ts	PR5.2
11	Dev Env	env sources diverge + late validation	onboarding breakage + prod drift	P2	Med	env.validation.ts + database.ts	PR4.1/PR4.2
12	Testing	Jest + Mocha split w/o strong boundaries	flaky CI + unclear test command	P3	Med	backend/jest.config.cjs, backend/tests/*	PR4.4

---

5) Phase-by-phase implementation (PR-ready specs)

This roadmap is designed so each PR is reviewable, testable, and can be deployed independently.

Phase 0 — Inventory + invariants (documentation + measurement)

PR0.1: System invariants + smoke-test runbook

Goal: Define “what must always be true” and how we verify it after each PR/deploy.\n\nDeliverables:\n- SYSTEM-INVARIANTS-AND-SMOKE-TESTS.md (new) under this folder.\n- A checklist used by reviewers and on-call.\n\nMust include invariants like:\n- Tool access policy enforcement is centralized.\n- Exactly one embedding per turn (when chunk retrieval enabled).\n- Background jobs are bounded.\n- Group chat payload parity with single.\n\nNo behavior changes.\n\n---\n\n### Phase 1 — Security & correctness (highest priority)\n\n#### PR1.1: Enforce explicit_user_selected tool policy\nGoal: Make tool access policy semantics match the catalog.\n\nScope:\n- Implement explicit_user_selected enforcement centrally in the tool executor layer.\n- Add tests for policy enforcement (both direct REST and LangChain tool turn if applicable).\n\nFiles likely touched:\n- backend/src/services/toolExecutor.service.ts\n- backend/src/config/toolCatalog.ts (confirm policy usage)\n- backend/src/controllers/toolPermissions.controller.ts / routes (if policy relies on user selection state)\n\nDefinition:\n- explicit_user_selected means: tool may only be executed if the request context indicates the user explicitly selected/approved it for this turn/session (exact storage TBD: request payload flag, session setting, or per-conversation tool-allowlist).\n\nSmoke tests:\n- Attempt to execute observer tools without explicit selection → 403.\n- Execute a normal allowed tool (quest/inventory/web) → OK.\n\n#### PR1.2: Route extract_and_save_user_facts through the Tool Platform\nGoal: Remove hidden bypasses.\n\nOptions (choose one and standardize):\n- A) Add it to toolCatalog with explicit policy (likely admin-only or self-only) and compile it like other tools.\n- B) If it must remain internal, still register in catalog as internal: true, excluded from prompt tool list by default, but enforce via the same policy/enablement path.\n\nSmoke tests:\n- Tool appears in canonical list and is auditable.\n- Policy enforcement applies consistently.\n\n#### PR1.3: Portal group chat correctness + payload parity\nGoal: Make group chat act like group chat, not “single chat with group socket”.\n\nScope:\n- Suggested prompts send via group pipeline when isGroup.\n- Group socket payload includes preferences, clientTimezone, clientLocale (or UI removes those options for group—pick one).\n- Implement group optimistic UI or explicitly rely on server echo (but consistent).\n- Fix ToolProgressPanel/status wiring to not read single state in group contexts.\n\nFiles likely touched:\n- aso-portal/src/components/chat/ChatWindow.tsx\n- aso-portal/src/components/chat/ToolProgressPanel.tsx\n- aso-portal/src/stores/utils/messageSendUtils.ts\n- aso-portal/src/services/groupSocketServices.ts\n\nSmoke tests:\n- Group suggested prompt posts into group history.\n- Web-search preference toggles affect group requests (if supported).\n- No stuck “thinking” on upload/send failure.\n\n---\n\n### Phase 2 — Consolidate tool schemas + eliminate multi-path drift\n\n#### PR2.1: Single schema registry + alias mapping\nGoal: toolSchemas.ts becomes the single schema source; alias normalization is centralized.\n\nScope:\n- Remove duplicate per-tool Zod schemas in legacy tool gateway or replace them with references to canonical schemas.\n- Centralize alias mapping to one place; ensure both LangChain and any legacy parsing uses it.\n\nPrimary files:\n- backend/src/services/tools/toolSchemas.ts\n- backend/src/services/toolExecutor.service.ts\n- backend/src/services/llmToolGateway.service.ts\n\n#### PR2.2: Legacy tool gateway becomes a thin adapter (or is retired)\nGoal: Either delete the legacy JSON tool-call stack, or constrain it to prompt-debug-only and ensure it cannot drift.\n\nScope:\n- If retained: parseLlmToolOutput is kept, but execution calls the canonical tool executor with canonical schema validation and policy enforcement.\n- Add tests proving parity with LangChain execution.\n\n#### PR2.3: One guardrails location\nGoal: avoid duplicate web/time “guardrails” across orchestration and tool-turn.\n\nDecision: place guardrails in exactly one layer:\n- Orchestration (aso.service.ts preflight) or\n- Agent runtime (langchainToolTurn.service.ts)\n\nAcceptance: traces show each guardrail at most once per turn.\n\n---\n\n### Phase 3 — Retrieval + ingestion reliability + cost\n\n#### PR3.1: One embedding per turn\nGoal: prevent re-embedding within retrievers.\n\nScope:\n- Retriever APIs accept queryEmbedding?: number[].\n- getThreeLayeredContext() uses the provided embedding for both chunk and non-chunk paths.\n- Instrument embedding calls per request.\n\n#### PR3.2: Atomic chunk ingestion\nGoal: eliminate delete-then-insert data loss.\n\nScope:\n- Wrap chunk replacement in a DB transaction.\n- Or UPSERT per (memoryId, chunkIndex) with cleanup of old rows.\n\n#### PR3.3: Bounded background jobs + per-key dedupe\nGoal: prevent overload on burst traffic.\n\nScope:\n- Introduce a concurrency limiter (global) and dedupe by (jobType, memoryId).\n- Longer-term: migrate to a persistent queue (BullMQ/pg-boss) if needed.\n\n#### PR3.4: Embedding HTTP timeouts\nGoal: avoid hung requests.\n\nScope:\n- Add axios timeout for OneAPI embedding calls.\n- (Optional) Add abort/timeouts for provider SDK calls where possible.\n\n---\n\n### Phase 4 — Env/config/scripts/tests/docs standardization\n\n#### PR4.1: Typed env module, early validation\nGoal: load and validate env once, before other imports consume it.\n\nScope:\n- Create backend/src/config/env.ts exporting env.\n- Remove service-local dotenv.config() (e.g. in embedding.service.ts).\n- Update server.ts startup sequence so env validation occurs before DB/app imports.\n\n#### PR4.2: Replace “env copy” files with .env.example\nGoal: stop committing env-like files; align onboarding.\n\nScope:\n- Replace backend/.env copy backend and aso-portal/.env copy - aso-portal with .env.example files.\n- Update SETUP-GUIDE.md accordingly.\n- Tighten .gitignore to prevent future variants.\n\n#### PR4.3: Remove duplicated build scripts; fail fast\nGoal: one blessed build path.\n\nScope:\n- Remove build1/copy-assets1 or alias them.\n- Ensure build fails on TS errors.\n\n#### PR4.4: Test runner strategy\nGoal: predictable testing.\n\nOptions:\n- A) Migrate Mocha/Chai → Jest (recommended).\n- B) Keep both, but hard-separate and provide explicit test:jest / test:mocha scripts.\n\n---\n\n### Phase 5 — Prompt system hardening + doc convergence\n\n#### PR5.1: Remove brittle prompt stripping\nGoal: stop using OUTPUT FORMAT fence stripping in runtime.\n\nScope:\n- Generate a LangChain-ready prompt variant that never includes sections needing stripping.\n\n#### PR5.2: Harden structured runtime-context extraction\nGoal: remove brace-based extraction fragility.\n\nScope:\n- Use an explicit sentinel payload (e.g. fenced JSON with exact markers) and strict schema validation.\n\n#### PR5.3: Docs hygiene\nGoal: remove references to ignored/stale sources and ignore build caches.\n\nScope:\n- Fix “Primary Source: docs/project_status.md” references if docs/ remains ignored.\n- Ignore docs-site/.vitepress/cache/**.\n\n---\n\n## 6) Metrics + instrumentation (must exist before large refactors)\n\n### Required counters (backend)\n- embedding.calls_per_turn (target: 1)\n- tool.execution.denied by reason (policy/protocol/enablement)\n- background_jobs.queued and background_jobs.running\n- memory_chunks.ingestion.failures\n- retriever.docs_returned by tier and source\n\n### Required user-visible behavior checks (portal)\n- No stuck “thinking” on failures.\n- Single and group payload parity is explicit.\n- Streaming and non-streaming flows show consistent tool progress.\n\n---\n\n## 7) “PR templates” (how each PR must be written)\n\nEach PR must include:\n- Summary (what/why)\n- Scope boundaries (what explicitly not touched)\n- Backwards-compat notes\n- Smoke test checklist (copy from invariants doc)\n- Rollback plan\n\n---\n\n## 8) Appendix: key hotspots\n\n### Backend hotspots\n- Tool enforcement: backend/src/services/toolExecutor.service.ts\n- LangChain tool assembly: backend/src/services/langchain/langchainToolTurn.service.ts\n- Legacy JSON tool parsing: backend/src/services/llmToolGateway.service.ts\n- Prompt stripping: backend/src/services/langchain/chains/promptRunnable.ts\n- Runtime context extraction: backend/src/services/langchain/structuredRuntimeContext.ts\n- Retrieval context: backend/src/services/aso/context.service.ts\n- Chunk ingestion: backend/src/services/aso/ingestion.service.ts\n- Background jobs: backend/src/services/backgroundJob.service.ts\n\n### Portal hotspots\n- Chat UI wiring: aso-portal/src/components/chat/ChatWindow.tsx\n- Unified send logic: aso-portal/src/stores/utils/messageSendUtils.ts\n- Group socket payload: aso-portal/src/services/groupSocketServices.ts\n- API env mismatch: aso-portal/src/api/axiosClient.ts, aso-portal/src/api/asoApi.ts\n+