Memory System Architecture

Last Updated: 2026-02-06
Content-Type: Explanation
Audience: Developers, Architects

---

Architecture Overview

The Memory System implements a three-tiered RAG (Retrieval-Augmented Generation) system with chunked retrieval for higher-quality recall and continuity.

---

Three-Tiered Memory Structure

---

Data Flow

Memory Retrieval Flow

Memory Retrieval Visualization

---

Key Components

Memory Service

File: backend/src/services/aso/memory.service.ts

The Memory Service handles all memory operations including:

• Memory storage and retrieval
• Embedding generation coordination
• Vector similarity search
• Memory categorization by source

Context Service

File: backend/src/services/aso/context.service.ts

The Context Service orchestrates context gathering:

• Retrieves memories from all three tiers
• Applies similarity search for relevant memories
• Combines memories with persona, emotions, quests, inventory
• Provides unified context for ASO responses

User Familiarity Graph (User Network Graph)

Files:

• backend/src/services/userGraph.service.ts
• backend/src/services/postResponsePipeline.service.ts

This is a lightweight, non-LLM “network graph” derived from user inputs (high-signal patterns like preferred name, interests, projects, style cues).

• Storage: persisted in aso_states.metadata.user_graph (per user)
• Update point: post-response pipeline (fire-and-forget) updates the graph after each turn
• Usage: injected into the prompt via sharedMemories as a compact summary so Yexian can talk in the user’s familiar style without inventing details

Embedding Service

File: backend/src/services/embedding.service.ts

The Embedding Service handles:

• Embedding generation (Gemini or OneAPI/OpenAI-compatible embeddings)
• In-process embedding caching for performance (env: EMBEDDING_CACHE_SIZE)
• Vector dimension management (env: VECTOR_DIM, default 1536)

Chunk Store + Ingestion

Files:

• backend/src/services/aso/ingestion.service.ts
• backend/src/models/memoryChunk.model.ts

Memories are chunked into memory_chunks to improve retrieval precision and allow hybrid retrieval.

Hybrid Retrieval (Vector + Keyword)

Files:

• backend/src/services/langchain/retrievers/pgvectorMemoryChunksRetriever.ts
• Migration index: memory_chunks_content_tsv_idx (GIN to_tsvector('simple', content))

The retriever:

• mixes vector similarity with keyword hits
• applies tier weighting + time decay
• optionally boosts chunks from the current conversationId (continuity)

RAG Debug Traces (Optional)

Files:

• backend/src/services/ragTrace.service.ts
• backend/src/models/ragTrace.model.ts

Enable with RAG_TRACE=true to persist the latest retrieved chunks for admin debugging.

---

System Relationships

The Memory System integrates with:

1. ASO Service - Main orchestrator that uses memory context
2. Timeline System - Memories linked to timeline events
3. Observer System - Level-8 observer can access all memories
4. File Handling System - Documents and images ingested as memories
5. Post-Response Pipeline - Saves memories after conversations

---

Design Decisions

Why Three Tiers?

• World Memory: Shared knowledge reduces duplication and ensures consistency
• ASO Core Memory: Maintains identity across all instances and users
• Shared Memory: Enables personalized experiences per user

Why Vector Similarity Search?

• Semantic understanding beyond keyword matching
• Efficient retrieval of relevant context
• Scales well with large memory databases
• Natural language queries work effectively

Why pgvector?

• Native PostgreSQL extension
• Efficient vector operations
• No separate vector database needed
• Integrated with existing data model

---

Performance Considerations

Caching

• Memory queries cached for frequently accessed content
• Embedding generation cached to reduce API calls
• Context gathering optimized for common patterns

Optimization

• pgvector indexes for fast similarity search
• Limit memory retrieval to top N results
• Batch embedding generation for multiple memories
• Async embedding generation to avoid blocking

---

Integration Points

ASO Service Integration

• handleNpcConversationTurn() calls gatherFullContext()
• Context includes memories from all three tiers
• Memories influence AI responses and behavior

Post-Response Pipeline

• After conversation, memories are saved via addMemory()
• Embeddings generated asynchronously
• Vision context and file context included
• User familiarity graph is updated from user input (non-blocking)

Observer System Integration

• Level-8 observer can access all memories
• Memories linked to timeline events
• Memory export for training future agents

---

Related Documentation

• API Reference - Function signatures and usage
• Data Models - Database schema details
• How to Add Memory - Practical usage examples
• Timeline System - Event tracking
• Observer System - Observer integration