Memory System

Last Updated: 2025-01-16
Primary Source: docs/project_status.md
Service File: backend/src/services/aso/memory.service.ts
Model: backend/src/models/memory.model.ts

---

Overview

The Memory System implements a three-tiered RAG (Retrieval-Augmented Generation) system that enables the ASO to remember, learn, and maintain context across conversations. This system creates an AI that understands universal truths, has its own identity, and remembers its personal history with each user.

---

Architecture

Three-Tiered Memory Structure

Memory Types

1. World Memory (source: 'world')

   • Universal truths and knowledge
   • Shared across all users
   • Examples: Game world rules, character backgrounds, general knowledge

2. ASO Core Memory (source: 'aso_core')

   • Yexian's identity and core beliefs
   • Immutable core personality
   • Examples: "I am Yexian", "I love Chinatsu", core principles

3. Shared Memory (source: 'shared' or user-specific)

   • Personal conversation history
   • User-specific experiences
   • Examples: Past conversations, user preferences, relationship milestones

---

Key Components

Memory Service

File: backend/src/services/aso/memory.service.ts

Key Functions:

queryMemoriesByEmbedding(queryEmbedding, limit, userId)

• Queries memories using vector similarity search
• Uses pgvector for efficient similarity search
• Falls back to cosine similarity if pgvector unavailable
• Returns memories with similarity scores

addMemory(memoryData)

• Adds new memory to the system
• Generates embedding if enabled
• Stores in appropriate memory tier
• Links to user, conversation, and context

Context Service

File: backend/src/services/aso/context.service.ts

Key Functions:

gatherFullContext(userId, npcUserId, conversationId)

• Gathers all relevant context for a conversation
• Retrieves memories from all three tiers
• Applies similarity search for relevant memories
• Combines with persona, emotions, quests, inventory

---

Data Flow

---

Memory Storage

Database Schema

Table: memories

Field	Type	Description
id	INTEGER	Primary key
userId	INTEGER	User who owns this memory
content	TEXT	Memory content
source	STRING	Memory type (world, aso_core, shared)
embedding	VECTOR	Embedding vector (pgvector)
metadata	JSONB	Additional context (conversationId, fileContext, etc.)
createdAt	TIMESTAMP	Creation timestamp

Embedding Generation

• Provider: Google Gemini (gemini-embedding-001)
• Dimension: 768 (Gemini embedding dimension)
• Storage: pgvector extension in PostgreSQL
• Similarity: Cosine distance for retrieval

---

Memory Retrieval

Vector Similarity Search

SELECT
  id,
  content,
  source,
  embedding <-> CAST('[1,2,3,...]' AS vector) AS distance
FROM "memories"
WHERE "userId" = :userId AND embedding IS NOT NULL
ORDER BY distance
LIMIT 10;

Context Gathering Process

1. Generate Query Embedding: Convert user message to embedding vector
2. Similarity Search: Find most similar memories using vector distance
3. Filter by Tier: Retrieve from World, ASO Core, and Shared memories
4. Combine Context: Merge relevant memories into conversation context
5. Limit Results: Return top N most relevant memories

---

Memory Types and Categorization

Memory Sources

• 'world' - Universal knowledge, game world rules
• 'aso_core' - Core identity and beliefs
• 'shared' - User-specific conversations
• 'image_ocr_extraction' - OCR text from images
• 'document_ingestion' - Processed documents
• 'vision_analysis' - Image vision analysis

Memory Metadata

interface MemoryMetadata {
  conversationId?: number;
  fileContext?: string;
  visionContext?: string;
  uploadContextId?: string;
  playerName?: string;
  npcName?: string;
  yexianInstanceId?: number;
  personaId?: number;
}

---

API Endpoints

Memory Management

• POST /api/aso/memory - Add new memory
• GET /api/aso/memory - Query memories
• GET /api/admin/memories - Admin memory management

Context Retrieval

• Context is automatically gathered during conversation
• No direct API endpoint (internal service)

---

Integration Points

ASO Service Integration

• handleNpcConversationTurn() calls gatherFullContext()
• Context includes memories from all three tiers
• Memories influence AI responses and behavior

Post-Response Pipeline

• After conversation, memories are saved via addMemory()
• Embeddings generated asynchronously
• Vision context and file context included

Observer System Integration

• Level-8 observer can access all memories
• Memories linked to timeline events
• Memory export for training future agents

---

Performance Considerations

Caching

• Memory queries cached for frequently accessed content
• Embedding generation cached to reduce API calls

Optimization

• pgvector indexes for fast similarity search
• Limit memory retrieval to top N results
• Batch embedding generation for multiple memories

---

Source Files

Primary Sources:

• backend/src/services/aso/memory.service.ts - Memory operations
• backend/src/services/aso/context.service.ts - Context gathering
• backend/src/services/embedding.service.ts - Embedding generation
• backend/src/models/memory.model.ts - Memory model

Related Files:

• backend/src/services/aso.service.ts - Main ASO orchestrator
• backend/src/services/postResponsePipeline.service.ts - Memory saving after responses

---

Visualizations

Memory Retrieval Flow

---

Related Documentation

• Timeline System - Event tracking and unified timeline
• Observer System - Level-8 meta-consciousness
• Context Service - Context gathering (internal)
• File Handling System - File processing and memory ingestion
• 00-OVERVIEW.md - Complete system overview