Training System

Last Updated: 2025-01-16
Primary Source: docs/project_status.md
Service Files: backend/src/services/training.service.ts, backend/src/services/imageOcr.service.ts, backend/src/services/document.service.ts
Socket Handler: backend/src/socket/trainingHandler.ts
Controller: backend/src/controllers/training.controller.ts

---

Overview

The Training System enables administrators to manually train NPCs (Yexian, Chinatsu, etc.) by ingesting text, images, and documents. It includes direct NPC chat for testing, OCR for image extraction, and "learn by heart" synthesis that converts raw content into first-person core beliefs.

---

Architecture

Training System Flow

---

Core Components

1. Training Service

File: backend/src/services/training.service.ts

Key Functions:

ingestTextAsMemory(npcUserId, text)

• Ingests plain text as memory
• Source: manual_ingestion_text
• Direct memory storage (no synthesis)

ingestScreenshotAsMemory(file, npcUserId)

• Extracts text from image using OCR
• Supports multiple languages
• Source: ingested_screenshot
• Returns extracted text

ingestFileByHeart(file, npcUserId)

• Processes file for "learn by heart"
• Extracts text from file
• Synthesizes into first-person core beliefs
• Source: core_belief
• Returns synthesized beliefs

ingestTextByHeart(text, npcUserId, sourceName)

• Synthesizes text into core beliefs
• Uses LLM to convert to first-person perspective
• Source: core_belief
• Returns synthesized beliefs

directChatWithNpc(npcUserId, playerUserId, message)

• Direct 1-on-1 chat with NPC
• Bypasses group chat director
• Forces NPC response
• Returns NpcResponse object

2. Image OCR Service

File: backend/src/services/imageOcr.service.ts

Key Functions:

getOcrLanguageConfig()

• Returns OCR language configuration
• Supports multiple languages (English, Japanese, Chinese, Nepali, Hindi, etc.)
• Returns language codes for Tesseract

3. Document Service

File: backend/src/services/document.service.ts

Key Functions:

extractTextFromFile(file)

• Extracts text from various file types
• Supports PDF, DOCX, TXT, etc.
• Returns extracted text content

4. Training Socket Handler

File: backend/src/socket/trainingHandler.ts

Socket Events:

directNpcMessage

• Client sends direct message to NPC
• Data: { npcUserId, message }
• Emits npcThinking while processing
• Emits directNpcResponse with response
• Emits chatError on error

---

Training Methods

1. Text Ingestion

Method: Direct text input

Process:

1. Admin enters text in training panel
2. Text saved as memory with source manual_ingestion_text
3. No synthesis - stored as-is

Use Cases:

• Quick facts
• Direct knowledge injection
• Simple updates

2. Image OCR

Method: Screenshot/image upload

Process:

1. Admin uploads image
2. OCR extracts text using Tesseract
3. Multi-language support
4. Extracted text saved with source ingested_screenshot
5. Returns extracted text

Use Cases:

• Screenshot training
• Image-based knowledge
• Multi-language content

3. File Ingestion (Learn by Heart)

Method: Document upload

Process:

1. Admin uploads file (PDF, DOCX, TXT, etc.)
2. Text extracted from file
3. LLM synthesizes into first-person core beliefs
4. Beliefs saved with source core_belief
5. Returns synthesized beliefs

Synthesis Prompt:

• Converts third-person content to first-person
• Maintains NPC personality
• Formats as personal beliefs/memories

Use Cases:

• Document training
• Comprehensive knowledge ingestion
• Personality-aligned learning

4. Text by Heart

Method: Text input with synthesis

Process:

1. Admin enters text
2. LLM synthesizes into first-person core beliefs
3. Beliefs saved with source core_belief
4. Returns synthesized beliefs

Use Cases:

• Inline text training
• Quick synthesis
• Personality-aligned learning

---

Direct NPC Chat

Purpose

Test NPC responses in isolation without group chat director

Implementation

Function: directChatWithNpc(npcUserId, playerUserId, message)

Features:

• Bypasses group chat orchestration
• Forces NPC response (no silent responses)
• Direct 1-on-1 conversation
• Returns NpcResponse object

Socket Events

Client → Server:

• directNpcMessage - Send message to NPC

Server → Client:

• npcThinking - NPC is processing
• directNpcResponse - NPC response
• chatError - Error occurred

---

Memory Sources

Training-Related Sources

1. manual_ingestion_text

   • Direct text ingestion
   • No synthesis
   • Quick knowledge injection

2. ingested_screenshot

   • OCR-extracted text
   • Image-based training
   • Multi-language support

3. core_belief

   • Synthesized beliefs
   • First-person perspective
   • Personality-aligned

---

API Endpoints

Training Endpoints

• POST /api/training/ingest-text - Ingest text as memory
• POST /api/training/ingest-screenshot - Ingest image via OCR
• POST /api/training/ingest-file - Ingest file by heart
• POST /api/training/ingest-text-by-heart - Ingest text by heart
• POST /api/training/direct-chat - Direct NPC chat

---

Socket.IO Events

Training Namespace

Client → Server:

directNpcMessage

• Send direct message to NPC
• Data: { npcUserId: number, message: string }

Server → Client:

npcThinking

• NPC is processing
• Data: {}

directNpcResponse

• NPC response
• Data: NpcResponse

chatError

• Error occurred
• Data: { message: string }

---

Frontend Integration

Training Center UI

Components:

• Manual Training Panel
• Simulation Panel
• Direct NPC Chat Interface
• File Upload Interface
• Image Upload Interface

Features:

• NPC selection
• Text input
• File upload
• Image upload
• Chat interface
• Response display

---

Data Flow

---

Source Files

Primary Sources:

• backend/src/services/training.service.ts - Main training service
• backend/src/services/imageOcr.service.ts - OCR processing
• backend/src/services/document.service.ts - File text extraction
• backend/src/socket/trainingHandler.ts - Socket handler
• backend/src/controllers/training.controller.ts - API controller

Related Files:

• backend/src/services/aso.service.ts - NPC conversation
• backend/src/services/memory.service.ts - Memory storage
• backend/src/services/embedding.service.ts - LLM calls

---

Related Documentation

• Memory System - Memory storage and retrieval
• File Handling System - File processing
• 04-FEATURES/Training-Center.md - Training Center UI
• 00-OVERVIEW.md - Complete system overview