File Handling System

Last Updated: 2025-01-16
Primary Source: docs/project_status.md, docs/PROVIDER_AGNOSTIC_FILE_REFERENCE_IMPLEMENTATION.md
Service File: backend/src/services/fileProcessing.service.ts, backend/src/services/fileReference.service.ts

---

Overview

The File Handling System implements a provider-agnostic file reference system that abstracts provider-specific differences, enabling clean, maintainable code while supporting multiple LLM providers and storage backends.

---

Architecture

File Processing Pipeline

---

Provider-Agnostic Design

Canonical File References

Concept: All files are stored with canonical internalFileId that maps to provider-specific IDs.

Model: backend/src/models/fileReference.model.ts

Table: file_references

Fields:

• id - Primary key (UUID)
• internal_file_id - Canonical ID (unique)
• firebase_url - Firebase storage URL
• mime_type - File MIME type
• file_size - File size in bytes
• checksum - File checksum
• openai_file_id - OpenAI file ID (if uploaded)
• gemini_file_uri - Gemini file URI (if uploaded)
• claude_file_id - Claude file ID (if uploaded)
• provider_used - Provider that was used
• upload_method - 'one-api' or 'direct'

Normalization Layer

Service: backend/src/services/fileNormalization.service.ts

Function: normalizeToProviderFormat(contentItems, provider)

Process:

1. Receives canonical format (file_reference, image_reference, image)
2. Detects provider from model
3. Converts to provider-specific format
4. Returns normalized payload

Supported Providers:

• OpenAI (GPT-4 Vision, etc.)
• Gemini (Gemini Pro Vision)
• DeepSeek
• Grok
• Fallback for unknown providers

---

File Upload Flow

1. File Upload

Endpoint: POST /api/files/upload

Process:

1. Client uploads file (multipart form)
2. Backend stores file in Firebase/S3
3. Creates canonical file reference
4. Returns uploadContextId and URLs

2. File Reference Creation

Service: backend/src/services/fileReference.service.ts

Function: createFileReference(fileData)

Process:

1. Generate internalFileId (UUID)
2. Store file metadata
3. Map to provider IDs (if uploaded to provider)
4. Store in file_references table

3. Provider Upload

Service: backend/src/services/llmFileUpload.service.ts

Hybrid Approach:

1. Try One-API first (/files endpoint)
2. Fallback to direct provider if One-API unavailable
3. Store both internal ID and provider ID
4. Track upload method

---

Multimodal Payload Building

Service

File: backend/src/services/multimodalPayload.service.ts

Functions:

• buildMultimodalPayload() - Build payload with canonical formats
• buildMultimodalPayloadNormalized() - Build and normalize to provider
• buildStructuredMultimodalPayloadNormalized() - Structured format

Process

1. Gather Content Items:

   • Text messages
   • File references (internalFileId)
   • Image references
   • Public image URLs

2. Normalize to Provider:

   • Convert canonical formats to provider format
   • Map internalFileId to provider file ID
   • Format images for provider (base64, URL, etc.)

3. Build Payload:

   • Combine text and media
   • Format for provider API
   • Return ready-to-send payload

---

File Types and Processing

Images

Processing:

• Instant Vision: Sent as Base64 for immediate AI vision
• Long-term Storage: Uploaded to Firebase for persistence
• Vision Analysis: analyzeImageContent() extracts context
• Memory Storage: Vision context saved with memory

Normalization:

• OpenAI: image_url with base64 or URL
• Gemini: inline_data with base64
• DeepSeek: Base64 in content

Documents

Supported Formats:

• PDF
• DOCX
• TXT
• RTF

Processing:

• Document Ingestion: Processed via RAG pipeline
• Text Extraction: Extracted text stored in memory
• Provider Upload: Large documents uploaded to provider (if supported)
• Memory Storage: Document content ingested into ASO memory

Screenshots

Tool: screenshot_ocr

Process:

1. User uploads screenshot
2. OCR extracts text
3. Text stored as user-scoped memory
4. OCR text appended to chat

---

Canonical Content Types

1. file_reference

Format:

{
  type: 'file_reference',
  internalFileId: string,
  mimeType?: string
}

Usage: Uploaded files with canonical ID

2. image_reference

Format:

{
  type: 'image_reference',
  internalFileId: string,
  mimeType?: string
}

Usage: Uploaded images with canonical ID

3. image

Format:

{
  type: 'image',
  url: string,
  mimeType?: string
}

Usage: Public image URLs

---

Data Flow

---

API Endpoints

File Management

• POST /api/files/upload - Upload file
• GET /api/files/:id - Get file reference
• DELETE /api/files/:id - Delete file

---

Source Files

Primary Sources:

• backend/src/services/fileProcessing.service.ts - File processing
• backend/src/services/fileReference.service.ts - File reference management
• backend/src/services/fileNormalization.service.ts - Provider normalization
• backend/src/services/multimodalPayload.service.ts - Payload building
• docs/PROVIDER_AGNOSTIC_FILE_REFERENCE_IMPLEMENTATION.md - Complete implementation guide

Related Files:

• backend/src/services/llmFileUpload.service.ts - Provider upload
• backend/src/services/vision.service.ts - Vision analysis
• backend/src/services/imageOcr.service.ts - OCR processing

---

Related Documentation

• Memory System - Memory storage and document ingestion
• 03-PLATFORMS/Web-Portal.md - File upload in web portal
• 03-PLATFORMS/Discord-Bot.md - File handling in Discord
• 06-ARCHITECTURE/Data-Flow.md - File processing flow
• 00-OVERVIEW.md - Complete system overview