Storage

Overview

Fiberwise provides two distinct storage systems with abstract providers that support local file storage, cloud storage, and scoped access patterns. Understanding these systems is crucial for proper deployment and development.

App Bundle Storage System

The app bundle storage system is the platform infrastructure that handles app installations, deployments, and agent code storage.

🗂️ What It Stores

• App Bundles: ZIP files containing complete applications
• Agent Code: Python, JavaScript, and other agent implementations
• Static Files: HTML, CSS, JS files for web apps
• Extracted Content: Unzipped files organized by app and version

📁 Storage Structure

# App Bundle Storage (Web Static Files)
/app_bundles/
├── apps/
│   └── {app_id}/
│       └── {version_id}/
│           ├── dist/           # Web UI files (HTML, CSS, JS)
│           ├── agents/         # Agent implementations
│           └── manifest.yaml   # App configuration

# Entity Bundle Storage (Executable Components)
/entity_bundles/
├── agents/
│   └── {agent_id}/
│       └── {version_id}/
│           └── agent.py        # Processed agent code
├── functions/
│   └── {function_id}/
│       └── {version_id}/
│           └── function.py     # Function implementation
├── pipelines/
│   └── {pipeline_id}/
│       └── {version_id}/
│           └── pipeline.yaml   # Pipeline configuration
└── workflows/
    └── {workflow_id}/
        └── {version_id}/
            └── workflow.yaml   # Workflow definition

⚙️ Configuration

Configure storage directories and provider type:

# Local storage directories (development)
export STORAGE_PROVIDER="local"
export APP_BUNDLES_DIR="./.fiberwise-core/app_bundles/apps"
export ENTITY_BUNDLES_DIR="./.fiberwise-core/entity_bundles"
export UPLOADS_DIR="./.fiberwise-core/uploads"
export TEMP_UPLOADS_DIR="./.fiberwise-core/temp"

# S3 storage (production)
export STORAGE_PROVIDER="s3"
export S3_BUCKET_NAME="your-fiberwise-bucket"
export S3_REGION="us-east-1"
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

Storage Provider Architecture

Both storage systems use the same abstract provider pattern:

🏗️ Provider Interface

class StorageProvider(ABC):
    """Abstract base class for storage providers"""

    async def upload_file(self, file_path: str, destination_path: str) -> str
    async def download_file(self, storage_path: str, local_path: str) -> str
    async def extract_archive(self, file_path: str, extract_dir: str) -> str
    async def list_files(self, prefix: str) -> List[Dict[str, Any]]
    async def delete_file(self, storage_path: str) -> bool
    async def file_exists(self, storage_path: str) -> bool

📂 Local Storage Provider

File system-based storage for development and on-premise deployments:

• Base Directory: Configurable storage root
• Path Management: Automatic directory creation
• Security: Path validation and sanitization
• Performance: Direct file system access

☁️ S3 Storage Provider

AWS S3 compatible storage for scalable cloud deployments:

• S3 Compatible: Works with AWS S3, MinIO, DigitalOcean Spaces
• Scalable: Handles large files and high throughput
• Reliable: Built-in redundancy and durability
• Cost-effective: Pay-per-use pricing model

Agent Runtime Storage System

The agent runtime storage system provides scoped, isolated storage for agents and applications during execution. This is separate from the app bundle storage and is injected into agents at runtime.

🔒 Automatic Scoping

Storage is automatically isolated by context to ensure security and prevent data leakage:

/storage/agents/{agent_id}/
├── files/           # Agent-created files
├── cache/           # Temporary cache data
└── logs/            # Agent execution logs

/storage/apps/{app_id}/
├── uploads/         # User uploaded files
├── shared/          # Cross-agent shared data
└── backups/         # Application backups

/storage/users/{user_id}/
├── documents/       # User documents
├── preferences/     # User settings
└── private/         # Private user data

🔑 Key Features

• Automatic Injection: Storage provider injected into agents at runtime
• Scope Enforcement: Agents can only access their scoped storage areas
• Independent Configuration: Can use different provider than app bundle storage
• API Integration: Available through REST API and SDKs

File Operations

Common storage operations available through the API:

📤 Upload Files

curl -X POST http://localhost:7001/api/v1/storage/upload \
  -H "Authorization: Bearer your-api-key" \
  -F "[email protected]" \
  -F "path=/documents/reports/"

📥 Download Files

curl -X GET http://localhost:7001/api/v1/storage/files/{file_id} \
  -H "Authorization: Bearer your-api-key" \
  -o downloaded_file.pdf

📋 List Files

curl -X GET http://localhost:7001/api/v1/storage/files?prefix=/documents/ \
  -H "Authorization: Bearer your-api-key"

Agent Integration

Use storage in your agents with automatic scoping:

🐍 Python SDK

from fiberwise_sdk import FiberAgent

class FileProcessorAgent(FiberAgent):
    def __init__(self):
        super().__init__()
        # Storage provider is automatically injected
        self.storage = self.get_storage_provider()

    def process(self, input_data):
        # Upload a file to agent's scoped storage
        file_url = await self.storage.upload_file(
            file_path=input_data['file_path'],
            destination_path='processed/output.txt'
        )

        # List files in agent's storage
        files = await self.storage.list_files(prefix='processed/')

        return {
            "uploaded_file": file_url,
            "file_count": len(files)
        }

Storage Configuration

Configure both storage systems independently for your deployment needs:

📦 App Bundle Storage Configuration

Controls where app bundles, agent code, and static files are stored:

# Local storage (development/on-premise)
export STORAGE_PROVIDER="local"
export APP_BUNDLES_DIR="/var/lib/fiberwise/app_bundles"

# S3 storage (production/cloud)
export STORAGE_PROVIDER="s3"
export S3_BUCKET_NAME="your-fiberwise-app-storage"
export S3_REGION="us-east-1"
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

# Optional: Custom S3 endpoint (for MinIO, etc.)
export S3_ENDPOINT_URL="https://minio.yourcompany.com"

🔒 Agent Runtime Storage Configuration

Controls agent data storage (can be different from app bundle storage):

# Agent storage configuration
export AGENT_STORAGE_PROVIDER="local"
export AGENT_STORAGE_BASE_PATH="/var/lib/fiberwise/agent_storage"
export AGENT_STORAGE_MAX_FILE_SIZE="100MB"

# For agent S3 storage
export AGENT_STORAGE_PROVIDER="s3"
export AGENT_S3_BUCKET="your-agent-data-bucket"
export AGENT_S3_REGION="us-east-1"

🏗️ Hybrid Configuration Example

You can mix storage types - for example, S3 for app bundles but local for agent data:

# App bundles in S3 (scalable, shared across instances)
export STORAGE_PROVIDER="s3"
export S3_BUCKET_NAME="company-app-bundles"

# Agent data local (fast access, instance-specific)
export AGENT_STORAGE_PROVIDER="local"
export AGENT_STORAGE_BASE_PATH="/opt/fiberwise/agent_data"

📄 Configuration File

# fiberwise.yaml
app_storage:
  provider: "s3"
  s3:
    bucket: "fiberwise-app-bundles"
    region: "us-east-1"
    endpoint_url: null  # Use AWS S3

agent_storage:
  provider: "local"
  local:
    base_path: "/var/lib/fiberwise/agent_storage"
    max_file_size: "100MB"
    allowed_extensions: [".pdf", ".txt", ".json", ".csv", ".png", ".jpg"]

Security Features