Docker Deployment Guide

Architecture Overview

The persistence system consists of two separate Docker containers:

1. Engine container (writes command logs)
2. Snapshotter container (generates snapshots and manages archival)

These containers must share access to the persistence volume, but can handle archived data separately.

Volume Configuration

Required Directory Structure

/app/persistence/           # Main persistence directory
└── archived/              # Archive subdirectory

Volume Considerations

• Engine requires read/write access to /app/persistence
• Snapshotter requires read/write access to both /app/persistence and /app/persistence/archived
• Archived directory can optionally be mounted as a separate volume

Cloud Deployment Considerations

Storage Classes

• Main persistence volume:

  • Requires high IOPS
  • Should use SSD-backed storage
  • Example: AWS EBS gp3 or io2
  • Size based on expected daily command log generation

• Archive volume:

  • Can use slower, cheaper storage
  • Consider automated lifecycle policies
  • Example: AWS EBS st1 or sc1
  • Size based on retention requirements

Volume Management

volumes:
  persistence:
    driver: ebs  # Example for AWS
    driver_opts:
      type: gp3
      size: 100
      iops: 3000
  archive:
    driver: ebs
    driver_opts:
      type: st1
      size: 500

Operational Considerations

Process Flow

1. Engine writes command logs to /app/persistence
2. Engine rotates to new command log periodically
3. Snapshotter detects completed command logs
4. Snapshotter generates new snapshot
5. Snapshotter moves older files to /app/persistence/archived

Latency and Throughput

1. Engine implements write-ahead, persisting all commands before processing
2. Persistence volume iops setting will impact latency.
3. Higher iops will enable lower latency through the engine.

Monitoring

Monitor:

• Available space in both volumes
• Rate of command log generation
• Snapshot generation frequency
• Archive directory growth

Backup Strategies

• Archive volume can be backed up independently
• Consider point-in-time snapshots of the persistence volume
• Implement retention policies for archived data