Skip to content
flow8

System Architecture Overview

Architecture at a Glance

flow8 is a modular, distributed workflow automation platform designed for enterprise IT, security, and business process automation. The system separates concerns into layers: HTTP routing (Gorgany framework), business logic (services), data persistence (MongoDB), and pluggable runtime capabilities (component system).

High-Level Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         CLIENTS                                  │
│        Browser (UI)  │  REST API  │  WebSocket  │  MCP Agent   │
└──────────┬──────────────┬─────────────┬──────────────┬───────────┘
           │              │             │              │
           └──────────────┴─────────────┴──────────────┴───────────┐
                                                                    │
           ┌────────────────────────────────────────────────────────┤
           │         HTTP Server (Gorgany Framework)                │
           │  Port 4454 (HTTP)  │  Port 4445 (MCP Server)         │
           └────────────────────────────────────────────────────────┘
                    │            │             │
        ┌───────────┴───┬────────┴────────┬───┴─────────┐
        │               │                 │             │
    ┌───▼────────┐  ┌──▼──────────┐  ┌──▼────────┐ ┌──▼──────────┐
    │ HTTP Routes│  │ Middleware   │  │ WebSocket │ │ MCP Tools   │
    │            │  │ (Auth, CORS) │  │ Handler   │ │ Execution   │
    └───┬────────┘  └──┬──────────┘  └──┬────────┘ └──┬──────────┘
        │              │                 │             │
        └──────────────┴─────────────────┴─────────────┴──────────┐
                                                                   │
           ┌──────────────────────────────────────────────────────┤
           │       Service Layer (70+ Domain Services)            │
           │  Flow  │  Play  │  Layer  │  AI  │  Scheduler  │    │
           │  Auth  │  Audit │  Config │ ...                     │
           └──────────────────┬───────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
        ┌─────▼─────┐   ┌────▼──────┐   ┌──▼────────────┐
        │  MongoDB   │   │ Component  │   │ Background    │
        │            │   │ Registry   │   │ Jobs (Cron,   │
        │ 43+        │   │            │   │ Cleanup)      │
        │ Collections│   │ Storage    │   │               │
        │            │   │ AI         │   │               │
        └────────────┘   │ Database   │   └───────────────┘
                         │ Providers  │
                         │ Custom     │
                         └────────────┘
                              │
              ┌───────────────┴────────────────┐
              │                                │
        ┌─────▼──────┐  ┌───────────────┐  ┌──▼──────────┐
        │ External    │  │ AWS S3,       │  │ Channels    │
        │ Services    │  │ Google Cloud  │  │ 7701-7799   │
        │             │  │ Microsoft     │  │ (Async      │
        │ OAuth2      │  │ Graph, etc.   │  │  Execution) │
        │ Providers   │  │               │  │             │
        └─────────────┘  └───────────────┘  └─────────────┘

Gorgany Framework

flow8 uses Gorgany (github.com/osbits/gorgany), a custom HTTP framework built on Go’s standard library. Gorgany provides:

  • Routing: Type-safe path and query parameter binding
  • Middleware pipeline: Request/response interception for auth, CORS, logging
  • Provider/DI system: Dependency injection via struct tags (container:"inject")
  • Error handling: Structured error responses with HTTP status codes
  • CORS support: Configurable allowed origins, credentials, methods

flow8 registers all controllers and middleware via the provider pattern in pkg/provider/, and the Gorgany app is bootstrapped in server.go.

Provider & Dependency Injection

flow8 uses Gorgany’s provider system for inversion of control:

type MyService struct {
    FlowService    *service.FlowService       `container:"inject"`
    Storage        component.StorageComponent `container:"inject"`
    Logger         *zerolog.Logger            `container:"inject"`
    DBClient       *mongo.Client              `container:"inject"`
}

Services are registered in pkg/provider/ and resolved at startup. This approach:

  • Decouples dependencies from concrete implementations
  • Enables testing via mock injection
  • Centralizes configuration management
  • Supports late-binding of component implementations

Key providers:

  • ProvideLogger() — zerolog instance
  • ProvideDBClient() — MongoDB connection
  • ProvideComponents() — named runtime capabilities
  • ProvideServices() — 70+ domain services

Component System

The component system is a critical abstraction layer that allows swappable implementations of runtime capabilities without code changes. Components are named, configurable, and selected at execution time.

Component Kinds

KindPurposeExamples
storageFile and artifact persistenceLocal filesystem, AWS S3, Google Cloud Storage
aiAI model providersOpenAI, Anthropic Claude, Mistral, Ollama
dbSQL database connectivityPostgreSQL, MySQL (for workflow execution)
requestHTTP client for external APIsHTTP with retries, TLS validation
consoleExecution logging and debuggingConsole output, structured logs

Configuration

Components are stored in the component_configs MongoDB collection with structure:

{
  "_id": "ObjectID",
  "name": "default-openai",
  "kind": "ai",
  "company_id": "...",
  "config": {
    "provider": "openai",
    "api_key": "[encrypted]",
    "model": "gpt-4",
    "temperature": 0.7
  },
  "is_default": true,
  "created_at": "...",
  "updated_at": "..."
}

Default vs Named Components

  • Default component: One per kind per installation. Selected if no override is specified.
  • Named component: Explicitly referenced by name in a flow or layer.
  • Per-layer override: A flowlet can override its AI provider via ComponentConfigIds map.

Example: A flowlet that uses a specific Mistral configuration:

{
  "component_config_ids": {
    "ai": "mistral-production"
  }
}

This decouples flow definitions from provider credentials and enables:

  • Zero-code provider switching: Change AI provider by updating component config
  • Multi-tenancy isolation: Each company has its own default components
  • Cost optimization: Route expensive operations to cheaper providers
  • Gradual migration: Test new providers on subset of flows before full rollout

MongoDB Data Model

flow8 uses MongoDB as the primary data store with 43+ collections organized by domain:

Core Flow Entities

  • flows (DBFlow) — Flow definitions, metadata, version history
  • flowlets (DBFlowlet) — Individual steps in a flow with module references
  • plays (DBPlay) — Individual execution instances with full state tracking
  • play_layers (DBPlayLayer) — Module execution records within a play

Configuration & Integration

  • companies — Multi-tenancy root
  • users — User accounts with encrypted credentials
  • component_configs — Named runtime capabilities
  • db_links (DBLink) — Encrypted OAuth2 and API key credentials
  • system_config — Global settings and overrides

Operational

  • audit_logs — Comprehensive audit trail (every request, CRUD, auth event)
  • flows_all — Retention-managed archive
  • kv_stores — Per-execution key-value state (scoped: global, flow, flow_group)
  • retention_policies — Automated cleanup rules

Metadata

  • modules (cached) — 135+ built-in module catalog
  • templates — Pre-built flow templates
  • flow_groups — Grouping and organization
  • tags — Custom metadata and categorization

Multi-tenancy is enforced at the query level: every read operation filters by company_id, and all mutations include company_id in writes.

Startup Sequence

flow8 initializes through a carefully ordered bootstrap process:

  1. Load environment variables — Read .env file via godotenv
  2. Initialize logging — Configure zerolog, set up audit logger with field sanitization
  3. Parse configuration — Load config/config.yml with Viper (supports ${VAR} substitution)
  4. Connect to MongoDB — Establish primary database connection with retry logic
  5. Initialize components — Load component registry, resolve default AI/storage/db providers
  6. Register services — Inject 70+ domain services via provider system
  7. Setup Gorgany app — Register HTTP routes, middleware, WebSocket handlers, MCP server
  8. Start HTTP listener — Begin accepting requests on port 4454 (configurable via SERVER_PORT)

Background jobs are spawned after HTTP startup:

  • Scheduler — Cron-based flow execution (gorhill/cronexpr)
  • Retention cleanup — Every 2 minutes, batch delete aged entries
  • Audit filter cache — Every 3 minutes, refresh cached audit query filters
  • TTL cache cleanup — Every 1 minute, evict expired test cases and modules

Port Layout

PortServiceProtocolPurpose
4454flow8 API & UIHTTP/WebSocketMain application (configurable via SERVER_PORT)
4445MCP ServerHTTPModel Context Protocol (JWT-authenticated)
7701–7799Channel PortsWebSocket/TCPAsync execution channels (100 concurrent per instance, range scales with instance count)

For multi-instance deployments, each instance reserves a block of channel ports. For example:

  • Instance 1: 7701–7799
  • Instance 2: 7800–7899
  • Instance 3: 7900–7999

Multi-Tenancy Architecture

Every flow8 entity includes a company_id field, and all database queries are automatically scoped to the authenticated user’s company. This ensures:

  • Isolation: Data from different organizations never mixes
  • Performance: Indexed queries filter by company first
  • Simplicity: No complex access control lists at the data layer

Users can belong to multiple companies, and the session context determines which company’s data is accessible in each request.

Key Design Principles

  1. Separation of Concerns: HTTP routing, business logic, and data persistence are clearly layered
  2. Pluggability: Components, modules, and external integrations are loosely coupled
  3. Observability: Comprehensive audit logging and zerolog structured logging throughout
  4. Security by Default: Field-level encryption, RBAC, multi-tenancy at the query layer
  5. Scalability: Stateless application layer, channel-based async execution, in-memory caching
  6. Developer Experience: Reflection-based module discovery, provider-based DI, clear package organization