Monitoring & Operational Logs

Application Logs

flow8 uses zerolog for structured JSON logging. Logs are written to:

Stdout — always (captured by Docker/systemd)
./storage/logs/ — rotated log files (path configurable via .config.yaml)

Log Format

Each log line is a JSON object:

{
  "level": "info",
  "time": "2024-01-15T10:30:00Z",
  "service": "flow_service",
  "flowId": "65abc123",
  "playId": "65abc456",
  "message": "play started"
}

Log Levels

Level	When Used
`debug`	Detailed execution tracing (disabled in production)
`info`	Normal operation events (flow started, completed, etc.)
`warn`	Recoverable issues (retry attempt, deprecated usage)
`error`	Failures requiring attention (DB error, module fail)
`fatal`	Startup failures (will exit the process)

Audit Logs

The audit MongoDB collection captures all significant platform actions. Every document includes:

Field	Description
`_id`	Document ID
`companyId`	Tenant context
`userId`	Actor
`timestamp`	When the action occurred
`action`	Action name (e.g., `flow.create`, `auth.login`, `play.run`)
`entityType`	Type of entity affected (flow, play, user, etc.)
`entityId`	ID of the affected entity
`before`	State before the action (for updates/deletes)
`after`	State after the action (for creates/updates)
`requestId`	Correlates with HTTP request
`ip`	Client IP address
`userAgent`	Client user agent

Logged Event Types

HTTP Requests — every inbound API request with method, path, status, duration
Auth Events — login, logout, token creation, token revocation
Entity CRUD — create/update/delete for: flows, groups, channels, schedules, users, KV entries, links, retention policies
Play Execution — flow triggered, play state transitions
Module Calls — component invocations during flow execution
Background Tasks — scheduler runs, retention cleanup cycles
System Events — startup, database registration, module availability checks

Querying Audit Logs

Via the API:

GET /api/audit?companyId=<id>&action=flow.create&from=2024-01-01&to=2024-01-31&size=50&page=1

Via MongoDB directly:

db.audit.find({
  companyId: ObjectId("..."),
  action: "play.run",
  timestamp: { $gte: ISODate("2024-01-01") }
}).sort({ timestamp: -1 }).limit(100)

Field Sanitization

The audit logger sanitizes sensitive fields before writing. Configured in pkg/logger/audit/sanitizer.go:

Passwords, tokens, and API keys are replaced with [REDACTED]
Fields exceeding AUDIT_LOG_FIELD_MAX_BYTES are truncated with a [TRUNCATED] suffix
The sanitization rules are entity-type-specific

Background Job Monitoring

Two background jobs run continuously:

Job	Schedule	What It Does
`CacheAuditFilters`	Every 3 minutes	Queries audit collection for distinct field values and caches them (4-minute TTL) for the filter dropdown UI
`ClearTTLCache`	Every 1 minute	Scans the in-memory TTL cache and evicts expired entries to prevent memory growth

The retention cleanup job runs every RETENTION_CLEANUP_INTERVAL (default: 2 minutes).

Monitor job execution by watching application logs for job-specific log entries.

Channel Runtime Monitoring

Each active channel holds a runtime state snapshot in the channels.runtime field. Check channel health via:

GET /api/channels/:id

The response includes current port allocation, active connection count, and last error (if any).

MongoDB Performance

Recommended monitoring approach:

# Real-time stats
mongotop --uri="mongodb://localhost:27017"
mongostat --uri="mongodb://localhost:27017"

Key collections to watch:

plays — grows fastest; ensure retention policies are running
audit — second largest; monitor size, enforce retention
logs — can grow large during heavy execution; consider TTL index

For Atlas deployments, use Atlas Performance Advisor and the built-in slow query log.

Alerting

Configure email alerts in .env:

ALERT_EMAIL_FROM=alerts@yourcompany.com
ALERT_EMAIL_HOST=smtp.yourcompany.com
ALERT_EMAIL_PORT=587
ALERT_EMAIL_USER=smtp_user
ALERT_EMAIL_PASS=smtp_password

Flow-level alert rules are defined per flow in the DBAlert model and managed via the UI or API.