Audit Logging

Audit Logging System

flow8 maintains a comprehensive audit trail of all significant system events. Every request, CRUD operation, authentication event, and module execution is logged with timestamps, user context, and field sanitization.

Audit Log Schema

type DBAuditLog struct {
    ID              primitive.ObjectID
    Timestamp       time.Time
    CompanyID       primitive.ObjectID
    UserID          primitive.ObjectID
    UserEmail       string
    UserIPAddress   string              // From X-Forwarded-For header
    UserAgent       string              // Browser/client info

    // HTTP Request context
    HTTPMethod      string              // GET, POST, PUT, DELETE, etc.
    HTTPPath        string              // /api/v1/flows, /api/v1/plays/:id
    HTTPStatus      int                 // 200, 404, 500, etc.
    RequestDuration time.Duration       // Execution time

    // Action details
    ActionType      string              // "flow_create", "flow_execute", "auth_login"
    ResourceType    string              // "flow", "play", "user", "integration"
    ResourceID      string              // ID of affected resource

    // State changes (for CRUD)
    BeforeState     map[string]interface{}  // [sanitized]
    AfterState      map[string]interface{}  // [sanitized]

    // Error tracking
    ErrorMessage    string              // If request failed
    ErrorType       string              // Type of error

    // Metadata
    CorrelationID   string              // Trace requests across services
    SessionID       string              // Session identifier
    Tags            map[string]string   // Custom tags for filtering
}

MongoDB Index

db.audit_logs.createIndex(
  { company_id: 1, timestamp: -1 },
  { background: true }
);

db.audit_logs.createIndex(
  { user_id: 1, timestamp: -1 },
  { background: true }
);

db.audit_logs.createIndex(
  { resource_type: 1, resource_id: 1, timestamp: -1 },
  { background: true }
);

db.audit_logs.createIndex(
  { expires_at: 1 },
  { expireAfterSeconds: 0 }  // TTL index for retention
);

Logged Events

Authentication Events

Event	Details
auth_login_success	User login with username/password
auth_login_failure	Failed login (wrong password)
auth_logout	User logout
auth_oauth_callback	OAuth2 successful authentication
auth_session_expired	Session TTL exceeded
auth_api_key_created	API key generated
auth_api_key_revoked	API key revoked
auth_mfa_enabled	Multi-factor authentication enabled
auth_mfa_disabled	MFA disabled

Example log entry:

{
  "_id": "ObjectID",
  "timestamp": "2026-04-04T10:23:45Z",
  "company_id": "company_123",
  "user_id": "user_456",
  "user_email": "alice@company.com",
  "user_ip_address": "203.0.113.42",
  "action_type": "auth_login_success",
  "resource_type": "user",
  "http_method": "POST",
  "http_path": "/api/v1/auth/login",
  "http_status": 200,
  "request_duration": 250000000,  // 250ms in nanoseconds
  "session_id": "sid_abc123..."
}

Entity CRUD Events

Event	Resource	Details
flow_created	flow	New flow definition created
flow_updated	flow	Flow edited
flow_deployed	flow	Flow published
flow_deleted	flow	Flow removed
flowlet_created	flowlet	Step added to flow
flowlet_updated	flowlet	Step configuration changed
user_created	user	New user account created
user_role_changed	user	User role/permissions modified
user_deleted	user	User account removed
integration_linked	integration	OAuth2/API key credential added
integration_unlinked	integration	Credential removed

Example: Flow update log entry:

{
  "timestamp": "2026-04-04T11:00:00Z",
  "company_id": "company_123",
  "user_id": "user_456",
  "user_email": "alice@company.com",
  "action_type": "flow_updated",
  "resource_type": "flow",
  "resource_id": "flow_789",
  "http_method": "PUT",
  "http_path": "/api/v1/flows/flow_789",
  "http_status": 200,
  "before_state": {
    "name": "ProcessInvoice",
    "status": "draft",
    "flowlet_count": 3
  },
  "after_state": {
    "name": "ProcessInvoice",
    "status": "published",
    "flowlet_count": 4,
    "updated_at": "2026-04-04T11:00:00Z"
  }
}

Execution Events

Event	Details
play_created	Flow execution instance started
play_started	Execution began
play_completed	Execution finished (DONE/FAIL/SKIP)
flowlet_executed	Individual step completed
flowlet_failed	Step error/timeout
flowlet_retried	Retry attempt after failure
flow_execution_timeout	Overall timeout exceeded
module_called	Module executed with input/output

Example: Play execution log entry:

{
  "timestamp": "2026-04-04T12:15:30Z",
  "company_id": "company_123",
  "user_id": "user_456",
  "action_type": "play_completed",
  "resource_type": "play",
  "resource_id": "play_999",
  "after_state": {
    "status": "DONE",
    "duration_ms": 3500,
    "flowlet_count": 4,
    "completed_at": "2026-04-04T12:15:30Z"
  },
  "tags": {
    "flow_id": "flow_789",
    "outcome": "success"
  }
}

System & Background Events

Event	Details
retention_cleanup_run	Data cleanup job executed
retention_entries_deleted	Old entries removed by retention policy
config_system_updated	System configuration changed
component_created	AI/storage/database component registered
component_updated	Component configuration modified
scheduler_job_started	Cron-scheduled flow started
scheduler_job_error	Scheduled flow failed

Field Sanitization

The audit logger automatically redacts sensitive information from all log entries. This prevents accidentally exposing secrets in logs.

Redaction Rules

Pattern	Redaction	Example
*password*	[REDACTED]	"password": "[REDACTED]"
*token*	[REDACTED]	"access_token": "[REDACTED]"
*key*	[REDACTED]	"api_key": "[REDACTED]"
*secret*	[REDACTED]	"client_secret": "[REDACTED]"
*credential*	[REDACTED]	"credentials": "[REDACTED]"
*oauth*	[REDACTED]	"oauth_token": "[REDACTED]"
email (PII)	Hashed	"email": "sha256:abc123..." (optional)
phone (PII)	Last 4 only	"phone": "***-***-5678" (optional)
SSN (PII)	Last 4 only	"ssn": "***-**-5678" (optional)

Configuration

config/config.yml

audit:
  field_sanitization:
    enabled: true
    redaction_mode: "redact"          # "redact" or "hash"
    redact_patterns:
      - "password"
      - "token"
      - "key"
      - "secret"
      - "credential"
      - "api_key"
      - "api_secret"
    pii_sanitization:
      enabled: true                   # Redact PII (email, phone, SSN)
      email_mode: "hash"              # "redact" or "hash"
      phone_mode: "partial"           # "partial" (last 4) or "redact"
      ssn_mode: "partial"
    max_field_bytes: 4096             # Truncate large fields

Example: Before & After Sanitization

Before (raw):

{
  "before_state": {
    "email": "alice@company.com",
    "api_key": "sk-1234567890abcdef",
    "phone": "555-123-4567",
    "integration_config": {
      "password": "super_secret_password_123"
    }
  }
}

After (sanitized):

{
  "before_state": {
    "email": "sha256:abc123def456...",
    "api_key": "[REDACTED]",
    "phone": "***-***-4567",
    "integration_config": {
      "password": "[REDACTED]"
    }
  }
}

Retention Policies

Audit logs are automatically cleaned up based on configurable retention policies to manage storage and comply with data minimization requirements.

Retention Configuration

config/config.yml

retention:
  cleanup_interval: "2m"              # Run cleanup every 2 minutes
  policies:
    audit_logs:
      cadence: "30d"                  # Keep 30 days
      min_entries: 10                 # Keep at least 10 entries
      enforced_minimum: "14d"         # Never delete logs < 14 days old
    flows_all:
      cadence: "90d"
      min_entries: 100
    flows_filtered:
      cadence: "30d"
      min_entries: 10
      enforced_minimum: "3d"          # GDPR: minimum 3 days for filtered flows
    plays:
      cadence: "7d"
      min_entries: 1000

Environment Overrides

RETENTION_CLEANUP_INTERVAL=5m
RETENTION_AUDIT_CADENCE=30d
RETENTION_FLOWS_FILTERED_CADENCE=7d
RETENTION_FLOWS_FILTERED_MIN_ENTRIES=50

Cleanup Job

The retention cleanup job runs every 2 minutes (configurable):

pkg/service/retention_policy_service.go

func (s *RetentionPolicyService) CleanupOldEntries(ctx context.Context, companyID string) error {
    policies := s.loadPolicies(companyID)

    for _, policy := range policies {
        cutoffTime := time.Now().Add(-policy.Cadence)

        // Count entries to delete
        count, _ := s.db.Collection(policy.Collection).CountDocuments(ctx, bson.M{
            "company_id": companyID,
            "created_at": bson.M{ "$lt": cutoffTime },
        })

        // Delete in batches of 500 to avoid locking
        if count > policy.MinEntries {
            s.db.Collection(policy.Collection).DeleteMany(ctx, bson.M{
                "company_id": companyID,
                "created_at": bson.M{ "$lt": cutoffTime },
            }, &options.DeleteOptions{})
        }
    }
}

Enforced Minimums

Retention policies have enforced minimums to ensure compliance and prevent accidental data loss:

Collection	Minimum Retention	Reason
audit_logs	14 days	Regulatory compliance, investigation window
flows (filtered)	3 days	GDPR right to deletion grace period
All collections	Min 10 entries	Prevent deleting entire datasets accidentally

Even if a policy specifies 1 day, audit logs will not be deleted until they are 14+ days old.

Querying Audit Logs

REST API

# Get all audit logs for current company (paginated)
GET /api/v1/audit?page=1&limit=50

# Filter by action type
GET /api/v1/audit?action=flow_created&page=1

# Filter by user
GET /api/v1/audit?user_id=user_456&page=1

# Filter by time range
GET /api/v1/audit?start_time=2026-04-01T00:00:00Z&end_time=2026-04-04T23:59:59Z

# Filter by resource
GET /api/v1/audit?resource_type=flow&resource_id=flow_789

# Combined filters
GET /api/v1/audit?action=flow_updated&user_id=user_456&start_time=2026-04-01T00:00:00Z&limit=100

Response:

{
  "logs": [
    {
      "id": "ObjectID",
      "timestamp": "2026-04-04T11:00:00Z",
      "action_type": "flow_updated",
      "user_email": "alice@company.com",
      "resource_type": "flow",
      "resource_id": "flow_789",
      "http_status": 200,
      "before_state": { ... },
      "after_state": { ... }
    }
  ],
  "total": 1234,
  "page": 1,
  "limit": 50
}

Direct MongoDB Query

// Find all flow creation events
db.audit_logs.find({
  company_id: ObjectId("company_123"),
  action_type: "flow_created"
}).sort({ timestamp: -1 }).limit(10)

// Find events by user in time range
db.audit_logs.find({
  company_id: ObjectId("company_123"),
  user_id: ObjectId("user_456"),
  timestamp: {
    $gte: ISODate("2026-04-01T00:00:00Z"),
    $lte: ISODate("2026-04-04T23:59:59Z")
  }
}).sort({ timestamp: -1 })

// Find failed login attempts (security)
db.audit_logs.find({
  action_type: "auth_login_failure",
  user_email: "alice@company.com"
}).sort({ timestamp: -1 }).limit(5)

Caching & Performance

Audit filters are cached in memory to avoid repeated database queries:

pkg/cache/audit_filter_cache.go

type AuditFilterCache struct {
    filters map[string][]string  // Cache of common filters
    ttl     time.Duration        // 5-minute cache
    mu      sync.RWMutex
}

// Background job refreshes cache every 3 minutes
func (c *AuditFilterCache) RefreshLoop(interval time.Duration) {
    ticker := time.NewTicker(interval)
    for range ticker.C {
        c.Refresh()  // Reload from MongoDB
    }
}

GDPR & Compliance

Data Subject Access Request (DSAR)

To export all audit logs and user data for a specific user:

POST /api/v1/privacy/export
{
  "user_id": "user_456",
  "data_types": ["audit_logs", "flows", "plays", "profile"]
}

Response:
{
  "export_id": "export_789",
  "status": "in_progress",
  "estimated_completion": "2026-04-04T14:00:00Z"
}

# Later, download ZIP
GET /api/v1/privacy/export/export_789/download

User Deletion

Audit logs are not deleted when a user is deleted (to maintain audit trail integrity):

DELETE /api/v1/users/user_456

# Deletes:
- User account
- Session records
- API keys

# Does NOT delete:
- Audit logs (user_id preserved for historical record)
- Flows authored by user (ownership transferred to admin)

Retention & Right to Deletion

Audit logs are retained for 14+ days regardless of retention policy to satisfy:

• Regulatory compliance (financial audits, incident investigations)
• Forensic analysis (security breach response)

However, personal data (email, phone) can be masked after retention period:

func (s *AuditService) MaskPersonalData(auditLog *DBAuditLog, olderThanDays int) {
    if time.Since(auditLog.Timestamp) > time.Duration(olderThanDays)*24*time.Hour {
        auditLog.UserEmail = maskEmail(auditLog.UserEmail)  // alice@company.com → a***@c***.com
        auditLog.UserIPAddress = maskIP(auditLog.UserIPAddress)  // 203.0.113.42 → 203.0.113.***
    }
}

Audit Log Export

Export to SIEM

flow8 can stream audit logs to external SIEM systems (Datadog, Splunk, ELK):

config/config.yml

audit_export:
  enabled: true
  backends:
    - type: "splunk"
      hec_url: "https://splunk.example.com:8088"
      hec_token: "[encrypted]"
      batch_size: 100
      flush_interval: "30s"
    - type: "datadog"
      api_key: "[encrypted]"
      site: "datadoghq.com"
      batch_size: 100

Syslog Export

audit_export:
  backends:
    - type: "syslog"
      address: "syslog.example.com:514"
      protocol: "tcp"
      format: "RFC5424"

Compliance Checklists

GDPR Compliance

• Audit logs retained only as long as necessary (14+ days minimum)
• PII redacted from logs after retention period
• Data subject access requests processed within 30 days
• User deletion removes personal data (but preserves audit trail)
• Consent logged for email/marketing communications
• Data processing agreement (DPA) signed with flow8

HIPAA Compliance

• All user actions logged (audit trail completeness)
• Audit logs encrypted at rest
• Access to audit logs restricted to authorized users
• Audit log integrity verified (no unauthorized changes)
• Audit logs retained for 6 years minimum
• Failed access attempts logged

SOC 2 Type II Compliance

• Audit logs cover 12+ months of testing period
• Audit logs demonstrate monitoring and alerting
• Access controls tested and verified
• Incident response procedures documented
• User authentication and authorization controls verified

Monitoring & Alerting

Key Metrics

flow8_audit_logs_total (counter)
  - company_id
  - action_type
  - http_status

flow8_audit_logs_latency_seconds (histogram)
  - query_type

flow8_auth_failures_total (counter)
  - failure_reason (wrong_password, account_locked, etc.)

flow8_retention_cleanup_duration_seconds (histogram)
  - collection

Alert Examples

# Alert on suspicious activity
- alert: HighFailedLoginRate
  expr: rate(flow8_auth_failures_total[5m]) > 0.5
  for: 5m
  annotations:
    summary: "High failed login rate detected"

# Alert on audit log age
- alert: AuditLogExceedsRetention
  expr: max(flow8_audit_log_age_days) > 30
  for: 1h
  annotations:
    summary: "Audit logs exceed 30-day retention"