Building from Source

Prerequisites

System Requirements

Component	Version	Purpose
Go	1.24.2+	Build and run source
Git	2.30+	Clone repository
Make	3.81+	Build automation (optional)
MongoDB	4.4+	Database

System Packages

flow8 requires document processing libraries for full functionality:

macOS (Homebrew):

brew install imagemagick ghostscript tesseract libreoffice

Ubuntu/Debian:

sudo apt-get update
sudo apt-get install -y \
  imagemagick \
  ghostscript \
  tesseract-ocr \
  libreoffice \
  liboffice-java-common \
  openssl \
  pkg-config

CentOS/RHEL:

sudo yum install -y \
  ImageMagick \
  ghostscript \
  tesseract \
  libreoffice \
  libreoffice-headless \
  openssl-devel \
  pkgconfig

macOS Silicon (M1/M2):

Ensure Tesseract is installed for ARM64:

# Using Rosetta for x86_64 dependencies
arch -x86_64 brew install tesseract

Go Setup

Install Go 1.24.2

macOS:

brew install go@1.24
brew link go@1.24
go version  # Should print go version go1.24.2

Linux (Ubuntu):

wget https://go.dev/dl/go1.24.2.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go1.24.2.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version

Docker (if using container):

FROM golang:1.24.2-alpine
RUN apk add --no-cache git make
WORKDIR /app

Go Environment Variables

# Workspace mode (Go 1.18+)
export GOWORK=off  # If using workspaces

# For private Go modules
export GOPRIVATE=github.com/osbits/*
export GONOSUMDB=github.com/osbits/*

# Build optimization
export CGO_ENABLED=1  # Required for SQLite/crypto
export GOOS=linux     # Target OS
export GOARCH=amd64   # Target architecture (or arm64 for Apple Silicon)

Private Module Access

flow8 uses private Go modules from github.com/osbits/. You must configure Git authentication:

Using SSH Keys

# Generate SSH key (if needed)
ssh-keygen -t ed25519 -f ~/.ssh/github_flow8 -C "your_email@example.com"

# Add to SSH agent
ssh-add ~/.ssh/github_flow8

# Configure Git
git config --global user.email "you@example.com"
git config --global user.name "Your Name"

# Test SSH access
ssh -T git@github.com

Using Personal Access Tokens

# Create PAT in GitHub settings (Settings → Developer settings → Personal access tokens)
# Scopes: repo, read:packages

# Configure Git credentials
git config --global credential.helper store
git config --global credential.useHttpPath true

# For go get, configure .netrc
cat >> ~/.netrc << EOF
machine github.com
login your-github-username
password ghp_your_personal_access_token
EOF
chmod 600 ~/.netrc

SSH Config

~/.ssh/config

Host github.com
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_flow8
    AddKeysToAgent yes
    IdentitiesOnly yes

Building from Source

Clone Repository

git clone git@github.com:osbits/flow8.git
cd flow8
git checkout develop  # Main development branch

Download Dependencies

go mod download
go mod verify  # Ensure checksums match

Run Server

Development (with hot reload):

# Simple run
go run server.go

# With Delve debugger
dlv debug -l localhost:2345 -- server.go

# With live reload (requires entr or similar)
find . -name '*.go' | entr go run server.go

Check health:

curl http://localhost:4454/health

Build Binary

Development build:

go build -o flow8 server.go

# Run binary
./flow8

Production build (optimized):

go build \
  -ldflags="-s -w -X main.Version=1.0.0 -X main.BuildTime=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" \
  -tags=netgo \
  -o flow8-release \
  server.go

# Strip binary
strip flow8-release

# Check size
ls -lh flow8-release

Cross-compile for Linux (from macOS):

GOOS=linux GOARCH=amd64 go build -o flow8-linux server.go

Configuration

Environment Setup

# Copy environment template
cp .env.example .env

# Edit with your values
export MONGODB_URI=mongodb://localhost:27017/flow8
export ENCRYPTION_KEY=$(openssl rand -hex 128)
export ENCRYPTION_KEY_SALT=$(openssl rand -hex 32)
export SERVER_PORT=4454

# Load into shell
set -a
source .env
set +a

Configuration File

# Create config directory
mkdir -p config

# Create config.yml
cat > config/config.yml << 'EOF'
server:
  port: 4454
  max_request_size_mb: 100

mongodb:
  uri: ${MONGODB_URI}
  database: flow8
  max_pool_size: 100

encryption:
  key_algorithm: argon2id

session:
  ttl_hours: 1
  cookie_secure: false  # OK for development
  cookie_http_only: true
EOF

Testing

Run All Tests

go test ./...

# With verbose output
go test -v ./...

# With coverage
go test -cover ./...

# Generate coverage HTML
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html

Run Specific Package Tests

# Test service package
go test ./pkg/service/...

# Test with regex filter
go test -run TestFlow ./...

# Test with benchmark
go test -bench=. -benchmem ./pkg/plugins/layers/v2/...

Run Single Test

go test -run TestIteratorModule -v ./pkg/plugins/layers/v2/

# With timeout
go test -timeout 30s -run TestLayerService ./pkg/service/...

Test with Integration Test Fixtures

# Tests use recorded fixtures (no external API calls)
go test -v ./pkg/service/flow_test_service.go

# Regenerate fixtures (requires credentials)
UPDATE_FIXTURES=true go test ./pkg/service/...

Development Workflow

Project Structure

flow8/
├── server.go              # Entry point
├── go.mod                 # Module definition
├── go.sum                 # Dependency checksums
├── .env.example           # Environment template
├── config/
│   └── config.yml         # YAML configuration
├── pkg/
│   ├── auth/              # Authentication
│   ├── cache/             # In-memory caching
│   ├── service/           # Business logic
│   ├── http/              # HTTP controllers
│   ├── model/             # MongoDB models
│   ├── plugin/            # Plugin system
│   ├── plugins/layers/v2/ # 135+ modules
│   └── ... (other packages)
├── storage/               # Runtime data (gitignored)
│   ├── db/
│   ├── data/
│   ├── logs/
│   └── tmp/
└── docs/                  # API documentation

Adding a New Module

Create file pkg/plugins/layers/v2/module_mymodule.go:

package v2

import (
    "github.com/osbits/gorgany"
)

type MyModule struct {
    *PluginModule
}

func NewModuleMymodule(pm *PluginModule) *MyModule {
    return &MyModule{PluginModule: pm}
}

func (m *MyModule) Execute(params Params) Response {
    // Implementation
    return Response{
        State: "DONE",
        Output: map[string]interface{}{
            "result": "success",
        },
    }
}

Register automatically via reflection discovery.

Code Style

Follow Go conventions:

# Format code
go fmt ./...

# Lint with golangci-lint
golangci-lint run ./...

# Vet for common mistakes
go vet ./...

# Run tests before commit
go test ./...

Install golangci-lint:

curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s -- -b $(go env GOPATH)/bin

Git Workflow

# Create feature branch
git checkout -b feature/my-feature

# Make changes
# ... edit code, run tests ...
go test ./...

# Commit
git add .
git commit -m "feat: describe the feature"

# Push
git push origin feature/my-feature

# Create pull request on GitHub

Module Catalog Generation

Generate CSV catalog of all 135+ modules:

go run cmd/modulescsv/main.go > modules.csv

# Output CSV with columns:
# - Name
# - Description
# - Category
# - Inputs
# - Outputs
# - Required

Dependency Management

Update Dependencies

# Check for available updates
go list -u -m all

# Update specific module
go get -u github.com/osbits/gorgany@latest

# Update all indirect dependencies
go get -u ./...

# Tidy unused dependencies
go mod tidy

# Verify integrity
go mod verify

Private Module Issues

If you get “no go-import meta tag” error:

go get github.com/osbits/gorgany
go: github.com/osbits/gorgany@v0.0.0: reading github.com/osbits/gorgany/go.mod at master:
    no go-import meta tags (404)

Solution:

# Ensure SSH key is added to SSH agent
ssh-add ~/.ssh/github_flow8

# Test SSH access
ssh -T git@github.com

# Re-run go get
go get github.com/osbits/gorgany

Debugging

Delve Debugger

# Install
go install github.com/go-delve/delve/cmd/dlv@latest

# Debug from source
dlv debug -l localhost:2345 -- server.go

# Connect from IDE (VS Code, GoLand, etc.)
# Port: 2345

IDE Setup

VS Code (.vscode/launch.json):

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Connect to Delve",
            "type": "go",
            "mode": "remote",
            "remotePath": "/app",
            "port": 2345,
            "host": "127.0.0.1",
            "showLog": true
        }
    ]
}

GoLand: Run → Edit Configurations → Go Remote

Logging & Debugging

Enable debug logging:

LOG_LEVEL=debug go run server.go

# Output includes:
# - HTTP request/response details
# - MongoDB query traces
# - Module execution steps

Production Build

Build Checklist

# Run all tests
go test ./...

# Check for vet issues
go vet ./...

# Run linter
golangci-lint run ./pkg/...

# Check test coverage
go test -cover ./... | grep coverage

# Update version
export VERSION=1.0.0

# Build binary
go build \
  -ldflags="-s -w -X main.Version=$VERSION" \
  -o flow8-$VERSION \
  server.go

# Create checksums
sha256sum flow8-$VERSION > flow8-$VERSION.sha256

# Sign binary (optional)
gpg --detach-sign flow8-$VERSION

Docker Build

# Build image
docker build -t flow8core:latest .

# Test image
docker run --rm -p 4454:4454 flow8core:latest

# Tag for registry
docker tag flow8core:latest ghcr.io/osbits/flow8core:latest

# Push to registry
docker push ghcr.io/osbits/flow8core:latest

Troubleshooting

Module Not Found

no required module providing package github.com/osbits/gorgany

Solution:

# Update go.sum
go mod tidy
go mod download

CGO Compilation Error

gcc: command not found

Solution:

# Disable CGO (not recommended, may break SQLite)
CGO_ENABLED=0 go build server.go

# Or install build tools
# Ubuntu: sudo apt-get install build-essential
# macOS: xcode-select --install

MongoDB Connection Refused

connect: connection refused

Solution:

# Start MongoDB
mongod --dbpath /data/db &

# Or use Docker
docker run -d -p 27017:27017 mongo:6.0

Out of Memory During Build

# Limit parallel builds
go build -p 1 server.go

# Or increase swap