Environment Separation Plan: Development vs Production

Status: Implemented and operational across all Frao stacks
Approach: Dual VPN isolation (frao.dev / frao.prod) with same ports across both stacks
Scope: All Frao Technologies services

Executive Summary

The Problem

Frao Technologies operates a complex microservice architecture. Initially, all services ran on a single Docker Compose stack. When an AI agent or developer ran docker compose up -d --build <service>, it rebuilt and restarted the only instance of that service — often the production one. There was no guard preventing accidental production disruption.

The Solution

We introduced two fully independent stacks that coexist on the same host, sharing nothing but the Docker daemon and the source tree:

Key design properties:

• Zero container name collisions — different COMPOSE_PROJECT_NAME
• Zero port collisions — same ports on different VPN interfaces
• Zero data collisions — dev uses separate volumes, databases, Redis instances
• AI agents default to dev — only explicit commands touch production

The Sacred Commandment

A core principle emerged: Dev-First, Prod-by-Intent.

THOU SHALT NOT rebuild, restart, or deploy to the production stack unless the user explicitly says "production", "prod", "deploy", or otherwise names the production stack.

The default target is ALWAYS the development stack.
Production is NEVER the default.

∀ rebuild_request ∈ {agent, user}:
  if ¬explicit_prod_intent(rebuild_request) → target = development

∀ deploy_action ∈ {agent}:
  confirm(user, "This affects PRODUCTION. Continue?") → proceed ∨ abort

Architecture

┌─────────────────────────────────────────────────────┐
│                    Docker Host                        │
│                                                       │
│  ┌────────────────────────────┐  ┌────────────────────────────┐
│  │   PRODUCTION (frao.prod)    │  │   DEVELOPMENT (frao.dev)    │
│  │   hedgefund_*               │  │   hedgefund_dev_*           │
│  │   Binds to: 10.128.0.5      │  │   Binds to: 10.64.0.5      │
│  │                              │  │                              │
│  │  db:5432                     │  │  db:5432                     │
│  │  redis:6379                  │  │  redis:6379                  │
│  │  backend:8080                │  │  backend:8080                │
│  │  frontend:3010               │  │  frontend:3010               │
│  │  company-portal:80           │  │  company-portal:80           │
│  │  ...                         │  │  ...                         │
│  └────────────────────────────┘  └────────────────────────────┘  │
│                                                       │
│  Both stacks share: Docker daemon, source code        │
│  Both stacks have: separate volumes, networks, ports  │
└─────────────────────────────────────────────────────┘

How Stack Selection Works

The stack is selected by which --env-file and which override compose file is passed:

# Production (explicit — requires human confirmation)
docker compose -f docker-compose.yml -f docker-compose.prod.yml --env-file .env up -d

# Development (default for AI agents)
docker compose -f docker-compose.yml -f docker-compose.dev.yml --env-file .env.dev up -d

A Makefile wraps these into simple commands — making the intent crystal clear.

Implementation Phases

The separation was implemented in 10 phases over several weeks:

1. .env.dev File — copied from .env, overridden with dev-specific ports and secrets
2. docker-compose.dev.yml — dev overrides for ports, env vars, hot-reload volumes
3. docker-compose.prod.yml — thin production overlay with explicit bindings
4. Base compose cleanup — removed environment-specific overrides from shared config
5. Makefile creation — make up-dev, make rebuild-dev <svc>, and all other commands
6. Build script updates — separate image tags for dev vs prod
7. Deploy script updates — explicit --env flag for deployment target
8. VPS setup/update scripts — always target prod stack explicitly
9. Skill & agent instruction updates — all AI agents trained to default to dev
10. Testing — verified isolation, port separation, data separation

Port Allocation

Ports are identical across both stacks. Isolation comes from WireGuard VPN interfaces binding to different IPs:

Since 10.64.0.5 and 10.128.0.5 are different network interfaces, they can use the same port numbers without conflict. This eliminates the +1000 offset scheme used previously.

Lessons Learned: Hardcoded IPs

During implementation, we discovered 33 hardcoded networking references across 4 repositories. These fell into categories:

Application Source Code

Services that hardcoded 10.64.0.5 as their bind address — which fails inside containers where that IP is on the host, not in the container’s network namespace.

Fix: Every service now reads its bind address from an environment variable with fallback to 0.0.0.0.

Frontend URL Construction

SvelteKit components constructing URLs with hardcoded WireGuard IPs instead of relative paths — breaking on the other stack.

Fix: All frontend code uses relative paths or window.location.host. Never construct URLs with hardcoded IPs.

Docker Compose Bindings

Port mappings hardcoded to 10.64.0.5 instead of using ${BIND_IP} variable syntax.

Fix: All compose files use ${BIND_IP:-<default>} set per-stack in .env / .env.dev.

AI Agent Integration

The most critical change was training all AI agents to default to the development stack. Every agent that works on Frao services now follows this flow:

1. Agent receives a change request
2. Agent determines: user didn't say "production" → targets DEVELOPMENT
3. Agent edits code on the `dev` branch
4. Agent runs: make rebuild-dev <service>
5. ✅ Development service rebuilds. Production untouched.

The Makefile provides unambiguous command pairs:

• make up / make up-dev — start stacks
• make rebuild <svc> / make rebuild-dev <svc> — rebuild services
• make logs <svc> / make logs-dev <svc> — view logs
• make down / make down-dev — stop stacks

Verification & Testing

Our isolation verification suite ensures both stacks work correctly:

Conclusion

The dual-stack architecture has been running successfully since implementation. Key takeaways:

1. Simple design works best — same ports on different VPN interfaces
2. Makefiles are the interface — agents never call docker compose directly
3. Training agents is as important as configuring infrastructure — explicit guardrails in skill definitions
4. Discover hardcoded IPs aggressively — rg -n '10\.(64|128)\.0\.\d+' is now standard procedure for new services

This pattern has become the canonical approach for all new Frao Technologies services: separate env files, separate compose overrides, Makefile wrappers, and AI agent training to default to development.

Originally documented in Frao Doc: "Environment Separation Plan: Development vs Production" (shortid: 040718ed)