gomesh

GoMesh - Service Mesh Implementation

A high-performance, distributed Service Mesh with a gRPC Control Plane built from scratch in Go.

Current Status: Phase 3 Part 4 - Proxy Config Streaming ✅

What Was Done:

• ✅ Phase 1: Basic reverse proxy with graceful shutdown
• ✅ Phase 2 Part 1: Structured JSON logging with Zap and logging middleware
• ✅ Phase 2 Part 2: Prometheus metrics with /metrics endpoint
• ✅ Phase 2 Part 3: Recovery middleware and middleware chaining
• ✅ Phase 2 Part 4: Distributed tracing with unique trace IDs
• ✅ Phase 3 Part 1: Protocol Buffer API definition and code generation
• ✅ Phase 3 Part 2: Control Plane Server with gRPC
• ✅ Phase 3 Part 3: Mandatory proxy registration with the control plane
• ✅ Phase 3 Part 4: Config streaming and in-memory storage in the proxy

Next Step:

• 📍 Phase 3 Part 5: Apply streamed config to proxy routing

Project Structure

GoMesh/
├── cmd/
│   ├── proxy/              # Data plane proxy binary
│   │   └── main.go
│   ├── controller/         # Control plane binary (Phase 3 Part 2)
│   │   └── main.go
│   └── backend/            # Test backend service
│       └── main.go
├── pkg/
│   ├── logging/            # Structured logging (Phase 2 Part 1)
│   │   └── logging.go      # Zap logger wrapper
│   ├── tracing/            # Distributed tracing (Phase 2 Part 4)
│   │   └── tracer.go       # Trace ID generation and propagation
│   ├── controlplane/       # Control plane logic (Phase 3 Part 2)
│   │   ├── server.go       # gRPC server implementation
│   │   └── config.go       # Configuration store with versioning
│   └── proxy/              # Proxy package
│       ├── client.go       # Control plane gRPC client
│       ├── config.go       # Configuration loader
│       ├── handler.go      # Reverse proxy logic
│       ├── middleware.go   # All middleware (logging, metrics, tracing, recovery)
│       ├── metrics.go      # Prometheus metrics (Phase 2 Part 2)
│       └── server.go       # HTTP server
├── api/
│   └── proto/              # gRPC API definitions (Phase 3 Part 1)
│       ├── mesh.proto      # Protocol Buffer definitions
│       ├── mesh.pb.go      # Generated: message types
│       └── mesh_grpc.pb.go # Generated: gRPC service interfaces
├── scripts/
│   └── generate-proto.sh   # Script to generate Go code from .proto files
├── config/
│   └── proxy.yaml          # Proxy configuration
├── go.mod
└── go.sum

How to Run

Step 1: Install Dependencies

cd gomesh
go mod download

Step 2: Start the Backend Service

In one terminal:

go run cmd/backend/main.go

You should see:

[BACKEND] Starting test backend on :3000
[BACKEND] Ready to receive requests from the proxy

Step 3: Start the Control Plane (Required for Phase 3)

In another terminal:

go run cmd/controller/main.go

You should see:

INFO    Control Plane starting...       {"port": 9090, "production": false}
INFO    gRPC server registered
INFO    control plane listening at      {"address": "[::]:9090"}

Flags:

• -port: Control plane port (default: 9090)
• -production: Use production logging (JSON) instead of development

Step 4: Start the Proxy

In another terminal:

go run cmd/proxy/main.go

You should see logs similar to:

INFO    Loading configuration file from path       {"path": "config/proxy.yaml"}
INFO    connecting to control plane                {"proxy_id": "proxy-1", "control_plane_addr": "localhost:9090"}
INFO    control plane connection established       {"proxy_id": "proxy-1", "control_plane_addr": "localhost:9090"}
INFO    registering proxy with control plane       {"proxy_id": "proxy-1", "version": "1.0.0", "listen_addr": "localhost:8000"}
INFO    proxy registered with control plane        {"proxy_id": "proxy-1", "message": "Proxy proxy-1 registered successfully!"}
INFO    subscribing to control plane config stream {"proxy_id": "proxy-1"}
INFO    control plane config stream established    {"proxy_id": "proxy-1"}
INFO    received config update from control plane  {"proxy_id": "proxy-1", "version": 1, "num_routes": 1}
INFO    proxy server starting                      {"port": 8000, "backend_url": "http://localhost:3000"}
INFO    metrics endpoint registered at /metrics    {"url": "http://localhost:8000/metrics"}

Step 5: Test It!

In a fourth terminal:

# Send a request to the proxy
curl http://localhost:8000/api/users

# You should see:
# {
#   "message": "Hello from the backend service!",
#   "timestamp": "2026-01-24T...",
#   "path": "/api/users",
#   "method": "GET"
# }

Step 6: Check Metrics!

# View Prometheus metrics
curl http://localhost:8000/metrics

# You'll see metrics like:
# gomesh_requests_total{service="backend",status="2xx"} 1
# gomesh_request_duration_seconds_bucket{service="backend",le="0.05"} 1
# gomesh_requests_in_flight 0

Step 7: Test Recovery Middleware!

# Trigger a panic in the backend (proxy should handle it gracefully)
curl http://localhost:8000/panic

# You'll see:
# - Client receives: "Internal Server Error" (HTTP 500)
# - Proxy logs the panic with full stack trace
# - Proxy KEEPS RUNNING (doesn't crash!) ✅

Step 8: Test Distributed Tracing!

# Make a request and check the trace ID in the response headers
curl -v http://localhost:8000/api/users

# You'll see in the response headers:
# X-Trace-ID: a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

# The response body also includes the trace ID:
# {
#   "message": "Hello from the backend service!",
#   "trace_id": "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6",
#   ...
# }

# You can also provide your own trace ID:
curl -H "X-Trace-ID: my-custom-trace-123" http://localhost:8000/api/users

# The backend will echo it back in logs and response

Step 9: Test the Control Plane! (Phase 3)

The control plane is now running and ready to manage proxy configurations:

Check Control Plane Logs:

When you start the control plane, you should see:

INFO    Control Plane starting...       {"port": 9090, "production": false}
INFO    gRPC server registered
INFO    control plane listening at      {"address": "[::]:9090"}

What Was Done:

The control plane is:

• ✅ Listening on port 9090 for gRPC connections
• ✅ Ready to accept proxy registrations
• ✅ Ready to stream configuration updates
• ✅ Storing a default route configuration (path: "/", backend: "localhost:3000")

Current Features:

1. Proxy Registration: Tracks connected proxies with unique IDs
2. Config Streaming: Sends initial config (version 1) to newly connected proxies
3. Versioned Config: Auto-increments version on each update
4. Thread-Safe: Handles concurrent proxy connections safely
5. Graceful Shutdown: Cleanly closes all proxy connections on SIGINT/SIGTERM

Next Step:

With proxy config streaming implemented, the next work is to:

• Apply streamed config to request routing
• Dynamically add/update/remove routes
• See real-time config propagation across all proxies

Working with Protocol Buffers (gRPC)

Prerequisites

Install the Protocol Buffer compiler and Go plugins:

# Install protoc compiler
# macOS:
brew install protobuf

# Linux:
# apt install -y protobuf-compiler

# Install Go plugins for protoc
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Verify installation
protoc --version  # Should show libprotoc 3.x or higher

Generating Go Code from .proto Files

Whenever you modify api/proto/mesh.proto, regenerate the Go code:

# Run the generation script
bash scripts/generate-proto.sh

# This generates:
# - api/proto/mesh.pb.go        (message types: ProxyInfo, ConfigUpdate, etc.)
# - api/proto/mesh_grpc.pb.go   (gRPC service interfaces: MeshControlServer, MeshControlClient)

Important: Never manually edit the generated .pb.go files. Always edit the .proto file and regenerate.

What's Happening?

Your Request
    ↓
    ↓ (HTTP GET /api/users)
    ↓
┌─────────────────┐
│  GoMesh Proxy   │  ← Listening on :8000
│  (localhost)    │
└─────────────────┘
    ↓
    ↓ (Forwards to backend)
    ↓
┌─────────────────┐
│  Backend App    │  ← Listening on :3000
│  (localhost)    │
└─────────────────┘
    ↓
    ↓ (Returns JSON)
    ↓
Your Response

Watch the Logs

In the proxy terminal, you'll see structured JSON logs:

{"level":"info","timestamp":"2026-01-25T10:30:05.200Z","msg":"Request starter","trace_id":"none","method":"GET","path":"/api/users","remote_addr":"127.0.0.1:53242"}
{"level":"info","timestamp":"2026-01-25T10:30:05.245Z","msg":"request completed","method":"GET","path":"/api/users","status":200,"latency_ms":"45ms","trace_id":"none"}

In the backend terminal, you'll see:

[BACKEND] Received: GET /api/users

What the Logs Tell You:

• trace_id: Unique 128-bit identifier for tracking requests across services
• method: HTTP method (GET, POST, etc.)
• path: Request path
• status: HTTP status code (200, 404, 500, etc.)
• latency_ms: Time taken to process the request
• timestamp: ISO8601 formatted timestamp
• remote_addr: Client IP address

Configuration

Edit config/proxy.yaml to change:

• Proxy listen port (default: 8000)
• Backend host/port (default: localhost:3000)
• Timeouts

What We've Learned

Phase 1: Basic Reverse Proxy ✅

• ✅ Go HTTP Server - Using net/http
• ✅ Reverse Proxy - Using httputil.ReverseProxy
• ✅ Configuration - YAML parsing with gopkg.in/yaml.v3
• ✅ Graceful Shutdown - Signal handling with os/signal
• ✅ Project Structure - Standard Go project layout
• ✅ Goroutines - Running server in background
• ✅ Channels - Communication between goroutines

Phase 2 Part 1: Structured Logging ✅

• ✅ Zap Logger - High-performance structured logging with go.uber.org/zap
• ✅ Middleware Pattern - Function wrapping: func(http.Handler) http.Handler
• ✅ Interface Wrapping - Custom responseWriter wraps http.ResponseWriter
• ✅ Request/Response Tracking - Capturing status codes and latency
• ✅ Defer Pattern - defer logger.Sync() ensures cleanup
• ✅ Structured Fields - JSON logs with typed fields (method, path, status, latency)

Phase 2 Part 2: Prometheus Metrics ✅

• ✅ Prometheus Client - Using github.com/prometheus/client_golang
• ✅ Metric Types - Counters (requests_total), Histograms (duration), Gauges (in_flight)
• ✅ Metric Labels - Multi-dimensional metrics (service, status, error_type)
• ✅ promauto Package - Automatic registration with default registry
• ✅ HTTP Multiplexer - Using http.NewServeMux() for multiple endpoints
• ✅ Middleware Chaining - Metrics → Logging → Proxy handler stack
• ✅ /metrics Endpoint - Standard Prometheus scraping endpoint

Phase 2 Part 3: Advanced Middleware ✅

• ✅ Panic Recovery - Using defer + recover() to catch runtime panics
• ✅ Stack Traces - runtime/debug.Stack() for debugging panics
• ✅ Resilient Design - Isolated request failures don't crash the entire proxy
• ✅ Middleware Chaining - Variadic function pattern for composable middleware
• ✅ Order-Aware Composition - Reverse loop to apply middleware in correct order
• ✅ Graceful Degradation - Return 500 on panic, log details, keep serving

Phase 2 Part 4: Distributed Tracing ✅

• ✅ Trace ID Generation - Using crypto/rand for cryptographically secure 128-bit IDs
• ✅ Header Propagation - X-Trace-ID header forwarded to backend services
• ✅ Request Correlation - Track requests across multiple services with unique IDs
• ✅ Response Headers - Return trace ID to clients for debugging
• ✅ Logging Integration - All logs include trace_id field for request correlation
• ✅ Fallback Mechanism - Timestamp-based fallback if crypto/rand fails

Phase 3 Part 1: gRPC API Definition ✅

• ✅ Protocol Buffers - Defined API contract in mesh.proto using proto3 syntax
• ✅ Service Definition - MeshControl service with RegisterProxy and StreamConfig RPCs
• ✅ Message Types - ProxyInfo, ConfigUpdate, Route, RegistrationResponse messages
• ✅ Code Generation - Automated Go code generation with protoc compiler
• ✅ Unary RPC - RegisterProxy for simple request-response pattern
• ✅ Server Streaming - StreamConfig for long-lived config update streams
• ✅ Build Script - generate-proto.sh for reproducible code generation

Phase 3 Part 2: Control Plane Server ✅

• ✅ gRPC Server - Implemented MeshControlServer interface with gRPC
• ✅ RegisterProxy RPC - Tracks connected proxies in memory with unique proxy IDs
• ✅ StreamConfig RPC - Long-lived server streaming for pushing config updates to proxies
• ✅ ConfigStore - Versioned configuration storage with thread-safe access
  • Default route to localhost:3000 backend
  • UpdateConfig() to replace entire config
  • AddRoute() to append new routes
  • Auto-incrementing version numbers on updates
• ✅ RWMutex - Reader-writer locks for concurrent access patterns
  • Read locks for GetConfig() and listing proxies
  • Write locks for updates and proxy registration
• ✅ BroadcastConfigUpdate - Push updates to all connected proxies simultaneously
• ✅ Graceful Shutdown - Signal handling (SIGINT, SIGTERM) for clean termination
• ✅ Connection Management - Automatic cleanup with defer when proxy disconnects
• ✅ Structured Logging - Zap logger with development/production modes
  • Development: Human-readable console output
  • Production: Structured JSON logs

Implementation Details:

The control plane runs as a standalone binary (cmd/controller/main.go) that:

1. Listens on port 9090 (configurable via -port flag)
2. Accepts proxy registrations and stores proxy info (ID, version, listen address)
3. Opens long-lived gRPC streams to send config updates
4. Maintains a map[string]*ProxyConnection to track all connected proxies
5. Sends initial config immediately when a proxy connects via StreamConfig()
6. Keeps connections alive and cleans up when proxies disconnect

The configuration store (pkg/controlplane/config.go) manages:

• Routing rules with path, backend, auth requirements, and timeouts
• Version tracking (incremented on every update)
• Thread-safe concurrent access with sync.RWMutex

The gRPC server (pkg/controlplane/server.go) implements:

• RegisterProxy(ProxyInfo) → RegistrationResponse - Unary RPC for registration
• StreamConfig(ProxyInfo) → stream ConfigUpdate - Server streaming for config push
• BroadcastConfigUpdate(ConfigUpdate) - Internal method to push updates to all proxies
• GetConnectedProxies() → []*ProxyInfo - Helper to list all connected proxies

Phase 3 Part 3: Proxy Registration ✅

• ✅ Bootstrap Config - Added proxy identity and control-plane bootstrap settings to config/proxy.yaml
• ✅ Validation - Startup now validates proxy identity, advertise address, and control-plane address
• ✅ gRPC Client - Added a dedicated control-plane client in pkg/proxy/client.go
• ✅ Mandatory Connection - Proxy startup waits for the control plane connection to become ready
• ✅ RegisterProxy RPC - Proxy registers itself before the HTTP server starts
• ✅ Fail-Fast Startup - Proxy exits if the control plane is unavailable or registration fails

Phase 3 Part 4: Proxy Config Streaming ✅

• ✅ StreamConfig RPC - Proxy opens the server-streaming RPC after registration
• ✅ Initial Config Gate - Proxy waits for the first ConfigUpdate before starting the HTTP server
• ✅ Config Storage - Latest control-plane config is stored safely in memory
• ✅ Runtime Monitoring - Proxy stops if the control-plane stream fails after startup
• ✅ Read API - GetConfig() returns a defensive copy for later route application work

Next Steps: Phase 3 - gRPC Control Plane (Continued)

Phase 3 Part 5: Dynamic Route Application 📍 NEXT

• Use streamed config inside the proxy handler
• Route requests from control-plane state instead of static backend YAML
• Apply route changes without restarting the proxy

Troubleshooting

Port already in use?

# Find what's using port 8000
lsof -i :8000

# Or change the port in config/proxy.yaml

Can't connect to backend?

# Make sure backend is running
curl http://localhost:3000/health

Commands Cheat Sheet

# Run proxy
go run cmd/proxy/main.go

# Run proxy with custom config
go run cmd/proxy/main.go -config /path/to/config.yaml

# Build the binary
go build -o bin/proxy cmd/proxy/main.go

# Run the binary
./bin/proxy

# Test the proxy
curl -v http://localhost:8000/test
curl -X POST http://localhost:8000/api/data -d '{"key":"value"}'

---

How the Middleware Stack Works

Understanding the request flow through our middleware chain:

Request from Client (with optional X-Trace-ID header)
        ↓
┌───────────────────────────┐
│  RecoveryMiddleware       │  ← OUTERMOST: Catches ALL panics (defer/recover)
│  (panic safety)           │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  TracingMiddleware        │  ← Generates/extracts trace ID
│  (distributed tracing)    │     Sets X-Trace-ID header in request & response
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  MetricsMiddleware        │  ← Increments in-flight counter, starts timer
│  (observability)          │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  LoggingMiddleware        │  ← Logs "request started" with trace_id
│  (structured logging)     │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  ProxyHandler             │  ← Forwards to backend with X-Trace-ID header
│  (httputil.ReverseProxy)  │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  Backend Service :3000    │  ← Receives trace ID, echoes in logs & response
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  LoggingMiddleware        │  ← Logs "request completed" with trace_id
│  (calculates latency)     │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  MetricsMiddleware        │  ← Records metrics (counter, histogram, gauge)
│  (records metrics)        │     Decrements in-flight counter
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  TracingMiddleware        │  ← X-Trace-ID already set in response header
│  (trace ID in response)   │
└───────────────────────────┘
        ↓
┌───────────────────────────┐
│  RecoveryMiddleware       │  ← If panic occurred, catches it here
│  (returns 500 if panic)   │     Logs with stack trace & trace_id, returns 500
└───────────────────────────┘
        ↓
Response to Client (with X-Trace-ID header)

Middleware Order Matters!

The order of middleware is critical:

1. Recovery (outermost) - Must catch panics from ALL inner middleware
2. Tracing - Generate/extract trace ID early so all inner middleware can use it
3. Metrics - Track all requests (even if they panic)
4. Logging - Log all requests with trace ID (even if they panic)
5. Proxy Handler - The actual reverse proxy logic

The Chain Function

Using the Chain helper makes middleware composition clean and readable:

// Before: Manual nesting (hard to read)
handler = RecoveryMiddleware(logger,
    TracingMiddleware(
        MetricsMiddleware(metrics,
            LoggingMiddleware(logger, handler))))

// After: Chain function (clean & clear)
handler = Chain(handler,
    RecoveryMiddleware(logger),    // First = outermost
    TracingMiddleware(),           // Second
    MetricsMiddleware(metrics),    // Third
    LoggingMiddleware(logger),     // Last = innermost
)

How Chain works:

func Chain(handler http.Handler, middlewares ...func(http.Handler) http.Handler) http.Handler {
    // Apply in reverse so first middleware becomes outermost
    for i := len(middlewares) - 1; i >= 0; i-- {
        handler = middlewares[i](handler)
    }
    return handler
}

This produces: Recovery(Tracing(Metrics(Logging(handler))))

Before vs After Logging

Phase 1 (Basic Logging):

[INFO] Forwarding: GET /api/users → localhost:3000

Phase 2 Part 1-4 (Structured Logging + Tracing):

{"level":"info","timestamp":"2026-01-26T10:30:05.200Z","msg":"Request starter","trace_id":"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6","method":"GET","path":"/api/users","remote_addr":"127.0.0.1:53242"}
{"level":"info","timestamp":"2026-01-26T10:30:05.245Z","msg":"request completed","method":"GET","path":"/api/users","status":200,"latency_ms":"45ms","trace_id":"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6"}

The structured logs are:

• Machine-readable - Easy to parse and analyze
• Searchable - Query by any field (status=500, latency>100ms, etc.)
• Standardized - Consistent format across all services

Prometheus Metrics

After sending some requests, check the /metrics endpoint:

curl http://localhost:8000/metrics

Key Metrics Available:

# Total number of requests (labeled by service and status bucket)
gomesh_requests_total{service="backend",status="2xx"} 5
gomesh_requests_total{service="backend",status="4xx"} 1
gomesh_requests_total{service="backend",status="5xx"} 0

# Request duration histogram (shows latency distribution)
gomesh_request_duration_seconds_bucket{service="backend",le="0.005"} 3
gomesh_request_duration_seconds_bucket{service="backend",le="0.01"} 5
gomesh_request_duration_seconds_bucket{service="backend",le="0.05"} 5
gomesh_request_duration_seconds_sum{service="backend"} 0.023
gomesh_request_duration_seconds_count{service="backend"} 5

# Number of requests currently being processed
gomesh_requests_in_flight 0

# Total errors (labeled by service and error type)
gomesh_errors_total{service="backend",error_type="timeout"} 0

What These Metrics Tell You:

• requests_total: Count requests by status code (2xx, 3xx, 4xx, 5xx)
• request_duration_seconds: Latency percentiles (p50, p95, p99) for SLA tracking
• requests_in_flight: Current load on the proxy
• errors_total: Error counts by type for alerting

Testing Recovery Middleware

The backend has a /panic endpoint that intentionally triggers a panic to test recovery:

# Trigger a panic
curl http://localhost:8000/panic

What happens:

1. Client receives:

   Internal Server Error
   

   HTTP status: 500

2. Proxy logs show:

   {
     "level": "error",
     "msg": "panic recovered",
     "error": "intentional panic for testing recovery middleware!",
     "path": "/panic",
     "method": "GET",
     "stack": "goroutine 123 [running]:\nruntime/debug.Stack()..."
   }
   

3. Proxy keeps running! ✅

   • Other requests still work
   • No downtime
   • Panic isolated to single request

Without recovery middleware, the entire proxy would crash. This is the difference between:

• ❌ One bad request → entire proxy down → manual restart
• ✅ One bad request → 500 error → logged → proxy still running

Distributed Tracing with Trace IDs

Every request gets a unique trace ID that follows it through your entire system:

# Make a request
curl -v http://localhost:8000/api/users

1. Proxy receives request:

{"level":"info","msg":"Request starter","trace_id":"a1b2c3d4...","method":"GET","path":"/api/users"}

2. Backend receives trace ID:

[BACKEND] Received: GET /api/users | Trace-ID: a1b2c3d4...

3. Backend returns response with trace ID:

{
  "message": "Hello from the backend service!",
  "trace_id": "a1b2c3d4...",
  ...
}

4. Client receives trace ID in header:

< X-Trace-ID: a1b2c3d4...

Why This Matters:

In a microservices architecture with multiple services:

Client → Proxy → Service A → Service B → Service C

Without trace IDs:

• ❌ Logs scattered across services
• ❌ Can't correlate which logs belong to same request
• ❌ Hard to debug issues spanning multiple services
• ❌ No visibility into request flow

With trace IDs:

• ✅ One ID tracks request through entire system
• ✅ Search logs for trace_id: a1b2c3d4... across all services
• ✅ See complete request journey
• ✅ Identify exactly where failures occurred
• ✅ Measure end-to-end latency

Example debugging scenario:

# User reports error, provides trace ID from response
# Search all logs for this trace ID:

# Proxy logs:
{"trace_id":"abc123","msg":"Request starter","path":"/api/order"}
{"trace_id":"abc123","msg":"request completed","status":500}

# Service A logs:
{"trace_id":"abc123","msg":"processing order","order_id":42}
{"trace_id":"abc123","msg":"calling payment service"}

# Service B logs:
{"trace_id":"abc123","error":"payment declined"}  ← Found the issue!

This is the foundation of observability in distributed systems!

gRPC Control Plane Architecture

The control plane allows dynamic configuration of proxies without restarts:

┌─────────────────────────────────────────────────┐
│          Control Plane (Port 9090)              │
│                                                  │
│  - Stores routing configuration                 │
│  - Tracks all registered proxies                │
│  - Pushes updates via gRPC streaming            │
│                                                  │
│  gRPC Server implements:                        │
│    • RegisterProxy(ProxyInfo)                   │
│    • StreamConfig(ProxyInfo) returns stream     │
└─────────────────────────────────────────────────┘
       ▲                    ▲                    ▲
       │ gRPC Stream        │ gRPC Stream        │ gRPC Stream
       │ (long-lived)       │ (long-lived)       │ (long-lived)
       │                    │                    │
   ┌────────┐          ┌────────┐          ┌────────┐
   │Proxy 1 │          │Proxy 2 │          │Proxy 3 │
   │ :8000  │          │ :8001  │          │ :8002  │
   └────────┘          └────────┘          └────────┘

Target Flow:

1. Proxy starts → Calls RegisterProxy() → Control plane tracks it
2. Proxy subscribes → Calls StreamConfig() → Opens long-lived stream
3. Control plane streams config → Sends initial ConfigUpdate (version 1) → Proxy receives routes
4. Proxy applies config → Updates routing table without restart
5. Admin updates config → Control plane calls BroadcastConfigUpdate() → All proxies receive new config instantly!

Current Implementation Status:

Phase 3 Part 4 (Proxy Config Streaming) is complete with:

• ✅ gRPC server running on port 9090
• ✅ Proxy bootstrap config for identity and control-plane address
• ✅ Mandatory control-plane connection during proxy startup
• ✅ Proxy registration via RegisterProxy()
• ✅ Config stream subscription via StreamConfig()
• ✅ Initial config receipt before serving traffic
• ✅ Latest config stored inside the proxy client
• ✅ Config store with default route to localhost:3000
• ✅ Thread-safe concurrent access
• ✅ Graceful shutdown on both binaries

Next Step:

• Apply streamed config to request routing
• Apply route changes without restarting the proxy

Protocol Buffers (.proto file):

The mesh.proto file defines the API contract:

service MeshControl {
    // Unary RPC: single request → single response
    rpc RegisterProxy(ProxyInfo) returns (RegistrationResponse);

    // Server Streaming RPC: single request → stream of responses
    rpc StreamConfig(ProxyInfo) returns (stream ConfigUpdate);
}

message ProxyInfo {
    string proxy_id = 1;        // e.g., "proxy-1"
    string version = 2;         // e.g., "1.0.0"
    string listen_addr = 3;     // e.g., "0.0.0.0:8000"
}

message ConfigUpdate {
    int64 version = 1;          // Config version (increments)
    repeated Route routes = 2;  // List of routes
}

message Route {
    string path = 1;            // e.g., "/api/users"
    string backend = 2;         // e.g., "localhost:3000"
    bool auth_required = 3;     // Require auth?
    int32 timeout_ms = 4;       // Request timeout
}

Why gRPC over REST?

Feature	gRPC	REST
Format	Binary (Protocol Buffers)	Text (JSON)
Speed	Fast (binary encoding)	Slower (JSON parsing)
Streaming	Built-in bidirectional streaming	Difficult (SSE/WebSockets)
Type Safety	Strong (generated code)	Weak (manual parsing)
HTTP	HTTP/2 (multiplexed)	HTTP/1.1 (one request at a time)

Before (Static YAML):

• ❌ Edit config/proxy.yaml file
• ❌ Restart proxy to apply changes
• ❌ Each proxy has its own file
• ❌ No central management

After (Dynamic gRPC):

• ✅ Control plane API to update config
• ✅ Proxies receive updates via stream
• ✅ No restarts needed
• ✅ Centralized configuration

---

Complete Roadmap

✅ Phase 1: Basic Reverse Proxy (Complete)

• HTTP reverse proxy
• YAML configuration
• Graceful shutdown

✅ Phase 2 Part 1: Structured Logging (Complete)

• Zap logger integration
• Logging middleware
• Request/response tracking

✅ Phase 2 Part 2: Prometheus Metrics (Complete)

• Metrics package with Prometheus client
• /metrics endpoint for scraping
• Request counters, histograms, and gauges
• Metrics middleware for automatic tracking

✅ Phase 2 Part 3: Advanced Middleware (Complete)

• Middleware chaining helper with Chain() function
• Recovery middleware with panic handling
• Stack trace logging for debugging
• Resilient proxy that doesn't crash on panics

✅ Phase 2 Part 4: Distributed Tracing (Complete)

• Cryptographically secure trace ID generation (128-bit)
• X-Trace-ID header propagation to backend services
• Trace ID included in all logs for request correlation
• Response headers include trace ID for client debugging

✅ Phase 3 Part 1: gRPC API Definition (Complete)

• Protocol Buffer API definition in mesh.proto
• MeshControl service with RegisterProxy and StreamConfig RPCs
• Message types for proxy info, config updates, and routes
• Automated code generation with protoc
• Build script for reproducible generation

✅ Phase 3 Part 2: Control Plane Server (Complete)

• ✅ Implement gRPC server with MeshControlServer interface
• ✅ Handle proxy registration via RegisterProxy() RPC
• ✅ Config streaming via StreamConfig() server-side streaming RPC
• ✅ Thread-safe proxy connection tracking with sync.RWMutex
• ✅ Versioned configuration store with ConfigStore
• ✅ Broadcast updates to all connected proxies via BroadcastConfigUpdate()
• ✅ Graceful shutdown with signal handling
• ✅ Default route configuration (localhost:3000)
• ✅ Structured logging with Zap (development/production modes)

✅ Phase 3 Part 3: Proxy Registration (Complete)

• ✅ Connect proxy to the control plane during startup
• ✅ Register proxy identity with RegisterProxy()
• ✅ Require control-plane availability before serving traffic
• ✅ Keep static backend routing in place until config streaming is implemented

✅ Phase 3 Part 4: Proxy Config Streaming (Complete)

• ✅ Subscribe to config updates via StreamConfig()
• ✅ Require the first config update before starting the proxy server
• ✅ Store the latest config in memory for later routing integration
• ✅ Stop the proxy if the config stream fails after startup

Phase 3 Part 5: Dynamic Route Application

• Use streamed config inside the request path
• Replace static backend routing with control-plane routes
• Apply route changes without restart

Phase 4: Service Discovery & Load Balancing

• Service registry
• Round-robin load balancing
• Health checking

Phase 5: Production Features

• mTLS encryption
• Circuit breaker
• Rate limiting
• JWT validation