e2e_harness

E2E Test Harness

End-to-end test harness for validating the federated query system, including the three-tier data architecture, CDC flushing, compaction, and data consistency.

Prerequisites

• Go 1.21+
• Docker - Required for running test containers (Postgres, RustFS/MinIO)
• Sufficient memory - Tests may require 2GB+ for container orchestration

Quick Start

# Run the minimal harness test (verifies infrastructure)
go test -v ./internal/e2e_harness/... -timeout=5m

# Run all federated E2E tests
go test -v ./internal/e2e_harness/federated/... -tags=e2e -timeout=30m

# Skip performance tests (faster CI runs)
go test -v ./internal/e2e_harness/federated/... -tags=e2e -short

Architecture

The E2E harness tests validate a three-tier federated query architecture:

flowchart TB
    subgraph FQ["Federated Query"]
        SQL["SELECT * FROM (base UNION ALL delta UNION ALL hot) QUALIFY ..."]
    end
    
    FQ --> Base
    FQ --> Delta
    FQ --> Hot
    
    Base["<b>S3 Base (Parquet)</b><p>• Oldest data</p><p>• Compacted</p><p>• Read-only</p>"]
    Delta["<b>S3 Delta (Parquet)</b><p>• Recent flushes</p><p>• Pre-compaction</p><p>• Append-only</p>"]
    Hot["<b>Postgres Hot Buffer</b><p>• Unflushed</p><p>• Real-time</p><p>• Read-write</p>"]

Tier Descriptions

Data Flow

1. Writes go to Postgres (Hot Buffer)
2. CDC Flush moves data from Hot Buffer to Delta (S3 Parquet)
3. Compaction merges Delta files into Base files
4. Queries federate across all three tiers with deduplication

sequenceDiagram
    participant Client as Client
    participant PG as Postgres<br/>(Hot Buffer)
    participant CDC as CDC Flusher
    participant S3Delta as S3 Delta<br/>(Parquet)
    participant Compactor as Compactor
    participant S3Base as S3 Base<br/>(Parquet)

    Note over Client,S3Base: Write Flow
    
    Client->>PG: INSERT/UPDATE/DELETE
    PG-->>Client: OK (unflushed)
    
    Note over PG,S3Delta: CDC Flush Triggers:<br/>• Records > 10,000<br/>• Oldest record > 1 hour
    
    CDC->>PG: Acquire Advisory Lock
    CDC->>PG: SELECT unflushed records
    PG-->>CDC: Return pending records
    CDC->>S3Delta: Write Parquet file
    S3Delta-->>CDC: OK
    CDC->>PG: UPDATE flushed_at timestamp
    CDC->>PG: Release Advisory Lock
    
    Note over S3Delta,S3Base: Compaction Trigger:<br/>• Dirty Ratio > 5%
    
    Compactor->>S3Delta: Read Delta files
    Compactor->>S3Base: Read Base files
    Compactor->>S3Base: Merge and write new Base file
    Compactor->>S3Delta: Delete merged Delta files

Test Categories

TC-01: Three-Tier Data Architecture

File: federated/data_tier_test.go

Validates that queries correctly access Base, Delta, and Hot tiers independently and together.

go test -v ./internal/e2e_harness/federated/... -run "TestDataTier" -tags=e2e

TC-02: Merge-on-Read Logic

File: federated/merge_on_read_test.go

Validates UNION ALL and merge behavior across tiers.

go test -v ./internal/e2e_harness/federated/... -run "TestMergeOnRead" -tags=e2e

TC-03: Global Deduplication

File: federated/deduplication_test.go

Validates QUALIFY ROW_NUMBER deduplication logic.

go test -v ./internal/e2e_harness/federated/... -run "TestDeduplication" -tags=e2e

TC-04: Soft Delete Filtering

File: federated/soft_delete_test.go

Validates deleted_at filtering logic.

go test -v ./internal/e2e_harness/federated/... -run "TestSoftDelete" -tags=e2e

TC-05: CDC Smart Flushing

File: federated/cdc_flush_test.go

Tests CDC flush triggers, advisory locks, and batch processing.

go test -v ./internal/e2e_harness/federated/... -run "TestCDCFlush" -tags=e2e

TC-06: Compaction Strategy

File: federated/compaction_test.go

Tests delta-to-base compaction behavior.

go test -v ./internal/e2e_harness/federated/... -run "TestCompaction" -tags=e2e

TC-07: Data Consistency

File: federated/consistency_test.go

Validates data consistency between Postgres and federated queries.

go test -v ./internal/e2e_harness/federated/... -run "TestConsistency" -tags=e2e

TC-08: Performance Benchmarks

File: federated/performance_test.go

Benchmarks query latency and throughput at medium scale (10K-20K records for tests, 100K-500K in production).

go test -v ./internal/e2e_harness/federated/... -run "TestPerformance" -tags=e2e -timeout=60m

Performance Thresholds

Simple Pagination Query:

Complex Filter Query:

Full Table Scan:

Concurrent Queries (20 Virtual Users):

CDC Flush (5K-20K records):

Compaction (20K records):

TC-09: Failure Modes

File: federated/failure_modes_test.go

Tests graceful degradation and error handling.

go test -v ./internal/e2e_harness/federated/... -run "TestFailureMode" -tags=e2e

Test Harness Components

TestHarness (Base)

Located in harness.go, provides:

• Postgres container management (via testcontainers)
• S3/RustFS container management
• DuckDB client initialization
• Basic seeding and parquet file operations

FederatedTestHarness (Extended)

Located in federated/harness.go, extends base harness with:

• Three-tier data seeding
• Federated query execution
• CDC flush simulation
• Compaction operations
• Data comparison utilities

Configuration

Key configuration options in the test harness:

CDCConfig{
    MinRecords:  100,    // Minimum unflushed records to trigger flush
    MaxAgeMs:    60000,  // Maximum age (ms) before flush triggered
    BatchSize:   1000,   // Records per flush batch
    S3Bucket:    "test-bucket",
    S3Prefix:    "test-project",
}

Running Specific Tests

# Run smoke test only
go test -v ./internal/e2e_harness/federated/... -run "TestSuite_Smoke" -tags=e2e

# Run integration test
go test -v ./internal/e2e_harness/federated/... -run "TestSuite_Integration" -tags=e2e

# Run benchmarks
go test -v ./internal/e2e_harness/federated/... -bench=. -tags=e2e -timeout=30m

# Run with verbose logging
LOG_LEVEL=debug go test -v ./internal/e2e_harness/federated/... -tags=e2e

# Run with race detection
go test -v -race ./internal/e2e_harness/federated/... -tags=e2e -timeout=45m

Troubleshooting

Container Startup Failures

If containers fail to start:

1. Ensure Docker daemon is running
2. Check for port conflicts on 5432 (Postgres) and 9000 (S3)
3. Increase container startup timeout if needed

Timeout Issues

Performance tests may timeout on slower machines:

# Increase timeout
go test -v ./internal/e2e_harness/federated/... -tags=e2e -timeout=60m

Memory Issues

If tests run out of memory:

1. Run test categories individually instead of all at once
2. Use -short flag to skip large-scale performance tests
3. Increase Docker memory allocation