benchmark

CronLite Benchmark and Diagnostic Harness

This tool measures the observable lifecycle of CronLite executions. It records API calls, scheduler timestamps, webhook callbacks, retry attempts, terminal status polling, optional PostgreSQL state, Prometheus metrics, correctness failures, and environment metadata.

It is test tooling. It does not change CronLite's API, schema, retry policy, scheduler behavior, or production defaults.

Quick start

With CronLite already running:

export CRONLITE_API_KEY="ec_..."

go run ./tools/benchmark \
  --base-url http://localhost:8080 \
  --api-key "$CRONLITE_API_KEY" \
  --scenario smoke \
  --output ./benchmark-output

The safe default is equivalent to --scenario smoke --sample-count 10. It starts an embedded receiver, creates one benchmark-owned job, manually triggers ten executions, waits for callbacks and terminal statuses, writes all three outputs, and deletes the job.

Prefer the CRONLITE_API_KEY environment variable instead of --api-key in shell history:

CRONLITE_API_KEY="ec_..." \
go run ./tools/benchmark --output ./benchmark-output

Managed local environment

The harness can start an isolated three-instance database-dispatch environment:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario smoke \
  --sample-count 10 \
  --output ./benchmark-output

This uses tools/benchmark/docker-compose.yml, a unique cronlite-benchmark-* Compose project, PostgreSQL 16, three CronLite instances, database dispatch, two workers per instance, Prometheus metrics, and the reconciler.

The environment is left running by default for inspection. To remove the harness-owned containers and volume after the run, explicitly authorize it:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario smoke \
  --cleanup-environment \
  --allow-disruptive \
  --output ./benchmark-output

Cleanup validation rejects project names that do not begin with cronlite-benchmark-. Process-control commands accept only the services defined in the benchmark Compose file.

Managed ports are:

Measurement modes

Black-box mode

Black-box mode is the default. It uses only interfaces available to a normal CronLite customer:

• REST API;
• health endpoints;
• execution inspection;
• webhook callbacks;
• Prometheus metrics when accessible.

It does not query CronLite's database.

Diagnostic mode

Enable read-only PostgreSQL inspection with:

CRONLITE_BENCHMARK_DATABASE_URL="postgres://user:pass@localhost/cronlite?sslmode=disable" \
go run ./tools/benchmark \
  --diagnostic \
  --scenario smoke \
  --output ./benchmark-output

Every diagnostic query runs in a transaction initialized with:

SET TRANSACTION READ ONLY

Diagnostic mode observes execution creation, scheduling, firing, claiming, attempt start/finish, status, queue depth, active executions, connection count, and database size where permissions allow.

Scenario reference

Select more than one scenario with a comma:

go run ./tools/benchmark \
  --scenario baseline,smoke,warm-sequential,concurrent \
  --sample-count 100 \
  --concurrency 1,5,10,25,50 \
  --output ./benchmark-output

--scenario all expands to every scenario and therefore requires --allow-disruptive. It includes real recurring ticks and production retry backoff, so it is intentionally long-running.

Full local suite profiles

Use separate profiles for steady-state, saturation, real retry timing, and fault injection. Keeping them separate prevents a destructive case from contaminating the latency baseline and makes failures easier to reproduce. The commands are standalone profiles; stop the prior managed stack or use --cleanup-environment --allow-disruptive before starting another one because managed stacks use fixed local ports.

Steady-state and edge-case coverage:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario baseline,smoke,cold-warm,warm-sequential,concurrent,control-plane,slow-receiver \
  --sample-count 100 \
  --concurrency 1,5,10,25,50,100 \
  --timeout 120s \
  --fail-on-correctness \
  --output ./benchmark-output/steady

Real minute-resolution scheduler ticks:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario recurring \
  --sample-count 3 \
  --timeout 4m \
  --fail-on-correctness \
  --output ./benchmark-output/recurring

The complete production retry policy takes roughly 15 minutes because attempts use the unchanged 30-second, 2-minute, and 10-minute backoffs:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario retry \
  --retry-profile real-policy \
  --timeout 120s \
  --fail-on-correctness \
  --output ./benchmark-output/retry

Saturation profile, producing 1,000 executions at each configured concurrency:

go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario load \
  --sample-count 1000 \
  --concurrency 100,250,500,1000 \
  --timeout 5m \
  --allow-disruptive \
  --fail-on-correctness \
  --output ./benchmark-output/saturation

Crash, failover, duplicate-race, and database-outage profile:

BENCHMARK_REQUEUE_THRESHOLD=5s \
go run ./tools/benchmark \
  --start-compose \
  --diagnostic \
  --scenario duplicate-race,crash-recovery,leader-failover,database-outage \
  --sample-count 1 \
  --timeout 120s \
  --requeue-threshold 5s \
  --allow-disruptive \
  --fail-on-correctness \
  --output ./benchmark-output/resilience

The five-second requeue threshold is deliberately unsafe and is accepted only by the managed CRONLITE_ENV=benchmark stack. CronLite emits a P0 warning when it is active. Normal environments reject a requeue threshold shorter than the complete dispatcher retry window.

Retry profiles

--retry-profile real-policy uses CronLite's unchanged policy:

attempt 1: immediate
attempt 2: 30 seconds
attempt 3: 2 minutes
attempt 4: 10 minutes

--retry-profile fast-test is accepted so automation can request the profile, but the retry scenario is marked skipped. CronLite currently has no runtime or build-tagged test-only retry injection. The harness does not modify production retry defaults to shorten a benchmark.

Recurring versus manual timing

CronLite recurring cron expressions have one-minute resolution. Recurring results measure scheduler accuracy and drift across real ticks. Manual triggers measure millisecond-level API, dispatch, callback, and persistence behavior. The report never merges the two sample groups.

Important flags

Run go run ./tools/benchmark --help for the complete list.

Correlation and receiver behavior

Every intended execution gets:

• benchmark run ID;
• scenario name;
• UUID benchmark correlation ID;
• job ID;
• execution ID;
• attempt/event ID;
• target URL;
• sample number and warm-up marker.

The receiver verifies X-CronLite-Signature using constant-time HMAC-SHA256, bounds callback bodies to 64 KiB, and records UTC nanosecond timestamps. It detects:

• repeated execution callbacks;
• repeated attempt IDs;
• simultaneous callbacks for one execution;
• callbacks after terminal status was observed;
• invalid signatures;
• callback payload changes.

Receiver observations are the authority for duplicate HTTP side effects. A single final execution row does not erase evidence that a webhook was delivered more than once.

Timestamp provenance

Each measurement is labeled:

Go's monotonic clock component measures durations inside the harness. UTC RFC3339-nanosecond timestamps correlate different processes. Cross-process durations therefore depend on clock synchronization.

Formulas

The report uses milliseconds:

api_trigger_latency_ms =
    trigger_response_received - trigger_request_start

scheduler_lag_ms =
    execution_created_at - intended_scheduled_at

queue_wait_ms =
    claimed_at - execution_created_at

claim_to_dispatch_ms =
    first_attempt_started_at - claimed_at

webhook_rtt_ms =
    attempt_finished_at - attempt_started_at

receiver_processing_ms =
    receiver_response_completed - receiver_arrival

retry_backoff_actual_ms =
    next_attempt_started_at - previous_attempt_finished_at

retry_backoff_error_ms =
    retry_backoff_actual_ms - configured_backoff_ms

end_to_end_delivery_ms (manual) =
    receiver_arrival - trigger_request_start

end_to_end_delivery_ms (recurring) =
    receiver_arrival - intended_scheduled_at

terminal_persistence_lag_ms =
    first_terminal_poll_observation - final_callback_response_completed

CronLite does not persist the exact terminal status update time. terminal_persistence_lag_ms is therefore an upper bound whose uncertainty includes --poll-interval.

Statistics

Every timing distribution includes:

• sample count;
• minimum and maximum;
• arithmetic mean;
• median;
• sample standard deviation;
• p50, p90, p95, and p99 using nearest-rank percentiles.

The report warns when p90 has fewer than 10 samples, p95 fewer than 20, or p99 fewer than 100. It also reports throughput, success rate, retry rate, permanent failure rate, duplicate rate, signature-failure rate, HTTP status distribution, error classification, and queue/resource observations when available.

Warm-up samples remain in raw output but are excluded from steady-state statistics.

Output files

Every completed run writes atomically:

benchmark-results.json
benchmark-results.csv
benchmark-report.md

JSON

The JSON document has schema_version: "1.0" and retains full raw:

• environment and redacted configuration;
• API, callback, metric, database, and process-control observations;
• execution and attempt identifiers;
• timestamps and polling bounds;
• measurements with provenance;
• scenario failures and skips;
• correctness findings;
• limitations.

CSV

The CSV has one row per delivery attempt, or one execution row when no attempt is observable. Its stable columns are:

run_id
scenario
correlation_id
job_id
execution_id
attempt_id
attempt
status_code
signature_valid
duplicate
warmup
trigger_type
final_status
error_class
scheduled_at
fired_at
execution_created_at
attempt_started_at
attempt_finished_at
receiver_arrived_at
api_trigger_latency_ms
scheduler_lag_ms
queue_wait_ms
claim_to_dispatch_ms
webhook_rtt_ms
receiver_processing_ms
terminal_persistence_lag_ms
end_to_end_delivery_ms
measurement_provenance

Markdown

The report puts correctness failures before performance numbers and contains:

1. Environment and configuration
2. Correctness findings
3. Scenario summary
4. Baseline HTTP latency
5. API latency
6. Scheduler accuracy
7. Delivery latency
8. Retry timing
9. Throughput
10. Failure analysis
11. Duplicate-delivery findings
12. Resource usage
13. Limitations
14. Reproduction instructions
15. Raw-file locations

The run header also records commit SHA, exact redacted command, start/finish times, and duration.

Exit codes

Outputs are still written after scenario failures whenever setup and output storage remain available.

Safety

• Non-loopback CronLite targets require --allow-non-local.
• Duplicate, load, crash, leader, and database-outage scenarios require --allow-disruptive.
• Environment deletion requires --start-compose --allow-disruptive.
• Compose deletion only targets cronlite-benchmark-* projects.
• Created jobs are deleted unless --keep-data is set.
• API keys, webhook secrets, authorization headers, PostgreSQL credentials, and receiver behavior tokens are redacted from output.
• Request/response bodies and Prometheus responses are bounded.
• Diagnostic SQL is parameterized and read-only.

Do not point aggressive workloads at a shared or production service without explicit authorization and capacity planning.

Current instrumentation gaps

The harness reports these gaps instead of changing CronLite:

1. Public execution responses omit delivery attempts even though the endpoint description says it returns them.
2. claimed_at exists in PostgreSQL but is not public.
3. Terminal status update time is not persisted.
4. Attempt start/finish times are stored but not returned by the public API.
5. Worker or instance identity is not attached to executions or attempts.
6. Webhook payloads have execution IDs but no customer-supplied correlation ID.
7. Retry backoff has no test-only runtime injection.
8. Exact final-response-to-terminal-persistence time is unavailable; API polling provides only a bound.
9. A database claim transaction commits before HTTP delivery, so stale requeue behavior must be verified from receiver side effects.

General limitations

• Recurring cron resolution is one minute.
• Cross-process timings depend on synchronized clocks.
• Same-machine Docker introduces virtualization and contention noise.
• Local results are regression baselines, not universal capacity guarantees.
• Tail percentiles need adequate samples.
• Production network conditions can differ substantially.
• Diagnostic database visibility is not customer-visible behavior.
• A benchmark can reveal duplicates but cannot mathematically prove every race is absent.
• Missing Docker, PostgreSQL, permissions, or platform features produce explicit skipped scenarios.

Example output

tools/benchmark/example-output contains a locally generated example. Read its environment and scenario status before interpreting measurements. If Docker was unavailable on the generating machine, the report marks the managed CronLite scenario as skipped/synthetic rather than inventing production measurements.