README
ΒΆ
Testcontainers-Go Usage Metrics
This directory contains the automation system for tracking testcontainers-go usage across GitHub repositories.
Overview
The system automatically collects three datasets: two usage metrics obtained by querying the GitHub Code Search API for references to testcontainers-go in go.mod files across public repositories, and the daily clone traffic of this repository obtained from the GitHub Traffic API. The data is visualised in interactive dashboards integrated into the main MkDocs documentation site at https://golang.testcontainers.org/usage-metrics/
Components
π Data Collection (collect.go)
A single Go program with three subcommands:
| Subcommand | Source | Flag |
|---|---|---|
versions |
Code Search: "testcontainers/testcontainers-go {version}" in go.mod |
-version |
modules |
Code Search: "testcontainers/testcontainers-go/modules/{module}" in go.mod |
-module |
clones |
Traffic API: GET /repos/{repo}/traffic/clones?per=day |
-repo (default testcontainers/testcontainers-go), -until (optional YYYY-MM-DD, record days up to and including it) |
The search subcommands exclude forks and testcontainers organisation repositories, retry on rate-limit errors (up to 5 passes, with inter-request and cooldown waits), and append results to a CSV file.
The clones subcommand fetches the last 14 days of clone traffic (the maximum GitHub retains) in a single request, retries retryable errors up to 5 passes, and upserts one row per day into its CSV: rows with a date already present are replaced, because consecutive 14-day windows overlap and GitHub revises recent days. The CSV header must be exactly date,count,uniques and the file is rewritten atomically through a temporary file. -until drops days after the given date; the workflow passes yesterday so the in-progress UTC day is never stored. Authentication errors, including any 403 that does not mention a rate limit, are not retried. The Traffic API requires push access to the repository, so this subcommand needs a token with the Administration: read repository permission.
πΎ Data Storage
| File | Description | Format |
|---|---|---|
docs/usage-metrics/core.csv |
Historical adoption data by library version | date,version,count |
docs/usage-metrics/modules.csv |
Historical import counts by module | date,module,count |
docs/usage-metrics/clones.csv |
Daily clone traffic of this repository | date,count,uniques |
All files are version-controlled for historical tracking and served directly by the MkDocs site.
π Website (integrated into docs/)
| File | Purpose |
|---|---|
docs/usage-metrics/index.md |
Core library dashboard (landing page for the section) |
docs/usage-metrics/modules.md |
Modules dashboard |
docs/usage-metrics/clones.md |
GitHub clones dashboard |
docs/js/usage-metrics.js |
Charts for the core library dashboard |
docs/js/modules-usage-metrics.js |
Charts for the modules dashboard |
docs/js/clones-metrics.js |
Charts for the GitHub clones dashboard, with a Day / Week / Month / Year toggle |
docs/css/usage-metrics.css |
Shared styles for all dashboards |
All dashboards use Chart.js for visualisations and are responsive for mobile and desktop.
π€ Automation
| Workflow | Schedule | Trigger input |
|---|---|---|
.github/workflows/usage-metrics.yml |
Monthly, 1st at 09:00 UTC | versions β comma-separated (empty = all from v0.13.0) |
.github/workflows/usage-metrics-modules.yml |
Monthly, 1st at 10:00 UTC | modules β comma-separated (empty = all modules under modules/) |
.github/workflows/github-clones.yml |
Daily at 03:00 UTC | none |
The two search workflows create a pull request for the metrics update rather than committing directly to main.
The clones workflow runs daily and maintains a monthly data branch named YYYY-MM-clones. On every run the branch is rebuilt on top of main: the run checks out main, restores clones.csv from the branch's previous tip (or from the previous month's branch when a new month starts), upserts the latest days up to yesterday, and force-pushes a single commit with --force-with-lease. The branch therefore always has exactly one commit over main and never conflicts with it. On the first run after a month ends, the previous month's branch is rebuilt the same way with -until set to the month's last day, so its final days carry complete numbers, and a pull request from it to main is opened; merging it publishes the data on the docs site. The check is repeated on every run, so a skipped schedule cannot lose the pull request. A concurrency group prevents two runs from pushing at the same time.
The clones workflow reads the Traffic API with the TRAFFIC_TOKEN repository secret: a fine-grained personal access token scoped to this repository with Administration: read and Metadata: read. Pushes and the pull request use the default GITHUB_TOKEN.
Data Formats
Core library (docs/usage-metrics/core.csv)
Tracks how many public repositories import each version of the core library.
date,version,count
2024-01-15,v0.27.0,133
2024-02-15,v0.27.0,145
2024-02-15,v0.28.0,12
- date: Collection date in
YYYY-MM-DDformat - version: Version string (e.g.
v0.27.0) - count: Number of repositories importing that version
Modules (docs/usage-metrics/modules.csv)
Tracks how many public repositories import each individual module.
date,module,count
2026-06-01,kafka,245
2026-06-01,postgres,1748
2026-06-01,redis,668
- date: Collection date in
YYYY-MM-DDformat - module: Module directory name (e.g.
kafka,postgres) - count: Number of repositories importing that module
GitHub clones (docs/usage-metrics/clones.csv)
Tracks how many times the repository was cloned each day, as reported by GitHub.
date,count,uniques
2026-09-16,5392,1372
2026-09-17,6073,1567
- date: Day of the clones in
YYYY-MM-DDformat (UTC) - count: Total clones on that day
- uniques: Unique cloners on that day, as counted by GitHub. Summing this column over several days overestimates the number of distinct cloners.
History starts on 2026-09-16. GitHub discards traffic older than 14 days, so earlier data cannot be recovered.
Usage
Manual Collection
cd usage-metrics
# Collect specific versions
go run collect.go versions -version v0.37.0 -version v0.38.0 -csv ../docs/usage-metrics/core.csv
# Collect specific modules
go run collect.go modules -module kafka -module redis -csv ../docs/usage-metrics/modules.csv
# Collect the last 14 days of clone traffic (needs a token with push access, see below)
GH_TOKEN=<token> go run collect.go clones -csv ../docs/usage-metrics/clones.csv
# Same, but ignore the in-progress day
GH_TOKEN=<token> go run collect.go clones -csv ../docs/usage-metrics/clones.csv -until "$(date -u -d yesterday +%Y-%m-%d)"
Flags can be repeated for multiple items. All subcommands accept a -csv flag to override the default output path.
Running Locally
# Serve the docs (from the repository root)
make serve-docs
# Core library dashboard: http://localhost:8000/usage-metrics/
# Modules dashboard: http://localhost:8000/usage-metrics/modules/
# GitHub clones dashboard: http://localhost:8000/usage-metrics/clones/
Manual Workflow Trigger
Core library versions:
- Go to Actions β "Update Usage Metrics"
- Click "Run workflow"
- Optionally specify versions (e.g.
v0.39.0,v0.38.0) or leave empty for all versions since v0.13.0
Modules:
- Go to Actions β "Update Modules Usage Metrics"
- Click "Run workflow"
- Optionally specify modules (e.g.
kafka,redis) or leave empty for all modules
GitHub clones:
- Go to Actions β "Update GitHub Clones Metrics"
- Click "Run workflow"
Rate Limiting
GitHub Code Search API limits:
- Unauthenticated: 10 requests/minute
- Authenticated: 30 requests/minute
The collection script queries items sequentially with:
- A 7-second wait between consecutive requests within a pass
- A 65-second cooldown after a rate-limit hit within a pass
- A 120-second wait between passes (up to 5 passes total)
The gh CLI is used under the hood, which automatically authenticates with the GITHUB_TOKEN available in the workflow environment.
Tests
Unit tests cover all logic that does not depend on the gh CLI:
cd usage-metrics
go test ./...
Customisation
Changing Collection Frequency
Edit the cron schedule in the relevant workflow file:
schedule:
- cron: '0 9 1 * *' # Monthly on the 1st at 9 AM UTC
Customising Charts
- Core library: edit
docs/js/usage-metrics.js - Modules: edit
docs/js/modules-usage-metrics.js
Changing the Version Range
By default, usage-metrics.yml queries all tags matching v0.X.Y from v0.13.0 onwards. To change the starting point, modify the awk pattern in the workflow file.
Adding or Removing Modules
The modules workflow auto-discovers all subdirectories under modules/ that contain a go.mod file. No manual list maintenance is needed.
Architecture Decisions
Why a single collect.go with subcommands?
Both collection jobs share the same retry logic, CSV helpers, and GitHub API call pattern. A single binary with versions and modules subcommands eliminates the duplication while keeping the two data models cleanly separated.
Why CSV?
- Simple and human-readable
- Version-controlled with Git for free historical diffing
- No database or external service required
- Suitable for the data volume (one row per item per month)
Why Go for Collection?
- Native to the project β no extra runtime or dependency
- Easy
ghCLI invocation for authenticated GitHub API calls - Good standard-library CSV support
Troubleshooting
API Rate Limiting
The collection script retries automatically (up to 5 passes). If it still fails:
- Check the workflow logs under the "Query versions" or "Query modules" step
- Re-run the workflow manually for the failed items only
CSV Not Updating
- Go to Actions and open the relevant workflow run
- Review the query step output for any errors
- Verify that
GH_TOKEN/GITHUB_TOKENhas Code Search access - For clones, verify that the
TRAFFIC_TOKENsecret exists, has not expired, and grantsAdministration: readon the repository. A403 Must have push accesserror in the "Collect clone traffic" step means the token lacks that permission.
Charts Not Displaying
- Verify the CSV file is at
docs/usage-metrics/core.csv,docs/usage-metrics/modules.csvordocs/usage-metrics/clones.csv - Open the browser console and look for fetch or JavaScript errors
- Ensure Chart.js and PapaParse CDN links are accessible from your browser
License
Same as the main testcontainers-go repository (MIT).
Documentation
ΒΆ
There is no documentation for this package.