verification

package
v0.0.0-...-e2f4ec3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 12, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

README

Verification Gate 2: Independent Recompute from Parquet

This package implements Verification Gate 2 from the commitgraph v2 plan (docs/plan/plan.md).

Purpose

For a sample of repos, recompute the rollup directly from the corpus Parquet with a separate implementation path and assert it matches what Postgres holds. This checks the pipeline against ground truth rather than against another pipeline, needs no predecessor, and keeps working indefinitely as a production audit — unlike Gate 1, which decays as the golden snapshot ages.

Key Requirement: True Independence

CRITICAL: This implementation shares NO code with the production write path (pkg/rollup and pkg/pg/rollup_write.go). This is intentional — the verification must be a truly independent implementation to catch bugs that would otherwise be missed by testing the production code against itself.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│ VERIFICATION GATE 2: INDEPENDENT RECOMPUTE                                │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  1. Read Parquet Artifact (ARMOR)                                           │
│     ├─ Per-repo: commitgraph/repo/<provider>/<repo>/commits.parquet        │
│     └─ Decrypt + Parse (PyArrow or Go Parquet library)                    │
│                                                                              │
│  2. Parse Commit Messages (Independent Detection)                           │
│     ├─ Co-Authored-By trailers                                              │
│     ├─ Body patterns                                                        │
│     └─ Author name patterns                                                 │
│                                                                              │
│  3. Apply Quarantine Filter (Independent Implementation)                      │
│     └─ Exclude commits outside [2005-01-01, today+1]                       │
│                                                                              │
│  4. Aggregate by (user_email, tool, day) → count                           │
│     └─ Group and count                                                      │
│                                                                              │
│  5. Resolve user_email → user_id (email_resolution)                          │
│     └─ Query Postgres email_resolution table                                │
│                                                                              │
│  6. Compare with Postgres Rollup                                            │
│     ├─ Read: repo_user_daily_tool table                                     │
│     ├─ Diff: recomputed vs postgres                                        │
│     └─ Report: differences + Claude Code canonical rule                     │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Components

  • recompute.go: Independent rollup recomputation logic

    • Recomputer: Parses commit messages, applies quarantine, aggregates rollups
    • PostgresReader: Reads live Postgres rollup data
    • Verifier: Orchestrates comparison and generates diffs

  • parquet_reader.go: Parquet artifact reading (ARMOR integration)

    • ParquetCommitReader: Reads commits from ARMOR Parquet artifacts
    • PostgresEmailResolver: Resolves author emails to user_ids
    • EnhancedVerifier: Full verification with email resolution

  • reporter.go: Verification result reporting

    • Reporter: Human-readable console reports
    • JSONReporter: Machine-readable JSON output
    • CIReporter: CI-friendly output format
    • BatchVerifier: Multi-repo verification with sampling

  • recompute_test.go: Unit tests for independent implementation

CLI Usage

# Verify a specific repo
go run cmd/verify-rollup -repo github.com/jedarden/commitgraph

# Verify top 10 repos by AI commit count
go run cmd/verify-rollup -sample 10 -strategy top_commits

# Verify repos with Claude Code activity (canonical rule testing)
go run cmd/verify-rollup -sample 20 -strategy claude_code

# CI-friendly output
go run cmd/verify-rollup -sample 5 -output ci -verbose

Environment Variables

  • COMMITGRAPH_POSTGRES_URL: Postgres connection string (required)

Sampling Strategies

  • random: Pick random repos from the corpus
  • top_commits: Pick repos with the most AI commits
  • recent: Pick the most recently scanned repos
  • claude_code: Pick repos with Claude Code activity (for canonical rule testing)

Claude Code Canonical Rule

For Claude Code discrepancies, claude-leaderboard is canonical. See docs/notes/cg-37gw-claude-leaderboard-canonical-rule.md.

When commitgraph and claude-leaderboard disagree on Claude Code counts:

  • Investigate commitgraph, not claude-leaderboard
  • claude-leaderboard is the authoritative source for Claude Code leaders

Exit Codes

  • 0: Success (all verifications passed)
  • 1: Flag parsing error
  • 2: Postgres connection error
  • 3: Verification failed (differences detected)

Implementation Status

✅ Complete
  • Independent rollup recomputation logic (no shared code with production)
  • Quarantine filtering (date bounds)
  • AI tool detection (Co-Authored-By trailers, body patterns)
  • Email resolution (Postgres email_resolution table)
  • Postgres rollup reading
  • Diff generation and reporting
  • Unit tests for core logic
  • CLI command skeleton
⚠️ In Progress (TODO)
  • ARMOR integration for Parquet reading
  • Repo sampling logic from Postgres
  • claude-leaderboard integration for canonical rule
  • Full end-to-end testing
🔄 Future Enhancements
  • Parquet streaming for large repos
  • Parallel verification (multiple repos at once)
  • Historical trend analysis (detect drift over time)
  • Integration with CI/CD pipeline
  • Automated remediation suggestions

Relationship to Production Code

This package is intentionally independent of the production pipeline:

Production Code Verification Code Purpose
pkg/rollup/rollup.go pkg/verification/recompute.go Separate aggregation logic
pkg/pg/rollup_write.go pkg/verification/parquet_reader.go Separate read paths
shared/detection.py pkg/verification/recompute.go Separate detection logic

This independence ensures that bugs in the production code are caught by verification, even if the same bug exists in both the write and read paths of the production system.

Testing

# Run unit tests
go test ./pkg/verification/...

# Run with coverage
go test -cover ./pkg/verification/...

# Run specific test
go test -v -run TestQuarantineBounds ./pkg/verification/...

Documentation References

  • Plan: docs/plan/plan.md — "Verification", "Gate 2 — independent recompute"
  • Canonical Rule: docs/notes/cg-37gw-claude-leaderboard-canonical-rule.md
  • Invariants: docs/plan/plan.md — "Invariants" (Invariant 1: Rollup matches artifact)

Contributing

When modifying this package, maintain the no shared code invariant:

  1. DO NOT import from pkg/rollup or pkg/pg/rollup_write.go
  2. DO NOT reuse production aggregation logic
  3. DO reimplement detection logic independently
  4. DO add tests that verify independence

The goal is to have two completely separate code paths that should produce identical results. Any divergence indicates a bug in one or both paths.

Documentation

Overview

Package verification provides independent recompute verification for commitgraph.

This package implements Verification Gate 2 from docs/plan/plan.md: "For a sample of repos, recompute the rollup directly from the corpus Parquet with a separate implementation path and assert it matches what Postgres holds."

CRITICAL: This implementation shares NO code with the production write path (pkg/rollup and pkg/pg/rollup_write.go). This is intentional — the verification must be a truly independent implementation to catch bugs in the production code.

The recompute logic: 1. Reads a repo's Parquet artifact from ARMOR 2. Parses each commit's message trailer for AI tool detection 3. Applies date quarantine filtering (exclude commits outside [2005-01-01, today+1]) 4. Aggregates by (author_email, tool, day) → count 5. Resolves author_email to user_id via email_resolution 6. Produces independent (user_id, repo_id, tool, day, count) rows

This is then compared against Postgres rollup data to detect divergence.

Index

Constants

View Source 
const (
	MinQuarantineDate = "2005-01-01T00:00:00Z"
)

Quarantine bounds match the production system exactly. See pkg/rollup/rollup.go for the production implementation.

Variables

This section is empty.

Functions

func ExampleVerification 

func ExampleVerification()

Example usage function (for documentation)

Types

type BatchResult 

type BatchResult struct {
	TotalRepos       int
	PassedRepos      int
	FailedRepos      int
	TotalCommits     int
	TotalDifferences int
	Duration         time.Duration
	RepoResults      []*DetailedVerificationResult
}

BatchResult contains aggregated batch verification results.

type BatchVerifier 

type BatchVerifier struct {
	// contains filtered or unexported fields
}

BatchVerifier runs verification on multiple repos and aggregates results.

func NewBatchVerifier 

func NewBatchVerifier(verifier *EnhancedVerifier, reporter *Reporter) *BatchVerifier

NewBatchVerifier creates a new batch verifier.

func (*BatchVerifier) VerifyBatch 

func (bv *BatchVerifier) VerifyBatch(ctx context.Context, repos []RepoKey) (*BatchResult, error)

VerifyBatch verifies multiple repositories in sequence.

type CIReporter 

type CIReporter struct {
	// contains filtered or unexported fields
}

CIReporter outputs verification results in a CI-friendly format.

func NewCIReporter 

func NewCIReporter(output io.Writer) *CIReporter

NewCIReporter creates a new CI reporter.

func (*CIReporter) ReportResult 

func (cir *CIReporter) ReportResult(result *DetailedVerificationResult) error

ReportResult writes a CI-friendly verification report.

type ClaudeCodeDiscrepancy 

type ClaudeCodeDiscrepancy struct {
	UserEmail              string
	RecomputedCount        int
	PostgresCount          int
	ClaudeLeaderboardCount int // Authoritative count from claude-leaderboard
	Day                    time.Time
	Conclusion             string // "commitgraph_correct", "claude_leaderboard_correct", "needs_investigation"
}

ClaudeCodeDiscrepancy details a Claude Code-specific discrepancy. See docs/notes/cg-37gw-claude-leaderboard-canonical-rule.md

type ClaudeLeaderboardResolver 

type ClaudeLeaderboardResolver struct {
}

ClaudeLeaderboardResolver handles the canonical rule for Claude Code discrepancies. See docs/notes/cg-37gw-claude-leaderboard-canonical-rule.md

func NewClaudeLeaderboardResolver 

func NewClaudeLeaderboardResolver() *ClaudeLeaderboardResolver

NewClaudeLeaderboardResolver creates a new resolver.

func (*ClaudeLeaderboardResolver) IsClaudeCodeTool 

func (clr *ClaudeLeaderboardResolver) IsClaudeCodeTool(tool string) bool

IsClaudeCodeTool returns true if the tool is Claude Code or a variant.

func (*ClaudeLeaderboardResolver) ResolveCanonicalCount 

func (clr *ClaudeLeaderboardResolver) ResolveCanonicalCount(ctx context.Context, repoID int64, userEmail string, day time.Time) (int, error)

ResolveCanonicalCount would query claude-leaderboard for the authoritative count. For now, it returns a placeholder implementation.

type Commit 

type Commit struct {
	SHA         string
	AuthorEmail string
	AuthorName  string
	CommittedAt time.Time
	Message     string
}

Commit represents a single commit read from Parquet for recomputation. This is intentionally independent of any production data structures.

type CommitReader 

type CommitReader interface {
	ReadCommits(repoID int64) ([]Commit, error)
}

CommitReader reads commit data from a Parquet file. This is a simplified interface — the actual implementation will need to handle ARMOR decryption and Parquet decoding.

type DetailedVerificationResult 

type DetailedVerificationResult struct {
	RepoID               int64
	Provider             string
	RepoFullName         string
	TotalCommits         int
	AICommits            int
	QuarantineExcluded   int
	TotalRecomputed      int
	TotalPostgres        int
	MatchCount           int
	MissingInPostgres    int
	MissingInRecompute   int
	CountMismatches      int
	UserMismatches       int
	ClaudeCodeDiffs      []ClaudeCodeDiscrepancy
	Differences          []RollupDifference
	VerificationDuration time.Duration
}

VerificationResult contains the detailed outcome of a verification run. This extends the basic VerificationResult with more detailed analysis.

func (*DetailedVerificationResult) MarshalJSON 

func (dvr *DetailedVerificationResult) MarshalJSON() ([]byte, error)

MarshalJSON implements JSON marshaling for DetailedVerificationResult.

func (*DetailedVerificationResult) Summary 

func (dvr *DetailedVerificationResult) Summary() string

Summary returns a human-readable summary of the verification result.

type EnhancedVerifier 

type EnhancedVerifier struct {
	// contains filtered or unexported fields
}

EnhancedVerifier provides detailed verification with email resolution and Claude Code canonical rule.

func NewEnhancedVerifier 

func NewEnhancedVerifier(
	recomputer *Recomputer,
	pgReader *PostgresReader,
	commitReader *ParquetCommitReader,
	emailResolver *PostgresEmailResolver,
) *EnhancedVerifier

NewEnhancedVerifier creates a new enhanced verifier.

func (*EnhancedVerifier) VerifyRepoDetailed 

func (ev *EnhancedVerifier) VerifyRepoDetailed(ctx context.Context, repo RepoKey) (*DetailedVerificationResult, error)

VerifyRepoDetailed performs a detailed verification of a single repository.

type JSONReporter 

type JSONReporter struct {
	// contains filtered or unexported fields
}

JSONReporter outputs verification results as JSON for machine processing.

func NewJSONReporter 

func NewJSONReporter(output io.Writer) *JSONReporter

NewJSONReporter creates a new JSON reporter.

func (*JSONReporter) ReportResult 

func (jr *JSONReporter) ReportResult(result *DetailedVerificationResult) error

ReportResult writes verification result as JSON.

type ParquetCommit 

type ParquetCommit struct {
	SchemaVersion int32  // Schema version identifier
	SHA           string // Commit SHA
	Provider      string // Git provider (e.g., "github")
	Repo          string // Repository full name (e.g., "owner/repo")
	Username      string // Git username (may be empty or noreply@)
	AuthorName    string // Git author name
	AuthorEmail   string // Git author email
	CommittedAt   int64  // Commit timestamp (microseconds since Unix epoch)
	Subject       string // Commit subject line
	Message       string // Full commit message
}

Commit represents a row from the Parquet commit artifact. This matches the production Parquet schema (see commitgraph-deprecated/containers/clone-worker/worker.py)

type ParquetCommitReader 

type ParquetCommitReader struct {
	// contains filtered or unexported fields
}

ParquetCommitReader reads commit data from Parquet artifacts stored in ARMOR.

This implementation is completely independent of any production code path that might read Parquet files. It implements only the minimal functionality needed for verification: reading per-repo Parquet artifacts and extracting commit SHA, author email/name, committed_at, and message.

The ARMOR key structure for per-rejo Parquet artifacts is: commitgraph/repo/<provider>/<repo_full_name>/commits.parquet

func NewParquetCommitReader 

func NewParquetCommitReader(config ParquetReaderConfig) *ParquetCommitReader

NewParquetCommitReader creates a new Parquet commit reader.

func (*ParquetCommitReader) ParquetArtifactKey 

func (r *ParquetCommitReader) ParquetArtifactKey(repo RepoKey) string

ParquetArtifactKey returns the ARMOR key for a repo's Parquet commit artifact.

func (*ParquetCommitReader) ReadCommits 

func (r *ParquetCommitReader) ReadCommits(ctx context.Context, repo RepoKey) ([]Commit, error)

ReadCommits reads all commits for a repository from its Parquet artifact.

This is a stub implementation. The real implementation would: 1. Construct the ARMOR object key from repo info 2. Make an HTTP GET request to ARMOR with decryption credentials 3. Stream the decrypted Parquet data 4. Use PyArrow or similar to read the Parquet file 5. Extract commit rows and return them as []Commit

For now, this returns an error to indicate it needs implementation.

type ParquetReaderConfig 

type ParquetReaderConfig struct {
	// ARMOREndpoint is the ARMOR proxy endpoint
	ARMOREndpoint string

	// ARMORBucket is the ARMOR bucket name
	ARMORBucket string

	// ARMORPrefix is the ARMOR prefix (e.g., "commitgraph/")
	ARMORPrefix string
}

ParquetReaderConfig configures the Parquet commit reader.

type PostgresEmailResolver 

type PostgresEmailResolver struct {
	// contains filtered or unexported fields
}

PostgresEmailResolver resolves author emails to user_ids using Postgres email_resolution table.

This is needed because the Postgres rollup table uses user_id (surrogate key) while the recomputed rollup starts from author_email. We need to resolve emails to user_ids for accurate comparison.

func NewPostgresEmailResolver 

func NewPostgresEmailResolver(conn *pgx.Conn) *PostgresEmailResolver

NewPostgresEmailResolver creates a new email resolver.

func (*PostgresEmailResolver) ResolveEmail 

func (per *PostgresEmailResolver) ResolveEmail(ctx context.Context, email string) (int64, bool, error)

ResolveEmail resolves a single author email to its user_id. Returns (user_id, true, nil) if resolved, (0, false, nil) if not found.

type PostgresReader 

type PostgresReader struct {
	// contains filtered or unexported fields
}

PostgresReader reads rollup data from Postgres for comparison.

func NewPostgresReader 

func NewPostgresReader(conn *pgx.Conn) *PostgresReader

NewPostgresReader creates a new Postgres rollup reader.

func (*PostgresReader) ReadRepoRollup 

func (pr *PostgresReader) ReadRepoRollup(ctx context.Context, repoID int64) ([]PostgresRollup, error)

ReadRepoRollup reads all rollup rows for a specific repo from Postgres.

type PostgresRollup 

type PostgresRollup struct {
	UserID int64     // Resolved user_id
	RepoID int64     // Repository ID
	Tool   string    // AI tool name
	Day    time.Time // Day (UTC midnight)
	Count  int       // Number of commits
}

PostgresRollup represents a rollup row read from Postgres.

type RecomputedRollup 

type RecomputedRollup struct {
	UserEmail string    // Author email (pre-resolution)
	RepoID    int64     // Repository ID
	Tool      string    // AI tool name
	Day       time.Time // Day (UTC midnight)
	Count     int       // Number of commits
}

RecomputedRollup represents a single independently-computed rollup row. This is the verification-side equivalent of pg.RollupRow but computed via a completely separate code path.

type Recomputer 

type Recomputer struct {
	// contains filtered or unexported fields
}

Recomputer handles independent rollup recomputation from Parquet data.

func NewRecomputer 

func NewRecomputer(today time.Time) *Recomputer

NewRecomputer creates a new Recomputer for verification.

func (*Recomputer) RecomputeFromCommits 

func (r *Recomputer) RecomputeFromCommits(commits []Commit, repoID int64) []RecomputedRollup

RecomputeFromCommits independently computes rollup rows from raw commits. This is a complete reimplementation of pkg/rollup.ComputeRollup to maintain true independence for verification.

type RepoKey 

type RepoKey struct {
	Provider     string // e.g., "github"
	RepoFullName string // e.g., "owner/repo"
	RepoID       int64  // Internal repository ID
}

RepoKey describes a repository's location in ARMOR.

type Reporter 

type Reporter struct {
	// contains filtered or unexported fields
}

Reporter formats and outputs verification results.

func NewReporter 

func NewReporter(output io.Writer, verbose bool) *Reporter

NewReporter creates a new verification reporter.

func (*Reporter) ReportResult 

func (r *Reporter) ReportResult(result *DetailedVerificationResult) error

ReportResult writes a human-readable verification report.

type RollupDifference 

type RollupDifference struct {
	Recomputed     *RecomputedRollup // May be nil if row only exists in Postgres
	Postgres       *PostgresRollup   // May be nil if row only exists in recomputed
	DifferenceType string            // "missing_in_postgres", "missing_in_recompute", "count_mismatch", "user_mismatch"
}

RollupDifference represents a single row difference between recomputed and Postgres rollup.

type SampleConfig 

type SampleConfig struct {
	Strategy   SampleStrategy
	Count      int   // Number of repos to sample
	Seed       int64 // Random seed (for reproducibility)
	MinCommits int   // Minimum AI commits to include (filters out inactive repos)
}

SampleConfig configures repo sampling for verification.

func DefaultSampleConfig 

func DefaultSampleConfig() SampleConfig

DefaultSampleConfig returns a reasonable default sampling configuration.

type SampleStrategy 

type SampleStrategy string

SampleStrategy defines how to sample repos for verification. 

const (
	// SampleRandom picks random repos from the corpus
	SampleRandom SampleStrategy = "random"
	// SampleTopByCommits picks repos with the most AI commits
	SampleTopByCommits SampleStrategy = "top_commits"
	// SampleRecent picks the most recently scanned repos
	SampleRecent SampleStrategy = "recent"
	// SampleClaudeCode picks repos with Claude Code activity (for canonical rule testing)
	SampleClaudeCode SampleStrategy = "claude_code"
)

type VerificationResult 

type VerificationResult struct {
	RepoID          int64
	TotalRecomputed int
	TotalPostgres   int
	MatchCount      int
	Differences     []RollupDifference
	ClaudeCodeNotes []string // Notes about Claude Code canonical rule application
}

VerificationResult contains the outcome of a verification run.

type Verifier 

type Verifier struct {
	// contains filtered or unexported fields
}

Verifier orchestrates the full verification process.

func NewVerifier 

func NewVerifier(recomputer *Recomputer, pgReader *PostgresReader, commitReader CommitReader) *Verifier

NewVerifier creates a new Verifier.

func (*Verifier) VerifyRepo 

func (v *Verifier) VerifyRepo(ctx context.Context, repoID int64) (*VerificationResult, error)

VerifyRepo verifies a single repository's rollup.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.