batchaccounting

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 26, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package batchaccounting settles usage and cost for provider batch jobs after they complete, out of band from the original request.

Two stores, by design

A batch job's lifecycle is split across two stores whose lifecycles are deliberately opposite:

  • Coordination state (BatchJobStore, backed by the config store) is a mutable state machine — pending → processing → accounted/unpriceable/error — that is UPDATE-d in place many times (poll rescheduling, ownership claims, settlement markers). It belongs in the relational config store alongside the sidekiq and dlock tables, not in the append-only log store (whose ClickHouse backend is a poor fit for in-place mutation).
  • The cost record (AggregateLogStore, backed by the log store) is written once, append-only, as a single aggregate row in the logs table — the durable financial artifact, sitting next to every other request cost row.

Because the two stores may be different physical databases, the aggregate-log write and the state markers are not transactional together; idempotency instead relies on CreateIfNotExists plus the AggregateLogWrittenAt / GovernanceReportedAt markers, so a retry after a partial settlement resumes rather than redoing work.

Settlement is at-least-once, not exactly-once

The aggregate cost log is genuinely idempotent: CreateIfNotExists keys on a deterministic id, so replaying it is a no-op.

Governance reporting is not, and the gap is deliberate. If ReportBatchUsage succeeds but the GovernanceReportedAt marker write then fails, the job is left retryable with the report already applied. Two things narrow this, neither closes it:

  • The reporter is expected to dedupe on BatchUsageReport.RequestID (a stable per-batch id). The governance plugin does, but only within one process.
  • Governance budget bumps are in-memory deltas flushed periodically, so a crash inside the window usually loses the bump too, making the replay a correction rather than a double-count.

The uncovered case is a marker failure WITHOUT a crash: the original node flushes its delta, and a later sweep on a DIFFERENT node retries with an empty dedupe set, double-counting that batch against one budget. This is accepted: the window requires a marker write to fail, the impact is bounded to one batch, and the synchronous request path alongside it is strictly less durable (it has no marker at all and simply under-counts on any crash). Closing it properly needs either a durable dedupe key persisted with the usage mutation, or marker-before-bump ordering (which trades double-count for under-count).

Governance sees a batch's usage only at settlement: cost recalculation is reporting-only and never re-reports to budgets, the same platform-wide invariant that holds for recalculating any ordinary request row.

Ownership fencing

Delayed accounting can run concurrently across nodes (the sweeper on one node, a user-triggered /results call on another). ClaimBatchJob transitions a job to "processing" under a runner id; every subsequent advance/complete is fenced on that runner id, and a job stuck in "processing" past a stale threshold can be re-claimed. This mirrors the sidekiq job runner rather than using a separate claim token.

Entry points

  • AccountBatchResults settles one completed batch: claim → price → write the aggregate cost log → report governance usage → complete. It is idempotent and safe to call from both the sweeper and request post-hooks.
  • Sweeper polls due jobs (ListDueBatchJobs), retrieves provider status, and invokes AccountBatchResults once a batch completes, rescheduling with capped retries and deterministic backoff/jitter until then.

Index

Constants

View Source
const (
	UnpriceableReasonNoResults           = "no_results"
	UnpriceableReasonNoUsage             = "no_usage"
	UnpriceableReasonMissingModel        = "missing_model"
	UnpriceableReasonMissingBatchPricing = "missing_batch_pricing"
	UnpriceableReasonMaxPollAttempts     = "max_poll_attempts"
	UnpriceableReasonResultParseErrors   = "result_parse_errors"
)

Variables

This section is empty.

Functions

func AccountingLogID

func AccountingLogID(provider schemas.ModelProvider, batchID string) string

AccountingLogID derives the id of the aggregate cost log for a batch.

The id is deterministic by design: CreateIfNotExists keys on it so a replayed settlement is a no-op, and it is reused as BatchUsageReport.RequestID so the governance reporter can dedupe. UUIDv5 gives that determinism while keeping the id a plain UUID like every other log id — ids travel back to the store through URL path parameters, and anything needing percent-encoding does not survive that round trip cleanly.

func BatchRequestType

func BatchRequestType(endpoint schemas.BatchEndpoint) schemas.RequestType

BatchRequestType maps a batch endpoint to the request type its results price under. Exported because the aggregate row records only the endpoint (its Object column is always "batch_results"), so a later repricing pass outside this package must reach the same modality settlement used.

func IsProviderSupported

func IsProviderSupported(provider schemas.ModelProvider) bool

Types

type AggregateLogEmitter

type AggregateLogEmitter interface {
	EmitBatchAggregateLog(ctx context.Context, entry *logstore.Log)
}

type AggregateLogStore

type AggregateLogStore interface {
	CreateIfNotExists(ctx context.Context, entry *logstore.Log) error
	// FindByID reads the aggregate cost row back, keyed by the same deterministic
	// id CreateIfNotExists wrote it under. Used only to mirror an already-settled
	// batch's price onto a caller that did not win this call's settlement claim
	// (see the !claimed branch in AccountBatchResults) — never to drive a write.
	FindByID(ctx context.Context, id string) (*logstore.Log, error)
}

AggregateLogStore writes the append-only aggregate batch cost record. It is satisfied by logstore.LogStore — the cost record lives in the log store next to every other request cost row.

type BatchJobStore

type BatchJobStore interface {
	UpsertBatchJob(ctx context.Context, job *cstables.TableBatchJob) error
	GetBatchJob(ctx context.Context, jobID string) (*cstables.TableBatchJob, error)
	// ClaimBatchJob fences the job on runnerID. allowUnpriceable relaxes the
	// terminal-state guard to admit "unpriceable" jobs: that state means "stop
	// polling", not "refuse money", so a caller that actually holds results may
	// re-drive one. Only settlement paths with results in hand pass true — the
	// sweeper's poll loop must keep skipping unpriceable jobs (it does, via
	// ListDueBatchJobs). "accounted" is terminal forever either way.
	ClaimBatchJob(ctx context.Context, jobID, runnerID string, staleBefore time.Time, allowUnpriceable bool) (bool, error)
	MarkBatchJobAggregateLogWritten(ctx context.Context, jobID, runnerID string) error
	MarkBatchJobGovernanceReported(ctx context.Context, jobID, runnerID string) error
	CompleteBatchJob(ctx context.Context, jobID, runnerID string) error
	MarkBatchJobUnpriceable(ctx context.Context, jobID, runnerID, reason string, err error) error
	FailBatchJob(ctx context.Context, jobID, runnerID string, err error) error
}

BatchJobStore is the mutable coordination-state store for delayed batch accounting. It is satisfied by configstore.ConfigStore (the batch lifecycle is a state machine, which belongs in the relational config store rather than the append-only log store). Ownership is fenced on a runner id.

type BatchModelUsage

type BatchModelUsage struct {
	Model      string
	Cost       float64
	TokensUsed int64
}

BatchModelUsage is one model's share of a settled batch.

type BatchResultFetcher

type BatchResultFetcher interface {
	RetrieveBatch(ctx context.Context, job *cstables.TableBatchJob) (*schemas.BifrostBatchRetrieveResponse, error)
	FetchBatchResults(ctx context.Context, job *cstables.TableBatchJob) (*schemas.BifrostBatchResultsResponse, error)
}

type BatchUsageReport

type BatchUsageReport struct {
	RequestID    string
	Provider     schemas.ModelProvider
	Model        string
	Cost         float64
	TokensUsed   int64
	BudgetIDs    []string
	RateLimitIDs []string
	UserID       string
	VirtualKeyID string
	ModelUsage   []BatchModelUsage
}

type ModelBreakdown

type ModelBreakdown = schemas.BatchModelBreakdown

ModelBreakdown is the per-model slice of a settled batch's usage and cost.

It is an alias rather than a local type because the breakdown is persisted on the aggregate log row (schemas.BifrostBatchDebug), and logstore cannot import this package — batchaccounting already imports logstore.

type PricingManager

type PricingManager interface {
	CalculateBatchCostDetailsForUsage(usage *schemas.BifrostLLMUsage, provider schemas.ModelProvider, model string, requestType schemas.RequestType, scopes *modelcatalog.PricingLookupScopes) modelcatalog.BatchCostDetails
}

type Request

type Request struct {
	Provider      schemas.ModelProvider
	BatchID       string
	FallbackModel string
	Endpoint      schemas.BatchEndpoint
	Results       []schemas.BatchResultItem
	ParseErrors   []schemas.BatchError
	RequestCounts *schemas.BatchRequestCounts
	BatchJob      *cstables.TableBatchJob
	BaseLog       *logstore.Log
	SourceLog     *logstore.Log
	Emitter       AggregateLogEmitter
	UsageReporter UsageReporter
	ClaimedBy     string
	Scopes        *modelcatalog.PricingLookupScopes
	Now           time.Time
}

type Summary

type Summary struct {
	JobID           string                    `json:"job_id"`
	LogID           string                    `json:"log_id"`
	Provider        schemas.ModelProvider     `json:"provider"`
	BatchID         string                    `json:"batch_id"`
	Cost            float64                   `json:"cost"`
	Usage           schemas.BifrostLLMUsage   `json:"usage"`
	ModelBreakdowns map[string]ModelBreakdown `json:"model_breakdowns"`
	PricedCount     int                       `json:"priced_count"`
	UnpricedCount   int                       `json:"unpriced_count"`
	FailedCount     int                       `json:"failed_count"`
	// UnpricedModel names the model behind wholly-unpriced usage when it is
	// unambiguous, so the logged row is attributable (and therefore backfillable).
	// Empty when the usage spans several models or carried no model at all.
	UnpricedModel string `json:"unpriced_model,omitempty"`
	// Status is the provider's batch lifecycle status as of this settlement attempt
	// (from req.BatchJob.ProviderStatus when claimed, or mirrored from the
	// already-written aggregate row otherwise). See BifrostBatchDebug.Status.
	Status string `json:"status,omitempty"`
	// Complete reports that every usage-bearing row priced, so Cost is the batch's
	// real total. When false the batch consumed tokens we could not price and Cost
	// is only the known part — it must never be persisted as the row's cost, and
	// governance must not be told the batch settled for it.
	Complete          bool   `json:"complete"`
	Accounted         bool   `json:"accounted"`
	Claimed           bool   `json:"claimed"`
	UnpriceableReason string `json:"unpriceable_reason,omitempty"`
}

func AccountBatchResults

func AccountBatchResults(ctx context.Context, stateStore BatchJobStore, logStore AggregateLogStore, pricing PricingManager, req Request) (*Summary, error)

AccountBatchResults settles a completed batch: it advances the coordination state in stateStore (configstore) and writes the append-only aggregate cost row via logStore (logstore). Ownership is fenced on req.ClaimedBy (the runner id).

type SweepStore

type SweepStore interface {
	BatchJobStore
	ListDueBatchJobs(ctx context.Context, provider string, now time.Time, limit int) ([]*cstables.TableBatchJob, error)
}

type Sweeper

type Sweeper struct {
	// contains filtered or unexported fields
}

func NewSweeper

func NewSweeper(store SweepStore, logStore AggregateLogStore, pricing PricingManager, fetcher BatchResultFetcher, emitter AggregateLogEmitter, usageReporter UsageReporter, config SweeperConfig) *Sweeper

func (*Sweeper) Run

func (s *Sweeper) Run(ctx context.Context)

func (*Sweeper) SweepOnce

func (s *Sweeper) SweepOnce(ctx context.Context)

type SweeperConfig

type SweeperConfig struct {
	Interval   time.Duration
	Limit      int
	ClaimedBy  string
	Provider   schemas.ModelProvider
	Scopes     *modelcatalog.PricingLookupScopes
	KVStore    schemas.KVStore
	KVLeaseTTL time.Duration
	Logger     schemas.Logger
}

type UsageReporter

type UsageReporter interface {
	ReportBatchUsage(ctx context.Context, usage BatchUsageReport) error
}

UsageReporter receives the settled usage/cost for a batch so it can be billed.

Implementations MUST be idempotent on BatchUsageReport.RequestID: settlement is at-least-once, so the same report can be delivered more than once when the durable "reported" marker fails to persist after a successful report. See the package doc for the exact window and its limits.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL