Documentation
¶
Overview ¶
Package batchaccounting settles usage and cost for provider batch jobs after they complete, out of band from the original request.
Two stores, by design ¶
A batch job's lifecycle is split across two stores whose lifecycles are deliberately opposite:
- Coordination state (BatchJobStore, backed by the config store) is a mutable state machine — pending → processing → accounted/unpriceable/error — that is UPDATE-d in place many times (poll rescheduling, ownership claims, settlement markers). It belongs in the relational config store alongside the sidekiq and dlock tables, not in the append-only log store (whose ClickHouse backend is a poor fit for in-place mutation).
- The cost record (AggregateLogStore, backed by the log store) is written once, append-only, as a single aggregate row in the logs table — the durable financial artifact, sitting next to every other request cost row.
Because the two stores may be different physical databases, the aggregate-log write and the state markers are not transactional together; idempotency instead relies on CreateIfNotExists plus the AggregateLogWrittenAt / GovernanceReportedAt markers, so a retry after a partial settlement resumes rather than redoing work.
Settlement is at-least-once, not exactly-once ¶
The aggregate cost log is genuinely idempotent: CreateIfNotExists keys on a deterministic id, so replaying it is a no-op.
Governance reporting is not, and the gap is deliberate. If ReportBatchUsage succeeds but the GovernanceReportedAt marker write then fails, the job is left retryable with the report already applied. Two things narrow this, neither closes it:
- The reporter is expected to dedupe on BatchUsageReport.RequestID (a stable per-batch id). The governance plugin does, but only within one process.
- Governance budget bumps are in-memory deltas flushed periodically, so a crash inside the window usually loses the bump too, making the replay a correction rather than a double-count.
The uncovered case is a marker failure WITHOUT a crash: the original node flushes its delta, and a later sweep on a DIFFERENT node retries with an empty dedupe set, double-counting that batch against one budget. This is accepted: the window requires a marker write to fail, the impact is bounded to one batch, and the synchronous request path alongside it is strictly less durable (it has no marker at all and simply under-counts on any crash). Closing it properly needs either a durable dedupe key persisted with the usage mutation, or marker-before-bump ordering (which trades double-count for under-count).
Governance sees a batch's usage only at settlement: cost recalculation is reporting-only and never re-reports to budgets, the same platform-wide invariant that holds for recalculating any ordinary request row.
Ownership fencing ¶
Delayed accounting can run concurrently across nodes (the sweeper on one node, a user-triggered /results call on another). ClaimBatchJob transitions a job to "processing" under a runner id; every subsequent advance/complete is fenced on that runner id, and a job stuck in "processing" past a stale threshold can be re-claimed. This mirrors the sidekiq job runner rather than using a separate claim token.
Entry points ¶
- AccountBatchResults settles one completed batch: claim → price → write the aggregate cost log → report governance usage → complete. It is idempotent and safe to call from both the sweeper and request post-hooks.
- Sweeper polls due jobs (ListDueBatchJobs), retrieves provider status, and invokes AccountBatchResults once a batch completes, rescheduling with capped retries and deterministic backoff/jitter until then.
Index ¶
- Constants
- func AccountingLogID(provider schemas.ModelProvider, batchID string) string
- func BatchRequestType(endpoint schemas.BatchEndpoint) schemas.RequestType
- func IsProviderSupported(provider schemas.ModelProvider) bool
- type AggregateLogEmitter
- type AggregateLogStore
- type BatchJobStore
- type BatchModelUsage
- type BatchResultFetcher
- type BatchUsageReport
- type ModelBreakdown
- type PricingManager
- type Request
- type Summary
- type SweepStore
- type Sweeper
- type SweeperConfig
- type UsageReporter
Constants ¶
const ( UnpriceableReasonNoResults = "no_results" UnpriceableReasonNoUsage = "no_usage" UnpriceableReasonMissingModel = "missing_model" UnpriceableReasonMissingBatchPricing = "missing_batch_pricing" UnpriceableReasonMaxPollAttempts = "max_poll_attempts" UnpriceableReasonResultParseErrors = "result_parse_errors" )
Variables ¶
This section is empty.
Functions ¶
func AccountingLogID ¶
func AccountingLogID(provider schemas.ModelProvider, batchID string) string
AccountingLogID derives the id of the aggregate cost log for a batch.
The id is deterministic by design: CreateIfNotExists keys on it so a replayed settlement is a no-op, and it is reused as BatchUsageReport.RequestID so the governance reporter can dedupe. UUIDv5 gives that determinism while keeping the id a plain UUID like every other log id — ids travel back to the store through URL path parameters, and anything needing percent-encoding does not survive that round trip cleanly.
func BatchRequestType ¶
func BatchRequestType(endpoint schemas.BatchEndpoint) schemas.RequestType
BatchRequestType maps a batch endpoint to the request type its results price under. Exported because the aggregate row records only the endpoint (its Object column is always "batch_results"), so a later repricing pass outside this package must reach the same modality settlement used.
func IsProviderSupported ¶
func IsProviderSupported(provider schemas.ModelProvider) bool
Types ¶
type AggregateLogEmitter ¶
type AggregateLogStore ¶
type AggregateLogStore interface {
CreateIfNotExists(ctx context.Context, entry *logstore.Log) error
// FindByID reads the aggregate cost row back, keyed by the same deterministic
// id CreateIfNotExists wrote it under. Used only to mirror an already-settled
// batch's price onto a caller that did not win this call's settlement claim
// (see the !claimed branch in AccountBatchResults) — never to drive a write.
FindByID(ctx context.Context, id string) (*logstore.Log, error)
}
AggregateLogStore writes the append-only aggregate batch cost record. It is satisfied by logstore.LogStore — the cost record lives in the log store next to every other request cost row.
type BatchJobStore ¶
type BatchJobStore interface {
UpsertBatchJob(ctx context.Context, job *cstables.TableBatchJob) error
GetBatchJob(ctx context.Context, jobID string) (*cstables.TableBatchJob, error)
// ClaimBatchJob fences the job on runnerID. allowUnpriceable relaxes the
// terminal-state guard to admit "unpriceable" jobs: that state means "stop
// polling", not "refuse money", so a caller that actually holds results may
// re-drive one. Only settlement paths with results in hand pass true — the
// sweeper's poll loop must keep skipping unpriceable jobs (it does, via
// ListDueBatchJobs). "accounted" is terminal forever either way.
ClaimBatchJob(ctx context.Context, jobID, runnerID string, staleBefore time.Time, allowUnpriceable bool) (bool, error)
MarkBatchJobAggregateLogWritten(ctx context.Context, jobID, runnerID string) error
MarkBatchJobGovernanceReported(ctx context.Context, jobID, runnerID string) error
CompleteBatchJob(ctx context.Context, jobID, runnerID string) error
MarkBatchJobUnpriceable(ctx context.Context, jobID, runnerID, reason string, err error) error
FailBatchJob(ctx context.Context, jobID, runnerID string, err error) error
}
BatchJobStore is the mutable coordination-state store for delayed batch accounting. It is satisfied by configstore.ConfigStore (the batch lifecycle is a state machine, which belongs in the relational config store rather than the append-only log store). Ownership is fenced on a runner id.
type BatchModelUsage ¶
BatchModelUsage is one model's share of a settled batch.
type BatchResultFetcher ¶
type BatchResultFetcher interface {
RetrieveBatch(ctx context.Context, job *cstables.TableBatchJob) (*schemas.BifrostBatchRetrieveResponse, error)
FetchBatchResults(ctx context.Context, job *cstables.TableBatchJob) (*schemas.BifrostBatchResultsResponse, error)
}
type BatchUsageReport ¶
type ModelBreakdown ¶
type ModelBreakdown = schemas.BatchModelBreakdown
ModelBreakdown is the per-model slice of a settled batch's usage and cost.
It is an alias rather than a local type because the breakdown is persisted on the aggregate log row (schemas.BifrostBatchDebug), and logstore cannot import this package — batchaccounting already imports logstore.
type PricingManager ¶
type PricingManager interface {
CalculateBatchCostDetailsForUsage(usage *schemas.BifrostLLMUsage, provider schemas.ModelProvider, model string, requestType schemas.RequestType, scopes *modelcatalog.PricingLookupScopes) modelcatalog.BatchCostDetails
}
type Request ¶
type Request struct {
Provider schemas.ModelProvider
BatchID string
FallbackModel string
Endpoint schemas.BatchEndpoint
Results []schemas.BatchResultItem
ParseErrors []schemas.BatchError
RequestCounts *schemas.BatchRequestCounts
BatchJob *cstables.TableBatchJob
BaseLog *logstore.Log
SourceLog *logstore.Log
Emitter AggregateLogEmitter
UsageReporter UsageReporter
ClaimedBy string
Scopes *modelcatalog.PricingLookupScopes
Now time.Time
}
type Summary ¶
type Summary struct {
JobID string `json:"job_id"`
LogID string `json:"log_id"`
Provider schemas.ModelProvider `json:"provider"`
BatchID string `json:"batch_id"`
Cost float64 `json:"cost"`
Usage schemas.BifrostLLMUsage `json:"usage"`
ModelBreakdowns map[string]ModelBreakdown `json:"model_breakdowns"`
PricedCount int `json:"priced_count"`
UnpricedCount int `json:"unpriced_count"`
FailedCount int `json:"failed_count"`
// UnpricedModel names the model behind wholly-unpriced usage when it is
// unambiguous, so the logged row is attributable (and therefore backfillable).
// Empty when the usage spans several models or carried no model at all.
UnpricedModel string `json:"unpriced_model,omitempty"`
// Status is the provider's batch lifecycle status as of this settlement attempt
// (from req.BatchJob.ProviderStatus when claimed, or mirrored from the
// already-written aggregate row otherwise). See BifrostBatchDebug.Status.
Status string `json:"status,omitempty"`
// Complete reports that every usage-bearing row priced, so Cost is the batch's
// real total. When false the batch consumed tokens we could not price and Cost
// is only the known part — it must never be persisted as the row's cost, and
// governance must not be told the batch settled for it.
Complete bool `json:"complete"`
Accounted bool `json:"accounted"`
Claimed bool `json:"claimed"`
UnpriceableReason string `json:"unpriceable_reason,omitempty"`
}
func AccountBatchResults ¶
func AccountBatchResults(ctx context.Context, stateStore BatchJobStore, logStore AggregateLogStore, pricing PricingManager, req Request) (*Summary, error)
AccountBatchResults settles a completed batch: it advances the coordination state in stateStore (configstore) and writes the append-only aggregate cost row via logStore (logstore). Ownership is fenced on req.ClaimedBy (the runner id).
type SweepStore ¶
type SweepStore interface {
BatchJobStore
ListDueBatchJobs(ctx context.Context, provider string, now time.Time, limit int) ([]*cstables.TableBatchJob, error)
}
type Sweeper ¶
type Sweeper struct {
// contains filtered or unexported fields
}
func NewSweeper ¶
func NewSweeper(store SweepStore, logStore AggregateLogStore, pricing PricingManager, fetcher BatchResultFetcher, emitter AggregateLogEmitter, usageReporter UsageReporter, config SweeperConfig) *Sweeper
type SweeperConfig ¶
type SweeperConfig struct {
Interval time.Duration
Limit int
ClaimedBy string
Provider schemas.ModelProvider
Scopes *modelcatalog.PricingLookupScopes
KVStore schemas.KVStore
KVLeaseTTL time.Duration
Logger schemas.Logger
}
type UsageReporter ¶
type UsageReporter interface {
ReportBatchUsage(ctx context.Context, usage BatchUsageReport) error
}
UsageReporter receives the settled usage/cost for a batch so it can be billed.
Implementations MUST be idempotent on BatchUsageReport.RequestID: settlement is at-least-once, so the same report can be delivered more than once when the durable "reported" marker fails to persist after a successful report. See the package doc for the exact window and its limits.