Documentation
¶
Overview ¶
Package usage answers a question filex could not answer at all: what does the storage behind it cost?
Two kinds of answer, never mixed ¶
A provider that publishes usage (Backblaze B2 does, as a daily CSV) is the accurate one: it is what the invoice is computed from, it reports byte-hours rather than a sampled size, and it knows which egress was free. Most filex installs are not on such a provider — they are on Hetzner, Garage, MinIO, a plain disk — and for those filex can only meter itself, which is an estimate.
Both are useful. Blending them silently is not: a number that looks authoritative and is not is worse than no number. So every row carries its Source, and nothing in this package adds a provider row to a measured one.
Scope, and the double-count that is easy to ship ¶
B2's report has an account-level row (empty bucket) alongside the per-bucket rows. Summing them adds the same transactions twice — by exactly the amount nobody notices, since the account row is usually small. Rows therefore carry a Scope, and Totals refuses to mix the two.
Index ¶
- Constants
- Variables
- func Month(t time.Time) (from, to time.Time)
- func SeedSettings(ctx context.Context, st dbsetting.Store)
- func SettingSpecs() []dbsetting.Seeder
- func SortDays(days []Day)
- type B2
- type Cost
- type Day
- type Pricing
- type Report
- type Scope
- type Service
- type Settings
- type Source
- type StorageLookup
- type Totals
Constants ¶
const ( // SourceProvider — read from the provider's own usage report. SourceProvider = "provider" // SourceFilex — counted by filex as it moved the bytes. An estimate: // anything written to the backend by another tool is invisible to it, and // stored size is a point-in-time reading rather than byte-hours. SourceFilex = "filex" )
Source kinds. A row's Source is how it was obtained, not who stored the bytes: a B2 row read from the daily report is SourceProvider, and a B2 storage metered by filex would be SourceFilex.
const ProviderB2 = "b2"
Known providers.
Variables ¶
var ( // SettingProvider is which provider publishes the report, or "" for none. SettingProvider = dbsetting.StringSpec{ Key: "usage.provider", EnvVar: "FILEX_USAGE_PROVIDER", Default: "", Normalize: func(raw string) string { return strings.ToLower(strings.TrimSpace(raw)) }, Check: func(v string) error { switch v { case "", ProviderB2: return nil } return fmt.Errorf("unknown usage provider %q (known: %s)", v, ProviderB2) }, } // SettingReportStorage names the filex STORAGE whose root is the provider's // report bucket. // // ⚠ A storage rather than a second set of credentials, on purpose: the // operator already knows how to add one, it is already encrypted the same // way, and the report bucket is genuinely just a bucket. Make it read-only // and it cannot be written to by anything here — this package only lists // and reads. SettingReportStorage = dbsetting.StringSpec{ Key: "usage.report_storage", EnvVar: "FILEX_USAGE_REPORT_STORAGE", Default: "", } // SettingAccountID is the provider's account id. Optional for B2 — it only // names the standalone file — but it is what an operator recognises. SettingAccountID = dbsetting.StringSpec{ Key: "usage.account_id", EnvVar: "FILEX_USAGE_ACCOUNT_ID", Default: "", } // SettingPrefix is where the dated folders live inside that storage. SettingPrefix = dbsetting.StringSpec{ Key: "usage.prefix", EnvVar: "FILEX_USAGE_PREFIX", Default: "", } // SettingPricing is the price table, as JSON. It is configuration because // prices change; see Pricing. SettingPricing = dbsetting.StringSpec{ Key: "usage.pricing", EnvVar: "FILEX_USAGE_PRICING", Default: "", Check: func(v string) error { if strings.TrimSpace(v) == "" { return nil } var p Pricing if err := json.Unmarshal([]byte(v), &p); err != nil { return fmt.Errorf("pricing must be a JSON object: %w", err) } return nil }, } )
The settings this feature owns. They are deliberately a handful of plain rows rather than one blob: an operator fixing a typo in an account id should not have to re-paste a JSON document, and the admin form can validate each field where it is typed.
Functions ¶
func Month ¶
Month returns the first and last day of the month containing t, in UTC — the range a "this month so far" view asks for.
func SeedSettings ¶
SeedSettings consumes the FILEX_USAGE_* variables into the settings table, first boot only. Call once at boot, before anything resolves them.
⚠ Every spec above declared its variable from the start and nothing called this: a compose file that set them came up with an unconfigured page and no word about why, and the docs had to say the variables did not exist.
func SettingSpecs ¶
SettingSpecs is the family, in the order they are seeded at boot.
Types ¶
type B2 ¶
type B2 struct {
// Driver is an initialized storage driver whose root is the reports
// bucket. Only List/Read are used.
Driver storage.Driver
// AccountID is the B2 account the report belongs to. Optional: it is used
// to build the standalone file name when the day's folder cannot be
// listed, and is otherwise only a label.
AccountID string
// Prefix is where the dated folders live inside the bucket. Empty is the
// bucket root, which is what B2 does today; it exists so an operator who
// mirrors the reports somewhere else can still point filex at them.
Prefix string
}
B2 reads Backblaze B2's daily usage report.
What it reads ¶
B2 writes one CSV per day into a restricted bucket of its own, `b2-reports-<accountId>`, under a `YYYY-MM-DD/` folder. The bucket is read-only from the outside — filex never writes there — and it is reachable over the same S3 API filex already speaks, so this needs no B2 SDK and no new dependency: it is handed a storage.Driver pointed at that bucket.
⚠ The file is NOT always `usage.account-<id>.csv`. Backblaze documents three shapes — `usage.account-{id}.csv` for a standalone account, `usage.{resource_name}.{reporting_location}.csv` for an organization and `usage.group-{groupId}.{reportingLocation}.csv` for a group — so the reader LISTS the day's folder instead of constructing one name. An operator whose account is in an organization would otherwise get an empty report and no error, which is the worst of both.
That also means it can be exercised against any S3 server. The tests run it against a local one with byte-exact copies of B2's own rows, which measures the part that can be measured without an account: the reading, the parsing and the arithmetic.
What it deliberately does not do ¶
It does not compute money. Prices change, allowances change, and a number baked into a binary is wrong the day after it ships — pricing is configuration elsewhere, and this type produces only what B2 reported.
func (B2) Fetch ¶
Fetch implements Source. It asks for one file per day in the range.
A day B2 has not published is absent from the result rather than present as a zero row: "nothing happened that day" and "the report is not out yet" are different statements, and today's report does not exist until B2 writes it.
type Cost ¶
type Cost struct {
Currency string `json:"currency"`
Days int `json:"days"`
// Storage.
AvgStoredGB float64 `json:"avg_stored_gb"`
BillableGBMon float64 `json:"billable_gb_months"`
StorageCost float64 `json:"storage_cost"`
FreeStorageGB float64 `json:"free_storage_gb"`
StorageCovered bool `json:"storage_covered"`
// Egress.
DownloadedGB float64 `json:"downloaded_gb"`
ProviderFreeGB float64 `json:"provider_free_gb"`
AllowanceGB float64 `json:"allowance_gb"`
BillableGB float64 `json:"billable_gb"`
EgressCost float64 `json:"egress_cost"`
EgressCovered bool `json:"egress_covered"`
// Transactions.
BillableOps int64 `json:"billable_ops"`
FreeOps int64 `json:"free_ops"`
OpsCost float64 `json:"ops_cost"`
Total float64 `json:"total"`
}
Cost is an estimate, broken into the lines an invoice has, with the free part of each shown beside the billable part.
⚠ It is an ESTIMATE even when it is computed from the provider's own report: allowances are applied over the period asked for rather than over the provider's billing month, taxes and minimums are not modelled, and a provider may round differently. It exists to answer "what is this costing me, roughly, and where is it going" — not to reconcile a bill.
type Day ¶
type Day struct {
Date time.Time `json:"date"`
Provider string `json:"provider"`
Source string `json:"source"`
Scope Scope `json:"scope"`
Bucket string `json:"bucket,omitempty"`
// Location is the provider's reporting region for the row, when it reports
// one. B2 prices regions separately, so two rows for one bucket are two
// rows, not one to be added up.
Location string `json:"location,omitempty"`
// StoredBytes is the size at the end of the day, where that is all the
// provider reports.
StoredBytes int64 `json:"stored_bytes"`
// ByteHours is storage integrated over the day — what storage is actually
// billed on. 0 when the provider does not report it.
ByteHours float64 `json:"byte_hours"`
UploadedBytes int64 `json:"uploaded_bytes"`
DeletedBytes int64 `json:"deleted_bytes"`
DownloadedBytes int64 `json:"downloaded_bytes"`
// FreeEgressBytes is the part of the download that the provider did not
// charge for (B2's CDN-partner egress). 0 elsewhere.
FreeEgressBytes int64 `json:"free_egress_bytes"`
// Transactions, in the classes every object store bills in even when it
// names them differently: A = writes, B = reads, C = listings/metadata,
// D = the provider's own extra class (B2 uses it for some downloads).
OpsA int64 `json:"ops_a"`
OpsB int64 `json:"ops_b"`
OpsC int64 `json:"ops_c"`
OpsD int64 `json:"ops_d"`
}
Day is one day of usage for one bucket (or the account), normalized across providers. Everything provider-specific is resolved before a value gets here, so a consumer never branches on which provider it came from.
⚠ Zero is a real value and "not reported" is a different thing. A provider that does not report byte-hours leaves ByteHours at 0 and fills StoredBytes; a consumer that wants "how much was stored" must prefer ByteHours when it is non-zero rather than assume both are always present.
type Pricing ¶
type Pricing struct {
// Currency is a display label. No conversion is ever performed.
Currency string `json:"currency"`
// StoragePerGBMonth is charged on the average stored over the period.
StoragePerGBMonth float64 `json:"storage_per_gb_month"`
// FreeStorageGB is deducted from the average before charging.
FreeStorageGB float64 `json:"free_storage_gb"`
// EgressPerGB is charged on downloads beyond the allowances.
EgressPerGB float64 `json:"egress_per_gb"`
// FreeEgressMultiple is the allowance expressed as a multiple of stored
// data — B2 gives 3x the average stored, per month. 0 disables it.
FreeEgressMultiple float64 `json:"free_egress_multiple"`
// Transactions, per ten thousand, by class.
ClassAPer10k float64 `json:"class_a_per_10k"`
ClassBPer10k float64 `json:"class_b_per_10k"`
ClassCPer10k float64 `json:"class_c_per_10k"`
ClassDPer10k float64 `json:"class_d_per_10k"`
// FreeClassPerDay is how many transactions of each PRICED class are free
// each day. B2 applies it to class D.
FreeClassPerDay int64 `json:"free_class_per_day"`
// Note carries where these numbers came from, so the next person can
// check them instead of trusting them.
Note string `json:"note,omitempty"`
}
Pricing is what a provider charges, as configuration rather than as code.
⚠ Nothing here is baked into the binary as a fact. Prices change, allowances change, and a number compiled in is wrong the day after it ships — with no symptom, because a plausible figure looks exactly like a correct one. The defaults below carry the date they were read and are meant to be edited by the operator, who is the one holding the invoice.
func BackblazeB2 ¶
func BackblazeB2() Pricing
BackblazeB2 is a starting point, not an authority: read from backblaze.com/cloud-storage/pricing on 2026-09-11. Check it against your own invoice before believing any figure computed from it.
type Report ¶
type Report struct {
Settings Settings `json:"settings"`
// Configured is false when nothing is set up yet. The page then shows the
// form, not an empty chart that looks like "you spent nothing".
Configured bool `json:"configured"`
From string `json:"from"`
To string `json:"to"`
Days []Day `json:"days"`
Totals Totals `json:"totals"`
Cost Cost `json:"cost"`
// Notes are things the operator should know about THIS answer: a range
// with no published report yet, a cached result, a provider that reports
// nothing for some days.
Notes []string `json:"notes,omitempty"`
}
Report is everything the admin page renders.
type Scope ¶
type Scope string
Scope separates the two row shapes a provider report carries.
const ( // ScopeBucket — one bucket's usage for one day. These sum. ScopeBucket Scope = "bucket" // ScopeAccount — the account's own line, which in B2's report carries the // transactions that belong to no bucket. It must NOT be added to the // bucket rows; it is reported beside them. ScopeAccount Scope = "account" )
type Service ¶
type Service struct {
Get dbsetting.Getter
Lookup StorageLookup
TTL time.Duration
// contains filtered or unexported fields
}
Service answers the admin surface's questions, with a small cache in front of the provider so opening a page does not re-download a month of reports.
func NewService ¶
NewService constructs the service. ttl <= 0 defaults to an hour, which is far below the daily cadence of the report and far above a page refresh.
func (*Service) Invalidate ¶
func (s *Service) Invalidate()
Invalidate drops the cache — called when the settings change, so an operator who fixes an account id sees the effect on the next load rather than an hour later.
type Settings ¶
type Settings struct {
Provider string `json:"provider"`
ReportStorage string `json:"report_storage"`
AccountID string `json:"account_id"`
Prefix string `json:"prefix"`
Pricing Pricing `json:"pricing"`
// PricingIsDefault says the operator has not saved a price table, so the
// numbers are ours and carry the date we read them. The page must say so:
// an estimate computed from somebody else's guess at a price is exactly
// the kind of number that gets quoted back later as fact.
PricingIsDefault bool `json:"pricing_is_default"`
}
Settings is the resolved configuration.
func (Settings) Configured ¶
Configured reports whether there is enough to fetch anything.
type Source ¶
type Source interface {
// Name identifies the provider ("b2", "filex", …).
Name() string
// Fetch returns every day it has in [from, to], inclusive of both ends and
// in date order. A day the provider has not published yet is simply
// absent — never a zero row, which would read as "nothing happened".
Fetch(ctx context.Context, from, to time.Time) ([]Day, error)
}
Source produces normalized days for one provider account.
type StorageLookup ¶
StorageLookup resolves a storage NAME to an initialized driver. The server has one; tests pass their own.
type Totals ¶
type Totals struct {
From, To time.Time `json:"-"`
Days int `json:"days"`
// AvgStoredBytes is the mean of the daily storage readings — byte-hours
// where the provider reports them, StoredBytes otherwise.
AvgStoredBytes float64 `json:"avg_stored_bytes"`
// PeakStoredBytes is the largest single-day reading.
PeakStoredBytes int64 `json:"peak_stored_bytes"`
UploadedBytes int64 `json:"uploaded_bytes"`
DeletedBytes int64 `json:"deleted_bytes"`
DownloadedBytes int64 `json:"downloaded_bytes"`
FreeEgressBytes int64 `json:"free_egress_bytes"`
OpsA int64 `json:"ops_a"`
OpsB int64 `json:"ops_b"`
OpsC int64 `json:"ops_c"`
OpsD int64 `json:"ops_d"`
// AccountOps holds the account-scoped transactions, kept beside the bucket
// totals rather than inside them. See Scope.
AccountOps struct {
A, B, C, D int64
} `json:"account_ops"`
}
Totals is a summary over a set of days.
func SumBuckets ¶
SumBuckets totals the BUCKET rows and reports the account rows separately.
⚠ It sums only rows whose Source agrees. Handing it a mixture returns the provider rows and ignores the measured ones, because adding an invoice to an estimate produces a number that is neither.
Source Files