stats

package
v0.1.84 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 3, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package stats maintains the killboard's aggregate counters.

Three tables, all keyed the same way — (entity_type, entity_id, period_type, period_start):

  • stats one row per entity per period, the headline counters
  • stats_breakdowns the same, split by a dimension (ship flown, region, who they died to) — the large one, hundreds of millions of rows
  • stats_leaderboards the top N entities per metric per period

Every killmail touches a lot of this. A single kill with fifty attackers updates the victim and fifty attackers, each across three period types, each with several breakdown dimensions — which is why the writer accumulates in memory and merges once rather than issuing an upsert per counter.

The enum values are stored as smallints and are a wire contract with the frontend and the API. They are never renumbered; the reserved gaps below are gaps on purpose.

Index

Constants

View Source
const (
	// DailyRetentionDays is how long per-day rows are kept. Every rolling
	// window the site offers is 365 days or less, so a daily row older than
	// that answers no question — and daily is by far the largest slice.
	DailyRetentionDays = 365

	// MonthlyRetentionMonths keeps a year and a half, enough for
	// year-over-year comparison with a margin.
	MonthlyRetentionMonths = 18

	// MonthlyRebuildMonths is how far back monthly rows are rebuilt. Six
	// months including the current one: older months cannot change, because
	// the daily rows behind them have been purged.
	MonthlyRebuildMonths = 5

	// BreakdownTopN caps how many dimensions survive per rolled-up period.
	// Without it a busy alliance's monthly row would carry every system it has
	// ever fought in, and the breakdown table is already the largest in the
	// database.
	BreakdownTopN = 20

	// LeaderboardTopN is how deep each ranking goes.
	LeaderboardTopN = 100
)

Retention and rollup windows.

View Source
const CatchupBatch = 5_000

CatchupBatch is how many killmails are held in memory at once.

The accumulator for a whole day of a busy killboard is large but bounded; the killmails and their attackers are not, so they are streamed in batches rather than read in one slice.

Variables

View Source
var ErrNotStored = errors.New("killmail is not stored")

ErrNotStored means the killmail the job names is not in the database.

Routine rather than exceptional: the stats job is dispatched alongside other follow-up work, and a killmail can be deleted between the dispatch and the run. Nothing to count, and nothing to retry.

Functions

func Load

func Load(ctx context.Context, pool *pgxpool.Pool, killmailID int64) (Killmail, []Attacker, error)

Load reads what the accumulator needs about one killmail.

Read back from the database rather than carried on the job so the payload is a single id — which keeps the queue small and makes a replay produce exactly the same counters as the original run, since both read the same stored row.

Types

type Accumulator

type Accumulator struct {
	Stats      map[StatsKey]*Row
	Breakdowns map[BreakdownKey]*Breakdown
}

Accumulator collects the counter updates for one or more killmails.

Reusable across a batch: accumulating a day of killmails and merging once is dramatically cheaper than merging per kill, because the same character appears on many kills and collapses to a single row.

func NewAccumulator

func NewAccumulator() *Accumulator

NewAccumulator returns an empty accumulator.

func (*Accumulator) Add

func (a *Accumulator) Add(km Killmail, attackers []Attacker)

Add folds one killmail into the accumulator.

func (*Accumulator) AddFactions added in v0.1.1

func (a *Accumulator) AddFactions(km Killmail, attackers []Attacker)

AddFactions folds only faction headline counters into the accumulator.

A faction-only backfill deliberately avoids constructing every other entity and breakdown row just to discard them. Keep this in lockstep with the faction portions of Add: one attacker-side kill per distinct faction on the mail, plus one victim-side loss.

func (*Accumulator) Empty

func (a *Accumulator) Empty() bool

Empty reports whether anything was accumulated.

func (*Accumulator) KeepEntityTypes added in v0.1.1

func (a *Accumulator) KeepEntityTypes(entityTypes []EntityType)

KeepEntityTypes removes aggregate rows outside an explicitly selected set. A nil/empty selection means all entity types, preserving the normal live and all-entity backfill behavior.

type Attacker

type Attacker struct {
	CharacterID   int32
	CorporationID int32
	AllianceID    int32
	FactionID     int32
	ShipTypeID    int32
	DamageDone    int64
	Points        int64
	FinalBlow     bool
}

Attacker is one participant.

type BackfillOptions

type BackfillOptions struct {
	FromMonth       time.Time
	ToMonth         time.Time
	DailyCutoff     time.Duration
	EntityTypes     []EntityType
	WantStats       bool
	WantBreakdowns  bool
	Reset           bool
	Reverse         bool
	SkipAggregation bool
	SkipRollup      bool
}

type BackfillResult

type BackfillResult struct {
	Months            int   `json:"months"`
	DailyMonths       int   `json:"daily_months"`
	MonthlyMonths     int   `json:"monthly_months"`
	Killmails         int64 `json:"killmails"`
	Stats             int64 `json:"stats"`
	Breakdowns        int64 `json:"breakdowns"`
	MonthlyStats      int64 `json:"monthly_stats"`
	MonthlyBreakdowns int64 `json:"monthly_breakdowns"`
	YearlyStats       int64 `json:"yearly_stats"`
	YearlyBreakdowns  int64 `json:"yearly_breakdowns"`
}

func Backfill

func Backfill(ctx context.Context, pool *pgxpool.Pool, opts BackfillOptions) (BackfillResult, error)

Backfill performs the same authoritative initial population as the TypeScript command. Old months are written directly at monthly granularity; recent months retain daily rows, then monthly and yearly rows are rebuilt from their lower-resolution source.

type Breakdown

type Breakdown struct {
	Kills        int64   `json:"kills"`
	Losses       int64   `json:"losses"`
	IskDestroyed float64 `json:"isk_destroyed"`
	IskLost      float64 `json:"isk_lost"`

	// LastKillmail records the most recent kill contributing to this
	// breakdown, which is what "last seen flying" is read from.
	LastKillmailID   int64 `json:"last_killmail_id"`
	LastKillmailTime int64 `json:"last_killmail_time"`
}

Breakdown is one dimension's counters.

func (*Breakdown) Add

func (b *Breakdown) Add(o Breakdown)

Add accumulates another breakdown, keeping the newer last-killmail marker.

type BreakdownKey

type BreakdownKey struct {
	EntityType  EntityType
	EntityID    int32
	DimCategory DimCategory
	DimID       int32
}

BreakdownKey identifies a breakdown row within one period.

type CatchupResult

type CatchupResult struct {
	Days       int   `json:"days"`
	Killmails  int64 `json:"killmails"`
	Deleted    int64 `json:"deleted"`
	Stats      int64 `json:"stats"`
	Breakdowns int64 `json:"breakdowns"`
}

CatchupResult reports what a catchup replaced.

func Catchup

func Catchup(ctx context.Context, pool *pgxpool.Pool, from, to time.Time) (CatchupResult, error)

Catchup rebuilds the daily rows for [from, to).

One day at a time, each in its own transaction. A range that fails partway leaves the days it finished correct rather than rolling back the lot — re-running covers the rest, and the days already done are replaced again to the same values.

func CatchupTargets

func CatchupTargets(
	ctx context.Context,
	pool *pgxpool.Pool,
	from, to time.Time,
	wantStats, wantBreakdowns bool,
) (CatchupResult, error)

CatchupTargets rebuilds either or both daily aggregate tables for [from, to). The selector exists for CLI parity with the TypeScript command, where operators can repair stats and breakdowns independently.

type Derived

type Derived struct {
	Efficiency       int     `json:"efficiency"`
	IskEfficiency    int     `json:"isk_efficiency"`
	SoloRatio        int     `json:"solo_ratio"`
	SoloLossRatio    int     `json:"solo_loss_ratio"`
	NPCLossRatio     int     `json:"npc_loss_ratio"`
	BlobFactor       float64 `json:"blob_factor"`
	AvgDamagePerKill int64   `json:"avg_damage_per_kill"`

	// DangerRatio is kills per loss. Zero losses with kills is reported as -1
	// rather than an infinity, because JSON has no infinity and encoding one
	// produces a document the frontend cannot parse.
	DangerRatio float64 `json:"danger_ratio"`
}

Derived are the metrics computed from a Row rather than stored.

Kept here so every consumer renders the same efficiency and the same blob factor; two endpoints computing "efficiency" slightly differently is exactly the sort of thing nobody notices until the numbers are compared.

func Derive

func Derive(r Row) Derived

Derive computes the metrics that are not stored.

type DimCategory

type DimCategory int16

DimCategory is what a breakdown row splits by.

The gaps are reserved rather than free. 20, 30 and 40–42 were character-level relationship dimensions that moved to Memgraph, where the graph shape fits better; rows may still exist for them, so the ids must not be reused for something else or old rows would be read as the new meaning.

const (
	DimShipFlown DimCategory = 0
	DimShipLost  DimCategory = 1

	DimSystem        DimCategory = 10
	DimConstellation DimCategory = 11
	DimRegion        DimCategory = 12

	DimDiesToCorporation DimCategory = 21
	DimDiesToAlliance    DimCategory = 22

	DimKilledCorporation DimCategory = 31
	DimKilledAlliance    DimCategory = 32
)

type EntityType

type EntityType int16

EntityType is what a stats row is about.

const (
	EntityCharacter     EntityType = 0
	EntityCorporation   EntityType = 1
	EntityAlliance      EntityType = 2
	EntityShip          EntityType = 3
	EntitySystem        EntityType = 4
	EntityConstellation EntityType = 5
	EntityRegion        EntityType = 6
	EntityFaction       EntityType = 7
)

func ParseEntityType added in v0.1.1

func ParseEntityType(value string) (EntityType, bool)

ParseEntityType resolves the stable name used by maintenance commands. "all" and an empty value deliberately mean no filter.

type Killmail

type Killmail struct {
	KillmailID      int64
	KillmailTime    time.Time
	SolarSystemID   int32
	ConstellationID int32
	RegionID        int32

	VictimCharacterID   int32
	VictimCorporationID int32
	VictimAllianceID    int32
	VictimFactionID     int32
	VictimShipTypeID    int32
	VictimDamageTaken   int64

	TotalValue    float64
	Points        int64
	AttackerCount int64
	IsNPC         bool
	IsSolo        bool
}

Killmail is what the accumulator needs about a kill.

Re-read from the database rather than carried on the job, so the job payload stays a single id and the handler is trivially replay-safe.

type LeaderboardMetric

type LeaderboardMetric int16

LeaderboardMetric is what a leaderboard ranks by.

const (
	MetricKills        LeaderboardMetric = 0
	MetricLosses       LeaderboardMetric = 1
	MetricIskDestroyed LeaderboardMetric = 2
	MetricIskLost      LeaderboardMetric = 3
	MetricSoloKills    LeaderboardMetric = 4
	MetricPoints       LeaderboardMetric = 5
	MetricFinalBlows   LeaderboardMetric = 6
)

type PeriodType

type PeriodType int16

PeriodType is the granularity of a row.

Daily rows are the source of truth and everything else is rebuilt from them: monthly from daily, yearly from monthly. That is why a purge of old daily rows has to run after the rebuild rather than before it.

const (
	PeriodDaily   PeriodType = 0
	PeriodMonthly PeriodType = 1
	PeriodYearly  PeriodType = 2
)

type PipelineResult

type PipelineResult struct {
	PurgedDaily       int64 `json:"purged_daily"`
	PurgedMonthly     int64 `json:"purged_monthly"`
	MonthlyStats      int64 `json:"monthly_stats"`
	MonthlyBreakdowns int64 `json:"monthly_breakdowns"`
	YearlyStats       int64 `json:"yearly_stats"`
	YearlyBreakdowns  int64 `json:"yearly_breakdowns"`
	Leaderboards      int64 `json:"leaderboards"`
}

PipelineResult reports what the nightly run did.

func RebuildPoints added in v0.1.57

func RebuildPoints(ctx context.Context, pool *pgxpool.Pool, opts RebuildPointOptions) (int64, PipelineResult, error)

RebuildPoints replaces combat points in the canonical daily/monthly source rows, then uses the normal pipeline to regenerate rollups and leaderboards.

func RunPipeline

func RunPipeline(ctx context.Context, pool *pgxpool.Pool) (PipelineResult, error)

RunPipeline rebuilds the rolled-up periods and the leaderboards.

type RebuildPointOptions added in v0.1.57

type RebuildPointOptions struct {
	FromMonth time.Time
	ToMonth   time.Time
	Workers   int
}

RebuildPointOptions controls the targeted point-only stats rebuild. Unlike a full stats reset it leaves kills, losses, ISK, damage, and breakdowns live.

type Row

type Row struct {
	Kills            int64   `json:"kills"`
	Losses           int64   `json:"losses"`
	SoloKills        int64   `json:"solo_kills"`
	SoloLosses       int64   `json:"solo_losses"`
	NPCLosses        int64   `json:"npc_losses"`
	FinalBlows       int64   `json:"final_blows"`
	Points           int64   `json:"points"`
	IskDestroyed     float64 `json:"isk_destroyed"`
	IskLost          float64 `json:"isk_lost"`
	DamageDealt      int64   `json:"damage_dealt"`
	DamageTaken      int64   `json:"damage_taken"`
	SumAttackerCount int64   `json:"sum_attacker_count"`
}

Row is the headline counters for one entity in one period.

func (*Row) Add

func (r *Row) Add(o Row)

Add accumulates another row into this one.

type StatsKey

type StatsKey struct {
	EntityType EntityType
	EntityID   int32
}

StatsKey identifies a stats row within one period.

type WriteResult

type WriteResult struct {
	Stats      int64 `json:"stats"`
	Breakdowns int64 `json:"breakdowns"`
}

WriteResult reports what a merge wrote.

func Write

func Write(ctx context.Context, pool *pgxpool.Pool, a *Accumulator, day time.Time) (WriteResult, error)

Write merges the accumulated counters for one day.

The period is always daily here. Monthly and yearly rows are rebuilt from these by the pipeline rather than written incrementally — incrementing three period types per counter would triple the write cost for aggregates nobody reads until the next day.

func WritePeriod

func WritePeriod(
	ctx context.Context,
	pool *pgxpool.Pool,
	a *Accumulator,
	periodStart time.Time,
	period PeriodType,
	wantStats bool,
	wantBreakdowns bool,
) (WriteResult, error)

WritePeriod merges an accumulator into one explicit period. It is used by the historical backfill to write old months directly as monthly rows rather than creating years of daily rows that the retention job would immediately purge.

func WritePeriodTx

func WritePeriodTx(
	ctx context.Context,
	tx pgx.Tx,
	a *Accumulator,
	periodStart time.Time,
	period PeriodType,
	wantStats bool,
	wantBreakdowns bool,
) (WriteResult, error)

WritePeriodTx is WritePeriod inside a caller-owned transaction.

The stats worker uses this so the additive counter updates and the killmail_processing ledger bit commit atomically. A River retry after the counters committed but before the bit did would otherwise count the same killmail twice.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL