zkb

package
v0.1.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package zkb reads zKillboard's R2Z2 feed.

R2Z2 is a sequential firehose: every killmail zKillboard receives is assigned a monotonically increasing sequence number and published at a URL derived from it. A consumer keeps a cursor, asks for cursor+1, and gets either the killmail or a 404 meaning "not yet". There is no long-poll and no websocket; the 404 is the backpressure.

The important property, and the reason this path exists at all next to the ESI killmail fetcher, is that R2Z2 embeds the full ESI killmail document in its response. A kill arriving this way costs one request and nothing from the ESI budget — no hash lookup, no /killmails/{id}/{hash}/ round trip, no error limit exposure. The ESI fetcher remains for backfill, where only an id and a hash are known.

Index

Constants

View Source
const DefaultBaseURL = "https://r2z2.zkillboard.com/ephemeral"

DefaultBaseURL is the ephemeral feed. "Ephemeral" is literal: entries are dropped after a retention window measured in hours, so a consumer that falls far enough behind can no longer catch up by following the sequence and has to be repaired from the history endpoint instead.

View Source
const DefaultHistoryURL = "https://r2z2.zkillboard.com/history"

DefaultHistoryURL serves one JSON object of id → hash per day, going back far enough to repair a multi-day outage.

View Source
const IngestWindowHours = 24

IngestWindowHours is how much history the counters keep.

View Source
const RateLimitPerSecond = 10

RateLimitPerSecond is what we allow ourselves. zKillboard permits 20; staying at half of that leaves room for the history cron and any ad-hoc repair running at the same time without the three of them together tripping a 429.

View Source
const RequestTimeout = 10 * time.Second

RequestTimeout bounds a single request. R2Z2 responds in milliseconds when healthy, so a request still open after ten seconds is not going to be answered usefully.

View Source
const SaveInterval = 50

SaveInterval is how many sequences pass between cursor writes.

The cursor is a performance optimisation, not a correctness mechanism: losing it costs a replay of up to fifty kills, all of which are rejected downstream as duplicates. Writing every sequence would be a database round trip per killmail for no benefit.

View Source
const WaitOnCaughtUp = 6 * time.Second

WaitOnCaughtUp is how long to pause after a 404. zKillboard's own guidance is six seconds; going faster earns a 429 and buys nothing, because the feed genuinely has nothing to hand over.

View Source
const WaitOnThrottled = 30 * time.Second

WaitOnThrottled is the penalty pause after a 429 or 403.

Variables

View Source
var ErrNotPublished = errors.New("sequence not published")

ErrNotPublished means the sequence number has not been filled yet — the consumer has caught up with the feed. Routine, and the signal to wait.

View Source
var ErrThrottled = errors.New("throttled by R2Z2")

ErrThrottled means R2Z2 asked us to slow down (429) or rejected the request outright (403, which in practice means a missing or unacceptable User-Agent). Both call for a long pause rather than the short catch-up wait.

Functions

func RecordIngest

func RecordIngest(ctx context.Context, rdb *redis.Client, kind IngestKind) error

RecordIngest increments the current hour's counter.

Errors are returned but every caller ignores them: a counter is telemetry, and failing to record one must never interrupt the ingestion it measures.

Types

type Client

type Client struct {
	BaseURL    string
	HistoryURL string
	UserAgent  string
	HTTP       *http.Client
	Limiter    Limiter
}

Client talks to R2Z2.

func New

func New(userAgent string) *Client

New builds a client with production defaults.

func (*Client) History

func (c *Client) History(ctx context.Context, day string) (map[int64]string, error)

History returns the id → hash map for one UTC day, keyed as R2Z2 spells it: YYYYMMDD with no separators.

This is the repair path. The ephemeral feed only retains hours, so anything missed during a longer outage is recovered from here, by id and hash, and refetched through ESI.

func (*Client) Killmail

func (c *Client) Killmail(ctx context.Context, sequence int64) (*Response, error)

Killmail fetches one sequence entry.

Returns ErrNotPublished when the entry does not exist yet, which is the normal steady state of a caught-up consumer rather than a failure.

func (*Client) LatestSequence

func (c *Client) LatestSequence(ctx context.Context) (int64, error)

LatestSequence reports the newest sequence number R2Z2 has published. Used once, to bootstrap a consumer that has no stored cursor.

func (*Client) Totals

func (c *Client) Totals(ctx context.Context) (map[string]int64, error)

Totals returns the killmail count R2Z2 holds for each available day, keyed YYYYMMDD.

This is the index of what history exists. A backfill reads it first to learn which days can be asked for at all, rather than walking the calendar and taking a 404 for every day before the archive begins — each of which would cost a request and tell it nothing.

type Event

type Event struct {
	Sequence   int64
	KillmailID int64
	Hash       string

	// Kind is "new", "repost", "caught-up", or "error".
	Kind string
	Err  error
}

Event is one thing the listener did, reported for display.

type IngestKind

type IngestKind string

IngestKind is what happened to a killmail from the feed.

const (
	// IngestNew is a killmail we did not already have.
	IngestNew IngestKind = "new"
	// IngestRepost is one we did — routine, since R2Z2 re-publishes during
	// backfeed storms and the same kill also arrives from the ESI backfill.
	IngestRepost IngestKind = "repost"
)

type IngestStats

type IngestStats struct {
	WindowHours int `json:"window_hours"`

	// Hourly counts, oldest first, always WindowHours long — an hour with no
	// traffic is a zero rather than a gap, so the series can be charted
	// directly.
	NewHourly    []int64 `json:"new_hourly"`
	RepostHourly []int64 `json:"repost_hourly"`

	NewTotal    int64 `json:"new_total"`
	RepostTotal int64 `json:"repost_total"`
}

IngestStats is the throughput over the window.

func ReadIngestStats

func ReadIngestStats(ctx context.Context, rdb *redis.Client) (*IngestStats, error)

ReadIngestStats returns the throughput over the window.

type Limiter

type Limiter interface {
	Wait(ctx context.Context) error
}

Limiter paces outbound requests.

An interface so tests can run the listener at full speed without either sleeping for real or having the client silently skip its own rate limiting — the limiter is exercised in both cases, just with time supplied differently.

type Listener

type Listener struct {
	Client *Client
	Store  Store

	// OnEvent is called for every entry. Optional.
	OnEvent func(Event)

	// Counter records ingest throughput for the status page. Optional; a nil
	// client makes RecordIngest a no-op.
	Counter *redis.Client

	// Sleep is the pause between iterations, injectable so tests do not spend
	// six real seconds per catch-up.
	Sleep func(context.Context, time.Duration) error
	// contains filtered or unexported fields
}

Listener follows the feed.

func (*Listener) Start

func (l *Listener) Start(ctx context.Context) (Stats, error)

Start bootstraps the cursor and follows the feed until the context is cancelled, which is the only way it stops — every error is a pause, not an exit. A killmail feed that dies on the first transient failure is a feed that needs a babysitter.

Returns the final stats and ctx.Err().

func (*Listener) Stats

func (l *Listener) Stats() Stats

Stats returns a copy of the running tally.

type PostgresStore

type PostgresStore struct {
	Pool    *pgxpool.Pool
	Deliver func(ctx context.Context, r *Response) error
}

PostgresStore is the production Store.

Deliver is a function rather than a method so this package does not depend on the queue: `work:zkb` hands it a River insert, a verification command hands it one that parses and stores inline, and neither arrangement requires the listener to know which it is talking to.

func (*PostgresStore) Accept

func (s *PostgresStore) Accept(ctx context.Context, r *Response) error

Accept hands the killmail on.

func (*PostgresStore) Cursor

func (s *PostgresStore) Cursor(ctx context.Context) (int64, error)

Cursor reads the stored sequence.

The key is the one the TypeScript listener uses, and that is deliberate: the Go listener has to resume where the process it replaces stopped, not restart at the head of the feed and leave a gap behind it.

func (*PostgresStore) Has

func (s *PostgresStore) Has(ctx context.Context, killmailID int64) (bool, error)

Has reports whether the killmail is already stored.

func (*PostgresStore) SaveCursor

func (s *PostgresStore) SaveCursor(ctx context.Context, sequence int64) error

SaveCursor persists the sequence.

type Response

type Response struct {
	KillmailID int64 `json:"killmail_id"`

	// Hash appears both at the top level and inside ZKB. They have always
	// agreed; Hash reports whichever is present, preferring the top-level one.
	Hash string `json:"hash"`

	ESI        killmail.ESIKillmail `json:"esi"`
	ZKB        ZKB                  `json:"zkb"`
	UploadedAt int64                `json:"uploaded_at"`
	SequenceID int64                `json:"sequence_id"`
}

Response is one entry in the feed.

func (*Response) KillmailHash

func (r *Response) KillmailHash() string

KillmailHash returns the hash to store, tolerating either placement.

type Stats

type Stats struct {
	Accepted int64 `json:"accepted"`
	Reposts  int64 `json:"reposts"`
	CaughtUp int64 `json:"caught_up"`
	Errors   int64 `json:"errors"`

	Sequence  int64 `json:"sequence"`
	StartedAt int64 `json:"started_at"`
}

Stats is a running tally.

type Store

type Store interface {
	// Cursor reads the stored sequence, returning 0 when there is none.
	Cursor(ctx context.Context) (int64, error)

	// SaveCursor persists the sequence.
	SaveCursor(ctx context.Context, sequence int64) error

	// Has reports whether the killmail is already stored. R2Z2 re-publishes
	// mails during backfeed storms, and the same kill also arrives from the
	// ESI backfill path, so this is a routine hit rather than an anomaly.
	Has(ctx context.Context, killmailID int64) (bool, error)

	// Accept takes delivery of a killmail that is not yet stored.
	Accept(ctx context.Context, r *Response) error
}

Store is what the listener needs from the rest of the system.

Kept this narrow deliberately — the listener has no opinion about whether a killmail is enqueued for a worker or processed on the spot, and tests supply an implementation that does neither.

type WindowLimiter

type WindowLimiter struct {
	// contains filtered or unexported fields
}

WindowLimiter allows at most n requests per sliding window.

It reproduces the TypeScript client's behaviour rather than using a token bucket: timestamps of recent requests are kept, anything older than the window is discarded, and a caller that would exceed the limit waits until the oldest timestamp ages out. The difference from a bucket matters at the boundary — a bucket refilling at 10/s permits a smooth 10 per second, whereas this permits 10 immediately and then nothing until the window slides. R2Z2 counts requests per second, so matching its own accounting is the safer reading.

The clock is injectable so tests can assert the pacing without spending real seconds doing it.

func NewWindowLimiter

func NewWindowLimiter(limit int, window time.Duration) *WindowLimiter

NewWindowLimiter builds a limiter permitting limit requests per window.

func (*WindowLimiter) Wait

func (l *WindowLimiter) Wait(ctx context.Context) error

Wait blocks until another request is permitted.

type ZKB

type ZKB struct {
	Hash string `json:"hash"`

	LocationID     int64    `json:"locationID"`
	FittedValue    float64  `json:"fittedValue"`
	DroppedValue   float64  `json:"droppedValue"`
	DestroyedValue float64  `json:"destroyedValue"`
	TotalValue     float64  `json:"totalValue"`
	Points         int32    `json:"points"`
	NPC            bool     `json:"npc"`
	Solo           bool     `json:"solo"`
	Awox           bool     `json:"awox"`
	Labels         []string `json:"labels"`
	AttackerCount  int32    `json:"attackerCount"`
	Href           string   `json:"href"`
}

ZKB is zKillboard's own metadata block.

Only Hash is load-bearing: it is the killmail hash, which the embedded ESI document does not carry and which the killmails table stores. Everything else is zKillboard's computed view of the kill — their ISK values, their points, their NPC and solo determination — and is deliberately not used. We compute all of it ourselves from the same ESI document, against our own price data, and storing theirs alongside ours would leave two disagreeing answers with no rule for which wins. It is decoded so the shape is documented and so a diagnostic can compare the two when a valuation looks wrong.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL