Documentation
¶
Overview ¶
Package zkb reads zKillboard's R2Z2 feed.
R2Z2 is a sequential firehose: every killmail zKillboard receives is assigned a monotonically increasing sequence number and published at a URL derived from it. A consumer keeps a cursor, asks for cursor+1, and gets either the killmail or a 404 meaning "not yet". There is no long-poll and no websocket; the 404 is the backpressure.
The important property, and the reason this path exists at all next to the ESI killmail fetcher, is that R2Z2 embeds the full ESI killmail document in its response. A kill arriving this way costs one request and nothing from the ESI budget — no hash lookup, no /killmails/{id}/{hash}/ round trip, no error limit exposure. The ESI fetcher remains for backfill, where only an id and a hash are known.
Index ¶
- Constants
- Variables
- func RecordIngest(ctx context.Context, rdb *redis.Client, kind IngestKind) error
- type Client
- func (c *Client) History(ctx context.Context, day string) (map[int64]string, error)
- func (c *Client) Killmail(ctx context.Context, sequence int64) (*Response, error)
- func (c *Client) LatestSequence(ctx context.Context) (int64, error)
- func (c *Client) Totals(ctx context.Context) (map[string]int64, error)
- type Event
- type IngestKind
- type IngestStats
- type Limiter
- type Listener
- type PostgresStore
- type Response
- type Stats
- type Store
- type WindowLimiter
- type ZKB
Constants ¶
const DefaultBaseURL = "https://r2z2.zkillboard.com/ephemeral"
DefaultBaseURL is the ephemeral feed. "Ephemeral" is literal: entries are dropped after a retention window measured in hours, so a consumer that falls far enough behind can no longer catch up by following the sequence and has to be repaired from the history endpoint instead.
const DefaultHistoryURL = "https://r2z2.zkillboard.com/history"
DefaultHistoryURL serves one JSON object of id → hash per day, going back far enough to repair a multi-day outage.
const IngestWindowHours = 24
IngestWindowHours is how much history the counters keep.
const RateLimitPerSecond = 10
RateLimitPerSecond is what we allow ourselves. zKillboard permits 20; staying at half of that leaves room for the history cron and any ad-hoc repair running at the same time without the three of them together tripping a 429.
const RequestTimeout = 10 * time.Second
RequestTimeout bounds a single request. R2Z2 responds in milliseconds when healthy, so a request still open after ten seconds is not going to be answered usefully.
const SaveInterval = 50
SaveInterval is how many sequences pass between cursor writes.
The cursor is a performance optimisation, not a correctness mechanism: losing it costs a replay of up to fifty kills, all of which are rejected downstream as duplicates. Writing every sequence would be a database round trip per killmail for no benefit.
const WaitOnCaughtUp = 6 * time.Second
WaitOnCaughtUp is how long to pause after a 404. zKillboard's own guidance is six seconds; going faster earns a 429 and buys nothing, because the feed genuinely has nothing to hand over.
const WaitOnThrottled = 30 * time.Second
WaitOnThrottled is the penalty pause after a 429 or 403.
Variables ¶
var ErrNotPublished = errors.New("sequence not published")
ErrNotPublished means the sequence number has not been filled yet — the consumer has caught up with the feed. Routine, and the signal to wait.
var ErrThrottled = errors.New("throttled by R2Z2")
ErrThrottled means R2Z2 asked us to slow down (429) or rejected the request outright (403, which in practice means a missing or unacceptable User-Agent). Both call for a long pause rather than the short catch-up wait.
Functions ¶
func RecordIngest ¶
RecordIngest increments the current hour's counter.
Errors are returned but every caller ignores them: a counter is telemetry, and failing to record one must never interrupt the ingestion it measures.
Types ¶
type Client ¶
type Client struct {
BaseURL string
HistoryURL string
UserAgent string
HTTP *http.Client
Limiter Limiter
}
Client talks to R2Z2.
func (*Client) History ¶
History returns the id → hash map for one UTC day, keyed as R2Z2 spells it: YYYYMMDD with no separators.
This is the repair path. The ephemeral feed only retains hours, so anything missed during a longer outage is recovered from here, by id and hash, and refetched through ESI.
func (*Client) Killmail ¶
Killmail fetches one sequence entry.
Returns ErrNotPublished when the entry does not exist yet, which is the normal steady state of a caught-up consumer rather than a failure.
func (*Client) LatestSequence ¶
LatestSequence reports the newest sequence number R2Z2 has published. Used once, to bootstrap a consumer that has no stored cursor.
func (*Client) Totals ¶
Totals returns the killmail count R2Z2 holds for each available day, keyed YYYYMMDD.
This is the index of what history exists. A backfill reads it first to learn which days can be asked for at all, rather than walking the calendar and taking a 404 for every day before the archive begins — each of which would cost a request and tell it nothing.
type Event ¶
type Event struct {
Sequence int64
KillmailID int64
Hash string
// Kind is "new", "repost", "caught-up", or "error".
Kind string
Err error
}
Event is one thing the listener did, reported for display.
type IngestKind ¶
type IngestKind string
IngestKind is what happened to a killmail from the feed.
const ( // IngestNew is a killmail we did not already have. IngestNew IngestKind = "new" // IngestRepost is one we did — routine, since R2Z2 re-publishes during // backfeed storms and the same kill also arrives from the ESI backfill. IngestRepost IngestKind = "repost" )
type IngestStats ¶
type IngestStats struct {
WindowHours int `json:"window_hours"`
// Hourly counts, oldest first, always WindowHours long — an hour with no
// traffic is a zero rather than a gap, so the series can be charted
// directly.
NewHourly []int64 `json:"new_hourly"`
RepostHourly []int64 `json:"repost_hourly"`
NewTotal int64 `json:"new_total"`
RepostTotal int64 `json:"repost_total"`
}
IngestStats is the throughput over the window.
func ReadIngestStats ¶
ReadIngestStats returns the throughput over the window.
type Limiter ¶
Limiter paces outbound requests.
An interface so tests can run the listener at full speed without either sleeping for real or having the client silently skip its own rate limiting — the limiter is exercised in both cases, just with time supplied differently.
type Listener ¶
type Listener struct {
Client *Client
Store Store
// OnEvent is called for every entry. Optional.
OnEvent func(Event)
// Counter records ingest throughput for the status page. Optional; a nil
// client makes RecordIngest a no-op.
Counter *redis.Client
// Sleep is the pause between iterations, injectable so tests do not spend
// six real seconds per catch-up.
Sleep func(context.Context, time.Duration) error
// contains filtered or unexported fields
}
Listener follows the feed.
func (*Listener) Start ¶
Start bootstraps the cursor and follows the feed until the context is cancelled, which is the only way it stops — every error is a pause, not an exit. A killmail feed that dies on the first transient failure is a feed that needs a babysitter.
Returns the final stats and ctx.Err().
type PostgresStore ¶
type PostgresStore struct {
Pool *pgxpool.Pool
Deliver func(ctx context.Context, r *Response) error
}
PostgresStore is the production Store.
Deliver is a function rather than a method so this package does not depend on the queue: `work:zkb` hands it a River insert, a verification command hands it one that parses and stores inline, and neither arrangement requires the listener to know which it is talking to.
func (*PostgresStore) Accept ¶
func (s *PostgresStore) Accept(ctx context.Context, r *Response) error
Accept hands the killmail on.
func (*PostgresStore) Cursor ¶
func (s *PostgresStore) Cursor(ctx context.Context) (int64, error)
Cursor reads the stored sequence.
The key is the one the TypeScript listener uses, and that is deliberate: the Go listener has to resume where the process it replaces stopped, not restart at the head of the feed and leave a gap behind it.
func (*PostgresStore) SaveCursor ¶
func (s *PostgresStore) SaveCursor(ctx context.Context, sequence int64) error
SaveCursor persists the sequence.
type Response ¶
type Response struct {
KillmailID int64 `json:"killmail_id"`
// Hash appears both at the top level and inside ZKB. They have always
// agreed; Hash reports whichever is present, preferring the top-level one.
Hash string `json:"hash"`
ESI killmail.ESIKillmail `json:"esi"`
ZKB ZKB `json:"zkb"`
UploadedAt int64 `json:"uploaded_at"`
SequenceID int64 `json:"sequence_id"`
}
Response is one entry in the feed.
func (*Response) KillmailHash ¶
KillmailHash returns the hash to store, tolerating either placement.
type Stats ¶
type Stats struct {
Accepted int64 `json:"accepted"`
Reposts int64 `json:"reposts"`
CaughtUp int64 `json:"caught_up"`
Errors int64 `json:"errors"`
Sequence int64 `json:"sequence"`
StartedAt int64 `json:"started_at"`
}
Stats is a running tally.
type Store ¶
type Store interface {
// Cursor reads the stored sequence, returning 0 when there is none.
Cursor(ctx context.Context) (int64, error)
// SaveCursor persists the sequence.
SaveCursor(ctx context.Context, sequence int64) error
// Has reports whether the killmail is already stored. R2Z2 re-publishes
// mails during backfeed storms, and the same kill also arrives from the
// ESI backfill path, so this is a routine hit rather than an anomaly.
Has(ctx context.Context, killmailID int64) (bool, error)
// Accept takes delivery of a killmail that is not yet stored.
Accept(ctx context.Context, r *Response) error
}
Store is what the listener needs from the rest of the system.
Kept this narrow deliberately — the listener has no opinion about whether a killmail is enqueued for a worker or processed on the spot, and tests supply an implementation that does neither.
type WindowLimiter ¶
type WindowLimiter struct {
// contains filtered or unexported fields
}
WindowLimiter allows at most n requests per sliding window.
It reproduces the TypeScript client's behaviour rather than using a token bucket: timestamps of recent requests are kept, anything older than the window is discarded, and a caller that would exceed the limit waits until the oldest timestamp ages out. The difference from a bucket matters at the boundary — a bucket refilling at 10/s permits a smooth 10 per second, whereas this permits 10 immediately and then nothing until the window slides. R2Z2 counts requests per second, so matching its own accounting is the safer reading.
The clock is injectable so tests can assert the pacing without spending real seconds doing it.
func NewWindowLimiter ¶
func NewWindowLimiter(limit int, window time.Duration) *WindowLimiter
NewWindowLimiter builds a limiter permitting limit requests per window.
type ZKB ¶
type ZKB struct {
Hash string `json:"hash"`
LocationID int64 `json:"locationID"`
FittedValue float64 `json:"fittedValue"`
DroppedValue float64 `json:"droppedValue"`
DestroyedValue float64 `json:"destroyedValue"`
TotalValue float64 `json:"totalValue"`
Points int32 `json:"points"`
NPC bool `json:"npc"`
Solo bool `json:"solo"`
Awox bool `json:"awox"`
Labels []string `json:"labels"`
AttackerCount int32 `json:"attackerCount"`
Href string `json:"href"`
}
ZKB is zKillboard's own metadata block.
Only Hash is load-bearing: it is the killmail hash, which the embedded ESI document does not carry and which the killmails table stores. Everything else is zKillboard's computed view of the kill — their ISK values, their points, their NPC and solo determination — and is deliberately not used. We compute all of it ourselves from the same ESI document, against our own price data, and storing theirs alongside ours would leave two disagreeing answers with no rule for which wins. It is decoded so the shape is documented and so a diagnostic can compare the two when a valuation looks wrong.