Documentation
¶
Overview ¶
Package everef imports the datasets EVE Ref publishes at data.everef.net: insurance payouts, market history, the sovereignty map, wars, and the daily killmail archives.
These are the datasets CCP either does not expose or exposes only as a live snapshot. EVE Ref keeps the history, which is the only way to backfill a killboard that intends to go back to 2007.
Every dataset is published the same handful of ways — a latest JSON, a daily file, a yearly tar.bz2, and an Apache-style directory index to discover what exists — so the transport lives here and each importer only describes its own data.
Index ¶
- Constants
- Variables
- func DateRange(from, to string) ([]string, error)
- func DayAfter(date string) (string, error)
- func DiscoverKillmailDays(ctx context.Context, client *Client, from, to string) ([]string, error)
- func LatestKillmailDate(ctx context.Context, pool *pgxpool.Pool) (string, error)
- func LatestPriceDate(ctx context.Context, pool *pgxpool.Pool) (string, error)
- func Today() string
- func Yesterday() string
- type Client
- func (c *Client) JSON(ctx context.Context, url string, out any) error
- func (c *Client) List(ctx context.Context, url string, pattern *regexp.Regexp) ([]string, error)
- func (c *Client) Stream(ctx context.Context, url string, fn func(io.Reader) error) error
- func (c *Client) WalkArchive(ctx context.Context, url string, fn func(name string, data []byte) error) error
- type KillmailImport
- type Result
- func ImportInsurance(ctx context.Context, pool *pgxpool.Pool, client *Client) (Result, error)
- func ImportPriceDay(ctx context.Context, pool *pgxpool.Pool, client *Client, date string) (Result, error)
- func ImportPrices(ctx context.Context, pool *pgxpool.Pool, client *Client, dates []string, ...) (Result, error)
- func ImportSovereigntyLatest(ctx context.Context, pool *pgxpool.Pool, client *Client) (Result, error)
- func ImportSovereigntyRange(ctx context.Context, pool *pgxpool.Pool, client *Client, from, to string, ...) (Result, error)
- type WarImport
Constants ¶
const DefaultBaseURL = "https://data.everef.net"
DefaultBaseURL is the root of every dataset.
const RegionTheForge = 10000002
RegionTheForge is the only region stored.
It contains Jita, which is the price every valuation in this codebase means. Keeping the other hundred-odd regions would multiply a 23-million-row table tenfold for data nothing reads.
Variables ¶
var ErrNotPublished = fmt.Errorf("not published")
ErrNotPublished marks a file EVE Ref does not have.
It is a routine outcome, not a failure: the most recent day is regularly absent for several hours, and whole years are missing from some datasets. An importer reports it and moves on.
Functions ¶
func DayAfter ¶
DayAfter returns the day following date, for resuming an import one past where it stopped.
func DiscoverKillmailDays ¶
DiscoverKillmailDays lists the archives published between two dates.
EVE Ref has gaps, so the listing is authoritative rather than a generated date range — asking for a day that was never published is a wasted request and a misleading failure count.
func LatestKillmailDate ¶
LatestKillmailDate reports the most recent stored kill, which is where a backfill resumes from.
func LatestPriceDate ¶
LatestPriceDate reports the most recent day held, which is where a backfill resumes from.
Types ¶
type Client ¶
type Client struct {
// BaseURL is where the datasets live. Overridable so the importers can be
// pointed at a fixture server; empty means DefaultBaseURL.
BaseURL string
UserAgent string
HTTP *http.Client
}
Client fetches from data.everef.net.
func NewClient ¶
NewClient builds a client. EVE Ref asks for an identifying user agent the same way CCP does.
func (*Client) JSON ¶
JSON fetches and decodes a JSON document, transparently decompressing a `.bz2` URL.
func (*Client) List ¶
List returns the hrefs in a directory index whose text matches pattern, in sorted order.
The capture group of pattern is what gets returned, so a caller asks for the date or the filename rather than the raw href.
func (*Client) Stream ¶
Stream hands the caller the decompressed body of a `.bz2` file, which is how anything large is read: a year of market history or a day of killmails is parsed as it arrives rather than buffered whole.
func (*Client) WalkArchive ¶
func (c *Client) WalkArchive(ctx context.Context, url string, fn func(name string, data []byte) error) error
WalkArchive calls fn for every .json member of a tar.bz2, in archive order.
fn receives the member's name and its already-read contents, because every caller needs the whole document to parse it and streaming a 2 KB file buys nothing. Returning an error from fn aborts the walk.
type KillmailImport ¶
type KillmailImport struct {
Pool *pgxpool.Pool
Client *Client
Cache *eve.Cache
Prices *eve.Prices
// SkipExisting reads the IDs already stored for the day and skips parsing
// them. The insert ignores conflicts either way, so this only saves the
// parse — worth one indexed query when re-running a day that is mostly
// duplicates, wasted work when the day is mostly new.
SkipExisting bool
}
KillmailImport carries what every day of the import needs.
func (*KillmailImport) ImportKillmailDay ¶
ImportKillmailDay imports one daily archive.
func (*KillmailImport) ImportKillmails ¶
func (k *KillmailImport) ImportKillmails(ctx context.Context, dates []string, progress func(Result)) (Result, error)
ImportKillmails imports each of the given days.
type Result ¶
type Result struct {
Name string `json:"name"`
// Seen is how many records the source offered, against which Rows shows how
// many were new or changed. The two together are what distinguish "nothing
// happened" from "nothing was fetched".
Seen int64 `json:"seen,omitempty"`
// Rows written to the importer's primary table.
Rows int64 `json:"rows"`
// Related rows written to a companion table: sovereignty's history log, or
// the killmails carried inside a war archive.
Related int64 `json:"related,omitempty"`
// Adjusted counts rows that already existed and were corrected in place —
// war ids filled in on killmails the live queue got to first.
Adjusted int64 `json:"adjusted,omitempty"`
Skipped int64 `json:"skipped,omitempty"`
Failed int64 `json:"failed,omitempty"`
Missing bool `json:"missing,omitempty"`
Elapsed string `json:"elapsed"`
}
Result reports what one import did. Every importer returns one so the CLI can render them all the same way.
func ImportInsurance ¶
ImportInsurance replaces the insurance table with the published snapshot.
func ImportPriceDay ¶
func ImportPriceDay(ctx context.Context, pool *pgxpool.Pool, client *Client, date string) (Result, error)
ImportPriceDay loads one day of market history.
A day EVE Ref has not published yet comes back with Missing set rather than an error — the most recent day is routinely absent for several hours.
func ImportPrices ¶
func ImportPrices(ctx context.Context, pool *pgxpool.Pool, client *Client, dates []string, progress func(Result)) (Result, error)
ImportPrices loads each of the given days, reporting progress as it goes.
func ImportSovereigntyLatest ¶
func ImportSovereigntyLatest(ctx context.Context, pool *pgxpool.Pool, client *Client) (Result, error)
ImportSovereigntyLatest applies the current published snapshot.
func ImportSovereigntyRange ¶
func ImportSovereigntyRange(ctx context.Context, pool *pgxpool.Pool, client *Client, from, to string, progress func(Result)) (Result, error)
ImportSovereigntyRange replays every snapshot between two dates.
The yearly archives carry hourly snapshots; only the first of each day is used, which is what the TypeScript does and is enough resolution for a history that is read by the day.
type WarImport ¶
type WarImport struct {
Pool *pgxpool.Pool
Client *Client
Cache *eve.Cache
Prices *eve.Prices
// SkipKillmails imports war metadata only. The killmails in these archives
// are largely the same ones the daily killmail archives carry, so skipping
// them is a reasonable way to refresh war records quickly.
SkipKillmails bool
}
WarImport carries what a war import needs.
func (*WarImport) ImportCurrentWars ¶
ImportCurrentWars applies the active-war snapshot. It carries no killmails.
func (*WarImport) ImportWarArchive ¶
ImportWarArchive imports one archive, whether yearly or daily.