everef

package
v0.1.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package everef imports the datasets EVE Ref publishes at data.everef.net: insurance payouts, market history, the sovereignty map, wars, and the daily killmail archives.

These are the datasets CCP either does not expose or exposes only as a live snapshot. EVE Ref keeps the history, which is the only way to backfill a killboard that intends to go back to 2007.

Every dataset is published the same handful of ways — a latest JSON, a daily file, a yearly tar.bz2, and an Apache-style directory index to discover what exists — so the transport lives here and each importer only describes its own data.

Index

Constants

View Source
const DefaultBaseURL = "https://data.everef.net"

DefaultBaseURL is the root of every dataset.

View Source
const RegionTheForge = 10000002

RegionTheForge is the only region stored.

It contains Jita, which is the price every valuation in this codebase means. Keeping the other hundred-odd regions would multiply a 23-million-row table tenfold for data nothing reads.

Variables

View Source
var ErrNotPublished = fmt.Errorf("not published")

ErrNotPublished marks a file EVE Ref does not have.

It is a routine outcome, not a failure: the most recent day is regularly absent for several hours, and whole years are missing from some datasets. An importer reports it and moves on.

Functions

func DateRange

func DateRange(from, to string) ([]string, error)

DateRange lists every day from..to inclusive, as YYYY-MM-DD.

func DayAfter

func DayAfter(date string) (string, error)

DayAfter returns the day following date, for resuming an import one past where it stopped.

func DiscoverKillmailDays

func DiscoverKillmailDays(ctx context.Context, client *Client, from, to string) ([]string, error)

DiscoverKillmailDays lists the archives published between two dates.

EVE Ref has gaps, so the listing is authoritative rather than a generated date range — asking for a day that was never published is a wasted request and a misleading failure count.

func LatestKillmailDate

func LatestKillmailDate(ctx context.Context, pool *pgxpool.Pool) (string, error)

LatestKillmailDate reports the most recent stored kill, which is where a backfill resumes from.

func LatestPriceDate

func LatestPriceDate(ctx context.Context, pool *pgxpool.Pool) (string, error)

LatestPriceDate reports the most recent day held, which is where a backfill resumes from.

func Today

func Today() string

Today and Yesterday are UTC, which is the only timezone EVE has.

func Yesterday

func Yesterday() string

Types

type Client

type Client struct {
	// BaseURL is where the datasets live. Overridable so the importers can be
	// pointed at a fixture server; empty means DefaultBaseURL.
	BaseURL   string
	UserAgent string
	HTTP      *http.Client
}

Client fetches from data.everef.net.

func NewClient

func NewClient(userAgent string) *Client

NewClient builds a client. EVE Ref asks for an identifying user agent the same way CCP does.

func (*Client) JSON

func (c *Client) JSON(ctx context.Context, url string, out any) error

JSON fetches and decodes a JSON document, transparently decompressing a `.bz2` URL.

func (*Client) List

func (c *Client) List(ctx context.Context, url string, pattern *regexp.Regexp) ([]string, error)

List returns the hrefs in a directory index whose text matches pattern, in sorted order.

The capture group of pattern is what gets returned, so a caller asks for the date or the filename rather than the raw href.

func (*Client) Stream

func (c *Client) Stream(ctx context.Context, url string, fn func(io.Reader) error) error

Stream hands the caller the decompressed body of a `.bz2` file, which is how anything large is read: a year of market history or a day of killmails is parsed as it arrives rather than buffered whole.

func (*Client) WalkArchive

func (c *Client) WalkArchive(ctx context.Context, url string, fn func(name string, data []byte) error) error

WalkArchive calls fn for every .json member of a tar.bz2, in archive order.

fn receives the member's name and its already-read contents, because every caller needs the whole document to parse it and streaming a 2 KB file buys nothing. Returning an error from fn aborts the walk.

type KillmailImport

type KillmailImport struct {
	Pool   *pgxpool.Pool
	Client *Client
	Cache  *eve.Cache
	Prices *eve.Prices

	// SkipExisting reads the IDs already stored for the day and skips parsing
	// them. The insert ignores conflicts either way, so this only saves the
	// parse — worth one indexed query when re-running a day that is mostly
	// duplicates, wasted work when the day is mostly new.
	SkipExisting bool
}

KillmailImport carries what every day of the import needs.

func (*KillmailImport) ImportKillmailDay

func (k *KillmailImport) ImportKillmailDay(ctx context.Context, date string) (Result, error)

ImportKillmailDay imports one daily archive.

func (*KillmailImport) ImportKillmails

func (k *KillmailImport) ImportKillmails(ctx context.Context, dates []string, progress func(Result)) (Result, error)

ImportKillmails imports each of the given days.

type Result

type Result struct {
	Name string `json:"name"`
	// Seen is how many records the source offered, against which Rows shows how
	// many were new or changed. The two together are what distinguish "nothing
	// happened" from "nothing was fetched".
	Seen int64 `json:"seen,omitempty"`
	// Rows written to the importer's primary table.
	Rows int64 `json:"rows"`
	// Related rows written to a companion table: sovereignty's history log, or
	// the killmails carried inside a war archive.
	Related int64 `json:"related,omitempty"`
	// Adjusted counts rows that already existed and were corrected in place —
	// war ids filled in on killmails the live queue got to first.
	Adjusted int64  `json:"adjusted,omitempty"`
	Skipped  int64  `json:"skipped,omitempty"`
	Failed   int64  `json:"failed,omitempty"`
	Missing  bool   `json:"missing,omitempty"`
	Elapsed  string `json:"elapsed"`
}

Result reports what one import did. Every importer returns one so the CLI can render them all the same way.

func ImportInsurance

func ImportInsurance(ctx context.Context, pool *pgxpool.Pool, client *Client) (Result, error)

ImportInsurance replaces the insurance table with the published snapshot.

func ImportPriceDay

func ImportPriceDay(ctx context.Context, pool *pgxpool.Pool, client *Client, date string) (Result, error)

ImportPriceDay loads one day of market history.

A day EVE Ref has not published yet comes back with Missing set rather than an error — the most recent day is routinely absent for several hours.

func ImportPrices

func ImportPrices(ctx context.Context, pool *pgxpool.Pool, client *Client, dates []string, progress func(Result)) (Result, error)

ImportPrices loads each of the given days, reporting progress as it goes.

func ImportSovereigntyLatest

func ImportSovereigntyLatest(ctx context.Context, pool *pgxpool.Pool, client *Client) (Result, error)

ImportSovereigntyLatest applies the current published snapshot.

func ImportSovereigntyRange

func ImportSovereigntyRange(ctx context.Context, pool *pgxpool.Pool, client *Client, from, to string, progress func(Result)) (Result, error)

ImportSovereigntyRange replays every snapshot between two dates.

The yearly archives carry hourly snapshots; only the first of each day is used, which is what the TypeScript does and is enough resolution for a history that is read by the day.

type WarImport

type WarImport struct {
	Pool   *pgxpool.Pool
	Client *Client
	Cache  *eve.Cache
	Prices *eve.Prices

	// SkipKillmails imports war metadata only. The killmails in these archives
	// are largely the same ones the daily killmail archives carry, so skipping
	// them is a reasonable way to refresh war records quickly.
	SkipKillmails bool
}

WarImport carries what a war import needs.

func (*WarImport) ImportCurrentWars

func (w *WarImport) ImportCurrentWars(ctx context.Context) (Result, error)

ImportCurrentWars applies the active-war snapshot. It carries no killmails.

func (*WarImport) ImportWarArchive

func (w *WarImport) ImportWarArchive(ctx context.Context, url, name string) (Result, error)

ImportWarArchive imports one archive, whether yearly or daily.

func (*WarImport) ImportWarYears

func (w *WarImport) ImportWarYears(ctx context.Context, fromYear, toYear int, reprocess bool, progress func(Result)) (Result, error)

ImportWarYears imports every archive covering the given years.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL