dataapi

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2026 License: MIT Imports: 35 Imported by: 0

Documentation

Overview

Package dataapi is Product A's HTTP face: the deterministic, no-LLM data endpoints (community-fit search, fit detail / stats / suggestions, SDE item search and the opt-in raw tool API) behind a stdlib http.ServeMux.

It is a library, not a binary: cmd/dataapi serves it standalone on /v1/, and services/native mounts the same handlers under its legacy /api/ paths (the Prefix option) so the web UI and the desktop app keep working unchanged.

Contract (every route):

  • Errors are JSON, always: {"error":{"code":"...","message":"..."}}. Codes are stable: not_found, method_not_allowed, invalid_request, payload_too_large, rate_limited, unavailable, upstream_error, unknown_tool, tool_disabled, api_key_required, invalid_api_key, key_not_permitted, janice_key_required, timeout, internal_error.
  • Every response carries X-Request-ID: the caller's value when it is a safe token ([A-Za-z0-9._-], 1-64 chars), otherwise a generated one. The same ID is attached to every log line the request produces.
  • Per-caller rate limits (sliding window, one limiter per route). The caller is named by a KeyResolver; the default keys an authenticated caller on its API key id and an anonymous one on the client IP, taken from Cloudflare's CF-Connecting-IP header and falling back to the TCP peer, never from X-Forwarded-For / X-Real-IP. A refused request is a 429 with a Retry-After header. CF-Connecting-IP is trusted as-is, so the service must only be reachable through the Cloudflare tunnel (or loopback).
  • Request bodies are capped (256 KiB for the fit routes, 64 KiB for the tool API); an oversize body is a 413.
  • Tool tiers (R3.4, core/catalog): on an internet-facing service a tool is public (anyone), keyed (a valid API key, from Config.Auth: Authorization: Bearer or X-API-Key), byo-key (appraise_items: the caller's own Janice key in X-Janice-Key, used for that request only and never logged or stored) or disabled. The same policy decides the REST tool endpoint and the MCP tool list per request.
  • The tool API answers a JSON envelope (Config.ToolFormat), except where a legacy mount asks for the bare text.
  • Optional machine-readable surfaces, all generated from the same route table and tool registry (core/catalog): Config.Docs serves GET {prefix}/openapi.yaml and openapi.json plus /llms.txt and /llms-full.txt at the root of the host; Config.MCP mounts an MCP streamable-HTTP handler at {prefix}/mcp behind the same request IDs and limiter. Both are off by default: cmd/dataapi turns them on, the native "/api" mount does not (its paths and tool responses are legacy, and the root-level files are not its to serve). Config.Docs also serves the human/developer index at GET / (HTML, or JSON for Accept: application/json) and the service terms at GET /terms (HTML) and /terms.md.
  • Cache-Control: descriptions, index and terms are public for an hour, pure-SDE reads (items search) for five minutes, and /health, every POST and the MCP endpoint are no-store. Only a 200 is cacheable; every error is no-store.

Dependencies arrive through Config; the package has no globals and imports only Product A packages (enforced by core/internal/tools/importcheck).

Index

Constants

View Source
const (
	// HeaderAPIKey carries an API key (the alternative to "Authorization: Bearer <key>").
	HeaderAPIKey = "X-API-Key"
	// HeaderJaniceKey carries the caller's own Janice key to a byo-key tool.
	HeaderJaniceKey = catalog.HeaderJaniceKey
)
View Source
const (
	// DefaultPrefix is the route prefix of the public API.
	DefaultPrefix = "/v1"

	// DefaultToolTimeout bounds one tool execution (the budget the retired
	// cmd/publicapi used).
	DefaultToolTimeout = 30 * time.Second
)
View Source
const (
	// DefaultCorpusProbeTTL is how long a reachability verdict is reused.
	DefaultCorpusProbeTTL = 30 * time.Second
)
View Source
const HeaderRequestID = "X-Request-ID"

HeaderRequestID carries the request ID on both the request and the response.

View Source
const SpecVersion = "0.1.0"

SpecVersion is the version of the API description (OpenAPI info.version). The /v1 path is the compatibility line; bump the minor when an operation or a field is added. The build version of the service is in every tool envelope instead, so the generated file does not change with each commit.

Variables

View Source
var ErrInvalidAPIKey = errors.New("invalid API key")

ErrInvalidAPIKey is what an Authenticator returns for a key that was presented but is not known (or is malformed). A request without any key is anonymous, not an error.

Functions

func APIKeyFrom

func APIKeyFrom(r *http.Request) (key string, present bool)

APIKeyFrom extracts the API key a request presents: "Authorization: Bearer <key>" first, else the X-API-Key header. present is false when neither header is set; a header that is set but empty or too long yields present with an empty key, which no store accepts.

func ClientIP

func ClientIP(r *http.Request) string

ClientIP returns the visitor's IP without a port. It prefers Cloudflare's CF-Connecting-IP header (Cloudflare overwrites it on every request and the origin is reachable only through the tunnel, so a caller cannot forge it) and otherwise uses the TCP peer. The client-controlled X-Forwarded-For, X-Real-IP and True-Client-IP are never read: any caller could rotate them to get a fresh bucket. A malformed CF-Connecting-IP is ignored rather than used as a key.

func GeneratedFiles

func GeneratedFiles() (map[string][]byte, error)

GeneratedFiles returns the generated description files of PublicSurface, keyed by their slash-separated path relative to the core module root.

func HashKey

func HashKey(key string) string

HashKey returns the lowercase hex SHA-256 of an API key: the only form of a key a key file (or any store) holds. Generate a key with `openssl rand -hex 32` and store `printf %s "$key" | sha256sum`.

func LLMsFullTxt

func LLMsFullTxt(s Surface) string

LLMsFullTxt returns the /llms-full.txt: everything an agent needs to use the service without fetching anything else.

func LLMsTxt

func LLMsTxt(s Surface) string

LLMsTxt returns the /llms.txt of the service described by s (https://llmstxt.org): a short index of the interfaces, endpoints and tools, linking to llms-full.txt for the detail. Links are root-relative unless Surface.PublicURL is set, which makes them absolute.

func OpenAPIJSON

func OpenAPIJSON(s Surface) ([]byte, error)

OpenAPIJSON returns the OpenAPI 3.1 document of the service described by s, as JSON. It is generated from the route table, the Go types the handlers encode and the tool registry (core/catalog); nothing in it is written by hand except the prose.

func OpenAPIYAML

func OpenAPIYAML(s Surface) ([]byte, error)

OpenAPIYAML returns the same document as YAML.

func RequestID

func RequestID(ctx context.Context) string

RequestID returns the ID of the request the context belongs to ("" outside a request served by this package).

func ValidatePublicURL

func ValidatePublicURL(raw string) (string, error)

ValidatePublicURL checks the public origin of the service (Config.PublicURL, DATAAPI_PUBLIC_URL) and returns it without a trailing slash: an absolute https URL with a host and nothing else (no path, query, fragment or credentials). The empty string is valid and means "relative links".

Types

type API

type API struct {
	// contains filtered or unexported fields
}

API is the data API. It implements http.Handler.

func New

func New(cfg Config) (*API, error)

New builds the API and its routes. It fails on an invalid Prefix, an unknown mode, or a combination of options that cannot be served (see Config.Docs).

func (*API) ServeHTTP

func (a *API) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements http.Handler: request ID, API-key authentication, panic recovery and the access log around the route mux.

type Authenticator

type Authenticator interface {
	Authenticate(r *http.Request) (Caller, error)
}

Authenticator turns a request into a Caller. It returns the anonymous Caller and a nil error when the request carries no API key, ErrInvalidAPIKey when it carries one that is not accepted. Implementations must be safe for concurrent use and must never log or return the key. R3.5's key store plugs in here.

type Caller

type Caller struct {
	// ID names the key (never the key itself); empty for an anonymous caller. It is the
	// rate-limit bucket of the caller (see CallerKeys) and what a log line may say.
	ID string
	// contains filtered or unexported fields
}

Caller is who a request is from, as the tool endpoint and the MCP endpoint see it. The zero Caller is anonymous.

func CallerFrom

func CallerFrom(ctx context.Context) Caller

CallerFrom returns the Caller of the request the context belongs to: anonymous outside a request served by this package, when no Authenticator is configured, and when the presented key was invalid (see Authenticator).

func NewCaller

func NewCaller(id string, tiers ...catalog.Tier) Caller

NewCaller returns the authenticated caller id, allowed the given tiers (TierKeyed for an ordinary key).

func (Caller) Allows

func (c Caller) Allows(t catalog.Tier) bool

Allows reports whether the caller may run a tool of tier t: public and byo-key tools are open to everyone (byo-key additionally needs the caller's own Janice key), keyed tools to a key whose allowance includes TierKeyed, disabled tools to nobody.

func (Caller) Authenticated

func (c Caller) Authenticated() bool

Authenticated reports whether the caller presented a valid key.

type CallerKeys

type CallerKeys struct{}

CallerKeys names the rate-limit bucket by the authenticated key ("key:<id>"), and by client IP (ClientIP) for an anonymous caller or an invalid key, so guessing keys is limited per IP. It is the default KeyResolver.

func (CallerKeys) Key

func (CallerKeys) Key(r *http.Request) string

Key implements KeyResolver.

type Config

type Config struct {
	Deps Deps

	// Prefix is the route prefix; empty means DefaultPrefix ("/v1"). It must start
	// with "/" and must not end with one. services/native sets "/api" to keep its
	// legacy paths.
	Prefix string

	// Limits are the per-route budgets; nil means DefaultLimits().
	Limits *Limits

	// Keys names the limiter bucket of a request; nil means CallerKeys (the key id of
	// an authenticated caller, else the client IP).
	Keys KeyResolver

	// Auth turns an API key into a Caller (see Authenticator, StaticKeys). Nil means no
	// keys exist: every caller is anonymous and keyed tools answer 401 api_key_required.
	// It applies to the tool endpoint and, through CallerFrom, to the MCP handler.
	Auth Authenticator

	// CorpusProbeTTL is how long /health reuses a corpus reachability verdict; zero
	// means DefaultCorpusProbeTTL. Only a Retriever that implements Pinger is probed.
	CorpusProbeTTL time.Duration

	// ToolAPI selects the raw tool endpoint mode; zero is off.
	ToolAPI ToolAPIMode

	// ToolFormat selects the body of a successful tool call; zero is the JSON
	// envelope. Errors are the JSON error shape in either format.
	ToolFormat ToolFormat

	// ToolTimeout bounds one tool execution; zero means DefaultToolTimeout.
	ToolTimeout time.Duration

	// Docs serves the generated descriptions of the API: GET {prefix}/openapi.yaml and
	// {prefix}/openapi.json, and GET /llms.txt and /llms-full.txt at the root of the
	// host (so the service must own the host's root, as cmd/dataapi does; a mount
	// under a sub-path never sees those two). They describe this service's own
	// configuration: the tool endpoint only when ToolAPI is on, the MCP endpoint only
	// when MCP is set. New fails if Docs is combined with ToolFormatText, which they do
	// not describe.
	Docs bool

	// PublicURL is the origin the service is reachable at, e.g. "https://data.eve-cyno.dev"
	// (https, no path or query; New fails otherwise). When set, the generated OpenAPI
	// `servers` entry and the links of llms.txt / llms-full.txt are absolute, which clients
	// that cannot resolve relative server URLs (ChatGPT Actions) need, and the index page
	// shows it in its examples. Empty keeps everything relative.
	PublicURL string

	// MCP, when non-nil, is mounted at {prefix}/mcp for every HTTP method, behind the
	// request ID, panic recovery, API-key authentication (an invalid key is a 401 before
	// the handler runs) and the Limits.MCP limiter. Build it with mcpserver.HTTPHandler,
	// which picks the tool set per request; give it Keyed: func(r) bool { return
	// CallerFrom(r.Context()).Allows(catalog.TierKeyed) }. Pass untyped nil, never a typed
	// nil pointer wrapped in the interface.
	MCP http.Handler

	// Logger receives the access log and handler diagnostics; every line carries
	// request_id. Nil discards.
	Logger *slog.Logger
}

Config configures New.

type Deps

type Deps struct {
	// Tools is the deterministic tool layer; Tools.SDE is the one SDE every route
	// reads (item search, hull-name resolution, EFT parsing).
	Tools *tools.Deps
	// Retriever backs /fits/search and the community frequencies of /fit/suggest.
	Retriever FitSearcher
	// Stats computes /fit/stats (core/fit/gofa).
	Stats fit.StatsProvider
}

Deps are the collaborators the handlers read. Any of them may be nil; the routes that need a missing one answer 503 `unavailable` instead of failing at startup, so a service with an SDE but no corpus still serves what it can.

Pass untyped nil, never a typed nil pointer wrapped in an interface (a nil *rag.QdrantRetriever assigned to Retriever is a non-nil interface).

func NewDeps

func NewDeps(t *tools.Deps, retr *rag.QdrantRetriever) Deps

NewDeps assembles Deps from the concrete deterministic layer the binaries build at startup (core/bootstrap): the Gofa stats engine is the tool layer's own (t.StatsEngine, over t.SDE), so /fit/stats and the fit-stat tools share one warm SDE memo, and a nil retriever stays an untyped nil interface (assigning a nil *rag.QdrantRetriever to Deps.Retriever directly would make it non-nil). Either argument may be nil; the routes that need what is missing answer 503.

type FitSearcher

type FitSearcher interface {
	SearchFits(ctx context.Context, q rag.FitSearchQuery) (rag.FitSearchResult, error)
}

FitSearcher is the community-fit corpus the search and suggestion routes read. *rag.QdrantRetriever implements it.

type IPKeys

type IPKeys struct{}

IPKeys keys every request by its client IP (see ClientIP).

func (IPKeys) Key

func (IPKeys) Key(r *http.Request) string

Key implements KeyResolver.

type KeyResolver

type KeyResolver interface {
	Key(r *http.Request) string
}

KeyResolver names the rate-limit bucket a request counts against. It is the seam for per-API-key limits: today every caller is keyed by IP (IPKeys); once keys exist (roadmap R3.5) a resolver returns e.g. "key:<id>" for an authenticated caller and falls back to IPKeys for anonymous ones. The key is opaque to the limiter; resolvers should prefix it so key and IP namespaces cannot collide.

type Limits

type Limits struct {
	FitsSearch  Rate
	FitsDetail  Rate
	FitStats    Rate
	FitSuggest  Rate
	ItemsSearch Rate
	// Tool applies only in ToolAPIPublic mode.
	Tool Rate
	// MCP counts every request to the MCP endpoint (a session is a handful of requests
	// plus one per tool call).
	MCP Rate
	// Docs covers the description routes (OpenAPI, llms.txt).
	Docs Rate
}

Limits holds the per-route budgets.

func DefaultLimits

func DefaultLimits() Limits

DefaultLimits are the budgets the endpoints had inside services/native.

type Pinger

type Pinger interface {
	Ping(ctx context.Context) error
}

Pinger is implemented by a corpus backend that can say whether it is reachable (*rag.QdrantRetriever does). A FitSearcher without Ping is assumed reachable.

type Rate

type Rate struct {
	Max    int
	Window time.Duration
}

Rate is one sliding-window budget per caller key: at most Max requests in any Window. A Max or Window <= 0 disables the limit (the zero Rate is unlimited).

type StaticKeys

type StaticKeys struct {
	// contains filtered or unexported fields
}

StaticKeys is an Authenticator over a fixed set of keys, held as SHA-256 hashes: for self-hosting and tests. Every key it knows may run the keyed tier.

func LoadKeyFile

func LoadKeyFile(path string) (*StaticKeys, error)

LoadKeyFile is ParseKeyFile over the file at path.

func ParseKeyFile

func ParseKeyFile(r io.Reader) (*StaticKeys, error)

ParseKeyFile reads a key file: one `id:sha256hex` per line (id is 1-64 characters from [A-Za-z0-9._-], the hash 64 hex digits); blank lines and lines starting with # are skipped. Anything else is an error that names the line number but never echoes the line, so a plaintext key pasted by mistake does not end up in a log. A file without a key is an error too.

func (*StaticKeys) Authenticate

func (k *StaticKeys) Authenticate(r *http.Request) (Caller, error)

Authenticate implements Authenticator.

func (*StaticKeys) Len

func (k *StaticKeys) Len() int

Len is the number of keys.

type Surface

type Surface struct {
	// Prefix is the route prefix ("" means DefaultPrefix).
	Prefix string
	// ToolAPI is the raw tool endpoint mode.
	ToolAPI ToolAPIMode
	// MCP is true when the MCP streamable-HTTP endpoint is mounted at {prefix}/mcp.
	MCP bool
	// Auth is true when the service accepts API keys, which makes the keyed tier
	// available and documented.
	Auth bool
	// Docs is true when the description routes (openapi.yaml, openapi.json, /llms.txt,
	// /llms-full.txt) are served.
	Docs bool
	// Limits are the per-route budgets stated in the documents.
	Limits Limits
	// PublicURL, when set (validated, no trailing slash), makes the OpenAPI servers entry
	// and the llms.txt links absolute. PublicSurface leaves it empty so the committed
	// generated files stay host-independent.
	PublicURL string
}

Surface says which optional parts a service exposes. The generated documents (OpenAPI, llms.txt) describe exactly the Surface they are given, and New registers exactly the routes the Surface of its Config names, from the one route table below.

func PublicSurface

func PublicSurface() Surface

PublicSurface is what the internet-facing cmd/dataapi serves with every optional part switched on, at the default prefix and default limits. The committed generated files (api/openapi.yaml, llms.txt, llms-full.txt) describe it.

type ToolAPIMode

type ToolAPIMode int

ToolAPIMode controls the raw tool endpoint POST {prefix}/tool/{name}.

const (
	// ToolAPIOff (the zero value) does not register the route at all: it answers
	// 404 like any unknown path. The endpoint runs raw tools for any caller, so it
	// is opt-in.
	ToolAPIOff ToolAPIMode = iota
	// ToolAPIPublic registers the route for an internet-facing service: the Tool
	// rate limit applies and the tool tiers are enforced (catalog.Tier): keyed tools
	// need an API key (401 api_key_required / invalid_api_key), appraise_items needs
	// the caller's own Janice key in X-Janice-Key (400 janice_key_required), disabled
	// tools answer 403 tool_disabled. The project's Janice key is never used.
	ToolAPIPublic
	// ToolAPILoopback registers the route with no limit and no tiers, for a
	// process that only serves its own user (the desktop app).
	ToolAPILoopback
)

type ToolFormat

type ToolFormat int

ToolFormat is the 200 body of the raw tool endpoint POST {prefix}/tool/{name}.

const (
	// ToolFormatEnvelope (the zero value) answers application/json:
	//
	//	{"tool": "...", "version": "...",
	//	 "result": {"text": "...", "data": {...} | null},
	//	 "attribution": [{"name": "...", "url": "...", "license": "..."}]}
	//
	// result.text is the LLM-oriented text the brain sees; result.data is the typed
	// result for the tools that have one (null for the rest); attribution names the
	// upstreams behind the answer. This is the public /v1 contract.
	ToolFormatEnvelope ToolFormat = iota
	// ToolFormatText answers the bare text, text/plain: the contract of the legacy
	// native /api/tool/{name}, which the desktop app and any script written against it
	// read as is. services/native sets it on its "/api" mount; nothing new should.
	ToolFormatText
)

type WindowLimiter

type WindowLimiter struct {
	// contains filtered or unexported fields
}

WindowLimiter is a sliding-window-log limiter: at most max events per key in any window. It is safe for concurrent use.

func NewWindowLimiter

func NewWindowLimiter(max int, window time.Duration) *WindowLimiter

NewWindowLimiter returns a limiter allowing max events per key per window.

func (*WindowLimiter) Allow

func (l *WindowLimiter) Allow(key string) (ok bool, retryAfter time.Duration)

Allow records an event for key and reports whether it is within budget. When it is not, retryAfter is how long until the oldest counted event leaves the window (always > 0). A refused event is not recorded.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL