Documentation
¶
Overview ¶
Package dataapi is Product A's HTTP face: the deterministic, no-LLM data endpoints (community-fit search, fit detail / stats / suggestions, SDE item search and the opt-in raw tool API) behind a stdlib http.ServeMux.
It is a library, not a binary: cmd/dataapi serves it standalone on /v1/, and services/native mounts the same handlers under its legacy /api/ paths (the Prefix option) so the web UI and the desktop app keep working unchanged.
Contract (every route):
- Errors are JSON, always: {"error":{"code":"...","message":"..."}}. Codes are stable: not_found, method_not_allowed, invalid_request, payload_too_large, rate_limited, unavailable, upstream_error, unknown_tool, tool_disabled, api_key_required, invalid_api_key, key_not_permitted, janice_key_required, timeout, internal_error.
- Every response carries X-Request-ID: the caller's value when it is a safe token ([A-Za-z0-9._-], 1-64 chars), otherwise a generated one. The same ID is attached to every log line the request produces.
- Per-caller rate limits (sliding window, one limiter per route). The caller is named by a KeyResolver; the default keys an authenticated caller on its API key id and an anonymous one on the client IP, taken from Cloudflare's CF-Connecting-IP header and falling back to the TCP peer, never from X-Forwarded-For / X-Real-IP. A refused request is a 429 with a Retry-After header. CF-Connecting-IP is trusted as-is, so the service must only be reachable through the Cloudflare tunnel (or loopback).
- Request bodies are capped (256 KiB for the fit routes, 64 KiB for the tool API); an oversize body is a 413.
- Tool tiers (R3.4, core/catalog): on an internet-facing service a tool is public (anyone), keyed (a valid API key, from Config.Auth: Authorization: Bearer or X-API-Key), byo-key (appraise_items: the caller's own Janice key in X-Janice-Key, used for that request only and never logged or stored) or disabled. The same policy decides the REST tool endpoint and the MCP tool list per request.
- The tool API answers a JSON envelope (Config.ToolFormat), except where a legacy mount asks for the bare text.
- Optional machine-readable surfaces, all generated from the same route table and tool registry (core/catalog): Config.Docs serves GET {prefix}/openapi.yaml and openapi.json plus /llms.txt and /llms-full.txt at the root of the host; Config.MCP mounts an MCP streamable-HTTP handler at {prefix}/mcp behind the same request IDs and limiter. Both are off by default: cmd/dataapi turns them on, the native "/api" mount does not (its paths and tool responses are legacy, and the root-level files are not its to serve). Config.Docs also serves the human/developer index at GET / (HTML, or JSON for Accept: application/json) and the service terms at GET /terms (HTML) and /terms.md.
- Cache-Control: descriptions, index and terms are public for an hour, pure-SDE reads (items search) for five minutes, and /health, every POST and the MCP endpoint are no-store. Only a 200 is cacheable; every error is no-store.
Dependencies arrive through Config; the package has no globals and imports only Product A packages (enforced by core/internal/tools/importcheck).
Index ¶
- Constants
- Variables
- func APIKeyFrom(r *http.Request) (key string, present bool)
- func ClientIP(r *http.Request) string
- func GeneratedFiles() (map[string][]byte, error)
- func HashKey(key string) string
- func LLMsFullTxt(s Surface) string
- func LLMsTxt(s Surface) string
- func OpenAPIJSON(s Surface) ([]byte, error)
- func OpenAPIYAML(s Surface) ([]byte, error)
- func RequestID(ctx context.Context) string
- func ValidatePublicURL(raw string) (string, error)
- type API
- type Authenticator
- type Caller
- type CallerKeys
- type Config
- type Deps
- type FitSearcher
- type IPKeys
- type KeyResolver
- type Limits
- type Pinger
- type Rate
- type StaticKeys
- type Surface
- type ToolAPIMode
- type ToolFormat
- type WindowLimiter
Constants ¶
const ( // HeaderAPIKey carries an API key (the alternative to "Authorization: Bearer <key>"). HeaderAPIKey = "X-API-Key" // HeaderJaniceKey carries the caller's own Janice key to a byo-key tool. HeaderJaniceKey = catalog.HeaderJaniceKey )
const ( // DefaultPrefix is the route prefix of the public API. DefaultPrefix = "/v1" // DefaultToolTimeout bounds one tool execution (the budget the retired // cmd/publicapi used). DefaultToolTimeout = 30 * time.Second )
const ( // DefaultCorpusProbeTTL is how long a reachability verdict is reused. DefaultCorpusProbeTTL = 30 * time.Second )
const HeaderRequestID = "X-Request-ID"
HeaderRequestID carries the request ID on both the request and the response.
const SpecVersion = "0.1.0"
SpecVersion is the version of the API description (OpenAPI info.version). The /v1 path is the compatibility line; bump the minor when an operation or a field is added. The build version of the service is in every tool envelope instead, so the generated file does not change with each commit.
Variables ¶
var ErrInvalidAPIKey = errors.New("invalid API key")
ErrInvalidAPIKey is what an Authenticator returns for a key that was presented but is not known (or is malformed). A request without any key is anonymous, not an error.
Functions ¶
func APIKeyFrom ¶
APIKeyFrom extracts the API key a request presents: "Authorization: Bearer <key>" first, else the X-API-Key header. present is false when neither header is set; a header that is set but empty or too long yields present with an empty key, which no store accepts.
func ClientIP ¶
ClientIP returns the visitor's IP without a port. It prefers Cloudflare's CF-Connecting-IP header (Cloudflare overwrites it on every request and the origin is reachable only through the tunnel, so a caller cannot forge it) and otherwise uses the TCP peer. The client-controlled X-Forwarded-For, X-Real-IP and True-Client-IP are never read: any caller could rotate them to get a fresh bucket. A malformed CF-Connecting-IP is ignored rather than used as a key.
func GeneratedFiles ¶
GeneratedFiles returns the generated description files of PublicSurface, keyed by their slash-separated path relative to the core module root.
func HashKey ¶
HashKey returns the lowercase hex SHA-256 of an API key: the only form of a key a key file (or any store) holds. Generate a key with `openssl rand -hex 32` and store `printf %s "$key" | sha256sum`.
func LLMsFullTxt ¶
LLMsFullTxt returns the /llms-full.txt: everything an agent needs to use the service without fetching anything else.
func LLMsTxt ¶
LLMsTxt returns the /llms.txt of the service described by s (https://llmstxt.org): a short index of the interfaces, endpoints and tools, linking to llms-full.txt for the detail. Links are root-relative unless Surface.PublicURL is set, which makes them absolute.
func OpenAPIJSON ¶
OpenAPIJSON returns the OpenAPI 3.1 document of the service described by s, as JSON. It is generated from the route table, the Go types the handlers encode and the tool registry (core/catalog); nothing in it is written by hand except the prose.
func OpenAPIYAML ¶
OpenAPIYAML returns the same document as YAML.
func RequestID ¶
RequestID returns the ID of the request the context belongs to ("" outside a request served by this package).
func ValidatePublicURL ¶
ValidatePublicURL checks the public origin of the service (Config.PublicURL, DATAAPI_PUBLIC_URL) and returns it without a trailing slash: an absolute https URL with a host and nothing else (no path, query, fragment or credentials). The empty string is valid and means "relative links".
Types ¶
type API ¶
type API struct {
// contains filtered or unexported fields
}
API is the data API. It implements http.Handler.
type Authenticator ¶
Authenticator turns a request into a Caller. It returns the anonymous Caller and a nil error when the request carries no API key, ErrInvalidAPIKey when it carries one that is not accepted. Implementations must be safe for concurrent use and must never log or return the key. R3.5's key store plugs in here.
type Caller ¶
type Caller struct {
// ID names the key (never the key itself); empty for an anonymous caller. It is the
// rate-limit bucket of the caller (see CallerKeys) and what a log line may say.
ID string
// contains filtered or unexported fields
}
Caller is who a request is from, as the tool endpoint and the MCP endpoint see it. The zero Caller is anonymous.
func CallerFrom ¶
CallerFrom returns the Caller of the request the context belongs to: anonymous outside a request served by this package, when no Authenticator is configured, and when the presented key was invalid (see Authenticator).
func NewCaller ¶
NewCaller returns the authenticated caller id, allowed the given tiers (TierKeyed for an ordinary key).
func (Caller) Allows ¶
Allows reports whether the caller may run a tool of tier t: public and byo-key tools are open to everyone (byo-key additionally needs the caller's own Janice key), keyed tools to a key whose allowance includes TierKeyed, disabled tools to nobody.
func (Caller) Authenticated ¶
Authenticated reports whether the caller presented a valid key.
type CallerKeys ¶
type CallerKeys struct{}
CallerKeys names the rate-limit bucket by the authenticated key ("key:<id>"), and by client IP (ClientIP) for an anonymous caller or an invalid key, so guessing keys is limited per IP. It is the default KeyResolver.
type Config ¶
type Config struct {
Deps Deps
// Prefix is the route prefix; empty means DefaultPrefix ("/v1"). It must start
// with "/" and must not end with one. services/native sets "/api" to keep its
// legacy paths.
Prefix string
// Limits are the per-route budgets; nil means DefaultLimits().
Limits *Limits
// Keys names the limiter bucket of a request; nil means CallerKeys (the key id of
// an authenticated caller, else the client IP).
Keys KeyResolver
// Auth turns an API key into a Caller (see Authenticator, StaticKeys). Nil means no
// keys exist: every caller is anonymous and keyed tools answer 401 api_key_required.
// It applies to the tool endpoint and, through CallerFrom, to the MCP handler.
Auth Authenticator
// CorpusProbeTTL is how long /health reuses a corpus reachability verdict; zero
// means DefaultCorpusProbeTTL. Only a Retriever that implements Pinger is probed.
CorpusProbeTTL time.Duration
// ToolAPI selects the raw tool endpoint mode; zero is off.
ToolAPI ToolAPIMode
// ToolFormat selects the body of a successful tool call; zero is the JSON
// envelope. Errors are the JSON error shape in either format.
ToolFormat ToolFormat
// ToolTimeout bounds one tool execution; zero means DefaultToolTimeout.
ToolTimeout time.Duration
// Docs serves the generated descriptions of the API: GET {prefix}/openapi.yaml and
// {prefix}/openapi.json, and GET /llms.txt and /llms-full.txt at the root of the
// host (so the service must own the host's root, as cmd/dataapi does; a mount
// under a sub-path never sees those two). They describe this service's own
// configuration: the tool endpoint only when ToolAPI is on, the MCP endpoint only
// when MCP is set. New fails if Docs is combined with ToolFormatText, which they do
// not describe.
Docs bool
// PublicURL is the origin the service is reachable at, e.g. "https://data.eve-cyno.dev"
// (https, no path or query; New fails otherwise). When set, the generated OpenAPI
// `servers` entry and the links of llms.txt / llms-full.txt are absolute, which clients
// that cannot resolve relative server URLs (ChatGPT Actions) need, and the index page
// shows it in its examples. Empty keeps everything relative.
PublicURL string
// MCP, when non-nil, is mounted at {prefix}/mcp for every HTTP method, behind the
// request ID, panic recovery, API-key authentication (an invalid key is a 401 before
// the handler runs) and the Limits.MCP limiter. Build it with mcpserver.HTTPHandler,
// which picks the tool set per request; give it Keyed: func(r) bool { return
// CallerFrom(r.Context()).Allows(catalog.TierKeyed) }. Pass untyped nil, never a typed
// nil pointer wrapped in the interface.
MCP http.Handler
// Logger receives the access log and handler diagnostics; every line carries
// request_id. Nil discards.
Logger *slog.Logger
}
Config configures New.
type Deps ¶
type Deps struct {
// Tools is the deterministic tool layer; Tools.SDE is the one SDE every route
// reads (item search, hull-name resolution, EFT parsing).
Tools *tools.Deps
// Retriever backs /fits/search and the community frequencies of /fit/suggest.
Retriever FitSearcher
// Stats computes /fit/stats (core/fit/gofa).
Stats fit.StatsProvider
}
Deps are the collaborators the handlers read. Any of them may be nil; the routes that need a missing one answer 503 `unavailable` instead of failing at startup, so a service with an SDE but no corpus still serves what it can.
Pass untyped nil, never a typed nil pointer wrapped in an interface (a nil *rag.QdrantRetriever assigned to Retriever is a non-nil interface).
func NewDeps ¶
func NewDeps(t *tools.Deps, retr *rag.QdrantRetriever) Deps
NewDeps assembles Deps from the concrete deterministic layer the binaries build at startup (core/bootstrap): the Gofa stats engine is the tool layer's own (t.StatsEngine, over t.SDE), so /fit/stats and the fit-stat tools share one warm SDE memo, and a nil retriever stays an untyped nil interface (assigning a nil *rag.QdrantRetriever to Deps.Retriever directly would make it non-nil). Either argument may be nil; the routes that need what is missing answer 503.
type FitSearcher ¶
type FitSearcher interface {
SearchFits(ctx context.Context, q rag.FitSearchQuery) (rag.FitSearchResult, error)
}
FitSearcher is the community-fit corpus the search and suggestion routes read. *rag.QdrantRetriever implements it.
type KeyResolver ¶
KeyResolver names the rate-limit bucket a request counts against. It is the seam for per-API-key limits: today every caller is keyed by IP (IPKeys); once keys exist (roadmap R3.5) a resolver returns e.g. "key:<id>" for an authenticated caller and falls back to IPKeys for anonymous ones. The key is opaque to the limiter; resolvers should prefix it so key and IP namespaces cannot collide.
type Limits ¶
type Limits struct {
FitsSearch Rate
FitsDetail Rate
FitStats Rate
FitSuggest Rate
ItemsSearch Rate
// Tool applies only in ToolAPIPublic mode.
Tool Rate
// MCP counts every request to the MCP endpoint (a session is a handful of requests
// plus one per tool call).
MCP Rate
// Docs covers the description routes (OpenAPI, llms.txt).
Docs Rate
}
Limits holds the per-route budgets.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits are the budgets the endpoints had inside services/native.
type Pinger ¶
Pinger is implemented by a corpus backend that can say whether it is reachable (*rag.QdrantRetriever does). A FitSearcher without Ping is assumed reachable.
type Rate ¶
Rate is one sliding-window budget per caller key: at most Max requests in any Window. A Max or Window <= 0 disables the limit (the zero Rate is unlimited).
type StaticKeys ¶
type StaticKeys struct {
// contains filtered or unexported fields
}
StaticKeys is an Authenticator over a fixed set of keys, held as SHA-256 hashes: for self-hosting and tests. Every key it knows may run the keyed tier.
func LoadKeyFile ¶
func LoadKeyFile(path string) (*StaticKeys, error)
LoadKeyFile is ParseKeyFile over the file at path.
func ParseKeyFile ¶
func ParseKeyFile(r io.Reader) (*StaticKeys, error)
ParseKeyFile reads a key file: one `id:sha256hex` per line (id is 1-64 characters from [A-Za-z0-9._-], the hash 64 hex digits); blank lines and lines starting with # are skipped. Anything else is an error that names the line number but never echoes the line, so a plaintext key pasted by mistake does not end up in a log. A file without a key is an error too.
func (*StaticKeys) Authenticate ¶
func (k *StaticKeys) Authenticate(r *http.Request) (Caller, error)
Authenticate implements Authenticator.
type Surface ¶
type Surface struct {
// Prefix is the route prefix ("" means DefaultPrefix).
Prefix string
// ToolAPI is the raw tool endpoint mode.
ToolAPI ToolAPIMode
// MCP is true when the MCP streamable-HTTP endpoint is mounted at {prefix}/mcp.
MCP bool
// Auth is true when the service accepts API keys, which makes the keyed tier
// available and documented.
Auth bool
// Docs is true when the description routes (openapi.yaml, openapi.json, /llms.txt,
// /llms-full.txt) are served.
Docs bool
// Limits are the per-route budgets stated in the documents.
Limits Limits
// PublicURL, when set (validated, no trailing slash), makes the OpenAPI servers entry
// and the llms.txt links absolute. PublicSurface leaves it empty so the committed
// generated files stay host-independent.
PublicURL string
}
Surface says which optional parts a service exposes. The generated documents (OpenAPI, llms.txt) describe exactly the Surface they are given, and New registers exactly the routes the Surface of its Config names, from the one route table below.
func PublicSurface ¶
func PublicSurface() Surface
PublicSurface is what the internet-facing cmd/dataapi serves with every optional part switched on, at the default prefix and default limits. The committed generated files (api/openapi.yaml, llms.txt, llms-full.txt) describe it.
type ToolAPIMode ¶
type ToolAPIMode int
ToolAPIMode controls the raw tool endpoint POST {prefix}/tool/{name}.
const ( // ToolAPIOff (the zero value) does not register the route at all: it answers // 404 like any unknown path. The endpoint runs raw tools for any caller, so it // is opt-in. ToolAPIOff ToolAPIMode = iota // ToolAPIPublic registers the route for an internet-facing service: the Tool // rate limit applies and the tool tiers are enforced (catalog.Tier): keyed tools // need an API key (401 api_key_required / invalid_api_key), appraise_items needs // the caller's own Janice key in X-Janice-Key (400 janice_key_required), disabled // tools answer 403 tool_disabled. The project's Janice key is never used. ToolAPIPublic // ToolAPILoopback registers the route with no limit and no tiers, for a // process that only serves its own user (the desktop app). ToolAPILoopback )
type ToolFormat ¶
type ToolFormat int
ToolFormat is the 200 body of the raw tool endpoint POST {prefix}/tool/{name}.
const ( // ToolFormatEnvelope (the zero value) answers application/json: // // {"tool": "...", "version": "...", // "result": {"text": "...", "data": {...} | null}, // "attribution": [{"name": "...", "url": "...", "license": "..."}]} // // result.text is the LLM-oriented text the brain sees; result.data is the typed // result for the tools that have one (null for the rest); attribution names the // upstreams behind the answer. This is the public /v1 contract. ToolFormatEnvelope ToolFormat = iota // ToolFormatText answers the bare text, text/plain: the contract of the legacy // native /api/tool/{name}, which the desktop app and any script written against it // read as is. services/native sets it on its "/api" mount; nothing new should. ToolFormatText )
type WindowLimiter ¶
type WindowLimiter struct {
// contains filtered or unexported fields
}
WindowLimiter is a sliding-window-log limiter: at most max events per key in any window. It is safe for concurrent use.
func NewWindowLimiter ¶
func NewWindowLimiter(max int, window time.Duration) *WindowLimiter
NewWindowLimiter returns a limiter allowing max events per key per window.
func (*WindowLimiter) Allow ¶
func (l *WindowLimiter) Allow(key string) (ok bool, retryAfter time.Duration)
Allow records an event for key and reports whether it is within budget. When it is not, retryAfter is how long until the oldest counted event leaves the window (always > 0). A refused event is not recorded.