Help us shape the future of Go! Take the 2026 Go Developer Survey

api

package
v0.0.0-...-5a4fc4a Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: AGPL-3.0 Imports: 82 Imported by: 0

Documentation

Index

Constants

View Source
const CacheInvalidatePath = "/api/v1/internal/cache/invalidate"

CacheInvalidatePath is the single source of truth for the /api/v1/internal/cache/invalidate route. The receiver registers it in CacheInvalidateHandler.Register; the cluster post-compaction sender (cmd/arc/main.go) and the auth middleware's PublicRoutes config both reference this constant so a future typo cannot silently break fan-out.

View Source
const ForwardedByHeader = "X-Arc-Forwarded-By"

ForwardedByHeader is the header used to detect forwarded requests and prevent routing loops.

View Source
const HeaderArcliInstallationID = "Arcli-Installation-Id"

HeaderArcliInstallationID is sent by arcli on every request with a random per-installation UUID (see arcli's README, "Privacy").

View Source
const IngestFencedHeader = "X-Arc-Ingest-Fenced"

IngestFencedHeader marks a 503 that refused a write because the node may not accept ingest (#1268): a node that forwarded the write recognises the expected refusal in the copied response and does not log it per request.

View Source
const IngestRefusedHeader = "X-Arc-Ingest-Refused"

IngestRefusedHeader marks a 503 that refused a write for a node-local, transient condition (ingest.WriteRefusal, #676); its value is the reason, e.g. "wal_disk". Nothing was written and a retry, to this node too, is safe. A forwarding node and the request logger recognise it like IngestFencedHeader.

Variables

View Source
var ErrCQNotCaughtUp = errors.New("this node has not caught up with the cluster's continuous queries yet; retry in a few seconds")

ErrCQNotCaughtUp: this node's tables are the cluster's (a mirror ran here), but its cluster state has not caught up since it started. The scheduler starts with no jobs and fills them in once it has.

Functions

func BuildHTTPRequest

func BuildHTTPRequest(c *fiber.Ctx) (*http.Request, error)

BuildHTTPRequest converts a Fiber context to a net/http Request for forwarding via the router.

Lifecycle: every caller in this codebase (lineprotocol.go, msgpack.go, import.go, tle.go, query.go) invokes the returned *http.Request synchronously within the Fiber handler — they pass it to router.RouteWrite/RouteQuery (which blocks on http.Client.Do; queries are buffered for retries, writes are streamed once and never retried, see internal/cluster/router.go), then read+close the response body inside CopyResponse, then return. fasthttp does not recycle the RequestCtx until after the handler returns (fasthttp server.go releaseCtx is post-handler), so wrapping c.Body() in bytes.NewReader directly is safe and avoids a copy that can be large at max payload (up to 1GB). c.Context() is a *fasthttp.RequestCtx; its Done() closes only on server shutdown, so a forward keeps running after its client leaves, as a local write would. Its value is only a parent for the forward's timeout.

If a future refactor moves any of this work to a background goroutine or stream writer, those properties no longer hold and the caller must take a defensive copy + use a non-pooled context. Today's callers are all synchronous; revisit this if that changes.

func CQTableLister

func CQTableLister(authManager *auth.AuthManager, dbPath string) cluster.LocalPolicyLister

CQTableLister is RetentionTableLister for continuous queries: a node where they are disabled still reports the ones its database holds.

func CheckWritePermissions

func CheckWritePermissions(c *fiber.Ctx, rbacManager RBACChecker, logger zerolog.Logger, database string, measurements []string) error

CheckWritePermissions checks if the token has write permission for the database and measurements. This is a shared implementation used by both LineProtocol and MsgPack handlers.

func CopyResponse

func CopyResponse(c *fiber.Ctx, resp *http.Response) error

CopyResponse writes an HTTP response from the router to a Fiber context. It copies the status code, headers (excluding hop-by-hop), and body.

func ForwardIdentityVerifier

func ForwardIdentityVerifier(coord ForwardIdentityChecker) func(c *fiber.Ctx) (bool, int64, string, error)

ForwardIdentityVerifier is the auth middleware's check of a forwarded request's signed identity (auth.MiddlewareConfig.ForwardIdentity). The request URI is the raw one received, which is what the peer signed.

func HandleRoutingError

func HandleRoutingError(c *fiber.Ctx, err error) error

HandleRoutingError returns an appropriate error response for routing failures.

func ListenAddr

func ListenAddr(host string, port int) (string, string)

ListenAddr canonicalizes (host, port) into the host:port string callers hand to a listener — Fiber on the HTTP path, the cluster coordinator's advertise/peer logic on the cluster path.

Behavior:

  • host="" preserves the historical wildcard (":<port>") so existing deployments keep dual-stack binding on Linux on upgrade.
  • net.JoinHostPort adds IPv6 brackets when needed; surrounding brackets in user input are stripped first so we never produce an invalid double-bracketed address like "[[::1]]:8000".

The canonicalized host is returned alongside the address so callers can use it for logging without re-parsing. Exported so non-api callers (cmd/arc) format the same address shape this server does — historically this was duplicated with fmt.Sprintf("%s:%d", …) which mis-formatted IPv6 literals.

func OptimizeLikePatterns

func OptimizeLikePatterns(sql string) (string, bool)

OptimizeLikePatterns rewrites SQL to reorder WHERE clause predicates for better query performance. Returns the optimized SQL and whether any changes were made.

func RespondAlreadyForwarded

func RespondAlreadyForwarded(c *fiber.Ctx) error

RespondAlreadyForwarded writes the error returned when a node receives a request marked as already-forwarded that it cannot serve locally (a routing loop or a spoofed X-Arc-Forwarded-By header). Uses 508 Loop Detected so the condition is distinguishable from a transient 503.

func RetentionTableLister

func RetentionTableLister(authManager *auth.AuthManager, dbPath string) cluster.LocalPolicyLister

RetentionTableLister answers the cluster's adoption on a node where retention is disabled: it runs nothing, but its database may still hold the policies (and history) of when it did, and those must count. Left out, the cluster could keep IDs that this node's own rows also use, and an old ID here would name a different policy once retention is turned back on. The table is read from the auth database, or from retention.db_path when that file exists; no table is no policies.

func RewriteRegexToStringFuncs

func RewriteRegexToStringFuncs(sql string) (string, bool)

RewriteRegexToStringFuncs rewrites regex function calls to equivalent string functions. This provides 2x performance improvement for common patterns like URL domain extraction. Returns the rewritten SQL and whether any rewrites were applied.

func ShouldForwardQuery

func ShouldForwardQuery(router *cluster.Router, c *fiber.Ctx) bool

ShouldForwardQuery reports whether a query request should be forwarded to another node. See ShouldForwardWrite for the return-value semantics.

func ShouldForwardWrite

func ShouldForwardWrite(router *cluster.Router, c *fiber.Ctx) bool

ShouldForwardWrite reports whether a write request should be forwarded to another node. It returns true only for ForwardToPeer; ForwardLocal and ForwardAlreadyForwarded both return false. Callers that need to distinguish the already-forwarded case (to emit an error rather than fall through to local processing) should use WriteForwardDecision instead.

func ValidateSQLRequest

func ValidateSQLRequest(sql string) error

ValidateSQLRequest validates an SQL query for common issues. Returns nil if valid, or an error with appropriate message. This is a shared function used by multiple query endpoints.

SECURITY: The denylist regex must run on a normalised version of the SQL — comments stripped (so `DROP /* */ TABLE x` does not bypass the `\b...\b` token boundary) and string literals masked (so `SELECT 'DROP TABLE x'` does not false-positive). The cost is one full pass over the SQL; for the 10KB cap that's microseconds.

Types

type AuditHandler

type AuditHandler struct {
	// contains filtered or unexported fields
}

AuditHandler handles audit log query endpoints

func NewAuditHandler

func NewAuditHandler(auditLogger *audit.Logger, authManager *auth.AuthManager, licenseClient *license.Client, logger zerolog.Logger) *AuditHandler

NewAuditHandler creates a new audit handler

func (*AuditHandler) RegisterRoutes

func (h *AuditHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers audit log query endpoints

type AuthHandler

type AuthHandler struct {
	// contains filtered or unexported fields
}

AuthHandler handles authentication-related endpoints

func NewAuthHandler

func NewAuthHandler(authManager *auth.AuthManager, logger zerolog.Logger) *AuthHandler

NewAuthHandler creates a new auth handler

func (*AuthHandler) RegisterRoutes

func (h *AuthHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers auth-related endpoints

func (*AuthHandler) RegisterTokenMembershipRoutes

func (h *AuthHandler) RegisterTokenMembershipRoutes(app *fiber.App)

RegisterTokenMembershipRoutes registers token membership endpoints (requires RBAC manager)

func (*AuthHandler) SetRBACManager

func (h *AuthHandler) SetRBACManager(rbacManager *auth.RBACManager)

SetRBACManager sets the RBAC manager for token membership endpoints

type BackupCoordinator

type BackupCoordinator interface {
	// IsPrimaryWriter reports whether this node may execute writer-only
	// mutations. A restore writes data files and (on a cluster) manifest
	// entries; a backup reads a listing that only the primary writer is
	// guaranteed to hold in full, and only one node should hold the slot.
	// Deletion sweeps the configured backup targets of the authoritative node.
	IsPrimaryWriter() bool
	// Role returns a human-readable role string for the rejection message.
	Role() string
}

BackupCoordinator is the minimal cluster interface the backup handler needs (#1083, #1134): which node may create, restore or delete a backup. nil = standalone mode, no gate. Same shape as DeleteCoordinator and RetentionCoordinator.

type BackupHandler

type BackupHandler struct {
	// contains filtered or unexported fields
}

BackupHandler handles backup and restore API operations.

func NewBackupHandler

func NewBackupHandler(manager *backup.Manager, authManager *auth.AuthManager, operationTimeout time.Duration, logger zerolog.Logger) *BackupHandler

NewBackupHandler creates a new backup handler.

operationTimeout bounds one backup or one restore run (backup.operation_timeout, default 2h). It is a constructor parameter rather than a settable field because both routes detach from the request context — Fiber recycles it — so this is the only thing that eventually releases the single-operation slot, and a zero value would expire the context before the run started. config.Load refuses a non-positive value; a caller that builds one by hand and leaves it zero is corrected here to the historical 2h rather than cancelling instantly.

func (*BackupHandler) CreateBackup

func (h *BackupHandler) CreateBackup(c *fiber.Ctx) error

CreateBackup triggers a new backup. POST /api/v1/backup

func (*BackupHandler) DeleteBackup

func (h *BackupHandler) DeleteBackup(c *fiber.Ctx) error

DeleteBackup removes a backup. DELETE /api/v1/backup/:id

func (*BackupHandler) GetBackup

func (h *BackupHandler) GetBackup(c *fiber.Ctx) error

GetBackup returns the manifest for a specific backup. GET /api/v1/backup/:id

func (*BackupHandler) GetStatus

func (h *BackupHandler) GetStatus(c *fiber.Ctx) error

GetStatus returns the progress of the current active operation. GET /api/v1/backup/status

func (*BackupHandler) ListBackups

func (h *BackupHandler) ListBackups(c *fiber.Ctx) error

ListBackups returns all available backups. GET /api/v1/backup

With several configured targets (#1085 stage B2b-2) the listing fans out over all of them and unions by backup ID. A target that cannot provide a complete listing, including one with an unreadable manifest, is NAMED in unreachable_targets and the response is still 200: one failed store must not hide the backups on the others. 503 is returned when no target can provide a complete listing; one unreadable destination is not a trustworthy answer for a single-target setup.

func (*BackupHandler) RegisterRoutes

func (h *BackupHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes registers backup and restore API routes.

func (*BackupHandler) RestoreBackup

func (h *BackupHandler) RestoreBackup(c *fiber.Ctx) error

RestoreBackup triggers a restore from a backup. POST /api/v1/backup/restore

func (*BackupHandler) SetCoordinator

func (h *BackupHandler) SetCoordinator(c BackupCoordinator)

SetCoordinator wires the cluster coordinator for the node gate. Callers pass it only when they hold a non-nil coordinator: an interface holding a typed nil pointer is not == nil (#713), and the gate would then call methods on a nil receiver. A nil interface is ignored here.

type CQClusterSource

type CQClusterSource interface {
	CQPoliciesSeeded() bool
	CQPoliciesCollecting() bool
	ClusterCQ(id int64) (raft.CQEntry, bool)
	ClusterCQByName(name string) (raft.CQEntry, bool)
	ClusterCQs() []raft.CQEntry
	CreateCQ(ctx context.Context, p raft.CreateCQPayload) error
	UpdateCQ(ctx context.Context, p raft.UpdateCQPayload) error
	DeleteCQ(ctx context.Context, id int64) error
	SetCQWatermark(ctx context.Context, id int64, from *time.Time, to time.Time) error
	CQReadBarrier(ctx context.Context) error
	CQMirrorState() raft.CQMirror
	LocalNodeID() string
}

CQClusterSource is the cluster state continuous queries read and write once adopted. *cluster.Coordinator satisfies it.

type CQCoordinator

type CQCoordinator interface {
	IsPrimaryWriter() bool
	Role() string
}

CQCoordinator is the minimal cluster interface the CQ handler needs to gate manual execute requests to the primary writer. nil = standalone mode (no gate).

type CQExecution

type CQExecution struct {
	ID                       int64   `json:"id"`
	QueryID                  int64   `json:"query_id"`
	ExecutionID              string  `json:"execution_id"`
	ExecutionTime            string  `json:"execution_time"`
	Status                   string  `json:"status"`
	StartTime                string  `json:"start_time"`
	EndTime                  string  `json:"end_time"`
	RecordsRead              *int64  `json:"records_read"`
	RecordsWritten           int64   `json:"records_written"`
	ExecutionDurationSeconds float64 `json:"execution_duration_seconds"`
	ErrorMessage             *string `json:"error_message"`
}

CQExecution represents an execution history record

type CQJobDef

type CQJobDef struct {
	ID       int64
	Name     string
	Interval string
	IsActive bool
}

CQJobDef is what a scheduler job needs of a continuous query.

type CQSchedulerInterface

type CQSchedulerInterface interface {
	Status() map[string]interface{}
	ReloadAll() error
	JobCount() int
}

CQSchedulerInterface defines the interface for CQ scheduler operations

type CQSchedulerReloader

type CQSchedulerReloader interface {
	ReloadCQ(cqID int64) error
	// StartJobDirect schedules a job using data already in hand, avoiding a
	// redundant SQLite read that could race with the just-committed INSERT.
	StartJobDirect(cqID int64, name, interval string, isActive bool) error
}

CQSchedulerReloader is an interface for reloading CQ schedules after updates. This avoids a circular import between api and scheduler packages.

type CacheInvalidateHandler

type CacheInvalidateHandler struct {
	// contains filtered or unexported fields
}

CacheInvalidateHandler serves the post-compaction cluster cache-invalidate endpoint. It exists only when cluster mode is active AND cluster.shared_secret is configured — callers in cmd/arc/main.go construct it conditionally and skip Register entirely otherwise.

Closes audit finding #3 from 2026-05-19: the previous gate was a static header (`X-Arc-Internal: cache-invalidate`) which any network-reachable caller could replay, forcing DuckDB's cache_httpfs glob results to repopulate from S3 — cost amplification + p99 spikes with no auth.

The "unconfigured = no route" shape (rather than a runtime secret-string check inside the handler) means an unconfigured Arc instance returns 404 for this path — there is nothing to probe, not even a 403. Configured-but-misauthed callers still get a uniform 403 across every rejection branch (see handle).

Per-decision: the audit deliberately assigned NO CVE to this finding because OSS does not reach the endpoint (post-compaction invalidation is in-process via db.ClearHTTPCache) and the endpoint carries no secrets — it is only a cluster-mode "clear your caches now" signal. Do NOT retroactively file a CVE without re-evaluating the threat model.

func NewCacheInvalidateHandler

func NewCacheInvalidateHandler(
	sharedSecret, clusterName, localNodeID string,
	nonceCache *security.NonceCache,
	tolerance time.Duration,
	onInvalidate func(),
	logger zerolog.Logger,
) *CacheInvalidateHandler

NewCacheInvalidateHandler constructs the handler. All arguments are required; the constructor panics on missing pieces so a misconfigured caller is loudly broken at startup instead of producing a silently half-armed endpoint. By construction, every field on the returned value is non-zero — handle() needs no nil-guards.

func (*CacheInvalidateHandler) Register

func (h *CacheInvalidateHandler) Register(app fiber.Router)

Register wires POST CacheInvalidatePath. Callers MUST add CacheInvalidatePath to the auth middleware's PublicRoutes — the endpoint authenticates via HMAC, not via the user-token auth middleware (cluster peers do not carry user auth).

type ClientRecorder

type ClientRecorder interface {
	RecordClient(id, version string)
}

ClientRecorder receives the identity of CLI installations that talk to this node. The telemetry collector implements it; nil disables the middleware entirely.

type ClusterHandler

type ClusterHandler struct {
	// contains filtered or unexported fields
}

ClusterHandler handles cluster management API endpoints.

func NewClusterHandler

func NewClusterHandler(
	coordinator *cluster.Coordinator,
	authManager *auth.AuthManager,
	licenseClient *license.Client,
	logger zerolog.Logger,
) *ClusterHandler

NewClusterHandler creates a new cluster handler. The coordinator can be nil if clustering is not enabled.

func (*ClusterHandler) RegisterRoutes

func (h *ClusterHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers cluster API routes.

type CompactionGate

type CompactionGate interface {
	// CanCompact reports whether the local node may run compaction. With
	// compactor failover configured this is the FSM's active-compactor
	// lease; otherwise it is the static role capability.
	CanCompact() bool
	// Role returns a human-readable role string for the rejection message.
	Role() string
	// LeaseHolder returns the node ID holding the compactor lease, or ""
	// when this cluster manages no lease at all.
	//
	// The rejection message needs this because CanCompact() is false for two
	// unrelated reasons and the remedy differs. With a lease, the operator
	// sends the trigger to its holder. Without one — the default, since
	// cluster.failover_enabled is false — the refusal came from the static
	// role, there is no lease to route to, and POST
	// /api/v1/cluster/compactor/assign cannot create the first one: it
	// answers "this cluster does not manage a compactor lease". Telling that
	// operator to assign a lease sends them to an endpoint that refuses in
	// exactly the state the message describes.
	LeaseHolder() string
}

CompactionGate is the minimal cluster interface the compaction handler needs (#1152): whether this node may run a compaction cycle. Same shape as compaction.ClusterGate, which the scheduler already consults, so the value main builds for the scheduler satisfies this with no adapter — and the two paths cannot drift apart. nil = OSS or standalone, no gate.

type CompactionHandler

type CompactionHandler struct {
	// contains filtered or unexported fields
}

CompactionHandler handles compaction API endpoints

func NewCompactionHandler

func NewCompactionHandler(manager *compaction.Manager, hourlyScheduler, dailyScheduler *compaction.Scheduler, authManager *auth.AuthManager, gate CompactionGate, logger zerolog.Logger) *CompactionHandler

NewCompactionHandler creates a new compaction handler.

gate is a constructor parameter rather than a setter (which is how the backup handler takes its coordinator) because main already holds the gate before it builds this handler: the value exists at the compaction-gate block and the handler is constructed much later. A setter would leave the wiring optional, which is the failure this gate exists to close — the trigger endpoint ran a full cycle on any node precisely because nothing required the role check to be wired.

func (*CompactionHandler) RegisterRoutes

func (h *CompactionHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers compaction endpoints

type ContinuousQuery

type ContinuousQuery struct {
	ID                     int64   `json:"id"`
	Name                   string  `json:"name"`
	Description            *string `json:"description"`
	Database               string  `json:"database"`
	SourceMeasurement      string  `json:"source_measurement"`
	DestinationMeasurement string  `json:"destination_measurement"`
	Query                  string  `json:"query"`
	Interval               string  `json:"interval"`
	// TagColumns are the output columns that are grouping dimensions (not
	// aggregates). They are written as Parquet `arc:tags` metadata and narrow
	// where compaction compares rows (#521). Compaction removes only rows
	// identical in every column (#1232): an identical copy of a window
	// collapses, and a re-emitted window whose values changed is kept beside
	// the original, with or without tag columns.
	TagColumns            []string `json:"tag_columns"`
	RetentionDays         *int     `json:"retention_days"`
	DeleteSourceAfterDays *int     `json:"delete_source_after_days"`
	IsActive              bool     `json:"is_active"`
	LastExecutionTime     *string  `json:"last_execution_time"`
	LastExecutionStatus   *string  `json:"last_execution_status"`
	LastProcessedTime     *string  `json:"last_processed_time"`
	LastRecordsWritten    *int64   `json:"last_records_written"`
	CreatedAt             string   `json:"created_at"`
	UpdatedAt             string   `json:"updated_at"`
}

ContinuousQuery represents a continuous query definition

type ContinuousQueryHandler

type ContinuousQueryHandler struct {
	// contains filtered or unexported fields
}

ContinuousQueryHandler handles continuous query operations

func NewContinuousQueryHandler

func NewContinuousQueryHandler(db *database.DuckDB, storage storage.Backend, arrowBuffer *ingest.ArrowBuffer, cfg *config.ContinuousQueryConfig, authManager *auth.AuthManager, logger zerolog.Logger) (*ContinuousQueryHandler, error)

func (*ContinuousQueryHandler) Close

func (h *ContinuousQueryHandler) Close() error

Close closes the database connection

func (*ContinuousQueryHandler) ExecuteCQ

func (h *ContinuousQueryHandler) ExecuteCQ(ctx context.Context, queryID int64) (*ExecuteCQResponse, error)

ExecuteCQ executes a continuous query by ID programmatically (used by scheduler) Returns the execution response and any error

func (*ContinuousQueryHandler) ForgetPendingWatermarks

func (h *ContinuousQueryHandler) ForgetPendingWatermarks()

ForgetPendingWatermarks drops every unsettled watermark: called when this node stops being the primary, so a later promotion never re-proposes one over whatever the cluster did meanwhile (another primary's runs, a reset that happened to land on the same value).

func (*ContinuousQueryHandler) GetActiveCQs

func (h *ContinuousQueryHandler) GetActiveCQs() ([]ContinuousQuery, error)

GetActiveCQs returns all active continuous queries (used by scheduler)

func (*ContinuousQueryHandler) GetCQ

func (h *ContinuousQueryHandler) GetCQ(queryID int64) (*ContinuousQuery, error)

GetCQ returns a continuous query by ID (used by scheduler)

func (*ContinuousQueryHandler) LocalPolicyLister

func (h *ContinuousQueryHandler) LocalPolicyLister() cluster.LocalPolicyLister

LocalPolicyLister reads this node's own continuous queries for the cluster's adoption, under localMu.

func (*ContinuousQueryHandler) MirrorClusterPolicies

func (h *ContinuousQueryHandler) MirrorClusterPolicies(ctx context.Context) error

MirrorClusterPolicies makes this node's own tables equal the cluster's continuous queries, watermarks included, once adopted (a no-op before), and then hands the scheduler the cluster's CQs.

func (*ContinuousQueryHandler) RegisterRoutes

func (h *ContinuousQueryHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers continuous query endpoints

func (*ContinuousQueryHandler) SetClusterSource

func (h *ContinuousQueryHandler) SetClusterSource(src CQClusterSource)

SetClusterSource makes continuous queries cluster-wide once the cluster adopts them. Called after construction in cluster mode.

func (*ContinuousQueryHandler) SetCoordinator

func (h *ContinuousQueryHandler) SetCoordinator(c CQCoordinator)

SetCoordinator sets the cluster coordinator for writer-gate checks. Called after coordinator creation since it depends on the handler.

func (*ContinuousQueryHandler) SetFieldSchema

func (h *ContinuousQueryHandler) SetFieldSchema(r *fieldschema.Registry)

executeAggregation runs the aggregation query and writes results. startTime is the window's inclusive left boundary — it stamps output rows when the aggregation query does not select a time column, so re-runs of the same window land on the same (tags, time), and an identical one collapses at compaction (#521, #1232). The window's end time is intentionally unused here: output rows are labelled by the window START (bucket-start convention), so only startTime is needed. SetFieldSchema installs the field schema registry (#914).

func (*ContinuousQueryHandler) SetOpenCompactions

func (h *ContinuousQueryHandler) SetOpenCompactions(oc OpenCompactions)

SetOpenCompactions wires the compaction jobs a source read must account for.

func (*ContinuousQueryHandler) SetScheduler

func (h *ContinuousQueryHandler) SetScheduler(s CQSchedulerReloader)

SetScheduler sets the CQ scheduler for reloading after updates. Called after scheduler creation since it depends on the handler.

type ContinuousQueryRequest

type ContinuousQueryRequest struct {
	Name                   optionalField[string]   `json:"name"`
	Description            optionalField[string]   `json:"description"`
	Database               optionalField[string]   `json:"database"`
	SourceMeasurement      optionalField[string]   `json:"source_measurement"`
	DestinationMeasurement optionalField[string]   `json:"destination_measurement"`
	Query                  optionalField[string]   `json:"query"`
	Interval               optionalField[string]   `json:"interval"`
	TagColumns             optionalField[[]string] `json:"tag_columns"`
	RetentionDays          optionalField[int]      `json:"retention_days"`
	DeleteSourceAfterDays  optionalField[int]      `json:"delete_source_after_days"`
	IsActive               optionalField[bool]     `json:"is_active"`
}

ContinuousQueryRequest represents a request to create/update a CQ.

Every field distinguishes an absent key from a null one (#1023). Create requires name, database, both measurements, query and interval. Update is a partial update: an absent key keeps the stored value, a present one is validated with the create rules and written, and null clears the four fields that can be empty (description, tag_columns, retention_days, delete_source_after_days); null on any other field is refused. Before #1023 every field was written unconditionally, so a PUT that left out is_active stored false and stopped the continuous query with a 200.

type CreateBackupRequest

type CreateBackupRequest struct {
	IncludeMetadata *bool `json:"include_metadata"` // default: true; false and refused when scoped
	// IncludeConfig default: true, EXCEPT false when the backup is scoped and
	// false when the destination is a remote target, whose own credentials
	// arc.toml carries (#1085 stage B2b-1). Setting it true against a remote
	// target is honoured and warned about.
	IncludeConfig *bool `json:"include_config"`
	// Databases scopes the backup to these databases (#1084): storage-root
	// segments, so an edge-sync spoke is named as the spoke. Empty or absent
	// is a whole-instance backup. At most maxScopeDatabases names, each a
	// safe storage path segment that names an existing database.
	Databases []string `json:"databases"`
}

CreateBackupRequest is the request body for POST /api/v1/backup.

type CreateDatabaseRequest

type CreateDatabaseRequest struct {
	Name string `json:"name"`
}

CreateDatabaseRequest represents a request to create a new database

type CreateTokenRequest

type CreateTokenRequest struct {
	Name        string    `json:"name"`
	Description string    `json:"description,omitempty"`
	Permissions *[]string `json:"permissions,omitempty"` // nil = default (read,write), empty array = no permissions (RBAC-only)
	ExpiresIn   string    `json:"expires_in,omitempty"`  // e.g., "24h", "7d", "30d"
}

CreateTokenRequest represents a token creation request

type DatabaseDeleteCoordinator

type DatabaseDeleteCoordinator interface {
	IsPrimaryWriter() bool
	Role() string
	GetFileManifestByDatabase(database string) []*raft.FileEntry
	BatchFileOpsInManifestContext(ctx context.Context, ops []raft.BatchFileOp) error
	SyncManifest(ctx context.Context) error
}

DatabaseDeleteCoordinator supplies the cluster-wide part of database deletion. A nil coordinator preserves standalone behavior.

type DatabaseInfo

type DatabaseInfo struct {
	Name             string `json:"name"`
	MeasurementCount int    `json:"measurement_count"`
	CreatedAt        string `json:"created_at,omitempty"`
}

DatabaseInfo represents information about a database

type DatabaseListResponse

type DatabaseListResponse struct {
	Databases []DatabaseInfo `json:"databases"`
	Count     int            `json:"count"`
}

DatabaseListResponse represents the response for listing databases

type DatabaseMeasurement

type DatabaseMeasurement struct {
	Name      string `json:"name"`
	FileCount int    `json:"file_count,omitempty"`
}

DatabaseMeasurement represents a measurement within a database

type DatabasePendingRegistrations

type DatabasePendingRegistrations interface {
	// record is the deletion's DELETE record (0 without one): batches of
	// another writer process settle as deleted by it.
	WithdrawDatabase(ctx context.Context, database string, record uint64) error
}

DatabasePendingRegistrations withdraws every pending registration of a database, so none registers after it is gone.

type DatabasesHandler

type DatabasesHandler struct {
	// contains filtered or unexported fields
}

DatabasesHandler handles database management API endpoints

func NewDatabasesHandler

func NewDatabasesHandler(storage storage.Backend, deleteConfig *config.DeleteConfig, authManager *auth.AuthManager, logger zerolog.Logger) *DatabasesHandler

func (*DatabasesHandler) RegisterRoutes

func (h *DatabasesHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers the database management routes.

Read-only routes (list, get, list measurements) pass through the global any-valid-token middleware mounted in cmd/arc/main.go — listing what exists is appropriate for any authenticated caller.

Mutating routes (Create, Delete) require admin permission. The global middleware would accept any-valid-token; the route-level admin wrapper tightens that to admin only. Without it, a read- scoped token could provision/destroy databases — a privilege escalation by configuration.

`withAdminAuth` returns auth.RequireAdmin(am) when am is non-nil and a passthrough when am is nil (OSS / auth-disabled). This matches the convention in import.go, lineprotocol.go, msgpack.go, and tle.go — single source of truth for the nil-am branching.

func (*DatabasesHandler) SetCoordinator

func (h *DatabasesHandler) SetCoordinator(c DatabaseDeleteCoordinator, gateNonPrimary bool)

SetCoordinator wires cluster manifest updates into database deletion, and with gateNonPrimary the primary-writer admission check. Pass gateNonPrimary under the same condition the tiering manager's writer gate is wired (shared storage or replication), so a cluster answers the same way whether tiered storage is enabled or not: without either, nodes share nothing, the manifest delete unlinks nothing on peers, and each node may delete its own files. Call before RegisterRoutes, alongside the other handler wiring.

func (*DatabasesHandler) SetFieldSchema

func (h *DatabasesHandler) SetFieldSchema(r *fieldschema.Registry)

SetFieldSchema installs the field schema registry so deleting a database also deletes its stored field schema anchors (#914).

func (*DatabasesHandler) SetFlusher

func (h *DatabasesHandler) SetFlusher(f DeleteFlusher)

SetFlusher installs the ingest buffer (nil on nodes that do not ingest).

func (*DatabasesHandler) SetIcebergDropper

func (h *DatabasesHandler) SetIcebergDropper(d IcebergCatalogDropper)

SetIcebergDropper wires catalog cleanup into database deletion.

func (*DatabasesHandler) SetOnFilesRemoved

func (h *DatabasesHandler) SetOnFilesRemoved(fn func())

SetOnFilesRemoved sets the function called after a database delete removed files: the query caches hold pruned partition lists naming them.

func (*DatabasesHandler) SetPendingRegistrations

func (h *DatabasesHandler) SetPendingRegistrations(p DatabasePendingRegistrations)

SetPendingRegistrations installs this node's registration outbox.

func (*DatabasesHandler) SetRBACManager

func (h *DatabasesHandler) SetRBACManager(rm RBACChecker)

SetRBACManager installs the RBAC checker for the listing endpoints.

MUST be called before RegisterRoutes: the route middleware captures this value when the route is registered, so a checker installed afterwards would gate nothing. A plain assignment is enough because there is no concurrent reader at startup. main.go guards the call with `authManager != nil && rbacManager != nil`, the same way it guards every other SetAuthAndRBAC: assigning a nil *auth.RBACManager into this interface field would make `h.rbacManager != nil` true for a typed nil and defeat the guards below.

func (*DatabasesHandler) SetTieringManager

func (h *DatabasesHandler) SetTieringManager(tm databaseTieringManager)

SetTieringManager sets the tiering manager for multi-tier database/measurement listing. This is called after initialization when tiering is enabled and licensed.

type DebugHandler

type DebugHandler struct {
	// contains filtered or unexported fields
}

DebugHandler exposes process- and DuckDB-level memory diagnostics. Used to attribute RSS to Go heap, DuckDB native heap, or glibc arenas during support cases where the customer reports unexplained memory growth.

func NewDebugHandler

func NewDebugHandler(db *database.DuckDB, authManager *auth.AuthManager, logger zerolog.Logger) *DebugHandler

func (*DebugHandler) RegisterRoutes

func (h *DebugHandler) RegisterRoutes(app *fiber.App)

type DeleteConfigResponse

type DeleteConfigResponse struct {
	Enabled               bool              `json:"enabled"`
	ConfirmationThreshold int               `json:"confirmation_threshold"`
	MaxRowsPerDelete      int               `json:"max_rows_per_delete"`
	Implementation        string            `json:"implementation"`
	PerformanceImpact     map[string]string `json:"performance_impact"`
}

DeleteConfigResponse represents delete configuration info

type DeleteCoordinator

type DeleteCoordinator interface {
	BatchFileOpsInManifest(ops []raft.BatchFileOp) error
	GetFileEntry(path string) (*raft.FileEntry, bool)
	IsPrimaryWriter() bool
	Role() string
	// LocalNodeID names this node in the manifest. A rewrite stamps it as the
	// origin of the new bytes (#976).
	LocalNodeID() string
}

DeleteCoordinator is the minimal cluster interface the delete handler needs to gate execution to the primary writer and propagate file manifest changes. nil = standalone mode (no gate, manifest not updated).

type DeleteFlusher

type DeleteFlusher interface {
	// FlushScopeAndWait returns, per buffer key, the last batch it handed
	// to the flush: the marks a DELETE record carries.
	FlushScopeAndWait(ctx context.Context, database, measurement string) (map[string]uint64, error)
	HasFlushFailure() bool
}

DeleteFlusher flushes the ingest buffers of a database so a DELETE sees every row acknowledged before it in a file.

type DeleteHandler

type DeleteHandler struct {
	// contains filtered or unexported fields
}

DeleteHandler handles delete operations using file rewrite strategy

func NewDeleteHandler

func NewDeleteHandler(db *database.DuckDB, storage storage.Backend, cfg *config.DeleteConfig, authManager *auth.AuthManager, tempDir string, logger zerolog.Logger) *DeleteHandler

NewDeleteHandler creates a new delete handler. tempDir MUST be the same path cmd/arc/main.go added to the DuckDB sandbox's allowed_directories, otherwise the COPY ... TO inside rewriteS3File fails on S3-backed deployments. Logs a Warn on empty tempDir so misconfigured deployments surface the issue at startup rather than at the first DELETE.

func (*DeleteHandler) DrainRetired

func (h *DeleteHandler) DrainRetired(ctx context.Context) int

DrainRetired settles the retired journal (retiredAction) and returns how many replaced files it removed.

func (*DeleteHandler) RegisterRoutes

func (h *DeleteHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers delete endpoints

func (*DeleteHandler) RewriteForRecords

func (h *DeleteHandler) RewriteForRecords(ctx context.Context, relativePath string, wheres []string) (cluster.DeleteRewriteResult, error)

RewriteForRecords applies DELETE records to a pending local file (the outbox's cluster.DeleteRewriter): the rows any of wheres matches are removed, exactly as the DELETE removes them from the files it rewrites. A WHERE the file cannot bind (a column it lacks) matches nothing in it, as a DELETE skips such a file.

func (*DeleteHandler) SetCoordinator

func (h *DeleteHandler) SetCoordinator(c DeleteCoordinator)

SetCoordinator wires the cluster coordinator for manifest updates and role gating. Called after construction when cluster mode is enabled.

func (*DeleteHandler) SetFlusher

func (h *DeleteHandler) SetFlusher(f DeleteFlusher)

SetFlusher installs the ingest buffer (nil on nodes that do not ingest).

func (*DeleteHandler) SetOnFilesRemoved

func (h *DeleteHandler) SetOnFilesRemoved(fn func())

SetOnFilesRemoved sets the function called after this handler removed or rewrote parquet files: the query caches hold pruned partition lists, and one naming a removed hour makes DuckDB report "No files found", which a query answers with a retryable 503 (classifyNoFiles) until the cache is cleared.

func (*DeleteHandler) SetPendingRegistrations

func (h *DeleteHandler) SetPendingRegistrations(p PendingRegistrations)

SetPendingRegistrations wires the registration outbox, so a rewrite of a file still pending registration is announced in its place (#1210).

func (*DeleteHandler) SetRetiredReads

func (h *DeleteHandler) SetRetiredReads(r *RetiredReads)

SetRetiredReads wires the standalone read source for replaced files.

func (*DeleteHandler) SetTieringManager

func (h *DeleteHandler) SetTieringManager(tm *tiering.Manager)

SetTieringManager wires rewrite/delete metadata updates into tiering.

func (*DeleteHandler) StartRetiredDrainer

func (h *DeleteHandler) StartRetiredDrainer(ctx context.Context)

StartRetiredDrainer runs DrainRetired now and every retiredDrainEvery until ctx ends (drainHere decides, at every tick, whether this node does).

type DeleteRequest

type DeleteRequest struct {
	Database    string `json:"database"`
	Measurement string `json:"measurement"`
	Where       string `json:"where"`
	DryRun      bool   `json:"dry_run"`
	Confirm     bool   `json:"confirm"`
	// AllowUnreachable runs the DELETE although files the cluster manifest
	// lists are not on this node (stream takeover): their rows are not
	// deleted, and the response names them.
	AllowUnreachable bool `json:"allow_unreachable,omitempty"`
}

DeleteRequest represents a delete operation request

type DeleteResponse

type DeleteResponse struct {
	Success         bool     `json:"success"`
	DeletedCount    int64    `json:"deleted_count"`
	AffectedFiles   int      `json:"affected_files"`
	RewrittenFiles  int      `json:"rewritten_files"`
	ExecutionTimeMs float64  `json:"execution_time_ms"`
	DryRun          bool     `json:"dry_run"`
	FilesProcessed  []string `json:"files_processed"`
	FailedFiles     []string `json:"failed_files,omitempty"`
	// UnreachableFiles are manifest files this node did not have: with
	// allow_unreachable, skipped; without it, why the DELETE was refused.
	UnreachableFiles []string `json:"unreachable_files,omitempty"`
	// UnflushedNodes are other nodes whose buffered rows of the scope could
	// not be flushed first (shared storage, #1265): with allow_unreachable,
	// the rows they hold are left; without it, why the DELETE was refused.
	UnflushedNodes []string `json:"unflushed_nodes,omitempty"`
	// Cold storage (#1350). ColdDeletedFiles: cold files whose rows all
	// matched, deleted (or, in a dry run, to be deleted); their rows are in
	// deleted_count. ColdFiles: cold files with only some matching rows, left
	// as they are; ColdRowsNotDeleted counts the matching rows left in cold
	// storage (never in deleted_count). ColdNotExamined: why cold storage was
	// not looked at, when it is configured but could not be.
	ColdDeletedFiles   []string `json:"cold_deleted_files,omitempty"`
	ColdFiles          []string `json:"cold_files,omitempty"`
	ColdRowsNotDeleted int64    `json:"cold_rows_not_deleted,omitempty"`
	ColdNotExamined    string   `json:"cold_not_examined,omitempty"`
	Error              string   `json:"error,omitempty"`
}

DeleteResponse represents a delete operation response

type EdgeSyncAdminHandler

type EdgeSyncAdminHandler struct {
	// contains filtered or unexported fields
}

EdgeSyncAdminHandler manages spoke registrations.

Separate from EdgeSyncHandler because the two have different audiences and different authentication. The sync endpoints are called by spokes and authenticated per-spoke by HMAC; these are called by an operator and authenticated by an Arc admin token. Mounting them on one handler would invite someone to reuse the wrong middleware.

func NewEdgeSyncAdminHandler

func NewEdgeSyncAdminHandler(registry *edgesync.Registry, authManager *auth.AuthManager, logger zerolog.Logger) (*EdgeSyncAdminHandler, error)

NewEdgeSyncAdminHandler validates configuration and returns a ready handler.

func (*EdgeSyncAdminHandler) RegisterRoutes

func (h *EdgeSyncAdminHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes mounts the spoke-management endpoints.

Every route is admin-only, including the read paths: the spoke list is a map of which edge deployments exist and when each last reported in, which is not something a read-scoped token should be able to enumerate.

Mounted at /api/v1/sync-spokes rather than under /api/v1/sync, deliberately.

Nesting them made behavior depend on registration order: Fiber binds a group's middleware only to routes registered afterwards on that group, so with admin registered first (as main.go does) the sync group's body limit never ran here — but swapping two lines in main.go would silently start applying it, and a registration whose JSON exceeds that limit would then get a 413 from the WRONG group before this handler's auth ever ran. An earlier version of this comment claimed the opposite, having tested the reversed order. A sibling path removes the coupling entirely rather than documenting which order is safe.

type EdgeSyncHandler

type EdgeSyncHandler struct {
	// contains filtered or unexported fields
}

EdgeSyncHandler serves the hub side of edge-to-cloud sync.

func NewEdgeSyncHandler

func NewEdgeSyncHandler(cfg EdgeSyncHandlerConfig) (*EdgeSyncHandler, error)

NewEdgeSyncHandler validates configuration and returns a ready handler.

Every dependency that carries a security property is required rather than optional: a nil secret lookup or replay guard would silently downgrade authentication, and a handler that half-works is worse than one that refuses to start.

func (*EdgeSyncHandler) RegisterRoutes

func (h *EdgeSyncHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes mounts the hub receive endpoints.

Two independent authentication layers, both mandatory:

  • Arc's API token auth, so the endpoint is not reachable by an anonymous caller even before sync-specific checks run.
  • Per-spoke HMAC inside the handler, which is what actually binds the request to a spoke identity and its content.

The token layer alone is insufficient (it proves a caller has *an* Arc token, not that it is a given spoke), and the HMAC alone would leave the endpoint reachable by anyone who can guess a spoke ID. Both, always.

type EdgeSyncHandlerConfig

type EdgeSyncHandlerConfig struct {
	Receiver *edgesync.Receiver

	// Reconciler answers batch discovery. Required — without it a spoke
	// returning from a long outage would have to probe file by file.
	Reconciler *edgesync.Reconciler

	// SpokeSecrets resolves a spoke's shared secret. Required — without it
	// every request would be unauthenticated.
	SpokeSecrets SpokeSecretLookup

	// Replay consumes nonces so a captured request cannot be replayed inside
	// the freshness window. Required.
	Replay security.ReplayGuard

	// HubID identifies this hub. It is bound into the request MAC, so a
	// request minted for another hub sharing the spoke's secret is rejected.
	HubID string

	// MaxFileBytes caps a single upload. Required and must be > 0.
	MaxFileBytes int64

	AuthManager *auth.AuthManager
	Logger      zerolog.Logger
}

EdgeSyncHandlerConfig configures the handler.

type EdgeSyncImportHandler

type EdgeSyncImportHandler struct {
	// contains filtered or unexported fields
}

EdgeSyncImportHandler exposes the hub's air-gap import controls.

Mounted at /api/v1/bundle-import, deliberately NOT under /api/v1/sync-*. Fiber's Group().Use() matches by string PREFIX, not path segment, so a group at "/api/v1/sync" also matches "/api/v1/sync-import/..." — the spoke-facing HMAC group's body limit and middleware would silently run on these operator routes. A prefix that is not a string-prefix of another group's is the only way to keep them uncoupled.

Different audience from the hub's receive endpoints: those are called by spokes and authenticated per-spoke by HMAC; these are called by an operator who has just plugged in a drive, and are authenticated by an admin token.

func NewEdgeSyncImportHandler

func NewEdgeSyncImportHandler(
	importer *edgesync.Importer,
	policy *edgesync.DestinationPolicy,
	index *edgesync.BundleIndex,
	authManager *auth.AuthManager,
	logger zerolog.Logger,
) (*EdgeSyncImportHandler, error)

NewEdgeSyncImportHandler validates configuration and returns a ready handler.

func (*EdgeSyncImportHandler) RegisterRoutes

func (h *EdgeSyncImportHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes mounts the import controls. Admin-only.

Importing writes a spoke's data into hub storage, and the history endpoint reveals which edge deployments have delivered and when. Neither is something a read-scoped token should reach.

type EdgeSyncSpokeHandler

type EdgeSyncSpokeHandler struct {
	// contains filtered or unexported fields
}

EdgeSyncSpokeHandler exposes the manual sync controls on a spoke.

Distinct from EdgeSyncHandler, which is the hub's receive side. The two have different audiences and different authentication — a hub's endpoints are called by spokes and authenticated per-spoke by HMAC; these are called by an operator on the edge box and authenticated by an Arc admin token. Mounting them together would invite the wrong middleware.

Mounted at /api/v1/spoke-sync, NOT under /api/v1/sync-*. Fiber's Group().Use() matches by string PREFIX, not by path segment, so a group at "/api/v1/sync" also matches "/api/v1/sync-spoke/..." — the hub's body limit and its middleware would silently run on these operator routes. A prefix that is not a string-prefix of another group's is the only way to keep the two genuinely uncoupled.

func NewEdgeSyncSpokeHandler

func NewEdgeSyncSpokeHandler(agent *edgesync.Agent, exporter *edgesync.Exporter, authManager *auth.AuthManager, logger zerolog.Logger) (*EdgeSyncSpokeHandler, error)

NewEdgeSyncSpokeHandler validates configuration and returns a ready handler.

func (*EdgeSyncSpokeHandler) RegisterRoutes

func (h *EdgeSyncSpokeHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes mounts the spoke's manual sync controls.

All admin-only. Triggering a sync moves data off this box and consumes whatever link budget it has; the ledger view exposes which measurements exist and how far behind they are. Neither is something a read-scoped token should reach.

type EstimateResponse

type EstimateResponse struct {
	Success         bool    `json:"success"`
	EstimatedRows   *int64  `json:"estimated_rows"`
	WarningLevel    string  `json:"warning_level"`
	WarningMessage  string  `json:"warning_message,omitempty"`
	ExecutionTimeMs float64 `json:"execution_time_ms"`
	Error           string  `json:"error,omitempty"`
}

EstimateResponse represents the response for query estimation

type ExecuteCQRequest

type ExecuteCQRequest struct {
	StartTime *string `json:"start_time"`
	EndTime   *string `json:"end_time"`
	DryRun    bool    `json:"dry_run"`
}

ExecuteCQRequest represents a request to execute a CQ

type ExecuteCQResponse

type ExecuteCQResponse struct {
	QueryID                int64   `json:"query_id"`
	QueryName              string  `json:"query_name"`
	ExecutionID            string  `json:"execution_id"`
	Status                 string  `json:"status"`
	StartTime              string  `json:"start_time"`
	EndTime                string  `json:"end_time"`
	RecordsRead            *int64  `json:"records_read"`
	RecordsWritten         int64   `json:"records_written"`
	ExecutionTimeSeconds   float64 `json:"execution_time_seconds"`
	DestinationMeasurement string  `json:"destination_measurement"`
	DryRun                 bool    `json:"dry_run"`
	ExecutedAt             string  `json:"executed_at"`
	ExecutedQuery          string  `json:"executed_query,omitempty"`
	// WatermarkStatus: in a cluster that adopted continuous queries, what a
	// manual run did to last_processed_time: "set", "moved" (another run or
	// primary moved it meanwhile; theirs is kept) or "pending" (the cluster
	// has not taken it yet; the next run settles it).
	WatermarkStatus string `json:"watermark_status,omitempty"`
}

ExecuteCQResponse represents the result of executing a CQ

type ExecuteRetentionRequest

type ExecuteRetentionRequest struct {
	DryRun  bool `json:"dry_run"`
	Confirm bool `json:"confirm"`
}

ExecuteRetentionRequest represents a request to execute a policy

type ExecuteRetentionResponse

type ExecuteRetentionResponse struct {
	PolicyID     int64  `json:"policy_id"`
	PolicyName   string `json:"policy_name"`
	DeletedCount int64  `json:"deleted_count"`
	FilesDeleted int    `json:"files_deleted"`
	// SkippedFiles counts files whose stored key could not be resolved to a
	// readable path, so they were neither examined nor deleted and never will
	// be. Omitted when zero, which is every healthy deployment.
	SkippedFiles  int    `json:"skipped_files,omitempty"`
	SkippedReason string `json:"skipped_reason,omitempty"`
	// Cold tier (#168). DeletedCount and FilesDeleted include these. In a dry
	// run they are what a real run would delete. ColdFilesSkipped counts cold
	// files retention could not examine or delete; ColdSkippedReason says why.
	ColdFilesDeleted     int      `json:"cold_files_deleted,omitempty"`
	ColdRowsDeleted      int64    `json:"cold_rows_deleted,omitempty"`
	ColdFilesSkipped     int      `json:"cold_files_skipped,omitempty"`
	ColdSkippedReason    string   `json:"cold_skipped_reason,omitempty"`
	ExecutionTimeMs      float64  `json:"execution_time_ms"`
	DryRun               bool     `json:"dry_run"`
	CutoffDate           string   `json:"cutoff_date"`
	AffectedMeasurements []string `json:"affected_measurements"`
}

ExecuteRetentionResponse represents the result of executing a policy

type ForwardDecision

type ForwardDecision int

ForwardDecision is the outcome of the pre-handler routing check. It separates the two reasons a request is not forwarded — "handle it here" versus "it was already forwarded and this node still cannot serve it" — so the second case produces a deterministic error instead of silently falling through to a local path that is structurally guaranteed to fail.

Splitting these two cases is also what makes the X-Arc-Forwarded-By loop guard non-abusable. The header is client-settable (it survives at the HTTP boundary and is only overwritten on the *outbound* forward in internal/cluster/router.go), so an authenticated caller can set it on a direct request. Previously that collapsed into "do not forward" and the node then attempted local processing it could not complete — a client-triggered routing bypass / self-inflicted failure. Now a present header on a node that cannot route locally yields ForwardAlreadyForwarded, which the handler answers with a clear 508-style error. A genuine peer->peer loop terminates the same way. On a node that *can* serve locally the header is irrelevant and ignored entirely (the common case).

const (
	// ForwardLocal: handle the request on this node.
	ForwardLocal ForwardDecision = iota
	// ForwardToPeer: forward the request to another node.
	ForwardToPeer
	// ForwardAlreadyForwarded: the request carries the forwarded-by marker
	// but this node cannot serve it locally — a loop or a spoofed header.
	// The handler must return an error, never fall through to local.
	ForwardAlreadyForwarded
)

func QueryForwardDecision

func QueryForwardDecision(router *cluster.Router, c *fiber.Ctx) ForwardDecision

QueryForwardDecision returns the full routing decision for a query.

func WriteForwardDecision

func WriteForwardDecision(router *cluster.Router, c *fiber.Ctx) ForwardDecision

WriteForwardDecision returns the full routing decision for a write.

type ForwardIdentityChecker

type ForwardIdentityChecker interface {
	VerifyForwardIdentity(values cluster.ForwardIdentityHeaderValues, method, requestURI string) (bool, cluster.ForwardToken, error)
}

ForwardIdentityChecker verifies a forwarded request's signed identity (*cluster.Coordinator).

type GovernanceHandler

type GovernanceHandler struct {
	// contains filtered or unexported fields
}

GovernanceHandler handles governance policy API operations.

func NewGovernanceHandler

func NewGovernanceHandler(manager *governance.Manager, authManager *auth.AuthManager, licenseClient *license.Client, logger zerolog.Logger) *GovernanceHandler

NewGovernanceHandler creates a new governance handler.

func (*GovernanceHandler) CreatePolicy

func (h *GovernanceHandler) CreatePolicy(c *fiber.Ctx) error

CreatePolicy creates a governance policy for a token. POST /api/v1/governance/policies

func (*GovernanceHandler) DeletePolicy

func (h *GovernanceHandler) DeletePolicy(c *fiber.Ctx) error

DeletePolicy removes the governance policy for a specific token. DELETE /api/v1/governance/policies/:token_id

func (*GovernanceHandler) GetPolicy

func (h *GovernanceHandler) GetPolicy(c *fiber.Ctx) error

GetPolicy returns the governance policy for a specific token. GET /api/v1/governance/policies/:token_id

func (*GovernanceHandler) GetUsage

func (h *GovernanceHandler) GetUsage(c *fiber.Ctx) error

GetUsage returns current usage statistics for a specific token. GET /api/v1/governance/usage/:token_id

func (*GovernanceHandler) ListPolicies

func (h *GovernanceHandler) ListPolicies(c *fiber.Ctx) error

ListPolicies returns all governance policies. GET /api/v1/governance/policies

func (*GovernanceHandler) RegisterRoutes

func (h *GovernanceHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes registers governance API routes.

func (*GovernanceHandler) UpdatePolicy

func (h *GovernanceHandler) UpdatePolicy(c *fiber.Ctx) error

UpdatePolicy updates the governance policy for a specific token. PUT /api/v1/governance/policies/:token_id

type IcebergCatalogDropper

type IcebergCatalogDropper interface {
	DropDatabase(ctx context.Context, database string) error
}

NewDatabasesHandler creates a new databases handler IcebergCatalogDropper removes a dropped database's Iceberg catalog artifacts (#639 item 3). Implemented by iceberg.Exporter; nil when Iceberg export is disabled.

type ImportHandler

type ImportHandler struct {
	// contains filtered or unexported fields
}

ImportHandler handles bulk CSV, Parquet, Line Protocol, and TLE file imports.

All formats parse rows in-process and ingest through the ArrowBuffer pipeline (the same path as streaming writes). No import path issues DuckDB queries against the uploaded file, so imports do not require the upload directory to be in the DuckDB sandbox allowlist. (The allowlist entry that still exists is retained solely for delete.go's S3 COPY ... TO staging — see cmd/arc/main.go.)

func NewImportHandler

func NewImportHandler(logger zerolog.Logger) *ImportHandler

NewImportHandler creates a new ImportHandler. The ArrowBuffer must be set via SetArrowBuffer before any import is served.

func (*ImportHandler) RegisterRoutes

func (h *ImportHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers import API routes. Import endpoints write historical data and use admin-tier auth (not write-tier) — bulk imports can rewrite or supplant existing partitions, so the stricter gate is appropriate.

func (*ImportHandler) SetArrowBuffer

func (h *ImportHandler) SetArrowBuffer(buf *ingest.ArrowBuffer)

SetArrowBuffer sets the ArrowBuffer for Line Protocol import

func (*ImportHandler) SetAuthAndRBAC

func (h *ImportHandler) SetAuthAndRBAC(authManager *auth.AuthManager, rbacManager RBACChecker)

SetAuthAndRBAC sets the auth and RBAC managers. See MsgPackHandler.SetAuthAndRBAC for the full rationale.

func (*ImportHandler) SetRouter

func (h *ImportHandler) SetRouter(router *cluster.Router)

SetRouter configures cluster write forwarding for import requests.

func (*ImportHandler) Stats

func (h *ImportHandler) Stats(c *fiber.Ctx) error

Stats returns import handler statistics

type ImportResult

type ImportResult struct {
	Database          string   `json:"database"`
	Measurement       string   `json:"measurement"`
	RowsImported      int64    `json:"rows_imported"`
	PartitionsCreated int      `json:"partitions_created"`
	TimeRangeMin      string   `json:"time_range_min,omitempty"`
	TimeRangeMax      string   `json:"time_range_max,omitempty"`
	Columns           []string `json:"columns"`
	DurationMs        int64    `json:"duration_ms"`
}

ImportResult holds the result of a bulk import operation

type LPImportResult

type LPImportResult struct {
	Database     string   `json:"database"`
	Measurements []string `json:"measurements"`
	RowsImported int64    `json:"rows_imported"`
	Precision    string   `json:"precision"`
	DurationMs   int64    `json:"duration_ms"`
}

LPImportResult holds the result of a Line Protocol bulk import operation

type LineProtocolHandler

type LineProtocolHandler struct {
	// contains filtered or unexported fields
}

LineProtocolHandler handles Line Protocol write requests

func NewLineProtocolHandler

func NewLineProtocolHandler(buffer *ingest.ArrowBuffer, logger zerolog.Logger) *LineProtocolHandler

NewLineProtocolHandler creates a new Line Protocol handler

func (*LineProtocolHandler) Flush

func (h *LineProtocolHandler) Flush(c *fiber.Ctx) error

Flush triggers a buffer flush

func (*LineProtocolHandler) GetStats

func (h *LineProtocolHandler) GetStats() map[string]int64

GetStats returns stats as a map (for programmatic access)

func (*LineProtocolHandler) Health

func (h *LineProtocolHandler) Health(c *fiber.Ctx) error

Health returns health status

func (*LineProtocolHandler) RegisterRoutes

func (h *LineProtocolHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers Line Protocol routes. All write endpoints are gated by auth.RequireWrite when auth is enabled; the /flush endpoint is admin-gated because it forces a global flush across all shards (operationally heavy + spammable). Per CLAUDE.md, mutating endpoints MUST have an explicit auth middleware.

func (*LineProtocolHandler) SetAuthAndRBAC

func (h *LineProtocolHandler) SetAuthAndRBAC(authManager *auth.AuthManager, rbacManager RBACChecker)

SetAuthAndRBAC sets the auth and RBAC managers. See MsgPackHandler.SetAuthAndRBAC for the full rationale.

func (*LineProtocolHandler) SetRouter

func (h *LineProtocolHandler) SetRouter(router *cluster.Router)

SetRouter sets the cluster router for request forwarding. When set, write requests from reader nodes will be forwarded to writer nodes.

func (*LineProtocolHandler) Stats

func (h *LineProtocolHandler) Stats(c *fiber.Ctx) error

Stats returns Line Protocol handler statistics

func (*LineProtocolHandler) WriteInfluxDB

func (h *LineProtocolHandler) WriteInfluxDB(c *fiber.Ctx) error

WriteInfluxDB handles InfluxDB 2.x compatible write requests POST /api/v2/write?org=myorg&bucket=mybucket&precision=ns

func (*LineProtocolHandler) WriteSimple

func (h *LineProtocolHandler) WriteSimple(c *fiber.Ctx) error

WriteSimple handles simple write requests without query parameters POST /api/v1/write/line-protocol

func (*LineProtocolHandler) WriteV1

func (h *LineProtocolHandler) WriteV1(c *fiber.Ctx) error

WriteV1 handles InfluxDB 1.x compatible write requests POST /write?db=telegraf&rp=default&precision=ns

type MQTTHandler

type MQTTHandler struct {
	// contains filtered or unexported fields
}

MQTTHandler handles MQTT stats and health endpoints

func NewMQTTHandler

func NewMQTTHandler(manager mqtt.Manager, authManager *auth.AuthManager, logger zerolog.Logger) *MQTTHandler

NewMQTTHandler creates a new MQTT handler

func (*MQTTHandler) RegisterRoutes

func (h *MQTTHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers the MQTT stats/health API routes

type MQTTSubscriptionHandler

type MQTTSubscriptionHandler struct {
	// contains filtered or unexported fields
}

MQTTSubscriptionHandler handles MQTT subscription REST API endpoints

func NewMQTTSubscriptionHandler

func NewMQTTSubscriptionHandler(manager mqtt.Manager, authManager *auth.AuthManager, logger zerolog.Logger) *MQTTSubscriptionHandler

NewMQTTSubscriptionHandler creates a new MQTT subscription handler

func (*MQTTSubscriptionHandler) RegisterRoutes

func (h *MQTTSubscriptionHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers the MQTT subscription API routes.

The requireEnabled middleware is applied to the route group so every current and future endpoint is automatically guarded against a nil manager. This is the project policy for MQTT API surfaces — see also MQTTHandler in mqtt.go.

type MeasurementInfo

type MeasurementInfo struct {
	Database    string  `json:"database"`
	Measurement string  `json:"measurement"`
	FileCount   int     `json:"file_count"`
	TotalSizeMB float64 `json:"total_size_mb"`
	StoragePath string  `json:"storage_path"`
}

MeasurementInfo represents information about a measurement

type MeasurementListResponse

type MeasurementListResponse struct {
	Database     string                `json:"database"`
	Measurements []DatabaseMeasurement `json:"measurements"`
	Count        int                   `json:"count"`
}

MeasurementListResponse represents the response for listing measurements

type MsgPackHandler

type MsgPackHandler struct {
	// contains filtered or unexported fields
}

MsgPackHandler handles MessagePack binary protocol endpoints

func NewMsgPackHandler

func NewMsgPackHandler(logger zerolog.Logger, arrowBuffer *ingest.ArrowBuffer, maxPayloadSize int64) *MsgPackHandler

NewMsgPackHandler creates a new MessagePack handler

func (*MsgPackHandler) RegisterRoutes

func (h *MsgPackHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers MessagePack endpoints. The write endpoint is gated by auth.RequireWrite when an auth manager is configured — per CLAUDE.md, mutating endpoints MUST have an explicit write-tier auth middleware. Stats and spec remain at any-authenticated-token level (gated by the global auth middleware in main.go).

func (*MsgPackHandler) SetAuthAndRBAC

func (h *MsgPackHandler) SetAuthAndRBAC(authManager *auth.AuthManager, rbacManager RBACChecker)

SetAuthAndRBAC sets the auth and RBAC managers. Takes the concrete *auth.AuthManager (not the local AuthManager interface) so route- level auth.RequireWrite middleware can be wired without a type assertion that could silently miss on a mock and disable protection. nil = auth disabled (OSS default).

func (*MsgPackHandler) SetRouter

func (h *MsgPackHandler) SetRouter(router *cluster.Router)

SetRouter sets the cluster router for request forwarding. When set, write requests from reader nodes will be forwarded to writer nodes.

type OpenCompactions

type OpenCompactions interface {
	ExcludedFor(ctx context.Context, prefix string) []string
	Open(prefix string) bool
	Generation() uint64
}

OpenCompactions is what the query layer needs from the compactions a read must account for: compaction.OpenJobs on this node, and in a cluster the coordinator's committed jobs and pending local deletes (CombineOpenCompactions).

func CombineOpenCompactions

func CombineOpenCompactions(sources ...OpenCompactions) OpenCompactions

CombineOpenCompactions unions the sources a read must account for (the compaction manager's own jobs, the cluster's staged jobs and pending local deletes). Nil members are skipped; with none left the result is nil, so reads are untouched.

type PeerFlusher

type PeerFlusher interface {
	PeerFlushActive() bool
	FlushPeersForDelete(ctx context.Context, database, measurement string) ([]string, error)
}

PeerFlusher is the coordinator's side of the flush a DELETE asks of the other nodes in shared storage (#1265).

type PendingRegistrations

type PendingRegistrations interface {
	ReplacePending(ctx context.Context, oldPath, newPath string, sizeBytes int64, sha256 string) (cluster.ReplaceOutcome, raft.FileEntry, error)
	Withdraw(ctx context.Context, path string) (bool, cluster.ReplaceOutcome, error)
}

PendingRegistrations is the slice of the registration outbox a DELETE needs: cluster.RegistrationOutbox.

type PolicyRequest

type PolicyRequest struct {
	HotOnly       bool `json:"hot_only,omitempty"`         // Exclude from tiering entirely
	HotMaxAgeDays *int `json:"hot_max_age_days,omitempty"` // nil = use global default
}

PolicyRequest represents a request to create/update a tiering policy 2-tier system: data older than HotMaxAgeDays moves from hot to cold

type PooledBuffer

type PooledBuffer struct {
	Data []byte // The decompressed data (slice of pooled buffer)
	// contains filtered or unexported fields
}

PooledBuffer wraps a decompression buffer that must be returned to pool after use This enables zero-copy decompression by returning the pooled buffer directly

func (*PooledBuffer) Release

func (pb *PooledBuffer) Release()

Release returns the buffer to the pool - MUST be called after use Safe to call multiple times (idempotent)

type QueryHandler

type QueryHandler struct {
	// contains filtered or unexported fields
}

QueryHandler handles SQL query endpoints

func NewQueryHandler

func NewQueryHandler(db *database.DuckDB, storage storage.Backend, logger zerolog.Logger, queryTimeoutSeconds int, slowQueryThresholdMs int) *QueryHandler

NewQueryHandler creates a new query handler queryTimeoutSeconds: timeout for query execution in seconds (0 = no timeout) slowQueryThresholdMs: threshold in milliseconds for slow query WARN logging (0 = disabled)

func (*QueryHandler) InvalidateCaches

func (h *QueryHandler) InvalidateCaches()

InvalidateCaches clears all internal caches (partition pruner and SQL transform cache). This should be called after compaction to prevent stale file references.

func (*QueryHandler) InvalidateCachesQuiet

func (h *QueryHandler) InvalidateCachesQuiet()

InvalidateCachesQuiet is InvalidateCaches without the Info line, for a caller that invalidates around every chunk of a long operation.

func (*QueryHandler) QueryGate503Total

func (h *QueryHandler) QueryGate503Total() int64

QueryGate503Total returns the total number of times the catch-up gate has fired since process start. Zero when the gate is disabled or has never fired. Exposed for Prometheus / metrics dashboards to alert on a non-zero rate without inferring from HTTP error logs.

func (*QueryHandler) RegisterRoutes

func (h *QueryHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers query endpoints

func (*QueryHandler) SetAuthAndRBAC

func (h *QueryHandler) SetAuthAndRBAC(am *auth.AuthManager, rm RBACChecker)

SetAuthAndRBAC sets the auth and RBAC managers for permission checking. Called once at startup before RegisterRoutes (see cmd/arc/main.go), so a plain pointer assignment is sufficient — there is no concurrent reader. Mirrors MsgPackHandler.SetAuthAndRBAC and the other ingest handlers.

func (*QueryHandler) SetCancelOnClientDisconnect

func (h *QueryHandler) SetCancelOnClientDisconnect(enabled bool)

SetCancelOnClientDisconnect controls whether query contexts are cancelled when the client connection closes before the response finishes. Some HTTP clients half-close their write side after sending a request; operators can disable this behavior for those clients.

func (*QueryHandler) SetCluster

func (h *QueryHandler) SetCluster(coordinator *cluster.Coordinator, queryGateOnCatchup bool)

SetCluster wires the cluster coordinator and the query-gate-on-catchup behavior flag into the query handler. Consulted by checkReplicationReady to short-circuit queries with 503 while peer file replication is still draining (#392). Both arguments are optional — nil coordinator OR gate=false makes the gate a no-op.

Initialization-only: must be called once during process startup before the HTTP server starts accepting requests. Order relative to RegisterRoutes does not matter (Fiber resolves field reads at request time, and request handling does not begin until server.Start). Runtime re-wiring is not supported — the fields are read lock-free per request.

The explicit nil branch protects against the typed-nil interface trap: assigning a nil *cluster.Coordinator to an interface field yields a non-nil interface that panics on method call. The current caller in main.go is gated on clusterCoordinator != nil so the branch is dead today, but we keep it so a future caller passing nil literally can't break the gate.

func (*QueryHandler) SetEmptyRangeAnchorScan

func (h *QueryHandler) SetEmptyRangeAnchorScan(enabled bool)

SetEmptyRangeAnchorScan enables answering a proven-empty time range from the measurement's schema anchor alone instead of scanning the whole measurement (#928). Requires the field schema registry; experimental.

func (*QueryHandler) SetFieldSchema

func (h *QueryHandler) SetFieldSchema(r *fieldschema.Registry)

SetFieldSchema installs the measurement field schema registry (#914). When set and enabled, every read_parquet the rewriter emits for a measurement lists the measurement's zero-row schema anchor first, so a field absent from the selected files binds as a typed NULL column instead of failing.

func (*QueryHandler) SetFileTimePruning

func (h *QueryHandler) SetFileTimePruning(enabled bool, margin time.Duration)

SetFileTimePruning enables file-level time pruning on the partition pruner (see internal/pruning/file_time_pruning.go). Called from main.go after construction, like SetStorageBackend.

func (*QueryHandler) SetGovernance

func (h *QueryHandler) SetGovernance(manager *governance.Manager, lc *license.Client)

SetGovernance sets the governance manager and license client for query rate limiting and quotas.

func (*QueryHandler) SetOpenCompactions

func (h *QueryHandler) SetOpenCompactions(oc OpenCompactions)

SetOpenCompactions wires the compaction jobs a read must account for.

func (*QueryHandler) SetQueryRegistry

func (h *QueryHandler) SetQueryRegistry(registry *queryregistry.Registry)

SetQueryRegistry sets the query registry for long-running query management.

func (*QueryHandler) SetRouter

func (h *QueryHandler) SetRouter(router *cluster.Router)

SetRouter sets the cluster router for request forwarding. When set, query requests from nodes that cannot handle queries will be forwarded. Note: Currently writers always process queries locally (CanQuery=true). This is provided for future extensibility (e.g., prefer_readers mode).

func (*QueryHandler) SetTieringManager

func (h *QueryHandler) SetTieringManager(manager *tiering.Manager)

SetTieringManager sets the tiering manager for multi-tier query routing. When set, queries will check both hot and cold tiers for data.

func (*QueryHandler) StartBackgroundWorkers

func (h *QueryHandler) StartBackgroundWorkers(ctx context.Context)

StartBackgroundWorkers spawns the long-lived goroutines the handler needs: today this is the partition pruner cache janitor, which sweeps expired entries from globCache + partitionCache so they don't accumulate over the process lifetime. (Both caches are TTL-only with no max-size cap and no read-side eviction — get() returns "expired" as a miss but leaves the stale entry in the map.)

The workers stop when ctx is cancelled. The handler itself remains usable after that, and InvalidateCaches() (called post-compaction) continues to reset both maps, but expired entries are no longer swept on a schedule — they accumulate until the next compaction flushes everything or the process exits. Production callers should pass a context tied to process lifetime via the shutdown coordinator.

type QueryManagementHandler

type QueryManagementHandler struct {
	// contains filtered or unexported fields
}

QueryManagementHandler handles query management API operations.

func NewQueryManagementHandler

func NewQueryManagementHandler(registry *queryregistry.Registry, authManager *auth.AuthManager, licenseClient *license.Client, logger zerolog.Logger) *QueryManagementHandler

NewQueryManagementHandler creates a new query management handler.

func (*QueryManagementHandler) RegisterRoutes

func (h *QueryManagementHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes registers query management API routes.

type QueryRequest

type QueryRequest struct {
	SQL string `json:"sql"`
}

QueryRequest represents a SQL query request

type QueryResponse

type QueryResponse struct {
	Success         bool                   `json:"success"`
	Columns         []string               `json:"columns"`
	Data            [][]interface{}        `json:"data"`
	RowCount        int                    `json:"row_count"`
	ExecutionTimeMs float64                `json:"execution_time_ms"`
	Timestamp       string                 `json:"timestamp"`
	Error           string                 `json:"error,omitempty"`
	Profile         *database.QueryProfile `json:"profile,omitempty"`
}

QueryResponse represents a SQL query response

type RBACChecker

type RBACChecker interface {
	IsRBACEnabled() bool
	CheckPermission(req *auth.PermissionCheckRequest) *auth.PermissionCheckResult
	CheckPermissionsBatch(reqs []*auth.PermissionCheckRequest) []*auth.PermissionCheckResult
	// CanAccessAnythingIn answers "may this caller enumerate inside this
	// database", which is weaker than CheckPermission with "*" and is the
	// only question a listing gate should ask. See its doc comment in
	// internal/auth.
	CanAccessAnythingIn(tokenInfo *auth.TokenInfo, database, permission string) bool
}

RBACChecker interface for RBAC permission checking

type RBACHandler

type RBACHandler struct {
	// contains filtered or unexported fields
}

RBACHandler handles RBAC-related endpoints

func NewRBACHandler

func NewRBACHandler(authManager *auth.AuthManager, rbacManager *auth.RBACManager, logger zerolog.Logger) *RBACHandler

NewRBACHandler creates a new RBAC handler

func (*RBACHandler) RegisterRoutes

func (h *RBACHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers RBAC-related endpoints

type ReconciliationHandler

type ReconciliationHandler struct {
	// contains filtered or unexported fields
}

ReconciliationHandler exposes the reconciliation engine over HTTP. Mirrors the SchedulerHandler shape: GET /status open to any authenticated token, mutating routes wrapped with RequireAdmin.

func NewReconciliationHandler

func NewReconciliationHandler(
	scheduler ReconciliationSchedulerInterface,
	licenseClient *license.Client,
	authManager *auth.AuthManager,
	logger zerolog.Logger,
) *ReconciliationHandler

NewReconciliationHandler constructs the handler. scheduler may be nil when reconciliation is not available (no clustering, no license, or not configured); in that case all routes return 503.

func (*ReconciliationHandler) RegisterRoutes

func (h *ReconciliationHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes wires the routes under /api/v1/reconciliation. Every route is gated by an explicit auth middleware per CLAUDE.md's "every new API endpoint MUST have auth middleware" rule, then by a license re-check (per CLAUDE.md "License middleware goes after auth middleware") on the mutating routes:

  • /status — RequireRead. NOT license-gated so an expired cluster can still report its disabled state to the dashboard.
  • /trigger — RequireAdmin + requireClusteringLicense (kicks off destructive cluster-wide deletes).
  • /runs/:id — RequireAdmin + requireClusteringLicense (run details include sample paths).

The authManager-nil branch exists for tests and OSS no-auth deployments where the operator has explicitly disabled auth.

type ReconciliationSchedulerInterface

type ReconciliationSchedulerInterface interface {
	IsRunning() bool
	Status() map[string]interface{}
	TriggerNow(ctx context.Context, dryRun bool) (*reconciliation.Run, error)
	FindRun(id string) (*reconciliation.Run, bool)
}

ReconciliationSchedulerInterface is the minimal surface the API needs. Defining it as an interface here (rather than depending on the concrete reconciliation.Scheduler) keeps the API package mockable in tests without dragging in the full reconciliation package.

type RestoreRequest

type RestoreRequest struct {
	BackupID        string `json:"backup_id"`
	RestoreData     *bool  `json:"restore_data"`     // default: true
	RestoreMetadata *bool  `json:"restore_metadata"` // default: true standalone, false on a cluster node (where true is refused)
	RestoreConfig   *bool  `json:"restore_config"`   // default: false (refused on a cluster node)
	Confirm         bool   `json:"confirm"`          // must be true
	// Mode is "merge" (additive, resurrects files deleted since the backup)
	// or "replace" (cluster nodes only: the current files of each restored
	// database are removed through the cluster manifest first). Absent, it
	// is merge, except that a scoped backup (#1084) restored on a cluster
	// node defaults to replace; the response echoes the effective mode.
	// Replace of a scoped backup that holds no data file for one of its
	// databases is refused (400), implicit or explicit: it would only delete.
	Mode string `json:"mode"`
}

RestoreRequest is the request body for POST /api/v1/backup/restore.

type RetentionClusterSource

type RetentionClusterSource interface {
	RetentionPoliciesSeeded() bool
	RetentionPoliciesCollecting() bool
	RetentionPolicy(id int64) (raft.RetentionPolicyEntry, bool)
	RetentionPolicyByName(name string) (raft.RetentionPolicyEntry, bool)
	RetentionPolicies() []raft.RetentionPolicyEntry
	CreateRetentionPolicy(ctx context.Context, p raft.CreateRetentionPolicyPayload) error
	UpdateRetentionPolicy(ctx context.Context, p raft.UpdateRetentionPolicyPayload) error
	DeleteRetentionPolicy(ctx context.Context, id int64) error
	RetentionMirrorState() raft.RetentionMirror
	LocalNodeID() string
}

RetentionClusterSource is the cluster state retention reads and writes once adopted. *cluster.Coordinator satisfies it.

type RetentionCoordinator

type RetentionCoordinator interface {
	BatchFileOpsInManifest(ops []raft.BatchFileOp) error
	// IsPrimaryWriter reports whether this node may execute writer-only mutations.
	// Returns true unconditionally for standalone (coordinator is nil).
	IsPrimaryWriter() bool
	// Role returns a human-readable role string for log messages.
	Role() string
}

RetentionCoordinator is the minimal cluster interface retention needs to propagate file deletes to the Raft manifest and gate execution to the primary writer. Using a minimal interface avoids a compile-time dependency on the cluster package.

type RetentionExecution

type RetentionExecution struct {
	ID                  int64   `json:"id"`
	PolicyID            int64   `json:"policy_id"`
	ExecutionTime       string  `json:"execution_time"`
	Status              string  `json:"status"`
	DeletedCount        int64   `json:"deleted_count"`
	CutoffDate          *string `json:"cutoff_date"`
	ExecutionDurationMs float64 `json:"execution_duration_ms"`
	ErrorMessage        *string `json:"error_message"`
}

RetentionExecution represents an execution history record

type RetentionHandler

type RetentionHandler struct {
	// contains filtered or unexported fields
}

RetentionHandler handles retention policy operations

func NewRetentionHandler

func NewRetentionHandler(storage storage.Backend, duckdb *database.DuckDB, cfg *config.RetentionConfig, licenseClient *license.Client, authManager *auth.AuthManager, logger zerolog.Logger) (*RetentionHandler, error)

NewRetentionHandler creates a new retention handler

func (*RetentionHandler) Close

func (h *RetentionHandler) Close() error

Close releases the database handle if this handler owns it.

When retention shares the auth database (the default), the handle is borrowed and its owner closes it later in the shutdown sequence.

func (*RetentionHandler) ExecutePolicy

func (h *RetentionHandler) ExecutePolicy(ctx context.Context, policyID int64) (*ExecuteRetentionResponse, error)

ExecutePolicy executes a retention policy by ID programmatically (used by scheduler) Returns the execution response and any error

func (*RetentionHandler) GetActivePolicies

func (h *RetentionHandler) GetActivePolicies() ([]RetentionPolicy, error)

GetActivePolicies returns all active retention policies (used by scheduler)

func (*RetentionHandler) GetPolicy

func (h *RetentionHandler) GetPolicy(policyID int64) (*RetentionPolicy, error)

GetPolicy returns a retention policy by ID (used by scheduler)

func (*RetentionHandler) LocalPolicyLister

func (h *RetentionHandler) LocalPolicyLister() cluster.LocalPolicyLister

LocalPolicyLister reads this node's own retention policies for the cluster's adoption, with how often this node ran each, under localMu so a local write is either in the answer or refused by the collect.

func (*RetentionHandler) MirrorClusterPolicies

func (h *RetentionHandler) MirrorClusterPolicies(ctx context.Context) error

MirrorClusterPolicies makes this node's own table equal the cluster's retention policies, once adopted (a no-op before). See reconcileMirror.

func (*RetentionHandler) RegisterRoutes

func (h *RetentionHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers retention endpoints

func (*RetentionHandler) SetClusterSource

func (h *RetentionHandler) SetClusterSource(src RetentionClusterSource)

SetClusterSource makes retention cluster-wide once the cluster adopts it. Called after construction in cluster mode.

func (*RetentionHandler) SetCoordinator

func (h *RetentionHandler) SetCoordinator(c RetentionCoordinator)

SetCoordinator wires the cluster coordinator for manifest updates. Called after construction when cluster mode is enabled.

func (*RetentionHandler) SetOnFilesRemoved

func (h *RetentionHandler) SetOnFilesRemoved(fn func())

SetOnFilesRemoved sets the function called after this handler removed or rewrote parquet files: the query caches hold pruned partition lists, and one naming a removed hour makes DuckDB report "No files found", which a query answers with a retryable 503 (classifyNoFiles) until the cache is cleared.

func (*RetentionHandler) SetTieringManager

func (h *RetentionHandler) SetTieringManager(m *tiering.Manager)

SetTieringManager makes retention apply its policies to the cold tier too.

type RetentionPolicy

type RetentionPolicy struct {
	ID                  int64   `json:"id"`
	Name                string  `json:"name"`
	Database            string  `json:"database"`
	Measurement         *string `json:"measurement"`
	RetentionDays       int     `json:"retention_days"`
	BufferDays          int     `json:"buffer_days"`
	IsActive            bool    `json:"is_active"`
	LastExecutionTime   *string `json:"last_execution_time"`
	LastExecutionStatus *string `json:"last_execution_status"`
	LastDeletedCount    *int64  `json:"last_deleted_count"`
	CreatedAt           string  `json:"created_at"`
	UpdatedAt           string  `json:"updated_at"`
}

RetentionPolicy represents a retention policy

type RetentionPolicyRequest

type RetentionPolicyRequest struct {
	Name          optionalField[string] `json:"name"`
	Database      optionalField[string] `json:"database"`
	Measurement   optionalField[string] `json:"measurement"`
	RetentionDays optionalField[int]    `json:"retention_days"`
	BufferDays    optionalField[int]    `json:"buffer_days"`
	IsActive      optionalField[bool]   `json:"is_active"`
}

RetentionPolicyRequest represents a request to create/update a policy.

Every field distinguishes an absent key from a null one (#1023). Create requires name, database and retention_days; buffer_days and is_active keep their long-standing create defaults of 0 and false when absent. Update is a partial update: an absent key keeps the stored value and a present one is validated and written. Only measurement can be null (it means the whole database, as an empty string always has).

Before #1023 update wrote every column from the body, so a PUT that left out measurement widened a measurement-pinned policy to delete across the whole database, one that left out buffer_days moved the cutoff in by the stored buffer, and one that left out is_active stopped the policy — all with a 200.

type RetentionSchedulerInterface

type RetentionSchedulerInterface interface {
	Status() map[string]interface{}
	TriggerNow(ctx context.Context) error
}

RetentionSchedulerInterface defines the interface for retention scheduler operations

type RetiredReads

type RetiredReads struct {
	// contains filtered or unexported fields
}

RetiredReads (standalone) leaves a file a row DELETE replaced out of reads while it is still in storage next to its replacement: the window between the rewrite's publish and the old file's removal, and, when that removal failed, until the drainer removes it. It is an OpenCompactions source (CombineOpenCompactions).

An entry is added once the replacement is in storage (the replacement holds every row the DELETE kept), so it is excluded at once, and dropped once the old file is gone. At start it is seeded from the retired journal (standalone records are written once the replacement is in storage), so a restart does not reopen the window.

func NewRetiredReads

func NewRetiredReads(store storage.Backend) *RetiredReads

NewRetiredReads returns an empty source over store.

func (*RetiredReads) ExcludedFor

func (r *RetiredReads) ExcludedFor(ctx context.Context, prefix string) []string

ExcludedFor returns the replaced files under prefix still in storage.

func (*RetiredReads) Generation

func (r *RetiredReads) Generation() uint64

Generation moves whenever an entry is added.

func (*RetiredReads) Open

func (r *RetiredReads) Open(prefix string) bool

Open reports whether reads under prefix must not be cached.

func (*RetiredReads) Seed

func (r *RetiredReads) Seed(ctx context.Context) error

Seed adds the journal's records.

func (*RetiredReads) SetOnSeen

func (r *RetiredReads) SetOnSeen(fn func())

SetOnSeen sets what runs the first time an entry is left out of a read (drop DuckDB's object-store listing cache when it is on: a listing cached before the replacement landed would lack it).

type SQLValidationError

type SQLValidationError struct {
	Message string
}

SQLValidationError represents an error from SQL validation

func (*SQLValidationError) Error

func (e *SQLValidationError) Error() string

type SchedulerHandler

type SchedulerHandler struct {
	// contains filtered or unexported fields
}

SchedulerHandler handles scheduler status API endpoints

func NewSchedulerHandler

func NewSchedulerHandler(
	cqScheduler CQSchedulerInterface,
	retentionScheduler RetentionSchedulerInterface,
	licenseClient *license.Client,
	authManager *auth.AuthManager,
	logger zerolog.Logger,
) *SchedulerHandler

NewSchedulerHandler creates a new scheduler handler

func (*SchedulerHandler) RegisterRoutes

func (h *SchedulerHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers scheduler API routes

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server represents the HTTP API server

func NewServer

func NewServer(config *ServerConfig, logger zerolog.Logger) *Server

NewServer creates a new HTTP server with Fiber

func (*Server) GetApp

func (s *Server) GetApp() *fiber.App

GetApp returns the underlying Fiber app (for registering custom routes)

func (*Server) GetMaxPayloadSize

func (s *Server) GetMaxPayloadSize() int64

GetMaxPayloadSize returns the configured maximum payload size in bytes

func (*Server) MarkNotReady

func (s *Server) MarkNotReady()

MarkNotReady flips the readiness flag to false. Called by the graceful-shutdown hook so the load balancer stops routing new traffic to this node before the HTTP listener closes. In-flight requests still drain via Fiber's normal shutdown handling.

func (*Server) MarkReady

func (s *Server) MarkReady()

MarkReady flips the readiness flag to true. Called by main.go once all blocking startup work (notably WAL recovery in Pattern 2 shared-storage multi-writer mode) has completed and the server is safe for the load balancer to route traffic to.

func (*Server) RegisterLogsRoute

func (s *Server) RegisterLogsRoute(authManager *auth.AuthManager)

RegisterLogsRoute registers the admin-authenticated application logs endpoint. It must be called AFTER the global auth middleware is installed so the route sits later in the Fiber stack than the middleware. withAdminAuth returns the admin guard when authManager is non-nil, or a passthrough when it is nil (authentication disabled), preserving no-auth deployment behaviour.

func (*Server) RegisterRoutes

func (s *Server) RegisterRoutes()

RegisterRoutes registers all API routes

func (*Server) SetDBStats

func (s *Server) SetDBStats(fn func() sql.DBStats)

SetDBStats wires the DuckDB connection-pool stats source (database.DuckDB.Stats) into the metrics endpoints.

Sampled when metrics are read rather than pushed on change: sql.DBStats is a point-in-time snapshot that is only meaningful at the instant it is observed, and a background ticker would either sample too coarsely to catch a saturation spike or burn CPU sampling a mutex-guarded struct nobody reads.

The server does not otherwise hold the *database.DuckDB handle, which is why this is a setter rather than a constructor field — same shape as SetStorageStatus above (#809).

func (*Server) SetLicenseStatus

func (s *Server) SetLicenseStatus(fn func() license.LicenseHealth)

SetLicenseStatus wires the /health "license" field source.

func (*Server) SetStorageStatus

func (s *Server) SetStorageStatus(fn func() map[string]database.StorageTierStatus, failReady bool)

SetStorageStatus wires the per-tier storage credential status source (database.DuckDB.StorageCredentialStatus) into /health and, when server.storage_credentials_fail_ready is set, /ready. failReady gates the readiness behavior.

func (*Server) SetWriteReadiness

func (s *Server) SetWriteReadiness(fn func() bool)

SetWriteReadiness wires the predicate behind /ready/write: whether writes should be routed to this node. Left unset — OSS, or any non-clustered deployment — the node accepts writes whenever it is ready at all.

func (*Server) Shutdown

func (s *Server) Shutdown(timeout time.Duration) error

Shutdown gracefully shuts down the server

func (*Server) Start

func (s *Server) Start() error

Start starts the HTTP server

func (*Server) WaitForShutdown

func (s *Server) WaitForShutdown(shutdownTimeout time.Duration)

WaitForShutdown blocks until shutdown signal is received

type ServerConfig

type ServerConfig struct {
	// Host is the bind address. Empty preserves the historical
	// dual-stack wildcard (":<port>" on Linux binds both IPv4 and
	// IPv6 via IPv4-mapped addresses). Set to a specific address
	// (e.g. "127.0.0.1", "::1", "192.0.2.10") to restrict the bind.
	// Explicit "0.0.0.0" forces IPv4-only.
	Host           string
	Port           int
	ReadTimeout    time.Duration
	WriteTimeout   time.Duration
	IdleTimeout    time.Duration
	MaxPayloadSize int64 // Maximum request payload size in bytes
	// NOTE: there is deliberately no ShutdownTimeout here. Shutdown takes its
	// budget as a parameter (see Server.Shutdown); a struct field was set by
	// callers and never read, which made server.shutdown_timeout look wired
	// when it was not (#805).
	// TLS Configuration
	TLSEnabled  bool
	TLSCertFile string
	TLSKeyFile  string

	// ClientRecorder, when set, receives the installation id + version
	// of arcli clients (Arcli-Installation-Id header) for telemetry.
	// nil (telemetry disabled) installs no middleware.
	ClientRecorder ClientRecorder
	// AuthRequired tells the client-identity middleware that the auth
	// middleware runs on this server, so only requests it validated are
	// recorded. False (auth disabled) trusts every served request.
	AuthRequired bool
}

ServerConfig holds server configuration

func DefaultServerConfig

func DefaultServerConfig() *ServerConfig

DefaultServerConfig returns default server configuration

type SpokeSecretLookup

type SpokeSecretLookup func(ctx context.Context, spokeID string) (secret string, ok bool)

SpokeSecretLookup returns the shared secret registered for a spoke.

Per-spoke secrets are the point: revoking one edge must not re-key the fleet. Returning ok=false rejects the request without disclosing whether the spoke is unknown or merely disabled.

func RegistrySpokeSecrets

func RegistrySpokeSecrets(reg *edgesync.Registry, logger zerolog.Logger) SpokeSecretLookup

RegistrySpokeSecrets adapts a spoke registry to the lookup the handler needs.

Every failure — unregistered, disabled, or an undecryptable secret — is collapsed into a single `false`. The handler turns that into the same 401 a bad MAC produces, so an attacker cannot use the response to learn which spoke IDs exist. The reason is logged hub-side, where an operator can see it.

func StaticSpokeSecrets

func StaticSpokeSecrets(secrets map[string]string) SpokeSecretLookup

StaticSpokeSecrets builds a lookup over an in-memory map.

Retained for tests and for a hub exercised without a registry. Production wiring uses RegistrySpokeSecrets.

type TLEHandler

type TLEHandler struct {
	// contains filtered or unexported fields
}

TLEHandler handles streaming TLE (Two-Line Element) write requests. TLE data is parsed into the satellite_tle measurement with orbital elements as fields and satellite identifiers as tags.

func NewTLEHandler

func NewTLEHandler(buffer *ingest.ArrowBuffer, logger zerolog.Logger) *TLEHandler

NewTLEHandler creates a new TLE handler

func (*TLEHandler) RegisterRoutes

func (h *TLEHandler) RegisterRoutes(app *fiber.App)

RegisterRoutes registers TLE routes. The write endpoint is gated by auth.RequireWrite when auth is enabled — per CLAUDE.md, mutating endpoints MUST have an explicit write-tier auth middleware.

func (*TLEHandler) SetAuthAndRBAC

func (h *TLEHandler) SetAuthAndRBAC(authManager *auth.AuthManager, rbacManager RBACChecker)

SetAuthAndRBAC sets the auth and RBAC managers. See MsgPackHandler.SetAuthAndRBAC for the full rationale.

func (*TLEHandler) SetRouter

func (h *TLEHandler) SetRouter(router *cluster.Router)

SetRouter sets the cluster router for request forwarding. When set, write requests from reader nodes will be forwarded to writer nodes.

func (*TLEHandler) Stats

func (h *TLEHandler) Stats(c *fiber.Ctx) error

Stats returns TLE handler statistics

type TLEImportResult

type TLEImportResult struct {
	Database       string   `json:"database"`
	Measurement    string   `json:"measurement"`
	SatelliteCount int      `json:"satellite_count"`
	RowsImported   int64    `json:"rows_imported"`
	ParseWarnings  []string `json:"parse_warnings,omitempty"`
	DurationMs     int64    `json:"duration_ms"`
}

TLEImportResult holds the result of a TLE bulk import operation

type TableReference

type TableReference struct {
	Database    string
	Measurement string
}

TableReference represents a database.measurement reference extracted from SQL

type TieringHandler

type TieringHandler struct {
	// contains filtered or unexported fields
}

TieringHandler handles tiered storage API operations

func NewTieringHandler

func NewTieringHandler(manager *tiering.Manager, authManager *auth.AuthManager, licenseClient *license.Client, logger zerolog.Logger) *TieringHandler

NewTieringHandler creates a new tiering handler

func (*TieringHandler) GetFiles

func (h *TieringHandler) GetFiles(c *fiber.Ctx) error

GetFiles returns files by tier GET /api/v1/tiering/files Query params: tier (optional), database (optional), limit (optional)

func (*TieringHandler) GetStats

func (h *TieringHandler) GetStats(c *fiber.Ctx) error

GetStats returns migration statistics GET /api/v1/tiering/stats

func (*TieringHandler) GetStatus

func (h *TieringHandler) GetStatus(c *fiber.Ctx) error

GetStatus returns the current tiering status GET /api/v1/tiering/status

func (*TieringHandler) RegisterRoutes

func (h *TieringHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes registers tiering API routes

func (*TieringHandler) ScanFiles

func (h *TieringHandler) ScanFiles(c *fiber.Ctx) error

ScanFiles registers every file in hot storage and, on any node with a usable cold tier, first reads the cold listing — which is what rebuilds cold rows this node has lost and, on a cluster, what shows it the files other nodes moved (#1179). POST /api/v1/tiering/scan

Bounded by tiered_storage.scan_timeout, the same budget the startup scan and the pre-migration scan use (#1154). The budget is read from the manager rather than carried on the handler so it cannot be a zero here.

Synchronous on purpose. c.Context() is the right parent because it is what makes a server shutdown cancel an in-flight scan, and the handler blocks until the scan returns, so the context never outlives the request. Note it is NOT cancelled when the client disconnects: fasthttp closes RequestCtx.Done only on shutdown, which is why a second scan is refused below rather than left to pile up behind an abandoned curl.

func (*TieringHandler) TriggerMigration

func (h *TieringHandler) TriggerMigration(c *fiber.Ctx) error

TriggerMigration triggers a manual migration POST /api/v1/tiering/migrate

type TieringPoliciesHandler

type TieringPoliciesHandler struct {
	// contains filtered or unexported fields
}

TieringPoliciesHandler handles tiering policy API operations

func NewTieringPoliciesHandler

func NewTieringPoliciesHandler(manager tieringPolicyManager, authManager *auth.AuthManager, licenseClient *license.Client, logger zerolog.Logger) *TieringPoliciesHandler

NewTieringPoliciesHandler creates a new tiering policies handler

func (*TieringPoliciesHandler) DeletePolicy

func (h *TieringPoliciesHandler) DeletePolicy(c *fiber.Ctx) error

DeletePolicy removes the custom policy for a database (reverts to global defaults) DELETE /api/v1/tiering/policies/:database

func (*TieringPoliciesHandler) GetEffectivePolicy

func (h *TieringPoliciesHandler) GetEffectivePolicy(c *fiber.Ctx) error

GetEffectivePolicy returns the effective policy for a database (resolved with global defaults) GET /api/v1/tiering/policies/:database/effective

func (*TieringPoliciesHandler) GetPolicy

func (h *TieringPoliciesHandler) GetPolicy(c *fiber.Ctx) error

GetPolicy returns the policy for a specific database GET /api/v1/tiering/policies/:database

func (*TieringPoliciesHandler) ListPolicies

func (h *TieringPoliciesHandler) ListPolicies(c *fiber.Ctx) error

ListPolicies returns all custom tiering policies GET /api/v1/tiering/policies

func (*TieringPoliciesHandler) RegisterRoutes

func (h *TieringPoliciesHandler) RegisterRoutes(app fiber.Router)

RegisterRoutes registers tiering policy API routes

func (*TieringPoliciesHandler) SetPolicy

func (h *TieringPoliciesHandler) SetPolicy(c *fiber.Ctx) error

SetPolicy creates or updates the policy for a database PUT /api/v1/tiering/policies/:database

type TriggerMigrationRequest

type TriggerMigrationRequest struct {
	FromTier    string `json:"from_tier,omitempty"`   // Optional: "hot" (2-tier system)
	ToTier      string `json:"to_tier,omitempty"`     // Optional: "cold" (2-tier system)
	Database    string `json:"database,omitempty"`    // Optional: filter by database
	Measurement string `json:"measurement,omitempty"` // Optional: filter by measurement
	DryRun      bool   `json:"dry_run,omitempty"`     // If true, only report what would be migrated
}

TriggerMigrationRequest represents a manual migration request

type UpdateTokenRequest

type UpdateTokenRequest struct {
	Name        *string   `json:"name,omitempty"`
	Description *string   `json:"description,omitempty"`
	Permissions *[]string `json:"permissions,omitempty"` // nil = don't change, empty array = clear permissions (RBAC-only)
	ExpiresIn   *string   `json:"expires_in,omitempty"`
}

UpdateTokenRequest represents a token update request

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL