query

package
v0.18.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 51 Imported by: 0

README

Portal query engine

This is the query engine vendored from github.com/lab47/portal/query at commit 09f7eaaea5b174cae1b56958405681bf89ea85e3 (v0.0.0-20261007004044-09f7eaaea5b1). It includes the upstream source and tests, without Portal's transport, authentication, or CLI. The package name remains query; import it as:

import query "miren.dev/runtime/pkg/portalquery"

query.Revision identifies this vendored language revision and its Miren adaptations. Update the upstream hash when re-vendoring and bump the Miren suffix for local changes to syntax, available fields, or query semantics. Runner query RPCs return this revision even on parse/execution errors, so clients can detect version skew with their local reference.

Local adaptations: external-package test imports and symbol-test function names use the Miren import path. The documented rollup example and its parser test use the local README and Miren CLI. The kernel loss-counter test filters its pinned thread rather than all process threads so runtime syscalls cannot inflate the expected count. PID-sensitive live eBPF tests require the host PID namespace, since kernel events use host IDs, and are skipped in nested namespaces such as iso. Process-symbol fixtures require matching procfs and filesystem executable identities; overlay filesystems that expose different identities are skipped. Tracepoint field offsets and sizes are parsed at their destination bit widths to reject integer overflow before building eBPF instructions. Upstream lint exceptions are scoped in .golangci.yml to preserve the other copied code. The upstream repository has no LICENSE or NOTICE file at this revision; no upstream licensing terms are inferred by this copy.

Rollups and computed columns

Use rollup by to keep a subset of a table's output group names, either when emitting it or on either side of a join. This avoids collecting the source twice just to get coarser groups. For example:

miren runner query node-a '
disk:completion where device_name = "nvme0n1" {
  @io[owner: io.cgroup.path, operation: operation] = {ops: count(), sectors: sum(sectors), p99: percentile(duration_ns, 99)}
}
cgroups {
  @kernel[owner: path] = {write_ops: rate(io.write_ios)}
}
after 30s {
  emit @io;
  emit @io rollup by owner full join @kernel on owner
    select ops_per_s = io.ops / window.seconds,
           kib_per_s = io.sectors * 512 / 1024 / window.seconds,
           ratio = ops_per_s / kernel.write_ops
    order by kib_per_s desc limit 20
}'

Inventory-driven snapshot queries (agent reference)

Use query.Engine to correlate a registered custom inventory snapshot with a built-in sampled source before collection and aggregation. This is a lookup, not a join of completed aggregate tables. No inventory names or fields are built into Portal.

Syntax

TARGET using (INVENTORY [where FIELD = VALUE [and ...]]) on TARGET_KEY = INVENTORY_KEY
  [where TARGET_FILTER = VALUE [and ...]]
  [FUNCTION(FIELD) [, FUNCTION(FIELD) ...] over WINDOW [every INTERVAL] [by FIELD, ...]]

Inventory predicates use the existing custom-source equality and membership syntax (app = "my-app", state in (running, starting)). Custom snapshot collectors must apply MonitorRequest.Filters, as for ordinary custom snapshots. The clause precedes the target's own where and action/aggregate. on accepts one exact string key, not numeric coercion, glob matching, composite keys, expressions, subqueries, or references to completed aggregate tables.

Targets are built-in snapshot sources with a string identity field:

Target Target key
cgroups path (not id)
network, cpu, sensors name
containers id
gpu uuid

The inventory can be any engine-local registered source with Snapshots. It need not have Events or support sampled aggregation. Nested using, event targets, aggregate inventories, and additional cgroups where path = ... filters are rejected. Non-cgroup target filters retain their normal semantics.

App I/O: exact examples for Miren

Given a registered sandboxes source whose flat rows contain app, cgroup, container_id, sandbox_id, version, pid, and state:

cgroups using (sandboxes where app = "my-app") on path = cgroup
  where result.format = rows
  rate(io.write_bytes), rate(io.write_ios) over 10s every 1s by inventory.app

This returns total charged bytes/s and operations/s for the selected app's containers, not their mean, and not lifetime counters divided by the window. Read I/O uses rate(io.read_bytes) / rate(io.read_ios); discard I/O uses rate(io.discard_bytes) / rate(io.discard_ios). There are at most 8 metrics.

Selector/block equivalent, with named output columns and app alias:

cgroups using (sandboxes where app = "my-app") on path = cgroup {
  @io[app: inventory.app] = {
    write_bytes_per_second: rate(io.write_bytes),
    write_ios_per_second: rate(io.write_ios)
  }
}
after 10s { emit @io }

Block syntax uses the default 1s sample interval. after returns one finite result; periodic reporting blocks are still event-only. Correlated selectors can also participate in ordinary multi-selector scripts and completed-table reports.

To inspect current lifetime counters per cgroup without sampling:

cgroups using (sandboxes where app = "my-app") on path = cgroup

Snapshot results use Snapshot.Data / JSON data, an array of flat objects with target fields (path, id, io.write_bytes, io.write_ios, etc.) and inventory fields qualified as inventory.app, inventory.container_id, etc. Dot-containing keys are literal flat keys; for jq use .data[] | .["io.write_bytes"], not .data[].io.write_bytes. Missing controller metrics remain absent, not zero. For this correlated snapshot there is no separate cgroups array.

Aggregates retain existing result shapes: compact syntax above uses Snapshot.Aggregation.Rows / JSON aggregation.rows; each row has a group object and metric values array in expression order. Block syntax uses named metric values. Inventory fields are dimensions for grouping/distinct count, not numeric metrics. Add path or inventory.container_id to by for per-container results. Use rate(counter) for app totals: avg(counter_per_second) computes an average of container observations, while sum(counter_per_second) sums over time as well as containers. Neither is an app's total window-average rate. Raw cumulative counters cannot be passed to sum/avg.

Execution and finite-window semantics

  1. Validate the complete request and bind its registered inventory source.
  2. Collect the filtered inventory once per correlated selector and freeze its rows for that selector's window. Multi-selector scripts resolve each selector's inventory independently, not as an atomic shared snapshot.
  3. Match inventory keys to target identities exactly. For cgroups, collect each unique selected path separately; never collect all cgroups then filter an already-aggregated total. Exact collection prunes other branches/descendants. Other built-ins collect their normally filtered snapshot and then match rows.
  4. Carry the frozen inventory fields onto matched target rows and apply the existing sampler/reducer, preserving target identity (cgroups: path + directory ID), not app name, as the counter-baseline key.

The aggregation window is [start, start + WINDOW), maximum 1h, and includes inventory collection time. After inventory completes, the first target snapshot is collected immediately if time remains. Later snapshots follow the original window's interval grid; slow reads skip ticks, never overlap or catch up in bursts. Default interval is 1s, minimum 100ms; derived rates require WINDOW > INTERVAL. Reads finishing at or beyond the window end are discarded. There is no mandatory end-boundary read. A window too short for two observations cannot yield a rate.

rate(counter) sums valid per-target deltas and divides by the union of their observed intervals within each output group. With concurrent container samples, this is the sum of container rates, not their mean. Gaps are not charged as zero and unobserved intervals are not extrapolated to the full window. Targets are read serially, not atomically; a sample's timestamp is collection completion.

New inventory rows appearing during the window are not discovered. Stopped rows remain selected: absent cgroups contribute no sample. Reappearing paths require fresh baselines after a missing observation; recreated directories have new IDs and cannot bridge lifetimes. Counter resets, missing controller/device counters, or changed device sets suppress the affected interval's rate. Lifetime snapshot counters include activity before the query, including descendants; they are not the bytes written during the observation window. Frozen inventory is attribution at query start, not a guarantee that a reused path still belongs to the same app.

No matches means an empty snapshot or grouped aggregate, not a synthetic zero app row. Ungrouped aggregates retain their existing empty-window defaults (e.g. count is 0 and rate is null). Missing/empty/null inventory keys are ignored. An inventory collector that returns a deadline error at the window end yields an empty aggregate; caller cancellation/deadline errors propagate, never a partial success. Other inventory or target errors fail the entire query. Collectors must honor the supplied context; the engine does not spawn detached goroutines to interrupt a collector that ignores cancellation. A non-aggregate snapshot has only its caller's deadline, not an implicit window.

Multiplicity, overlap, and collection limits

  • Multiple disjoint cgroups for one app all contribute, including pause containers if the inventory selects them. App filtering does not infer container roles.
  • Identical inventory rows with the same target key are deduplicated. Different rows sharing a target key fail as ambiguous ownership, even if only an unused inventory field differs. Filter or deduplicate inventory rather than multiply target metrics. Duplicate physical target identities also fail.
  • Selecting a cgroup and any descendant fails rather than summing overlapping hierarchical accounting. This also applies across different selected apps. Choose disjoint workload roots; the engine neither subtracts child counters nor silently discards either owner. A parent alone includes its descendants.
  • Cgroup keys must be canonical absolute paths relative to the visible cgroup v2 mount (e.g. /apps/worker), not host filesystem paths or globs. Symlink aliases, .., repeated/trailing slashes, and glob metacharacters are not selectors.
  • Inventory data must be a JSON array of flat objects, containing only declared scalar fields (strings, numbers, booleans, null); the matching key must be a string. A typed Go slice of structs with flat JSON tags works too. Return [] for an empty inventory, not null; null rows are also rejected.
  • At most 4096 inventory rows before deduplication, 7 MiB serialized inventory, 4096 target records per collected snapshot, and 4096 matched records per tick. Up to 4096 distinct cgroup paths are read serially each tick. Narrow inventory filters for large apps; slow collection reduces sampling coverage.
  • Existing reducer limits remain: 4096 retained metric groups, 65536 retained values, at most 4 grouping fields, and 4096 bytes of query text. Remote output retains the transport's 7 MiB limit. Limits fail, not silently truncate; a result limit affects presentation, not collection or ownership validation.

Go integration

Register inventory before using the engine, then use engine.ParseMonitorQuery (not package-level query.ParseMonitorQuery, which cannot know custom sources) and engine.Query(ctx, request). The serialized API equivalent is:

request := query.MonitorRequest{
    Source: "cgroups", Mode: "aggregate",
    Using: &query.SnapshotCorrelation{
        Inventory: query.MonitorRequest{
            Source: "sandboxes", Mode: "snapshot",
            Filters: []query.SourceFilter{{Field: "app", Values: []string{"my-app"}}},
        },
        Field: "path", Key: "cgroup",
    },
    Aggregation: &query.AggregationRequest{
        Function: "rate", Field: "io.write_bytes",
        Window: 10*time.Second, Every: time.Second,
        GroupBy: []string{"inventory.app"}, Compact: true,
    },
}
result, err := engine.Query(ctx, request)

Engine.Validate and Engine.Metadata rebind inventory registrations after JSON transport. Inventory fields appear in grouping metadata. Engines lacking the registration reject the request. Update Portal on both sides of any request transport to retain the new using field. Miren uses this vendored engine with the new expression/result decoding, not an event implementation or a sampler for sandboxes. Its existing snapshot collector must honor filters, return the declared flat fields, and use canonical cgroup paths. Portal does not add Miren-specific authorization or inventory discovery.

Documentation

Index

Constants

View Source
const (
	MaxAggregateGroups  = maxAggregateGroups
	MaxAggregateValues  = maxAggregateValues
	MaxAggregateMetrics = maxAggregateMetrics
)

Aggregation resource limits apply equally to local and remote execution.

View Source
const DefaultSampleInterval = time.Second
View Source
const MinSampleInterval = 100 * time.Millisecond
View Source
const Revision = "09f7eaaea5b174cae1b56958405681bf89ea85e3-miren.1"

Revision identifies the vendored query language and its local adaptations. Update the upstream hash when re-vendoring and bump the Miren suffix whenever local changes alter accepted syntax, fields, or query semantics.

Variables

This section is empty.

Functions

func CollectEvents

func CollectEvents(ctx context.Context, request MonitorRequest, emit func(Event) error) error

CollectEvents is the built-in EventSource. It accepts an already validated event selection; use Engine.Monitor for validation, filtering and timestamps.

func FormatTAI64N

func FormatTAI64N(t time.Time) string

FormatTAI64N formats a UTC time as a resumable event timestamp.

func StampEvent

func StampEvent(event *Event, last *time.Time)

StampEvent assigns a strictly increasing cursor for a stream. A tie or backward adjustment advances the previous timestamp by one nanosecond.

func ValidateTAI64N

func ValidateTAI64N(text string) error

ValidateTAI64N validates a timestamp cursor.

Types

type AggregateCapability

type AggregateCapability struct {
	Name        string `json:"name"`
	Syntax      string `json:"syntax"`
	FieldType   string `json:"field_type"` // none, scalar, number (event metrics remain integers)
	Description string `json:"description"`
}

type AggregateComputedColumn

type AggregateComputedColumn struct {
	Name       string              `json:"name"`
	Expression AggregateExpression `json:"expression"`
}

type AggregateCount

type AggregateCount struct {
	Group map[string]json.RawMessage `json:"group"`
	Count uint64                     `json:"count"`
}

type AggregateExpression

type AggregateExpression struct {
	Op    string                `json:"op"`
	Value string                `json:"value,omitempty"`
	Args  []AggregateExpression `json:"args,omitempty"`
}

AggregateExpression is a bounded arithmetic tree over completed numeric metrics. Op is number, field, neg, +, -, *, or /; leaf nodes use Value.

type AggregateMetric

type AggregateMetric struct {
	Name       string  `json:"name,omitempty"`
	Function   string  `json:"function"`
	Field      string  `json:"field,omitempty"`
	Percentile float64 `json:"percentile,omitempty"`
}

AggregateMetric selects one reduction in a shared window and grouping.

type AggregateReport

type AggregateReport struct {
	Left        string                    `json:"left"`
	Right       string                    `json:"right,omitempty"`
	Kind        string                    `json:"kind,omitempty"` // inner, left, full; empty without Right
	On          []string                  `json:"on,omitempty"`
	LeftRollup  []string                  `json:"left_rollup,omitempty"`
	RightRollup []string                  `json:"right_rollup,omitempty"`
	Select      []AggregateComputedColumn `json:"select,omitempty"` // appended numeric columns, evaluated before sorting/limit
	Sort        string                    `json:"sort,omitempty"`
	Limit       int                       `json:"limit,omitempty"`
	Ascending   bool                      `json:"ascending,omitempty"`
}

AggregateReport emits a table or a one-to-one join of two completed tables. Sort is a metric name (unambiguous) or TABLE.METRIC; limits apply after joining.

type AggregateRow

type AggregateRow struct {
	Group  map[string]json.RawMessage `json:"group"`
	Values []json.RawMessage          `json:"values"` // Go values align with Columns; named table JSON uses a metric-name object, legacy JSON an array
}

type AggregateValue

type AggregateValue struct {
	Group map[string]json.RawMessage `json:"group"`
	Value json.RawMessage            `json:"value"` // JSON number, histogram object, or null
}

type AggregationRequest

type AggregationRequest struct {
	Window       time.Duration     `json:"window"` // nanoseconds
	GroupBy      []string          `json:"group_by,omitempty"`
	Function     string            `json:"function,omitempty"` // count, sum, avg, min, max, count_distinct, percentile, hist, rate
	Field        string            `json:"field,omitempty"`
	Percentile   float64           `json:"percentile,omitempty"` // 0–100, only for percentile
	Every        time.Duration     `json:"every,omitempty"`      // Snapshot sampling interval; zero defaults to 1s for snapshot-only sources.
	Metrics      []AggregateMetric `json:"metrics,omitempty"`    // Alternative to Function/Field/Percentile; all share Window/Every/GroupBy.
	Compact      bool              `json:"compact,omitempty"`
	Limit        int               `json:"limit,omitempty"`       // first rows in SortMetric order; zero is unlimited
	Nonzero      bool              `json:"nonzero,omitempty"`     // omit rows whose every metric is numeric zero
	SortMetric   int               `json:"sort_metric,omitempty"` // zero-based metric index, default first
	Ascending    bool              `json:"ascending,omitempty"`   // compact rows sorted low-to-high; default descending, nulls always last
	Table        string            `json:"table,omitempty"`
	GroupAliases map[string]string `json:"group_aliases,omitempty"`
	ReportEvery  time.Duration     `json:"report_every,omitempty"` // finite tumbling event buckets; Window is total observation duration
}

AggregationRequest reduces matching events or sampled snapshots during a new window. An omitted Function selects count for compatibility.

type AggregationResult

type AggregationResult struct {
	Table                string                          `json:"table,omitempty"`
	Join                 *AggregateReport                `json:"join,omitempty"`
	Start                time.Time                       `json:"start"`
	End                  time.Time                       `json:"end"`
	GroupBy              []string                        `json:"group_by"`
	Counts               []AggregateCount                `json:"counts,omitempty"`
	Function             string                          `json:"function,omitempty"`
	Field                string                          `json:"field,omitempty"`
	Percentile           float64                         `json:"percentile,omitempty"`
	Values               []AggregateValue                `json:"values,omitempty"`
	Every                time.Duration                   `json:"every,omitempty"`
	Collection           *CollectionStats                `json:"collection,omitempty"`
	Collections          map[string]*CollectionStats     `json:"collections,omitempty"` // join: separate source diagnostics, never summed
	StackCoverage        *StackCoverageReport            `json:"stack_coverage,omitempty"`
	StackCoverageByTable map[string]*StackCoverageReport `json:"stack_coverage_by_table,omitempty"` // join: preserve each source's received-event coverage
	Metrics              []*AggregationResult            `json:"metrics,omitempty"`                 // In request order; each has the same window/grouping. Single queries retain their old shape.
	Columns              []AggregateMetric               `json:"columns,omitempty"`
	Rows                 []AggregateRow                  `json:"rows,omitempty"`
	TotalGroups          int                             `json:"total_groups,omitempty"`
	OmittedZeroGroups    int                             `json:"omitted_zero_groups,omitempty"`
	// contains filtered or unexported fields
}

AggregationResult describes a half-open server ingestion window [Start, End).

func (AggregationResult) MarshalJSON

func (a AggregationResult) MarshalJSON() ([]byte, error)

Keep the selected result collection visible even for an empty grouped window.

func (*AggregationResult) UnmarshalJSON

func (a *AggregationResult) UnmarshalJSON(data []byte) error

Decode both row wire formats into the same column-aligned Go representation.

type CPUInfo

type CPUInfo struct {
	Name   string  `json:"name"`
	User   float64 `json:"user"`
	System float64 `json:"system"`
	Idle   float64 `json:"idle"`
	IOWait float64 `json:"iowait"`
	Steal  float64 `json:"steal"`
	Total  float64 `json:"total"` // Includes nice/IRQ/softIRQ accounting, without double-counting guest time.
}

CPUInfo contains cumulative seconds, not instantaneous utilization percentages.

type Capabilities

type Capabilities struct {
	Version     int                   `json:"version"`
	OS          string                `json:"os"`
	Arch        string                `json:"arch"`
	Syntax      string                `json:"syntax"`
	Notes       []string              `json:"notes"`
	Sources     []SourceCapability    `json:"sources"`
	Aggregates  []AggregateCapability `json:"aggregates"`
	Limits      CapabilityLimits      `json:"limits"`
	EventFields []FieldCapability     `json:"event_fields"`
}

Capabilities is a versioned, server-generated query reference. Authorization is evaluated for the requesting certificate; runtime dependencies are not probed.

type CapabilityLimits

type CapabilityLimits struct {
	QueryBytes              int    `json:"query_bytes"`
	MaxWindow               string `json:"max_window"`
	GroupByFields           int    `json:"group_by_fields"`
	Groups                  int    `json:"groups"`
	RetainedAggregateValues int    `json:"retained_aggregate_values"`
	AggregateMetrics        int    `json:"aggregate_metrics"`
	TracepointFields        int    `json:"tracepoint_fields"`
	SyscallFilters          int    `json:"syscall_filters"`
	PacketCaptureBytes      int    `json:"packet_capture_bytes"`
	DefaultMonitorTTL       string `json:"default_monitor_ttl"`
	RegisteredMonitors      int    `json:"registered_monitors"`
	MonitorRingEvents       int    `json:"monitor_ring_events"`
	MonitorEventBytes       int    `json:"monitor_event_bytes"`
}

type CapturedStack

type CapturedStack struct {
	Frames            []SymbolFrame `json:"frames,omitempty"`
	Error             string        `json:"error,omitempty"`
	CaptureErrorCode  int64         `json:"capture_error_code,omitempty"`  // signed bpf_get_stackid result, not a symbolization error
	DepthLimitReached bool          `json:"depth_limit_reached,omitempty"` // possibly truncated; exact-depth complete stacks are indistinguishable
}

type CgroupIO

type CgroupIO struct {
	ReadBytes    *uint64          `json:"read_bytes,omitempty"`
	WriteBytes   *uint64          `json:"write_bytes,omitempty"`
	DiscardBytes *uint64          `json:"discard_bytes,omitempty"`
	ReadIOs      *uint64          `json:"read_ios,omitempty"`
	WriteIOs     *uint64          `json:"write_ios,omitempty"`
	DiscardIOs   *uint64          `json:"discard_ios,omitempty"`
	Devices      []CgroupIODevice `json:"devices"`
}

type CgroupIODevice

type CgroupIODevice struct {
	Device   string            `json:"device"`
	Counters map[string]uint64 `json:"counters"`
}

type CgroupInfo

type CgroupInfo struct {
	Path             string    `json:"path"`
	ID               string    `json:"id"` // Filesystem device/inode identity, used to detect recreated groups.
	CPUSeconds       *float64  `json:"cpu_seconds,omitempty"`
	CPULimitCores    *float64  `json:"cpu_limit_cores,omitempty"`
	MemoryBytes      *uint64   `json:"memory_bytes,omitempty"`
	MemoryLimitBytes *uint64   `json:"memory_limit_bytes,omitempty"`
	MemoryAnonBytes  *uint64   `json:"memory_anon_bytes,omitempty"`
	MemoryFileBytes  *uint64   `json:"memory_file_bytes,omitempty"`
	PIDsCurrent      *uint64   `json:"pids_current,omitempty"` // Counts tasks, including threads.
	IO               *CgroupIO `json:"io,omitempty"`
}

CgroupInfo describes a visible Linux cgroup v2 directory. Usage includes descendants; limits are configured locally, not effective ancestor limits.

type CollectionStats

type CollectionStats struct {
	RingBufferDropped       uint64 `json:"ring_buffer_dropped"`
	StackCaptureFailures    uint64 `json:"stack_capture_failures"`
	StackCollisions         uint64 `json:"stack_collisions"`
	PairingFailures         uint64 `json:"pairing_failures"`
	UnmatchedExits          uint64 `json:"unmatched_exits"`
	BlockIssues             uint64 `json:"block_issues,omitempty"`
	BlockCompletions        uint64 `json:"block_completions,omitempty"`
	BlockReissues           uint64 `json:"block_reissues,omitempty"`
	BlockPartialCompletions uint64 `json:"block_partial_completions,omitempty"`
	BlockFinalCompletions   uint64 `json:"block_final_completions,omitempty"`
}

CollectionStats are cumulative subscription counters, not per-event deltas. StackCaptureFailures includes StackCollisions. Kernel drops are not per-group.

type ContainerInfo

type ContainerInfo struct {
	ID    string `json:"id"`
	Name  string `json:"name"`
	Image string `json:"image"`
	State string `json:"state"`
}

type CustomSource

type CustomSource struct {
	Fields        []string
	NumericFields []string
	Events        EventSource
	Snapshots     SnapshotSource
}

CustomSource defines an integration's source. Fields are the flat keys in Event.Fields; NumericFields is the subset usable in numeric aggregations and comparisons. Events and Snapshots independently enable streaming/aggregation and one-shot snapshots. Snapshot collectors apply Filters themselves. Configure Sources before using an Engine and do not mutate it while in use.

type DiskEvent

type DiskEvent struct {
	Device        uint32  `json:"device"`
	Sector        uint64  `json:"sector"`
	Sectors       uint32  `json:"sectors"`
	Operation     string  `json:"operation"`
	RWBS          string  `json:"rwbs,omitempty"`
	DeviceName    string  `json:"device_name,omitempty"`
	RequestFlags  *uint32 `json:"request_flags,omitempty"`
	DurationNS    *uint64 `json:"duration_ns,omitempty"`
	Status        *uint32 `json:"status,omitempty"`       // completion blk_status_t; zero is success, not errno
	IOCgroupID    *uint64 `json:"io_cgroup_id,omitempty"` // kernel charged blkcg kernfs ID, not issuing task membership
	IOCgroupPath  string  `json:"io_cgroup_path,omitempty"`
	IOCgroupError string  `json:"io_cgroup_error,omitempty"`
}

DiskEvent describes a block request issued to a device. Sector units are 512 bytes.

type DiskFilter

type DiskFilter struct {
	Device    uint32 `json:"device,omitempty"`
	Operation string `json:"operation,omitempty"` // read, write, discard, or flush
}

DiskFilter selects block requests by kernel device ID and operation.

type Engine

type Engine struct {
	Events    EventSource
	Snapshots SnapshotSource
	Sources   map[string]CustomSource // engine-local names; built-in names are reserved
}

Engine executes local queries. Nil sources use the built-in collectors.

func (Engine) Metadata

func (e Engine) Metadata(request MonitorRequest) (SourceMetadata, error)

Metadata returns field metadata including this engine's custom sources.

func (Engine) Monitor

func (e Engine) Monitor(ctx context.Context, request MonitorRequest, onEvent func(Event) error) error

Monitor validates and streams filtered, timestamped events.

func (Engine) ParseMonitorQuery

func (e Engine) ParseMonitorQuery(text string) (MonitorRequest, error)

ParseMonitorQuery compiles the DSL with this engine's custom sources.

func (Engine) Query

func (e Engine) Query(ctx context.Context, request MonitorRequest) (Snapshot, error)

Query normalizes query mode, resolves syscall names for the native ABI, and dispatches snapshots, sampled aggregates, and event aggregates.

func (Engine) Validate

func (e Engine) Validate(request MonitorRequest) error

Validate validates a request against this engine's registered sources.

type Event

type Event struct {
	Time        time.Time        `json:"time"`
	TAI64N      string           `json:"tai64n"`
	PID         uint32           `json:"pid,omitempty"`
	TID         uint32           `json:"tid,omitempty"`
	Name        string           `json:"name,omitempty"`         // current task comm, up to 15 bytes; may differ between threads
	ProcessName string           `json:"process_name,omitempty"` // best-effort full executable basename, resolved on receipt
	NameGroup   string           `json:"name_group,omitempty"`   // kernel workers normalized for grouping; otherwise task name
	CgroupPath  string           `json:"cgroup_path,omitempty"`  // best-effort unified cgroup path of the event thread at receipt
	Phase       string           `json:"phase,omitempty"`
	DurationNS  *uint64          `json:"duration_ns,omitempty"`
	ReturnValue *int64           `json:"return_value,omitempty"`
	File        *SyscallFile     `json:"file,omitempty"`
	Collection  *CollectionStats `json:"collection,omitempty"` // cumulative subscription counters; not additive
	Kind        string           `json:"kind,omitempty"`       // collection_stats for diagnostic-only records
	Syscall     int              `json:"syscall,omitempty"`
	Packet      *PacketEvent     `json:"packet,omitempty"`
	Process     *ProcessEvent    `json:"process,omitempty"`
	Disk        *DiskEvent       `json:"disk,omitempty"`
	Tracepoint  *TracepointEvent `json:"tracepoint,omitempty"`
	UserStack   *CapturedStack   `json:"user_stack,omitempty"`
	KernelStack *CapturedStack   `json:"kernel_stack,omitempty"`
	Data        any              `json:"data,omitempty"`   // integration-specific typed payload
	Fields      map[string]any   `json:"fields,omitempty"` // flat custom-source query fields
}

Event is a source data record or a collection_stats diagnostic.

func (Event) MarshalJSON

func (e Event) MarshalJSON() ([]byte, error)

MarshalJSON keeps syscall number zero visible without putting a spurious syscall field on non-syscall events.

type EventSource

type EventSource func(context.Context, MonitorRequest, func(Event) error) error

EventSource supplies raw events for an event-mode request.

type FieldCapability

type FieldCapability struct {
	Path         string `json:"path"`
	Type         string `json:"type"`
	QueryField   string `json:"query_field,omitempty"` // DSL alias when filterable/groupable.
	Optional     bool   `json:"optional,omitempty"`
	Unit         string `json:"unit,omitempty"`
	Description  string `json:"description,omitempty"`
	Aggregatable bool   `json:"aggregatable"` // At least one field-taking aggregate; numeric_fields/semantics constrain which.
}

FieldCapability describes an output JSON path, not necessarily a DSL field.

type FilterCapability

type FilterCapability struct {
	Field       string   `json:"field"`
	Type        string   `json:"type"`
	Operators   []string `json:"operators"`
	Values      []string `json:"values,omitempty"`
	Modes       []string `json:"modes"`
	Description string   `json:"description"`
}

type GPUInfo

type GPUInfo struct {
	Index       int      `json:"index"`
	UUID        string   `json:"uuid"`
	Name        string   `json:"name"`
	Temperature *float64 `json:"temperature_celsius,omitempty"`
	Utilization *float64 `json:"utilization_percent,omitempty"`
	MemoryUsed  *uint64  `json:"memory_used_mib,omitempty"`
	MemoryTotal *uint64  `json:"memory_total_mib,omitempty"`
	Power       *float64 `json:"power_watts,omitempty"`
}

type Histogram

type Histogram struct {
	Count     uint64            `json:"count"`
	ZeroCount uint64            `json:"zero_count"`
	Buckets   []HistogramBucket `json:"buckets"`
}

type HistogramBucket

type HistogramBucket struct {
	Lower json.Number `json:"lower"`
	Upper json.Number `json:"upper"`
	Count uint64      `json:"count"`
}

Histogram bounds are in the field's native units, lower inclusive/upper exclusive. Zero is counted separately; only occupied buckets are returned.

type InterfaceInfo

type InterfaceInfo struct {
	Name        string   `json:"name"`
	Index       int      `json:"index"`
	MTU         int      `json:"mtu"`
	Flags       []string `json:"flags"`
	Addresses   []string `json:"addresses"`
	BytesSent   uint64   `json:"bytes_sent"`
	BytesRecv   uint64   `json:"bytes_recv"`
	PacketsSent uint64   `json:"packets_sent"`
	PacketsRecv uint64   `json:"packets_recv"`
}

InterfaceInfo includes cumulative traffic counters, not per-second rates.

type KernelCounters

type KernelCounters struct {
	ContextSwitches  int `json:"context_switches"`
	ProcessesRunning int `json:"processes_running"`
	ProcessesBlocked int `json:"processes_blocked"`
}

type KernelInfo

type KernelInfo struct {
	BootTime      time.Time       `json:"boot_time"`
	UptimeSeconds uint64          `json:"uptime_seconds"`
	Load1         float64         `json:"load_1"`
	Load5         float64         `json:"load_5"`
	Load15        float64         `json:"load_15"`
	Counters      *KernelCounters `json:"counters,omitempty"`
}

type MemoryInfo

type MemoryInfo struct {
	Total     uint64 `json:"total"`
	Available uint64 `json:"available"`
	Used      uint64 `json:"used"`
	SwapTotal uint64 `json:"swap_total"`
	SwapUsed  uint64 `json:"swap_used"`
}

type MonitorRequest

type MonitorRequest struct {
	Source       string               `json:"source"`
	Mode         string               `json:"mode,omitempty"` // empty for events, "snapshot" or "aggregate" for queries
	Aggregation  *AggregationRequest  `json:"aggregation,omitempty"`
	Probes       []MonitorRequest     `json:"probes,omitempty"`  // script: independent selections sharing aggregation timing
	Reports      []AggregateReport    `json:"reports,omitempty"` // script: post-aggregation emits, including joins
	PID          uint32               `json:"pid,omitempty"`
	Syscalls     []int                `json:"syscalls,omitempty"`
	SyscallNames []string             `json:"syscall_names,omitempty"` // resolved against the server's native Linux ABI
	Phase        string               `json:"phase,omitempty"`         // syscalls/disk: entry (default) or completion
	Paths        bool                 `json:"paths,omitempty"`         // best-effort syscall FD path resolution
	Packet       *PacketFilter        `json:"packet,omitempty"`
	Process      *ProcessFilter       `json:"process,omitempty"`
	Disk         *DiskFilter          `json:"disk,omitempty"`
	Tracepoint   *TracepointFilter    `json:"tracepoint,omitempty"`
	Symbols      *SymbolRequest       `json:"symbols,omitempty"`
	Stacks       *StackCapture        `json:"stacks,omitempty"`
	Name         string               `json:"name,omitempty"`          // network interface or sensor key (exact or edge glob)
	Path         string               `json:"path,omitempty"`          // cgroup path relative to the visible v2 mount (exact or edge glob)
	EventFilters map[string]string    `json:"event_filters,omitempty"` // task-context event strings; AND, exact or one edge glob
	Comparisons  []NumericComparison  `json:"comparisons,omitempty"`   // event numeric predicates; AND, applied before delivery/reduction
	FileDepth    int                  `json:"file_depth,omitempty"`    // aggregate file.dir prefix depth; zero retains full directory
	Filters      []SourceFilter       `json:"filters,omitempty"`       // custom-source predicates
	Using        *SnapshotCorrelation `json:"using,omitempty"`         // frozen inventory selection before snapshot collection
	// contains filtered or unexported fields
}

MonitorRequest selects a source and its server-side filters. Source is "syscalls", "packets", "process", "disk", "tracepoint", or a snapshot-only source; unrelated filters must be omitted. "script" selects concurrent named aggregates in Probes, with shared timing in Aggregation and no root-level filters.

func ParseMonitorQuery

func ParseMonitorQuery(query string) (MonitorRequest, error)

ParseMonitorQuery compiles the text DSL into the validated, signed request used by Client.Monitor or Client.Query. The syntax is SOURCE [where FIELD = VALUE [and ...]], plus "syscall in (NUMBER, :NAME, ...)" and "tracepoint where event = CATEGORY:NAME and fields in (NAME, ...) [and field.NAME = NUMBER]". Append "count over DURATION [by FIELD, ...]", "FUNCTION(FIELD) over ...", or "percentile(FIELD, PERCENT) over ..." for a one-shot aggregation query. Comma-separated functions share one window, sampling interval and grouping.

func ResolveSyscallNames

func ResolveSyscallNames(r MonitorRequest, serverArch string) (MonitorRequest, error)

Resolve only on the server, after authentication and before any collection or userspace filtering. The client architecture must not determine syscall IDs. ResolveSyscallNames resolves symbolic syscall filters for a server ABI.

func (MonitorRequest) Matches

func (r MonitorRequest) Matches(event Event) bool

Matches reports whether an event satisfies the request's user-space filters.

func (MonitorRequest) Validate

func (r MonitorRequest) Validate() error

Validate checks that the request selects a supported source and that all source-specific filters and controls are internally consistent.

type NumericComparison

type NumericComparison struct {
	Field string `json:"field"`
	Op    string `json:"op"` // >, >=, <, <=
	Value string `json:"value"`
}

NumericComparison compares an event field to an exact numeric threshold. Value is textual to preserve full-width integer precision in signed requests.

type PacketEvent

type PacketEvent struct {
	Protocol        string `json:"protocol"`
	Direction       string `json:"direction"`
	SourceIP        string `json:"source_ip"`
	DestinationIP   string `json:"destination_ip"`
	SourcePort      uint16 `json:"source_port,omitempty"`
	DestinationPort uint16 `json:"destination_port,omitempty"`
	Length          int    `json:"length"`
	Data            []byte `json:"data"`
}

PacketEvent contains packet endpoints and up to 2048 raw link-layer bytes. Data is base64-encoded in JSON. No TCP stream reassembly is performed.

type PacketFilter

type PacketFilter struct {
	Protocol        string `json:"protocol,omitempty"`
	Direction       string `json:"direction,omitempty"`
	SourceIP        string `json:"source_ip,omitempty"`
	DestinationIP   string `json:"destination_ip,omitempty"`
	SourcePort      uint16 `json:"source_port,omitempty"`
	DestinationPort uint16 `json:"destination_port,omitempty"`
}

PacketFilter matches packet headers. Empty fields match any value. Direction is "incoming" or "outgoing" relative to the server's network interface. Protocol is "tcp" or "udp"; port filters only match those protocols.

type ProcessEvent

type ProcessEvent struct {
	Name   string `json:"name"`
	Action string `json:"action"`
}

ProcessEvent describes a start or exit observed by the process poller.

type ProcessFilter

type ProcessFilter struct {
	Name   string `json:"name,omitempty"`   // exact, prefix*, or *suffix
	Action string `json:"action,omitempty"` // start or exit
}

ProcessFilter selects process lifecycle events. Empty fields match all.

type ProcessInfo

type ProcessInfo struct {
	PID         uint32    `json:"pid"`
	Name        string    `json:"name"`
	Started     time.Time `json:"started"`
	CPUSeconds  *float64  `json:"cpu_seconds,omitempty"` // Cumulative user + system CPU time, excluding children.
	RSSBytes    *uint64   `json:"rss_bytes,omitempty"`
	VMSBytes    *uint64   `json:"vms_bytes,omitempty"`
	User        string    `json:"user,omitempty"`
	State       string    `json:"state,omitempty"`
	Threads     *int32    `json:"threads,omitempty"`
	CommandLine string    `json:"command_line,omitempty"`
}

ProcessInfo is a process visible to the server at snapshot time.

type SampleField

type SampleField struct {
	FieldCapability
	Semantics string `json:"semantics"` // identity, gauge, counter, rate, utilization
}

type SamplingCapability

type SamplingCapability struct {
	DefaultInterval string        `json:"default_interval"`
	MinInterval     string        `json:"min_interval"`
	Fields          []SampleField `json:"fields"`
	GroupByFields   []string      `json:"group_by_fields"`
	NumericFields   []string      `json:"numeric_fields"`
}

type SensorInfo

type SensorInfo struct {
	Name        string  `json:"name"`
	Temperature float64 `json:"temperature_celsius"`
	High        float64 `json:"high_celsius"`
	Critical    float64 `json:"critical_celsius"`
}

type Snapshot

type Snapshot struct {
	Source       string               `json:"source"`
	Time         time.Time            `json:"time"`
	Processes    []ProcessInfo        `json:"processes,omitempty"`
	CPU          []CPUInfo            `json:"cpu,omitempty"`
	Memory       *MemoryInfo          `json:"memory,omitempty"`
	Network      []InterfaceInfo      `json:"network,omitempty"`
	Kernel       *KernelInfo          `json:"kernel,omitempty"`
	Sensors      []SensorInfo         `json:"sensors,omitempty"`
	Containers   []ContainerInfo      `json:"containers,omitempty"`
	Cgroups      []CgroupInfo         `json:"cgroups,omitempty"`
	GPUs         []GPUInfo            `json:"gpus,omitempty"`
	Aggregation  *AggregationResult   `json:"aggregation,omitempty"`
	Windows      []*AggregationResult `json:"windows,omitempty"` // finite periodic reports, returned together at query completion
	Tables       []Snapshot           `json:"tables,omitempty"`  // multi-selector results in selector order
	Capabilities *Capabilities        `json:"capabilities,omitempty"`
	Symbols      *SymbolResult        `json:"symbols,omitempty"`
	Data         any                  `json:"data,omitempty"` // integration-specific snapshot payload
	// contains filtered or unexported fields
}

Snapshot is a one-shot view, not a stream of lifecycle events.

func (Snapshot) MarshalJSON

func (s Snapshot) MarshalJSON() ([]byte, error)

MarshalJSON keeps empty collections visible for the selected source while excluding fields from unrelated sources.

type SnapshotCorrelation

type SnapshotCorrelation struct {
	Inventory MonitorRequest `json:"inventory"`
	Field     string         `json:"field"` // target string identity field
	Key       string         `json:"key"`   // inventory string field
}

SnapshotCorrelation selects target identities from a frozen custom snapshot. Inventory.Data must be a JSON array of flat objects with declared scalar fields. Matching is exact, not a glob or a completed-aggregate join.

type SnapshotSource

type SnapshotSource func(context.Context, MonitorRequest) (Snapshot, error)

SnapshotSource collects one snapshot for a snapshot-mode request.

type SourceCapability

type SourceCapability struct {
	Name              string              `json:"name"`
	Description       string              `json:"description"`
	Modes             []string            `json:"modes"` // events, snapshot, aggregate
	PlatformSupported bool                `json:"platform_supported"`
	Authorized        bool                `json:"authorized"`
	UnavailableReason string              `json:"unavailable_reason,omitempty"`
	Requirements      []string            `json:"requirements"`
	Fields            []FieldCapability   `json:"fields"`
	Filters           []FilterCapability  `json:"filters"`
	GroupByFields     []string            `json:"group_by_fields"`
	NumericFields     []string            `json:"numeric_fields"`
	Examples          []string            `json:"examples"`
	Sampling          *SamplingCapability `json:"sampling,omitempty"`
}

type SourceFilter

type SourceFilter struct {
	Field  string   `json:"field"`
	Values []string `json:"values"`
}

SourceFilter is an equality or membership predicate for a custom field. Values are textual DSL values, compared to fmt.Sprint of the event field.

type SourceMetadata

type SourceMetadata struct {
	Sampled             bool
	SampleFields        []SampleField
	SampleGroupByFields []string
	SampleNumericFields []string
	GroupByFields       []string
	NumericFields       []string
	DefaultInterval     string
	MinimumInterval     string
}

SourceMetadata exposes the engine-derived field metadata needed to build transport capability documents without exposing the engine's internal maps.

func Metadata

func Metadata(request MonitorRequest) (SourceMetadata, error)

Metadata returns aggregation and sampling metadata for a request's source.

type StackCapture

type StackCapture struct {
	User        bool        `json:"user,omitempty"`
	Kernel      bool        `json:"kernel,omitempty"`
	Depth       int         `json:"depth,omitempty"` // default 32, maximum 64
	Symbolize   bool        `json:"symbolize,omitempty"`
	UserShape   *StackShape `json:"user_shape,omitempty"`
	KernelShape *StackShape `json:"kernel_shape,omitempty"`
}

StackCapture opts in to bounded best-effort eBPF stack capture.

func (StackCapture) Validate

func (s StackCapture) Validate() error

type StackCoverage

type StackCoverage struct {
	Events                uint64            `json:"events"`
	Captured              uint64            `json:"captured"` // at least one raw frame, even if symbolization failed
	Missing               uint64            `json:"missing"`
	Empty                 uint64            `json:"empty"`
	CaptureFailures       uint64            `json:"capture_failures"`
	HelperErrors          map[string]uint64 `json:"helper_errors,omitempty"` // signed decimal error codes
	LookupFailures        uint64            `json:"lookup_failures"`
	DepthLimitReached     uint64            `json:"depth_limit_reached"`
	Frames                uint64            `json:"frames"`
	NamedFrames           uint64            `json:"named_frames"`
	UnresolvedFrames      uint64            `json:"unresolved_frames"`
	FullySymbolized       uint64            `json:"fully_symbolized"`
	PartiallySymbolized   uint64            `json:"partially_symbolized"`
	Unsymbolized          uint64            `json:"unsymbolized"`
	SymbolizationFailures uint64            `json:"symbolization_failures"` // stack-level symbolizer errors; frame gaps are counted separately
	SymbolizationDisabled uint64            `json:"symbolization_disabled"`
}

StackCoverage describes matching received events before shaping and output limits. Kernel-wide failures (including undelivered events) remain in CollectionStats.

type StackCoverageReport

type StackCoverageReport struct {
	User   *StackCoverage `json:"user,omitempty"`
	Kernel *StackCoverage `json:"kernel,omitempty"`
}

type StackShape

type StackShape struct {
	DropOffsets  bool     `json:"drop_offsets,omitempty"`
	DropBottom   int      `json:"drop_bottom,omitempty"`
	DropTop      int      `json:"drop_top,omitempty"`
	Top          int      `json:"top,omitempty"`           // zero keeps all remaining frames
	Until        string   `json:"until,omitempty"`         // exact function name or one edge glob; matching frame is included
	From         string   `json:"from,omitempty"`          // discard leaf-side frames before the first match, inclusive match retained
	FromPatterns []string `json:"from_patterns,omitempty"` // alternative to From; first matching frame wins, not pattern order
}

StackShape projects leaf-first frames into aggregation keys, without changing raw event frames. Order: DropBottom, DropTop, From, Until, Top, DropOffsets.

type SymbolFrame

type SymbolFrame struct {
	Address    string  `json:"address"`
	Name       string  `json:"name,omitempty"`
	Module     string  `json:"module,omitempty"`
	Offset     uint64  `json:"offset,omitempty"`
	FileOffset *uint64 `json:"file_offset,omitempty"`
	Error      string  `json:"error,omitempty"`
}

type SymbolRecord

type SymbolRecord struct {
	Address    string  `json:"address"`
	Name       string  `json:"name"`
	Module     string  `json:"module,omitempty"`
	Size       uint64  `json:"size,omitempty"`
	FileOffset *uint64 `json:"file_offset,omitempty"`
}

type SymbolRequest

type SymbolRequest struct {
	Target    string   `json:"target"`
	PID       uint32   `json:"pid,omitempty"`
	Path      string   `json:"path,omitempty"`
	Addresses []uint64 `json:"addresses,omitempty"`
	Name      string   `json:"name,omitempty"`
	Limit     int      `json:"limit,omitempty"`
}

SymbolRequest describes a standalone symbol lookup. Name is exact, prefix*, or *suffix. A zero Limit uses 256.

func (SymbolRequest) Validate

func (r SymbolRequest) Validate() error

type SymbolResult

type SymbolResult struct {
	Frames  []SymbolFrame  `json:"frames,omitempty"`
	Symbols []SymbolRecord `json:"symbols,omitempty"`
}

func InspectSymbols

func InspectSymbols(ctx context.Context, request SymbolRequest) (SymbolResult, error)

InspectSymbols resolves symbols without reading process or kernel memory.

type SyscallFile

type SyscallFile struct {
	FD    int32  `json:"fd"`
	Path  string `json:"path,omitempty"`
	Error string `json:"error,omitempty"`
}

SyscallFile is a best-effort kernel path snapshot captured at syscall entry and retained through completion. Concurrent rename/descriptor changes may race.

type TracepointEvent

type TracepointEvent struct {
	Event  string                 `json:"event"`
	Fields map[string]json.Number `json:"fields"`
}

TracepointEvent contains values interpreted using the server kernel's format. json.Number preserves full-width signed and unsigned 64-bit values on clients.

type TracepointFilter

type TracepointFilter struct {
	Event  string            `json:"event"` // category:name
	Fields []string          `json:"fields"`
	Equals map[string]string `json:"equals,omitempty"`
}

TracepointFilter selects scalar fields from a Linux tracepoint. Equals values use decimal or 0x notation; every filtered field must also be selected. An empty selection captures task identity/stacks without reading payload fields.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL