conformance

package
v0.25.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: GPL-3.0 Imports: 56 Imported by: 0

Documentation

Overview

Package conformance is the sensor protocol conformance suite (RFC-026 WP-S3, RFC-023 D23), starting with results ingest.

It has two halves:

  • FakePlatform, a control plane that implements the v2 results contract (RFC-026 §3) strictly and the v2 control plane (RFC-029), records every request and can be told to fail in each §3.8 class. The SDK's own tests run against it: they check the SDK sends the right headers and digest, retries only what it should, splits on 413, never resends a partially accepted report and reuses report_id on retry.
  • The live suite (live_test.go), which runs the same contract against a real API: OPENCTEM_CONFORMANCE_URL and OPENCTEM_CONFORMANCE_KEY (a sensor key) select it; see the package README in the test file.

Stability: Stable (docs/STABILITY.md).

Index

Constants

View Source 
const FutureMember = "x_openctem_future"

FutureMember is the member FutureFields adds to every JSON object answer.

View Source 
const HeaderV3Binding = "X-Fake-V3-Binding"

HeaderV3Binding marks the fake's in-process v2 requests that carried a v3 call, with the binding ("grpc" or "https"); Requests shows it.

View Source 
const SensorALPN = "openctem-sensor/3"

SensorALPN is the ALPN id the router passes through to the gRPC binding (client.SensorALPN; repeated here so the fake does not import the client).

View Source 
const UnknownCommandType = "x_conformance_unknown"

UnknownCommandType is the command type the suite dispatches to check that a sensor fails a command it does not know instead of hanging or crashing.

Variables

This section is empty.

Functions

func FuzzOutput added in v0.19.0 

func FuzzOutput(t T, m tool.Manifest, dir string, d time.Duration)

FuzzOutput fuzzes the parser of an exec-profile tool's output format for d: seeds are the files of the tool's fuzz/ directory and a minimal valid document, mutated with a fixed-seed generator. A panic, two parses of the same input that differ, or a result above the record cap fails; the input is saved under fuzz/crashers/. Adapter tools (records checked one by one by the runtime) are not fuzzed here.

func RunContractSuite added in v0.19.0 

func RunContractSuite(t T, manifestPath string, opts ContractOptions)

RunContractSuite checks a tool against the contract of the capabilities it implements, the way the sensor runs it (every task through the runtime's host and sandbox):

  • Lint: LintDescriptor (warnings are notes).
  • FixtureContract: every self-test fixture, run as a task of each implemented capability, has no contract violation (required output paths, allowed outputs).
  • Capability suites (opts.Capability): per capability, fixture targets on loopback and checks of what the tool found.
  • Scope (opts.Scope): no connection to an address it was not given.
  • Fuzz (opts.Fuzz): FuzzOutput.

func RunReachSuite added in v0.25.0 

func RunReachSuite(t T, probe ReachProber)

RunReachSuite checks a sensor's reach probe (RFC-089 §4.3 in openctemio/openctem):

  • OnlyAdmitted: with a recording dialer, the probe never dials an address its guard did not admit: a literal outside the scope, the out-of-scope half of a range, a denied address inside a range, a name resolving to a denied address, IPv6, link-local, a name, never a name (only address literals reach the dialer).
  • Bounds: at most reach.MaxProbeAddrs addresses, at most reach.MaxProbeRate connects in any second, one connect per address at a time.
  • Classes: a refused port is reached (port_closed), no route and timeouts are unreached, and the summary is bounded and stable.
  • Unroutable: a target behind no route is reported unreached, never as reached or as nothing.
  • Canceled: a canceled probe stops dialing.

func RunSensorSuite added in v0.16.0 

func RunSensorSuite(t *testing.T, start StartSensor, opts SuiteOptions)

RunSensorSuite checks a sensor against the platform contract every sensor must keep, whatever tools it runs, so a third-party sensor can prove it in its own tests: 

func TestConformance(t *testing.T) {
	httpsec.AllowLoopback = true // the fake listens on 127.0.0.1
	conformance.RunSensorSuite(t, func(t *testing.T, p conformance.SensorEndpoint) func() {
		kit, err := sensorkit.New(sensorkit.Options{Name: "my-sensor", Version: "1.0.0",
			APIURL: p.URL, APIKey: p.APIKey, StateDir: t.TempDir(),
			Outbox: sensorkit.OutboxSettings{Dir: t.TempDir()}})
		if err != nil {
			t.Fatal(err)
		}
		kit.AddScanner(myScanner)
		ctx, cancel := context.WithCancel(context.Background())
		done := make(chan error, 1)
		go func() { done <- kit.Run(ctx) }()
		return func() { cancel(); <-done }
	}, conformance.SuiteOptions{ScanPayload: json.RawMessage(`{"scanner":"my-tool","target":"..."}`)})
}

Each check runs against its own FakePlatform (protocol v2, control plane and manifest on):

  • Heartbeat: the sensor heartbeats and names the SDK it is built on.
  • Manifest: a sensor that announces a manifest digest registers the manifest when the platform asks for it.
  • UnknownCommand: a command of a type the sensor does not handle is left pending for a sensor that does (or failed), never claimed and abandoned, and the sensor keeps polling.
  • ForwardCompatible: the same with every answer carrying a JSON member no SDK knows (FutureMember), as a newer platform's would: the sensor ignores it and keeps working (and runs its own command, with ScanPayload).
  • Command (with ScanPayload): the command completes, and every report sent for it is committed before the command is completed.

func RunToolSuite added in v0.18.0 

func RunToolSuite(t T, manifestPath string, opts ToolSuiteOptions)

RunToolSuite checks a tool that runs as its own program (a tool.yaml with a run section, any language) against adapter protocol v1, the way the sensor's runtime will run it:

  • Manifest: the tool.yaml loads strictly and validates, and has a run section.
  • Handshake: the program answers the runtime's hello with protocol 1, ignores a message type it does not know, and answers describe with exactly its tool.yaml (the run section aside).
  • Validate: a configuration with an unknown key is refused.
  • StdoutIsProtocol: everything it writes to standard output is a protocol message (diagnostics belong on standard error).
  • ExitsOnEOF: it exits, with status 0, when its input ends.
  • Cancel: after run and cancel it sends its result, or exits, within the 10-second grace.
  • Fixtures: each self-test fixture (manifest selftest: a task file and the CTIS it must produce) runs through the runtime's own host, with every runtime check, and produces exactly the expected normalized report, with no invalid message.

Use it from go test (conformance.RunToolSuite(t, "tool.yaml", conformance.ToolSuiteOptions{})) or with the command cmd/openctem-conformance.

func SDKReachProber added in v0.25.0 

func SDKReachProber(ctx context.Context, guard reach.Guard, dial reach.DialFunc, targets []string) (reach.ProbeResult, error)

SDKReachProber runs reach.Probe with the suite's guard and dialer.

Types

type Bump added in v0.19.0 

type Bump int

Bump is the semantic version part a descriptor change needs. 

const (
	BumpNone Bump = iota
	BumpPatch
	BumpMinor
	BumpMajor
)

Bumps, in order.

func (Bump) String added in v0.19.0 

func (b Bump) String() string

type ContractOptions added in v0.19.0 

type ContractOptions struct {
	// Timeout bounds each run (zero: 30s).
	Timeout time.Duration
	// Credentials are what an operator would store for the tool.
	Credentials map[string]string
	// Capability runs the per-capability suites: fixture targets on
	// loopback (open and closed ports, an HTTP app with known issues, a
	// repository with a fake secret, ...) and the capability's checks.
	Capability bool
	// Scope runs the scope check: the tool is given target A while an
	// address B listens; any connection to B fails.
	Scope bool
	// Fuzz, when positive, fuzzes the parser of the tool's output format
	// for this long (see FuzzOutput).
	Fuzz time.Duration
	// Env is extra environment for the tool (tests).
	Env map[string]string
	// ScopeListenAddr is the address B listens on ("": 127.0.0.2:0).
	ScopeListenAddr string
}

ContractOptions configure RunContractSuite.

type Diff added in v0.19.0 

type Diff struct {
	// Breaking changes: a workflow, a pinned node or a stored setting
	// that worked with the old version can fail with the new one.
	Breaking []string
	// Additions: new capabilities, inputs, outputs or settings.
	Additions []string
	// Required is the smallest version bump the changes need.
	Required Bump
	// Problems are the reasons the new version is not acceptable: a name
	// change, or a version that does not bump enough.
	Problems []string
}

Diff is what changed between two versions of a tool descriptor.

func DiffManifests added in v0.19.0 

func DiffManifests(oldM, newM tool.Manifest) Diff

DiffManifests compares two versions of a tool's descriptor. A change that narrows what the tool takes or emits (an implemented capability, a consumed or produced type, a param mapping, a supported value or bound, an output shape), adds a required config key, removes a config key, raises the tier or adds a side effect is breaking and needs a major bump; an addition needs a minor bump; any other change to the descriptor needs at least a patch bump.

func (Diff) OK added in v0.19.0 

func (d Diff) OK() bool

OK reports whether the new descriptor is an acceptable successor.

type FakePlatform 

type FakePlatform struct {
	Server *httptest.Server
	// APIKey is the sensor key it accepts.
	APIKey string
	// V2 serves the v2 routes and advertises them; false behaves like a
	// platform from before RFC-026 (404 on /api/v2).
	V2 bool
	// Limits are published on hello.
	Limits protov2.Limits
	// Tools are the sensor's declared tools (empty: any).
	Tools []string
	// Control serves the api RFC-029 control plane on v2 (heartbeat,
	// commands, suppressions, fingerprints, keys) and lists it on hello.
	// False (with V2) is a platform from api v0.8: v2 results only.
	Control bool
	// Paused answers heartbeats as for a disabled sensor (v2: 200 + pause).
	Paused bool
	// Manifest serves PUT /manifest (api RFC-033) with Control and lists it
	// on hello: the fake stores the manifest's digest and asks for it again
	// (send_manifest) when a heartbeat names another one.
	Manifest bool
	// FutureFields adds a member no SDK knows to every JSON object the
	// fake answers with, as a newer platform would: a sensor must ignore
	// it (forward compatibility, docs/STABILITY.md).
	FutureFields bool
	// contains filtered or unexported fields
}

FakePlatform is an in-process control plane. Safe for concurrent use.

func NewFakePlatform 

func NewFakePlatform(v2 bool) *FakePlatform

NewFakePlatform starts a fake platform. Close it with Close. v2 false is a platform without sensor protocol v2 (every v2 route answers 404), against which the SDK must fail with ErrV2Unsupported; v2 true serves results and the RFC-029 control plane (SetControl(false) leaves only results).

func (*FakePlatform) AcceptedFindings 

func (f *FakePlatform) AcceptedFindings() int

AcceptedFindings counts the findings of committed reports.

func (*FakePlatform) AcceptedReports added in v0.18.0 

func (f *FakePlatform) AcceptedReports() []*ctis.Report

AcceptedReports returns the reports the fake accepted: every segment of each committed report.

func (*FakePlatform) Close 

func (f *FakePlatform) Close()

Close stops the server.

func (*FakePlatform) CommandLogs added in v0.18.0 

func (f *FakePlatform) CommandLogs(id string) []protov2.CommandLogLine

CommandLogs returns the log lines received for a command, in seq order.

func (*FakePlatform) CommandResult added in v0.9.0 

func (f *FakePlatform) CommandResult(id string) json.RawMessage

CommandResult returns the result a completed command stored.

func (*FakePlatform) CommandState 

func (f *FakePlatform) CommandState(id string) (string, string)

CommandState returns a command's state and its error message.

func (*FakePlatform) ConfigReports added in v0.18.0 

func (f *FakePlatform) ConfigReports() []json.RawMessage

ConfigReports returns the config reports received, in order.

func (*FakePlatform) EnableV3 added in v0.19.0 

func (f *FakePlatform) EnableV3(pub ed25519.PublicKey, grpc bool) error

EnableV3 serves protocol v3 for the sensor holding key pub: the HTTPS binding, and with grpc the gRPC binding on its own TLS listener. The v2 hello lists both. The fake's API key is used for the v2 routes the calls are carried through.

func (*FakePlatform) ForgetConfigReport added in v0.18.0 

func (f *FakePlatform) ForgetConfigReport()

ForgetConfigReport drops the stored config report digest, as a platform that lost it: the next heartbeat naming one is asked to send it again.

func (*FakePlatform) ForgetManifest added in v0.14.0 

func (f *FakePlatform) ForgetManifest()

ForgetManifest drops the stored digest, as a platform that lost it: the next heartbeat naming it is asked for the manifest again.

func (*FakePlatform) HeartbeatBuilds added in v0.10.0 

func (f *FakePlatform) HeartbeatBuilds() []HeartbeatBuild

HeartbeatBuilds returns the "sdk" and "sensor" blocks of every heartbeat received, in order.

func (*FakePlatform) HeartbeatOutbox 

func (f *FakePlatform) HeartbeatOutbox() []json.RawMessage

HeartbeatOutbox returns the outbox objects heartbeats carried.

func (*FakePlatform) Heartbeats added in v0.10.0 

func (f *FakePlatform) Heartbeats() []json.RawMessage

Heartbeats returns the heartbeat bodies received, in order.

func (*FakePlatform) HideV3 added in v0.19.0 

func (f *FakePlatform) HideV3()

HideV3 keeps protocol v3 off the v2 hello (signed requests of the trusted key still pass): a platform from before v3.

func (*FakePlatform) LeaseEpoch added in v0.16.0 

func (f *FakePlatform) LeaseEpoch(id string) int

LeaseEpoch returns the command's lease epoch (0 before its first claim).

func (*FakePlatform) Manifests added in v0.14.0 

func (f *FakePlatform) Manifests() []json.RawMessage

Manifests returns the manifests registered, in order.

func (*FakePlatform) OpenCommand 

func (f *FakePlatform) OpenCommand(id string)

OpenCommand registers a running command assigned to the sensor.

func (*FakePlatform) QueueCommand added in v0.9.0 

func (f *FakePlatform) QueueCommand(id string)

QueueCommand adds a pending command a poll offers.

func (*FakePlatform) QueueCommandPayload added in v0.16.0 

func (f *FakePlatform) QueueCommandPayload(id, typ string, payload json.RawMessage)

QueueCommandPayload adds a pending command of the given type ("" is "scan") and payload, as the platform dispatches it.

func (*FakePlatform) ReclaimCommand added in v0.16.0 

func (f *FakePlatform) ReclaimCommand(id string)

ReclaimCommand simulates a lease loss (api RFC-035 D6): the command was re-queued and claimed again, so its lease epoch moves on while its state stays. A complete or fail that echoes the old epoch is refused with invalid-transition, as the platform answers it.

func (*FakePlatform) RefuseV3Identity added in v0.19.0 

func (f *FakePlatform) RefuseV3Identity(on bool)

RefuseV3Identity makes every v3 call answer UNAUTHENTICATED: a sensor must not fall back to a weaker transport.

func (*FakePlatform) Releases added in v0.10.0 

func (f *FakePlatform) Releases() []Release

Releases returns the releases received, in order.

func (*FakePlatform) Reports 

func (f *FakePlatform) Reports() map[string]*StoredReport

Reports returns the accepted v2 reports by id.

func (*FakePlatform) Requests 

func (f *FakePlatform) Requests() []Request

Requests returns the recorded requests.

func (*FakePlatform) RequestsTo 

func (f *FakePlatform) RequestsTo(method, prefix string) []Request

RequestsTo returns the recorded requests whose path has prefix.

func (*FakePlatform) RouteV3GRPCByALPN added in v0.21.0 

func (f *FakePlatform) RouteV3GRPCByALPN() error

RouteV3GRPCByALPN puts ALPN routing in front of the gRPC binding: a ClientHello that offers SensorALPN is passed through to the listener byte for byte; any other meets a server that terminates TLS with a certificate of its own (the web UI behind the gateway). The v2 hello then names the router as grpc_alpn_endpoint only, and no grpc_endpoint.

func (*FakePlatform) SetAPIKey 

func (f *FakePlatform) SetAPIKey(key string)

SetAPIKey replaces the accepted sensor key (a key rotation).

func (*FakePlatform) SetConfigReport added in v0.18.0 

func (f *FakePlatform) SetConfigReport(on bool)

SetConfigReport lists or stops listing the "config_report" feature on hello and serves PUT /config-report (api RFC-033, config report): the fake stores each report and asks for it again (send_config_report) when a heartbeat names another digest.

func (*FakePlatform) SetControl added in v0.9.0 

func (f *FakePlatform) SetControl(on bool)

SetControl switches the RFC-029 control plane on v2 on or off.

func (*FakePlatform) SetFault 

func (f *FakePlatform) SetFault(fl Fault)

SetFault installs a fault injector (nil removes it).

func (*FakePlatform) SetFutureFields added in v0.16.0 

func (f *FakePlatform) SetFutureFields(on bool)

SetFutureFields switches FutureFields.

func (*FakePlatform) SetLocalPolicy added in v0.18.0 

func (f *FakePlatform) SetLocalPolicy(on bool)

SetLocalPolicy lists or stops listing the "local_policy" feature on hello (api RFC-040 §5.7): with it, heartbeats and manifests carry the sensor's local policy report.

func (*FakePlatform) SetLogs added in v0.18.0 

func (f *FakePlatform) SetLogs(on bool)

SetLogs makes the fake serve POST /commands/{id}/logs and list "logs" on hello.

func (*FakePlatform) SetManifest added in v0.14.0 

func (f *FakePlatform) SetManifest(on bool)

SetManifest serves or stops serving the sensor manifest (api RFC-033).

func (*FakePlatform) SetManifestPolicy added in v0.15.0 

func (f *FakePlatform) SetManifestPolicy(allowedTools []string, omitInventory bool)

SetManifestPolicy sets what the manifest answers say from now on (api RFC-033 §6.12): the allowed tools (nil: every tool of the manifest) and whether heartbeats may leave the inventory out.

func (*FakePlatform) SetNetwork added in v0.25.0 

func (f *FakePlatform) SetNetwork(on bool)

SetNetwork lists or stops listing the "network" feature on hello and serves GET /egress-echo (api RFC-089): with it, manifests carry the sensor's network report.

func (*FakePlatform) SetPaused added in v0.9.0 

func (f *FakePlatform) SetPaused(on bool)

SetPaused answers heartbeats as for a disabled sensor.

func (*FakePlatform) SetPosture added in v0.19.0 

func (f *FakePlatform) SetPosture(on bool)

SetPosture lists or stops listing the "posture" feature on hello (api RFC-040): with it, manifests carry the sensor's posture (platform TLS pin, tool sandbox).

func (*FakePlatform) SetV2 

func (f *FakePlatform) SetV2(on bool)

SetV2 switches the v2 routes on or off.

func (*FakePlatform) StopV3GRPC added in v0.19.0 

func (f *FakePlatform) StopV3GRPC()

StopV3GRPC closes the gRPC binding's listener (the hello still names it): a sensor must fall back to the HTTPS binding.

func (*FakePlatform) URL 

func (f *FakePlatform) URL() string

URL is the base URL.

func (*FakePlatform) V3GRPCEndpoint added in v0.19.0 

func (f *FakePlatform) V3GRPCEndpoint() string

V3GRPCEndpoint is host:port of the fake's gRPC binding ("" without one).

func (*FakePlatform) V3OfferedALPN added in v0.21.0 

func (f *FakePlatform) V3OfferedALPN() []string

V3OfferedALPN is the ALPN ids of the last ClientHello the router saw.

type Fault 

type Fault func(r *http.Request, seq int) *FaultAnswer

Fault makes the fake answer a request with an error instead of handling it. Return nil to handle it normally.

type FaultAnswer 

type FaultAnswer struct {
	Status     int
	Problem    protov2.ProblemType // "" for a plain (non-problem) answer
	RetryAfter string
	// Drop closes the connection without answering (a network error, the
	// sender cannot tell whether the request was processed). With Process
	// the request is handled first: "the response was lost".
	Drop    bool
	Process bool
}

FaultAnswer is an injected answer.

type HeartbeatBuild added in v0.10.0 

type HeartbeatBuild struct {
	SDK *struct {
		Name    string `json:"name"`
		Version string `json:"version"`
	} `json:"sdk"`
	Sensor *struct {
		Name      string `json:"name"`
		Version   string `json:"version"`
		Commit    string `json:"commit"`
		BuildTime string `json:"build_time"`
	} `json:"sensor"`
}

HeartbeatBuild is what a heartbeat says about the SDK and the sensor binary ("sdk" and "sensor"); nil where the heartbeat did not carry it.

type LintIssue added in v0.19.0 

type LintIssue struct {
	// Level is "error" or "warning".
	Level   string
	Path    string
	Message string
}

LintIssue is one descriptor lint finding.

func LintDescriptor added in v0.19.0 

func LintDescriptor(m tool.Manifest) []LintIssue

LintDescriptor checks what Validate accepts but a good descriptor should not leave out: the contract (implements), self-test fixtures, a display name, an engine version probe for a wrapped program, a rate key for a capability with a rate param, and the deprecated keys.

func (LintIssue) String added in v0.19.0 

func (l LintIssue) String() string

type ReachProber added in v0.25.0 

type ReachProber func(ctx context.Context, guard reach.Guard, dial reach.DialFunc, targets []string) (reach.ProbeResult, error)

ReachProber is a sensor's reach probe as the suite drives it: it must dial only through guard and dial, and only for targets. SDKReachProber is the SDK's own (reach.Probe).

type Release added in v0.10.0 

type Release struct {
	CommandID, Reason string
}

Release is one release a sensor sent (api RFC-030).

type Request 

type Request struct {
	Method, Path string
	Header       http.Header
	Body         []byte // as sent (compressed when Content-Encoding is set)
	Status       int    // what the fake answered
}

Request is one request the fake received.

type SensorEndpoint added in v0.16.0 

type SensorEndpoint struct {
	URL    string
	APIKey string
}

SensorEndpoint is the platform RunSensorSuite points the sensor under test at: the fake's base URL and the sensor key it accepts.

type StartSensor added in v0.16.0 

type StartSensor func(t *testing.T, p SensorEndpoint) (stop func())

StartSensor starts the sensor under test against p (in process, or as a child process with SENSOR_API_URL / SENSOR_API_KEY set) and returns a function that stops it and waits until it has exited. The fake listens on 127.0.0.1: an SDK-based sensor needs httpsec.AllowLoopback (or OPENCTEM_SDK_HTTPSEC_ALLOW_LOOPBACK=1 for a child process) to reach it.

type StoredReport 

type StoredReport struct {
	ReportID  string
	CommandID string
	Segments  map[int]segment
	Committed bool
	Status    protov2.Status
}

StoredReport is a report the fake accepted.

type SuiteOptions added in v0.16.0 

type SuiteOptions struct {
	// ScanType and ScanPayload are a command the sensor can run to
	// completion against the fake ("" ScanType is "scan"). With no
	// ScanPayload the command checks are skipped.
	ScanType    string
	ScanPayload json.RawMessage
	// Timeout bounds every wait. Zero is 60s.
	Timeout time.Duration
}

SuiteOptions configures RunSensorSuite.

type T added in v0.18.0 

type T interface {
	Helper()
	Logf(format string, args ...any)
	Errorf(format string, args ...any)
	Fatalf(format string, args ...any)
}

T is what the tool suite reports to: *testing.T, or the conformance command's printer.

type ToolSuiteOptions added in v0.18.0 

type ToolSuiteOptions struct {
	// Timeout bounds each check (zero: 30s).
	Timeout time.Duration
	// Credentials are what an operator would store for the tool; only
	// those the manifest declares are delivered.
	Credentials map[string]string
	// Update rewrites each fixture's expect file with what the tool
	// produced (also OPENCTEM_UPDATE_GOLDEN=1).
	Update bool
}

ToolSuiteOptions configure RunToolSuite.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.