consume

package
v1.0.54 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 21, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package consume implements the consumer-side process of `dws event consume`: dial the bus, send Hello, read Event frames, format them, and write them out (stdout / file / dir). v1 (P3) implements the minimal path — NDJSON to stdout. P4 adds filter/format/route/compact pipeline.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrForceRequiresForeground is the plan §3.1 contract: --force only
	// makes sense in foreground mode where the bus runs in the current
	// process (then --force skips the single-instance lock so a second
	// foreground bus can co-exist for a brief debug window). Outside of
	// --foreground, --force would silently produce two daemons writing
	// to the same socket — refuse upfront.
	ErrForceRequiresForeground = &ValidationError{
		Msg: "--force is only meaningful with --foreground (in daemon mode it would produce multiple bus instances; cloud events would be randomly split across connections). To restart the bus: preview dws event stop --all --dry-run, confirm with dws event stop --all --yes, then run dws event consume",
	}

	// ErrJSONFormatRequiresBounded is the plan §3.1 contract: --format
	// json renders each event as a multi-line JSON object suitable for
	// human inspection. With an unbounded stream the output mixes events
	// without delimiters. Force the user to bound the run.
	ErrJSONFormatRequiresBounded = &ValidationError{
		Msg: "--format json requires --max-events or --duration (an unbounded JSON stream is not parseable). Use --format ndjson for unbounded streams.",
	}
)

validation sentinels (cobra layer uses errors.Is to set the exit code).

View Source
var ErrPipeClosed = errors.New("sink: downstream pipe closed")

ErrPipeClosed is returned by stdout-style sinks when the downstream reader closed its end (SIGPIPE / EPIPE on Unix, ERROR_BROKEN_PIPE on Windows). The pipeline catches this sentinel and exits cleanly without surfacing it as a fatal error.

Functions

func IsValidationError

func IsValidationError(err error) bool

IsValidationError reports whether err is a flag-level user error. Cobra command handlers use this to map validation errors to exit code 2.

func PrintDryRun

func PrintDryRun(w io.Writer, cfg Config)

PrintDryRun writes the resolved configuration to w in a single human-readable block. Called by Run when cfg.DryRun is true. Format avoids JSON so users can `dws event consume --dry-run | head` cleanly.

Secret-bearing fields are never present in Config (credentials never reach this layer), so no redaction is required here.

func Run

func Run(ctx context.Context, cfg Config) error

Run dials the bus (forking one if necessary), sends Hello, and writes each received Event frame as one NDJSON line to stdout. Blocks until ctx is cancelled, MaxEvents is reached, the bus sends Bye, or the stream is interrupted.

Returns nil on graceful exits (ctx done, max-events reached, bye received, stdout pipe closed). Returns a non-nil error only for connection / protocol failures.

func ValidateConfig

func ValidateConfig(cfg Config) error

ValidateConfig performs all pre-flight validation that does not require disk / network I/O. Returns a *ValidationError for any rule violation; returns nil if the cfg is launchable. The cobra layer calls this BEFORE calling Run so the user gets clear errors at parse time.

Rules implemented:

  1. WorkDir / IPCEndpoint / ClientID non-empty
  2. --force requires --foreground (plan §3.1)
  3. --format json requires --max-events OR --duration (bounded)
  4. Routes already pre-parsed (any parse error is reported by ParseRoutes)
  5. --output-dir conflict with global --output (caller-supplied flag — we expose ValidateNoOutputConflict separately because global -o is a cobra-layer concern)

Rules NOT enforced here (deferred to caller / Run):

  • Credentials presence (auth.ResolveAppCredentialsStrict already reports a typed error)
  • bus availability (busctl.Discover handles)

func ValidateNoOutputConflict

func ValidateNoOutputConflict(cfg Config, globalOutput string) error

ValidateNoOutputConflict ensures --output-dir / --route (event-stream sinks) are not combined with the dws global hidden -o/--output flag (request-output to file). The cobra layer reads the global output flag from inherited flags and passes its value here; an empty globalOutput means the flag was unset.

Types

type Config

type Config struct {
	// WorkDir is the bus working directory:
	// <ConfigDir>/events/<edition>/<source_kind>/<identity_hash>/
	WorkDir string
	// IPCEndpoint is the Unix socket path / Windows pipe name. Caller
	// computes from WorkDir on Unix, from edition+hash on Windows.
	IPCEndpoint string
	// ClientID is forwarded to busctl.Spawn so it can pass --client-id
	// when forking _bus.
	ClientID string
	// SpawnExtraArgs are forwarded to the hidden _bus process when consume.Run
	// needs to start a daemon. Used for source-mode options that must be
	// reproduced in the child process, including portal ticket mode and
	// personal_stream.
	SpawnExtraArgs []string

	// EventTypes / Filter / Compact are forwarded to the bus via Hello
	// for server-side pushdown filtering.
	EventTypes  []string
	Filter      string
	SubscribeID string
	Compact     bool

	// MaxEvents: stop after receiving this many events. 0 = no limit.
	MaxEvents int

	// EventKey is the single event key being consumed. Used only for the
	// AI-subprocess contract stderr lines (`[event] ready event_key=...`).
	// Empty → the key is omitted from the marker.
	EventKey string

	// Duration: wall-clock budget for the consume run. After this elapses,
	// Run returns nil (clean exit, exit code 0). Zero = no limit.
	//
	// Note: this is event-consume specific and intentionally NOT named
	// "Timeout" — global dws --timeout is HTTP request timeout (int
	// seconds) which would collide if reused. See plan §1 决策
	// "事件运行时长 flag 不复用全局 --timeout".
	Duration time.Duration

	// DryRun, when true, prints the resolved configuration to Stderr and
	// returns nil without dialing the bus. Used by the cobra layer to
	// preview configuration with `--dry-run` (plan §3.1).
	DryRun bool

	// Foreground hint, passed through to status output but otherwise has
	// no behavioural effect inside consume.Run — the cobra layer decides
	// whether to call this Run or to bus.Run directly when --foreground
	// is set.
	Foreground bool
	// Force, like Foreground, is informational at this layer. The cobra
	// layer enforces the "--force requires --foreground" rule before
	// calling Run.
	Force bool

	// --- Output / Sink config (P4) ---
	// Format controls the per-event output shape (ndjson/json/pretty/raw/
	// compact). The cobra layer maps --format string → Format via
	// NormalizeFormat; an empty Format here defaults to NDJSON inside
	// BuildPipeline.
	Format Format
	// Flatten records whether the caller explicitly selected a structured
	// business projection. Projector remains the executable behavior; this
	// field is surfaced in dry-run output so users can verify the final mode.
	Flatten bool
	// OutputDir, if non-empty, switches the fallback sink from stdout to
	// "file per event" under this directory.
	OutputDir string
	// Routes are pre-parsed --route specs. Empty = no routing.
	Routes []Route
	// Projector, when set, maps transport envelopes to the public value used
	// by all structured formats. Raw format always bypasses it.
	Projector Projector

	// ReadySubscribeID identifies the personal subscription in the stable
	// ready marker emitted after HelloAck. It is separate from SubscribeID
	// because debug raw mode deliberately clears the latter's local filter.
	ReadySubscribeID string

	// Stdin, when non-nil, is watched for EOF: closing stdin triggers a
	// graceful shutdown (reason: signal). This wires the AI-subprocess
	// contract — a parent closes stdin to stop the consumer. The cobra
	// layer passes os.Stdin; tests inject a controllable reader. nil →
	// stdin is not watched (backward-compatible default for callers that
	// do not opt in).
	//
	// Note: `< /dev/null` EOFs immediately and exits at once. To stay
	// resident feed a never-EOF stdin (`< <(tail -f /dev/null)`) or run
	// bounded (--max-events / --duration).
	Stdin io.Reader

	// Stdout sink; nil → os.Stdout. Injected for tests.
	Stdout io.Writer
	// Stderr sink for status lines (HelloAck info, bye reason); nil → os.Stderr.
	// Set to io.Discard when --quiet is in effect.
	Stderr io.Writer

	// Quiet suppresses stderr status writes (the HelloAck / bye banners).
	Quiet bool
}

Config holds everything Run needs. Built by the cobra command handler (P5) from flag values + strict resolver output.

type Format

type Format string

Format identifies the wire shape `dws event consume` writes per event. Values mirror dws's global -f/--format flag vocabulary (defined in internal/output) with the subset that makes sense for streaming.

const (
	// FormatNDJSON is the default: one compact JSON object per line.
	// Pipe-friendly; one event per `read` line. Recommended for agents.
	FormatNDJSON Format = "ndjson"
	// FormatJSON pretty-prints each event as multi-line JSON. NOT a
	// JSON array — still NDJSON-style (one document per output unit),
	// just with indentation. See plan §3.1 输出约束 note about why we
	// do not emit a JSON array for an unbounded stream.
	FormatJSON Format = "json"
	// FormatPretty is the same as FormatJSON in v1; reserved for future
	// human-friendly colorisation. Kept distinct so we never silently
	// degrade `--format pretty` to compact-ndjson.
	FormatPretty Format = "pretty"
	// FormatRaw writes only the SDK's original Data string (one per
	// event, newline-terminated). Useful when piping into jq / a tool
	// that wants the cloud payload verbatim without our envelope.
	FormatRaw Format = "raw"
	// FormatCompact emits one compact JSON line. Personal streams use their
	// configured business projection; other streams use the registry processor.
	FormatCompact Format = "compact"
)

func NormalizeFormat

func NormalizeFormat(raw string) (f Format, fellback bool)

NormalizeFormat maps a raw flag value to a supported Format. Values outside the event command's supported set fall back to NDJSON with the fallback flag set true — callers SHOULD warn on stderr when fallback is true and the original value was non-empty (e.g. user passed --format table which has no meaning for an event stream).

Empty input maps to NDJSON without a fallback warning.

type Formatter

type Formatter interface {
	Render(ev transport.Event) ([]byte, error)
}

Formatter renders a transport.Event into the byte stream the sink writes out. Implementations append their own line terminator when appropriate (NDJSON / Raw add '\n'; Pretty/JSON embed newlines in the JSON itself).

func NewFormatter

func NewFormatter(format Format, opts ...FormatterOption) (Formatter, error)

NewFormatter returns a Formatter for the given Format. When no projector is configured, compact dispatches through registry.LookupProcessor. Returns an error only if format is internally unsupported.

type FormatterOption added in v1.0.54

type FormatterOption func(*formatterConfig)

FormatterOption configures structured event rendering. Raw output always bypasses these options and preserves the source Data string.

func WithProjectionWarnings added in v1.0.54

func WithProjectionWarnings(w io.Writer) FormatterOption

func WithProjector added in v1.0.54

func WithProjector(projector Projector) FormatterOption

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline is the consumer-side delivery chain: format → route → sink. Each delivered event goes through formatting once; the routed sink then dispatches the formatted bytes to either a route-specific directory or the fallback (stdout/file).

A Pipeline is bound to a single Config snapshot. Reconfiguring (changing format / routes mid-stream) is out of scope for v1.

func BuildPipeline

func BuildPipeline(format Format, outputDir string, routes []Route, stdoutW io.Writer, formatterOpts ...FormatterOption) (*Pipeline, error)

BuildPipeline constructs a Pipeline from the cobra-side flag bundle. The cobra command first parses --format / --output-dir / --route into the derived inputs here so this function stays free of cobra dependencies.

Sink selection rules (plan §3.1 输出约束):

  • --route present → routed sink with per-rule dirs; fallback is --output-dir if set, else stdout
  • --output-dir only → file-per-event sink at the dir
  • neither → stdout sink with stdoutW

stdoutW is injected for tests (os.Stdout in production). When nil it defaults to io.Discard so a misconfigured pipeline never writes to the host process's actual stdout.

func NewPipeline

func NewPipeline(formatter Formatter, sink Sink) *Pipeline

NewPipeline builds a Pipeline for the given formatter and sink.

func (*Pipeline) Close

func (p *Pipeline) Close() error

Close releases sink resources. Safe to call multiple times because underlying Sink Close methods are idempotent.

func (*Pipeline) Deliver

func (p *Pipeline) Deliver(ev transport.Event) error

Deliver renders ev with the configured formatter and hands the result to the sink. Returns ErrPipeClosed (re-raised from the sink) when the downstream stdout pipe closed; otherwise returns whatever formatting or sink error surfaced.

type Projector added in v1.0.54

type Projector func(ev transport.Event) (any, error)

Projector maps a transport envelope to the public value rendered by structured formats. Returning a value together with an error means the value is a safe fallback and should still be emitted after a warning.

type Route

type Route struct {
	Pattern *regexp.Regexp
	Dir     string
	// Raw is the original CLI spec; preserved for status / debug output.
	Raw string
}

Route describes one --route rule. The CLI accepts the wire form `<regex>=dir:<path>`; ParseRoute compiles regex once at startup so the hot path is just a Match.

Pattern matches against event.EventType (NOT the whole event JSON). The first matching rule in CLI order wins; unmatched events fall through to the default sink (stdout or --output-dir).

func ParseRoute

func ParseRoute(spec string) (Route, error)

ParseRoute parses one `<regex>=dir:<path>` spec. Returns a typed error for bad inputs so the CLI can render a clear "did you mean" message.

Wire grammar:

spec   = regex "=dir:" path
regex  = any chars except literal "=" (use \= to escape)  (v1: no escape)
path   = any string (no validation here; sink validates at write time)

Examples:

"^im\\.message=dir:./im/"
"^approval\\.=dir:./approval/"

func ParseRoutes

func ParseRoutes(specs []string) ([]Route, error)

ParseRoutes parses many specs in CLI order. On any parse failure returns the partial parse so far and the error — the caller decides whether to continue. (The cobra layer treats any parse error as fatal validation.)

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router decides which sink an event goes to. Match returns the directory of the first matching Route, or empty string when no rule matches (fall through to default sink).

func NewRouter

func NewRouter(rules []Route) *Router

NewRouter constructs a router from pre-parsed rules.

func (*Router) Match

func (r *Router) Match(ev transport.Event) string

Match returns the destination directory for the event's type, or "" if no rule matched. Iterates rules in CLI order (first match wins).

func (*Router) Rules

func (r *Router) Rules() []Route

Rules returns the parsed routes for status / debug output. Caller MUST NOT mutate the returned slice.

type Sink

type Sink interface {
	// Write places one event on the sink. Returns an error for IO failures;
	// returns ErrPipeClosed when the downstream consumer closed the pipe
	// (typical pattern: `dws event consume | head -1`).
	Write(ev transport.Event, formatted []byte) error
	// Close releases sink-owned resources. Idempotent.
	Close() error
}

Sink is what a Pipeline writes formatted event bytes to. Implementations take both the event (for filename derivation in file sinks) and the already-formatted bytes (so the same event can be rendered with different formats per call without re-running the formatter inside the sink).

func NewFileDirSink

func NewFileDirSink(dir string) Sink

NewFileDirSink returns a sink that writes each event to its own file under dir, naming files `{type}_{id}_{ts}.json`. The directory is mkdir'd on first write so callers don't have to ensure it themselves.

Filename pieces are sanitised: characters that would escape the directory (path separators) or break shell globbing are replaced with '_'. `ts` is the ReceivedAtUnixMS (or current time if zero) so two events with the same id (re-delivery, dedup-defeated edge cases) don't collide.

func NewRoutedSink

func NewRoutedSink(router *Router, fallback Sink) Sink

NewRoutedSink composes a Router with per-route dir sinks plus a fallback. On each Write, Router.Match decides the target dir; if non-empty, the event is written there; otherwise the fallback sink handles it. The fallback is typically NewStdoutSink (default) or NewFileDirSink (--output-dir mode).

func NewStdoutSink

func NewStdoutSink(w io.Writer) Sink

NewStdoutSink wraps the given writer (typically os.Stdout) in a Sink that writes formatted bytes verbatim. Detects broken-pipe on the host platform and returns ErrPipeClosed so the caller can exit code 0.

type ValidationError

type ValidationError struct{ Msg string }

ValidationError represents a flag-level user error. It wraps a clear, actionable message — the cobra command layer surfaces it to the user with exit code 2 (validation error).

func (*ValidationError) Error

func (e *ValidationError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL