Documentation
¶
Overview ¶
Package consume implements the consumer-side process of `dws event consume`: dial the bus, send Hello, read Event frames, format them, and write them out (stdout / file / dir). v1 (P3) implements the minimal path — NDJSON to stdout. P4 adds filter/format/route/compact pipeline.
Index ¶
- Variables
- func IsValidationError(err error) bool
- func PrintDryRun(w io.Writer, cfg Config)
- func Run(ctx context.Context, cfg Config) error
- func ValidateConfig(cfg Config) error
- func ValidateNoOutputConflict(cfg Config, globalOutput string) error
- type Config
- type Format
- type Formatter
- type FormatterOption
- type Pipeline
- type Projector
- type Route
- type Router
- type Sink
- type ValidationError
Constants ¶
This section is empty.
Variables ¶
var ( // ErrForceRequiresForeground is the plan §3.1 contract: --force only // makes sense in foreground mode where the bus runs in the current // process (then --force skips the single-instance lock so a second // foreground bus can co-exist for a brief debug window). Outside of // --foreground, --force would silently produce two daemons writing // to the same socket — refuse upfront. ErrForceRequiresForeground = &ValidationError{ Msg: "--force is only meaningful with --foreground (in daemon mode it would produce multiple bus instances; cloud events would be randomly split across connections). To restart the bus: preview dws event stop --all --dry-run, confirm with dws event stop --all --yes, then run dws event consume", } // ErrJSONFormatRequiresBounded is the plan §3.1 contract: --format // json renders each event as a multi-line JSON object suitable for // human inspection. With an unbounded stream the output mixes events // without delimiters. Force the user to bound the run. ErrJSONFormatRequiresBounded = &ValidationError{ Msg: "--format json requires --max-events or --duration (an unbounded JSON stream is not parseable). Use --format ndjson for unbounded streams.", } )
validation sentinels (cobra layer uses errors.Is to set the exit code).
var ErrPipeClosed = errors.New("sink: downstream pipe closed")
ErrPipeClosed is returned by stdout-style sinks when the downstream reader closed its end (SIGPIPE / EPIPE on Unix, ERROR_BROKEN_PIPE on Windows). The pipeline catches this sentinel and exits cleanly without surfacing it as a fatal error.
Functions ¶
func IsValidationError ¶
IsValidationError reports whether err is a flag-level user error. Cobra command handlers use this to map validation errors to exit code 2.
func PrintDryRun ¶
PrintDryRun writes the resolved configuration to w in a single human-readable block. Called by Run when cfg.DryRun is true. Format avoids JSON so users can `dws event consume --dry-run | head` cleanly.
Secret-bearing fields are never present in Config (credentials never reach this layer), so no redaction is required here.
func Run ¶
Run dials the bus (forking one if necessary), sends Hello, and writes each received Event frame as one NDJSON line to stdout. Blocks until ctx is cancelled, MaxEvents is reached, the bus sends Bye, or the stream is interrupted.
Returns nil on graceful exits (ctx done, max-events reached, bye received, stdout pipe closed). Returns a non-nil error only for connection / protocol failures.
func ValidateConfig ¶
ValidateConfig performs all pre-flight validation that does not require disk / network I/O. Returns a *ValidationError for any rule violation; returns nil if the cfg is launchable. The cobra layer calls this BEFORE calling Run so the user gets clear errors at parse time.
Rules implemented:
- WorkDir / IPCEndpoint / ClientID non-empty
- --force requires --foreground (plan §3.1)
- --format json requires --max-events OR --duration (bounded)
- Routes already pre-parsed (any parse error is reported by ParseRoutes)
- --output-dir conflict with global --output (caller-supplied flag — we expose ValidateNoOutputConflict separately because global -o is a cobra-layer concern)
Rules NOT enforced here (deferred to caller / Run):
- Credentials presence (auth.ResolveAppCredentialsStrict already reports a typed error)
- bus availability (busctl.Discover handles)
func ValidateNoOutputConflict ¶
ValidateNoOutputConflict ensures --output-dir / --route (event-stream sinks) are not combined with the dws global hidden -o/--output flag (request-output to file). The cobra layer reads the global output flag from inherited flags and passes its value here; an empty globalOutput means the flag was unset.
Types ¶
type Config ¶
type Config struct {
// WorkDir is the bus working directory:
// <ConfigDir>/events/<edition>/<source_kind>/<identity_hash>/
WorkDir string
// IPCEndpoint is the Unix socket path / Windows pipe name. Caller
// computes from WorkDir on Unix, from edition+hash on Windows.
IPCEndpoint string
// ClientID is forwarded to busctl.Spawn so it can pass --client-id
// when forking _bus.
ClientID string
// SpawnExtraArgs are forwarded to the hidden _bus process when consume.Run
// needs to start a daemon. Used for source-mode options that must be
// reproduced in the child process, including portal ticket mode and
// personal_stream.
SpawnExtraArgs []string
// EventTypes / Filter / Compact are forwarded to the bus via Hello
// for server-side pushdown filtering.
EventTypes []string
Filter string
SubscribeID string
Compact bool
// MaxEvents: stop after receiving this many events. 0 = no limit.
MaxEvents int
// EventKey is the single event key being consumed. Used only for the
// AI-subprocess contract stderr lines (`[event] ready event_key=...`).
// Empty → the key is omitted from the marker.
EventKey string
// Duration: wall-clock budget for the consume run. After this elapses,
// Run returns nil (clean exit, exit code 0). Zero = no limit.
//
// Note: this is event-consume specific and intentionally NOT named
// "Timeout" — global dws --timeout is HTTP request timeout (int
// seconds) which would collide if reused. See plan §1 决策
// "事件运行时长 flag 不复用全局 --timeout".
Duration time.Duration
// DryRun, when true, prints the resolved configuration to Stderr and
// returns nil without dialing the bus. Used by the cobra layer to
// preview configuration with `--dry-run` (plan §3.1).
DryRun bool
// Foreground hint, passed through to status output but otherwise has
// no behavioural effect inside consume.Run — the cobra layer decides
// whether to call this Run or to bus.Run directly when --foreground
// is set.
Foreground bool
// Force, like Foreground, is informational at this layer. The cobra
// layer enforces the "--force requires --foreground" rule before
// calling Run.
Force bool
// --- Output / Sink config (P4) ---
// Format controls the per-event output shape (ndjson/json/pretty/raw/
// compact). The cobra layer maps --format string → Format via
// NormalizeFormat; an empty Format here defaults to NDJSON inside
// BuildPipeline.
Format Format
// Flatten records whether the caller explicitly selected a structured
// business projection. Projector remains the executable behavior; this
// field is surfaced in dry-run output so users can verify the final mode.
Flatten bool
// OutputDir, if non-empty, switches the fallback sink from stdout to
// "file per event" under this directory.
OutputDir string
// Routes are pre-parsed --route specs. Empty = no routing.
Routes []Route
// Projector, when set, maps transport envelopes to the public value used
// by all structured formats. Raw format always bypasses it.
Projector Projector
// ReadySubscribeID identifies the personal subscription in the stable
// ready marker emitted after HelloAck. It is separate from SubscribeID
// because debug raw mode deliberately clears the latter's local filter.
ReadySubscribeID string
// Stdin, when non-nil, is watched for EOF: closing stdin triggers a
// graceful shutdown (reason: signal). This wires the AI-subprocess
// contract — a parent closes stdin to stop the consumer. The cobra
// layer passes os.Stdin; tests inject a controllable reader. nil →
// stdin is not watched (backward-compatible default for callers that
// do not opt in).
//
// Note: `< /dev/null` EOFs immediately and exits at once. To stay
// resident feed a never-EOF stdin (`< <(tail -f /dev/null)`) or run
// bounded (--max-events / --duration).
Stdin io.Reader
// Stdout sink; nil → os.Stdout. Injected for tests.
Stdout io.Writer
// Stderr sink for status lines (HelloAck info, bye reason); nil → os.Stderr.
// Set to io.Discard when --quiet is in effect.
Stderr io.Writer
// Quiet suppresses stderr status writes (the HelloAck / bye banners).
Quiet bool
}
Config holds everything Run needs. Built by the cobra command handler (P5) from flag values + strict resolver output.
type Format ¶
type Format string
Format identifies the wire shape `dws event consume` writes per event. Values mirror dws's global -f/--format flag vocabulary (defined in internal/output) with the subset that makes sense for streaming.
const ( // FormatNDJSON is the default: one compact JSON object per line. // Pipe-friendly; one event per `read` line. Recommended for agents. FormatNDJSON Format = "ndjson" // FormatJSON pretty-prints each event as multi-line JSON. NOT a // JSON array — still NDJSON-style (one document per output unit), // just with indentation. See plan §3.1 输出约束 note about why we // do not emit a JSON array for an unbounded stream. FormatJSON Format = "json" // FormatPretty is the same as FormatJSON in v1; reserved for future // human-friendly colorisation. Kept distinct so we never silently // degrade `--format pretty` to compact-ndjson. FormatPretty Format = "pretty" // FormatRaw writes only the SDK's original Data string (one per // event, newline-terminated). Useful when piping into jq / a tool // that wants the cloud payload verbatim without our envelope. FormatRaw Format = "raw" // FormatCompact emits one compact JSON line. Personal streams use their // configured business projection; other streams use the registry processor. FormatCompact Format = "compact" )
func NormalizeFormat ¶
NormalizeFormat maps a raw flag value to a supported Format. Values outside the event command's supported set fall back to NDJSON with the fallback flag set true — callers SHOULD warn on stderr when fallback is true and the original value was non-empty (e.g. user passed --format table which has no meaning for an event stream).
Empty input maps to NDJSON without a fallback warning.
type Formatter ¶
Formatter renders a transport.Event into the byte stream the sink writes out. Implementations append their own line terminator when appropriate (NDJSON / Raw add '\n'; Pretty/JSON embed newlines in the JSON itself).
func NewFormatter ¶
func NewFormatter(format Format, opts ...FormatterOption) (Formatter, error)
NewFormatter returns a Formatter for the given Format. When no projector is configured, compact dispatches through registry.LookupProcessor. Returns an error only if format is internally unsupported.
type FormatterOption ¶ added in v1.0.54
type FormatterOption func(*formatterConfig)
FormatterOption configures structured event rendering. Raw output always bypasses these options and preserves the source Data string.
func WithProjectionWarnings ¶ added in v1.0.54
func WithProjectionWarnings(w io.Writer) FormatterOption
func WithProjector ¶ added in v1.0.54
func WithProjector(projector Projector) FormatterOption
type Pipeline ¶
type Pipeline struct {
// contains filtered or unexported fields
}
Pipeline is the consumer-side delivery chain: format → route → sink. Each delivered event goes through formatting once; the routed sink then dispatches the formatted bytes to either a route-specific directory or the fallback (stdout/file).
A Pipeline is bound to a single Config snapshot. Reconfiguring (changing format / routes mid-stream) is out of scope for v1.
func BuildPipeline ¶
func BuildPipeline(format Format, outputDir string, routes []Route, stdoutW io.Writer, formatterOpts ...FormatterOption) (*Pipeline, error)
BuildPipeline constructs a Pipeline from the cobra-side flag bundle. The cobra command first parses --format / --output-dir / --route into the derived inputs here so this function stays free of cobra dependencies.
Sink selection rules (plan §3.1 输出约束):
- --route present → routed sink with per-rule dirs; fallback is --output-dir if set, else stdout
- --output-dir only → file-per-event sink at the dir
- neither → stdout sink with stdoutW
stdoutW is injected for tests (os.Stdout in production). When nil it defaults to io.Discard so a misconfigured pipeline never writes to the host process's actual stdout.
func NewPipeline ¶
NewPipeline builds a Pipeline for the given formatter and sink.
func (*Pipeline) Close ¶
Close releases sink resources. Safe to call multiple times because underlying Sink Close methods are idempotent.
type Projector ¶ added in v1.0.54
Projector maps a transport envelope to the public value rendered by structured formats. Returning a value together with an error means the value is a safe fallback and should still be emitted after a warning.
type Route ¶
type Route struct {
Pattern *regexp.Regexp
Dir string
// Raw is the original CLI spec; preserved for status / debug output.
Raw string
}
Route describes one --route rule. The CLI accepts the wire form `<regex>=dir:<path>`; ParseRoute compiles regex once at startup so the hot path is just a Match.
Pattern matches against event.EventType (NOT the whole event JSON). The first matching rule in CLI order wins; unmatched events fall through to the default sink (stdout or --output-dir).
func ParseRoute ¶
ParseRoute parses one `<regex>=dir:<path>` spec. Returns a typed error for bad inputs so the CLI can render a clear "did you mean" message.
Wire grammar:
spec = regex "=dir:" path regex = any chars except literal "=" (use \= to escape) (v1: no escape) path = any string (no validation here; sink validates at write time)
Examples:
"^im\\.message=dir:./im/" "^approval\\.=dir:./approval/"
func ParseRoutes ¶
ParseRoutes parses many specs in CLI order. On any parse failure returns the partial parse so far and the error — the caller decides whether to continue. (The cobra layer treats any parse error as fatal validation.)
type Router ¶
type Router struct {
// contains filtered or unexported fields
}
Router decides which sink an event goes to. Match returns the directory of the first matching Route, or empty string when no rule matches (fall through to default sink).
type Sink ¶
type Sink interface {
// Write places one event on the sink. Returns an error for IO failures;
// returns ErrPipeClosed when the downstream consumer closed the pipe
// (typical pattern: `dws event consume | head -1`).
Write(ev transport.Event, formatted []byte) error
// Close releases sink-owned resources. Idempotent.
Close() error
}
Sink is what a Pipeline writes formatted event bytes to. Implementations take both the event (for filename derivation in file sinks) and the already-formatted bytes (so the same event can be rendered with different formats per call without re-running the formatter inside the sink).
func NewFileDirSink ¶
NewFileDirSink returns a sink that writes each event to its own file under dir, naming files `{type}_{id}_{ts}.json`. The directory is mkdir'd on first write so callers don't have to ensure it themselves.
Filename pieces are sanitised: characters that would escape the directory (path separators) or break shell globbing are replaced with '_'. `ts` is the ReceivedAtUnixMS (or current time if zero) so two events with the same id (re-delivery, dedup-defeated edge cases) don't collide.
func NewRoutedSink ¶
NewRoutedSink composes a Router with per-route dir sinks plus a fallback. On each Write, Router.Match decides the target dir; if non-empty, the event is written there; otherwise the fallback sink handles it. The fallback is typically NewStdoutSink (default) or NewFileDirSink (--output-dir mode).
func NewStdoutSink ¶
NewStdoutSink wraps the given writer (typically os.Stdout) in a Sink that writes formatted bytes verbatim. Detects broken-pipe on the host platform and returns ErrPipeClosed so the caller can exit code 0.
type ValidationError ¶
type ValidationError struct{ Msg string }
ValidationError represents a flag-level user error. It wraps a clear, actionable message — the cobra command layer surfaces it to the user with exit code 2 (validation error).
func (*ValidationError) Error ¶
func (e *ValidationError) Error() string