stream

package
v0.0.20 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 20, 2025 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package stream provides streaming encode/decode for Tony documents.

The stream package provides structural event-based encoding and decoding optimized for streaming use cases like snapshot indexing. It only supports bracketed structures ({...} and [...]) and does not handle formatting options like colors, comments, or block style.

For general parsing/encoding with full feature support, use the parse and encode packages instead.

Example: Encoding 

enc, err := stream.NewEncoder(writer, stream.WithBrackets())
if err != nil {
    return err
}
enc.BeginObject()
enc.WriteKey("name")
enc.WriteString("value")
enc.EndObject()

Example: Encoding with Tags 

enc, err := stream.NewEncoder(writer, stream.WithBrackets())
if err != nil {
    return err
}
enc.BeginObject()
enc.WriteKey("name")
enc.Tag("!schema(string)")
enc.WriteString("John")
enc.EndObject()

Example: Decoding 

dec, err := stream.NewDecoder(reader, stream.WithBrackets())
if err != nil {
    return err
}
event, _ := dec.ReadEvent()  // EventBeginObject (with Tag field if present)
event, _ := dec.ReadEvent()  // EventKey("name")
event, _ := dec.ReadEvent()  // EventString("value", with Tag field if present)
event, _ := dec.ReadEvent()  // EventEndObject

Comments

The API is comment-ready (aligned with IR specification):

  • Head comments: precede a value (IR: CommentType node with 1 value in Values)
  • Line comments: on same line as value (IR: CommentType node in Comment field)

Comment support is deferred to Phase 2. In Phase 1, comment methods are no-ops and comment tokens are skipped.

Tags

Tags are supported and appear before values in the Tony format (e.g., !schema(...) value). When encoding, call Tag() before writing a value to set the tag for that value. The tag is automatically cleared after the value is written. When decoding, tags are automatically read and included in the Event.Tag field.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeNode 

func DecodeNode(r io.Reader, opts ...StreamOption) (*ir.Node, error)

DecodeNode decodes bytes to ir.Node using Decoder. Convenience function: Decoder + EventsToNode.

func EncodeNode 

func EncodeNode(node *ir.Node, w io.Writer, opts ...StreamOption) error

EncodeNode encodes an ir.Node to bytes using Encoder. Convenience function: NodeToEvents + Encoder.

func EventsToNode 

func EventsToNode(events []Event) (*ir.Node, error)

EventsToNode converts a sequence of events to an ir.Node. Takes events read from Decoder.

Phase 1: Comment events are not present (comments skipped). Phase 2: Comment events are converted to IR comment nodes.

Types

type BinaryEventReader added in v0.0.15 

type BinaryEventReader struct {
	// contains filtered or unexported fields
}

BinaryEventReader reads events from an io.Reader using binary format.

func NewBinaryEventReader added in v0.0.15 

func NewBinaryEventReader(r io.Reader) *BinaryEventReader

NewBinaryEventReader creates an event reader from a reader positioned at binary events.

func (*BinaryEventReader) ReadEvent added in v0.0.15 

func (r *BinaryEventReader) ReadEvent() (*Event, error)

ReadEvent reads the next event using binary format.

type BinaryEventWriter added in v0.0.15 

type BinaryEventWriter struct {
	// contains filtered or unexported fields
}

BinaryEventWriter writes events to an io.Writer using binary format.

func NewBinaryEventWriter added in v0.0.15 

func NewBinaryEventWriter(w io.Writer) *BinaryEventWriter

NewBinaryEventWriter creates an event writer that writes to w.

func (*BinaryEventWriter) WriteEvent added in v0.0.15 

func (w *BinaryEventWriter) WriteEvent(ev *Event) error

WriteEvent writes an event in binary format to the writer.

type BufferEventSink added in v0.0.15 

type BufferEventSink struct {
	// contains filtered or unexported fields
}

BufferEventSink writes events to a buffer using compact binary encoding.

func NewBufferEventSink added in v0.0.15 

func NewBufferEventSink(buf *bytes.Buffer) *BufferEventSink

NewBufferEventSink creates an event sink that writes to a byte buffer.

func (*BufferEventSink) WriteEvent added in v0.0.15 

func (s *BufferEventSink) WriteEvent(ev *Event) error

WriteEvent writes an event in binary format to the buffer.

type Decoder 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder provides structural event-based decoding. Only supports bracketed structures ({...} and [...]). Block style (TArrayElt) is not supported.

func NewDecoder 

func NewDecoder(r io.Reader, opts ...StreamOption) (*Decoder, error)

NewDecoder creates a new Decoder reading from r. Requires bracketed format (use WithBrackets() or WithWire()). Returns error if bracketing not specified.

func (*Decoder) CurrentIndex 

func (d *Decoder) CurrentIndex() (int, bool)

CurrentIndex returns the current array index (if in array).

func (*Decoder) CurrentKey 

func (d *Decoder) CurrentKey() (string, bool)

CurrentKey returns the current object key (if in object).

func (*Decoder) CurrentPath 

func (d *Decoder) CurrentPath() string

CurrentPath returns the current kinded path (e.g., "", "key", "key[0]").

func (*Decoder) Depth 

func (d *Decoder) Depth() int

Depth returns the current nesting depth (0 = top level).

func (*Decoder) IsInArray 

func (d *Decoder) IsInArray() bool

IsInArray returns true if currently inside an array.

func (*Decoder) IsInObject 

func (d *Decoder) IsInObject() bool

IsInObject returns true if currently inside an object.

func (*Decoder) ReadEvent 

func (d *Decoder) ReadEvent() (*Event, error)

ReadEvent reads the next structural event from the stream. Returns structural events (BeginObject, Key, String, etc.) that correspond to the encoder's API. Low-level tokens (commas, colons) are elided. Returns io.EOF when stream is exhausted.

Phase 1: Comment tokens are skipped (no comment events emitted). Phase 2: Comment tokens are converted to EventHeadComment or EventLineComment.

func (*Decoder) Reset 

func (d *Decoder) Reset(r io.Reader, opts ...StreamOption) error

Reset resets the decoder to read from a new reader.

type EmptyEventReader added in v0.0.15 

type EmptyEventReader struct{}

EmptyEventReader provides an empty event stream (for null state).

func NewEmptyEventReader added in v0.0.15 

func NewEmptyEventReader() *EmptyEventReader

NewEmptyEventReader creates an empty event reader.

func (*EmptyEventReader) ReadEvent added in v0.0.15 

func (r *EmptyEventReader) ReadEvent() (*Event, error)

ReadEvent returns io.EOF immediately (empty stream).

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder provides explicit stack management for streaming Tony document encoding. Only supports bracketed structures ({...} and [...]). Block style (TArrayElt) is not supported.

func NewEncoder 

func NewEncoder(w io.Writer, opts ...StreamOption) (*Encoder, error)

NewEncoder creates a new Encoder writing to w. Requires bracketed format (use WithBrackets() or WithWire()). Returns error if bracketing not specified.

func (*Encoder) BeginArray 

func (e *Encoder) BeginArray() error

BeginArray begins a regular array. Uses the tag set by Tag() if one was set.

func (*Encoder) BeginObject 

func (e *Encoder) BeginObject() error

BeginObject begins an object (or sparse array). Note: Sparse arrays use BeginObject/EndObject (semantic distinction at parse layer). Uses the tag set by Tag() if one was set.

func (*Encoder) CurrentIndex 

func (e *Encoder) CurrentIndex() (int, bool)

CurrentIndex returns the current array index (if in array).

func (*Encoder) CurrentKey 

func (e *Encoder) CurrentKey() (string, bool)

CurrentKey returns the current object key (if in object).

func (*Encoder) CurrentPath 

func (e *Encoder) CurrentPath() string

CurrentPath returns the current kinded path (e.g., "", "key", "key[0]").

func (*Encoder) CurrentTag added in v0.0.13 

func (e *Encoder) CurrentTag() string

CurrentTag returns the currently pending tag (if any). Returns empty string if no tag is pending.

func (*Encoder) Depth 

func (e *Encoder) Depth() int

Depth returns the current nesting depth (0 = top level).

func (*Encoder) EndArray 

func (e *Encoder) EndArray() error

EndArray ends an array.

func (*Encoder) EndObject 

func (e *Encoder) EndObject() error

EndObject ends an object.

func (*Encoder) Flush 

func (e *Encoder) Flush() error

Flush flushes any buffered data.

func (*Encoder) IsInArray 

func (e *Encoder) IsInArray() bool

IsInArray returns true if currently inside an array.

func (*Encoder) IsInObject 

func (e *Encoder) IsInObject() bool

IsInObject returns true if currently inside an object.

func (*Encoder) Offset 

func (e *Encoder) Offset() int64

Offset returns the byte offset in the output stream.

func (*Encoder) Reset 

func (e *Encoder) Reset(w io.Writer, opts ...StreamOption) error

Reset resets the encoder to write to a new writer.

func (*Encoder) Tag added in v0.0.13 

func (e *Encoder) Tag(tag string) error

Tag sets the tag for the next value to be written. The tag will be applied to the next call to BeginObject, BeginArray, WriteString, WriteInt, WriteFloat, WriteBool, or WriteNull. After writing the value, the tag is cleared. Returns an error if a tag is already pending (not yet consumed by a value).

func (*Encoder) TagCompose added in v0.0.13 

func (e *Encoder) TagCompose(tag string, args []string) error

TagCompose composes a tag with arguments and the currently pending tag (if any). If there's a pending tag, it composes the new tag with the pending tag. The new tag comes first in the composition (e.g., if pending is "!schema" and you call TagCompose("!bracket", nil), the result is "!bracket.schema"). If there's no pending tag, it just sets the new tag. The tag will be applied to the next call to BeginObject, BeginArray, WriteString, WriteInt, WriteFloat, WriteBool, or WriteNull. After writing the value, the tag is cleared.

func (*Encoder) WriteBool 

func (e *Encoder) WriteBool(value bool) error

WriteBool writes a boolean value. Uses the tag set by Tag() if one was set.

func (*Encoder) WriteFloat 

func (e *Encoder) WriteFloat(value float64) error

WriteFloat writes a float value. Uses the tag set by Tag() if one was set.

func (*Encoder) WriteHeadComment 

func (e *Encoder) WriteHeadComment(lines []string) error

WriteHeadComment writes a head comment (precedes a value). IR: CommentType node with 1 value in Values. Phase 1: No-op (comment support deferred).

func (*Encoder) WriteInt 

func (e *Encoder) WriteInt(value int64) error

WriteInt writes an integer value. Uses the tag set by Tag() if one was set.

func (*Encoder) WriteIntKey added in v0.0.13 

func (e *Encoder) WriteIntKey(key int) error

func (*Encoder) WriteKey 

func (e *Encoder) WriteKey(key string) error

WriteKey writes an object key.

func (*Encoder) WriteLineComment 

func (e *Encoder) WriteLineComment(lines []string) error

WriteLineComment writes a line comment (on same line as value). IR: CommentType node in Comment field. Phase 1: No-op (comment support deferred).

func (*Encoder) WriteNull 

func (e *Encoder) WriteNull() error

WriteNull writes a null value. Uses the tag set by Tag() if one was set.

func (*Encoder) WriteString 

func (e *Encoder) WriteString(value string) error

WriteString writes a string value. Uses the tag set by Tag() if one was set.

type Error 

type Error struct {
	Msg string
}

Error represents a stream error.

func (*Error) Error 

func (e *Error) Error() string

type Event 

type Event struct {
	Type EventType `tony:"field=t"`

	// Tag field (applies to value events: String, Int, Float, Bool, Null, BeginObject, BeginArray)
	Tag string `tony:"field=a optional"`

	// Value fields (only one is set based on Type)
	Key    string  `tony:"field=k optional"`
	IntKey int64   `tony:"field=ik optional"`
	String string  `tony:"field=s optional"`
	Int    int64   `tony:"field=i optional"`
	Float  float64 `tony:"field=f optional"`
	Bool   bool    `tony:"field=b optional"`

	// Comment fields (for EventHeadComment and EventLineComment)
	CommentLines []string `tony:"field=c optional"` // Comment text lines (from IR Node.Lines)
}

Event represents a structural event from the decoder. Events correspond to the encoder's API methods, providing a symmetric encode/decode interface.

func NodeToEvents 

func NodeToEvents(node *ir.Node) ([]Event, error)

NodeToEvents converts an ir.Node to a sequence of events. Returns events that can be written via Encoder.

Phase 2: Comments are converted to EventHeadComment or EventLineComment. Head comments (CommentType node with 1 value) emit EventHeadComment before the value. Line comments (CommentType node in Comment field) emit EventLineComment after the value.

func (*Event) FromTony added in v0.0.13 

func (s *Event) FromTony(data []byte, opts ...gomap.UnmapOption) error

FromTony parses Tony format bytes and populates Event.

func (*Event) FromTonyIR added in v0.0.13 

func (s *Event) FromTonyIR(node *ir.Node, opts ...gomap.UnmapOption) error

FromTonyIR populates Event from a Tony IR node.

func (*Event) IsValueStart added in v0.0.13 

func (e *Event) IsValueStart() bool

IsValueStart returns true if this event starts a value (as opposed to a key, end marker, or comment). Value-starting events are: BeginObject, BeginArray, String, Int, Float, Bool, Null.

func (*Event) ReadBinary added in v0.0.15 

func (e *Event) ReadBinary(r io.Reader) error

ReadBinary reads event in compact binary format.

func (*Event) ToTony added in v0.0.13 

func (s *Event) ToTony(opts ...gomap.MapOption) ([]byte, error)

ToTony converts Event to Tony format bytes.

func (*Event) ToTonyIR added in v0.0.13 

func (s *Event) ToTonyIR(opts ...gomap.MapOption) (*ir.Node, error)

ToTonyIR converts Event to a Tony IR node.

func (*Event) WriteBinary added in v0.0.15 

func (e *Event) WriteBinary(w io.Writer) error

WriteBinary writes event in compact binary format. Format: [type:1byte][fields based on type] No per-event length prefix - snapshots use index chunk sizes.

type EventReader added in v0.0.15 

type EventReader interface {
	ReadEvent() (*Event, error)
}

EventReader provides events from a source (snapshot, empty stream, etc.).

type EventType 

type EventType int

EventType represents the type of a structural event. 

const (
	EventBeginObject EventType = iota
	EventEndObject
	EventBeginArray
	EventEndArray
	EventKey
	EventIntKey
	EventString
	EventInt
	EventFloat
	EventBool
	EventNull
	EventHeadComment // Head comment (precedes a value) - IR: CommentType node with 1 value in Values
	EventLineComment // Line comment (on same line as value) - IR: CommentType node in Comment field
)

func (EventType) IsKey added in v0.0.13 

func (t EventType) IsKey() bool

func (EventType) MarshalText added in v0.0.13 

func (t EventType) MarshalText() ([]byte, error)

func (EventType) String 

func (t EventType) String() string

func (*EventType) UnmarshalText added in v0.0.13 

func (t *EventType) UnmarshalText(d []byte) error

type EventWriter added in v0.0.16 

type EventWriter interface {
	WriteEvent(*Event) error
}

EventWriter receives events (builder, writer, etc.).

type State 

type State struct {
	// contains filtered or unexported fields
}

State provides minimal stack/state/path management. Just processes tokens and tracks state - no tokenization, no io.Reader. Use this if you already have tokens.

Only tracks bracketed structures ({...} and [...]). Block-style arrays (TArrayElt) are not tracked.

func KPathState added in v0.0.13 

func KPathState(kp string) (*State, error)

KPathState creates a State positioned to process events at the given path.

For leaf array elements, positions one element BEFORE the target so that processing the event at that offset advances to the correct position. For non-leaf array elements, uses the actual index for path matching.

Examples: 

KPathState("users[3]")      → positioned at "users[2]" (leaf)
KPathState("users[0]")      → positioned at "users" (leaf at index 0)
KPathState("users[0].name") → positioned at "users[0].name" (non-leaf)

Returns an error if the kpath string is invalid.

func NewState 

func NewState() *State

NewState creates a new State for tracking structure state.

func (*State) CurrentIndex 

func (s *State) CurrentIndex() (int, bool)

CurrentIndex returns the current array index (if in array), -1 otherwise

func (*State) CurrentIntKey added in v0.0.13 

func (s *State) CurrentIntKey() (int, bool)

func (*State) CurrentKey 

func (s *State) CurrentKey() (string, bool)

CurrentKey returns the current object key (if in object).

func (*State) CurrentPath 

func (s *State) CurrentPath() string

CurrentPath returns the current kinded path (e.g., "", "key", "key[0]").

Skips nil segments (arrays before first element processed). nil segments only occur at the top of the stack during normal processing, so this has no effect. Skipping allows manually-constructed stacks to include nil segments in the middle without breaking path construction.

func (*State) Depth 

func (s *State) Depth() int

Depth returns the current nesting depth (0 = top level).

func (*State) IsInArray 

func (s *State) IsInArray() bool

IsInArray returns true if currently inside an array.

func (*State) IsInObject 

func (s *State) IsInObject() bool

IsInObject returns true if currently inside an object.

func (*State) IsInSparseArray added in v0.0.13 

func (s *State) IsInSparseArray() bool

func (*State) ProcessEvent 

func (s *State) ProcessEvent(event *Event) error

ProcessEvent processes an event and updates state/path tracking. Call this for each event in order.

type StreamOption 

type StreamOption func(*streamOpts)

StreamOption configures Encoder/Decoder behavior.

func WithBrackets 

func WithBrackets() StreamOption

WithBrackets forces bracketed style encoding/decoding.

func WithWire 

func WithWire() StreamOption

WithWire enables wire format (implies brackets).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.