Documentation
¶
Overview ¶
Package packstream implements the PackStream binary serialisation format used by the Bolt protocol.
PackStream is a strongly typed, binary format built around a set of marker bytes that describe the type and size of the value that follows. The Encoder and Decoder types in this package provide low-level, zero-copy access to individual values; the higher-level WriteValue/ReadValue helpers operate on the Value sum type for convenience.
Concurrency: Encoder and Decoder are NOT safe for concurrent use. Each goroutine must hold its own instance. Use EncodePool/DecodePool to amortise allocation costs.
Index ¶
- Variables
- type DecodePool
- type Decoder
- func (d *Decoder) PeekType() (Type, error)
- func (d *Decoder) ReadBool() (bool, error)
- func (d *Decoder) ReadBytes() ([]byte, error)
- func (d *Decoder) ReadFloat() (float64, error)
- func (d *Decoder) ReadInt() (int64, error)
- func (d *Decoder) ReadListHeader() (int, error)
- func (d *Decoder) ReadMapHeader() (int, error)
- func (d *Decoder) ReadNull() error
- func (d *Decoder) ReadString() (string, error)
- func (d *Decoder) ReadStructHeader() (tag byte, n int, err error)
- func (d *Decoder) ReadValue() (Value, error)
- func (d *Decoder) Reset(r io.Reader)
- type EncodePool
- type Encoder
- func (e *Encoder) Flush() error
- func (e *Encoder) Reset(w io.Writer)
- func (e *Encoder) WriteBool(v bool) error
- func (e *Encoder) WriteBytes(v []byte) error
- func (e *Encoder) WriteFloat(v float64) error
- func (e *Encoder) WriteInt(v int64) error
- func (e *Encoder) WriteListHeader(n int) error
- func (e *Encoder) WriteMapHeader(n int) error
- func (e *Encoder) WriteNull() error
- func (e *Encoder) WriteString(v string) error
- func (e *Encoder) WriteStructHeader(tag byte, n int) error
- func (e *Encoder) WriteValue(v Value) error
- type Struct
- type Type
- type Value
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrDecodedMemoryExceeded = errors.New("packstream: decoded collection size exceeds memory budget")
ErrDecodedMemoryExceeded is returned by the decoder when the cumulative decoded size of the collections (List/Map/Struct) in a single message would exceed the decoder's decoded-memory budget. Inspect with errors.Is.
The byte budget behind ErrLengthExceedsInput bounds what a count prefix may *claim* relative to the bytes still on the wire, but it does not bound memory amplification: a List element can be a single wire byte (0xC0 NULL) while its decoded slot costs at least 16 bytes (one interface value), so a structurally valid 16 MiB message of ~16.7M NULLs would still coerce a make([]Value, ~16.7M) of roughly 256 MB — a ~16x amplification, multiplied again by every concurrent connection. Map entries are worse: two wire bytes (empty-string key plus NULL) decode into roughly 48 bytes of map storage, a ~24x amplification.
The decoder therefore charges every collection header's declared element count against a per-message decoded-memory budget (see [maxDecodedCollectionBytes]) before allocating. The charge is cumulative across the whole message — nested and sibling collections all draw from the same budget — so nesting cannot multiply the bound away. A message whose collections would decode to more than the budget is rejected with this error. This is reachable pre-authentication during the first HELLO decode, so the bound is a hard security boundary, not a convenience.
var ErrLengthExceedsInput = errors.New("packstream: declared length exceeds remaining input")
ErrLengthExceedsInput is returned by the decoder when a length or count prefix declares more elements than can possibly remain in the current message. Inspect with errors.Is.
PackStream sizes Bytes/String payloads and List/Map/Struct collections with a length prefix that is up to a uint32 (~4.29e9). The 16 MiB message cap in github.com/FlavioCFOliveira/GoGraph/bolt/proto.ChunkedReader bounds the bytes a client may send, but it does NOT bound the allocation those bytes can *request*: a 5-byte frame such as 0xCE 0xFF 0xFF 0xFF 0xFF claims a ~4.29 GB Bytes payload, and a 5-byte List32 header claims billions of 16-byte interface slots (~64 GB). The eager make() that follows would OOM the process before the inevitable short read failed.
The decoder defends against this by carrying a per-message byte budget (see [Decoder.remaining]) and rejecting any prefix whose minimum on-wire cost exceeds the bytes still available, BEFORE allocating. Every byte/string of length n needs n payload bytes; every List of count n needs at least n bytes (each element is at least one wire byte); every Map of count n needs at least 2n bytes (each entry is a key plus a value). This is reachable pre-authentication during the first HELLO decode, so the bound is a hard security boundary, not a convenience.
The byte budget bounds what a prefix may claim, but not how much memory a structurally valid message decodes into; that amplification is bounded separately by the decoded-memory budget. See ErrDecodedMemoryExceeded.
Independently of the budgets, any 32-bit length/count prefix above math.MaxInt32 is rejected with this error while the value is still unsigned: Go sizes allocations with int, and on a 32-bit platform such a prefix would otherwise wrap to a negative int, slip past the budget comparisons above, and panic the subsequent make() with "len out of range". See wire32MaxLen.
var ErrNestingTooDeep = errors.New("packstream: value nesting too deep")
ErrNestingTooDeep is returned by ReadValue when a composite value (List/Map/Structure) nests deeper than maxValueDepth. It guards against stack-overflow denial-of-service from maliciously crafted messages; see maxValueDepth for the rationale.
var ErrPayloadTooLarge = fmt.Errorf("packstream: payload length exceeds 32-bit length prefix (max %d)", uint64(math.MaxUint32))
ErrPayloadTooLarge is returned by WriteBytes/WriteString/WriteListHeader/ WriteMapHeader when the supplied length exceeds the largest size the PackStream wire format can encode in its 32-bit length field. Bolt payloads in practice are bounded well below this limit by the server's frame-size limit; the explicit check guards against silent truncation when callers pass an oversize slice from upstream batching code.
Functions ¶
This section is empty.
Types ¶
type DecodePool ¶
type DecodePool struct {
// contains filtered or unexported fields
}
DecodePool is a pool of Decoders backed by bytes.Reader readers. It is safe for concurrent use.
func (*DecodePool) Get ¶
func (dp *DecodePool) Get(src *bytes.Reader) *Decoder
Get retrieves a Decoder from the pool, resetting it to read from src. The caller must call Put when done.
func (*DecodePool) Put ¶
func (dp *DecodePool) Put(dec *Decoder)
Put returns dec to the pool for reuse.
type Decoder ¶
type Decoder struct {
// contains filtered or unexported fields
}
Decoder reads PackStream-encoded values from an underlying buffered reader. It is NOT safe for concurrent use.
Memory contract: decoding the values of one message allocates at most the message's wire size in payload bytes (Bytes/String data, bounded 1:1 by the wire byte budget behind ErrLengthExceedsInput) plus 128 MiB of collection storage (List/Map/Struct containers and their element slots, bounded cumulatively — across nesting and sibling collections alike — by the decoded-memory budget behind ErrDecodedMemoryExceeded), plus a small constant factor for per-element boxing. Input that would exceed either budget is rejected with the corresponding typed error before the offending allocation is made. Decoder.Reset restores both budgets in full.
func NewDecoder ¶
NewDecoder returns a new Decoder that reads from r.
When r exposes its length (it is a *bytes.Reader, *bytes.Buffer, or *strings.Reader — as the Bolt server's per-message decode path is, reading a reassembled message from a bytes.Reader), the decoder uses that length as an exact byte budget and rejects any length/count prefix that exceeds the bytes actually remaining. For any other reader the length is unknown and the decoder falls back to capping allocations at the default 16 MiB message ceiling.
func (*Decoder) PeekType ¶
PeekType returns the PackStream type of the next value without consuming it.
func (*Decoder) ReadInt ¶
ReadInt reads and returns an Integer value, regardless of width.
PackStream defines INT_8/INT_16/INT_32/INT_64 as fixed-width signed two's-complement integers. The byte/uint→int reinterpretation casts below preserve the wire bit pattern: they are the canonical decode and not unchecked overflows; gosec G115 is a false positive at each site.
func (*Decoder) ReadListHeader ¶
ReadListHeader reads the list marker and returns the number of elements. The caller is responsible for reading exactly that many values.
The count is validated before it is returned: each element occupies at least one wire byte, so a count exceeding the bytes still available is rejected with ErrLengthExceedsInput, and the count's decoded slot cost (listElemCost per element) is charged against the per-message decoded-memory budget, rejecting with ErrDecodedMemoryExceeded when it does not fit. A returned count is therefore always safe to pre-size with.
func (*Decoder) ReadMapHeader ¶
ReadMapHeader reads the map marker and returns the number of key/value pairs. The caller is responsible for reading exactly 2*n items.
The count is validated before it is returned: each entry is a key plus a value and so occupies at least two wire bytes, so a count exceeding half the bytes still available is rejected with ErrLengthExceedsInput, and the count's decoded entry cost (mapEntryCost per entry) is charged against the per-message decoded-memory budget, rejecting with ErrDecodedMemoryExceeded when it does not fit. A returned count is therefore always safe to pre-size with.
func (*Decoder) ReadNull ¶
ReadNull consumes the NULL marker. Returns an error if the next value is not NULL.
func (*Decoder) ReadString ¶
ReadString reads and returns a String value.
func (*Decoder) ReadStructHeader ¶
ReadStructHeader reads the struct marker and returns the tag byte and the number of fields. PackStream v2 supports only TinyStruct (0..15 fields).
The field count is validated before it is returned: each field occupies at least one wire byte, so a count exceeding the bytes still available is rejected with ErrLengthExceedsInput, and the count's decoded slot cost (listElemCost per field) is charged against the per-message decoded-memory budget. With at most 15 fields a single struct cannot amplify, but charging it keeps the budget an honest cumulative account across deeply structured messages.
func (*Decoder) ReadValue ¶
ReadValue reads a single PackStream value from the stream, returning it as a Go value. The concrete types returned are:
nil → NULL bool → Boolean int64 → Integer float64 → Float []byte → Bytes string → String []Value → List map[string]Value → Map Struct → Structure
func (*Decoder) Reset ¶
Reset points the decoder at a new underlying reader and recomputes the per-message byte budget from r (see NewDecoder). The decoded-memory budget (see ErrDecodedMemoryExceeded) is restored to its full per-message value. It is used by the decode pool to reuse Decoder objects across messages.
type EncodePool ¶
type EncodePool struct {
// contains filtered or unexported fields
}
EncodePool is a pool of Encoders backed by bytes.Buffer writers. It is safe for concurrent use.
func (*EncodePool) Get ¶
func (ep *EncodePool) Get(dst *bytes.Buffer) *Encoder
Get retrieves an Encoder from the pool, resetting it to write into dst. The caller must call Put when done.
func (*EncodePool) Put ¶
func (ep *EncodePool) Put(enc *Encoder)
Put returns enc to the pool for reuse.
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder writes PackStream-encoded values to an underlying buffered writer. It is NOT safe for concurrent use.
func NewEncoder ¶
NewEncoder returns a new Encoder that writes to w. The caller retains ownership of w.
func (*Encoder) Reset ¶
Reset points the encoder at a new underlying writer. Used by pool helpers to reuse Encoder objects.
func (*Encoder) WriteBytes ¶
WriteBytes encodes a PackStream Bytes value. It selects BYTES8, BYTES16, or BYTES32 based on the slice length. Returns ErrPayloadTooLarge when len(v) exceeds math.MaxUint32.
func (*Encoder) WriteFloat ¶
WriteFloat encodes a PackStream Float64 value (8-byte big-endian IEEE-754).
func (*Encoder) WriteInt ¶
WriteInt encodes a PackStream Integer value using the most compact representation: TinyInt for values in [-16, 127], then INT8/16/32/64.
The numeric reinterpretation casts below (byte(v), int8(v), int16(v), int32(v), uint64(v)) are bit-pattern-preserving two's-complement conversions mandated by the PackStream wire format; gosec G115 is a false positive here because the enclosing switch has already bounded v to the destination type's range before each cast.
Example ¶
ExampleEncoder_WriteInt shows the wire form of a TINY_INT. PackStream encodes any integer in the range -16..127 as a single byte equal to its two's complement, so 42 serialises to one byte: 0x2a.
package main
import (
"bytes"
"fmt"
"github.com/FlavioCFOliveira/GoGraph/bolt/packstream"
)
func main() {
var buf bytes.Buffer
enc := packstream.NewEncoder(&buf)
if err := enc.WriteInt(42); err != nil {
fmt.Println("write:", err)
return
}
if err := enc.Flush(); err != nil {
fmt.Println("flush:", err)
return
}
fmt.Printf("bytes: % x\n", buf.Bytes())
}
Output: bytes: 2a
func (*Encoder) WriteListHeader ¶
WriteListHeader writes the PackStream list marker for a list with n elements. The caller is responsible for encoding exactly n elements after this call. n must be non-negative and at most math.MaxUint32; otherwise the function returns ErrPayloadTooLarge.
func (*Encoder) WriteMapHeader ¶
WriteMapHeader writes the PackStream map marker for a map with n key/value pairs. The caller is responsible for encoding exactly 2n items (alternating keys and values). Returns ErrPayloadTooLarge when n exceeds math.MaxUint32.
func (*Encoder) WriteString ¶
WriteString encodes a PackStream String value. It selects TinyString (len 0..15), STRING8, STRING16, or STRING32. Returns ErrPayloadTooLarge when len(v) exceeds math.MaxUint32.
func (*Encoder) WriteStructHeader ¶
WriteStructHeader writes the PackStream struct marker byte (TinyStruct) and the tag byte. PackStream v2 supports only TinyStruct (0..15 fields). n must be in [0, 15].
func (*Encoder) WriteValue ¶
WriteValue encodes v into the stream using the Encoder. It dispatches on the concrete type of v.
Example ¶
ExampleEncoder_WriteValue round-trips a heterogeneous list through the Value-level helpers. WriteValue dispatches on the Go type; ReadValue reconstructs the same values (integers come back as int64, strings as string), so the decoded list equals the input.
package main
import (
"bytes"
"fmt"
"github.com/FlavioCFOliveira/GoGraph/bolt/packstream"
)
func main() {
in := []packstream.Value{int64(42), "go", true}
var buf bytes.Buffer
enc := packstream.NewEncoder(&buf)
if err := enc.WriteValue(in); err != nil {
fmt.Println("write:", err)
return
}
if err := enc.Flush(); err != nil {
fmt.Println("flush:", err)
return
}
dec := packstream.NewDecoder(&buf)
out, err := dec.ReadValue()
if err != nil {
fmt.Println("read:", err)
return
}
list := out.([]packstream.Value)
fmt.Println("len:", len(list))
fmt.Printf("values: %v %q %v\n", list[0], list[1], list[2])
}
Output: len: 3 values: 42 "go" true
type Struct ¶
type Struct struct {
// Tag is the single-byte structure signature.
Tag byte
// Fields contains the structure's fields in declaration order.
Fields []Value
}
Struct represents a PackStream Structure with its tag byte and ordered fields.
type Type ¶
type Type uint8
Type identifies the PackStream type of the next value in the stream.
const ( // TypeNull represents the PackStream NULL type. TypeNull Type = iota // TypeBool represents the PackStream Boolean type. TypeBool // TypeInt represents the PackStream Integer type. TypeInt // TypeFloat represents the PackStream Float type (Float64). TypeFloat // TypeBytes represents the PackStream Bytes type. TypeBytes // TypeString represents the PackStream String type. TypeString // TypeList represents the PackStream List type. TypeList // TypeMap represents the PackStream Map type. TypeMap // TypeStruct represents the PackStream Structure type. TypeStruct )
Source Files