framer

framer — message boundaries over stream I/O

Languages: English | 简体中文 | 日本語 | Español | Français

Portable message framing for Go. Preserve one-message-per-Read/Write over stream transports.

Scope: framer solves message boundary preservation across stream transports.

At a glance

• Solve message boundary problems for byte streams (TCP, Unix stream, pipes).
• Pass-through on boundary-preserving transports (UDP, Unix datagram, WebSocket, SCTP).
• Portable wire format; configurable byte order.

Why

Many transports are byte streams (TCP, Unix stream, pipes). A single Read may return a partial application message, or several messages concatenated. framer restores message boundaries: in stream mode, one Read returns exactly one message payload, and one Write emits exactly one framed message.

Protocol adaptation

• BinaryStream (stream transports: TCP, TLS-over-TCP, Unix stream, pipes): adds a length prefix; reads/writes whole messages.
• SeqPacket (e.g., SCTP, WebSocket): pass-through; the transport already preserves boundaries.
• Datagram (e.g., UDP, Unix datagram): pass-through; boundary already preserved.

Select at construction time via WithProtocol(...) (read/write variants exist) or via transport helpers (see Options).

Wire format

Compact variable-length length prefix, followed by payload bytes. Byte order for the extended length is configurable: WithByteOrder, or per-direction WithReadByteOrder / WithWriteByteOrder.

Frame data format

The framing scheme used by framer is intentionally compact:

• Header byte H0 + optional extended length bytes.
• Let L be the payload length in bytes.
  • If 0 ≤ L ≤ 253 (0x00..0xFD): H0 = L. No extra length bytes.
  • If 254 ≤ L ≤ 65535 (0x0000..0xFFFF): H0 = 0xFE and the next 2 bytes encode L as an unsigned 16‑bit integer in the configured byte order.
  • If 65536 ≤ L ≤ 2^56-1: H0 = 0xFF and the next 7 bytes carry L as a 56‑bit integer, laid out in the configured byte order.
    • Big‑endian: bytes [1..7] are the big‑endian lower 56 bits of L.
    • Little‑endian: bytes [1..7] are the little‑endian lower 56 bits of L.

Limits and errors:

• The maximum supported payload length is 2^56-1; larger values result in framer.ErrTooLong.
• When a read‑side limit is configured (WithReadLimit), lengths exceeding the limit fail with framer.ErrTooLong.

Quick start

Install with go get:

go get code.hybscloud.com/framer

c1, c2 := net.Pipe()
defer c1.Close()
defer c2.Close()

w := framer.NewWriter(c1, framer.WithWriteTCP())
r := framer.NewReader(c2, framer.WithReadTCP())

go func() { _, _ = w.Write([]byte("hello")) }()

buf := make([]byte, 64)
n, err := r.Read(buf)
if err != nil {
    panic(err)
}
fmt.Printf("got: %q\n", buf[:n])

Options

• WithProtocol(proto Protocol) — choose BinaryStream, SeqPacket, or Datagram (read/write variants available).
• Byte order: WithByteOrder, or WithReadByteOrder / WithWriteByteOrder.
• WithReadLimit(n int) — cap maximum message payload size when reading.
• WithRetryDelay(d time.Duration) — configure would-block policy; helpers: WithNonblock() / WithBlock().

Transport helpers (presets):

• WithReadTCP / WithWriteTCP (BinaryStream, network‑order BigEndian)
• WithReadUDP / WithWriteUDP (Datagram, BigEndian)
• WithReadWebSocket / WithWriteWebSocket (SeqPacket, BigEndian)
• WithReadSCTP / WithWriteSCTP (SeqPacket, BigEndian)
• WithReadUnix / WithWriteUnix (BinaryStream, BigEndian)
• WithReadUnixPacket / WithWriteUnixPacket (Datagram, BigEndian)
• WithReadLocal / WithWriteLocal (BinaryStream, native byte order)

Everything else: see GoDoc: https://pkg.go.dev/code.hybscloud.com/framer

Semantics Contract

Error taxonomy

Outcome tables

Reader.Read(p []byte) (n int, err error) — BinaryStream mode

Writer.Write(p []byte) (n int, err error) — BinaryStream mode

Reader.WriteTo(dst io.Writer) (n int64, err error)

Writer.ReadFrom(src io.Reader) (n int64, err error)

Forwarder.ForwardOnce() (n int, err error)

Operation classification

Blocking policy

By default, framer is non-blocking (WithNonblock()): ErrWouldBlock is returned immediately.

• WithBlock() — yield (runtime.Gosched) and retry on would-block
• WithRetryDelay(d) — sleep d and retry on would-block
• Negative RetryDelay (default) — return ErrWouldBlock immediately

No method hides blocking unless explicitly configured.

Fast paths

framer implements stdlib copy fast paths to interoperate with io.Copy-style engines and iox.CopyPolicy:

• (*Reader).WriteTo(io.Writer) — efficiently transfers framed message payloads to dst.

  • Stream (BinaryStream): processes one framed message at a time and writes only the payload bytes to dst. If ReadLimit == 0, an internal default cap (64KiB) is used; messages larger than this cap return framer.ErrTooLong.
  • Packet (SeqPacket/Datagram): pass-through (reads bytes, writes bytes).
  • Semantic errors framer.ErrWouldBlock and framer.ErrMore are propagated unchanged with the progress count reflecting bytes written.

• (*Writer).ReadFrom(io.Reader) — chunk-to-message: each successful Read chunk from src is encoded as a single framed message.

  • This is efficient but does not preserve application message boundaries from src.
  • On boundary-preserving protocols it effectively behaves like pass-through.
  • Semantic errors framer.ErrWouldBlock and framer.ErrMore are propagated unchanged with progress counts.

Recommendation: prefer iox.CopyPolicy with a retry-aware policy (e.g., PolicyRetry) in non-blocking loops so ErrWouldBlock / ErrMore are handled explicitly.

Forwarding

• Wire proxying (byte engines): use iox.CopyPolicy and standard io fast paths (WriterTo/ReaderFrom). This maximizes throughput when you don't need to preserve higher-level boundaries.
• Message relay (preserve boundaries): use framer.NewForwarder(dst, src, ...) and call ForwardOnce() in your poll loop. It decodes exactly one framed message from src and re-encodes it as exactly one framed message to dst.
  • Non-blocking semantics: ForwardOnce returns (n>0, framer.ErrWouldBlock|framer.ErrMore) when partial progress happened; retry the same Forwarder instance later to complete.
  • Limits: io.ErrShortBuffer when the internal buffer is too small for the message; framer.ErrTooLong when a message exceeds the configured WithReadLimit.
  • Zero‑alloc steady state after construction; the internal scratch buffer is reused per message.

License

MIT — see LICENSE.