codec

package
v0.27.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package codec — declares the optional Appender extension that hot-path encoders (NDJSON, JSON, text) implement so callers can write into a caller-supplied buffer without paying for an intermediate allocation.

Codecs that satisfy Codec but do NOT implement Appender remain valid; the logger and other consumers detect support with a runtime type assertion and fall back to Marshal + copy when the optional interface is absent.

Package codec declares the domain contracts every format codec in the SDK implements. Consumers interact with codecs through the pkg/v1/data/codec facade; concrete implementations live in internal/service/data/codec/<format>/.

This package is interface-only: no runtime state, no registrations, no format-specific knowledge. Codecs register themselves via the registry in registry.go when their service subpackage is imported.

Package codec — range 0.2.2.* (ADR 0005 core/data/codec block).

Package codec — declares the registry's boot-time sentinel and the family's three misuse refusals, EmptyInput, TargetInvalid and ValueInvalid (#1101). Each var name equals its Reason in SCREAMING_SNAKE form.

Package codec — declares the typed Format string and its helpers. The zero Format ("") is invalid — consumers obtain a Format via the pkg/v1/data/codec constants or via the FromMIME / FromExtension helpers.

Package codec — holds the process-wide Codec registry. Service-level codec packages register themselves via package-level var initialisers when the package is imported.

Index

Constants

View Source
const (
	// FieldOffset is the key of a refusal's 0-based byte offset.
	FieldOffset = "offset"
	// FieldLine is the key of a refusal's 1-based line.
	FieldLine = "line"
	// FieldColumn is the key of a refusal's 1-based column, counted as the format
	// counts it: characters for yaml and toml, bytes for csv.
	FieldColumn = "column"
)

The keys a decode refusal carries its position under (#1125): every codec writes them through PositionFields or by these names, so PositionOf reads any codec's position back.

View Source
const CodeBoundInvalid errs.Code = 0x00_02_02_07 // 0.2.2.7

CodeBoundInvalid identifies a byte ceiling of zero or below passed to BoundedStreamingCodec.NewBoundedDecoder: a bound is positive, there is no value meaning unlimited (ADR 0031, #891).

View Source
const CodeCodecNil errs.Code = 0x00_02_02_05 // 0.2.2.5

CodeCodecNil identifies a Register call made with a nil Codec — a nil-argument programmer error distinct from a duplicate registration. Surfaced via panic at boot (see registry.go), mirroring writer.CodeWriterNil.

View Source
const CodeDuplicateRegistration errs.Code = 0x00_02_02_01 // 0.2.2.1

CodeDuplicateRegistration identifies an init-time collision on the codec registry; two codecs tried to claim the same Name.

View Source
const CodeEmptyInput errs.Code = 0x00_02_02_02 // 0.2.2.2

CodeEmptyInput identifies Unmarshal being called with a zero-byte slice for a format that has no empty document: json, xml, bson, cbor, msgpack, tlv.

View Source
const CodeKeyEmpty errs.Code = 0x00_02_02_06 // 0.2.2.6

CodeKeyEmpty identifies a Register call whose codec declares an empty Name, MIME type or extension — one that is empty once normalised the way its Lookup normalises it. Such a key is what a package variable read before its initialiser ran holds, so refusing it names the codec at boot instead of letting two of them collide on "". Surfaced via panic (see registry.go), like CodeCodecNil.

View Source
const CodeTargetInvalid errs.Code = 0x00_02_02_03 // 0.2.2.3

CodeTargetInvalid identifies Unmarshal or Decode being called with a target that is not a non-nil pointer to a storable value — a programmer's error every codec reports before reading a byte (#1101).

View Source
const CodeValueInvalid errs.Code = 0x00_02_02_04 // 0.2.2.4

CodeValueInvalid identifies Marshal being called on a value no codec can serialise — a channel, a function, an unsafe pointer —, refused before encoding (#1101).

Variables

View Source
var (
	// DuplicateRegistration is the conflict Register refuses at boot: a distinct
	// codec already holds the Format name, MIME type or extension a second one
	// claims. It is never returned — Register panics with conflictText of it, so
	// the panic carries the dotted-quad header AND the key that collided.
	DuplicateRegistration = errs.Static(CodeDuplicateRegistration, "DUPLICATE_REGISTRATION",
		"A codec is already registered under that name, MIME type or extension",
		"core/data/codec.Register: a distinct codec already holds this Format name, MIME type or extension",
		0,
		0)

	// EmptyInput is what Unmarshal returns for zero bytes in a format that has
	// no empty document — json, xml, bson, cbor, msgpack, tlv —, before any
	// parsing (#1101). yaml and toml read an empty document and leave the
	// target unchanged.
	EmptyInput = errs.Static(CodeEmptyInput, "EMPTY_INPUT",
		"codec input is empty",
		"core/data/codec: Unmarshal called with no bytes for a format that has no empty document",
		400,
		0)

	// TargetInvalid is what CheckTarget returns, and so every codec's Unmarshal
	// and Decode, for a target that is nil or not a pointer (#1101). Its "target"
	// field names the type.
	TargetInvalid = errs.Static(CodeTargetInvalid, "TARGET_INVALID",
		"codec target must be a non-nil pointer",
		"core/data/codec: Unmarshal or Decode called with a nil or non-pointer target; the target field names its type",
		0,
		0)

	// ValueInvalid is what CheckValue returns for a value no codec encodes — a
	// channel, a function, an unsafe pointer (#1101). Its "type" field names the
	// type.
	ValueInvalid = errs.Static(CodeValueInvalid, "VALUE_INVALID",
		"codec value cannot be encoded",
		"core/data/codec: Marshal called with a value no codec encodes; the type field names it",
		0,
		0)
)
View Source
var BoundInvalid = errs.Static(CodeBoundInvalid, "BOUND_INVALID",
	"A byte ceiling must be positive",
	"core/data/codec: NewBoundedDecoder was given a byte ceiling of zero or below",
	0,
	0)

BoundInvalid is the refusal of a byte ceiling of zero or below by BoundedStreamingCodec.NewBoundedDecoder (ADR 0031, #891).

Functions

func CheckTarget added in v0.27.0

func CheckTarget(v any) error

CheckTarget refuses v as the target of an Unmarshal or a Decode unless it is a non-nil pointer: TargetInvalid, its "target" field naming v's type, "nil" for an untyped nil. Every codec calls it before reading a byte (#1101) — bson also takes a non-nil map, flatbuffers a BytesAcceptor that is not a pointer —, so a programmer's error is never reported as corrupt input. It allocates only when it refuses.

func CheckValue added in v0.27.0

func CheckValue(v any) error

CheckValue refuses v as the value of a Marshal when its kind is one no codec encodes — a channel, a function, an unsafe pointer —: ValueInvalid, its "type" field naming v's type. The Marshal of the codecs that encode any Go value (json, xml, yaml, toml, cbor, msgpack, bson) calls it first (#1101); Append, the hot path, leaves the refusal to the encoder. It allocates only when it refuses.

func IsMisuse added in v0.27.0

func IsMisuse(err error) bool

IsMisuse reports whether err, or an error it wraps, is a codec refusing how it was called rather than a byte it read: EMPTY_INPUT, TARGET_INVALID or VALUE_INVALID, the family's codes 0.2.2.2 to 0.2.2.4 (#1101). A TARGET_INVALID or a VALUE_INVALID is the program's own bug, an HTTP server answers it with 500; an EMPTY_INPUT is a body that never came, 400.

func IsTooDeep added in v0.27.0

func IsTooDeep(err error) bool

IsTooDeep reports whether err, or an error it wraps, is a codec's refusal of input nested past a depth bound: the family's DEPTH_EXCEEDED reason (#894).

func IsTooLarge added in v0.27.0

func IsTooLarge(err error) bool

IsTooLarge reports whether err, or an error it wraps, is a codec's refusal of input past a size bound — the family's SIZE_EXCEEDED reason, which every bounded codec gives its byte, node, pair and entry bounds (#894), or the two older spellings kept as they were: multipart's LIMIT_EXCEEDED and strictjson's DOCUMENT_TOO_LARGE. An HTTP server answers it with 413.

func PositionFields added in v0.27.0

func PositionFields(p PositionValue) []errs.FieldValue

PositionFields returns the fields a decode refusal carries p under, for errs.Wrap: FieldOffset, FieldLine and FieldColumn, each only when p knows it — Line and Column when not zero, Offset when not zero or when p has no line (see PositionValue). Every codec places a refusal through it or through those three keys, so PositionOf reads any codec's position back (#1125).

Types

type Appender

type Appender interface {
	Codec
	// Append encodes v and appends the bytes onto dst. The returned slice
	// MUST contain the prior contents of dst followed by the encoded form.
	Append(dst []byte, v any) (out []byte, err error)
}

Appender is the optional extension implemented by codecs that can encode directly into a caller-supplied byte slice instead of allocating their own. Hot paths (logger record formatting, NDJSON streams, batch sinks) consume this interface to avoid the buffer-copy cost imposed by Marshal.

Implementations MUST:

  • Append the encoded representation of v onto dst (typically via the standard append(dst, …) idiom).
  • Return the (possibly re-allocated) buffer back to the caller.
  • Be safe for concurrent use by multiple goroutines.

The returned buffer's length grows by the number of bytes written; the underlying array MAY be a fresh allocation when dst's capacity is too small. Callers retain ownership of dst and SHOULD reuse it across calls.

type BoundedStreamingCodec added in v0.27.0

type BoundedStreamingCodec interface {
	StreamingCodec
	NewBoundedDecoder(r io.Reader, maxBytes int64) (dec Decoder, err error)
}

BoundedStreamingCodec is the optional extension of a streaming codec whose Decoder holds a byte ceiling the caller chooses (#891) — a sibling of StreamingCodec, never a wider Decoder (ADR 0039).

NewBoundedDecoder returns a Decoder that refuses a value longer than maxBytes — counted from the end of the previous value to the end of this one, so a long stream of small values is never refused — with the codec's SIZE_EXCEEDED, which IsTooLarge reads, after reading at most maxBytes+1 bytes of it: a hostile stream grows the process by the ceiling, not by the size of its value. A ceiling of zero or below is refused with BoundInvalid: a bound has no value meaning unlimited (ADR 0031).

type Codec

type Codec interface {
	Name() string
	MIMETypes() []string
	Extensions() []string
	Marshal(v any) (data []byte, err error)
	Unmarshal(data []byte, v any) (err error)
}

Codec is the minimum contract every format codec satisfies. Instances MUST be safe for concurrent use; format-specific options are supplied via the codec's constructor, not via the Codec interface.

func Lookup

func Lookup(f Format) (c Codec, ok bool)

Lookup returns the codec registered under f.

IFACE-PLUGIN: the registry stores plug-in codec instances behind the Codec interface — concrete types are intentionally unexported per-format.

func LookupExt

func LookupExt(ext string) (c Codec, ok bool)

LookupExt returns the codec whose extension list contains ext.

IFACE-PLUGIN: the registry stores plug-in codec instances behind the Codec interface — concrete types are intentionally unexported per-format.

func LookupMIME

func LookupMIME(raw string) (c Codec, ok bool)

LookupMIME returns the codec whose MIME list matches. MIME parameters (e.g. "; charset=utf-8") are stripped before the index lookup so that "application/json; charset=utf-8" resolves like "application/json".

IFACE-PLUGIN: the registry stores plug-in codec instances behind the Codec interface — concrete types are intentionally unexported per-format.

func Register

func Register(c Codec) Codec

Register inserts c into the registry and returns it so callers can bind the singleton to a typed package-level variable like `var Codec codec.Codec = codec.Register(&jsonCodec{})`. Panics on nil argument, on an empty Name / MIME / extension (KEY_EMPTY), or on a duplicate Name / MIME / extension.

Register reads c.Name, c.MIMETypes and c.Extensions through the Codec interface, so the initialiser of the calling package variable does NOT depend, in the sense of the Go specification's package initialization rules, on any package variable those methods read: a dependency is only recorded for a method referenced through a non-interface type. A codec whose methods read package variables must therefore make that dependency visible in the initialiser itself — or, as the SDK's codecs do, return fresh literals from the methods — otherwise those variables may still be unset when Register runs (the gc compiler hides this by initialising constant composite literals statically; garble -literals does not).

IFACE-PLUGIN: the registry hands plug-in codec instances back to callers so each domain can keep its concrete codec type unexported; the only stable contract is the Codec interface itself.

"Nil" here means UNUSABLE, not only an untyped nil: a typed nil pointer and a plug-in whose type is not comparable both satisfy the port and neither can serve one call (see internal/kernel/plugin).

type Decoder

type Decoder interface {
	Decode(v any) (err error)
	More() (ok bool)
}

Decoder reads one or more values from the wrapped reader.

type Encoder

type Encoder interface {
	Encode(v any) (err error)
	Close() (err error)
}

Encoder writes one or more values to the wrapped writer.

type Format

type Format string

Format is the typed string identifier of a codec.

func Available

func Available() []Format

Available returns the sorted list of registered Formats, or nil before any Register.

func Detect added in v0.27.0

func Detect(prefix []byte) (f Format, ok bool)

Detect returns the Format whose registered codec recognises prefix — the first bytes of an input, of which it reads at most 512 as http.DetectContentType does — through its Sniffer (#909). It asks every codec implementing Sniffer, in registration order, and answers only when exactly one claims the bytes: none, or two, is (Format(""), false). It allocates nothing.

func (Format) Known

func (f Format) Known() bool

Known reports whether f has been registered in the codec registry.

func (Format) String

func (f Format) String() string

String implements fmt.Stringer and returns the raw identifier.

type PositionValue added in v0.27.0

type PositionValue struct {
	// Offset is the 0-based byte offset of the problem in the input.
	Offset int
	// Line is the 1-based line of the problem, zero when unknown.
	Line int
	// Column is the 1-based column of the problem, zero when unknown.
	Column int
}

PositionValue places a decode refusal in its input (#1125). Offset is the 0-based byte, Line and Column are 1-based, a zero Line or Column is unknown. A text format places by line and column — characters for yaml and toml, bytes for csv — and adds the offset when it knows a non-zero one; a binary format places by offset alone, so with no Line the Offset is meant even when zero: the input's first byte. json carries the offset and xml the line their standard library's errors give.

func PositionOf added in v0.27.0

func PositionOf(err error) (p PositionValue, ok bool)

PositionOf reads back the position a codec's refusal carries, in err or an error it wraps: ok is false when it carries none. A key set twice along the chain keeps its first value — the origin's, as errs.Wrap keeps the origin. A key whose value is not an integer is not a position.

type Sniffer added in v0.27.0

type Sniffer interface {
	Codec
	Sniff(prefix []byte) (confident bool)
}

Sniffer is the optional extension of a codec whose encoding opens with a recognisable head (#909). Sniff reports whether prefix — the first bytes of an input, at most 512 of them, possibly cut inside a value — is confidently this codec's: a head no other format's output starts with, not merely bytes the codec could parse. It reads nothing past prefix, keeps nothing and allocates nothing. Detect asks every Sniffer and answers only when exactly one claims the bytes.

type StreamingCodec

type StreamingCodec interface {
	Codec
	NewEncoder(w io.Writer) Encoder
	NewDecoder(r io.Reader) Decoder
}

StreamingCodec is the optional extension for formats that support incremental encode/decode over io.Writer / io.Reader. Codecs that CAN stream SHOULD implement it; consumers detect support with a type assertion.

Directories

Path Synopsis
Package asn1 declares the error codes and the sentinels of the ASN.1 DER codec, internal/service/data/codec/asn1 — range 0.3.9.* (ADR 0005 service/data/codec/asn1 block).
Package asn1 declares the error codes and the sentinels of the ASN.1 DER codec, internal/service/data/codec/asn1 — range 0.3.9.* (ADR 0005 service/data/codec/asn1 block).
Package baseenc declares the error codes and the sentinels of the base-N codec family (base64, base64url, base32, base16, hex, ascii85, base45, base58, base62), internal/service/data/codec/baseenc — range 0.3.24.* (ADR 0006 service/data/codec/baseenc block).
Package baseenc declares the error codes and the sentinels of the base-N codec family (base64, base64url, base32, base16, hex, ascii85, base45, base58, base62), internal/service/data/codec/baseenc — range 0.3.24.* (ADR 0006 service/data/codec/baseenc block).
Package bson declares the error codes and the sentinels of the BSON codec, internal/service/data/codec/bson — range 0.3.36.* (ADR 0021 service/data/codec/bson block).
Package bson declares the error codes and the sentinels of the BSON codec, internal/service/data/codec/bson — range 0.3.36.* (ADR 0021 service/data/codec/bson block).
Package cbor declares the error codes and the sentinels of the CBOR codec, internal/service/data/codec/cbor — range 0.3.6.* (ADR 0005 service/data/codec/cbor block).
Package cbor declares the error codes and the sentinels of the CBOR codec, internal/service/data/codec/cbor — range 0.3.6.* (ADR 0005 service/data/codec/cbor block).
Package csv declares the error codes and the sentinels of the CSV codec, internal/service/data/codec/csv — range 0.3.8.* (ADR 0005 service/data/codec/csv block).
Package csv declares the error codes and the sentinels of the CSV codec, internal/service/data/codec/csv — range 0.3.8.* (ADR 0005 service/data/codec/csv block).
Package flatbuffers declares the error codes and the sentinels of the FlatBuffers passthrough codec, internal/service/data/codec/flatbuffers — range 0.3.23.* (ADR 0006 service/data/codec/flatbuffers block).
Package flatbuffers declares the error codes and the sentinels of the FlatBuffers passthrough codec, internal/service/data/codec/flatbuffers — range 0.3.23.* (ADR 0006 service/data/codec/flatbuffers block).
Package form declares the error codes and the sentinels of the application/x-www-form-urlencoded codec, internal/service/data/codec/form — range 0.3.40.* (ADR 0005 service/data/codec/form block).
Package form declares the error codes and the sentinels of the application/x-www-form-urlencoded codec, internal/service/data/codec/form — range 0.3.40.* (ADR 0005 service/data/codec/form block).
Package jcs declares the error codes and the sentinels of the RFC 8785 canonicalization package, internal/service/data/codec/jcs — range 0.3.102.* (ADR 0102 amendment of 2026-11-03).
Package jcs declares the error codes and the sentinels of the RFC 8785 canonicalization package, internal/service/data/codec/jcs — range 0.3.102.* (ADR 0102 amendment of 2026-11-03).
Package json declares the error codes and the sentinels of the JSON codec, internal/service/data/codec/json — range 0.3.2.* (ADR 0005 service/data/codec/json block).
Package json declares the error codes and the sentinels of the JSON codec, internal/service/data/codec/json — range 0.3.2.* (ADR 0005 service/data/codec/json block).
Package jsonpatch declares the error codes and the sentinels of the JSON-difference package (RFC 6902 operations, not a codec), internal/service/data/codec/jsonpatch — range 0.3.90.* (ADR 0143 service/data/codec/jsonpatch block).
Package jsonpatch declares the error codes and the sentinels of the JSON-difference package (RFC 6902 operations, not a codec), internal/service/data/codec/jsonpatch — range 0.3.90.* (ADR 0143 service/data/codec/jsonpatch block).
Package jsonschema declares the error codes and the sentinels of the JSON Schema 2020-12 validator, internal/service/data/codec/jsonschema — range 0.3.103.* (ADR 0210).
Package jsonschema declares the error codes and the sentinels of the JSON Schema 2020-12 validator, internal/service/data/codec/jsonschema — range 0.3.103.* (ADR 0210).
Package msgpack declares the error codes and the sentinels of the MessagePack codec, internal/service/data/codec/msgpack — range 0.3.7.* (ADR 0005 service/data/codec/msgpack block).
Package msgpack declares the error codes and the sentinels of the MessagePack codec, internal/service/data/codec/msgpack — range 0.3.7.* (ADR 0005 service/data/codec/msgpack block).
Package multipart declares the error codes and the sentinels of the multipart/form-data codec, internal/service/data/codec/multipart — range 0.3.41.* (ADR 0005 service/data/codec/multipart block).
Package multipart declares the error codes and the sentinels of the multipart/form-data codec, internal/service/data/codec/multipart — range 0.3.41.* (ADR 0005 service/data/codec/multipart block).
Package ndjson declares the error codes and the sentinels of the NDJSON codec, internal/service/data/codec/ndjson — range 0.3.11.* (ADR 0005 service/data/codec/ndjson block).
Package ndjson declares the error codes and the sentinels of the NDJSON codec, internal/service/data/codec/ndjson — range 0.3.11.* (ADR 0005 service/data/codec/ndjson block).
Package pem declares the error codes and the sentinels of the PEM codec, internal/service/data/codec/pem — range 0.3.10.* (ADR 0005 service/data/codec/pem block).
Package pem declares the error codes and the sentinels of the PEM codec, internal/service/data/codec/pem — range 0.3.10.* (ADR 0005 service/data/codec/pem block).
Package scratch provides shared, size-bounded recycling primitives for the service codecs.
Package scratch provides shared, size-bounded recycling primitives for the service codecs.
Package strictjson declares the error codes and the sentinels of the strict JSON decoder (not a codec), internal/service/data/codec/strictjson — range 0.3.72.* (ADR 0102 service/data/codec/strictjson block).
Package strictjson declares the error codes and the sentinels of the strict JSON decoder (not a codec), internal/service/data/codec/strictjson — range 0.3.72.* (ADR 0102 service/data/codec/strictjson block).
Package tlv declares the error codes and the sentinels of the TLV codec, internal/service/data/codec/tlv — range 0.3.22.* (ADR 0006 service/data/codec/tlv block).
Package tlv declares the error codes and the sentinels of the TLV codec, internal/service/data/codec/tlv — range 0.3.22.* (ADR 0006 service/data/codec/tlv block).
Package toml declares the error codes and the sentinels of the TOML codec, internal/service/data/codec/toml — range 0.3.5.* (ADR 0005 service/data/codec/toml block).
Package toml declares the error codes and the sentinels of the TOML codec, internal/service/data/codec/toml — range 0.3.5.* (ADR 0005 service/data/codec/toml block).
Package xml declares the error codes and the sentinels of the XML codec, internal/service/data/codec/xml — range 0.3.3.* (ADR 0005 service/data/codec/xml block).
Package xml declares the error codes and the sentinels of the XML codec, internal/service/data/codec/xml — range 0.3.3.* (ADR 0005 service/data/codec/xml block).
Package yaml declares the error codes and the sentinels of the YAML codec, internal/service/data/codec/yaml — range 0.3.4.* (ADR 0005 service/data/codec/yaml block).
Package yaml declares the error codes and the sentinels of the YAML codec, internal/service/data/codec/yaml — range 0.3.4.* (ADR 0005 service/data/codec/yaml block).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL