codec

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: MIT Imports: 30 Imported by: 0

README

codec

import "github.com/kitsunium/sdk/pkg/v1/data/codec"

Package codec is the universal encoder/decoder dispatch facade.

One verb, twenty-four formats. Marshal and Unmarshal reach every encoding the SDK ships — JSON, YAML, CBOR, MessagePack, BSON, NDJSON, XML, TOML, CSV, urlencoded forms, multipart/form-data, ASN.1 DER, PEM, TLV, FlatBuffers, plus the nine base-N variants (base64, base64url, base32, base16, base45, base58, base62, hex, ascii85). Format-swap at runtime is a single string change.

Goals
  • One verb, many formats. codec.Marshal(f, v) reaches every registered wire format — your call site doesn't change when you swap JSON for CBOR.
  • Stable Format strings. Once registered, a Format string is frozen across minor versions. Code that compiles today keeps compiling.
  • Streaming when it pays. Codecs that implement StreamingCodec get NewEncoder / NewDecoder automatically — no need to buffer megabytes.
  • Zero registration code. Blank-import the package; the 16 codecs register themselves as Go initialises them. No Register() calls in consumer code. A program that needs one format imports that format's package beneath this one instead, and links it alone.
  • Append for hot paths. Codecs implementing Appender let you reuse a []byte buffer across calls — handy in tight loops.
  • Broadcast marshal. MarshalMany(v, formats...) serialises the same value to N formats at once — content negotiation, replication, multi-protocol message buses.
What's shipped — all 24 formats

Single blank import (`import _ "github.com/kitsunium/sdk/pkg/v1/data/codec"`) activates the entire list. The Streaming column marks codecs that implement core/data/codec.StreamingCodec and unlock NewEncoder / NewDecoder. Every codec implements core/data/codec.Appender.

| Format       | Go constant       | MIME                       | Extension           | Streaming | Best for |
|--------------|-------------------|----------------------------|---------------------|:---------:|----------|
| json         | codec.JSON        | application/json           | .json               | yes       | API responses, public configs |
| ndjson       | codec.NDJSON      | application/x-ndjson       | .ndjson / .jsonl    | yes       | streaming logs, line-oriented batches |
| xml          | codec.XML         | application/xml            | .xml                | yes       | legacy integrations |
| csv          | codec.CSV         | text/csv                   | .csv                | yes       | tabular exports |
| form         | codec.Form        | application/x-www-form-urlencoded | .form / .urlencoded | —  | HTML form bodies, query strings |
| asn1-der     | codec.ASN1DER     | application/pkix-cert      | .der / .cer         | —         | crypto / X.509 artefacts |
| pem          | codec.PEM         | application/x-pem-file     | .pem / .crt / .key  | yes       | block-wrapped DER (certs, keys) |
| yaml         | codec.YAML        | application/yaml           | .yaml / .yml        | yes       | human-edited configs |
| toml         | codec.TOML        | application/toml           | .toml               | yes       | app configs (strict typing) |
| cbor         | codec.CBOR        | application/cbor           | .cbor               | yes       | IoT / mobile (RFC 8949) |
| msgpack      | codec.MsgPack     | application/msgpack        | .msgpack / .mpk     | yes       | RPC payloads |
| bson         | codec.BSON        | application/bson           | .bson               | —         | MongoDB documents (top level must be a document) |
| multipart    | codec.Multipart   | multipart/form-data        | —                   | yes       | uploads / form posts (native shape codec.MultipartForm) |
| tlv          | codec.TLV         | application/x-tlv          | .tlv                | yes       | custom binary streams, self-describing |
| flatbuffers  | codec.FlatBuffers | application/x-flatbuffers  | .fbs / .bin         | —         | zero-copy passthrough |
| base64       | codec.Base64      | application/base64         | .b64 / .base64      | yes       | text-safe wrap (JSON → base-N) |
| base64url    | codec.Base64URL   | application/base64url      | .b64url             | yes       | URL-safe base64 |
| base32       | codec.Base32      | application/base32         | .b32                | yes       | larger alphabet for human-typed tokens |
| base16       | codec.Base16      | application/base16         | .b16                | yes       | hex-like text (uppercase) |
| base45       | codec.Base45      | application/base45         | .b45                | yes       | QR-code alphanumeric payloads (RFC 9285) |
| base58       | codec.Base58      | application/base58         | .b58                | yes       | Bitcoin-style short IDs (4 KiB input cap) |
| base62       | codec.Base62      | application/base62         | .b62                | yes       | alphanumeric short IDs (4 KiB input cap) |
| hex          | codec.Hex         | application/hex            | .hex                | yes       | classic hex pair encoding (lowercase) |
| ascii85      | codec.ASCII85     | application/ascii85        | .a85                | yes       | 4-byte to 5-char text packing |

Runtime introspection: codec.Available() returns the live registry list, codec.FromMIME and codec.FromExtension resolve from external metadata.

Quick start
package main

import (
    "fmt"

    "github.com/kitsunium/sdk/pkg/v1/data/codec"
)

type User struct {
    Name string `json:"name" cbor:"name" yaml:"name"`
    Age  int    `json:"age"  cbor:"age"  yaml:"age"`
}

func main() {
    u := User{Name: "Ada", Age: 36}
    for _, f := range []codec.Format{codec.JSON, codec.CBOR, codec.YAML, codec.MsgPack, codec.Base64} {
        data, _ := codec.Marshal(f, u)
        var back User
        _ = codec.Unmarshal(f, data, &back)
        fmt.Printf("%-10s %d bytes  → %#v\n", f, len(data), back)
    }
}
Activation

A single blank import activates the full registry:

import _ "github.com/kitsunium/sdk/pkg/v1/data/codec"

The package is the aggregate of the sixteen per-format packages beneath it — asn1, baseenc, bson, cbor, csv, flatbuffers, form, json, msgpack, multipart, ndjson, pem, tlv, toml, xml and yaml — and blank-imports each. Every one of them imports its codec, which registers itself in the registry as Go initialises it, once however many packages import it. Consumers never call a Register() function — registration is a side-effect. Importing one of those packages instead registers its format alone, links nothing of the others, and names that format's error codes (ADR 0134).

Extension interfaces

A registered codec MAY implement one or both of these optional interfaces (assert at the call site):

// Append into a caller-supplied buffer — drops the fresh-slice
// allocation Marshal returns (see BENCH.md for per-codec numbers;
// the remaining allocs are the encoder's, not the facade's).
type Appender interface {
    Append(dst []byte, v any) ([]byte, error)
}

// Streaming — open Encoder/Decoder around an io.Writer/io.Reader.
type StreamingCodec interface {
    NewEncoder(w io.Writer) Encoder
    NewDecoder(r io.Reader) Decoder
}

// Bounded streaming — a Decoder refusing any value past maxBytes
// (JSON, XML, CBOR; reached through NewBoundedDecoder).
type BoundedStreamingCodec interface {
    StreamingCodec
    NewBoundedDecoder(r io.Reader, maxBytes int64) (Decoder, error)
}

// Sniffing — a confident claim on an input's first bytes, which
// Detect asks of every codec (JSON, NDJSON, XML, PEM, BSON).
type Sniffer interface {
    Sniff(prefix []byte) (confident bool)
}

StreamingUnsupported is returned from NewEncoder / NewDecoder when the resolved codec does not satisfy StreamingCodec, and from NewBoundedDecoder when it does not satisfy BoundedStreamingCodec.

Durations

A time.Duration follows one rule per kind of format (#1126). The formats people edit by hand — YAML and TOML — write its String(), "1h30m0s", and read it with time.ParseDuration, refusing a string without its unit. The binary formats — CBOR, MessagePack, BSON — write an integer of int64 nanoseconds, no standard tag carrying a duration. JSON keeps the integer encoding/json writes, the standard library's output being its contract, and every format promoted through JSON inherits it. A configuration converted between YAML and TOML keeps "30m" meaning thirty minutes.

Errors

Dispatch failures carry typed dotted-quad codes under range 1.2.0.*:

Per-codec failures carry the codec's own range (0.3.* for service codecs), which each per-format package names — cbor.CodeUnmarshalFailed, yaml.CodeAnchorRefused. Inspect with the accessors in github.com/kitsunium/sdk/pkg/v1/errs:

if errs.HasCode(err, codec.CodeUnknownFormat) { … }
if errs.HasReason(err, "UNKNOWN_FORMAT")       { … }

Package codec — range 1.2.0.* (ADR 0005 pkg/v1/data/codec block).

Package codec — compressed-frame verbs (ADR 0014 D1). MarshalCompressed and UnmarshalCompressed wrap the universal codec dispatch in a self-describing, version-tagged compression frame, so any value can be compressed in any registered Format without a parallel API. Compression is a parallel transform registry, never a codec Format (ADR 0014 §Why-not), so the algorithm is named separately from the wire Format.

Package codec — declares the sentinel *errs.Error values the facade emits when dispatch fails.

Package codec — the multipart/form-data value types, and the one helper a consumer needs to send what Marshal(Multipart, …) returns. The format is a container whose delimiter lives in the Content-Type header, which the Codec contract cannot carry, so the facade publishes the two shapes the codec speaks natively and the function that recovers that header from the body — the multipart package's own Form, Part and ContentType, under the names this package has always given them.

Package codec — JSON-bridge promotion path for codecs whose runtime preconditions reject the public Marshal(F, any) / Unmarshal(F, *, any) contract. Six of the twenty-four registered codecs constrain their input shape: csv expects [][]string or a slice of flat structs, ndjson expects []T, pem expects *pem.Block, flatbuffers expects []byte or BytesProvider, form expects url.Values, tlv's decoder cannot project composites into typed targets. Without promotion the facade's "format-swap is a single string change" promise is a lie for 6/24. Promotion intercepts the VALUE_INVALID / FLATBUFFERS_BAD_* / UNMARSHAL_FAILED responses, encodes the value to JSON, wraps the bytes in a codec-specific container the codec will accept, and reverses the pipeline on Unmarshal. The fast (native-shape) path is untouched so existing callers see zero overhead. See TestUniversalRoundtripAllCodecs for the contract pin.

Index

Constants

const ASCII85 Format = "ascii85"

ASCII85 denotes the JSON-mediated Adobe Ascii85 text-safe wrap.

const ASN1DER Format = "asn1-der"

ASN1DER denotes the stdlib encoding/asn1 DER wire format.

const BSON Format = "bson"

BSON denotes the MongoDB binary document format (top-level must be a document — struct or map — not a scalar). ADR 0021.

const Base16 Format = "base16"

Base16 denotes the JSON-mediated base16 (uppercase hex) wrap.

const Base32 Format = "base32"

Base32 denotes the JSON-mediated base32 (std) text-safe wrap.

const Base45 Format = "base45"

Base45 denotes the JSON-mediated RFC 9285 Base45 wrap (QR-code safe).

const Base58 Format = "base58"

Base58 denotes the JSON-mediated Bitcoin Base58 wrap (short inputs; base-conversion is O(n²) so a 4 KiB input cap applies).

const Base62 Format = "base62"

Base62 denotes the JSON-mediated Base62 wrap (short inputs; same O(n²) 4 KiB input cap as Base58).

const Base64 Format = "base64"

Base64 denotes the JSON-mediated base64 (std) text-safe wrap.

const Base64URL Format = "base64url"

Base64URL denotes the JSON-mediated base64 (URL-safe alphabet) wrap.

const CBOR Format = "cbor"

CBOR denotes the RFC 8949 wire format, encoded and decoded on the standard library alone.

const CSV Format = "csv"

CSV denotes the stdlib encoding/csv wire format.

const CodeCodecUnavailable v1errs.Code = 16908290

CodeCodecUnavailable fires when a codec entry exists but is unusable (e.g., blank import missing or build tag not active).

Code 1.2.0.2

const CodePromoteFailed v1errs.Code = 16908292

CodePromoteFailed fires when the facade's JSON-bridge promotion path cannot serve the request (unknown Format with no promotion strategy, a malformed container after the codec populated it, or encoding/json failing on the bridge itself).

Code 1.2.0.4

const CodeStreamingUnsupported v1errs.Code = 16908291

CodeStreamingUnsupported fires when NewEncoder / NewDecoder is called on a codec that does not implement StreamingCodec, or NewBoundedDecoder on one that does not implement BoundedStreamingCodec.

Code 1.2.0.3

const CodeUnknownFormat v1errs.Code = 16908289

CodeUnknownFormat fires when a Format is not registered in the registry.

Code 1.2.0.1

const FlatBuffers Format = "flatbuffers"

FlatBuffers denotes the passthrough codec for already-encoded FlatBuffer payloads — schema-typed reads stay in the caller's flatc-generated accessors.

const Flate CompressAlgorithm = "flate"

Flate selects the raw-DEFLATE compressor (frame algID 0x02).

const Form Format = "form"

Form denotes application/x-www-form-urlencoded, the encoding every HTML form POSTs. Its native Go shape is url.Values: a repeated key carries multiple values, which is the only array syntax the format has.

const Gzip CompressAlgorithm = "gzip"

Gzip selects the gzip compressor (frame algID 0x01).

const Hex Format = "hex"

Hex denotes the JSON-mediated lowercase-hex text-safe wrap.

const JSON Format = "json"

JSON denotes the stdlib encoding/json wire format.

const MsgPack Format = "msgpack"

MsgPack denotes the MessagePack wire format (msgpack.org specification), implemented natively on the standard library.

const Multipart Format = "multipart"

Multipart denotes RFC 7578 multipart/form-data. Its native Go shape is a MultipartForm of MultipartPart sections; any other value travels as a single JSON-mediated part. The RFC 2046 boundary lives in the Content-Type header, which the Codec contract cannot carry — the codec re-emits it in the body, and MultipartContentType recovers the header value from the bytes Marshal returned.

const NDJSON Format = "ndjson"

NDJSON denotes the newline-delimited JSON dialect.

const PEM Format = "pem"

PEM denotes the stdlib encoding/pem block format.

const TLV Format = "tlv"

TLV denotes the self-describing Type-Length-Value reflection codec (one record per encode; nested composites supported).

const TOML Format = "toml"

TOML denotes TOML v1.0.0, read and written by the SDK's stdlib-only codec.

const XML Format = "xml"

XML denotes the stdlib encoding/xml wire format.

const YAML Format = "yaml"

YAML denotes YAML as the SDK reads and writes it: a named subset of YAML 1.2.2 on the standard library alone, which refuses anchors, tags, merge keys and the other constructs it leaves out by name. The full reader is the opt-in "yaml-full" Format of third-party/codec/yaml.

Variables

var CodecUnavailable *errs.Error

CodecUnavailable is returned when the registry entry exists but the codec itself cannot be used (reserved for future gating).

Code 1.2.0.2 · reason CODEC_UNAVAILABLE · public "codec is registered but unavailable"

var PromoteFailed *errs.Error

PromoteFailed is returned when the facade's JSON-bridge promotion cannot serve the request — the Format has no promotion strategy registered, the codec populated a container whose shape no longer matches what the wrap stage produced (an NDJSON stream of more than one record included), or encoding/json refused the value or the target on the bridge, its error kept as the cause.

Code 1.2.0.4 · reason PROMOTE_FAILED · public "codec promotion failed"

var StreamingUnsupported *errs.Error

StreamingUnsupported is returned when NewEncoder / NewDecoder is called on a codec that does not implement StreamingCodec.

Code 1.2.0.3 · reason STREAMING_UNSUPPORTED · public "codec does not support streaming"

var UnknownFormat *errs.Error

UnknownFormat is returned when Marshal / Unmarshal / NewEncoder / NewDecoder receives a Format that no codec has registered.

Code 1.2.0.1 · reason UNKNOWN_FORMAT · public "codec format is not registered"

func Available

func Available() []Format

Available returns the sorted list of registered formats.

func Detect

func Detect(prefix []byte) (f Format, ok bool)

Detect returns the Format the first bytes of an input are written in, without a MIME type or a file extension (#909): JSON, NDJSON (two records or more), XML, PEM or BSON, recognised by their head. It reads at most 512 bytes of prefix, as http.DetectContentType does, answers (Format(""), false) when no format — or more than one — claims them, and allocates nothing. A frame MarshalCompressed wrote is no Format — UnmarshalCompressed reads it — so Detect never claims one; compressed bytes are DetectCompression's.

func DetectCompression

func DetectCompression(prefix []byte) (algo CompressAlgorithm, ok bool)

DetectCompression returns the compression the first bytes of an input open with, by its magic number (#909): "gzip" (1f 8b 08), "zlib" (an RFC 1950 header whose check bits hold), "zstd" (28 b5 2f fd) or "s2" (the framed stream identifier of S2 or Snappy). It answers ("", false) for anything else — raw DEFLATE ("flate") and S2's block form have no magic, and a frame MarshalCompressed wrote is UnmarshalCompressed's — and allocates nothing. zstd and s2 are named, not decompressed, here: they live in third-party/transform (ADR 0066). Decompress, then Detect what is inside.

func FromExtension

func FromExtension(ext string) (f Format, ok bool)

FromExtension resolves a file extension to its registered Format.

func FromMIME

func FromMIME(mime string) (f Format, ok bool)

FromMIME resolves a MIME string to its registered Format.

func IsMisuse

func IsMisuse(err error) bool

IsMisuse reports whether err is a codec refusing how it was called rather than a byte it read: EMPTY_INPUT (no bytes for a format with no empty document), TARGET_INVALID (a nil or non-pointer Unmarshal or Decode target), VALUE_INVALID (a channel, a function, an unsafe pointer to Marshal). The last two are the program's bug: answer them with 500.

func IsTooDeep

func IsTooDeep(err error) bool

IsTooDeep reports whether err is a codec's refusal of input nested past a depth bound.

func IsTooLarge

func IsTooLarge(err error) bool

IsTooLarge reports whether err is a codec's refusal of input past a size bound — SIZE_EXCEEDED in every bounded format, multipart's LIMIT_EXCEEDED, strictjson's DOCUMENT_TOO_LARGE: answer it with 413.

func Marshal

func Marshal(f Format, v any) (encoded []byte, err error)

Marshal serialises v using the codec registered under f. The codec's native input shape is tried first (fast path, zero overhead); if the codec rejects v as the wrong shape (csv requires [][]string or a slice of flat structs, pem requires *pem.Block, etc.) the facade promotes v via json-encode + codec-specific wrap so every Format accepts any Go value — see promote.go for the per-format strategies and the uniform-contract rationale. A value whose type alone rules it out — a struct given to Form, CSV, NDJSON or PEM — is promoted at once, without the codec building a refusal first; the bytes are the same.

func MarshalCompressed

func MarshalCompressed(f Format, algo CompressAlgorithm, v any) (box []byte, err error)

MarshalCompressed serialises v with the codec registered under f, compresses the result with the transform registered under algo, and wraps both in a self-describing frame so UnmarshalCompressed needs no Format or Algorithm argument. It returns UnknownCompressor when algo has no frame id or no registered body, CompressedFrameInvalid when f exceeds the addressable length, and forwards any codec / compressor error untouched (origin wins).

func MarshalMany

func MarshalMany(v any, formats ...Format) (encodedByFormat map[Format][]byte, err error)

MarshalMany serialises v into every format in formats and returns a map keyed by Format with the encoded bytes. The same logical value goes out in N different wire encodings — handy for HTTP content negotiation, multi-protocol message buses, archival doubling, or cross-region replication where each region speaks a different codec.

Per-format errors are joined under errors.Join and returned as a single error; partial results stay in the returned map so callers can decide whether to ship the formats that succeeded or fail the whole batch. An unknown Format in the list surfaces UnknownFormat / CodeUnknownFormat inside that joined error (and leaves no entry in the map for that Format).

Calling with formats == nil (or empty) returns an empty map and no error — the no-op semantics mirror calling Marshal zero times.

out, err := codec.MarshalMany(payload, codec.JSON, codec.CBOR, codec.MsgPack)
// out[codec.JSON], out[codec.CBOR], out[codec.MsgPack] all populated
// when err == nil.

func MultipartContentType

func MultipartContentType(body []byte) (value string, err error)

MultipartContentType returns the Content-Type header value — "multipart/form-data; boundary=…", quoted when the boundary needs it — for a body Marshal(Multipart, …) returned. Send it alongside the bytes: the boundary is announced in that header, and since the Codec contract has nowhere to hand one back, it is read off the body's first delimiter line. A body with no recoverable delimiter is refused with the codec's BOUNDARY_INVALID.

A caller streaming through NewEncoder(Multipart, w) must set the header before the first byte instead: the Encoder returned has a Boundary() string method, and mime.FormatMediaType builds the same value from it.

func NewBoundedDecoder

func NewBoundedDecoder(f Format, r io.Reader, maxBytes int64) (dec Decoder, err error)

NewBoundedDecoder returns a streaming decoder for the codec registered under f that refuses a value longer than maxBytes after reading at most maxBytes+1 bytes of it (#891): the decoder to put in front of a socket. The refusal is the codec's SIZE_EXCEEDED, which IsTooLarge reads; a maxBytes of zero or below is refused by the codec with BOUND_INVALID. JSON, XML and CBOR implement it; a Format whose codec does not returns StreamingUnsupported — TOML, MessagePack and YAML bound their NewDecoder with fixed ceilings of their own.

func NewDecoder

func NewDecoder(f Format, r io.Reader) (dec Decoder, err error)

NewDecoder returns a streaming decoder for the codec registered under f.

func NewEncoder

func NewEncoder(f Format, w io.Writer) (enc Encoder, err error)

NewEncoder returns a streaming encoder for the codec registered under f.

func PositionOf

func PositionOf(err error) (p Position, ok bool)

PositionOf returns where in its input a codec refused to decode — the line and column, or the byte offset, the format knows —, whatever the format: ok is false when the refusal carries no position.

func Unmarshal

func Unmarshal(f Format, data []byte, v any) error

Unmarshal parses data into v using the codec registered under f. As with Marshal, the codec's native target shape is tried first; on shape mismatch the facade promotes via JSON-bridge so every Format can decode into any Go target.

func UnmarshalCompressed

func UnmarshalCompressed(box []byte, v any) error

UnmarshalCompressed parses a frame produced by MarshalCompressed, decompresses the payload under a bomb guard, and decodes it into v using the inner Format the frame records. A malformed frame, an unknown algorithm id, or a payload that decompresses past the bomb guard returns CompressedFrameInvalid; codec and compressor faults are forwarded untouched.

type Codec

Codec re-exports core/data/codec.Codec for consumers who want direct access.

type Codec = corecodec.Codec

Codec is an alias of github.com/kitsunium/sdk/internal/core/data/codec.Codec, declared as:

type Codec interface {
	Extensions() []string
	MIMETypes() []string
	Marshal(v any) (data []byte, err error)
	Name() string
	Unmarshal(data []byte, v any) (err error)
}

func (Codec) Extensions
func (Codec) Extensions() []string

func (Codec) MIMETypes
func (Codec) MIMETypes() []string

func (Codec) Marshal
func (Codec) Marshal(v any) (data []byte, err error)

func (Codec) Name
func (Codec) Name() string

func (Codec) Unmarshal
func (Codec) Unmarshal(data []byte, v any) (err error)

type CompressAlgorithm

CompressAlgorithm is the transform package's Algorithm, so consumers name a compressor without importing internal/*. Use the Gzip / Flate constants.

type CompressAlgorithm = transform.Algorithm

CompressAlgorithm is an alias of github.com/kitsunium/sdk/internal/core/data/transform.Algorithm, declared as:

type CompressAlgorithm string

func (CompressAlgorithm) Known
func (CompressAlgorithm) Known() bool

Known reports whether a has been registered in the Compressor registry.

func (CompressAlgorithm) String
func (CompressAlgorithm) String() string

String implements fmt.Stringer and returns the raw identifier.

type Decoder

Decoder re-exports core/data/codec.Decoder for streaming callers.

type Decoder = corecodec.Decoder

Decoder is an alias of github.com/kitsunium/sdk/internal/core/data/codec.Decoder, declared as:

type Decoder interface {
	Decode(v any) (err error)
	More() (ok bool)
}

func (Decoder) Decode
func (Decoder) Decode(v any) (err error)

func (Decoder) More
func (Decoder) More() (ok bool)

type Encoder

Encoder re-exports core/data/codec.Encoder for streaming callers.

type Encoder = corecodec.Encoder

Encoder is an alias of github.com/kitsunium/sdk/internal/core/data/codec.Encoder, declared as:

type Encoder interface {
	Close() (err error)
	Encode(v any) (err error)
}

func (Encoder) Close
func (Encoder) Close() (err error)

func (Encoder) Encode
func (Encoder) Encode(v any) (err error)

type Format

Format re-exports core/data/codec.Format so consumers only depend on pkg/v1.

type Format = corecodec.Format

Format is an alias of github.com/kitsunium/sdk/internal/core/data/codec.Format, declared as:

type Format string

func (Format) Known
func (Format) Known() bool

Known reports whether f has been registered in the codec registry.

func (Format) String
func (Format) String() string

String implements fmt.Stringer and returns the raw identifier.

type MultipartForm

MultipartForm is the native Go shape of the Multipart format: a whole multipart/form-data body — its RFC 2046 boundary and its parts, in wire order. Marshal one to build an upload; Unmarshal into a *MultipartForm to read one. An empty Boundary asks Marshal to generate a delimiter; Unmarshal always fills it with the one it recovered, so a decode → encode round trip keeps the original delimiter — and reproduces the whole body byte for byte only for a body this codec encoded, since a part header other than the name, filename and media type is not carried through, and a filename is decoded to its last path element.

type MultipartForm = multipart.Form

MultipartForm is an alias of github.com/kitsunium/sdk/internal/service/data/codec/multipart.FormValue, declared as:

type MultipartForm struct {
	// Boundary is the RFC 2046 delimiter framing the parts.
	Boundary string
	// Parts are the body's sections, in wire order.
	Parts    []PartValue
}

type MultipartPart

MultipartPart is one section of a MultipartForm: a named field, optionally a filename and a media type, and the bytes themselves — a file upload is a part with FileName and ContentType set. Marshal also accepts a single MultipartPart or a []MultipartPart.

Name is required. Name, FileName and ContentType are written into the part's header block, so a CR, an LF or a NUL in any of them is refused — never escaped — with the codec's VALUE_INVALID naming the field. A UTF-8 filename is written as-is (RFC 7578 §4.2).

Decoding keeps only a filename's last path element, reading both '/' and '\' as separators on every operating system — RFC 7578 §4.2 has the receiver drop the directory information a sender may include, and the sender writes it with its own separator. `C:\Users\ada\cv.pdf` decodes to `cv.pdf` on Linux exactly as on Windows. Nothing else is removed, so a caller turning FileName into a path still validates it first.

type MultipartPart = multipart.Part

MultipartPart is an alias of github.com/kitsunium/sdk/internal/service/data/codec/multipart.PartValue, declared as:

type MultipartPart struct {
	// Name is the form field name (Content-Disposition name=). Required: Encode
	// refuses a part without one, and decoding refuses a body carrying one.
	Name        string
	// FileName is the optional filename= parameter; empty for a plain field.
	// UTF-8 is written as-is (RFC 7578 §4.2); only CR, LF and NUL are refused.
	//
	// Decoding keeps only its LAST PATH ELEMENT, and reads both '/' and '\' as
	// separators on every operating system: RFC 7578 §4.2 has the receiver drop
	// the directory information a sender may include, and the sender writes it
	// with its own separator. `C:\Users\ada\cv.pdf` and `dir/cv.pdf` both
	// decode to `cv.pdf`, wherever the decoder runs. Nothing else is removed —
	// "." and ".." come back as sent — so a caller turning it into a path still
	// validates it first.
	FileName    string
	// ContentType is the optional per-part media type; empty omits the header.
	ContentType string
	// Data is the part body.
	Data        []byte
}

type Position

Position places a decode refusal in its input: Offset the 0-based byte, Line and Column 1-based, a zero Line or Column unknown. yaml, toml and csv place by line and column, json by offset, xml by line, and the binary formats — cbor, msgpack, bson, tlv, flatbuffers — by offset alone, where an Offset of zero is the first byte. Read it with PositionOf.

type Position = corecodec.PositionValue

Position is an alias of github.com/kitsunium/sdk/internal/core/data/codec.PositionValue, declared as:

type Position struct {
	// Offset is the 0-based byte offset of the problem in the input.
	Offset int
	// Line is the 1-based line of the problem, zero when unknown.
	Line   int
	// Column is the 1-based column of the problem, zero when unknown.
	Column int
}

Generated from docs/api by tools/genindex.

Documentation

Overview

Package codec is the universal encoder/decoder dispatch facade.

One verb, twenty-four formats. Marshal and Unmarshal reach every encoding the SDK ships — JSON, YAML, CBOR, MessagePack, BSON, NDJSON, XML, TOML, CSV, urlencoded forms, multipart/form-data, ASN.1 DER, PEM, TLV, FlatBuffers, plus the nine base-N variants (base64, base64url, base32, base16, base45, base58, base62, hex, ascii85). Format-swap at runtime is a single string change.

Goals

  • One verb, many formats. codec.Marshal(f, v) reaches every registered wire format — your call site doesn't change when you swap JSON for CBOR.
  • Stable Format strings. Once registered, a Format string is frozen across minor versions. Code that compiles today keeps compiling.
  • Streaming when it pays. Codecs that implement StreamingCodec get NewEncoder / NewDecoder automatically — no need to buffer megabytes.
  • Zero registration code. Blank-import the package; the 16 codecs register themselves as Go initialises them. No Register() calls in consumer code. A program that needs one format imports that format's package beneath this one instead, and links it alone.
  • Append for hot paths. Codecs implementing Appender let you reuse a []byte buffer across calls — handy in tight loops.
  • Broadcast marshal. MarshalMany(v, formats...) serialises the same value to N formats at once — content negotiation, replication, multi-protocol message buses.

What's shipped — all 24 formats

Single blank import (`import _ "github.com/kitsunium/sdk/pkg/v1/data/codec"`) activates the entire list. The Streaming column marks codecs that implement core/data/codec.StreamingCodec and unlock NewEncoder / NewDecoder. Every codec implements core/data/codec.Appender.

| Format       | Go constant       | MIME                       | Extension           | Streaming | Best for |
|--------------|-------------------|----------------------------|---------------------|:---------:|----------|
| json         | codec.JSON        | application/json           | .json               | yes       | API responses, public configs |
| ndjson       | codec.NDJSON      | application/x-ndjson       | .ndjson / .jsonl    | yes       | streaming logs, line-oriented batches |
| xml          | codec.XML         | application/xml            | .xml                | yes       | legacy integrations |
| csv          | codec.CSV         | text/csv                   | .csv                | yes       | tabular exports |
| form         | codec.Form        | application/x-www-form-urlencoded | .form / .urlencoded | —  | HTML form bodies, query strings |
| asn1-der     | codec.ASN1DER     | application/pkix-cert      | .der / .cer         | —         | crypto / X.509 artefacts |
| pem          | codec.PEM         | application/x-pem-file     | .pem / .crt / .key  | yes       | block-wrapped DER (certs, keys) |
| yaml         | codec.YAML        | application/yaml           | .yaml / .yml        | yes       | human-edited configs |
| toml         | codec.TOML        | application/toml           | .toml               | yes       | app configs (strict typing) |
| cbor         | codec.CBOR        | application/cbor           | .cbor               | yes       | IoT / mobile (RFC 8949) |
| msgpack      | codec.MsgPack     | application/msgpack        | .msgpack / .mpk     | yes       | RPC payloads |
| bson         | codec.BSON        | application/bson           | .bson               | —         | MongoDB documents (top level must be a document) |
| multipart    | codec.Multipart   | multipart/form-data        | —                   | yes       | uploads / form posts (native shape codec.MultipartForm) |
| tlv          | codec.TLV         | application/x-tlv          | .tlv                | yes       | custom binary streams, self-describing |
| flatbuffers  | codec.FlatBuffers | application/x-flatbuffers  | .fbs / .bin         | —         | zero-copy passthrough |
| base64       | codec.Base64      | application/base64         | .b64 / .base64      | yes       | text-safe wrap (JSON → base-N) |
| base64url    | codec.Base64URL   | application/base64url      | .b64url             | yes       | URL-safe base64 |
| base32       | codec.Base32      | application/base32         | .b32                | yes       | larger alphabet for human-typed tokens |
| base16       | codec.Base16      | application/base16         | .b16                | yes       | hex-like text (uppercase) |
| base45       | codec.Base45      | application/base45         | .b45                | yes       | QR-code alphanumeric payloads (RFC 9285) |
| base58       | codec.Base58      | application/base58         | .b58                | yes       | Bitcoin-style short IDs (4 KiB input cap) |
| base62       | codec.Base62      | application/base62         | .b62                | yes       | alphanumeric short IDs (4 KiB input cap) |
| hex          | codec.Hex         | application/hex            | .hex                | yes       | classic hex pair encoding (lowercase) |
| ascii85      | codec.ASCII85     | application/ascii85        | .a85                | yes       | 4-byte to 5-char text packing |

Runtime introspection: codec.Available() returns the live registry list, codec.FromMIME and codec.FromExtension resolve from external metadata.

Quick start

package main

import (
    "fmt"

    "github.com/kitsunium/sdk/pkg/v1/data/codec"
)

type User struct {
    Name string `json:"name" cbor:"name" yaml:"name"`
    Age  int    `json:"age"  cbor:"age"  yaml:"age"`
}

func main() {
    u := User{Name: "Ada", Age: 36}
    for _, f := range []codec.Format{codec.JSON, codec.CBOR, codec.YAML, codec.MsgPack, codec.Base64} {
        data, _ := codec.Marshal(f, u)
        var back User
        _ = codec.Unmarshal(f, data, &back)
        fmt.Printf("%-10s %d bytes  → %#v\n", f, len(data), back)
    }
}

Activation

A single blank import activates the full registry:

import _ "github.com/kitsunium/sdk/pkg/v1/data/codec"

The package is the aggregate of the sixteen per-format packages beneath it — asn1, baseenc, bson, cbor, csv, flatbuffers, form, json, msgpack, multipart, ndjson, pem, tlv, toml, xml and yaml — and blank-imports each. Every one of them imports its codec, which registers itself in the registry as Go initialises it, once however many packages import it. Consumers never call a Register() function — registration is a side-effect. Importing one of those packages instead registers its format alone, links nothing of the others, and names that format's error codes (ADR 0134).

Extension interfaces

A registered codec MAY implement one or both of these optional interfaces (assert at the call site):

// Append into a caller-supplied buffer — drops the fresh-slice
// allocation Marshal returns (see BENCH.md for per-codec numbers;
// the remaining allocs are the encoder's, not the facade's).
type Appender interface {
    Append(dst []byte, v any) ([]byte, error)
}

// Streaming — open Encoder/Decoder around an io.Writer/io.Reader.
type StreamingCodec interface {
    NewEncoder(w io.Writer) Encoder
    NewDecoder(r io.Reader) Decoder
}

// Bounded streaming — a Decoder refusing any value past maxBytes
// (JSON, XML, CBOR; reached through NewBoundedDecoder).
type BoundedStreamingCodec interface {
    StreamingCodec
    NewBoundedDecoder(r io.Reader, maxBytes int64) (Decoder, error)
}

// Sniffing — a confident claim on an input's first bytes, which
// Detect asks of every codec (JSON, NDJSON, XML, PEM, BSON).
type Sniffer interface {
    Sniff(prefix []byte) (confident bool)
}

StreamingUnsupported is returned from NewEncoder / NewDecoder when the resolved codec does not satisfy StreamingCodec, and from NewBoundedDecoder when it does not satisfy BoundedStreamingCodec.

Durations

A time.Duration follows one rule per kind of format (#1126). The formats people edit by hand — YAML and TOML — write its String(), "1h30m0s", and read it with time.ParseDuration, refusing a string without its unit. The binary formats — CBOR, MessagePack, BSON — write an integer of int64 nanoseconds, no standard tag carrying a duration. JSON keeps the integer encoding/json writes, the standard library's output being its contract, and every format promoted through JSON inherits it. A configuration converted between YAML and TOML keeps "30m" meaning thirty minutes.

Errors

Dispatch failures carry typed dotted-quad codes under range 1.2.0.*:

Per-codec failures carry the codec's own range (0.3.* for service codecs), which each per-format package names — cbor.CodeUnmarshalFailed, yaml.CodeAnchorRefused. Inspect with the accessors in github.com/kitsunium/sdk/pkg/v1/errs:

if errs.HasCode(err, codec.CodeUnknownFormat) { … }
if errs.HasReason(err, "UNKNOWN_FORMAT")       { … }

Package codec — range 1.2.0.* (ADR 0005 pkg/v1/data/codec block).

Package codec — compressed-frame verbs (ADR 0014 D1). MarshalCompressed and UnmarshalCompressed wrap the universal codec dispatch in a self-describing, version-tagged compression frame, so any value can be compressed in any registered Format without a parallel API. Compression is a parallel transform registry, never a codec Format (ADR 0014 §Why-not), so the algorithm is named separately from the wire Format.

Package codec — declares the sentinel *errs.Error values the facade emits when dispatch fails.

Package codec — the multipart/form-data value types, and the one helper a consumer needs to send what Marshal(Multipart, …) returns. The format is a container whose delimiter lives in the Content-Type header, which the Codec contract cannot carry, so the facade publishes the two shapes the codec speaks natively and the function that recovers that header from the body — the multipart package's own Form, Part and ContentType, under the names this package has always given them.

Package codec — JSON-bridge promotion path for codecs whose runtime preconditions reject the public Marshal(F, any) / Unmarshal(F, *, any) contract. Six of the twenty-four registered codecs constrain their input shape: csv expects [][]string or a slice of flat structs, ndjson expects []T, pem expects *pem.Block, flatbuffers expects []byte or BytesProvider, form expects url.Values, tlv's decoder cannot project composites into typed targets. Without promotion the facade's "format-swap is a single string change" promise is a lie for 6/24. Promotion intercepts the VALUE_INVALID / FLATBUFFERS_BAD_* / UNMARSHAL_FAILED responses, encodes the value to JSON, wraps the bytes in a codec-specific container the codec will accept, and reverses the pipeline on Unmarshal. The fast (native-shape) path is untouched so existing callers see zero overhead. See TestUniversalRoundtripAllCodecs for the contract pin.

Index

Constants

View Source
const CodeCodecUnavailable v1errs.Code = 0x01_02_00_02 // 1.2.0.2

CodeCodecUnavailable fires when a codec entry exists but is unusable (e.g., blank import missing or build tag not active).

View Source
const CodePromoteFailed v1errs.Code = 0x01_02_00_04 // 1.2.0.4

CodePromoteFailed fires when the facade's JSON-bridge promotion path cannot serve the request (unknown Format with no promotion strategy, a malformed container after the codec populated it, or encoding/json failing on the bridge itself).

View Source
const CodeStreamingUnsupported v1errs.Code = 0x01_02_00_03 // 1.2.0.3

CodeStreamingUnsupported fires when NewEncoder / NewDecoder is called on a codec that does not implement StreamingCodec, or NewBoundedDecoder on one that does not implement BoundedStreamingCodec.

View Source
const CodeUnknownFormat v1errs.Code = 0x01_02_00_01 // 1.2.0.1

CodeUnknownFormat fires when a Format is not registered in the registry.

Variables

View Source
var (
	// UnknownFormat is returned when Marshal / Unmarshal / NewEncoder /
	// NewDecoder receives a Format that no codec has registered.
	UnknownFormat = errs.Static(CodeUnknownFormat, "UNKNOWN_FORMAT",
		"codec format is not registered",
		"pkg/v1/data/codec: Lookup returned false for the requested Format",
		0,
		0)

	// CodecUnavailable is returned when the registry entry exists but the
	// codec itself cannot be used (reserved for future gating).
	CodecUnavailable = errs.Static(CodeCodecUnavailable, "CODEC_UNAVAILABLE",
		"codec is registered but unavailable",
		"pkg/v1/data/codec: registered codec cannot serve the request",
		0,
		0)

	// StreamingUnsupported is returned when NewEncoder / NewDecoder is
	// called on a codec that does not implement StreamingCodec.
	StreamingUnsupported = errs.Static(CodeStreamingUnsupported, "STREAMING_UNSUPPORTED",
		"codec does not support streaming",
		"pkg/v1/data/codec: target codec is not a core/data/codec.StreamingCodec",
		0,
		0)

	// PromoteFailed is returned when the facade's JSON-bridge promotion
	// cannot serve the request — the Format has no promotion strategy
	// registered, the codec populated a container whose shape no longer
	// matches what the wrap stage produced (an NDJSON stream of more than
	// one record included), or encoding/json refused the value or the
	// target on the bridge, its error kept as the cause.
	PromoteFailed = errs.Static(CodePromoteFailed, "PROMOTE_FAILED",
		"codec promotion failed",
		"pkg/v1/data/codec: JSON-bridge promotion path could not serve the request",
		0,
		0)
)

Functions

func IsMisuse added in v0.12.0

func IsMisuse(err error) bool

IsMisuse reports whether err is a codec refusing how it was called rather than a byte it read: EMPTY_INPUT (no bytes for a format with no empty document), TARGET_INVALID (a nil or non-pointer Unmarshal or Decode target), VALUE_INVALID (a channel, a function, an unsafe pointer to Marshal). The last two are the program's bug: answer them with 500.

func IsTooDeep added in v0.12.0

func IsTooDeep(err error) bool

IsTooDeep reports whether err is a codec's refusal of input nested past a depth bound.

func IsTooLarge added in v0.12.0

func IsTooLarge(err error) bool

IsTooLarge reports whether err is a codec's refusal of input past a size bound — SIZE_EXCEEDED in every bounded format, multipart's LIMIT_EXCEEDED, strictjson's DOCUMENT_TOO_LARGE: answer it with 413.

func Marshal

func Marshal(f Format, v any) (encoded []byte, err error)

Marshal serialises v using the codec registered under f. The codec's native input shape is tried first (fast path, zero overhead); if the codec rejects v as the wrong shape (csv requires [][]string or a slice of flat structs, pem requires *pem.Block, etc.) the facade promotes v via json-encode + codec-specific wrap so every Format accepts any Go value — see promote.go for the per-format strategies and the uniform-contract rationale. A value whose type alone rules it out — a struct given to Form, CSV, NDJSON or PEM — is promoted at once, without the codec building a refusal first; the bytes are the same.

func MarshalCompressed

func MarshalCompressed(f Format, algo CompressAlgorithm, v any) (box []byte, err error)

MarshalCompressed serialises v with the codec registered under f, compresses the result with the transform registered under algo, and wraps both in a self-describing frame so UnmarshalCompressed needs no Format or Algorithm argument. It returns UnknownCompressor when algo has no frame id or no registered body, CompressedFrameInvalid when f exceeds the addressable length, and forwards any codec / compressor error untouched (origin wins).

func MarshalMany

func MarshalMany(v any, formats ...Format) (encodedByFormat map[Format][]byte, err error)

MarshalMany serialises v into every format in formats and returns a map keyed by Format with the encoded bytes. The same logical value goes out in N different wire encodings — handy for HTTP content negotiation, multi-protocol message buses, archival doubling, or cross-region replication where each region speaks a different codec.

Per-format errors are joined under errors.Join and returned as a single error; partial results stay in the returned map so callers can decide whether to ship the formats that succeeded or fail the whole batch. An unknown Format in the list surfaces UnknownFormat / CodeUnknownFormat inside that joined error (and leaves no entry in the map for that Format).

Calling with formats == nil (or empty) returns an empty map and no error — the no-op semantics mirror calling Marshal zero times.

out, err := codec.MarshalMany(payload, codec.JSON, codec.CBOR, codec.MsgPack)
// out[codec.JSON], out[codec.CBOR], out[codec.MsgPack] all populated
// when err == nil.

func MultipartContentType

func MultipartContentType(body []byte) (value string, err error)

MultipartContentType returns the Content-Type header value — "multipart/form-data; boundary=…", quoted when the boundary needs it — for a body Marshal(Multipart, …) returned. Send it alongside the bytes: the boundary is announced in that header, and since the Codec contract has nowhere to hand one back, it is read off the body's first delimiter line. A body with no recoverable delimiter is refused with the codec's BOUNDARY_INVALID.

A caller streaming through NewEncoder(Multipart, w) must set the header before the first byte instead: the Encoder returned has a Boundary() string method, and mime.FormatMediaType builds the same value from it.

func Unmarshal

func Unmarshal(f Format, data []byte, v any) error

Unmarshal parses data into v using the codec registered under f. As with Marshal, the codec's native target shape is tried first; on shape mismatch the facade promotes via JSON-bridge so every Format can decode into any Go target.

func UnmarshalCompressed

func UnmarshalCompressed(box []byte, v any) error

UnmarshalCompressed parses a frame produced by MarshalCompressed, decompresses the payload under a bomb guard, and decodes it into v using the inner Format the frame records. A malformed frame, an unknown algorithm id, or a payload that decompresses past the bomb guard returns CompressedFrameInvalid; codec and compressor faults are forwarded untouched.

Types

type Codec

type Codec = corecodec.Codec

Codec re-exports core/data/codec.Codec for consumers who want direct access.

type CompressAlgorithm

type CompressAlgorithm = transform.Algorithm

CompressAlgorithm is the transform package's Algorithm, so consumers name a compressor without importing internal/*. Use the Gzip / Flate constants.

const (
	// Gzip selects the gzip compressor (frame algID 0x01).
	Gzip CompressAlgorithm = "gzip"
	// Flate selects the raw-DEFLATE compressor (frame algID 0x02).
	Flate CompressAlgorithm = "flate"
)

func DetectCompression added in v0.12.0

func DetectCompression(prefix []byte) (algo CompressAlgorithm, ok bool)

DetectCompression returns the compression the first bytes of an input open with, by its magic number (#909): "gzip" (1f 8b 08), "zlib" (an RFC 1950 header whose check bits hold), "zstd" (28 b5 2f fd) or "s2" (the framed stream identifier of S2 or Snappy). It answers ("", false) for anything else — raw DEFLATE ("flate") and S2's block form have no magic, and a frame MarshalCompressed wrote is UnmarshalCompressed's — and allocates nothing. zstd and s2 are named, not decompressed, here: they live in third-party/transform (ADR 0066). Decompress, then Detect what is inside.

type Decoder

type Decoder = corecodec.Decoder

Decoder re-exports core/data/codec.Decoder for streaming callers.

func NewBoundedDecoder added in v0.12.0

func NewBoundedDecoder(f Format, r io.Reader, maxBytes int64) (dec Decoder, err error)

NewBoundedDecoder returns a streaming decoder for the codec registered under f that refuses a value longer than maxBytes after reading at most maxBytes+1 bytes of it (#891): the decoder to put in front of a socket. The refusal is the codec's SIZE_EXCEEDED, which IsTooLarge reads; a maxBytes of zero or below is refused by the codec with BOUND_INVALID. JSON, XML and CBOR implement it; a Format whose codec does not returns StreamingUnsupported — TOML, MessagePack and YAML bound their NewDecoder with fixed ceilings of their own.

func NewDecoder

func NewDecoder(f Format, r io.Reader) (dec Decoder, err error)

NewDecoder returns a streaming decoder for the codec registered under f.

type Encoder

type Encoder = corecodec.Encoder

Encoder re-exports core/data/codec.Encoder for streaming callers.

func NewEncoder

func NewEncoder(f Format, w io.Writer) (enc Encoder, err error)

NewEncoder returns a streaming encoder for the codec registered under f.

type Format

type Format = corecodec.Format

Format re-exports core/data/codec.Format so consumers only depend on pkg/v1.

const (
	// JSON denotes the stdlib encoding/json wire format.
	JSON Format = "json"
	// NDJSON denotes the newline-delimited JSON dialect.
	NDJSON Format = "ndjson"
	// XML denotes the stdlib encoding/xml wire format.
	XML Format = "xml"
	// CSV denotes the stdlib encoding/csv wire format.
	CSV Format = "csv"
	// Form denotes application/x-www-form-urlencoded, the encoding every
	// HTML form POSTs. Its native Go shape is url.Values: a repeated key
	// carries multiple values, which is the only array syntax the format
	// has.
	Form Format = "form"
	// ASN1DER denotes the stdlib encoding/asn1 DER wire format.
	ASN1DER Format = "asn1-der"
	// PEM denotes the stdlib encoding/pem block format.
	PEM Format = "pem"
	// YAML denotes YAML as the SDK reads and writes it: a named subset of
	// YAML 1.2.2 on the standard library alone, which refuses anchors, tags,
	// merge keys and the other constructs it leaves out by name. The full
	// reader is the opt-in "yaml-full" Format of third-party/codec/yaml.
	YAML Format = "yaml"
	// TOML denotes TOML v1.0.0, read and written by the SDK's stdlib-only codec.
	TOML Format = "toml"
	// CBOR denotes the RFC 8949 wire format, encoded and decoded on the
	// standard library alone.
	CBOR Format = "cbor"
	// MsgPack denotes the MessagePack wire format (msgpack.org specification),
	// implemented natively on the standard library.
	MsgPack Format = "msgpack"

	// TLV denotes the self-describing Type-Length-Value reflection
	// codec (one record per encode; nested composites supported).
	TLV Format = "tlv"
	// FlatBuffers denotes the passthrough codec for already-encoded
	// FlatBuffer payloads — schema-typed reads stay in the caller's
	// flatc-generated accessors.
	FlatBuffers Format = "flatbuffers"

	// Base64 denotes the JSON-mediated base64 (std) text-safe wrap.
	Base64 Format = "base64"
	// Base64URL denotes the JSON-mediated base64 (URL-safe alphabet) wrap.
	Base64URL Format = "base64url"
	// Base32 denotes the JSON-mediated base32 (std) text-safe wrap.
	Base32 Format = "base32"
	// Base16 denotes the JSON-mediated base16 (uppercase hex) wrap.
	Base16 Format = "base16"
	// Hex denotes the JSON-mediated lowercase-hex text-safe wrap.
	Hex Format = "hex"
	// ASCII85 denotes the JSON-mediated Adobe Ascii85 text-safe wrap.
	ASCII85 Format = "ascii85"
	// Base45 denotes the JSON-mediated RFC 9285 Base45 wrap (QR-code safe).
	Base45 Format = "base45"
	// Base58 denotes the JSON-mediated Bitcoin Base58 wrap (short inputs;
	// base-conversion is O(n²) so a 4 KiB input cap applies).
	Base58 Format = "base58"
	// Base62 denotes the JSON-mediated Base62 wrap (short inputs; same O(n²)
	// 4 KiB input cap as Base58).
	Base62 Format = "base62"
	// BSON denotes the MongoDB binary document format (top-level must be a
	// document — struct or map — not a scalar). ADR 0021.
	BSON Format = "bson"
	// Multipart denotes RFC 7578 multipart/form-data. Its native Go shape is
	// a [MultipartForm] of [MultipartPart] sections; any other value travels
	// as a single JSON-mediated part. The RFC 2046 boundary lives in the
	// Content-Type header, which the Codec contract cannot carry — the codec
	// re-emits it in the body, and [MultipartContentType] recovers the header
	// value from the bytes Marshal returned.
	Multipart Format = "multipart"
)

Known format constants — string values are part of the public contract. Prefer the typed constants over string literals at call sites: the IDE catches typos at compile time, autocomplete surfaces the full list, and the underlying string value (frozen post-v1.0.0) stays available via `string(codec.JSON)` whenever raw access is needed.

func Available

func Available() []Format

Available returns the sorted list of registered formats.

func Detect added in v0.12.0

func Detect(prefix []byte) (f Format, ok bool)

Detect returns the Format the first bytes of an input are written in, without a MIME type or a file extension (#909): JSON, NDJSON (two records or more), XML, PEM or BSON, recognised by their head. It reads at most 512 bytes of prefix, as http.DetectContentType does, answers (Format(""), false) when no format — or more than one — claims them, and allocates nothing. A frame MarshalCompressed wrote is no Format — UnmarshalCompressed reads it — so Detect never claims one; compressed bytes are DetectCompression's.

func FromExtension

func FromExtension(ext string) (f Format, ok bool)

FromExtension resolves a file extension to its registered Format.

func FromMIME

func FromMIME(mime string) (f Format, ok bool)

FromMIME resolves a MIME string to its registered Format.

type MultipartForm

type MultipartForm = multipart.Form

MultipartForm is the native Go shape of the Multipart format: a whole multipart/form-data body — its RFC 2046 boundary and its parts, in wire order. Marshal one to build an upload; Unmarshal into a *MultipartForm to read one. An empty Boundary asks Marshal to generate a delimiter; Unmarshal always fills it with the one it recovered, so a decode → encode round trip keeps the original delimiter — and reproduces the whole body byte for byte only for a body this codec encoded, since a part header other than the name, filename and media type is not carried through, and a filename is decoded to its last path element.

type MultipartPart

type MultipartPart = multipart.Part

MultipartPart is one section of a MultipartForm: a named field, optionally a filename and a media type, and the bytes themselves — a file upload is a part with FileName and ContentType set. Marshal also accepts a single MultipartPart or a []MultipartPart.

Name is required. Name, FileName and ContentType are written into the part's header block, so a CR, an LF or a NUL in any of them is refused — never escaped — with the codec's VALUE_INVALID naming the field. A UTF-8 filename is written as-is (RFC 7578 §4.2).

Decoding keeps only a filename's last path element, reading both '/' and '\' as separators on every operating system — RFC 7578 §4.2 has the receiver drop the directory information a sender may include, and the sender writes it with its own separator. `C:\Users\ada\cv.pdf` decodes to `cv.pdf` on Linux exactly as on Windows. Nothing else is removed, so a caller turning FileName into a path still validates it first.

type Position added in v0.12.0

type Position = corecodec.PositionValue

Position places a decode refusal in its input: Offset the 0-based byte, Line and Column 1-based, a zero Line or Column unknown. yaml, toml and csv place by line and column, json by offset, xml by line, and the binary formats — cbor, msgpack, bson, tlv, flatbuffers — by offset alone, where an Offset of zero is the first byte. Read it with PositionOf.

func PositionOf added in v0.12.0

func PositionOf(err error) (p Position, ok bool)

PositionOf returns where in its input a codec refused to decode — the line and column, or the byte offset, the format knows —, whatever the format: ok is false when the refusal carries no position.

Directories

Path Synopsis
Package asn1 registers the ASN.1 DER codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package asn1 registers the ASN.1 DER codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package baseenc registers the base-N codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package baseenc registers the base-N codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package bson is BSON, MongoDB's binary document format, implemented with the standard library alone.
Package bson is BSON, MongoDB's binary document format, implemented with the standard library alone.
Package cbor registers the CBOR codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package cbor registers the CBOR codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package csv registers the CSV codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package csv registers the CSV codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package flatbuffers registers the FlatBuffers codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package flatbuffers registers the FlatBuffers codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package form registers the application/x-www-form-urlencoded codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package form registers the application/x-www-form-urlencoded codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package jcs writes the JSON Canonicalization Scheme of RFC 8785: one byte form for a JSON document — members sorted, no whitespace, numbers and strings spelled one way — so it can be signed or hashed, and checked by anyone who canonicalizes the same document.
Package jcs writes the JSON Canonicalization Scheme of RFC 8785: one byte form for a JSON document — members sorted, no whitespace, numbers and strings spelled one way — so it can be signed or hashed, and checked by anyone who canonicalizes the same document.
Package json registers the JSON codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package json registers the JSON codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package jsonpatch computes the structural difference between two JSON documents as RFC 6902 operations — add, remove and replace, each at an RFC 6901 JSON Pointer — with the value each operation writes and the value it replaces or removes (ADR 0143).
Package jsonpatch computes the structural difference between two JSON documents as RFC 6902 operations — add, remove and replace, each at an RFC 6901 JSON Pointer — with the value each operation writes and the value it replaces or removes (ADR 0143).
Package jsonschema validates JSON documents somebody else wrote against a JSON Schema 2020-12 they publish — a webhook's payload, an OpenAPI request body, another program's configuration — with the standard library alone.
Package jsonschema validates JSON documents somebody else wrote against a JSON Schema 2020-12 they publish — a webhook's payload, an OpenAPI request body, another program's configuration — with the standard library alone.
Package jsonshape describes how values of a Go type look on the wire under encoding/json — the JSON kind each value takes, the members an object has and under which names, which may be missing and which may be null — for a reader to show: an API's documentation, a developer console, a schema generator (ADR 0133).
Package jsonshape describes how values of a Go type look on the wire under encoding/json — the JSON kind each value takes, the members an object has and under which names, which may be missing and which may be null — for a reader to show: an API's documentation, a developer console, a schema generator (ADR 0133).
Package msgpack registers the MessagePack codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package msgpack registers the MessagePack codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package multipart registers the multipart/form-data codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package multipart registers the multipart/form-data codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package ndjson registers the NDJSON codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package ndjson registers the NDJSON codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package pem registers the PEM codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package pem registers the PEM codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package strictjson decodes exactly one JSON document into a Go value, within a byte bound, refusing what a permissive decoder lets through — and no refusal ever repeats a byte of the document.
Package strictjson decodes exactly one JSON document into a Go value, within a byte bound, refusing what a permissive decoder lets through — and no refusal ever repeats a byte of the document.
httpbody
Package httpbody decodes the JSON body of an HTTP request with the strict decoder of github.com/kitsunium/sdk/pkg/v1/data/codec/strictjson, adding what only an HTTP body has.
Package httpbody decodes the JSON body of an HTTP request with the strict decoder of github.com/kitsunium/sdk/pkg/v1/data/codec/strictjson, adding what only an HTTP body has.
Package tlv registers the TLV codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package tlv registers the TLV codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package toml registers the TOML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package toml registers the TOML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package xml registers the XML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package xml registers the XML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package yaml registers the YAML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):
Package yaml registers the YAML codec with the SDK's codec registry — and no other codec — when it is imported (ADR 0134):

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL