Documentation
¶
Overview ¶
Package format bridges Codec[T] to concrete serialization formats.
A [Codec][T] works with an intermediate representation (map[string]any) that is format-agnostic. Format wraps that intermediate layer so the same codec can read and write multiple wire formats without any changes to the codec itself.
File I/O has moved ¶
The declarative typed file descriptor (File[T], NewFile, Read/Write/Update/ Patch, PatchEncoded, FilePathParam, and the file error types) now lives in the ports package — see [ports.File] — since it is a protocol-agnostic addressing descriptor bound via [ports.FilePattern] to adapters/file, the same role [ports.Cache] plays for adapters/redis. This package still provides Format.IsPatchable, Format.PatchInto, Format.Codec, Format.UnmarshalRaw, Format.MarshalRaw, and DeepMerge — the lower-level primitives [ports.File.Patch]/[ports.PatchEncoded] are built on.
Embedded format codecs — JSON/YAML/TOML within a string field ¶
Some APIs and protocols store a serialised document inside a string field (CloudEvents data-as-string, database JSONB via REST, Kafka message headers, device-twin configuration). EmbeddedJSON, EmbeddedYAML, and EmbeddedTOML return a codex.Codec[T] that treats the wire string as a nested document:
// Wire: {"event":"user.created","payload":"{\"id\":\"123\",\"name\":\"Alice\"}"}
var eventCodec = codex.Struct[Event](
codex.RequiredField("event", codex.String(), ...),
codex.RequiredField("payload", format.EmbeddedJSON(userCodec), ...),
)
// Compose with ports.File[T] — the outer format handles the file bytes;
// EmbeddedJSON handles the string-to-struct field conversion.
var eventFile = ports.NewFile("events/user.json", format.JSON(eventCodec))
event, err := eventFile.Read(nil, ports.FileOptions{})
Decode: wire string → format unmarshal → inner.Decode → T Encode: inner.Encode → format marshal → wire string
Format parse failures return EmbeddedDecodeError{Format, Err}; marshal failures return EmbeddedEncodeError{Format, Err}. Both implement slog.LogValuer. Inner codec validation errors propagate unchanged.
Built-in formats ¶
Text-based formats (JSON, YAML, TOML) pass through the map[string]any intermediate:
format.JSON(codec) // application/json format.YAML(codec) // application/yaml format.TOML(codec) // application/toml
Binary formats ¶
Binary formats bypass the intermediate and operate on the typed value directly:
Gob — uses encoding/gob framing. Suitable for Go-to-Go communication and binary caching. NOT suitable for writing files that must be readable by other tools (image viewers, PDF readers), because Gob adds its own framing bytes.
Binary — writes and reads []byte as-is, without any encoding. Suitable for raw binary file I/O (PNG, JPEG, PDF, WAV…) and HTTP binary bodies. Unlike Gob, Binary produces files that any tool understanding the underlying format can open.
Choosing between Binary and Gob ¶
- Use Binary when the file must be byte-identical to the original format (PNG, PDF…).
- Use Gob for internal Go-to-Go serialization where framing overhead is acceptable.
Custom formats ¶
For formats not covered by the built-ins, use:
- New — map-based intermediate (CBOR, MessagePack, XML, …)
- NewTyped — typed T directly, without map[string]any (templ HTML, Protobuf, image.Image)
- NewStreamed — streams to io.Writer without buffering (SSR streaming, chunked exports)
Binary is convenience sugar over NewTyped[[]byte] with identity marshal/unmarshal. Use NewTyped directly when T ≠ []byte or when marshal/unmarshal perform real encoding.
Index ¶
- Variables
- func DeepMerge(dst, src map[string]any)
- func EmbeddedJSON[T any](inner codex.Codec[T]) codex.Codec[T]
- func EmbeddedTOML[T any](inner codex.Codec[T]) codex.Codec[T]
- func EmbeddedYAML[T any](inner codex.Codec[T]) codex.Codec[T]
- type EmbeddedDecodeError
- type EmbeddedEncodeError
- type Format
- func Binary(c codex.Codec[[]byte]) Format[[]byte]
- func Gob[T any](c codex.Codec[T]) Format[T]
- func JSON[T any](c codex.Codec[T]) Format[T]
- func New[T any](c codex.Codec[T], marshal func(any) ([]byte, error), ...) Format[T]
- func NewStreamed[T any](c codex.Codec[T], marshalTo func(T, io.Writer) error, ...) Format[T]
- func NewTyped[T any](c codex.Codec[T], marshal func(T) ([]byte, error), ...) Format[T]
- func TOML[T any](c codex.Codec[T]) Format[T]
- func YAML[T any](c codex.Codec[T]) Format[T]
- func (f Format[T]) Codec() codex.Codec[T]
- func (f Format[T]) ContentType() string
- func (f Format[T]) IsPatchable() bool
- func (f Format[T]) IsStreamable() bool
- func (f Format[T]) Marshal(v T) ([]byte, error)
- func (f Format[T]) MarshalRaw(v any) ([]byte, error)
- func (f Format[T]) MarshalTo(v T, w io.Writer) error
- func (f Format[T]) PatchInto(existing []byte, patch map[string]any) (T, error)
- func (f Format[T]) Schema() schema.Schema
- func (f Format[T]) Unmarshal(data []byte) (T, error)
- func (f Format[T]) UnmarshalRaw(data []byte) (any, error)
- func (f Format[T]) Validate(v T) error
- func (f Format[T]) WithContentType(ct string) Format[T]
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNotStreamable = errors.New("format: not streamable — use NewStreamed to create a streaming format")
ErrNotStreamable is returned by Format.MarshalTo when the format was not created with NewStreamed and therefore does not support streaming output.
Functions ¶
func DeepMerge ¶ added in v0.12.0
DeepMerge applies src over dst in place. Nested maps at the same key are recursively merged. Scalar and array values in src overwrite those in dst. Exposed for callers that need to merge two map[string]any intermediates themselves (see [ports.PatchEncoded]); Format.PatchInto uses it internally.
func EmbeddedJSON ¶ added in v0.11.0
EmbeddedJSON returns a codex.Codec[T] where the wire type is a JSON-encoded string. The inner codec's constraints and schema run on the decoded value.
Decode path: JSON string → json.Unmarshal → map[string]any → inner.Decode → T Encode path: T → inner.Encode → map[string]any → json.Marshal → JSON string
This pattern is common in systems where a structured value is serialised into a string field: CloudEvents data-as-string, database JSONB via REST APIs, Kafka message headers, and similar double-encoded formats.
On format parse failure, returns EmbeddedDecodeError{Format:"json"}. On marshal failure, returns EmbeddedEncodeError{Format:"json"}. Codec validation errors from inner propagate unchanged.
Example — CloudEvents-style event with a typed payload in a string field:
var eventCodec = codex.Struct[Event](
codex.RequiredField("type", codex.String(), ...),
codex.RequiredField("payload", format.EmbeddedJSON(userCodec), ...),
)
// Wire: {"type":"user.created","payload":"{\"id\":\"123\",\"name\":\"Alice\"}"}
// Go: Event{Type:"user.created", Payload:User{ID:"123", Name:"Alice"}}
func EmbeddedTOML ¶ added in v0.11.0
EmbeddedTOML returns a codex.Codec[T] where the wire type is a TOML-encoded string. The inner codec's constraints run on the decoded value.
Decode path: TOML string → toml.Decode → map[string]any → inner.Decode → T Encode path: T → inner.Encode → map[string]any → toml.Encode → TOML string
TOML integers decode as int64 and TOML floats as float64 — both are handled correctly by the built-in codex primitives.
Note: TOML requires all keys to be strings and does not support top-level arrays. Use EmbeddedTOML with struct codecs (not SliceOf).
On format parse failure, returns EmbeddedDecodeError{Format:"toml"}. On marshal failure, returns EmbeddedEncodeError{Format:"toml"}.
func EmbeddedYAML ¶ added in v0.11.0
EmbeddedYAML returns a codex.Codec[T] where the wire type is a YAML-encoded string. The inner codec's constraints run on the decoded value.
Decode path: YAML string → yaml.Unmarshal → map[string]any → inner.Decode → T Encode path: T → inner.Encode → map[string]any → yaml.Marshal → YAML string
YAML integers decode as int and YAML floats as float64 — both are handled correctly by the built-in codex primitives (Int, Float64, etc.).
On format parse failure, returns EmbeddedDecodeError{Format:"yaml"}. On marshal failure, returns EmbeddedEncodeError{Format:"yaml"}.
Types ¶
type EmbeddedDecodeError ¶ added in v0.11.0
type EmbeddedDecodeError struct {
// Format is the wire format that failed to parse ("json", "yaml", "toml").
Format string
// Err is the underlying parse error from the format library.
Err error
}
EmbeddedDecodeError is returned by EmbeddedJSON, EmbeddedYAML, and EmbeddedTOML when the string value cannot be parsed as the expected format. Codec validation errors from the inner codec propagate unchanged.
var dec format.EmbeddedDecodeError
if errors.As(err, &dec) {
slog.Warn("embedded field parse failed", "error", dec)
}
func (EmbeddedDecodeError) Error ¶ added in v0.11.0
func (e EmbeddedDecodeError) Error() string
func (EmbeddedDecodeError) LogValue ¶ added in v0.11.0
func (e EmbeddedDecodeError) LogValue() slog.Value
LogValue implements slog.LogValuer for structured logging.
type EmbeddedEncodeError ¶ added in v0.11.0
type EmbeddedEncodeError struct {
// Format is the wire format that failed to marshal ("json", "yaml", "toml").
Format string
// Err is the underlying marshal error from the format library.
Err error
}
EmbeddedEncodeError is returned by EmbeddedJSON, EmbeddedYAML, and EmbeddedTOML when the Go value cannot be marshalled to the target format string.
var enc format.EmbeddedEncodeError
if errors.As(err, &enc) {
slog.Warn("embedded field marshal failed", "error", enc)
}
func (EmbeddedEncodeError) Error ¶ added in v0.11.0
func (e EmbeddedEncodeError) Error() string
func (EmbeddedEncodeError) LogValue ¶ added in v0.11.0
func (e EmbeddedEncodeError) LogValue() slog.Value
LogValue implements slog.LogValuer for structured logging.
type Format ¶
type Format[T any] struct { // contains filtered or unexported fields }
Format binds a Codec[T] to a specific serialization format. Use JSON, YAML, TOML, Gob, or Binary to construct one. For formats that operate on the typed value directly (e.g. HTML rendering), use NewTyped. For streaming output (write to io.Writer without buffering), use NewStreamed.
func Binary ¶ added in v0.11.0
Binary returns a Format that reads and writes raw binary data.
It is equivalent to NewTyped[[]byte] with identity marshal and validate+identity unmarshal — bytes are stored and read exactly as they are, without any encoding. Unlike Gob (which adds framing bytes), Binary writes raw bytes that any tool understanding the underlying format (PNG viewer, PDF reader, etc.) can open.
The codec validates the []byte value via [Codec.Refine] constraints on both paths:
- Write: constraints run before the bytes are written; failure returns ports.FileEncodeError (via [ports.File]).
- Read: constraints run after the bytes are read; failure returns ports.FileDecodeError (via [ports.File]).
ContentType defaults to "application/octet-stream"; override with Format.WithContentType.
For typed binary formats where the Go type is not []byte (e.g. image.Image, Protobuf structs) or for write-only formats, use NewTyped directly.
Typical usage:
var pngSignature = []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A}
var pngFile = ports.NewFile(
"images/{name}.png",
format.Binary(
codex.Bytes().
Refine(validate.MaxBytes(5*1024*1024)).
Refine(validate.HasPrefix(pngSignature)),
).WithContentType("image/png"),
ports.FilePathParam{Name: "name"},
)
func Gob ¶ added in v0.9.0
Gob returns a Format[T] that serialises T using encoding/gob.
Unlike JSON, YAML, and TOML — which pass a map[string]any intermediate through the codec — Gob encodes and decodes the typed value directly. Codec constraints are enforced on both marshal (before encoding) and unmarshal (after decoding).
Suitable for internal Go-to-Go communication: forge pipelines, MQTT between Go services, and binary caching. Not suitable for REST content negotiation, human-readable output, or cross-language interoperability.
Requirements and limitations:
- All struct fields that gob encodes must be exported; unexported fields are silently skipped.
- For interface values, register concrete types with encoding/gob.Register before use.
- "application/gob" is a conventional content type; it is not an IANA-registered MIME type.
- Observability is handled by the adapter layer — no special configuration needed.
ContentType is "application/gob".
func JSON ¶
JSON returns a Format that reads and writes JSON. ContentType is "application/json".
Example ¶
package main
import (
"fmt"
"github.com/DaniDeer/go-codex/codex"
"github.com/DaniDeer/go-codex/format"
)
func main() {
type Item struct {
Name string
Price float64
}
itemCodec := codex.Struct[Item](
codex.RequiredField("name", codex.String(),
func(i Item) string { return i.Name },
func(i *Item, v string) { i.Name = v },
),
codex.RequiredField("price", codex.Float64(),
func(i Item) float64 { return i.Price },
func(i *Item, v float64) { i.Price = v },
),
)
j := format.JSON(itemCodec)
// Marshal a Go value to JSON bytes.
data, _ := j.Marshal(Item{Name: "Widget", Price: 9.99})
fmt.Println(string(data))
// Unmarshal JSON bytes back to the typed value.
item, _ := j.Unmarshal(data)
fmt.Printf("%s: %.2f\n", item.Name, item.Price)
}
Output: {"name":"Widget","price":9.99} Widget: 9.99
func New ¶
func New[T any](c codex.Codec[T], marshal func(any) ([]byte, error), unmarshal func([]byte) (any, error)) Format[T]
New creates a Format from a codec and custom marshal/unmarshal functions. Use this to integrate formats not covered by the built-in constructors. ContentType is empty by default; call Format.WithContentType to set it.
func NewStreamed ¶ added in v0.8.0
func NewStreamed[T any](c codex.Codec[T], marshalTo func(T, io.Writer) error, unmarshal func([]byte) (T, error), contentType string) Format[T]
NewStreamed creates a Format where responses are written directly to an io.Writer without buffering to a []byte intermediate. This enables chunked or streaming responses for large payloads (HTML pages, JSON arrays, CSV exports).
The codec is used for validation: Format.MarshalTo runs all Refine constraints on the value before calling marshalTo, so invalid data is rejected before any bytes are written.
unmarshal is used for Format.Unmarshal (reading streaming formats is rarely needed; pass a function that returns an error when not applicable).
Use Format.IsStreamable to detect streaming formats. The adapter calls Format.MarshalTo instead of Format.Marshal and writes response headers before streaming, so partial output is never flushed on validation failure.
Example — streaming a templ component without buffering:
streamFmt := format.NewStreamed(
propsCodec,
func(props Props, w io.Writer) error {
return component(props).Render(context.Background(), w)
},
func([]byte) (Props, error) {
var zero Props
return zero, errors.New("HTML is not decodable")
},
"text/html; charset=utf-8",
)
func NewTyped ¶ added in v0.8.0
func NewTyped[T any](c codex.Codec[T], marshal func(T) ([]byte, error), unmarshal func([]byte) (T, error), contentType string) Format[T]
NewTyped creates a Format where the marshal and unmarshal functions operate on the typed value directly, rather than on the intermediate representation. The codec is still used for validation: Format.Marshal runs all Refine constraints on the value before calling marshal, and Format.Validate works as normal.
Use this when the wire format cannot be represented via a map[string]any intermediate — for example, rendering HTML via a templ component:
htmlFormat := format.NewTyped(
propsCodec,
func(props Props) ([]byte, error) {
var buf bytes.Buffer
err := component(props).Render(context.Background(), &buf)
return buf.Bytes(), err
},
func([]byte) (Props, error) {
var zero Props
return zero, errors.New("HTML is not decodable")
},
"text/html; charset=utf-8",
)
func (Format[T]) Codec ¶ added in v0.12.0
Codec returns the codec this format validates through. Exposed for callers (like [ports.File]/[ports.PatchEncoded]) that need to validate/decode an intermediate value directly, outside a full Marshal/Unmarshal round trip.
func (Format[T]) ContentType ¶ added in v0.8.0
ContentType returns the MIME type associated with this format (e.g. "application/json"). Empty string means the format has no registered content type.
func (Format[T]) IsPatchable ¶ added in v0.11.0
IsPatchable reports whether partial-patch operations (used by [ports.File.Patch]/[ports.PatchEncoded]) are supported for this format. Only map-based formats (JSON, YAML, TOML, and formats created with New) return true — they use a [map[string]any] intermediate that can be partially overridden. Gob, Binary, NewTyped, and NewStreamed formats return false.
func (Format[T]) IsStreamable ¶ added in v0.8.0
IsStreamable reports whether this format supports streaming output via Format.MarshalTo. Streaming formats are created with NewStreamed.
func (Format[T]) Marshal ¶
Marshal encodes v to bytes using the codec and then the format serializer. If the format was created with NewTyped, the codec validates v first and then the typed marshal function is called directly (bypassing the intermediate). For streaming formats created with NewStreamed, use Format.MarshalTo instead.
func (Format[T]) MarshalRaw ¶ added in v0.12.0
MarshalRaw serializes an already-encoded intermediate value directly, WITHOUT running it through the codec's Encode. Only valid when Format.IsPatchable is true — exposed for callers that need to write back a merged intermediate map that may carry fields outside the file's own codec (see [ports.PatchEncoded]). Prefer Format.Marshal for a normal full encode.
func (Format[T]) MarshalTo ¶ added in v0.8.0
MarshalTo validates v via the codec and then writes the serialized form directly to w without buffering to a []byte intermediate. Use this with formats created via NewStreamed; call Format.IsStreamable first. Returns ErrNotStreamable if the format has no streaming marshal function. Validation errors are returned before any bytes are written to w.
func (Format[T]) PatchInto ¶ added in v0.12.0
PatchInto parses existing bytes into a map[string]any intermediate, deep-patches it with the provided patch map, and decodes the result through the codec to validate all Refine constraints. Only valid for formats where Format.IsPatchable is true — used by [ports.File.Patch].
func (Format[T]) Unmarshal ¶
Unmarshal deserializes data into an intermediate and then decodes it via the codec. If the format was created with NewTyped, the typed unmarshal function is used directly.
func (Format[T]) UnmarshalRaw ¶ added in v0.12.0
UnmarshalRaw deserializes data into the format's untyped intermediate (e.g. map[string]any for JSON/YAML/TOML), WITHOUT running it through the codec's Decode. Only valid when Format.IsPatchable is true — exposed for callers that need the raw intermediate to deep-merge a partial patch before validating (see [ports.PatchEncoded]). Prefer Format.Unmarshal for a normal full decode.
func (Format[T]) Validate ¶
Validate checks v against the codec's constraints without serializing to bytes. It delegates to Codec.Validate — see its documentation for the rationale.
func (Format[T]) WithContentType ¶ added in v0.8.0
WithContentType returns a copy of the format with the given MIME content type set. Use this when registering custom formats for content negotiation.