msgpack

msgpack (Quad4 fork)

This repository is a fork of github.com/vmihailenco/msgpack (v5 API), maintained by Quad4 at git.quad4.io/Go-Libs/msgpack.

The wire format and public API are unchanged: every Marshal / Unmarshal / Encoder / Decoder call signature, struct tag, and option from upstream v5.4.1 continues to work. This fork adds security, correctness, and performance fixes; it does not introduce breaking changes.

Drop-in migration from github.com/vmihailenco/msgpack/v5

1. Replace the import path:

   // before
   import "github.com/vmihailenco/msgpack/v5"
   
   // after
   import "git.quad4.io/Go-Libs/msgpack/v5/pkg/msgpack"
   

   The package is still imported as msgpack, so call sites do not need to change.

2. Pull the module:

   go get git.quad4.io/Go-Libs/msgpack/v5@latest
   

3. (Optional) For users of the App Engine helpers:

   // before
   import "github.com/vmihailenco/msgpack/v5/msgpappengine"
   
   // after
   import "git.quad4.io/Go-Libs/msgpack/extra/msgpappengine"
   

   This is a separate Go module; install with go get git.quad4.io/Go-Libs/msgpack/extra/msgpappengine@latest.

The wire codes (subpackage msgpcode) move from github.com/vmihailenco/msgpack/v5/msgpcode to git.quad4.io/Go-Libs/msgpack/v5/pkg/msgpack/msgpcode. Constants are unchanged.

What this fork fixes vs. upstream v5.4.1

The upstream module has been effectively unmaintained for several years. This fork addresses, without changing the API:

Security

• OOM via forged length prefixes. Four decode paths (bytes, bytesPtr, decodeSlice, decodeSliceValue, DecodeMap, DecodeUntypedMap) trusted the on-wire length and allocated up front, so a single hostile bin32 / str32 / array32 / map32 header could request a multi-gigabyte allocation before the underlying short read failed. All four now clamp the initial allocation to the documented per-decoder limit and grow incrementally as real bytes arrive. The disableAllocLimitFlag toggle (Decoder.DisableAllocLimit(true)) preserves the legacy unbounded behaviour for callers that want it.
• Stack exhaustion and 32-bit length overflow hardening. Decoder recursion now enforces a depth cap (SetDecodeDepthLimit, default 10,000) so hostile deeply nested payloads fail with an error instead of consuming unbounded stack. str32 / bin32 / array32 / map32 / ext32 length parsing now rejects uint32 lengths that overflow int on 32-bit builds rather than silently wrapping.
• disableAllocLimitFlag was a no-op in decodeSliceValue. Upstream compared the bit flag against the literal 1, but the flag is 1 << 1 == 2, so the limit was never applied along the typed-slice path. The comparison is now != 0.
• Goroutine and memory leak in the per-type preallocator. The upstream cachedValue spawned one perpetual goroutine for every distinct reflect.Type ever decoded into, plus a 256-slot buffered channel each, and held a global sync.RWMutex on the hot decode path. A long-running process that ever decoded into N distinct Go types retained N goroutines forever. The preallocator is now backed by a sync.Map of *sync.Pool; idle entries are reclaimed by the GC, the global mutex is gone from the lookup path, and goroutine count stays flat under churn. Verified by TestNoGoroutineLeakOnDistinctTypes and TestConcurrentDistinctTypesPreallocate in pkg/msgpack/leak_test.go.

Correctness

• invalid code=cb decoding int64 on JSON-sourced data. (*Decoder).int and (*Decoder).uint now accept msgpcode.Float / msgpcode.Double payloads when the destination is a Go integer, provided the value is finite, integer-valued, and in range. This unblocks the common JSON -> map[string]any -> msgpack -> struct round-trip (encoding/json decodes JSON numbers into float64 regardless of declared field type). NaN, infinities, fractional values, negative values into uint64, and out-of-range magnitudes still error rather than silently truncating.
• Pooled *bytes.Reader does not retain caller data. Marshal / Unmarshal reuse internal buffers via sync.Pool; the wrappers are reset to nil before being returned to the pool so they cannot leak the previous caller's slice into a subsequent call. Verified by TestPoolDoesNotRetainCallerData and TestInvariantPoolDoesNotAlias.

Performance (vs. upstream v5.4.1, geomean over 5 x 2s benchstat runs)

• Unmarshal: pooled *bytes.Reader wrapper. BenchmarkStructUnmarshal -50% B/op (96 -> 48), -1 alloc/op, -5.16% time. BenchmarkStructUnmarshalPartially -75% B/op (64 -> 16), -1 alloc/op, -6.68% time.
• Marshal: pre-grows the encode buffer to 64 bytes, skipping the first one or two backing-array doublings for typical small payloads. The returned slice still owns its backing array; aliasing semantics are preserved.
• AppendMarshal / (*Encoder).Append: caller-owned destination-buffer APIs for hot paths that reuse output capacity; with a warm buffer they run at zero allocs/op on both scalar and representative struct benchmarks.
• byteWriter.WriteByte: writes through a 1-byte field on the wrapper struct instead of allocating a fresh []byte{c} per call. BenchmarkDiscard -100% B/op, -100% allocs/op.
• Pooled *Encoder and *Decoder (GetEncoder / PutEncoder, GetDecoder / PutDecoder) work as before; reuse benchmarks added in pkg/msgpack/bench_test.go.

Install

go get git.quad4.io/Go-Libs/msgpack/v5@latest

import "git.quad4.io/Go-Libs/msgpack/v5/pkg/msgpack"

The module path is git.quad4.io/Go-Libs/msgpack/v5 (the /v5 suffix matches the major version). Source lives under pkg/msgpack/; subpackage msgpcode is at pkg/msgpack/msgpcode.

Features

• Primitives, arrays, maps, structs, time.Time, and interface{}.
• Allocation-aware API surface: Marshal for convenience and AppendMarshal / (*Encoder).Append for caller-managed reusable output buffers.
• App Engine *datastore.Key and datastore.Cursor via extra/msgpappengine (optional module).
• CustomEncoder / CustomDecoder for custom encoding.
• Extensions, struct tags (msgpack:"..."), omitempty, sorted map keys, array-encoded structs, and Decoder.Query-style path queries.

Layout

Testing

• Unit tests in pkg/msgpack/*_test.go cover the decoder/encoder surface, struct round-trips, time, ext, intern, and query paths.
• Property-based tests via pbt (git.quad4.io/Go-Libs/pbt/pkg/pbt) in pbt_test.go (roundtrip properties for []byte, string, []int, []string, map[string]int, map[string]string).
• Fuzz targets in fuzz_test.go: FuzzMarshalUnmarshalRoundtrip, FuzzUnmarshalArbitrary, FuzzDecoderQuery, FuzzDecodeIntoStruct, FuzzDecodeExtHeader, FuzzDecodeTime, FuzzDecodeInternedString, FuzzDecodeSkip, FuzzDecodeMulti. Regression corpora for the allocation-limit fixes are committed under pkg/msgpack/testdata/fuzz/.
• Stress tests in stress_test.go: concurrent Marshal / Unmarshal under -race, large byte slice / string round-trips, deeply nested map and slice round-trips, repeated reuse of pooled encoder/decoder.
• Invariant tests in invariant_test.go: Unmarshal(nil) / empty input never panics; Marshal(nil) is a single msgpcode.Nil byte; bit-exact round-trip of int64 min, uint64 max, NaN, -Inf; pool aliasing checks.
• Leak tests in leak_test.go: per-type preallocator must not retain goroutines or values across decoder churn.

Run go test -race ./... for the full suite; make runs go vet plus tests.

License

BSD 2-clause; see LICENSE. Original copyright remains with the vmihailenco authors; fork maintenance is attributed in this README.