dynamicpb

Overview ¶

Package dynamicpb provides a dynamic protobuf message implementation that constructs messages from descriptors at runtime without generated code. It enables creating, populating, serializing, and deserializing protobuf messages when the message schema is only known at runtime, such as when processing descriptors obtained from a protobuf registry or a gRPC reflection server.

The core type is Message, a concrete struct that implements both protoreflect.Message (12 reflection methods including Get, Set, Has, Clear, Range, and WhichOneof) and proto.Message (9 embedded interfaces including MarshalAppender, Unmarshaler, Merger, Cloner, and Validator). This dual conformance gives dynamic messages full interoperability with the proto ecosystem: they can be marshaled, unmarshaled, merged, cloned, validated, and reflected upon using the same interfaces as generated message types.

Two container types support composite fields:

• List is a repeated field container implementing protoreflect.List. It provides indexed access to repeated elements with Get, Set, Append, and Truncate, following Go slice semantics.

• Map is a map field container implementing protoreflect.Map. It provides key-based access to map entries with Get, Set, Has, Clear, and Range, following Go map semantics with undefined iteration order.

The New function is the primary constructor. It accepts any MessageDescriptorAccessor and returns a fully initialized Message ready for use:

msg := dynamicpb.New(messageDescriptor)
msg.Set(fieldDesc, protoreflect.ValueOfString("hello"))
data, err := msg.Marshal()

NewList and NewMap construct standalone containers for repeated and map fields respectively.

Primary use cases include gRPC reflection servers that process messages without compile-time type knowledge, schema-driven middleware that inspects and transforms messages in transit, protobuf registries that store and retrieve arbitrary message types, and serialization pipelines that decode wire data, inspect or transform fields, and re-encode without losing information.

Field values are stored as protoreflect.Value entries in a map keyed by field number, providing O(1) field access for Get, Set, Has, and Clear. The constructor pre-builds a lookup table mapping field numbers to their FieldDescriptorAccessor, enabling O(1) dispatch during decode without repeated descriptor iteration. The values map and oneof tracking map are lazily initialized on first Set to avoid allocation for messages that are only read, such as default value sentinels returned by Get on unset message fields.

The package follows a split error convention. Programmer errors such as passing a field descriptor from a different message to Get, Set, Has, or Clear, or passing a nil descriptor to New, cause panics prefixed with "dynamicpb: " for easy identification (e.g., "dynamicpb: field \"name\" does not belong to message \"example.Foo\""). Data-driven failures return errors: Unmarshal returns errors from the decode package for malformed wire data, Merge returns an error when the source and destination have incompatible descriptors, and Validate returns an error listing missing required fields or invalid closed enum values.

This package does not provide thread safety. Dynamic messages are not safe for concurrent mutation, matching the semantics of Go maps and slices. Extension fields and well-known type special handling (Any, Timestamp, Duration, Struct) are out of scope and planned for future work.

This package depends on descriptor, protoreflect, proto, encode, decode, scalar, and wire from this module, plus fmt, math, sort, and errors from the standard library. It has zero external dependencies.

message.go defines the Message struct and its constructors for dynamic protobuf messages. Field lookup tables, oneof tracking, and container factory helpers live here; the protocol-specific methods (marshal, unmarshal, reflect, merge/clone) are in their respective split files. The init()-based registration of newMessageFunc is in init_gc.go; TinyGo consumers should call Register() from register.go instead.

message_fields.go implements Merge, Clone, Reset, Validate, and their supporting deep-copy and validation helpers for dynamic messages.

message_marshal.go implements Size, MarshalAppend, and Marshal for dynamic messages, along with the scalar-to-uint64 conversion helpers used by both size computation and wire encoding.

message_reflect.go implements the protoreflect.Message interface methods for dynamic messages: Descriptor, New, Interface, ProtoReflect, IsValid, Get, Set, Has, Clear, Range, WhichOneof, GetUnknown, and SetUnknown.

message_unmarshal.go implements Unmarshal and its supporting decode methods for dynamic messages, handling singular, repeated, packed, and map field decoding from wire-format bytes.