README
¶
serial
This package is a proof-of-concept modeling a general purpose serialization strategy after the articles by Glenn Fiedler.
It is intended to demonstrate the serialization pattern described, which aims to tightly pack bytes to minimize transport costs in a UDP communication system, while also providing a single function for both read and write to reduce potential for human error when modifications to the structure are made.
The code has seen several revisions, balancing execution performance, convenience, and the smallest byte packing possible while using the encoding/binary package in LittleEndian. The latest iteration has traded for convenience at the cost of reduced built-in functionality. This is mostly due to the lack of sane alternatives when dealing with variable size types, most especially complex combinations such as maps or unsized slices.
While it can achieve significantly smaller byte sizes than tools like encoding/gob or even github.com/tinylib/msgp, it comes at the expense of spending many hours fine-tuning and writing the serialization logic by hand.
This code comes with a full set of unit tests, and the benchmarks have been cleanly separated to create a more well defined example without mixing up code.
conclusions
With the MTU of only 1500 bytes, every byte counts when trying to create a communication protocol over UDP that does not fragment.
I have to agree with Glenn Fiedler regarding the fact that the best performance can only be achieved by explicitly defining serialization per structure.
However, depending on your use case the msgp package offers absolutely stellar performance and very low memory consumption. It produces results just under 2x the size of manual serialization, but at zero cognitive overhead. Using this lets you focus on the other parts of the application, and fits very nicely within the same overall packet-based strategy.
On the other hand, I cannot recommend the encoding/gob package. It consumes significantly more memory, and sometimes upwards of 6x the amount of bytes due to the amount of metadata generated. Additionally it is more geared towards streaming, and has heavy dependency on a per-instance cache. This model conflicts with packet based UDP, and new instances are an order of magnitude slower to parse or write messages.
To summarize, if you want to focus on the rest of your application just use the msgp package. If you get to a point where you need to optimize, it takes very little effort to exchange it for custom serialization. However, writing custom serialization is very expensive. It is best to plan according to your applications needs.
references
Documentation
¶
Overview ¶
This package is a proof-of-concept demonstrating a serialization pattern as described by Glenn Fiedlers articles on UDP network traffic.
It has gone through several iterations to try new things, both to improve the performance, as well as the byte-packing.
This latest iteration discards variable support in favor of simplification.
The benchmarks have been enhanced to randomly populate a complex structure and use both tinylib/msgp and encoding/gob to compare both performance and byte size.
While I have strategies to running serialization against a variety of data I have not yet identified a sane pattern for complex variable size types, such as maps, and slices of variable length or that contain variable length data.
The benchmarks show significant packing when using this serialization tool, which can be further optimized through careful consideration when writing the serialization logic per structure. However, this process is cognitively expensive.
I cannot recommend the gob package. It has enormous variable sizes due to the amount of metadata stored, and in scenarios where you cannot rely on an existing instance it performs quite horribly by depending on a cache per instance, without which significantly more effort (likely via reflect) and more allocations are required.
The msgp solution is highly recommended for many reasons. First it performs far better with zero effort than manual serialization. Anywhere from 30% to several times faster. It also has roughly 5 times less allocations, which is much lighter on memory consumption. It supports a very similar pattern as the serialization strategy, and would be trivial to connect to a gzip package for example. The first downside is that it consumes around twice as many bytes as serialization. The second is that it produces a sizable amount of generated code. The trade-off is that generated code takes zero effort away from the core development. At worst, msgp is the absolute best way to get started with a project that needs good network communication.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Read ¶
type Read struct {
// contains filtered or unexported fields
}
A serialization writer which exposes the Serialize function.
func (*Read) Serialize ¶
This method directly funnels all input into binary.Write, which only accepts fixed size data types, and will not work on variable length data such as int, uint, string, slices of those types, and structures that contain them.
It will return immediately on the first error encountered.
While it can accept a slice and populate the existing instances, but it does not restore dynamic sized records. To work around this, expect the size to be stored and load that first.
type Write ¶
type Write struct {
// contains filtered or unexported fields
}
A serialization writer which exposes the Serialize function.
func (*Write) Serialize ¶
This method directly funnels all input into binary.Write, which only accepts fixed size data types, and will not work on variable length data such as int, uint, string, slices of those types, and structures that contain them.
It will return immediately on the first error encountered.
While slices of fixed-size types are accepted, the size of that slice is not stored with the data. To work around this, store the size first.
Source Files
Directories