binary

Binary to Map Encoder

Encode and decode arbitrary binary structure data in golang.

Notes and Caveats

• Bitfields are not support at this time
• Presently this library only supports decoding from arbitary binary structures to a go map[string]interface{}. Encoding is in progress.
• This project is based on the kaitai project and supports a subset of the primitives. Using simple yaml based spec, conversion from binary to other structures or json can be performed simply. The primary difference between this an kaitai is the ablity to decode arbitrary data without have to pre-compile structure source.

Sample Data

The examples use the following C stucture, encoded to base64.

C Structure

#include <string.h>
#include <stdint.h>

typedef struct tagSensors {
    double ph;
    double ec;
} Sensors;

typedef struct tagReading {
    char probe_id[32];
    double temp;
    double humidity;
    Sensors sensors;
} Reading;

Reading N={ "000001", 12.0, 0.88, { 0.34, 0.45 }};

Base64 Value

The Reading data would have this value on the wire (base64).

MDAwMDAxAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAKAAAAAAAAD/sKPXCj1wpP9XCj1wo9cM=

Sample Spec

The spec follows the kaitai format as closely as possible. There are two additional fields for sequence values that are supported.

1. omit - The value will be omitted from the final input.
2. encodedValue - evaluation expression for the encoding the value (not yet supported).

meta:
  id: iot_reading
  endian: be
seq:
  - id: probe_id
    type: str
    size: 32
  - id: temp
    type: f8
    doc: the temperature
  - id: humidity_raw
    type: f8
    doc: the raw humidity
    omit: true
  - id: sensors
    type: sensor_data
types:
  sensor_data:
    seq:
      - id: ph_raw
        type: f8
        omit: true
      - id: ec_raw
        type: f8
        omit: true
    instances:
      pH:
        value: seq(sensors, 'ph_raw') * 100
      ec:
        value: ec_decoder() / 2.3
instances:
  humidity:
    value: humidity_raw * 100

Spec Components

Sequences

The spec seq is an array of binary data types defined in order in the data, starting at index 0. Each sequence must have an id and an optional type. The default type will return a []byte. If type is omitted, size should be provided, otherwise it is assumed the sequence continues to the end-of-stream (eos).

Sequences can be either primitives or user defined types. Primitives are defined in the format:

<type><width><endian>

• u - unsigned integer
• s - signed integer
• f - floating point

Signed (s) and Unsigned (u) integers support 1 (8-bit), 2 (16-bit), 4 (32-bit), and 8 (64-bit) widths.

Floating points support 4 (32-bit) and double 8 (64-bit) widths.

• be - Big Endian
• le - Little Endian (default)

String Types

str - The str type will decode a string of UTF-8 byte characters. This type requires the size attribute be specified in the sequence.

strz - The strz type will decode a null \0 terminated string.

Instances

The spec instances hash contains named components that either have relative positioning in the data, or a derived value.

User defined types

The spec types is a hash of named types that can be use in the sequences or other types.

Initializing and Decoding the Spec

specData, err := ioutil.ReadFile("spec.yaml")
if err != nil {
    panic(err)
}

spec, err := binary.NewSpec(specData)
if err != nil {
    panic(err)
}

Initializing with Custom Functions

spec, err := binary.NewSpec(specData, map[string]binary.EvalFunc{
    "ec_decoder":
        func(ctx types.StringMap, args ...interface{}) (interface{}, error) {
            return ctx.Float64("sensors.ec_raw") * 1000, nil
    },
})
if err != nil {
    panic(err)
}

Decoding the Data and converting to JSON

Data is expected to be raw bytes.

// data is the raw []byte, base64 decoded value
rval, err := spec.Decode(data)
if err != nil {
    panic(err)
}

out, err := json.MarshalIndent(rval, "", "\t")
if err != nil {
    panic(err)
}

Output:

{
  "humidity": 88,
  "probe_id": "000001",
  "sensors": {
    "ec": 195.6521739130435,
    "pH": 34
  },
  "temp": 12
}

Decoding to a struct

import "gitlab.com/ModelRocket/sparks/util"

type reading struct {
    ProbeID  string `json:"probe_id"`
    Humidity int64
    Sensors  struct {
        EC float64
        PH float64
    }
    Temp float64
}

func main() {
    // ...

    r := &reading{}
    if err := util.MapStruct(r, rval); err != nil {
        panic(err)
    }
}

Advanced Sequence and Instance Values

Repeating and array values are supported to kaitai spec. For example this C structure:

#include <string.h>
#include <stdint.h>

typedef struct tagSensors {
    char name[16];
    double value;
} Sensors;

typedef struct tagReading {
    int8_t vals_len;
    Sensors vals[2];
} Reading;

Reading N={1, {{"ph", 12.0}, { "ec", 0.88}}};

Would be represented as:

meta:
  id: iot_reading
  endian: be
seq:
  - id: vals_len
    type: s1
    omit: true
  - id: vals
    type: sensor_data
    repeat: expr
    repeat-expr: vals_len
types:
  sensor_data:
    seq:
      - id: name
        type: str
        size: 16
      - id: value
        type: f8
    instances:
      value_converted:
        value: context('vals.0.value') * 100

Accessing other values in selectors

There are two special builtin function that allows for accesing values inside of other structs:

• seq(member, key) - returns a sub for an embedded member

Example: seq(sensors, 'ph_raw')

• context(key) - returns a value in the current context

Example: context('_inst.value')

Note: in this example, the special key _inst is used to access the current array value being operated on. A companion key to this is _index which gives the current index of the array value being operated on.

types:
  sensor_data:
    seq:
      - id: name
        type: str
        size: 16
      - id: value
        type: f8
    instances:
      value_real:
        value: context('_inst.value') * 100

{
  "vals": [
    {
      "name": "ph",
      "value": 12,
      "value_real": 1200
    },
    {
      "name": "ec",
      "value": 0.88,
      "value_real": 88
    }
  ]
}

Accessing values in arrays

Arrays can be accessed using a subscript in the dot notation i.e. context('vals.0.value') * 100

types:
  sensor_data:
    seq:
      - id: name
        type: str
        size: 16
      - id: value
        type: f8
    instances:
      value_total:
        value: context('vals.0.value') * 100