binary

Binary to Map Encoder

Encode and decode arbitrary binary structure data in golang.

Notes and Caveats

• Bitfields are not support at this time
• Presently this library only supports decoding from arbitary binary structures to a go map[string]interface{}. Encoding is in progress.
• This project is based on the kaitai project and supports a subset of the primities. Using simple yaml based spec, conversion from binary to other structures or json can be performed simply. The primary difference between this an kaitai is the ablity to decode arbitrary data without have to pre-compile structure source.

Sample Data

The examples use the following C stucture, encoded to base64.

C Structure

#include <string.h>
#include <stdint.h>

typedef struct tagSensors {
    double ph;
    double ec;
} Sensors;

typedef struct tagReading {
    char probe_id[32];
    double temp;
    double humidity;
    Sensors sensors;
} Reading;

Reading N={ "000001", 12.0, 0.88, { 0.34, 0.45 }};

Base64 Value

The Reading data would have this value on the wire (base64).

MDAwMDAxAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAKAAAAAAAAD/sKPXCj1wpP9XCj1wo9cM=

Sample Spec

The spec follows the kaitai format as closely as possible. There are two additional fields for sequence values that are supported.

1. omit - The value will be omitted from the final input.
2. encodedValue - evaluation expression for the encoding the value (not yet supported).

meta:
  id: iot_reading
  endian: be
seq:
  - id: probe_id
    type: str
    size: 32
  - id: temp
    type: f8
    doc: the temperature
  - id: humidity_raw
    type: f8
    doc: the raw humidity
    omit: true
  - id: sensors
    type: sensor_data
types:
  sensor_data:
    seq:
      - id: ph_raw
        type: f8
        omit: true
      - id: ec_raw
        type: f8
        omit: true
    instances:
      pH:
        value: seq(sensors, 'ph_raw') * 100
      ec:
        value: ec_decoder() / 2.3
instances:
  humidity:
    value: humidity_raw * 100

Initializing and Decoding the Spec

specData, err := ioutil.ReadFile("spec.yaml")
if err != nil {
    panic(err)
}

spec, err := binary.NewSpec(specData)
if err != nil {
    panic(err)
}

Initializing with Custom Functions

spec, err := binary.NewSpec(specData, map[string]binary.EvalFunc{
    "ec_decoder":
        func(ctx types.StringMap, args ...interface{}) (interface{}, error) {
            return ctx.Float64("sensors.ec_raw") * 1000, nil
    },
})
if err != nil {
    panic(err)
}

Decoding the Data and converting to JSON

Data is expected to be raw bytes.

// data is the raw []byte, base64 decoded value
rval, err := spec.Decode(data)
if err != nil {
    panic(err)
}

out, err := json.MarshalIndent(rval, "", "\t")
if err != nil {
    panic(err)
}

Output:

{
  "humidity": 88,
  "probe_id": "000001",
  "sensors": {
    "ec": 195.6521739130435,
    "pH": 34
  },
  "temp": 12
}

Decoding to a struct

import "gitlab.com/ModelRocket/sparks/util"

type reading struct {
    ProbeID  string `json:"probe_id"`
    Humidity int64
    Sensors  struct {
        EC float64
        PH float64
    }
    Temp float64
}

func main() {
    // ...

    r := &reading{}
    if err := util.MapStruct(r, rval); err != nil {
        panic(err)
    }
}

Advanced Sequence and Instance Values

Repeating and array values are supported to kaitai spec. For example this C structure:

#include <string.h>
#include <stdint.h>

typedef struct tagSensors {
    char name[16];
    double value;
} Sensors;

typedef struct tagReading {
    int8_t vals_len;
    Sensors vals[2];
} Reading;

Reading N={1, {{"ph", 12.0}, { "ec", 0.88}}};

Would be represented as:

meta:
  id: iot_reading
  endian: be
seq:
  - id: vals_len
    type: s1
    omit: true
  - id: vals
    type: sensor_data
    repeat: expr
    repeat-expr: vals_len
types:
  sensor_data:
    seq:
      - id: name
        type: str
        size: 16
      - id: value
        type: f8
    instances:
      value_converted:
        value: context('vals.0.value') * 100

Accessing other values in selectors

There are two special builtin function that allows for accesing values inside of other structs:

• seq(member, key) - returns a sub for an embedded member

Example: seq(sensors, 'ph_raw')

• context(key) - returns a value in the current context

Example: context('_inst.value')

Note: in this example, the special key _inst is used to access the current array value being operated on.

types:
  sensor_data:
    seq:
      - id: name
        type: str
        size: 16
      - id: value
        type: f8
    instances:
      value_real:
        value: context('_inst.value') * 100

{
  "vals": [
    {
      "name": "ph",
      "value": 12,
      "value_real": 1200
    },
    {
      "name": "ec",
      "value": 0.88,
      "value_real": 88
    }
  ]
}