jsonData

jsonData — Developer Usage Guide

A small, dependency-light Go library for reading and writing dynamic JSON documents. The centrepiece is the Typ struct (to be read as type making the package look like jsonData.Typ and sound like JSON Data Type), a thin, type-aware wrapper around map[string]any (aliased as StringAnyMap).

The library exists to solve one specific pain point: working with dynamic JSON content in Go.

Table of contents

1. Why does it exist
2. The Typ type
3. Creating a Typ
4. JPath syntax
5. Reading values — GetValueByJPath
6. Writing values
7. Top-level JSON arrays
8. Serialisation & string methods
9. Using Typ with database/sql
10. JSON marshal / unmarshal
11. Important caveats
12. Quick recipes

Why does this library exist

Go is a strictly typed language, while JSON is a very flexible, and rather lightly structured data type.

For example, an array is always homogenous in Go. It is not so in JSON. The following is a valid JSON array:

[1, 2, "vaibhav", {"city": "Ranchi"}]

But you cannot have any type in Go to handle this automatically using the standard Marshal and Unmarshal functions from Go's stdlib.

In addition, if you are dealing with a JSON documents with varying structure then you cannot have a concrete struct into which you can Unmarshal a JSON text. The option that remains is to cast it to either an []any type or a map[string]any type. However, in both the cases, writing reliable code to navigate those entities is much harder and prone to errors; that is if you are ready to deal with how ugly that looks, especially for deeply nested objects.

Despite all this, JSON is part and parcle of modern development. Web apps, mobile apps, databases, caches - all of varying type - they all support JSON in one form or another. This library alleviates the pain of dealing with JSON in go.

This library provides utility functions to deal with JSON data from APIs, and can also deal with database json/jsonb columns that otherwise might come back from the DB driver as strings. If you send that string straight out of a REST API, every quote and newline gets escaped (\", \\n) and clients (Postman, test suites, etc.) have a hard time parsing it. Typ keeps the data as a real in-memory structure and re-encodes it cleanly on the way out.

The 3 primary pain points that it deals with:

1. Clean handling of JSON/JSONB columns in databases (as a database/sql driver) without writing custom logic.
2. Getting a value from JSON using a single path expression (called JPath, explained later).
3. Setting a scalar value into JSON using a single path expression (using JPath).

1. The Typ type

type (
    StringAnyMap map[string]any
    Typ          struct {
        Valid            bool
        hasTopLevelArray bool
        StringAnyMap
    }
)

• StringAnyMap — the underlying map holding the parsed JSON. It is embedded, so a Typ is a map[string]any plus metadata.
• Valid — false means the object is "null"/empty/invalid. true means it holds real JSON.
• hasTopLevelArray — internal flag set when the source JSON is a top-level array (e.g. [1,2,3,4]). See section 6.

A Typ is value-passable, but because it embeds a map (a reference type), be aware of aliasing — see caveats.

The well-known invalid singleton is:

var NullJsonData Typ   // Valid == false, empty map

⚠️ Do not treat NullJsonData as a reusable template: mutating a Typ through it mutates shared state. Create fresh objects instead (below).

2. Creating a Typ

From any value — ToJsonData

The most common entry point. Accepts a JSON string, a []byte, or any Go value (struct, map, slice…) and returns a Typ.

// From a JSON string
jo, err := ToJsonData(`{"name": "vaibhav", "age": 30, "tags": ["go", "json"]}`)
if err != nil {
    // handle
}

// From a Go struct — it is marshalled to JSON first, then re-parsed
type User struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}
jo, err = ToJsonData(User{Name: "Kaushal", Age: 28})

From a single key/value — NewJsonData

jo := NewJsonData("name", "Vaibhav")
// -> {"name":"Vaibhav"}

⚠️ If the key is blank (or only whitespace), NewJsonData silently returns an empty (but valid) object instead of an error. Prefer ToJsonData when you might pass a blank key

A fresh, empty, valid object — EmptyNotNullJsonData

jo := EmptyNotNullJsonData()
// -> a valid Typ wrapping an empty map  ({}), safe to mutate in any goroutine

EmptyNotNullJsonData is a function, not a global var, on purpose: it hands out a brand-new map every call so concurrent goroutines never share/alias the same underlying map.

3. JPath syntax

A JPath is a dotted path that navigates the JSON document. It is not JSONPath — it is this library's own, simpler expression language.

Rules & limits

• Segment separator is .. A key that itself contains . cannot be addressed (there is no escaping).
• Array indices are zero-based, non-negative integers only. [-1], [1.5], [abc], [ ] (blank) are all rejected.
• [] (append) is only valid as the last segment of a path and only for append operation.
• Malformed paths return an error, they do not panic:
  • leading/trailing . or .. (empty segment)
  • [ without a closing ]
  • non-numeric index inside [ ]
• Validate a path string on its own with ValidateJPathSyntax(path) error before using it (e.g. for user-supplied paths).

if err := ValidateJPathSyntax("arrObjs.[2].objName"); err != nil {
    // path is syntactically bad
}

4. Reading values — GetValueByJPath

func (j *Typ) GetValueByJPath(path string) (dataType string, value any, err appError.Typ)

Returns three values:

1. dataType — a string naming the detected type (one of the Type* constants below).
2. value — the raw value as any.
3. err — an appError.Typ. If err.IsNotBlank() (i.e. not blank), an error occurred and the first two values must be ignored.

Detecting success

typ, val, err := jo.GetValueByJPath("obj.nestedObj.child0.lvl")
if err.IsNotBlank() {
    // path invalid / element not found / wrong shape — handle it
    return
}
// safe to use typ + val

Data-type → Go-type mapping

The value you get back should be type-asserted according to dataType:

Examples

jo, _ := ToJsonData(`{"str":"x","n":5,"arr":[1,2,3],"obj":{"a":"b"}}`)

// A string
typ, val, err := jo.GetValueByJPath("str")   // typ == "string"
s := val.(string)                            // "x"

// A nested key
_, nested, err := jo.GetValueByJPath("obj.a")  // "b"

// An array element
typ, val, err = jo.GetValueByJPath("arr.[1]")  // typ == "float64" (see caveat)
arr1 := val.()                           // [1 2 3]

⚠️ JSON integers come back as float64. Because the value is parsed from JSON into map[string]any, every number is a float64. So GetValueByJPath("n") returns dataType == "float64" and val == float64(5), not an int. Only values that were set programmatically as Go int/[]int report as "int". Always check dataType before asserting, or accept both.

5. Writing values

There are two styles: a receiver method (mutates in place) and free functions (return a new Typ, leaving the input untouched).

A. Receiver — SetValueByJPath (mutates the object)

func (j *Typ) SetValueByJPath(path string, valueToSet any) error

jo, _ := ToJsonData(`{"str":"old","arrInts":[1,2,3]}`)
_ = jo.SetValueByJPath("str", "new")        // str -> "new"
_ = jo.SetValueByJPath("arrInts.[2]", 99)    // arrInts -> [1,2,99]
_ = jo.SetValueByJPath("arrInts.[]", 4)      // arrInts -> [1,2,3,4]  (append)

B. Free functions (return a new Typ)

func SetValueInJsonDataByJPath(obj Typ, path string, valueToSet any) (Typ, error)
func SetValueAndOverrideInJsonDataByJPath(obj Typ, path string, valueToSet any, override bool) (Typ, error)

• SetValueInJsonDataByJPath is a thin wrapper around SetValueAndOverrideInJsonDataByJPath(..., override=false).
• The override flag controls what happens when a path segment does not exist yet or has an incompatible type (e.g. replacing an object with an array, or extending a non-existent nested path).

override = false vs override = true

Creating a brand-new nested path (requires override = true)

jo, _ := ToJsonData(`{"obj":{"a":1}}`)

// "obj.b.c.d" does not exist yet — this would ERROR with override=false
newJo, err := SetValueAndOverrideInJsonDataByJPath(jo, "obj.b.c.d", 42, true)
if err != nil { /* ... */ }

_, v, e := newJo.GetValueByJPath("obj.b.c.d")
_ = v.(int) // 42

Appending to an array (works in both modes when the array exists)

jo, _ := ToJsonData(`{"tags":["go"]}`)
_, _ = jo.SetValueByJPath("tags.[]", "json")   // -> ["go","json"]

Setting an index that is beyond the current length

When override = true, the missing intermediate nodes are created; a deep index like .[3] will create the array long enough to hold index 3 (zero-filling the gaps). This is what powers the "create non-existing object with array" tests.

jo, _ := ToJsonData(`{"o":{"x":1}}`)
// "o" is an object, but we ask for o.[2] — override turns it into an array
newJo, _ := SetValueAndOverrideInJsonDataByJPath(jo, "o.[2]", 10.0, true)

Type strictness is off

SetValueByJPath does not check that the new value's type matches the existing one. You can set a string where an int used to be, and it just works — the old value is replaced. The tests deliberately exercise this ("bool1" ← a string, etc.).

This is again, one of the core reasons why this library was created. Because JSON data is extremely flexible and things like these can't be easily done using typical Go semantics.

Setting from a Go struct

If valueToSet is a Go struct, it is automatically marshalled to a map[string]any and set in place:

type Address struct {
    City    string `json:"city"`
    ZipCode int    `json:"zipCode"`
}
_ = jo.SetValueByJPath("user.address", Address{City: "Bengaluru", ZipCode: 560001})

6. Top-level JSON arrays

The type is fundamentally object-oriented — it cannot natively represent a document that starts with [. To work around this, when the source JSON is a top-level array, it is wrapped internally under a private synthetic key (topLevelArrayKey). From the outside, the API behaves as if the array is at the root:

jo, _ := ToJsonData(`[1,2,3,4]`)

// Read by index — the array is treated as the root
_, val, err := jo.GetValueByJPath("[2]")     // val == 3.0 (float64)

// Overwrite an element
_ = jo.SetValueByJPath("[2]", "hello")       // -> [1,2,"hello",4]

// Append to the root array
_ = jo.SetValueByJPath("[]", "world")        // -> [1,2,"hello",4,"world"]

jo.HasTopLevelArray()                        // true

• HasTopLevelArray() reports whether the current object wraps a top-level array.
• SetNewTopLevelElement refuses to add a named key to a top-level-array object (there's no sensible place to put it) and returns an error.

7. Serialisation & string methods

jo, _ := ToJsonData(`{"a":1}`)
fmt.Println(jo.String())            // {"a":1}
fmt.Println(jo.StringOrBlankObject()) // {"a":1} (or {} if invalid)

8. Using Typ with database/sql

Typ implements the two interfaces database/sql needs, so it can be a field type in a DB model and used directly with database/sql or sqlx:

• Value() (driver.Value, error) — driver.Valuer. Returns the JSON bytes to store. An invalid object returns SQL NULL; a valid empty object returns {}.
• Scan(value any) error — sql.Scanner. Decodes a DB value (string or []byte, or nil) into the Typ. A nil DB value yields an invalid Typ.

// In a model:
type Event struct {
    ID       int64
    MoreData jsonData.Typ   // json/jsonb column
}

// Scanning a row:
var e Event
err := row.Scan(&e.ID, &e.MoreData)   // MoreData is populated via Scan

// Later, reading a field out of the stored JSON:
_, val, appErr := e.MoreData.GetValueByJPath("settings.theme")
if appErr.IsNotBlank() {
    // key missing or path bad
}

// Writing back: the driver calls Value() and sends proper JSON, not an
// escaped string — this is one of the biggest reasons the type exists.

Because Scan/Value round-trip through json, integers read from the DB come back as float64 (see caveats).

Advantage: You can very much use this type against a string based column (such as TEXT or VARCHAR) if you have JSON data in it. It is especially useful if you are using a database like SQLite that does not have a strict JSON type.

9. JSON marshal / unmarshal

Typ implements json.Marshaler and json.Unmarshaler, so it embeds cleanly into other structs that are themselves JSON-encoded:

func (j Typ) MarshalJSON() ([]byte, error)  // -> null when !Valid; the JSON when valid
func (j *Typ) UnmarshalJSON(data []byte) error

• MarshalJSON on an invalid object emits null.
• UnmarshalJSON of null produces an invalid Typ (where Valid is set to false); of a JSON object produces a valid one; of a bare scalar/array (other than an object) returns an error.

// Embedding into a larger payload that is then json-encoded:
payload := struct {
    Name   string
    Extra  jsonData.Typ `json:"extra"`
}{Name: "x", Extra: /* a Typ */}
b, _ := json.Marshal(payload)   // Extra is emitted as raw JSON, not a string

10. Important caveats

Read these before you build on this library.

1. Numbers from JSON are float64. Any value that came from parsing JSON (DB, ToJsonData on a string) — including integers — is a float64. Check the returned dataType or accept float64 when asserting.

2. GetValueByJPath returns references, not copies. The returned []any / map[string]any is the same underlying data as inside the Typ. Mutating the returned value mutates the Typ.

3. Not goroutine-safe. Typ wraps a map and is used with database/sql across goroutines. There is a commented-out sync.Mutex in the struct and goroutine safe behavior is planned for future, but is not avaialable right now. If you share one Typ across goroutines, add locking or hand out fresh copies yourself.

4. Top-level arrays are a special case. The type cannot natively represent a document that starts with [; it is wrapped under a synthetic key. See section 6.

5. NewJsonData silently swallows blank keys. This is because it is assumed that someone creating a new JSON Object would likely have a name for their key. We will probably add checks or return an error in a future version.

6. No key escaping. Keys cannot contain ., [, ] in a way the JPath can address.

11. Quick recipes

// 1. Build one from a string
jo, err := ToJsonData(`{"user":{"name":"vaibhav","roles":["admin"]}}`)
if err != nil { /* ... */ }

// 2. Read a value
typ, val, e := jo.GetValueByJPath("user.name")
if e.IsNotBlank() {
    // missing / bad path
}
name := val.(string) // "vaibhav"

// 3. Read a nested array element
_, roles, e := jo.GetValueByJPath("user.roles")
if e.IsBlank() {
    roleList := roles.([]any)
    // ["admin"]
}

// 4. Update an existing value (mutates jo)
_ = jo.SetValueByJPath("user.name", "archana")

// 5. Append to an array
_ = jo.SetValueByJPath("user.roles.[]", "editor")   // ["admin","editor"]

// 6. Create a deep, non-existent path (needs override)
newJo, err := SetValueAndOverrideInJsonDataByJPath(jo, "user.meta.a.b.c", 1, true)
if err != nil { /* ... */ }

// 7. Fill a Go struct out of a Typ
var user struct {
    Name string `json:"name"`
}
err = FillFromJsonData(jo, &user)

// 8. Get compact JSON for an HTTP response
body := jo.AsByteSlice()   // proper JSON, not an escaped string