jsonData

jsonData — Developer Usage Guide

A small, dependency-light Go library for reading and writing dynamic JSON documents. The centrepiece is the Typ struct (to be read as type making the package look like jsonData.Typ and sound like JSON Data Type), a thin, type-aware wrapper around map[string]any (aliased as StringAnyMap).

The library exists to solve one specific pain point: working with dynamic JSON content in Go.

---

Table of contents

1. Why does it exist
2. The Typ type
3. Creating a Typ
4. JPath syntax
5. Reading values — GetValueByJPath
6. Writing values
7. Top-level JSON arrays
8. Serialisation & string methods
9. Using Typ with database/sql
10. JSON marshal / unmarshal
11. Important caveats
12. Quick recipes
13. Edge cases & test coverage
14. Benchmark reference
15. Building, testing & benchmarking

---

Why does this library exist

Go is a strictly typed language, while JSON is a very flexible, and rather lightly structured data type.

For example, an array is always homogenous in Go. It is not so in JSON. The following is a valid JSON array:

[1, 2, "vaibhav", {"city": "Ranchi"}]

But you cannot have any type in Go to handle this automatically using the standard Marshal and Unmarshal functions from Go's stdlib.

In addition, if you are dealing with a JSON documents with varying structure then you cannot have a concrete struct into which you can Unmarshal a JSON text. The option that remains is to cast it to either an []any type or a map[string]any type. However, in both the cases, writing reliable code to navigate those entities is much harder and prone to errors; that is if you are ready to deal with how ugly that looks, especially for deeply nested objects.

Despite all this, JSON is part and parcle of modern development. Web apps, mobile apps, databases, caches - all of varying type - they all support JSON in one form or another. This library alleviates the pain of dealing with JSON in go.

This library provides utility functions to deal with JSON data from APIs, and can also deal with database json/jsonb columns that otherwise might come back from the DB driver as strings. If you send that string straight out of a REST API, every quote and newline gets escaped (\", \\n) and clients (Postman, test suites, etc.) have a hard time parsing it. Typ keeps the data as a real in-memory structure and re-encodes it cleanly on the way out.

The 3 primary pain points that it deals with:

1. Clean handling of JSON/JSONB columns in databases (as a database/sql driver) without writing custom logic.
2. Getting a value from JSON using a single path expression (called JPath, explained later).
3. Setting a scalar value into JSON using a single path expression (using JPath).

---

1. The Typ type

type (
    StringAnyMap map[string]any
    Typ          struct {
        Valid            bool
        hasTopLevelArray bool
        StringAnyMap
    }
)

• StringAnyMap — the underlying map holding the parsed JSON. It is embedded, so a Typ is a map[string]any plus metadata.
• Valid — false means the object is "null"/empty/invalid. true means it holds real JSON.
• hasTopLevelArray — internal flag set when the source JSON is a top-level array (e.g. [1,2,3,4]). See section 6.

A Typ is value-passable, but because it embeds a map (a reference type), be aware of aliasing — see caveats.

The well-known invalid singleton is:

var NullJsonData Typ   // Valid == false, empty map

⚠️ Do not treat NullJsonData as a reusable template: mutating a Typ through it mutates shared state. Create fresh objects instead (below).

---

2. Creating a Typ

From any value — ToJsonData

The most common entry point. Accepts a JSON string, a []byte, or any Go value (struct, map, slice…) and returns a Typ.

// From a JSON string
jo, err := ToJsonData(`{"name": "vaibhav", "age": 30, "tags": ["go", "json"]}`)
if err != nil {
    // handle
}

// From a Go struct — it is marshalled to JSON first, then re-parsed
type User struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}
jo, err = ToJsonData(User{Name: "Kaushal", Age: 28})

From a single key/value — NewJsonData

jo := NewJsonData("name", "Vaibhav")
// -> {"name":"Vaibhav"}

⚠️ If the key is blank (or only whitespace), NewJsonData silently returns an empty (but valid) object instead of an error. Prefer ToJsonData when you might pass a blank key

A fresh, empty, valid object — EmptyNotNullJsonData

jo := EmptyNotNullJsonData()
// -> a valid Typ wrapping an empty map  ({}), safe to mutate in any goroutine

EmptyNotNullJsonData is a function, not a global var, on purpose: it hands out a brand-new map every call so concurrent goroutines never share/alias the same underlying map.

---

3. JPath syntax

A JPath is a dotted path that navigates the JSON document. It is not JSONPath — it is this library's own, simpler expression language.

Syntax	Meaning	Example	Resolves to
key	a top-level key	str	the value at str
a.b.c	nested keys	user.settings.reachout.email	nested value
arr.[i]	array element by index i	favFoods.[1]	2nd element
a.[i].b	array element then a key	favFoods.[1].name	"Biryani"
a.[i].[j]	nested array indices	profile.[0].deepMetrics.[2].[3].[1].[2]	deep value
[]	append a value to an array (must be last segment)	nicknames.[]	append
[i] (no key before)	index into a top-level array	[2]	3rd element

Rules & limits

• Segment separator is .. A key that itself contains . cannot be addressed (there is no escaping).
• Array indices are zero-based, non-negative integers only. [-1], [1.5], [abc], [ ] (blank) are all rejected.
• [] (append) is only valid as the last segment of a path and only for append operation.
• Malformed paths return an error, they do not panic:
  • leading/trailing . or .. (empty segment)
  • [ without a closing ]
  • non-numeric index inside [ ]
• Validate a path string on its own with ValidateJPathSyntax(path) error before using it (e.g. for user-supplied paths).

if err := ValidateJPathSyntax("arrObjs.[2].objName"); err != nil {
    // path is syntactically bad
}

---

4. Reading values — GetValueByJPath

func (j *Typ) GetValueByJPath(path string) (dataType string, value any, err appError.Typ)

Returns three values:

1. dataType — a string naming the detected type (one of the Type* constants below).
2. value — the raw value as any.
3. err — an appError.Typ. If err.IsNotBlank() (i.e. not blank), an error occurred and the first two values must be ignored.

Detecting success

typ, val, err := jo.GetValueByJPath("obj.nestedObj.child0.lvl")
if err.IsNotBlank() {
    // path invalid / element not found / wrong shape — handle it
    return
}
// safe to use typ + val

Data-type → Go-type mapping

The value you get back should be type-asserted according to dataType:

dataType (returned string)	Constant	Underlying Go type of value
"string"	TypeString	string
"int"	TypeInt	int
"float64"	TypeFloat64	float64
"bool"	TypeBool	bool
"nil"	TypeNil	nil
"object"	TypeObject	map[string]any
"array/any"	TypeArrayAny	[]any
"array/int"	TypeArrayInt	[]int
"array/float64"	TypeArrayFloat64	[]float64
"array/string"	TypeArrayString	[]string
"array/bool"	TypeArrayBool	[]bool
"array/object"	TypeArrayObject	[]map[string]any
"any"	TypeAny	any
"unknown"	TypeUnknown	(edge case)

Examples

jo, _ := ToJsonData(`{"str":"x","n":5,"arr":[1,2,3],"obj":{"a":"b"}}`)

// A string
typ, val, err := jo.GetValueByJPath("str")   // typ == "string"
s := val.(string)                            // "x"

// A nested key
_, nested, err := jo.GetValueByJPath("obj.a")  // "b"

// An array element
typ, val, err = jo.GetValueByJPath("arr.[1]")  // typ == "float64" (see caveat)
arr1 := val.()                           // [1 2 3]

⚠️ JSON integers come back as float64. Because the value is parsed from JSON into map[string]any, every number is a float64. So GetValueByJPath("n") returns dataType == "float64" and val == float64(5), not an int. Only values that were set programmatically as Go int/[]int report as "int". Always check dataType before asserting, or accept both.

---

5. Writing values

There are two styles: a receiver method (mutates in place) and free functions (return a new Typ, leaving the input untouched).

A. Receiver — SetValueByJPath (mutates the object)

func (j *Typ) SetValueByJPath(path string, valueToSet any, override ...bool) error

The variadic override flag (default true) controls what happens when a path segment does not exist yet or has an incompatible shape:

• true (default, and the historical behaviour) — create the missing intermediate objects/arrays and override mismatched shapes.
• false — return an error instead (strict mode).

override is variadic purely for backward compatibility, so existing SetValueByJPath(path, value) callers are unaffected.

jo, _ := ToJsonData(`{"str":"old","arrInts":[1,2,3]}`)
_ = jo.SetValueByJPath("str", "new")        // str -> "new"
_ = jo.SetValueByJPath("arrInts.[2]", 99)    // arrInts -> [1,2,99]
_ = jo.SetValueByJPath("arrInts.[]", 4)      // arrInts -> [1,2,3,4]  (append)

B. Free function — SetJsonDataByJPath (returns a new Typ)

func SetJsonDataByJPath(obj Typ, path string, valueToSet any, override bool) (Typ, error)

SetJsonDataByJPath is the non-mutating counterpart to the receiver: it returns a new Typ and leaves the input untouched. Both entry points delegate to the same private core, applySetByJPath.

• The override flag controls what happens when a path segment does not exist yet or has an incompatible type (e.g. replacing an object with an array, or extending a non-existent nested path).

override = false vs override = true

Situation	override = false	override = true
Setting an existing key/value	✅ update	✅ update
Appending to an existing array ([])	✅ append	✅ append
A key in the middle of the path does not exist	❌ returns an error	✅ creates the missing intermediate object/array
A node is the wrong shape (e.g. object where an array index is used)	❌ returns an error	✅ overrides it with the new shape

Creating a brand-new nested path (requires override = true)

jo, _ := ToJsonData(`{"obj":{"a":1}}`)

// "obj.b.c.d" does not exist yet — this would ERROR with override=false
newJo, err := SetJsonDataByJPath(jo, "obj.b.c.d", 42, true)
if err != nil { /* ... */ }

_, v, e := newJo.GetValueByJPath("obj.b.c.d")
_ = v.(int) // 42

Appending to an array (works in both modes when the array exists)

jo, _ := ToJsonData(`{"tags":["go"]}`)
_, _ = jo.SetValueByJPath("tags.[]", "json")   // -> ["go","json"]

Setting an index that is beyond the current length

When override = true, the missing intermediate nodes are created; a deep index like .[3] will create the array long enough to hold index 3 (zero-filling the gaps). This is what powers the "create non-existing object with array" tests.

jo, _ := ToJsonData(`{"o":{"x":1}}`)
// "o" is an object, but we ask for o.[2] — override turns it into an array
newJo, _ := SetJsonDataByJPath(jo, "o.[2]", 10.0, true)

Type strictness is off

SetValueByJPath does not check that the new value's type matches the existing one. You can set a string where an int used to be, and it just works — the old value is replaced. The tests deliberately exercise this ("bool1" ← a string, etc.).

This is again, one of the core reasons why this library was created. Because JSON data is extremely flexible and things like these can't be easily done using typical Go semantics.

Setting from a Go struct

If valueToSet is a Go struct, it is automatically marshalled to a map[string]any and set in place:

type Address struct {
    City    string `json:"city"`
    ZipCode int    `json:"zipCode"`
}
_ = jo.SetValueByJPath("user.address", Address{City: "Bengaluru", ZipCode: 560001})

---

6. Top-level JSON arrays

The type is fundamentally object-oriented — it cannot natively represent a document that starts with [. To work around this, when the source JSON is a top-level array, it is wrapped internally under a private synthetic key (topLevelArrayKey). From the outside, the API behaves as if the array is at the root:

jo, _ := ToJsonData(`[1,2,3,4]`)

// Read by index — the array is treated as the root
_, val, err := jo.GetValueByJPath("[2]")     // val == 3.0 (float64)

// Overwrite an element
_ = jo.SetValueByJPath("[2]", "hello")       // -> [1,2,"hello",4]

// Append to the root array
_ = jo.SetValueByJPath("[]", "world")        // -> [1,2,"hello",4,"world"]

jo.HasTopLevelArray()                        // true

• HasTopLevelArray() reports whether the current object wraps a top-level array.
• SetNewTopLevelElement refuses to add a named key to a top-level-array object (there's no sensible place to put it) and returns an error.

---

7. Serialisation & string methods

Method	Returns	Notes
String()	compact JSON string	"" when !Valid
PrettyString()	JSON string formatted	"" when !Valid
StringOrBlankObject()	"{}" when !Valid, else JSON	used by the generator
StringOrNil()	*string (JSON) or nil	nil when !Valid
AsByteSlice()	[]byte of String()	convenience for HTTP bodies

jo, _ := ToJsonData(`{"a":1}`)
fmt.Println(jo.String())            // {"a":1}
fmt.Println(jo.StringOrBlankObject()) // {"a":1} (or {} if invalid)

---

8. Using Typ with database/sql

Typ implements the two interfaces database/sql needs, so it can be a field type in a DB model and used directly with database/sql or sqlx:

• Value() (driver.Value, error) — driver.Valuer. Returns the JSON bytes to store. An invalid object returns SQL NULL; a valid empty object returns {}.
• Scan(value any) error — sql.Scanner. Decodes a DB value (string or []byte, or nil) into the Typ. A nil DB value yields an invalid Typ.

// In a model:
type Event struct {
    ID       int64
    MoreData jsonData.Typ   // json/jsonb column
}

// Scanning a row:
var e Event
err := row.Scan(&e.ID, &e.MoreData)   // MoreData is populated via Scan

// Later, reading a field out of the stored JSON:
_, val, appErr := e.MoreData.GetValueByJPath("settings.theme")
if appErr.IsNotBlank() {
    // key missing or path bad
}

// Writing back: the driver calls Value() and sends proper JSON, not an
// escaped string — this is one of the biggest reasons the type exists.

Because Scan/Value round-trip through json, integers read from the DB come back as float64 (see caveats).

Advantage: You can very much use this type against a string based column (such as TEXT or VARCHAR) if you have JSON data in it. It is especially useful if you are using a database like SQLite that does not have a strict JSON type.

---

9. JSON marshal / unmarshal

Typ implements json.Marshaler and json.Unmarshaler, so it embeds cleanly into other structs that are themselves JSON-encoded:

func (j Typ) MarshalJSON() ([]byte, error)  // -> null when !Valid; the JSON when valid
func (j *Typ) UnmarshalJSON(data []byte) error

• MarshalJSON on an invalid object emits null.
• UnmarshalJSON of null produces an invalid Typ (where Valid is set to false); of a JSON object produces a valid one; of a bare scalar/array (other than an object) returns an error.

// Embedding into a larger payload that is then json-encoded:
payload := struct {
    Name   string
    Extra  jsonData.Typ `json:"extra"`
}{Name: "x", Extra: /* a Typ */}
b, _ := json.Marshal(payload)   // Extra is emitted as raw JSON, not a string

---

10. Important caveats

Read these before you build on this library.

1. Numbers from JSON are float64. Any value that came from parsing JSON (DB, ToJsonData on a string) — including integers — is a float64. Check the returned dataType or accept float64 when asserting.

2. GetValueByJPath returns references, not copies. The returned []any / map[string]any is the same underlying data as inside the Typ. Mutating the returned value mutates the Typ.

3. Not goroutine-safe. Typ wraps a map and is used with database/sql across goroutines. There is a commented-out sync.Mutex in the struct and goroutine safe behavior is planned for future, but is not avaialable right now. If you share one Typ across goroutines, add locking or hand out fresh copies yourself.

4. Top-level arrays are a special case. The type cannot natively represent a document that starts with [; it is wrapped under a synthetic key. See section 6.

5. NewJsonData silently swallows blank keys. This is because it is assumed that someone creating a new JSON Object would likely have a name for their key. We will probably add checks or return an error in a future version.

6. No key escaping. Keys cannot contain ., [, ] in a way the JPath can address.

---

11. Quick recipes

// 1. Build one from a string
jo, err := ToJsonData(`{"user":{"name":"vaibhav","roles":["admin"]}}`)
if err != nil { /* ... */ }

// 2. Read a value
typ, val, e := jo.GetValueByJPath("user.name")
if e.IsNotBlank() {
    // missing / bad path
}
name := val.(string) // "vaibhav"

// 3. Read a nested array element
_, roles, e := jo.GetValueByJPath("user.roles")
if e.IsBlank() {
    roleList := roles.([]any)
    // ["admin"]
}

// 4. Update an existing value (mutates jo)
_ = jo.SetValueByJPath("user.name", "archana")

// 5. Append to an array
_ = jo.SetValueByJPath("user.roles.[]", "editor")   // ["admin","editor"]

// 6. Create a deep, non-existent path (needs override)
newJo, err := SetJsonDataByJPath(jo, "user.meta.a.b.c", 1, true)
if err != nil { /* ... */ }

// 7. Fill a Go struct out of a Typ
var user struct {
    Name string `json:"name"`
}
err = FillFromJsonData(jo, &user)

// 8. Get compact JSON for an HTTP response
body := jo.AsByteSlice()    // proper JSON, not an escaped string

---

12. Edge cases & test coverage

The following behaviours are locked in by the test suite in json_data_edge_test.go. They are the "unhappy paths" and special cases a caller should be aware of. All of them are verified to return an error (or a non-blank appError.Typ) rather than panic.

Malformed JPaths are rejected, never panic

Path	Why it is invalid
arrInts.[-1]	negative array index (only non-negative integers are allowed)
arrInts.[1.5]	decimal index — the . also splits the path into a malformed [1 segment
arrInts.[abc]	non-numeric index
.str	leading . → empty first segment
str.	trailing . → empty last segment
obj..nestedObj	double . → empty middle segment
""	empty path
arrObjs.[][0]	[] (append) used as a non-last segment

The append token [] is valid only as the last segment of a path. Using it anywhere else — on either the read path (GetValueByJPath) or the write path (SetValueByJPath) — returns an error.

Setting nil at a leaf

SetValueByJPath("str", nil) turns that key into JSON null. Reading it back yields dataType == "nil" (TypeNil) with a nil value and a blank error.

The free function does not understand top-level arrays

Only the receiver method (SetValueByJPath) and the reader (GetValueByJPath) are aware of top-level arrays (they prepend the synthetic topLevelArrayKey). The free function is not:

jo, _ := ToJsonData(`[1,2,3,4]`)
_, err := SetJsonDataByJPath(jo, "[2]", 99, false) // err != nil
// -> "Cannot treat entity of type object as array"

Use the receiver method for top-level-array writes.

UnmarshalJSON

Input	Result
{"a":1} (object)	valid Typ, no error
null	invalid Typ (Valid == false), no error
[1,2,3] (bare array)	error
42 (bare scalar)	error

FillFromJsonData

Returns an error (never panics) for:

• a non-pointer argument (e.g. a struct value),
• a nil (typed) pointer, and
• a pointer to a non-struct (e.g. *int).

Keys that contain non-. characters

Because . is the only JPath separator and there is no escaping, a key containing other characters (e.g. a/b) is treated as a single, addressable segment. SetJsonDataByJPath(jo, "a/b", 5, true) followed by GetValueByJPath("a/b") round-trips to 5.

---

13. Benchmark reference

Benchmarks live in json_data_bench_test.go. Run them with:

go test -run 'xxxNoTestxxx' -bench 'Benchmark' -benchmem .

Every benchmark calls b.ReportAllocs(), so the report includes ns/op, B/op and allocs/op. Read-only benchmarks (get / string / value / marshal) parse once outside the loop; mutating benchmarks (set) re-parse inside the loop, because a Typ embeds a map (a reference type) and reusing one instance across iterations would alias the same underlying map.

Benchmark	What it measures
BenchmarkToJsonData	parsing a JSON object string into a Typ
BenchmarkToJsonDataTopLevelArray	parsing a top-level array document
BenchmarkGetValueByJPath	deep nested read (objects + arrays)
BenchmarkGetValueByJPathTopLevelArray	deep read inside a top-level array
BenchmarkSetValueByJPath	receiver set (mutates in place)
BenchmarkSetJsonDataByJPath	non-mutating set that creates a deep, non-existing path
BenchmarkString	compact JSON serialisation
BenchmarkMarshalJSON	json.Marshaler path
BenchmarkScan	sql.Scanner (DB value → Typ)
BenchmarkValue	driver.Valuer (Typ → DB value)

Illustrative numbers (Apple M1, Go 1.27.1 — treat as relative, not absolute):

BenchmarkToJsonData                              ~29.8 us/op   ~16.6 KB/op   ~501 allocs/op
BenchmarkGetValueByJPath                          ~0.25 us/op     120 B/op      2 allocs/op
BenchmarkSetValueByJPath                          ~34.1 us/op   ~21.4 KB/op   ~514 allocs/op
BenchmarkSetJsonDataByJPath                       ~32.1 us/op     ~23.0 KB/op     ~523 allocs/op
BenchmarkString                                   ~10.0 us/op    2.7 KB/op     27 allocs/op
BenchmarkMarshalJSON                              ~ 9.5 us/op    1.5 KB/op     26 allocs/op
BenchmarkScan                                     ~31.4 us/op   ~16.6 KB/op   ~500 allocs/op
BenchmarkValue                                    ~ 9.6 us/op    1.6 KB/op     27 allocs/op

The parse / scan / set paths are the expensive ones (they rebuild the in-memory tree), while reads and serialisation are comparatively cheap.

---

14. Building, testing & benchmarking

Local build cache (important). The Go build cache defaults to a directory outside the project (e.g. ~/Library/Caches/go-build). In a sandboxed or permission-restricted environment that write is denied and the build fails with operation not permitted. Point GOCACHE at a project-local directory instead (per the project convention, use .gocache):

mkdir -p .gocache
export GOCACHE="$PWD/.gocache"
export GOTOOLCHAIN=local    # avoid trying to download a toolchain

With that in place:

# Build
go build ./...

# Run the full unit-test suite
go test -count=1 .

# Run only the new edge-case / deep top-level-array tests
go test -count=1 -run 'TestEdge|TestDeepTopLevelArray' -v .

# Run the benchmarks
go test -run 'xxxNoTestxxx' -bench 'Benchmark' -benchmem .