Documentation
¶
Overview ¶
Package parserkit loads a JSON list from one or more sources, in priority order, with either fallback or merge semantics.
Layout ¶
The root package depends on nothing outside the standard library. Everything that needs a third-party client lives in a subpackage, so a program links only what it actually loads from:
parserkit the loader, the file source, stdlib only parserkit/redissource a Redis source (go-redis) parserkit/remotesource an HTTP source (http-kit)
Importing the root package alone costs nothing beyond the standard library. A service that reads its rules from a file on disk does not link go-redis, http-kit or OpenTelemetry, and none of them appear in its go.sum.
Sources ¶
A source is anything that can produce bytes: a Fetcher. The root package ships File; the subpackages ship the rest, and so can you.
loader, err := parserkit.NewLoader[Rule](nil)
rules, err := loader.Load(ctx,
parserkit.At(0, parserkit.File("/etc/app/rules.json")),
parserkit.At(1, redissource.New(rdb, "app:rules")),
)
Priority is the order the loader tries sources in -- lower first. What it does with each result is LoadStrategy: LoadStrategyFallback returns the first source that yields data, LoadStrategyMerge combines them all and deduplicates with LoadOptions.KeyFunc.
Writing a fetcher ¶
A Fetcher returns the raw bytes of one source and enforces the byte budget it is handed. ReadLimited does the second part correctly -- it reads one byte past the limit so that an oversized source is reported as ErrSourceTooLarge rather than silently truncated into a JSON syntax error. Decoding, normalization, priority and merge semantics are the loader's job, not the fetcher's.
Example ¶
Load reads from the highest-priority source that yields data.
package main
import (
"context"
"fmt"
"os"
"path/filepath"
parserkit "github.com/soulteary/parser-kit/v2"
)
type Rule struct {
ID string `json:"id"`
Action string `json:"action"`
}
func main() {
dir, _ := os.MkdirTemp("", "parserkit-example-*")
defer func() { _ = os.RemoveAll(dir) }()
path := filepath.Join(dir, "rules.json")
_ = os.WriteFile(path, []byte(`[{"id":"r1","action":"allow"}]`), 0o600)
loader, err := parserkit.NewLoader[Rule](nil)
if err != nil {
panic(err)
}
rules, err := loader.Load(context.Background(),
parserkit.At(0, parserkit.File(path)),
)
if err != nil {
panic(err)
}
fmt.Println(len(rules), rules[0].ID, rules[0].Action)
}
Output: 1 r1 allow
Index ¶
Examples ¶
Constants ¶
const DefaultMaxBytes int64 = 10 * 1024 * 1024 // 10MB
DefaultMaxBytes is the byte budget a source gets when LoadOptions.MaxBytes is left unset.
Variables ¶
var ErrNoFetcher = errors.New("parserkit: source has no fetcher")
ErrNoFetcher reports a Source with no Fetcher in it, which is a programming error rather than a source that failed to load.
var ErrSourceTooLarge = errors.New("parserkit: source exceeds MaxBytes")
ErrSourceTooLarge reports that a source exceeded LoadOptions.MaxBytes.
It exists so the overrun is distinguishable from malformed content: io.LimitReader truncates silently, so an oversized file previously surfaced as a JSON parse error.
Functions ¶
func ReadLimited ¶
ReadLimited reads r into memory, refusing to return more than maxBytes.
It reads one byte past the limit, because io.LimitReader truncates silently: without the extra byte an oversized source comes back as a JSON syntax error, indistinguishable from genuinely malformed content. An overrun is reported as an error wrapping ErrSourceTooLarge.
maxBytes of zero or less means DefaultMaxBytes, so a Fetcher used on its own still has a bound. Callers should add their own context:
raw, err := parserkit.ReadLimited(resp.Body, maxBytes)
if err != nil {
return nil, fmt.Errorf("response from %q: %w", url, err)
}
Types ¶
type BytesFetcher ¶
type BytesFetcher []byte
BytesFetcher serves a payload already in memory. It is what tests and callers that fetched the bytes themselves reach for.
type DataLoader ¶
type DataLoader[T any] interface { // LoadOne loads data from a single source, applying the byte limit and // the normalization function but none of the priority or merge handling. LoadOne(ctx context.Context, fetcher Fetcher) ([]T, error) // Load loads data from multiple sources. // Sources are processed in priority order (lower number = higher priority). // Behavior depends on LoadStrategy: // - LoadStrategyFallback: returns data from the first successful source // - LoadStrategyMerge: merges data from all successful sources with deduplication Load(ctx context.Context, sources ...Source) ([]T, error) }
DataLoader loads a JSON list of T from one or more sources.
func NewLoader ¶
func NewLoader[T any](opts *LoadOptions[T]) (DataLoader[T], error)
NewLoader creates a new generic data loader
func NewLoaderWithNormalize ¶
func NewLoaderWithNormalize[T any](opts *LoadOptions[T], normalizeFunc NormalizeFunc[T]) (DataLoader[T], error)
NewLoaderWithNormalize creates a new generic data loader with normalization function
type Fetcher ¶
Fetcher retrieves the raw, still-encoded bytes of a single source.
It is the whole extension point: a Redis key, an HTTP endpoint, an S3 object or a test fixture are all just Fetchers, and the loader does not need to know which. The root package ships File; parserkit/redissource and parserkit/remotesource ship the two that need a third-party client.
maxBytes is LoadOptions.MaxBytes. An implementation must not return more than that, and must report an overrun as an error wrapping ErrSourceTooLarge rather than truncating -- ReadLimited does both. The loader re-checks the length it gets back, so a fetcher that ignores the budget fails loudly instead of quietly.
Returning no bytes and no error means the source is absent -- a file that does not exist, a key that was never set -- and the loader reads that as an empty result, which LoadOptions.AllowEmptyData then governs.
Example ¶
Fetcher is the extension point: a source parser-kit has never heard of works the same as the ones it ships.
package main
import (
"context"
"fmt"
"os"
parserkit "github.com/soulteary/parser-kit/v2"
)
type Rule struct {
ID string `json:"id"`
Action string `json:"action"`
}
// envFetcher reads a source out of the environment. Any type with a Fetch
// method is a source; ReadLimited is not needed for bytes already in memory,
// but the budget still has to be honoured.
type envFetcher string
func (e envFetcher) Fetch(_ context.Context, maxBytes int64) ([]byte, error) {
raw := []byte(os.Getenv(string(e)))
if int64(len(raw)) > maxBytes {
return nil, fmt.Errorf("$%s: %w", string(e), parserkit.ErrSourceTooLarge)
}
return raw, nil
}
func main() {
_ = os.Setenv("APP_RULES", `[{"id":"env","action":"allow"}]`)
defer func() { _ = os.Unsetenv("APP_RULES") }()
loader, err := parserkit.NewLoader[Rule](nil)
if err != nil {
panic(err)
}
rules, err := loader.LoadOne(context.Background(), envFetcher("APP_RULES"))
if err != nil {
panic(err)
}
fmt.Println(rules[0].ID, rules[0].Action)
}
Output: env allow
type FileFetcher ¶
type FileFetcher struct {
// contains filtered or unexported fields
}
FileFetcher reads a source from the local filesystem. Use File to build one.
func (*FileFetcher) AllowMissing ¶
func (f *FileFetcher) AllowMissing() *FileFetcher
AllowMissing makes a file that does not exist report an absent source rather than an error, so the loader moves on to the next source without the missing file showing up as the reason everything failed.
It says nothing about a file that exists and is unreadable: a permission error is still an error.
Example ¶
A missing primary source falls through to the next one instead of failing the whole load.
package main
import (
"context"
"fmt"
parserkit "github.com/soulteary/parser-kit/v2"
)
type Rule struct {
ID string `json:"id"`
Action string `json:"action"`
}
func main() {
loader, err := parserkit.NewLoader[Rule](nil)
if err != nil {
panic(err)
}
rules, err := loader.Load(context.Background(),
parserkit.At(0, parserkit.File("/nonexistent/rules.json").AllowMissing()),
parserkit.At(1, parserkit.BytesFetcher(`[{"id":"builtin","action":"deny"}]`)),
)
if err != nil {
panic(err)
}
fmt.Println(rules[0].ID, rules[0].Action)
}
Output: builtin deny
func (*FileFetcher) Path ¶
func (f *FileFetcher) Path() string
Path returns the path this fetcher reads.
type KeyFunc ¶
KeyFunc extracts a unique key from an item for deduplication. Returns the key and true if the item should be included, false otherwise.
type LoadOptions ¶
type LoadOptions[T any] struct { // MaxBytes limits how much a single source may return (default: // DefaultMaxBytes). It applies to every source, not just files: a Fetcher // is handed this budget and must report ErrSourceTooLarge rather than // return more. MaxBytes int64 // AllowEmptyData if true, a source that succeeds with zero items is a // result rather than a failure (default: false). // // With the default, an empty-but-successful source is treated as a failure // and the next source is tried. That is dangerous for allow/deny lists: // clearing the primary source falls back to a stale copy elsewhere, so // entries that were just deleted come back. Set this when an empty result // is a legitimate state -- which for a policy list it almost always is. AllowEmptyData bool // LoadStrategy determines how to combine data from multiple sources // (default: LoadStrategyFallback). LoadStrategy LoadStrategy // KeyFunc is required when LoadStrategy is LoadStrategyMerge. // It extracts a unique key from each item for deduplication. KeyFunc KeyFunc[T] }
LoadOptions configures the behavior of Load operations.
It is generic in T so that KeyFunc is checked by the compiler. Timeouts, retries and TLS settings are not here: they belong to the source that has them, so they are options on that source -- see parserkit/remotesource.
func DefaultLoadOptions ¶
func DefaultLoadOptions[T any]() *LoadOptions[T]
DefaultLoadOptions returns default load options
type LoadStrategy ¶
type LoadStrategy string
LoadStrategy determines how data from multiple sources should be combined
const ( // LoadStrategyFallback returns data from the first successful source (default) LoadStrategyFallback LoadStrategy = "fallback" // LoadStrategyMerge merges data from all successful sources with deduplication LoadStrategyMerge LoadStrategy = "merge" )
type NormalizeFunc ¶
type NormalizeFunc[T any] func([]T) []T
NormalizeFunc is a function type for normalizing data after parsing. It accepts a slice of any type and returns a normalized slice.
type Source ¶
type Source struct {
// Fetcher retrieves the bytes.
Fetcher Fetcher
// Priority orders the source against the others in one Load call.
// Lower number = higher priority (0 is highest). Sources sharing a
// priority keep the order they were passed in.
Priority int
}
Source is a Fetcher with a priority attached.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package redissource loads parser-kit data from a Redis key.
|
Package redissource loads parser-kit data from a Redis key. |
|
Package remotesource loads parser-kit data from an HTTP endpoint.
|
Package remotesource loads parser-kit data from an HTTP endpoint. |