parserkit

package module
v3.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 21, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

README

Parser Kit

Go Reference Go Report Card License codecov

中文文档

A generic data loader that reads a JSON list from several sources — a local file, Redis, an HTTP endpoint — in priority order, with fallback or merge semantics.

Features

  • Multi-source support: local files, Redis and remote HTTP endpoints, plus anything else you write a Fetcher for
  • Priority-based fallback: fall through to the next source when one fails
  • Merge strategy: combine every source that answered, deduplicated by key
  • Generic design: works with any JSON-serializable type, and LoadOptions[T] makes KeyFunc a compile-time concern
  • You link what you load: the root package depends on nothing outside the standard library
  • Size limits: every source is bounded, and an overrun is reported as ErrSourceTooLarge rather than as malformed JSON
  • HTTP retry: exponential backoff with jitter for remote sources
  • Normalization support: optional post-parse normalization

Layout

Everything that needs a third-party client lives in a subpackage, so a program links only what it actually reads from:

Package Contents Cost
parser-kit/v3 the loader, File, BytesFetcher, Fetcher standard library only
parser-kit/v3/redissource the Redis source go-redis
parser-kit/v3/remotesource the HTTP source http-kit/v2

Measured for a program that loads from a file, v1.8.0 against v2.0.0: the binary goes from 9,699,758 to 3,952,636 bytes, linked packages from 248 to 78, and its go.sum from 20 modules to 2. See CHANGELOG.md for the full table, including what the split costs a program that does use all three sources.

Requirements

  • Go 1.27+ (go.mod declares go 1.27.0)
  • github.com/redis/go-redis/v9 — only if you import redissource
  • github.com/soulteary/http-kit/v2 — only if you import remotesource

Installation

go get github.com/soulteary/parser-kit/v3

Upgrading from v1 or v2? The import path changes for everyone; see Upgrade Notes.

Usage

Basic Example
package main

import (
    "context"

    "github.com/redis/go-redis/v9"
    parserkit "github.com/soulteary/parser-kit/v3"
    "github.com/soulteary/parser-kit/v3/redissource"
    "github.com/soulteary/parser-kit/v3/remotesource"
)

type User struct {
    ID    string `json:"id"`
    Email string `json:"email"`
    Phone string `json:"phone"`
}

func main() {
    loader, err := parserkit.NewLoader[User](nil)
    if err != nil {
        panic(err)
    }

    rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})

    remote, err := remotesource.New("https://api.example.com/users",
        remotesource.WithAuthorization("Bearer token"),
    )
    if err != nil {
        panic(err)
    }

    // Lower priority number = tried first.
    users, err := loader.Load(context.Background(),
        parserkit.At(0, redissource.New(rdb, "users:cache")),
        parserkit.At(1, remote),
        parserkit.At(2, parserkit.File("/path/to/users.json")),
    )
    if err != nil {
        panic(err)
    }

    _ = users
}

A program with no Redis and no remote endpoint imports neither subpackage, and links neither dependency:

loader, _ := parserkit.NewLoader[User](nil)
users, err := loader.Load(ctx,
    parserkit.At(0, parserkit.File("/etc/app/users.json")),
    parserkit.At(1, parserkit.BytesFetcher(builtinDefaults)),
)
Individual Source Loading
users, err := loader.LoadOne(ctx, parserkit.File("/path/to/users.json"))
users, err := loader.LoadOne(ctx, redissource.New(rdb, "users:cache"))
users, err := loader.LoadOne(ctx, remote)
Custom Options
opts := &parserkit.LoadOptions[User]{
    MaxBytes:       20 * 1024 * 1024, // 20MB
    AllowEmptyData: true,
}

loader, err := parserkit.NewLoaderWithNormalize[User](opts, func(users []User) []User {
    for i := range users {
        users[i].Phone = strings.TrimSpace(users[i].Phone)
    }
    return users
})

Timeouts, retries and TLS belong to the source that has them, so they are options on that source rather than on the loader:

remote, err := remotesource.New("https://api.example.com/users",
    remotesource.WithTimeout(10*time.Second),
    remotesource.WithRetry(remotesource.RetryPolicy{
        MaxRetries:    5,
        RetryDelay:    2 * time.Second,
        MaxRetryDelay: 30 * time.Second,
    }),
)

Sources

File Source
parserkit.At(2, parserkit.File("/path/to/data.json"))

// A file that does not exist is an absent source rather than an error, so the
// loader moves on to the next one:
parserkit.At(2, parserkit.File("/path/to/data.json").AllowMissing())
Redis Source

Reads a Redis key holding JSON. The size is checked with STRLEN before the value is read, so an oversized value is refused rather than pulled into memory.

parserkit.At(0, redissource.New(rdb, "data:cache"))
parserkit.At(0, redissource.New(rdb, "data:cache").AllowMissing())

redissource.New takes a redissource.Getter — the two commands the source issues — not a concrete client:

type Getter interface {
    Get(ctx context.Context, key string) *redis.StringCmd
    StrLen(ctx context.Context, key string) *redis.IntCmd
}

*redis.Client, *redis.ClusterClient, *redis.Ring and redis.UniversalClient all satisfy it, so a cluster or Sentinel deployment works with the same source. A nil client — including a nil *redis.Client stored in the interface, which would otherwise panic — is reported as redissource.ErrClientNil.

Remote Source
remote, err := remotesource.New("https://api.example.com/data",
    remotesource.WithAuthorization("Bearer token"),
    remotesource.WithHeader("X-Tenant", "acme"),
    remotesource.WithTimeout(5*time.Second),
)

Each remotesource.Fetcher builds its own HTTP client. To share one connection pool across several remote sources — or to bring your own mTLS, proxy or transport configuration — build the client once and pass it in:

client, err := httpkit.NewClient(&httpkit.Options{
    BaseURL: "https://api.example.com",
    Timeout: 5 * time.Second,
})
a, _ := remotesource.New("https://api.example.com/users", remotesource.WithClient(client))
b, _ := remotesource.New("https://api.example.com/groups", remotesource.WithClient(client))
Security note for remote sources

remotesource uses the URL as given and sends the value from WithAuthorization as an Authorization header. New checks that the URL is an absolute http or https URL and nothing more, so a caller-controlled URL still makes this an SSRF primitive that forwards your credentials to wherever the URL points.

Validate the URL before passing it in, and close the DNS-rebinding window on the dial:

import "github.com/soulteary/cli-kit/validator"

opts := &validator.URLOptions{AllowedSchemes: []string{"https"}}
if err := validator.ValidateURL(remoteURL, opts); err != nil {
    return err
}
// and use validator.SSRFDialControl(opts) on the transport that fetches it,
// passed in with remotesource.WithClient

remotesource.WithInsecureSkipVerify() disables TLS verification entirely — development only.

Writing your own source

A source is anything with a Fetch method. ReadLimited enforces the byte budget the way the built-in sources do — it reads one byte past the limit, so an oversized source is reported as ErrSourceTooLarge instead of being silently truncated into a JSON syntax error.

type s3Source struct {
    bucket, key string
    client      *s3.Client
}

func (s s3Source) Fetch(ctx context.Context, maxBytes int64) ([]byte, error) {
    out, err := s.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: &s.bucket, Key: &s.key,
    })
    if err != nil {
        return nil, err
    }
    defer out.Body.Close()

    raw, err := parserkit.ReadLimited(out.Body, maxBytes)
    if err != nil {
        return nil, fmt.Errorf("s3://%s/%s: %w", s.bucket, s.key, err)
    }
    return raw, nil
}

users, err := loader.Load(ctx, parserkit.At(0, s3Source{...}))

Returning no bytes and no error means the source is absent; the loader reads that as an empty result, which AllowEmptyData then governs.

Priority System

Sources are processed in priority order:

  • Lower priority number = higher priority
  • Priority 0 is the highest priority
  • If a source fails, the loader automatically tries the next source
  • Behavior depends on LoadStrategy (see below)

Sorting is stable, so two sources sharing a priority keep the order you passed them in — the same input always produces the same order.

Load Strategy

Fallback (default)

LoadStrategyFallback: returns data from the first successful source. Use it for cache → remote → file style loading.

Merge

LoadStrategyMerge: merges data from every successful source, deduplicated by KeyFunc. The first source to define a key fixes its position; a later one replaces its value.

opts := parserkit.DefaultLoadOptions[User]()
opts.LoadStrategy = parserkit.LoadStrategyMerge
opts.KeyFunc = func(u User) (string, bool) { return u.Phone, u.Phone != "" } // key, include

loader, err := parserkit.NewLoader[User](opts)
users, err := loader.Load(ctx, sources...)

KeyFunc is KeyFunc[T], so a mismatched signature is a compile error at the call site rather than an error returned from NewLoader.

Options Reference

LoadOptions[T]
Option Default Description
MaxBytes 10MB Max bytes any single source may return
AllowEmptyData false When false, treat an empty source as a failure and try the next
LoadStrategy fallback fallback or merge
KeyFunc nil Required for merge; func(T) (string, bool)
remotesource options
Option Default Description
WithTimeout(d) 5s Bounds one request, retries included
WithRetry(RetryPolicy{MaxRetries}) 3 Retries after a failed request
WithRetry(RetryPolicy{RetryDelay}) 1s Delay before the first retry; doubles from there. Must be positive — see below
WithRetry(RetryPolicy{MaxRetryDelay}) 30s Ceiling on the backoff. Must be positive — see below
WithAuthorization(v) — Authorization header, used verbatim
WithHeader(k, v) — Extra request header; may be repeated
WithUserAgent(ua) — User-Agent for this source's requests
WithInsecureSkipVerify() off Skip TLS verification (dev only)
WithClient(c) — Use an existing http-kit v2 *httpkit.Client instead of building one
WithPropagator(p) none Inject cross-process context — trace headers, baggage, a request ID — into every request; otelprop.Global() for OpenTelemetry

RetryDelay and MaxRetryDelay are not optional in the way a zero value usually is. http-kit computes RetryDelay × 2^attempt and clamps the result to MaxRetryDelay unconditionally, so a zero in either field means every retry fires immediately and a failing remote source is retried as fast as the network allows. New replaces a non-positive value in either field with its default, which is why WithRetry(RetryPolicy{MaxRetries: 5}) is safe to write.

NewLoader and NewLoaderWithNormalize work on a copy of your LoadOptions, so filling in defaults does not mutate the struct you passed.

Error Handling

  • If every source fails, Load returns an error carrying the last one.
  • LoadOne returns its source's error directly.
  • A Source with no Fetcher reports ErrNoFetcher.
  • A missing file or Redis key is an error by default; AllowMissing() on that source makes it an absent source instead, and the loader moves on.
Oversized sources

A source larger than MaxBytes fails with ErrSourceTooLarge, from every source type:

data, err := loader.Load(ctx, sources...)
if errors.Is(err, parserkit.ErrSourceTooLarge) {
    // the source exists and is reachable, but exceeds MaxBytes
}

Match it with errors.Is rather than inspecting the message — an oversized source and a malformed one are different problems with different fixes.

Pass math.MaxInt64 as MaxBytes to mean "no limit"; the one-byte overrun margin saturates rather than overflowing.

An empty source counts as a failure

With AllowEmptyData: false (the default), a source that loads successfully but yields no entries is treated as a failure, and the fallback strategy moves on to the next source.

That is worth thinking about for an allow list or deny list: clearing the primary source falls back to a stale copy in Redis or on a remote, and entries you just deleted come back. Set AllowEmptyData: true when an empty result is a legitimate state.

Testing

Tests do not require a real Redis instance or a real HTTP server. The suite uses miniredis and net/http/httptest, so you can run tests and coverage locally without any external services:

go test ./...
go test -race -coverprofile=coverage.out -covermode=atomic ./...
go tool cover -func=coverage.out

Dependencies

The root package has none beyond the standard library.

  • github.com/redis/go-redis/v9 — used by redissource
  • github.com/soulteary/http-kit/v2 — used by remotesource

Test-only: github.com/alicebob/miniredis/v2 for in-process Redis, and github.com/stretchr/testify.

Both moved dependencies keep their minimum versions in this module's go.mod, and minimum version selection still passes those minimums to anyone who imports the subpackages. What the split removes is the requirement for everyone else.

Upgrade Notes

v3.0.0

The import path changes for every user, including programs that only read files, because Go encodes the major version in the module path:

import parserkit "github.com/soulteary/parser-kit/v3"

remotesource moved from http-kit v1.5.0 to http-kit/v2 v2.0.0, so remotesource.WithClient takes an http-kit v2 *httpkit.Client. Go resolves the two http-kit paths as unrelated modules, so a caller holding a v1 client no longer compiles — and a program that imported remotesource alongside http-kit/v2 no longer links two copies of the same client.

v2 v3
import parserkit "github.com/soulteary/parser-kit/v2" import parserkit "github.com/soulteary/parser-kit/v3"
import httpkit "github.com/soulteary/http-kit" import httpkit "github.com/soulteary/http-kit/v2"
WithClient(c) with a v1 *httpkit.Client WithClient(c) with a v2 *httpkit.Client

That is the only exported signature that changed. The one behaviour change is easy to miss, because it compiles either way:

  • remotesource no longer propagates trace context unless asked. Until http-kit v2 this package called otel.GetTextMapPropagator() on every fetch, so a service with a global OpenTelemetry propagator got trace headers without asking — and every service importing remotesource linked OpenTelemetry whether it traced or not. Nothing errors; the fetches simply stop carrying the headers. One line brings it back:

    import "github.com/soulteary/http-kit/v2/otelprop"
    
    remotesource.New(url, remotesource.WithPropagator(otelprop.Global()))
    

    A caller supplying its own client through WithClient was never affected and still is not: it sets Propagator on that client's httpkit.Options.

See CHANGELOG.md for what the split off OpenTelemetry is worth — six modules and 27 linked packages for a program that does not trace.

v2.0.0

The import path changes for every user, including programs that only read files, because Go encodes the major version in the module path:

import parserkit "github.com/soulteary/parser-kit/v2"

The Redis and HTTP sources moved into subpackages so the root package could stop importing go-redis and http-kit. Deprecated shims were not an option: a shim for FromRedis has to import go-redis, which relinks it and gives the entire benefit back.

v1 v2
loader.FromFile(ctx, path) loader.LoadOne(ctx, parserkit.File(path))
loader.FromRedis(ctx, client, key) loader.LoadOne(ctx, redissource.New(client, key))
loader.FromRemote(ctx, url, auth) loader.LoadOne(ctx, remotesource.New(url, remotesource.WithAuthorization(auth)))
Source{Type: SourceTypeFile, Priority: n, Config: SourceConfig{FilePath: p}} parserkit.At(n, parserkit.File(p))
Source{Type: SourceTypeRedis, …RedisClient: c, RedisKey: k} parserkit.At(n, redissource.New(c, k))
Source{Type: SourceTypeRemote, …RemoteURL: u, AuthorizationHeader: a} parserkit.At(n, remote) from remotesource.New(u, remotesource.WithAuthorization(a))
LoadOptions LoadOptions[T]
DefaultLoadOptions() DefaultLoadOptions[T]()
LoadOptions.KeyFunc interface{} LoadOptions[T].KeyFunc KeyFunc[T]
LoadOptions.MaxFileSize LoadOptions[T].MaxBytes
LoadOptions.AllowEmptyFile parserkit.File(path).AllowMissing()
LoadOptions.HTTPTimeout remotesource.WithTimeout(d)
LoadOptions.MaxRetries / .RetryDelay / .MaxRetryDelay remotesource.WithRetry(remotesource.RetryPolicy{…})
LoadOptions.InsecureSkipVerify remotesource.WithInsecureSkipVerify()

Also in v2, and easy to miss:

  • A zero RetryDelay no longer means "retry immediately". v1.7.0 fixed this for MaxRetryDelay and left the same hole one field over, so &LoadOptions{MaxRetries: 3} retried as fast as the network allowed.
  • A zero-byte source decodes to an empty list rather than "unexpected end of JSON input". With the default AllowEmptyData: false a fallback chain behaves exactly as before.
  • redissource accepts any go-redis client shape. v1 asserted client.(*redis.Client) and rejected a *redis.ClusterClient at run time.
v1.8.0

Dependency refresh only. No API was removed and no call needs rewriting.

  • Test Redis is miniredis v2.39.0 (was v2.36.1).
  • Transitive yuin/gopher-lua is v1.1.2 (was v1.1.1), matching the other kits.
  • Still requires http-kit v1.5.0. That module had no newer published release.
v1.7.0
  • Remote retries back off again. FromRemote never set MaxRetryDelay, and http-kit clamps every computed delay to that ceiling unconditionally — so a ceiling of zero made all three retries fire immediately, turning a failing source into a tight request loop. With the defaults the delays are now 1s, 2s and 4s rather than none at all. A remote source that used to fail fast now takes a few seconds to exhaust its retries; lower RetryDelay or MaxRetries if you depended on the old timing.
  • LoadOptions.MaxRetryDelay is new (default 30s). NewLoader and NewLoaderWithNormalize fill it in when you leave it zero, for the reason above: a zero you pass is replaced rather than honoured, because it means "no backoff", not "no ceiling". Set it explicitly to choose a different ceiling.
v1.6.0

One sentinel was added; nothing was removed. Two error paths report differently.

  • An oversized source is reported as ErrSourceTooLarge, not as malformed data. io.LimitReader truncates silently, so a file or HTTP response larger than MaxFileSize came back as invalid JSON with no way to tell the two apart. Both paths now read one byte past the limit and report the sentinel — matching FromRedis, which already checked STRLEN up front, so the three sources behaved differently for the same condition. If you matched on "invalid JSON" to detect an oversized source, match errors.Is(err, ErrSourceTooLarge) instead.
  • FromRedis wraps the sentinel. It reported an oversized value with a plain formatted error, so errors.Is(err, ErrSourceTooLarge) was false for the one source that had always detected the condition.
  • MaxFileSize: math.MaxInt64 works. The one-byte overrun margin (MaxFileSize+1) wrapped to math.MinInt64, io.LimitReader returned EOF immediately, and every source — however small — came back as malformed JSON. The limit saturates now.
  • Source ordering is stable. Sources were ordered with sort.Slice, so two sources sharing a priority took an arbitrary order that could differ between runs on identical input.
  • NewLoader/NewLoaderWithNormalize no longer mutate your LoadOptions. They wrote their defaults back into the caller's struct.
  • Documented, not changed: an empty-but-successful source counts as a failure when AllowEmptyData is false — see An empty source counts as a failure — and FromRemote is an SSRF primitive for a caller-controlled URL.

License

Apache License 2.0 — see LICENSE for details.

Documentation

Overview

Package parserkit loads a JSON list from one or more sources, in priority order, with either fallback or merge semantics.

Layout

The root package depends on nothing outside the standard library. Everything that needs a third-party client lives in a subpackage, so a program links only what it actually loads from:

parserkit                  the loader, the file source, stdlib only
parserkit/redissource      a Redis source            (go-redis)
parserkit/remotesource     an HTTP source            (http-kit)

Importing the root package alone costs nothing beyond the standard library. A service that reads its rules from a file on disk does not link go-redis or http-kit, and neither appears in its go.sum. Since http-kit v2, importing parserkit/remotesource no longer links OpenTelemetry either -- trace propagation is opt-in, through remotesource.WithPropagator.

Sources

A source is anything that can produce bytes: a Fetcher. The root package ships File; the subpackages ship the rest, and so can you.

loader, err := parserkit.NewLoader[Rule](nil)
rules, err := loader.Load(ctx,
	parserkit.At(0, parserkit.File("/etc/app/rules.json")),
	parserkit.At(1, redissource.New(rdb, "app:rules")),
)

Priority is the order the loader tries sources in -- lower first. What it does with each result is LoadStrategy: LoadStrategyFallback returns the first source that yields data, LoadStrategyMerge combines them all and deduplicates with LoadOptions.KeyFunc.

Writing a fetcher

A Fetcher returns the raw bytes of one source and enforces the byte budget it is handed. ReadLimited does the second part correctly -- it reads one byte past the limit so that an oversized source is reported as ErrSourceTooLarge rather than silently truncated into a JSON syntax error. Decoding, normalization, priority and merge semantics are the loader's job, not the fetcher's.

Example

Load reads from the highest-priority source that yields data.

package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	parserkit "github.com/soulteary/parser-kit/v3"
)

type Rule struct {
	ID     string `json:"id"`
	Action string `json:"action"`
}

func main() {
	dir, _ := os.MkdirTemp("", "parserkit-example-*")
	defer func() { _ = os.RemoveAll(dir) }()
	path := filepath.Join(dir, "rules.json")
	_ = os.WriteFile(path, []byte(`[{"id":"r1","action":"allow"}]`), 0o600)

	loader, err := parserkit.NewLoader[Rule](nil)
	if err != nil {
		panic(err)
	}

	rules, err := loader.Load(context.Background(),
		parserkit.At(0, parserkit.File(path)),
	)
	if err != nil {
		panic(err)
	}

	fmt.Println(len(rules), rules[0].ID, rules[0].Action)
}
Output:
1 r1 allow

Index

Examples

Constants

View Source
const DefaultMaxBytes int64 = 10 * 1024 * 1024 // 10MB

DefaultMaxBytes is the byte budget a source gets when LoadOptions.MaxBytes is left unset.

Variables

View Source
var ErrNoFetcher = errors.New("parserkit: source has no fetcher")

ErrNoFetcher reports a Source with no Fetcher in it, which is a programming error rather than a source that failed to load.

View Source
var ErrSourceTooLarge = errors.New("parserkit: source exceeds MaxBytes")

ErrSourceTooLarge reports that a source exceeded LoadOptions.MaxBytes.

It exists so the overrun is distinguishable from malformed content: io.LimitReader truncates silently, so an oversized file previously surfaced as a JSON parse error.

Functions

func ReadLimited

func ReadLimited(r io.Reader, maxBytes int64) ([]byte, error)

ReadLimited reads r into memory, refusing to return more than maxBytes.

It reads one byte past the limit, because io.LimitReader truncates silently: without the extra byte an oversized source comes back as a JSON syntax error, indistinguishable from genuinely malformed content. An overrun is reported as an error wrapping ErrSourceTooLarge.

maxBytes of zero or less means DefaultMaxBytes, so a Fetcher used on its own still has a bound. Callers should add their own context:

raw, err := parserkit.ReadLimited(resp.Body, maxBytes)
if err != nil {
	return nil, fmt.Errorf("response from %q: %w", url, err)
}

Types

type BytesFetcher

type BytesFetcher []byte

BytesFetcher serves a payload already in memory. It is what tests and callers that fetched the bytes themselves reach for.

func (BytesFetcher) Fetch

func (b BytesFetcher) Fetch(_ context.Context, maxBytes int64) ([]byte, error)

Fetch returns the bytes, subject to maxBytes.

type DataLoader

type DataLoader[T any] interface {
	// LoadOne loads data from a single source, applying the byte limit and
	// the normalization function but none of the priority or merge handling.
	LoadOne(ctx context.Context, fetcher Fetcher) ([]T, error)

	// Load loads data from multiple sources.
	// Sources are processed in priority order (lower number = higher priority).
	// Behavior depends on LoadStrategy:
	//   - LoadStrategyFallback: returns data from the first successful source
	//   - LoadStrategyMerge: merges data from all successful sources with deduplication
	Load(ctx context.Context, sources ...Source) ([]T, error)
}

DataLoader loads a JSON list of T from one or more sources.

func NewLoader

func NewLoader[T any](opts *LoadOptions[T]) (DataLoader[T], error)

NewLoader creates a new generic data loader

func NewLoaderWithNormalize

func NewLoaderWithNormalize[T any](opts *LoadOptions[T], normalizeFunc NormalizeFunc[T]) (DataLoader[T], error)

NewLoaderWithNormalize creates a new generic data loader with normalization function

type Fetcher

type Fetcher interface {
	Fetch(ctx context.Context, maxBytes int64) ([]byte, error)
}

Fetcher retrieves the raw, still-encoded bytes of a single source.

It is the whole extension point: a Redis key, an HTTP endpoint, an S3 object or a test fixture are all just Fetchers, and the loader does not need to know which. The root package ships File; parserkit/redissource and parserkit/remotesource ship the two that need a third-party client.

maxBytes is LoadOptions.MaxBytes. An implementation must not return more than that, and must report an overrun as an error wrapping ErrSourceTooLarge rather than truncating -- ReadLimited does both. The loader re-checks the length it gets back, so a fetcher that ignores the budget fails loudly instead of quietly.

Returning no bytes and no error means the source is absent -- a file that does not exist, a key that was never set -- and the loader reads that as an empty result, which LoadOptions.AllowEmptyData then governs.

Example

Fetcher is the extension point: a source parser-kit has never heard of works the same as the ones it ships.

package main

import (
	"context"
	"fmt"
	"os"

	parserkit "github.com/soulteary/parser-kit/v3"
)

type Rule struct {
	ID     string `json:"id"`
	Action string `json:"action"`
}

// envFetcher reads a source out of the environment. Any type with a Fetch
// method is a source; ReadLimited is not needed for bytes already in memory,
// but the budget still has to be honoured.
type envFetcher string

func (e envFetcher) Fetch(_ context.Context, maxBytes int64) ([]byte, error) {
	raw := []byte(os.Getenv(string(e)))
	if int64(len(raw)) > maxBytes {
		return nil, fmt.Errorf("$%s: %w", string(e), parserkit.ErrSourceTooLarge)
	}
	return raw, nil
}

func main() {
	_ = os.Setenv("APP_RULES", `[{"id":"env","action":"allow"}]`)
	defer func() { _ = os.Unsetenv("APP_RULES") }()

	loader, err := parserkit.NewLoader[Rule](nil)
	if err != nil {
		panic(err)
	}

	rules, err := loader.LoadOne(context.Background(), envFetcher("APP_RULES"))
	if err != nil {
		panic(err)
	}

	fmt.Println(rules[0].ID, rules[0].Action)
}
Output:
env allow

type FileFetcher

type FileFetcher struct {
	// contains filtered or unexported fields
}

FileFetcher reads a source from the local filesystem. Use File to build one.

func File

func File(path string) *FileFetcher

File returns a Fetcher that reads path.

func (*FileFetcher) AllowMissing

func (f *FileFetcher) AllowMissing() *FileFetcher

AllowMissing makes a file that does not exist report an absent source rather than an error, so the loader moves on to the next source without the missing file showing up as the reason everything failed.

It says nothing about a file that exists and is unreadable: a permission error is still an error.

Example

A missing primary source falls through to the next one instead of failing the whole load.

package main

import (
	"context"
	"fmt"

	parserkit "github.com/soulteary/parser-kit/v3"
)

type Rule struct {
	ID     string `json:"id"`
	Action string `json:"action"`
}

func main() {
	loader, err := parserkit.NewLoader[Rule](nil)
	if err != nil {
		panic(err)
	}

	rules, err := loader.Load(context.Background(),
		parserkit.At(0, parserkit.File("/nonexistent/rules.json").AllowMissing()),
		parserkit.At(1, parserkit.BytesFetcher(`[{"id":"builtin","action":"deny"}]`)),
	)
	if err != nil {
		panic(err)
	}

	fmt.Println(rules[0].ID, rules[0].Action)
}
Output:
builtin deny

func (*FileFetcher) Fetch

func (f *FileFetcher) Fetch(_ context.Context, maxBytes int64) ([]byte, error)

Fetch reads the file, subject to maxBytes.

func (*FileFetcher) Path

func (f *FileFetcher) Path() string

Path returns the path this fetcher reads.

type KeyFunc

type KeyFunc[T any] func(T) (string, bool)

KeyFunc extracts a unique key from an item for deduplication. Returns the key and true if the item should be included, false otherwise.

type LoadOptions

type LoadOptions[T any] struct {
	// MaxBytes limits how much a single source may return (default:
	// DefaultMaxBytes). It applies to every source, not just files: a Fetcher
	// is handed this budget and must report ErrSourceTooLarge rather than
	// return more.
	MaxBytes int64

	// AllowEmptyData if true, a source that succeeds with zero items is a
	// result rather than a failure (default: false).
	//
	// With the default, an empty-but-successful source is treated as a failure
	// and the next source is tried. That is dangerous for allow/deny lists:
	// clearing the primary source falls back to a stale copy elsewhere, so
	// entries that were just deleted come back. Set this when an empty result
	// is a legitimate state -- which for a policy list it almost always is.
	AllowEmptyData bool

	// LoadStrategy determines how to combine data from multiple sources
	// (default: LoadStrategyFallback).
	LoadStrategy LoadStrategy

	// KeyFunc is required when LoadStrategy is LoadStrategyMerge.
	// It extracts a unique key from each item for deduplication.
	KeyFunc KeyFunc[T]
}

LoadOptions configures the behavior of Load operations.

It is generic in T so that KeyFunc is checked by the compiler. Timeouts, retries and TLS settings are not here: they belong to the source that has them, so they are options on that source -- see parserkit/remotesource.

func DefaultLoadOptions

func DefaultLoadOptions[T any]() *LoadOptions[T]

DefaultLoadOptions returns default load options

type LoadStrategy

type LoadStrategy string

LoadStrategy determines how data from multiple sources should be combined

const (
	// LoadStrategyFallback returns data from the first successful source (default)
	LoadStrategyFallback LoadStrategy = "fallback"
	// LoadStrategyMerge merges data from all successful sources with deduplication
	LoadStrategyMerge LoadStrategy = "merge"
)

type NormalizeFunc

type NormalizeFunc[T any] func([]T) []T

NormalizeFunc is a function type for normalizing data after parsing. It accepts a slice of any type and returns a normalized slice.

type Source

type Source struct {
	// Fetcher retrieves the bytes.
	Fetcher Fetcher

	// Priority orders the source against the others in one Load call.
	// Lower number = higher priority (0 is highest). Sources sharing a
	// priority keep the order they were passed in.
	Priority int
}

Source is a Fetcher with a priority attached.

func At

func At(priority int, fetcher Fetcher) Source

At pairs a fetcher with a priority.

loader.Load(ctx,
	parserkit.At(0, parserkit.File("/etc/app/rules.json")),
	parserkit.At(1, redissource.New(rdb, "app:rules")),
)

Directories

Path Synopsis
Package redissource loads parser-kit data from a Redis key.
Package redissource loads parser-kit data from a Redis key.
Package remotesource loads parser-kit data from an HTTP endpoint.
Package remotesource loads parser-kit data from an HTTP endpoint.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL