parserkit

package module
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 12, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

README

Parser Kit

Go Reference Go Report Card License codecov

中文文档

A generic data loader kit that supports loading data from multiple sources (file, Redis, remote HTTP) with priority-based fallback strategy.

Features

  • Multi-source support: Load data from local files, Redis, or remote HTTP endpoints
  • Priority-based fallback: Automatically fallback to next source if previous one fails
  • Generic design: Works with any JSON-serializable type
  • HTTP retry mechanism: Automatic retry with exponential backoff for remote requests
  • Size limits: Protection against memory exhaustion attacks (file/remote; Redis uses MaxFileSize with a size check)
  • Normalization support: Optional data normalization after parsing

Requirements

  • Go 1.27+ (go.mod declares go 1.27.0)
  • github.com/redis/go-redis/v9 for the Redis source

Installation

go get github.com/soulteary/parser-kit

Usage

Basic Example
package main

import (
    "context"
    "github.com/soulteary/parser-kit"
    "github.com/soulteary/redis-kit/client"
)

type User struct {
    ID    string `json:"id"`
    Email string `json:"email"`
    Phone string `json:"phone"`
}

func main() {
    // Create loader
    loader, err := parserkit.NewLoader[User](nil)
    if err != nil {
        panic(err)
    }

    // Define sources with priority (lower number = higher priority)
    sources := []parserkit.Source{
        {
            Type:     parserkit.SourceTypeRedis,
            Priority: 0, // Highest priority
            Config: parserkit.SourceConfig{
                RedisKey:    "users:cache",
                RedisClient: redisClient, // *redis.Client
            },
        },
        {
            Type:     parserkit.SourceTypeRemote,
            Priority: 1,
            Config: parserkit.SourceConfig{
                RemoteURL:           "https://api.example.com/users",
                AuthorizationHeader: "Bearer token",
            },
        },
        {
            Type:     parserkit.SourceTypeFile,
            Priority: 2, // Lowest priority (fallback)
            Config: parserkit.SourceConfig{
                FilePath: "/path/to/users.json",
            },
        },
    }

    // Load data (will try Redis first, then remote, then file)
    ctx := context.Background()
    users, err := loader.Load(ctx, sources...)
    if err != nil {
        panic(err)
    }

    // Use users...
}
Individual Source Loading
// Load from file
users, err := loader.FromFile(ctx, "/path/to/users.json")

// Load from remote
users, err := loader.FromRemote(ctx, "https://api.example.com/users", "Bearer token")

// Load from Redis
users, err := loader.FromRedis(ctx, redisClient, "users:cache")
Custom Options
opts := &parserkit.LoadOptions{
    MaxFileSize:  20 * 1024 * 1024, // 20MB
    MaxRetries:   5,
    RetryDelay:   2 * time.Second,
    HTTPTimeout:  10 * time.Second,
}

normalizeFunc := func(users []User) []User {
    // Normalize data after parsing
    for i := range users {
        // Apply normalization logic
    }
    return users
}

loader, err := parserkit.NewLoaderWithNormalize[User](opts, normalizeFunc)

Source Types

File Source

Loads data from a local JSON file.

{
    Type: parserkit.SourceTypeFile,
    Priority: 2,
    Config: parserkit.SourceConfig{
        FilePath: "/path/to/data.json",
    },
}
Redis Source

Loads data from a Redis key (must contain JSON).

{
    Type: parserkit.SourceTypeRedis,
    Priority: 0,
    Config: parserkit.SourceConfig{
        RedisKey:    "data:cache",
        RedisClient: redisClient, // *redis.Client from redis-kit
    },
}
Remote Source

Loads data from a remote HTTP/HTTPS endpoint. Make sure RemoteURL is trusted or validated by the caller to avoid SSRF.

{
    Type: parserkit.SourceTypeRemote,
    Priority: 1,
    Config: parserkit.SourceConfig{
        RemoteURL:           "https://api.example.com/data",
        AuthorizationHeader: "Bearer token", // Optional
        Timeout:             5 * time.Second, // Optional, uses default if not set
    },
}
// Note: InsecureSkipVerify is set in LoadOptions at loader creation, not per source.

Note: InsecureSkipVerify is applied at loader creation time via LoadOptions. Per-source values are ignored; create separate loaders if you need different TLS behavior per source.

Security note for remote sources

FromRemote uses the URL as given and sends auth as an Authorization header, with no validation. A caller-controlled URL therefore makes it an SSRF primitive that forwards your credentials to wherever the URL points.

Validate the URL before passing it in, and close the DNS-rebinding window on the dial:

import "github.com/soulteary/cli-kit/validator"

opts := &validator.URLOptions{AllowedSchemes: []string{"https"}}
if err := validator.ValidateURL(remoteURL, opts); err != nil {
    return err
}
// and use validator.SSRFDialControl(opts) on the transport that fetches it

InsecureSkipVerify disables TLS verification entirely — development only.

Priority System

Sources are processed in priority order:

  • Lower priority number = higher priority
  • Priority 0 is the highest priority
  • If a source fails, the loader automatically tries the next source
  • Behavior depends on LoadStrategy (see below)

Sorting is stable, so two sources sharing a priority keep the order you passed them in — the same input always produces the same order.

Load Strategy

Two strategies control how data from multiple sources is combined:

Fallback (default)

LoadStrategyFallback: Returns data from the first successful source. Use for cache → remote → file style loading.

Merge

LoadStrategyMerge: Merges data from all successful sources with deduplication. Use when you need "remote + local supplement" (e.g. Warden's REMOTE_FIRST). Requires KeyFunc to extract a unique key per item.

keyFunc := func(u User) (string, bool) { return u.Phone, true } // key, include
opts := parserkit.DefaultLoadOptions()
opts.LoadStrategy = parserkit.LoadStrategyMerge
opts.KeyFunc = keyFunc
loader, _ := parserkit.NewLoader[User](opts)

// Load merges file1 + file2; same key overwrites (later source wins)
users, _ := loader.Load(ctx, sources...)

Options Reference

Option Default Description
MaxFileSize 10MB Max bytes to read from file/response
MaxRetries 3 Retries for remote requests
RetryDelay 1s Base delay between retries
MaxRetryDelay 30s Ceiling on the retry backoff. Must be positive — see below
HTTPTimeout 5s Timeout for remote requests
InsecureSkipVerify false Skip TLS verification (dev only)
AllowEmptyFile false Return [] when file not found instead of error
AllowEmptyData false When false, treat empty source as failure and try next
LoadStrategy fallback fallback or merge
KeyFunc nil Required for merge; func(T) (string, bool)

MaxRetryDelay is not optional in the way a zero value usually is. http-kit clamps every computed backoff to it unconditionally, so a zero ceiling means every retry fires immediately — RetryDelay and the backoff multiplier look configured and do nothing, and a failing remote source is retried as fast as the network allows. NewLoader fills the 30s default when it is unset, so this only bites if you set it to zero on purpose.

Use DefaultLoadOptions() and override the fields you need, so MaxFileSize and similar are set rather than zero. MaxFileSize also guards the Redis value size (checked with STRLEN before loading).

NewLoader and NewLoaderWithNormalize work on a copy of your LoadOptions, so filling in defaults does not mutate the struct you passed:

opts := parserkit.DefaultLoadOptions()
loader, err := parserkit.NewLoader[User](opts)

// NewLoaderWithNormalize post-processes every successful load
loader, err = parserkit.NewLoaderWithNormalize[User](opts, func(users []User) []User {
    for i := range users {
        users[i].Phone = strings.TrimSpace(users[i].Phone)
    }
    return users
})

Error Handling

  • If every source fails, Load() returns an error carrying the last one encountered.
  • The individual source methods (FromFile, FromRemote, FromRedis) return their error immediately.
  • File not found is an error by default; AllowEmptyFile: true returns [].
Oversized sources

A source larger than MaxFileSize fails with ErrSourceTooLarge, from all three source types:

data, err := loader.Load(ctx, sources...)
if errors.Is(err, parserkit.ErrSourceTooLarge) {
    // the source exists and is reachable, but exceeds MaxFileSize
}

Match it with errors.Is rather than inspecting the message — an oversized source and a malformed one are different problems with different fixes.

Pass math.MaxInt64 as MaxFileSize to mean "no limit"; the one-byte overrun margin saturates rather than overflowing.

An empty source counts as a failure

With AllowEmptyData: false (the default), a source that loads successfully but yields no entries is treated as a failure, and the fallback strategy moves on to the next source.

That is worth thinking about for an allow list or deny list: clearing the primary source falls back to a stale copy in Redis or on a remote, and entries you just deleted come back. Set AllowEmptyData: true when an empty result is a legitimate state.

Testing

Tests do not require a real Redis instance. The suite uses miniredis for Redis-related tests, so you can run tests and coverage locally without any external services:

go test ./...
go test -coverprofile=coverage.out -covermode=atomic ./...
go tool cover -func=coverage.out

Dependencies

  • github.com/soulteary/http-kit - For HTTP client and retry logic
  • github.com/redis/go-redis/v9 - For Redis operations

Test-only: github.com/alicebob/miniredis/v2 for in-process Redis in tests.

Upgrade Notes (v1.6.0)

One sentinel was added; nothing was removed. Two error paths report differently.

  • An oversized source is reported as ErrSourceTooLarge, not as malformed data. io.LimitReader truncates silently, so a file or HTTP response larger than MaxFileSize came back as invalid JSON with no way to tell the two apart. Both paths now read one byte past the limit and report the sentinel — matching FromRedis, which already checked STRLEN up front, so the three sources behaved differently for the same condition. If you matched on "invalid JSON" to detect an oversized source, match errors.Is(err, ErrSourceTooLarge) instead.
  • FromRedis wraps the sentinel. It reported an oversized value with a plain formatted error, so errors.Is(err, ErrSourceTooLarge) was false for the one source that had always detected the condition.
  • MaxFileSize: math.MaxInt64 works. The one-byte overrun margin (MaxFileSize+1) wrapped to math.MinInt64, io.LimitReader returned EOF immediately, and every source — however small — came back as malformed JSON. The limit saturates now.
  • Source ordering is stable. Sources were ordered with sort.Slice, so two sources sharing a priority took an arbitrary order that could differ between runs on identical input.
  • NewLoader/NewLoaderWithNormalize no longer mutate your LoadOptions. They wrote their defaults back into the caller's struct.
  • Documented, not changed: an empty-but-successful source counts as a failure when AllowEmptyData is false — see An empty source counts as a failure — and FromRemote is an SSRF primitive for a caller-controlled URL.

License

Apache License 2.0 — see LICENSE for details.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrSourceTooLarge = errors.New("parserkit: source exceeds MaxFileSize")

ErrSourceTooLarge reports that a source exceeded LoadOptions.MaxFileSize.

It exists so the overrun is distinguishable from malformed content: io.LimitReader truncates silently, so an oversized file previously surfaced as a JSON parse error.

Functions

This section is empty.

Types

type DataLoader

type DataLoader[T any] interface {
	// FromFile loads data from a local file
	FromFile(ctx context.Context, path string) ([]T, error)

	// FromRemote loads data from a remote URL
	FromRemote(ctx context.Context, url, auth string) ([]T, error)

	// FromRedis loads data from Redis
	FromRedis(ctx context.Context, client interface{}, key string) ([]T, error)

	// Load loads data from multiple sources
	// Sources are processed in priority order (lower number = higher priority)
	// Behavior depends on LoadStrategy:
	// - LoadStrategyFallback: returns data from the first successful source
	// - LoadStrategyMerge: merges data from all successful sources with deduplication
	Load(ctx context.Context, sources ...Source) ([]T, error)
}

DataLoader is a generic interface for loading data from various sources

func NewLoader

func NewLoader[T any](opts *LoadOptions) (DataLoader[T], error)

NewLoader creates a new generic data loader

func NewLoaderWithNormalize

func NewLoaderWithNormalize[T any](opts *LoadOptions, normalizeFunc NormalizeFunc[T]) (DataLoader[T], error)

NewLoaderWithNormalize creates a new generic data loader with normalization function

type KeyFunc

type KeyFunc[T any] func(T) (string, bool)

KeyFunc extracts a unique key from an item for deduplication Returns the key and true if the item should be included, false otherwise

type LoadOptions

type LoadOptions struct {
	// MaxFileSize limits the maximum file size to read (default: 10MB)
	MaxFileSize int64

	// MaxRetries for remote requests (default: 3)
	MaxRetries int

	// RetryDelay for remote requests (default: 1s)
	RetryDelay time.Duration

	// MaxRetryDelay caps the backoff between remote retries (default: 30s).
	//
	// It must be positive. http-kit's CalculateRetryDelay clamps every computed
	// delay to this ceiling unconditionally, so a zero value does not mean
	// "no ceiling" -- it means every retry happens immediately, which turns a
	// failing remote source into a tight request loop. NewLoader fills the
	// default when this is unset for exactly that reason.
	MaxRetryDelay time.Duration

	// HTTPTimeout for remote requests (default: 5s)
	HTTPTimeout time.Duration

	// InsecureSkipVerify allows skipping TLS certificate verification (default: false)
	// Only use in development environments
	InsecureSkipVerify bool

	// AllowEmptyFile if true, returns empty slice instead of error when file not found (default: false)
	AllowEmptyFile bool

	// AllowEmptyData if true, continues to next source even if current source returns empty data (default: false)
	AllowEmptyData bool

	// LoadStrategy determines how to combine data from multiple sources (default: LoadStrategyFallback)
	// - LoadStrategyFallback: return data from first successful source
	// - LoadStrategyMerge: merge data from all successful sources with deduplication
	LoadStrategy LoadStrategy

	// KeyFunc is required when LoadStrategy is LoadStrategyMerge
	// It extracts a unique key from each item for deduplication
	// Note: This is stored as interface{} and will be type-asserted in the loader
	KeyFunc interface{} // Should be KeyFunc[T] but we can't use generics in struct fields
}

LoadOptions configures the behavior of Load operations

func DefaultLoadOptions

func DefaultLoadOptions() *LoadOptions

DefaultLoadOptions returns default load options

type LoadStrategy

type LoadStrategy string

LoadStrategy determines how data from multiple sources should be combined

const (
	// LoadStrategyFallback returns data from the first successful source (default)
	LoadStrategyFallback LoadStrategy = "fallback"
	// LoadStrategyMerge merges data from all successful sources with deduplication
	LoadStrategyMerge LoadStrategy = "merge"
)

type NormalizeFunc

type NormalizeFunc[T any] func([]T) []T

NormalizeFunc is a function type for normalizing data after parsing It accepts a slice of any type and returns a normalized slice

type Source

type Source struct {
	Type     SourceType
	Priority int // Lower number = higher priority (0 is highest)
	Config   SourceConfig
}

Source represents a data source with priority

type SourceConfig

type SourceConfig struct {
	// For file source
	FilePath string

	// For Redis source
	RedisKey    string
	RedisClient interface{} // *redis.Client from redis-kit

	// For remote source
	RemoteURL           string
	AuthorizationHeader string
	Timeout             time.Duration
	InsecureSkipVerify  bool
}

SourceConfig holds configuration for a data source

type SourceType

type SourceType string

SourceType represents the type of data source

const (
	// SourceTypeFile represents a local file source
	SourceTypeFile SourceType = "file"
	// SourceTypeRedis represents a Redis source
	SourceTypeRedis SourceType = "redis"
	// SourceTypeRemote represents a remote HTTP/HTTPS source
	SourceTypeRemote SourceType = "remote"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL