xconf

package module
v1.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: MIT Imports: 14 Imported by: 0

README

xconf

Load configuration from explicit sources, merge them in priority order, and apply a schemapb schema. Supports typed values, live reload, subscriptions and field provenance. Invalid updates keep the last valid config.

Go 1.25.7+. Root module: github.com/gopherex/xconf. Sources and decoders are separate contrib modules. The rewritten API starts at v1.2.0 and requires migration from earlier releases.

Quick start

Install the library and the integrations used below in your application's Go module:

go get github.com/gopherex/xconf@v1.2.0 \
  github.com/gopherex/xconf/contrib/sources/file@v1.2.0 \
  github.com/gopherex/xconf/contrib/sources/env@v1.2.0 \
  github.com/gopherex/xconf/contrib/decoders/json@v1.2.0

A complete application reading config.json, an optional local file and environment:

package main

import (
    "context"
    "fmt"
    "log"

    sp "github.com/gopherex/schemapb/go/schemapb"
    "github.com/gopherex/xconf"
    jsondecode "github.com/gopherex/xconf/contrib/decoders/json"
    "github.com/gopherex/xconf/contrib/sources/env"
    "github.com/gopherex/xconf/contrib/sources/file"
)

type Config struct {
    Server struct {
        Host string `json:"host" schemapb:"default=127.0.0.1"`
        Port int64  `json:"port" schemapb:"default=8080;gte=1;lte=65535"`
    } `json:"server" schemapb:"default={}"`
}

func main() {
    schema, err := sp.ReflectType[Config](sp.ID("app", "config", sp.Ver(1, 0, 0)))
    if err != nil {
        log.Fatal(err)
    }
    cfg, err := xconf.LoadAs[Config](context.Background(), schema,
        file.New("config.json", jsondecode.Decode),
        xconf.Optional(file.New("config.local.json", jsondecode.Decode)),
        env.New(env.Prefix("APP_")),
    )
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s:%d\n", cfg.Server.Host, cfg.Server.Port)
}

Save this as main.go and create config.json:

{"server":{"port":9090}}

Run go run . to get 127.0.0.1:9090. APP_SERVER_PORT=7000 overrides the port; APP_SERVER_PORT=70000 fails validation. An existing *schemapb.Schema can be passed directly instead of reflecting a struct.

Struct tags

json names fields; schemapb defines defaults and constraints. Separate attributes with ;. Reflection builds the schema; defaults and constraints apply after merging.

Tag Meaning
json:"port" Key in documents; becomes APP_PORT with the APP_ env prefix
json:"port,omitempty" Makes an absent field optional; explicit zero remains present
json:"-" Excludes the field
schemapb:"default=8080;gte=1;lte=65535" Integer default and inclusive bounds
schemapb:"default=30s;gt=0s" Duration default and positive constraint
schemapb:"default=false" Boolean default; explicit false is preserved
schemapb:"required=true;min_len=1" Requires a string field and rejects an empty string
schemapb:"default={}" Creates a missing object so child defaults can apply
schemapb:"min_items=1;max_items=8" List size bounds
schemapb:"in=[\"fast\",\"safe\"];default=safe" Allowed string values and default
schemapb:"secret=true" Marks schema metadata as secret; does not encrypt or erase the value
schemapb:"description=API address" Field description

Non-pointer fields without omitempty are required by default. Pointers are optional and nullable; required=true and nullable=false can override those properties independently. Defaults apply to missing keys, never explicit null, zero, false or empty strings. List/map defaults and nonempty object defaults are not supported by the pinned schemapb version.

The supported validate vocabulary includes required, numeric comparisons, string lengths, list min/max, oneof, and formats such as email, url and uuid. It is not the full go-playground validator. Use schemapb attributes for explicit schema rules. See tag examples and limitations.

Priority and merge rules

Pass sources from lowest to highest priority. Objects/maps merge recursively; scalars, null and lists replace previous values. Zero, false, "" and [] are explicit overrides. {} preserves lower object keys; use a Replace edit to clear an object. Delete removes a key before schema defaults run. Changing a OneOf discriminator replaces the previous variant's object.

Optional suppresses only ErrNotFound, not malformed input or unavailable stores. Removing an override reveals lower layers or defaults on the next successful reload. Unknown fields follow schema strictness. Reflection enables coercion for textual sources; enable it explicitly in hand-built schemas.

xconf.AllowPaths(source, xconf.Path{"server", "limit"}, xconf.Path{"logging"}) restricts any source to selected paths and their subtrees, for example live settings from Consul over a local file. Other fields keep their lower-layer values. Paths use literal key segments; watch notifications and source provenance are preserved. See path filtering.

Reload and subscriptions

Using the imports and Config above:

func watch(ctx context.Context, schema *sp.Schema) error {
    runtime, err := xconf.OpenAs[Config](ctx, schema,
        file.New("config.json", jsondecode.Decode), env.New(env.Prefix("APP_")),
    )
    if err != nil {
        return err
    }
    defer runtime.Close()
    for event := range runtime.Subscribe(ctx) {
        if event.Err != nil {
            log.Printf("configuration update rejected: %v", event.Err)
            continue
        }
        cfg, err := xconf.Decode[Config](event.Snapshot)
        if err != nil {
            return err
        }
        fmt.Printf("server=%s:%d\n", cfg.Server.Host, cfg.Server.Port)
        // Reconfigure application components here.
    }
    return nil
}

Cancel ctx to stop. Subscribers receive the current snapshot immediately. Current() returns an independent typed value; Reload(ctx) requests a rebuild. Initial failure aborts OpenAs, except as held below; later failures retain the last valid snapshot. Slow subscribers receive only the latest event, so reconcile full state. See runtime semantics and custom sources.

xconf.Resilient(source) keeps a remote source working while it is unavailable, at startup or later: a failed read serves the last good layer (or an empty one) with Layer.Stale set, and never fails Open itself. event.Snapshot.Degraded() maps stale source names to their raw errors (may contain secrets); becoming stale or recovering publishes an event. Reads and a failing watch registration retry with xconf.Backoff(min, max) (default 1s..30s) until Close. Use xconf.Resilient(xconf.Optional(source)) so a missing key is empty, not degraded. A required value served only by a degraded source holds Open/OpenAs until the source recovers or ctx ends — use a ctx deadline to bound startup (ctx is also the runtime lifetime: cancel it from a timer stopped once Open returns). Validation or typed decode failures while a source is degraded return *xconf.DegradedError (wrapping the *xconf.ValidationError); its text names the sources, never their errors. See unavailable remote sources.

Sources: where to load

Module paths start with github.com/gopherex/xconf/contrib/sources/. Document sources accept a decoder; environment and field-oriented sources supply values directly.

Module Input Updates
file File with a supplied decoder Polling
fs fs.FS, including embedded files Manual / xconf.Poll
reader Reader factory or captured stream Manual / xconf.Poll
env Process or injected environment Manual / xconf.Poll
dotenv .env file, without modifying process env Polling
envfile Explicit *_FILE variables pointing to files Polling
directory One field per file, including mounted secrets Polling
flags Explicit arguments and field bindings Fixed arguments
pflag Changed pflag/Cobra flags Captured after parsing
http HTTP document Conditional polling
consul KV document or prefix mapped to fields Blocking queries
etcd Document in an etcd v3 key Watch
kubernetes ConfigMap/Secret fields or document Watch
nats JetStream KV document Watch
s3 S3-compatible object Polling
vault Vault KV v2 data Polling

Environment paths map to names such as server.port → APP_SERVER_PORT. env.Bind supplies explicit mappings; env.Strict() rejects unknown prefixed names. Containers use JSON unless env.Split is configured; an empty container variable is unset, as orchestrators render an absent optional variable. Errors name the variable, never its value, and show in the SourceError text. Remote clients, authentication and TLS belong to the caller. See source options.

Decoders: how to parse

Module paths start with github.com/gopherex/xconf/contrib/decoders/. Each Decode implements xconf.Decoder and performs no I/O or schema validation.

Module Format
json One JSON object; exact numeric tokens
yaml One mapping; bounded aliases and source locations; no merge keys
toml TOML document, including date/time values

For example, httpconf.New(client, url, jsondecode.Decode) combines an HTTP source with a JSON decoder. sources/json, sources/yaml and sources/toml are file convenience wrappers: jsonconf.File(path) combines file with the JSON decoder. The shared internal/document helper is private to the root module.

API and development

API Result
LoadAs[T](ctx, schema, sources...) Typed config
Load(ctx, schema, sources...) Snapshot with diagnostics and provenance
Decode[T](snapshot) Independently owned typed value
OpenAs[T](ctx, schema, sources...) Runtime with typed checks before publication
Open(ctx, schema, sources...) Untyped runtime
New(schema, compileOptions...) Reusable compiled loader
Resilient(source, Backoff(min, max)) Source serving its last good layer while failing
snapshot.Degraded() Raw errors of stale sources, keyed by name
DegradedError Validation/decode failure while sources are stale

snapshot.Explain("server", "port") returns one field's provenance; snapshot.Origins() returns a copied map[string][]Origin for all JSON-pointer paths. Both omit values. Raw snapshot diagnostics and unwrapped provider errors may contain secrets.

make test builds, vets and race-tests every module. make tidy updates dependencies. Local contrib replace directives support checkout development with GOWORK=off; consumers use published versions. make release updates requirements in all modules, including the example, then tidies and tests before committing and publishing library tags. Examples are not tagged. python3 scripts/test_release.py checks this workflow against a local test remote. Verify released consumers without local replacements.

The old xconf DSL, xconfgen, pkg/load and pkg/structconf are removed. Migrate schemas to schemapb and loading to LoadAs / OpenAs; existing applications can keep their pinned releases until migrated.

Documentation

Overview

Package xconf assembles ordered configuration layers against a schemapb schema. Defaults, coercion and validation run once after merging. Reloads publish only complete, validated snapshots.

Index

Constants

This section is empty.

Variables

View Source
var ErrClosed = errors.New("xconf: runtime closed")
View Source
var ErrNotFound = errors.New("xconf: source not found")

Functions

func Decode added in v1.2.0

func Decode[T any](s *Snapshot) (T, error)

func LoadAs added in v1.2.0

func LoadAs[T any](ctx context.Context, schema *sp.Schema, sources ...Source) (T, error)

func Public added in v1.4.2

func Public(err error) error

Public marks err, whose text holds no configuration value, as safe to show in SourceError; nil stays nil.

Types

type Decoder added in v1.2.0

type Decoder func([]byte) (map[string]any, map[string]Location, error)

Decoder converts one document into an object and optional JSON-pointer locations. Decoders preserve numeric precision and do not apply schema defaults or validation. Each call must return independently owned data.

type DegradedError added in v1.4.0

type DegradedError struct {
	Degraded map[string]error
	Err      error
}

DegradedError wraps a validation or typed decode failure that occurred while sources served stale layers, so recovery may resolve it. Degraded holds their raw errors (may contain secrets); Error() names the sources only.

func (*DegradedError) Error added in v1.4.0

func (e *DegradedError) Error() string

func (*DegradedError) Unwrap added in v1.4.0

func (e *DegradedError) Unwrap() error

type Edit added in v1.2.0

type Edit struct {
	Kind  EditKind
	Path  Path
	Value any
}

Edit operates on object members. Lists are replaced as a whole.

type EditKind added in v1.2.0

type EditKind uint8
const (
	Replace EditKind = iota + 1
	Delete
)

type Event added in v1.2.0

type Event struct {
	Snapshot *Snapshot
	Err      error
	Changed  []Path
}

Event carries the current valid snapshot, even after a failed reload. Changed paths are relative to the previous published version, not the last event a subscriber read. Slow subscribers receive the latest event, dropping intermediates.

type Layer added in v1.2.0

type Layer struct {
	Values    map[string]any
	Revision  string
	Locations map[string]Location // keys are JSON pointers
	Edits     []Edit              // applied in order after Values
	// Stale reports a failing source: the layer is its last good one (or empty).
	// The raw error is exposed only through Snapshot.Degraded.
	Stale error
}

type Loader added in v1.2.0

type Loader struct {
	// contains filtered or unexported fields
}

Loader holds an owned schema and compiled engine for repeated loads. Compile options (including custom formats and CEL cost limits) pass to schemapb.

func New added in v1.2.0

func New(schema *sp.Schema, options ...sp.CompileOption) (*Loader, error)

func (*Loader) Load added in v1.2.0

func (l *Loader) Load(ctx context.Context, sources ...Source) (*Snapshot, error)

func (*Loader) Open added in v1.2.0

func (l *Loader) Open(ctx context.Context, sources ...Source) (*Runtime, error)

Open starts a runtime with this loader's schema and compile options.

type Location added in v1.2.0

type Location struct {
	Name         string
	Line, Column int
}

Location describes a source position, never its value.

type Memory added in v1.2.0

type Memory struct {
	// contains filtered or unexported fields
}

Memory is a concurrent, watchable source useful for overrides and tests.

func NewMemory added in v1.2.0

func NewMemory(name string, values map[string]any) (*Memory, error)

func (*Memory) Name added in v1.2.0

func (m *Memory) Name() string

func (*Memory) Read added in v1.2.0

func (m *Memory) Read(ctx context.Context, _ *sp.Schema) (Layer, error)

func (*Memory) Set added in v1.2.0

func (m *Memory) Set(layer Layer) error

func (*Memory) Watch added in v1.2.0

func (m *Memory) Watch(_ context.Context, notify func()) (func(), error)

type Origin added in v1.2.0

type Origin struct {
	Source, Revision, Operation string
	Location                    Location
}

type Path added in v1.2.0

type Path []string

Path addresses an object member or list index. String returns a JSON pointer.

func (Path) String added in v1.2.0

func (p Path) String() string

type PublicError added in v1.4.2

type PublicError struct{ Err error }

PublicError is a source error whose text names keys, paths or variables but never holds a configuration value, so SourceError shows it.

func (*PublicError) Error added in v1.4.2

func (e *PublicError) Error() string

func (*PublicError) Unwrap added in v1.4.2

func (e *PublicError) Unwrap() error

type ResilientOption added in v1.3.0

type ResilientOption func(*resilient)

func Backoff added in v1.3.0

func Backoff(min, max time.Duration) ResilientOption

Backoff bounds the retry delay, doubling from min to max. Default 1s..30s.

type Runtime added in v1.2.0

type Runtime struct {
	// contains filtered or unexported fields
}

func Open added in v1.2.0

func Open(ctx context.Context, schema *sp.Schema, sources ...Source) (*Runtime, error)

func (*Runtime) Close added in v1.2.0

func (r *Runtime) Close() error

Close cancels reads/watchers and waits for shutdown. Sources must honor context.

func (*Runtime) Reload added in v1.2.0

func (r *Runtime) Reload(ctx context.Context) (*Snapshot, error)

Reload serializes full reads and keeps the last valid snapshot on every failure. A successful rebuild with identical values, provenance and degraded sources keeps its version; a source becoming stale or recovering publishes an event.

func (*Runtime) Snapshot added in v1.2.0

func (r *Runtime) Snapshot() *Snapshot

func (*Runtime) Subscribe added in v1.2.0

func (r *Runtime) Subscribe(ctx context.Context) <-chan Event

Subscribe immediately delivers the current snapshot. Cancel ctx to unsubscribe.

type Snapshot added in v1.2.0

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot owns immutable state. Accessors returning mutable data clone it.

func Load added in v1.2.0

func Load(ctx context.Context, schema *sp.Schema, sources ...Source) (*Snapshot, error)

func (*Snapshot) Baked added in v1.2.0

func (s *Snapshot) Baked() *sp.Baked

func (*Snapshot) Decode added in v1.2.0

func (s *Snapshot) Decode(target any) error

func (*Snapshot) Degraded added in v1.3.0

func (s *Snapshot) Degraded() map[string]error

Degraded returns the raw errors of sources serving stale layers, keyed by source name. It is empty when every source is healthy. Errors may contain secrets.

func (*Snapshot) Explain added in v1.2.0

func (s *Snapshot) Explain(path ...string) []Origin

Explain returns source writes followed by schema operations at this exact path. Records contain no values, including overridden values.

func (*Snapshot) LoadedAt added in v1.2.0

func (s *Snapshot) LoadedAt() time.Time

func (*Snapshot) Origins added in v1.2.2

func (s *Snapshot) Origins() map[string][]Origin

Origins returns all provenance keyed by JSON pointer, including schema operations. The map and each history slice are independent copies. Records contain no values.

func (*Snapshot) Report added in v1.2.0

func (s *Snapshot) Report() *sp.ResolveReport

func (*Snapshot) Revisions added in v1.2.0

func (s *Snapshot) Revisions() map[string]string

func (*Snapshot) Validation added in v1.2.0

func (s *Snapshot) Validation() *sp.ValidationResult

func (*Snapshot) Version added in v1.2.0

func (s *Snapshot) Version() uint64

type Source added in v1.2.0

type Source interface {
	Name() string
	Read(context.Context, *sp.Schema) (Layer, error)
}

Source reads a complete partial layer. Read must honor ctx and return owned data (or data it will not mutate). The schema argument is an isolated copy. Sources must not apply schema defaults or require a complete configuration.

func AllowPaths added in v1.2.3

func AllowPaths(source Source, paths ...Path) Source

AllowPaths restricts a source to the given object/map paths and their subtrees. Segments are literal keys, not globs or dotted paths. No paths allows nothing; an empty Path allows everything. Paths are captured when the wrapper is made. Lists must be selected as a whole. Selecting a OneOf discriminator requires selecting its whole object, since changing it replaces the previous variant. Ancestor edits are projected onto allowed subtrees, preserving their siblings. Name, Revision, read errors and watch notifications pass through unchanged.

func Optional added in v1.2.0

func Optional(source Source) Source

Optional suppresses only ErrNotFound. Parse, permission and network errors fail.

func Poll added in v1.2.0

func Poll(source Source, interval time.Duration) Source

Poll adds periodic invalidation to a source. It does not cache failed reads.

func Resilient added in v1.3.0

func Resilient(source Source, opts ...ResilientOption) Source

Resilient keeps a remote source in effect while it is unavailable. Any Read error yields the last good layer (or an empty one) with Layer.Stale set, so the source never fails Open or Reload; Snapshot.Degraded reports it. While stale, the wrapper invalidates the runtime with exponential backoff until a Read succeeds. A failing inner Watch registration is retried in the background, followed by an invalidation to cover changes missed before it. ErrNotFound is a failure too; compose Resilient(Optional(source)) to treat a missing source as empty instead.

type SourceError added in v1.2.0

type SourceError struct {
	Source, Stage string
	Cause         error
}

SourceError omits provider error text, which may contain secrets, unless the cause is a PublicError. The original error remains available via errors.Is/As/Unwrap.

func (*SourceError) Error added in v1.2.0

func (e *SourceError) Error() string

func (*SourceError) Unwrap added in v1.2.0

func (e *SourceError) Unwrap() error

type SourceFunc added in v1.2.0

type SourceFunc struct {
	ID       string
	ReadFunc func(context.Context, *sp.Schema) (Layer, error)
}

SourceFunc adapts a custom reader. Name must be stable and unique in a loader.

func (SourceFunc) Name added in v1.2.0

func (s SourceFunc) Name() string

func (SourceFunc) Read added in v1.2.0

func (s SourceFunc) Read(ctx context.Context, schema *sp.Schema) (Layer, error)

type TypedRuntime added in v1.2.0

type TypedRuntime[T any] struct{ *Runtime }

TypedRuntime validates destination decoding before publishing each snapshot. Current returns a fresh value; callers can freely mutate its maps and slices.

func OpenAs added in v1.2.0

func OpenAs[T any](ctx context.Context, schema *sp.Schema, sources ...Source) (*TypedRuntime[T], error)

func (*TypedRuntime[T]) Current added in v1.2.0

func (r *TypedRuntime[T]) Current() (T, error)

type ValidationError added in v1.2.0

type ValidationError struct {
	Result *sp.ValidationResult
	Report *sp.ResolveReport
}

ValidationError carries diagnostics without including input values in Error().

func (*ValidationError) Error added in v1.2.0

func (e *ValidationError) Error() string

type Watcher added in v1.2.0

type Watcher interface {
	Watch(context.Context, func()) (stop func(), err error)
}

Watcher registers notifications synchronously, before the initial Read. Notifications invalidate the whole layer. stop must release resources and wait for callbacks to finish. Read and notifications may run concurrently.

Directories

Path Synopsis
contrib
decoders/json module
decoders/toml module
decoders/yaml module
sources/env module
sources/etcd module
sources/file module
sources/flags module
sources/fs module
sources/http module
sources/json module
sources/nats module
sources/pflag module
sources/s3 module
sources/toml module
sources/vault module
sources/yaml module
internal
document
Package document contains shared bounded document reading for source adapters.
Package document contains shared bounded document reading for source adapters.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL