fastconf

module
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 14, 2026 License: MIT

README

FastConf — 强类型 · 无锁 · Kustomize 风格配置框架

fastconf 是一个基于 Go 1.26 泛型的强类型、无锁、Kustomize 风格动态配置加载框架。

一句话说明:把 YAML / JSON 文件、环境变量、命令行参数、远程 KV 像 Kustomize 一样叠加成一个强类型 Go 结构体,并在 K8s ConfigMap 改动时安全地热重载。

当前版本:v0.8.1;变更细节见 CHANGELOG.md

Go Reference CI


目录


0. 架构概览

FastConf 把配置加载分成三层:声明(Options)→ 构建(New)→ 运行(reload loop)。Options 是纯值(functional option pattern),可组合、可覆盖;New[T] 首次同步执行一遍完整 reload pipeline 来生成初始 *State[T];此后 reload loop 是单写者 goroutine,所有改动通过 reloadCh 串行消费。

┌──────────────────────────────────────────────────────────────────┐
│  User Code                                                       │
│  cfg, _ := fastconf.New[T](ctx, opts...);  app := cfg.Get()     │
└──────────────────────────────┬───────────────────────────────────┘
                               │
                               ▼
       ┌───────────────────────────────────────────────────────┐
       │  Manager[T]                                           │
       │  ┌── atomic.Pointer[State[T]]  ◄── commit()          │
       │  │                                                    │
       │  │  reloadCh ◄── fsnotify + debounce (500 ms)        │
       │  │           ◄── provider events (backoff + drop)     │
       │  │           ◄── ManualReload(ctx, requestID)         │
       │  │                                                    │
       │  │  reload pipeline (single writer goroutine):        │
       │  │    stageMerge   → stageAssemble                    │
       │  │    stageMigrate → stageTransform                   │
       │  │    stageDecode  → stageValidate → stagePolicy      │
       │  │                                                    │
       │  │    ↑ fail at any stage → preserve old *State[T]   │
       │  │    ↓ success → canonicalHashBytes → deduplicate    │
       │  │              → atomic.Pointer.Store(newState)      │
       │  │              → history.push → audit fan-out        │
       │  │              → fireWatches (PartHashes diff)        │
       └───────────────────────────────────────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────────────────────────┐
│  State[T]  (immutable per-reload snapshot)                       │
│    Value       *T                  // zero-alloc Get() returns   │
│    Hash        [32]byte            // SHA-256 of canonical JSON  │
│    PartHashes  map[string][32]byte // per top-level field        │
│    Sources     []SourceRef         // all merged layers          │
│    Generation  uint64              // monotonically increments   │
│    Origins     *OriginIndex        // field provenance (opt-in)  │
│    Cause       ReloadCause         // audit trail                │
└──────────────────────────────────────────────────────────────────┘

Reload pipeline 六个 stagedefaultStages[T]() 单点声明,commitPlan 共用同一组):

Stage 作用 失败行为
stageMerge file discovery → decode → kustomize-style overlay merge + RFC 6902 patch abort, keep old state
stageAssemble provider Load() → merge on top of file layers by Priority abort, keep old state
stageMigrate WithMigrations 回调,重写 map[string]any 键路径 abort, keep old state
stageTransform WithTransformers 列表(Defaults / EnvSubst / Aliases / DeletePaths / custom) abort, keep old state
stageDecode JSON round-trip → *T;填充 fastconf:"default=…" 结构体 tag abort, keep old state
stageValidate WithValidator[T] 函数列表;stagePolicy 紧跟其后评估 Policy abort, keep old state

模块拓扑(8 个 go.mod):

位置 角色 独立 go.mod
fastconf/ 主包:Manager / State / Option / Preset / Stage / Provenance / History / Tenant / AuditSink / Tracer ✅ 根模块
fastconf/contracts/ Provider / SnapshotProvider / Resumable / Event / Codec / Source / Priority 根模块内
fastconf/pkg/ 13 个可复用实现原语 根模块内
fastconf/{bus,render} 配置变更总线与模板渲染 根模块内
fastconf/providers/{vault,consul,http} 一线 Provider 实现(build tag 可裁剪) 根模块内
fastconf/validate/playground go-playground/validator 适配 根模块内
fastconf/otel OpenTelemetry Tracer 适配 ✅ 独立
fastconf/policy/{cue,opa} 策略引擎(CUE / OPA) ✅ 独立(各一个)
fastconf/metrics/prometheus Prometheus counter/histogram adapter ✅ 独立
fastconf/validate/cue/cuelang CUE Schema 校验后端 ✅ 独立
cmd/fastconfctl / cmd/fastconfgen 管理 CLI / 代码生成器 ✅ 独立(各一个)

1. 设计哲学

原则 含义
强类型读路径 Get() *T 杜绝 GetString("a.b.c") 这类字符串拼写错误;类型由编译器检查。
无锁热路径 Manager.Get() 是一条 atomic.Pointer.Load(),O(1)、零分配、零锁竞争。
失败安全 reload pipeline 任意 stage 失败 → 不替换 atomic.Pointer,用户侧 Get() 继续返回旧值。
单写者 只有一个 reload goroutine;所有触发源(fsnotify / provider event / ManualReload)通过 reloadCh 串行化,消除竞态。
Kustomize 风格叠加 conf.d/base + conf.d/overlays/<profile> 按文件名字典序叠加;支持 RFC 6902 JSON Patch(_patch.json)。
依赖最少的核心 主 go.mod 只依赖 yaml.v3 + json-patch + fsnotify + contracts;重依赖全部在独立 sub-module,用 build tag 或 go get 按需引入。
一个拼写,一个概念 Option 只有 WithXxx 形式;核心动词只有 New / Get / Reload / Watch / Plan / Snapshot / Close

2. 快速上手

2.1 最小示例
package main

import (
    "context"
    "log"

    "github.com/fastabc/fastconf/fastconf"
)

type AppConfig struct {
    Server struct {
        Addr string `json:"addr" yaml:"addr"`
    } `json:"server" yaml:"server"`
    Database struct {
        DSN  string `json:"dsn"  yaml:"dsn"`
        Pool int    `json:"pool" yaml:"pool"`
    } `json:"database" yaml:"database"`
}

func main() {
    cfg, err := fastconf.New[AppConfig](context.Background(),
        fastconf.WithDir("conf.d"),
        fastconf.WithProfileEnv("APP_PROFILE"),
        fastconf.WithDefaultProfile("dev"),
        fastconf.WithWatch(true),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer cfg.Close()

    app := cfg.Get()          // *AppConfig, O(1), lock-free, zero-alloc
    log.Println("addr:", app.Server.Addr)
    log.Println("pool:", app.Database.Pool)
}
2.2 目录约定
conf.d/
  base/
    00-app.yaml          # 基础默认值
    10-db.yaml
  overlays/
    dev/
      50-overrides.yaml  # dev 环境覆盖
    prod/
      50-overrides.yaml  # prod 环境覆盖
      _meta.yaml         # 可选元信息(match 表达式 / appendSlices 等)
    _patch.json          # RFC 6902 补丁(适用所有 profile)

APP_PROFILE=prod 时,FastConf 按 base/*.yaml(字典序)→ overlays/prod/*.yaml(字典序)叠加。

2.3 Profile 感知的配置示例
# conf.d/base/00-app.yaml
server:
  addr: ":8080"
database:
  pool: 10

# conf.d/overlays/prod/50-overrides.yaml
server:
  addr: ":8443"
database:
  pool: 32
// APP_PROFILE=prod → app.Server.Addr == ":8443", app.Database.Pool == 32
app := cfg.Get()
2.4 Codec bridge 说明

默认 decode bridge 走 JSON round-trip;如果现有结构体只有 yaml:"..." tag,请显式加 json:"..." tag,或使用 WithCodecBridge(fastconf.BridgeYAML) 回退旧行为。


3. 包结构

fastconf/
  manager.go           Manager[T] 核心:New / Get / Close / ManualReload
  reload_loop.go       单写者 reload goroutine + reloadCh 消费逻辑
  pipeline.go          runStages[T] + pipelineCtx[T] 定义
  pipeline_stages.go   6 个 Stage[T] 实现(Merge/Assemble/Migrate/Transform/Decode/Validate)
  pipeline_helpers.go  canonicalHashBytes / decodeInto / stateFromCtx 等辅助
  pipeline_plan.go     Plan(ctx) dry-run 路径
  options.go           所有 WithXxx Option + types(Transformer / MigrationApplier / …)
  state.go             State[T] + ReloadCause + Origins/Explain/Lookup + History
  watch.go             Watch / WatchTyped / fieldHasher / computePartHashes
  watcher.go           fsnotify startWatcher + parent-dir symlink 处理
  provider_watch.go    provider 事件订阅 + 指数退避 + drop-on-full
  presets.go           PresetK8s / PresetSidecar / PresetTesting
  registry.go          RegisterProviderFactory / WithProviderByName
  defaults.go          fastconf:"default=…" struct tag + constants
  secret.go            fastconf:"secret" + SecretRedactor
  errors.go            ErrFastConf sentinel + 所有命名错误
  obs_observability.go AuditSink / MetricsSink / Tracer(含 noop 实现)
  tenant.go            TenantManager[T]
  policy.go            WithPolicy[T] + PolicyError
  doc.go               package-level godoc

fastconf/pkg/
  debounce/            事件去抖动工具
  decoder/             YAML/JSON codec 注册表
  discovery/           conf.d 目录扫描 + _meta.yaml 解析
  fingerprint/         SHA-256 指纹计算
  mappath/             dotted-path Get/Set/Delete 工具
  merger/              Kustomize 风格 map[string]any 叠加
  migration/           Chain + Step(From/To/Apply struct)
  profile/             profile 表达式编译器(&/|/!/())
  provider/            内置 Env / CLI / Bytes / File Provider
  transform/           Defaults / SetIfAbsent / EnvSubst / DeletePaths / Aliases
  typeinfo/            共享 reflect walker + Cache[V]
  validate/            Validator + ValidatorReport
  watcher/             fsnotify 底层封装

fastconf/contracts/
  provider.go          Provider / Resumable / ErrResumeUnsupported
  event.go             Event(Change/Delete/Noop)
  snapshot.go          SnapshotProvider / Snapshot{Map,Revision,Stale}
  codec.go             Codec / RegisterCodecExt / CodecExtFunc
  source.go            Source(Name/Codec/Data)
  priority.go          Priority 常量(File/Env/CLI/Remote)

cmd/
  fastconfd/           Sidecar HTTP + SSE 服务(/healthz /version /config /events)
  fastconfctl/         管理 CLI(reload / plan / snapshot / rollback)
  fastconfgen/         配置结构体骨架代码生成器

4. 核心抽象

4.1 Manager[T] — 配置管理器
type Manager[T any] struct { /* unexported */ }

// 构造(首次 reload 同步执行)
func New[T any](ctx context.Context, opts ...Option) (*Manager[T], error)

// 读路径(lock-free, O(1), zero-alloc)
func (m *Manager[T]) Get() *T

// 写路径(触发 pipeline 异步执行;等待结果)
func (m *Manager[T]) Reload(ctx context.Context) error

// Dry-run(不更新指针;收集全部 ValidatorReport)
func (m *Manager[T]) Plan(ctx context.Context) (*PlanResult[T], error)

// 当前快照(State[T] + Sources + Origins)
func (m *Manager[T]) Snapshot() *State[T]

// 模块级订阅(只在指定 top-level 字段变化时触发)
func (m *Manager[T]) Watch(module string, fn func(*T)) (cancel func())

// 强类型订阅(M 为 T 的子结构;extractor 提取子视图)
func WatchTyped[T, M any](m *Manager[T], extract func(*T) M, fn func(M)) (cancel func())

// 历史与回滚
func (m *Manager[T]) History() []*State[T]
func (m *Manager[T]) Rollback(generation uint64) error
func (m *Manager[T]) PauseWatch()
func (m *Manager[T]) ResumeWatch()

// 手动触发(支持 requestID 追踪)
func (m *Manager[T]) ManualReload(ctx context.Context, requestID string) error

// 生命周期
func (m *Manager[T]) Close()
4.2 State[T] — 不可变快照
type State[T any] struct {
    Value      *T                  // 强类型业务结构体(Get() 直接返回)
    Hash       [32]byte            // 全局 SHA-256 指纹
    PartHashes map[string][32]byte // 每个 top-level 字段的 SHA-256
    LoadedAt   int64               // unix nanoseconds
    Sources    []SourceRef         // 参与本次合并的所有 layer
    Generation uint64              // 单调递增版本号
    Cause      ReloadCause         // 触发原因 + Revisions + RequestID
    // origins: 字段级来源追踪(ProvenanceTopLevel / ProvenanceFull 时填充)
}

// 字段来源追踪
func (s *State[T]) Explain(path string) []Origin        // 最老→最新的覆盖链
func (s *State[T]) Lookup(path string) []Origin         // 同 Explain
func (s *State[T]) LookupStrict(path string) ([]Origin, error)
func (s *State[T]) Origins() *OriginIndex
4.3 SourceRef — Layer 元信息
type SourceRef struct {
    Name     string    // 文件路径 / provider 名称
    Kind     LayerKind // LayerFile / LayerProvider / LayerBytes / LayerCLI / LayerEnv
    Priority int
    LoadedAt int64
}
4.4 ReloadCause — 触发原因审计
type ReloadCause struct {
    Reason    string            // "initial" / "watcher" / "provider:vault://…" / "manual"
    At        int64             // reload pipeline 启动时间(unix ns)
    Revisions map[string]string // 每个 provider 的 revision(Resumable WatchFrom 用)
    RequestID string            // ManualReload 传入的追踪 ID
    Tenant    string            // TenantManager 多租户标识
}

5. Option 参考

所有 WithXxx 函数都接受 Option 类型,可以任意组合传给 New[T]。后者按调用顺序 last-write-wins 应用。

5.1 文件系统
Option 说明 默认值
WithDir(dir string) 配置根目录 "conf.d"
WithFS(fs.FS) 替代 dir 的 fs.FS(测试用)
WithStrict(bool) 未知字段是否报错 false
WithLogger(*slog.Logger) 注入 logger slog.Default()
WithSlog(slog.Handler) 用 Handler 构建 logger
WithCodecBridge(codecBridge) BridgeJSON(默认)/ BridgeYAML BridgeJSON
WithMultiAxisOverlays(axes ...OverlayAxis) 多轴 Overlay(region / zone / host 等)
WithRawMapAccess(func(map[string]any)) 解码前读取完整 merged map(见 §5.1.1)
5.1.1 WithRawMapAccess — 原始 map 访问钩子

WithRawMapAccess 解决下游接入时 codec bridge 的两类类型不匹配问题:

场景 问题 解决方案
BridgeJSON + time.Duration 字段 YAML "30s" 被 JSON decoder 当作字符串,无法直接赋给 time.Duration(int64) 通过钩子拿到原始字符串,在 Validator 里手动解析
BridgeYAML + json.RawMessage 字段 YAML !!map 无法 unmarshal 到 json.RawMessage 钩子中取出子树,json.Marshal 后注入 Protocols 字段
var rawProto map[string]any

fastconf.New[Config](ctx,
    fastconf.WithRawMapAccess(func(root map[string]any) {
        // 钩子在所有 Transformer 之后、codec 解码之前调用
        // 不得持有 root 的引用,不得修改 root(若需修改请用 WithTransformers)
        if p, ok := root["protocols"].(map[string]any); ok {
            rawProto = p
        }
    }),
    fastconf.WithValidator(func(cfg *Config) error {
        if rawProto != nil {
            b, err := json.Marshal(rawProto)
            if err != nil {
                return err
            }
            cfg.Protocols = json.RawMessage(b)
        }
        return nil
    }),
)
5.2 Watch
Option 说明 默认值
WithWatch(bool) 启用 fsnotify false
WithDebounceInterval(time.Duration) 去抖动窗口 500ms
WithWatchPaths(paths ...string) 额外监视路径
5.3 Profile
Option 说明
WithProfile(p string) 显式设置单一 profile
WithProfiles(p ...string) 多 profile 模式(overlay 用 _meta.yaml.match 表达式匹配)
WithProfileEnv(name string) 从环境变量读取 profile
WithDefaultProfile(p string) 环境变量为空时的 fallback
WithProfileExpr(expr string) 全局 profile 匹配表达式(覆盖每个 overlay 的默认 membership 逻辑)
5.4 Provider
Option 说明
WithProvider(contracts.Provider) 注册外部 provider
WithEnvProvider(prefix string) 等价于 WithProvider(provider.NewEnv(prefix))
WithCLIProvider(data map[string]any) 等价于 WithProvider(provider.NewCLI(data))
WithBytes(name, codec string, data []byte) 内存 layer(测试 / quickstart)
WithSource(contracts.Source) 自描述 Source 对象
WithProviderByName(name string, cfg map[string]any) 通过 Factory Registry 按名称构建 provider
5.5 Pipeline 增强
Option 说明
WithTransformers(t ...Transformer) post-merge / pre-decode 变换链
WithMigrations(func(map[string]any) error) 模式迁移回调(在 Transformer 之前执行)
WithRawMapAccess(func(map[string]any)) 解码前只读钩子,访问完整 merged map
WithValidator[T](func(*T) error) decode 后的强类型校验(失败保留旧状态)
WithPolicy[T](policy.Policy[T]) validate 后的策略评估(SeverityError 中止 reload)
5.6 可观测性
Option 说明
WithMetrics(MetricsSink) 注入 metrics sink
WithAuditSink(AuditSink) 每次成功 reload 后回调(多个 sink fan-out)
WithTracer(Tracer) OTel-compatible span tracer
WithProvenance(ProvenanceLevel) ProvenanceOff(默认)/ ProvenanceTopLevel / ProvenanceFull
WithHistory(n int) 保留最近 n 个成功状态(History ring)
WithSecretRedactor(SecretRedactor) 日志和快照中的 secret 脱敏

6. Reload 流水线

6.1 触发源
                          ┌── fsnotify events → debounce 500ms ──┐
                          │                                       │
ManualReload(ctx, rid) ───┤    reloadCh chan reloadRequest       ├──► reloadLoop
                          │                                       │    (single writer)
provider.Watch events ────┘── backoff + drop-on-full ──────────┘
6.2 Pipeline 执行序列
reloadCh.recv(req)
  │
  ├─ stageMerge:    discovery.Scan(dir) → decode files → merger.Merge(layers)
  │                 apply _meta.yaml (appendSlices / profileEnv / match)
  │                 apply _patch.json (RFC 6902)
  │
  ├─ stageAssemble: for each provider: Load(ctx) → merge by Priority
  │
  ├─ stageMigrate:  opts.migrationRun.Migrate(merged)  [optional]
  │
  ├─ stageTransform: for each transformer: t.Transform(merged)
  │
  ├─ stageDecode:   json.Marshal(merged) → json.Unmarshal(→ *T)
  │                 apply fastconf:"default=…" struct tags
  │
  ├─ stageValidate: for each validator: v(*T)
  │   stagePolicy:  for each policy:    p.Evaluate(ctx, *T, reason, tenant)
  │
  └─ commit:
       canonicalHashBytes(mergedJSON) → SHA-256 dedup
       atomic.Pointer.Store(newState)
       history.push(newState)
       for each AuditSink: Audit(ctx, cause)
       fireWatches(oldPartHashes, newPartHashes)
6.3 失败保留语义

任意 stage 返回非 nil 错误时:

  • atomic.Pointer 不更新Get() 继续返回旧值;
  • Generation 不递增
  • 错误通过 reloadCh.reply 返回给调用方(ManualReload / Reload 可观察);
  • AuditSink 不调用(只有成功 commit 才触发 Audit);
  • MetricsSink.ReloadFinished(ok=false, dur) 被调用。

7. Profile 与 Overlay

7.1 目录结构
conf.d/
  base/                   # 所有 profile 共享的基础值
    00-defaults.yaml
    10-feature-flags.yaml
  overlays/
    dev/                  # 仅当 profile == "dev" 时叠加
      50-dev.yaml
    prod/
      50-prod.yaml
      _meta.yaml          # profile 匹配表达式
    staging/
      50-staging.yaml
      _meta.yaml
7.2 _meta.yaml 字段
schemaVersion: "1"
profileEnv: "APP_PROFILE"     # 读取 profile 的环境变量(优先级低于 WithProfileEnv)
defaultProfile: "dev"         # 兜底 profile
appendSlices: true            # slice 字段追加而非覆盖
match: "prod | staging"       # 布尔 profile 表达式(& | ! () 均支持)

match 表达式由 fastconf/pkg/profile 包编译,支持:

语法 含义
prod profile 集合包含 "prod"
prod | staging 包含 prod 或 staging
prod & !debug 包含 prod 且不包含 debug
(eu-west | eu-east) & !debug 复合表达式
7.3 RFC 6902 JSON Patch

在任意 overlay 目录下放 _patch.json,FastConf 在文件叠加完成后应用:

[
  { "op": "replace", "path": "/server/addr", "value": ":8443" },
  { "op": "add",     "path": "/feature/darkMode", "value": true },
  { "op": "remove",  "path": "/legacy/key" }
]
7.4 多 Profile 模式
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithProfiles("prod", "eu-west", "canary"),
    // overlays 目录:match: "prod" 的会被包含;match: "canary" 的也会被包含
)

WithProfilesWithProfile 互斥;多 profile 模式下每个 overlay 的 _meta.yaml.match 用于判断是否包含。


8. Provider 系统

8.1 内置 Provider
Provider 构造 说明
Env provider.NewEnv("APP_") 读取 APP_FOO_BARfoo.bar_.,前缀去掉)
CLI provider.NewCLI(map[string]any) 直接传入 map;来自命令行 flag 解析结果
Bytes WithBytes(name, codec, data) 内存 layer;测试最常用
File provider.NewFile(path, codec) 读取单个文件
8.2 contracts.Provider 接口
type Provider interface {
    Name()     string
    Priority() int

    // Load 返回一次性快照(reload 时被调用)。
    Load(ctx context.Context) (map[string]any, error)

    // Watch 返回变更事件 channel;无通知能力的 provider 返回 (nil, nil)。
    Watch(ctx context.Context) (<-chan Event, error)
}
8.3 Priority 常量
const (
    PriorityFile     = 100  // 文件系统 layer 默认
    PriorityEnv      = 200  // Env provider 默认
    PriorityCLI      = 300  // CLI provider 默认(最高,用户显式覆盖)
    PriorityRemote   = 150  // Vault / Consul 建议区间
)

较高 Priority 的 provider 层覆盖较低 Priority 的层。

8.4 Resumable(断点续订)
type Resumable interface {
    // lastRev 为空时等价于 Watch(冷订阅)。
    // lastRev 非空时从该 revision 之后的变更开始推送。
    // 若 revision 已被压缩,返回 ErrResumeUnsupported,框架回退到冷订阅。
    WatchFrom(ctx context.Context, lastRev string) (<-chan Event, error)
}

框架自动记忆每个 provider 最后观测到的 Event.Revision,在断线重连时传给 WatchFrom

8.5 Provider Factory Registry
// 注册(通常在 init() 或 TestMain 中)
fastconf.RegisterProviderFactory("vault", func(cfg map[string]any) (contracts.Provider, error) {
    addr := cfg["addr"].(string)
    path := cfg["path"].(string)
    return vault.New(addr, path)
})

// 使用(支持从 YAML 配置中读取 provider 配置)
mgr, err := fastconf.New[AppConfig](ctx,
    fastconf.WithProviderByName("vault", map[string]any{
        "addr": "https://vault.svc",
        "path": "kv/data/myapp",
    }),
)
8.6 内置 Provider 包

这三个 Provider 是根模块内的包(无独立 go.mod),随根模块版本一起发布。直接 go get 根模块即可:

go get github.com/fastabc/fastconf@v0.8.1
Provider 包路径 构造示例
Vault (KV v2) fastconf/providers/vault vault.New(vault.Options{...})
Consul KV fastconf/providers/consul consul.New(consul.Options{...})
HTTP/SSE fastconf/providers/http httpprovider.New(url)

使用 build tag 可在编译时裁剪不需要的 Provider:

go build -tags no_provider_vault,no_provider_consul,no_provider_http ./...

9. Transformer 与 Migration

9.1 Transformer 接口
type Transformer interface {
    Transform(root map[string]any) error
    Name() string
}

Transformer 在 merge 完成、decode 开始之前运行,接收 map[string]any,可以安全地修改树结构。

9.2 内置 Transformer
import "github.com/fastabc/fastconf/fastconf/pkg/transform"

fastconf.WithTransformers(
    // 把 defaults map 中的缺失键填入树(递归合并,不覆盖已有值)
    transform.Defaults(map[string]any{
        "server": map[string]any{"timeout": "30s"},
    }),

    // 只在路径不存在时设置单个值
    transform.SetIfAbsent("server.timeout", "30s"),

    // 替换字符串中的 ${VAR} / ${VAR:-default}
    transform.EnvSubst(),

    // 删除指定路径
    transform.DeletePaths("internal.debug", "legacy.key"),

    // 旧路径 → 新路径重命名(目标已有值时不覆盖)
    transform.Aliases(map[string]string{
        "db.url":      "database.dsn",
        "server.port": "server.addr",
    }),

    // 自定义 Transformer
    transform.TransformerFunc{
        NameStr: "my-custom",
        Fn: func(root map[string]any) error {
            root["injected"] = "value"
            return nil
        },
    },
)
9.3 fastconf:"default=…" 结构体 Tag
type AppConfig struct {
    Server struct {
        Addr    string        `json:"addr"    fastconf:"default=:8080"`
        Timeout time.Duration `json:"timeout" fastconf:"default=30s"`
    } `json:"server"`
    LogLevel string `json:"log_level" fastconf:"default=info"`
}

struct tag default 在 stageDecode 之后、进入 stageValidate 之前应用,只填充零值字段。

9.4 fastconf:"secret" Tag
type AppConfig struct {
    Database struct {
        DSN string `json:"dsn" fastconf:"secret"` // 日志 / 快照中自动脱敏
    } `json:"database"`
}

mgr, err := fastconf.New[AppConfig](ctx,
    fastconf.WithSecretRedactor(fastconf.DefaultRedactor), // 替换为 "***"
)
9.5 Migration(模式迁移)
fastconf.WithMigrations(func(root map[string]any) error {
    // 将旧版 "db_url" 迁移到新版 "database.dsn"
    if v, ok := root["db_url"]; ok {
        db, _ := root["database"].(map[string]any)
        if db == nil {
            db = map[string]any{}
            root["database"] = db
        }
        if _, exists := db["dsn"]; !exists {
            db["dsn"] = v
        }
        delete(root, "db_url")
    }
    return nil
})

对于多版本迁移链,使用 pkg/migration.Chain

import "github.com/fastabc/fastconf/fastconf/pkg/migration"

chain := migration.NewChain(
    migration.Step{From: "1", To: "2", Apply: migrateV1toV2},
    migration.Step{From: "2", To: "3", Apply: migrateV2toV3},
)
fastconf.WithMigrations(chain.Migrate)

10. Watch 热重载

10.1 文件系统 Watch
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithWatch(true),
    fastconf.WithDebounceInterval(500*time.Millisecond), // default
)
// K8s ConfigMap symlink 原子交换(..data)被父目录 fsnotify 正确处理
10.2 模块级订阅
// 只在 T.Server 字段发生变化时触发(基于 PartHashes SHA-256 diff)
cancel := cfg.Watch("server", func(app *AppConfig) {
    log.Println("server config changed:", app.Server.Addr)
})
defer cancel()

Watch callback 同步执行于 reload goroutine,recover() 隔离 panic。长耗时操作请自行 go func() 异步处理。

10.3 强类型订阅
// WatchTyped 提取子视图,仅在 extractor 返回值变化时触发
cancel := fastconf.WatchTyped(cfg,
    func(app *AppConfig) AppConfig.DatabaseConfig {
        return app.Database
    },
    func(db AppConfig.DatabaseConfig) {
        reconnect(db.DSN)
    },
)
defer cancel()
10.4 手动触发
// ManualReload 等待 reload 完成,支持追踪 ID
if err := cfg.ManualReload(ctx, "admin-triggered-reload-001"); err != nil {
    log.Println("reload failed:", err)
}
10.5 PauseWatch / ResumeWatch
cfg.PauseWatch()
// 在 pause 期间,文件变化不触发 reload
applyBatchUpdate()
cfg.ResumeWatch()

11. Provenance、History 与 Rollback

11.1 Provenance(字段来源追踪)
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithProvenance(fastconf.ProvenanceFull), // Off / TopLevel / Full
)

state := cfg.Snapshot()

// 获取 "server.addr" 字段的完整覆盖链(oldest → newest)
origins := state.Explain("server.addr")
for _, o := range origins {
    fmt.Printf("layer=%s priority=%d value=%v\n", o.Source.Name, o.Source.Priority, o.Value)
}

// 严格查询(区分"未开启 provenance"和"路径不存在")
origins, err := state.LookupStrict("database.dsn")

ProvenanceLevel 对比:

Level 开销 能追踪什么
ProvenanceOff
ProvenanceTopLevel O(top-level keys) 每个顶层字段最终来自哪个 layer
ProvenanceFull O(leaves) 每个叶子字段的完整覆盖链 + 每层的原始值
11.2 History Ring
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithHistory(10), // 保留最近 10 次成功状态
)

// 获取历史快照列表(从旧到新)
history := cfg.History() // []*State[T]
for _, s := range history {
    fmt.Printf("gen=%d loaded_at=%d hash=%x\n",
        s.Generation, s.LoadedAt, s.Hash[:4])
}
11.3 Rollback
// 回滚到指定 Generation(Generation 来自 State[T].Generation)
if err := cfg.Rollback(targetGeneration); err != nil {
    log.Println("rollback failed:", err)
}

Rollback 把历史 *State[T] 重新发布到 atomic.Pointer,不重新执行 pipeline,不增加 Generation,但会触发 Watch callback(如果 PartHashes 不同)。


12. 可观测性

12.1 AuditSink
// 每次成功 reload 后被调用(在 reload goroutine 中同步执行)
type AuditSink interface {
    Audit(ctx context.Context, cause ReloadCause) error
}

// 内置实现:写 JSON 行到 io.Writer
sink := fastconf.NewJSONAuditSink(os.Stderr)
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithAuditSink(sink),
    // 多个 sink fan-out
    fastconf.WithAuditSink(myRemoteSink),
)

输出示例:

{"reason":"watcher","at":"2026-05-14T08:00:00Z","revisions":{"vault":"42"}}
12.2 MetricsSink
type MetricsSink interface {
    ReloadStarted()
    ReloadFinished(ok bool, dur time.Duration)
    // 可选扩展接口:ProviderMetricsSink / StageMetricsSink
}

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithMetrics(myPromMetrics),
)

Prometheus 实现在 fastconf/metrics/prometheus sub-module:

import prommetrics "github.com/fastabc/fastconf/fastconf/metrics/prometheus"

cfg, _ := fastconf.New[AppConfig](ctx,
    fastconf.WithMetrics(prommetrics.New()),
)
12.3 Tracer(OTel)
// 默认 noopTracer;-tags fastconf_otel 启用 enrichSpan 属性注入
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithTracer(myOtelTracer),
)

OTel SDK 集成在 fastconf/otel sub-module(独立 go.mod,不污染主包依赖):

import fastconfotel "github.com/fastabc/fastconf/fastconf/otel"

tracer := fastconfotel.NewTracer(otel.GetTracerProvider())
cfg, _ := fastconf.New[AppConfig](ctx,
    fastconf.WithTracer(tracer),
)
12.4 Policy(策略引擎)
import "github.com/fastabc/fastconf/fastconf/policy"

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithPolicy(policy.Func[AppConfig]{
        N: "deny-debug-in-prod",
        Fn: func(_ context.Context, in policy.Input[AppConfig]) ([]policy.Violation, error) {
            if in.Config.Env == "prod" && in.Config.Debug {
                return []policy.Violation{{
                    Rule:     "deny-debug-in-prod",
                    Path:     "debug",
                    Message:  "debug mode must be false in prod",
                    Severity: policy.SeverityError, // 中止 reload
                }}, nil
            }
            return nil, nil
        },
    }),
)

CUE 和 OPA 实现在独立 sub-module:

// CUE 策略
import cuepolicy "github.com/fastabc/fastconf/fastconf/policy/cue"

// OPA 策略
import opapolicy "github.com/fastabc/fastconf/fastconf/policy/opa"

Violation Severity:

Severity Plan 行为 Reload 行为
SeverityWarning 记录警告,继续 记录警告,继续
SeverityError 降级为警告,继续(dry-run) 中止 reload,保留旧状态

13. 多租户(TenantManager)

tm := fastconf.NewTenantManager[AppConfig]()

// 注册 tenant(每个 tenant 有独立的 Manager,故障互不影响)
mgr, err := tm.Add(ctx, "tenant-a",
    fastconf.WithDir("/etc/config/tenant-a"),
    fastconf.WithProfileEnv("TENANT_A_PROFILE"),
)

mgr, err = tm.Add(ctx, "tenant-b",
    fastconf.WithDir("/etc/config/tenant-b"),
    fastconf.WithProvider(tenantBVaultProvider),
)

// 读取指定 tenant 的配置
app, err := tm.Get("tenant-a") // *AppConfig, error
if err != nil {
    // fastconf.ErrUnknownTenant
}

// 移除 tenant(调用内部 Manager.Close())
_ = tm.Remove("tenant-a")

// 关闭所有 tenant
tm.Close()

TenantManager 不跨 tenant 共享任何 Option,每个 tenant 完全隔离。AuditSink 自动注入 Cause.Tenant = id


14. Preset 与 Cookbook

14.1 Preset:快速组合常用 Option

PresetK8s — Kubernetes ConfigMap 标准部署:

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.PresetK8s(fastconf.K8sOpts{
        Dir:        "/etc/config",    // ConfigMap 挂载路径
        ProfileEnv: "APP_PROFILE",    // 从 Pod env 读取 profile
        Default:    "default",        // 默认 profile
        Watch:      true,             // 开启 fsnotify 热重载
    }),
    fastconf.WithStrict(false), // 覆盖 Preset 的 strict=true
)

PresetSidecarfastconfd Sidecar 守护进程:

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.PresetSidecar(fastconf.SidecarOpts{
        Dir:      "/etc/fastconfd",
        HistoryN: 16,        // SSE 消费者可回放最近 16 个版本
        Watch:    true,
        Strict:   false,
    }),
)

PresetTesting — 单元测试(内存 layer,零文件 I/O):

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.PresetTesting(fastconf.TestingOpts{
        Data: map[string]any{
            "server": map[string]any{"addr": ":9090"},
        },
    }),
)
14.2 Cookbook

场景 1:环境变量覆盖数据库密码

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithEnvProvider("APP_"),     // APP_DATABASE_DSN → database.dsn
    fastconf.WithSecretRedactor(fastconf.DefaultRedactor),
)

场景 2:Vault 动态密钥 + 热更新

import vault "github.com/fastabc/fastconf/fastconf/providers/vault"

vp, _ := vault.New(vault.Options{
    Addr:   "https://vault.svc",
    Path:   "kv/data/myapp",
    Token:  os.Getenv("VAULT_TOKEN"),
})
cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithProvider(vp),
    fastconf.WithWatch(true),
)

场景 3:cross-field 校验

cfg, err := fastconf.New[AppConfig](ctx,
    fastconf.WithDir("conf.d"),
    fastconf.WithValidator(func(app *AppConfig) error {
        if app.TLS.Enabled && app.TLS.CertFile == "" {
            return errors.New("tls.cert_file is required when tls.enabled=true")
        }
        return nil
    }),
)

场景 4:Plan 预演(CI/CD 检查)

result, err := cfg.Plan(ctx)
if err != nil {
    log.Fatal("plan failed:", err)
}
for _, report := range result.ValidatorReports {
    log.Printf("validator %s: %v\n", report.Name, report.Issues)
}

场景 5:订阅数据库子模块并自动重连

fastconf.WatchTyped(cfg,
    func(app *AppConfig) string { return app.Database.DSN },
    func(dsn string) {
        go dbPool.Reconnect(dsn)
    },
)

15. Sub-module 生态矩阵

根模块内置包(随根模块版本发布,import 路径不变)
路径 说明
contracts fastconf/contracts Provider / Codec / Source / Event 接口定义
http fastconf/providers/http HTTP / SSE Provider(build tag: no_provider_http 可裁剪)
vault fastconf/providers/vault HashiCorp Vault KV v2 Provider(build tag: no_provider_vault 可裁剪)
consul fastconf/providers/consul Consul KV Provider(build tag: no_provider_consul 可裁剪)
playground fastconf/validate/playground go-playground/validator 校验适配
bus fastconf/bus 配置变更事件总线
render fastconf/render 模板渲染扩展
独立 Sub-module(需单独 go get,含重型依赖)
Sub-module 路径 主要依赖 说明
prometheus fastconf/metrics/prometheus prometheus/client_golang ~10 MB Prometheus 指标适配
otel fastconf/otel OpenTelemetry SDK ~15 MB OpenTelemetry Tracer 适配
cue-policy fastconf/policy/cue cuelang.org/go ~50 MB CUE 策略引擎实现
opa-policy fastconf/policy/opa open-policy-agent/opa ~100 MB+ Open Policy Agent 实现
cue-validate fastconf/validate/cue/cuelang cuelang.org/go ~50 MB CUE Schema 校验后端

独立 sub-module 用 go get github.com/fastabc/fastconf/fastconf/<name> 按需拉取,不影响根模块依赖图。

统一打 tag(使用 scripts/tag-release.sh

# 为所有模块打本地 tag(8 个模块一次完成)
./scripts/tag-release.sh v0.8.1

# 同时推送到 origin
./scripts/tag-release.sh v0.8.1 --push

# 强制覆盖已存在的 tag(本地 + 推送时同步删除远端)
./scripts/tag-release.sh v0.8.1 --force --push

Build tag 裁剪(根模块内置 Provider 二进制瘦身):

# 排除 vault、consul、http provider(减少编译产物)
go build -tags no_provider_vault,no_provider_consul,no_provider_http ./...

16. 扩展指南

16.1 实现自定义 Provider
// 实现 contracts.Provider 接口
type RedisProvider struct {
    client *redis.Client
    key    string
    ch     chan contracts.Event
}

func (p *RedisProvider) Name()     string { return "redis:" + p.key }
func (p *RedisProvider) Priority() int    { return contracts.PriorityRemote }

func (p *RedisProvider) Load(ctx context.Context) (map[string]any, error) {
    data, err := p.client.Get(ctx, p.key).Bytes()
    if err != nil {
        return nil, err
    }
    var result map[string]any
    return result, json.Unmarshal(data, &result)
}

func (p *RedisProvider) Watch(ctx context.Context) (<-chan contracts.Event, error) {
    go p.watchLoop(ctx)
    return p.ch, nil
}

// 注册工厂(init() 中调用)
func init() {
    fastconf.RegisterProviderFactory("redis", func(cfg map[string]any) (contracts.Provider, error) {
        addr, _ := cfg["addr"].(string)
        key, _ := cfg["key"].(string)
        return NewRedisProvider(addr, key)
    })
}
16.2 实现自定义 Transformer
// 满足 fastconf.Transformer = transform.Transformer 接口
type PrefixTransformer struct {
    Prefix string
}

func (t PrefixTransformer) Name() string { return "prefix:" + t.Prefix }

func (t PrefixTransformer) Transform(root map[string]any) error {
    if v, ok := root["app_name"].(string); ok {
        root["app_name"] = t.Prefix + "-" + v
    }
    return nil
}

// 使用
fastconf.WithTransformers(PrefixTransformer{Prefix: "myorg"})
16.3 实现自定义 Codec
import "github.com/fastabc/fastconf/fastconf/contracts"

// 注册 TOML codec(在 init() 中调用)
contracts.RegisterCodecExt("toml", contracts.CodecExtFunc(func(data []byte) (map[string]any, error) {
    var result map[string]any
    return result, toml.Unmarshal(data, &result)
}))

// 之后 conf.d/base/config.toml 会被自动识别并解码
16.4 依赖方向约束
fastconf  →  fastconf/pkg/discovery
          →  fastconf/pkg/decoder
          →  fastconf/pkg/merger
          →  fastconf/pkg/provider
          →  fastconf/pkg/watcher
          →  fastconf/pkg/fingerprint
          →  fastconf/pkg/validate
          →  fastconf/contracts

fastconf/pkg/* 之间不得相互依赖(仅可依赖标准库与 pkg/debounce 这类纯工具包)。
例外:pkg/discovery 依赖 pkg/profile;
      pkg/provider 依赖 pkg/decoder。

CI 通过 go list -deps 自动校验上述方向,违反即 fail。


17. CLI 工具

17.1 fastconfd — Sidecar 服务
fastconfd --dir=/etc/config --profile=prod --addr=:8081

HTTP 端点:

端点 方法 说明
/healthz GET 健康检查;{"status":"ok","generation":N}
/version GET 当前 State 版本(Hash + Generation)
/config GET 当前配置 JSON(secret 字段已脱敏)
/reload POST 触发手动 reload;接受 {"request_id":"…"}
/events GET SSE 流;每次成功 reload 推送 ReloadCause JSON
17.2 fastconfctl — 管理 CLI
# 显示当前配置快照
fastconfctl snapshot --addr=:8081

# 触发 reload(等待完成)
fastconfctl reload --addr=:8081 --request-id=deploy-123

# 执行 Plan dry-run 并输出 ValidatorReport
fastconfctl plan --addr=:8081

# 回滚到指定 Generation
fastconfctl rollback --addr=:8081 --generation=42

# 查看 sources(参与合并的所有 layer)
fastconfctl sources --addr=:8081
17.3 fastconfgen — 代码生成器
# 从 YAML 文件生成 Go 结构体骨架
fastconfgen generate --input=conf.d/base/00-app.yaml --pkg=config --out=config/config_gen.go

18. 命令速查

# 拉取依赖
go mod tidy

# 构建全部
make build
# 或
go build ./...

# 测试(含 race detector)
make test
# 或
go test -race -count=1 ./...

# 测试(包含 cmd/ 子模块)
make test-all

# Lint(需要 golangci-lint)
make lint

# 性能基准(BenchmarkGet ≈ 0.43 ns/op, 0 alloc)
go test -bench=BenchmarkGet -benchmem ./fastconf/...

# 生成依赖关系图
make graph
# 输出:docs/graph/{deps.dot,deps.svg,owners.md}

# CI 防线脚本
bash scripts/bench-guard.sh       # ns/op + allocs 阈值检查
bash scripts/loc-budget.sh        # fastconf/ 主包 ≤3,700 LOC
bash scripts/total-loc-budget.sh  # 全树 ≤8,800 LOC
bash scripts/audit-phase-comments.sh --strict  # Phase NN 考古注释检查

快速链接


License

MIT

Directories

Path Synopsis
cmd
fastconfd command
fastconfd is the Phase 26 sidecar daemon.
fastconfd is the Phase 26 sidecar daemon.
internal/cli
Package cli centralises the FastConf command-line flag set so every cmd/* binary registers -dir / -profile / -strict / -watch with identical defaults and semantics, and constructs the Manager via a single canonical path.
Package cli centralises the FastConf command-line flag set so every cmd/* binary registers -dir / -profile / -strict / -watch with identical defaults and semantics, and constructs the Manager via a single canonical path.
Package fastconf 提供基于 Go 1.26 泛型的强类型、无锁化、Kustomize 风格的 动态配置加载叠加框架。
Package fastconf 提供基于 Go 1.26 泛型的强类型、无锁化、Kustomize 风格的 动态配置加载叠加框架。
bus
Package bus provides a small message-bus abstraction for FastConf.
Package bus provides a small message-bus abstraction for FastConf.
contracts
Package contracts is the **public, stable** surface of FastConf interfaces.
Package contracts is the **public, stable** surface of FastConf interfaces.
pkg/debounce
Package debounce provides a single-writer trailing-edge debouncer used by the watcher subsystem to coalesce bursty filesystem events into one reload trigger.
Package debounce provides a single-writer trailing-edge debouncer used by the watcher subsystem to coalesce bursty filesystem events into one reload trigger.
pkg/decoder
Package decoder 把不同编码(yaml/json/...)的字节流解码为统一的 map[string]any 中间表示。
Package decoder 把不同编码(yaml/json/...)的字节流解码为统一的 map[string]any 中间表示。
pkg/discovery
Package discovery 扫描配置目录并产出按优先级排序的 layer 流。
Package discovery 扫描配置目录并产出按优先级排序的 layer 流。
pkg/fingerprint
Package fingerprint hosts reusable fingerprinting primitives for FastConf.
Package fingerprint hosts reusable fingerprinting primitives for FastConf.
pkg/mappath
Package mappath provides dotted-path helpers for map[string]any trees.
Package mappath provides dotted-path helpers for map[string]any trees.
pkg/merger
Package merger 实现 Kustomize 风格的 map[string]any 深度合并。
Package merger 实现 Kustomize 风格的 map[string]any 深度合并。
pkg/migration
Package migration lets FastConf rewrite the merged map from one schema version to another before it is decoded into the strongly typed snapshot.
Package migration lets FastConf rewrite the merged map from one schema version to another before it is decoded into the strongly typed snapshot.
pkg/profile
Package profile implements FastConf's tiny boolean profile-expression language (Phase 13).
Package profile implements FastConf's tiny boolean profile-expression language (Phase 13).
pkg/provider
Package provider abstracts external configuration sources (env, CLI, KV, Vault, ...).
Package provider abstracts external configuration sources (env, CLI, KV, Vault, ...).
pkg/transform
Package transform provides composable, post-merge / pre-decode transformations on the merged configuration tree.
Package transform provides composable, post-merge / pre-decode transformations on the merged configuration tree.
pkg/typeinfo
Package typeinfo provides a single, cached reflect.Type walker for FastConf's per-T metadata extraction (secret paths, default tags, top-level field hashers).
Package typeinfo provides a single, cached reflect.Type walker for FastConf's per-T metadata extraction (secret paths, default tags, top-level field hashers).
pkg/validate
Package validate hosts reusable validation primitives for FastConf.
Package validate hosts reusable validation primitives for FastConf.
pkg/watcher
Package watcher subscribes to filesystem changes and triggers reloads.
Package watcher subscribes to filesystem changes and triggers reloads.
policy
Package policy defines the Phase 23 policy interface.
Package policy defines the Phase 23 policy interface.
providers/consul
Package consul is a first-party Consul KV provider for FastConf.
Package consul is a first-party Consul KV provider for FastConf.
providers/http
Package http is a first-party HTTP/HTTPS provider for FastConf.
Package http is a first-party HTTP/HTTPS provider for FastConf.
providers/vault
Package vault is a first-party HashiCorp Vault KV v2 provider for FastConf.
Package vault is a first-party HashiCorp Vault KV v2 provider for FastConf.
render
Package render plugs FastConf into the long tail of legacy daemons that only consume on-disk configuration files (nginx.conf, envoy.yaml, postgresql.conf, ...).
Package render plugs FastConf into the long tail of legacy daemons that only consume on-disk configuration files (nginx.conf, envoy.yaml, postgresql.conf, ...).
validate/playground
Package playground adapts go-playground/validator/v10 to the FastConf validator contract: a function (*T) error called by Manager.commit before publishing a new state.
Package playground adapts go-playground/validator/v10 to the FastConf validator contract: a function (*T) error called by Manager.commit before publishing a new state.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL