Documentation
¶
Overview ¶
Package sources runs the commands a base declares and files what they print.
There is no provider code in fkf. A source is an entry in the base's own fkf.yaml: a command that prints JSON, and jq-subset paths naming the fields worth joining on. That choice is what removes credentials from this program entirely — the CLI a source names already holds the login — and it is why adding a source is a YAML pull request.
Both `run:` and `body:` are direct argv. A base helper's shebang chooses its interpreter; fkf never reparses arguments as shell syntax. `run:` receives only values fkf computes, while `body:` may receive charset-checked values from one stored record.
Index ¶
- Constants
- Variables
- func CollectWindow(ctx context.Context, runner Runner, source *core.Source, env Environment, ...) (map[string]*Document, error)
- func DecodeFragment(fragment string) (string, error)
- func DescribePolicy(source *core.Source) string
- func EncodeDocument(document *Document) ([]byte, error)
- func EncodeFragment(id string) string
- func EnsureBinDir(root string) error
- func EventDocumentURI(date, source string) string
- func IndexDocumentURI(source string) string
- func ParseDay(value string) (time.Time, error)
- func ParseRecordTime(value string) (time.Time, error)
- func VerifyRecords(document *Document) error
- func WriteDocument(path string, document *Document) error
- type Command
- type Document
- func Collect(ctx context.Context, runner Runner, source *core.Source, env Environment, ...) (*Document, error)
- func DecodeDocument(data []byte, path string) (*Document, error)
- func DecodeDocumentContext(ctx context.Context, data []byte, path string) (*Document, error)
- func ReadDocument(path string) (*Document, error)
- func ReadDocumentContext(ctx context.Context, path string) (*Document, error)
- type Environment
- type Fields
- type Pacer
- type PolicyRunner
- type Record
- type Runner
- type RunnerFunc
- type Schema
- type Window
Constants ¶
const SchemaVersion = 1
SchemaVersion is the permanent v1 evidence-envelope marker. Configuration and stored documents deliberately use the same fkf: 1 marker value; the containing file identifies which contract applies. Evolution inside this marker is additive: older readers ignore fields they do not know, and newer readers accept documents that omit later optional fields. Generated caches remain disposable, but collected evidence must not depend on a provider retaining enough history to re-create it.
Variables ¶
var ErrCivilDateDoesNotExist = errors.New("civil date does not exist in the local timezone")
ErrCivilDateDoesNotExist reports a date label skipped entirely by the local timezone. It is distinct from malformed input so planners can step over the missing label and continue to the previous completed civil day.
var ErrIncomplete = errors.New("collection is incomplete")
ErrIncomplete marks every completeness failure, so `sync` can report the day as failed without inspecting messages.
var ErrUnknownSchema = errors.New("unknown document schema")
ErrUnknownSchema reports a document from a different generation.
Functions ¶
func CollectWindow ¶
func CollectWindow( ctx context.Context, runner Runner, source *core.Source, env Environment, rangeWindow Window, dates []string, timeout time.Duration, now time.Time, ) (map[string]*Document, error)
CollectWindow runs a `window: true` source ONCE for the whole requested range and partitions what it returns into one Document per requested day, bucketed by each record's own declared `fields.time`. It is the range analogue of Collect: the same completeness rule applies to the whole range rather than to one day, because the command that produced every record ran once — a record whose time falls outside every requested day fails the range entirely, the same way a record with no identity fails a day today.
dates must be requested in ascending order and represent a contiguous, gap-free span: the caller (services/sync.go) derives {{start}}/{{end}} from the first and last day, so a day silently missing from the middle of dates would still be covered by the command's own output and its records would have nowhere to land.
func DecodeFragment ¶
DecodeFragment reverses EncodeFragment.
func DescribePolicy ¶
DescribePolicy renders a source's invocation policy for the trust listing, or "" when it declares none. The review has to say how a command will be invoked, not only what it is.
func EncodeDocument ¶
EncodeDocument renders a document for storage: two-space indented JSON with a trailing newline, byte-identical for identical input.
func EncodeFragment ¶
EncodeFragment percent-encodes a record identity for a URI fragment, leaving the characters that read naturally in an owner/name or an email address. It is the inverse of DecodeFragment.
func EnsureBinDir ¶
EnsureBinDir creates the base's script directory so a `run:` line that calls a helper does not fail on a base scaffolded before that preset existed.
func EventDocumentURI ¶
EventDocumentURI is where an events source's day is filed.
func IndexDocumentURI ¶
IndexDocumentURI is where an index source's snapshot is filed.
func ParseDay ¶
ParseDay reads a YYYY-MM-DD day in the local zone, anchored at noon rather than midnight. No real transition falls at noon, so the returned time's calendar date is always exactly the one requested — which a direct midnight parse cannot promise, since a nonexistent local midnight is what DayWindow's own fix exists to route around. Every caller already reduces this to a date label or hands it straight to DayWindow, so the shifted hour is invisible.
func ParseRecordTime ¶
ParseRecordTime reads the timestamp at a source's declared `time` path. Epoch values are accepted because provider CLIs and local databases emit seconds, milliseconds, microseconds, and nanoseconds. Consecutive powers of 1000 are distinguishable by magnitude for every instant between 1973 and the year 5138.
func VerifyRecords ¶
VerifyRecords enforces the rules that make a record addressable: it has the identity its source declared, that identity is unique within the document, and — for a dated document — a timestamp fkf can read and that belongs to the civil day the document declares.
func WriteDocument ¶
WriteDocument replaces a document atomically, so a reader sees the previous complete document or the new one and a crash never leaves a partial day that a skip-if-exists run would later treat as collected.
Types ¶
type Command ¶
type Command struct {
Argv []string `json:"argv"`
Dir string `json:"dir"`
ForbiddenRoot string `json:"-"`
Stdin string `json:"-"`
Env map[string]string `json:"env,omitempty"`
Timeout time.Duration `json:"timeout"`
}
Command is one execution request. It is a value so a test can assert exactly what would have run, which is what makes `--dry-run` verifiable rather than merely plausible.
func BuildRunCommand ¶
func BuildRunCommand(source *core.Source, env Environment, window Window, timeout time.Duration) Command
BuildRunCommand substitutes fkf-owned placeholders in each declared argument and returns exactly what exec receives. No shell parses the result.
func FetchBody ¶
func FetchBody( ctx context.Context, runner Runner, source *core.Source, fields Fields, env Environment, record Record, timeout time.Duration, ) (string, Command, error)
FetchBody runs a source's `body:` command for one record. Every placeholder value comes from collected data, so each is checked against the charset before exec and the argv is passed to the kernel without a shell. The output is printed and never stored: a body is evidence to read once, not a second copy of the provider's database.
type Document ¶
type Document struct {
FKF int `json:"fkf"`
Source string `json:"source"`
Layer core.Layer `json:"layer"`
Date string `json:"date,omitempty"`
WindowStart string `json:"window_start,omitempty"`
WindowEnd string `json:"window_end,omitempty"`
CollectedAt string `json:"collected_at"`
Schema Schema `json:"schema"`
Fields Fields `json:"fields"`
Body bool `json:"body"`
Count int `json:"count"`
Records []Record `json:"records"`
}
Document is one complete collection: every record a source produced for one day, or the point-in-time snapshot of an index source. It is complete or absent, never partial.
func Collect ¶
func Collect( ctx context.Context, runner Runner, source *core.Source, env Environment, window Window, timeout time.Duration, now time.Time, ) (*Document, error)
Collect runs one source for one window and returns the document it would file. It writes nothing: the caller decides where a complete document goes, which is what keeps the runner pure enough to golden-test.
func DecodeDocument ¶
DecodeDocument parses a stored document and refuses an unrecognised schema marker.
func DecodeDocumentContext ¶
DecodeDocumentContext is DecodeDocument with cooperative cancellation during JSON parsing. Stored documents are bounded but can be large enough that a cancelled status or retrieval request must stop before decoding all records.
func ReadDocument ¶
ReadDocument loads one stored document under the whole-file bound.
func ReadDocumentContext ¶
ReadDocumentContext loads and decodes one bounded stored document with cancellation.
func (*Document) FindRecord ¶
FindRecord returns the record whose declared identity matches.
type Environment ¶
Environment is the machine-local context a base's commands run in.
func NewEnvironment ¶
func NewEnvironment(config *core.Config) Environment
NewEnvironment resolves the execution context from a base's configuration. The base's own bin/ comes first on PATH so a preset's helper script is reachable without installation, and the declared bin: entries follow — both ahead of the inherited PATH, because a base is allowed to pin the tool its own sources expect. Inherited entries below the base, including symlinks back into it, are removed first: only the separately prepended, trust-digested bin/ may make repository content executable.
func (Environment) LookPath ¶
func (e Environment) LookPath(name string) (string, bool)
LookPath resolves a command name against the PATH this environment will actually give the subprocess — the base's own bin/ first, then the declared bin: entries, then the inherited PATH. It delegates to core so `fkf status` and the runner cannot answer differently.
type Fields ¶
Fields is the open field map a document carries. It travels with the records so a read never depends on the live fkf.yaml, and editing a path in configuration never rewrites history.
type Pacer ¶
type Pacer struct {
// Now and Sleep are injected for the same reason PolicyRunner's Sleep is.
Now func() time.Time
Sleep func(ctx context.Context, d time.Duration) error
// contains filtered or unexported fields
}
Pacer enforces a source's `min_interval:` across the whole sync, which retry cannot: retry spaces the attempts of one failing call, while a provider's rate limit counts every call. A source collecting thirty days at concurrency four makes thirty calls that retry never sees as related.
It is a value on the sync run rather than a package global so two syncs in one process — a test suite — never pace each other.
type PolicyRunner ¶
type PolicyRunner struct {
// Sleep is the wait between attempts. It is a field so a test can assert the schedule
// without spending it, and it takes a context so a cancelled sync stops waiting.
Sleep func(ctx context.Context, d time.Duration) error
// contains filtered or unexported fields
}
PolicyRunner applies one source's declared back-pressure around another Runner. It is created per unit of work, so Attempts reports what that unit actually cost.
func NewPolicyRunner ¶
func NewPolicyRunner(inner Runner, source *core.Source) *PolicyRunner
NewPolicyRunner wraps inner with source's declared retry policy.
func (*PolicyRunner) Attempts ¶
func (p *PolicyRunner) Attempts() int
Attempts is how many times the command was actually run. A retried failure must never be quieter than a first-try one, so the sync unit reports this even when the run succeeded.
type Record ¶
Record is one collected item. Every decoded field and value is retained without redaction, dropping, or inference; storage then uses the document's canonical indented JSON encoding, so provider whitespace, object-key order, and escape spellings are not a byte-level contract. Numbers use json.Number so even their lexical form survives a round trip that float64 would round.
func DecodeRecords ¶
DecodeRecords turns one command's stdout into records under the source's declared format.
The empty-output rule differs by format on purpose. A CLI invoked for one JSON document prints `[]` for an empty result, so silence means the command was cut short and must fail. A paginating CLI streaming NDJSON legitimately prints nothing for a day that held nothing, so silence there is an empty day.
type Runner ¶
Runner executes one command and returns its stdout. It is an interface so every test in this repository can inject a fake: the suite must never be able to reach a provider.
func ExecRunner ¶
func ExecRunner() Runner
ExecRunner is the real runner: bounded output, honoured cancellation, and a per-command timeout.
func NewPacingRunner ¶
NewPacingRunner wraps the actual invocation boundary. It must sit inside PolicyRunner so the first try and every retry reserve separate provider-rate-limit slots shared with all concurrent units for the same source.
type RunnerFunc ¶
RunnerFunc adapts a function to Runner.
type Window ¶
Window is the local day an events source collects, and the values its placeholders expand to. The bounds are half-open in UTC so a source that filters on timestamps and one that filters on dates select the same records.
func DayWindow ¶
DayWindow builds the window for one local day. day is expected to be NOON-anchored — what ParseDay returns — because reconstructing literal midnight directly from a Y/M/D is exactly the operation that fails silently on a day whose midnight does not exist.
A civil day is not always 24 hours: a spring-forward transition at local midnight (several zones move their clocks at 00:00, not 02:00 — Brazil did through 2018) makes that day 23 hours long, and `time.Date` given a wall-clock time inside the gap does not error. It normalizes to a DIFFERENT, earlier, valid instant — one that formats as the PREVIOUS calendar day. Reconstructing "midnight" from day.Year/Month/Day would reproduce the same non-existent request and the same silent misfile, so the boundary is found instead: the earliest local instant whose calendar date actually equals the one asked for.
Source Files