Documentation
¶
Overview ¶
Package httpsource translates a datatug project's HTTP-type QueryDefs into a dalgo2http-backed dal.DB: one dalgo2http.Collection per query (see BuildCollection for the translation and its sensible-default policy for RowsPath/KeyField, since the QueryDef JSON schema does not yet carry those explicitly), snapshot fallback wired to the project's fixtures/http/ directory (see fixtureFS for how its flatly-named fixture files are bridged to dalgo2http's per-request snapshot keying), live-then-snapshot mode.
This package is pkg/dbcopy/url.go's http:// / https:// backend (see parseHTTPSource there); it has no dependency on dbcopy and can also be used directly by a caller — a server endpoint handler, a future datatug-cli command — that wants an HTTP-sourced dal.DB for a project.
Index ¶
- func BuildCollection(def *datatug.QueryDef, urlTemplate string, sample map[string]any) (dalgo2http.Collection, error)
- func ContextWithDispatch(ctx context.Context, mode dalgo2http.Mode, offline bool, timeout time.Duration) context.Context
- func LoadURLTemplate(projectDir, folderPath, id string) (string, error)
- func Open(ctx context.Context, projectDir string, opts ...Option) (dal.DB, error)
- func SnapshotIdentity(projectDir, queryID string) (id string, recordedAt time.Time, ok bool)
- type LoadedQuery
- type Option
- type Result
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildCollection ¶
func BuildCollection(def *datatug.QueryDef, urlTemplate string, sample map[string]any) (dalgo2http.Collection, error)
BuildCollection translates one HTTP QueryDef into a dalgo2http.Collection.
urlTemplate is the sibling .query.http file's content (see LoadURLTemplate). sample, when non-nil, is the query's recorded fixture body (see readFixtureSample) decoded as JSON; it is used ONLY to infer the RowsPath and KeyField defaults documented on inferRowsPath/inferKeyField — never to shape any other field of the returned descriptor, and a nil sample degrades to those functions' no-sample answers rather than failing.
func ContextWithDispatch ¶ added in v0.22.0
func ContextWithDispatch(ctx context.Context, mode dalgo2http.Mode, offline bool, timeout time.Duration) context.Context
ContextWithDispatch returns ctx carrying explicit HTTP-source dispatch options for Open to read instead of its own defaults.
- mode == "" keeps Open's own default (dalgo2http.ModeLive — see Open's doc comment for why that default changed from ModeLiveThenSnapshot).
- offline, when true, makes Open wire a Client that fails every live request without ever dialing (see offlineTransport) — `datatug serve --http-offline`'s mechanism.
- timeout, when > 0, overrides BuildCollection's own defaultTimeout for every collection Open builds from projectDir.
func LoadURLTemplate ¶
LoadURLTemplate reads the URL template for the HTTP query id found in folderPath (as returned by LoadHTTPQueries), trimming surrounding whitespace (the sibling .query.http files in the demo project end with a trailing newline).
func Open ¶
Open builds a dal.DB from every HTTP QueryDef declared under <projectDir>/queries/**. It fails with an error naming projectDir when no HTTP QueryDef is found there: an http(s):// db-copy source with nothing to serve is a configuration mistake (the wrong project path, most often), not a validly-empty database.
func SnapshotIdentity ¶ added in v0.22.0
SnapshotIdentity reports the stable identity pkg/server/endpoints' exec/run_query uses for the one recorded snapshot this demo project ships per HTTP query — see fixtureFS's own doc comment for why there is exactly one, independent of the parameter value actually requested. The identity is "<queryID>@<recordedAt RFC3339>" (lead assumption 2026-09-10, pending founder confirmation — see the contract amendment in spec/features/core-investigation-loop/api-contract.md): stable across process restarts (recordedAt never changes without a new fixture file replacing the old one), and the exact string a client must echo back as ExecutionRequest.SnapshotID to select this snapshot explicitly (api-contract.md "mode:snapshot and a configured snapshotId"). ok is false when projectDir has no recorded fixture for queryID (fixtures/http/<queryID>.json does not exist), in which case id and recordedAt are the zero value and must not be used.
Types ¶
type LoadedQuery ¶
LoadedQuery is one HTTP-type QueryDef found by LoadHTTPQueries, together with the folder it was found in (relative to <projectDir>/queries) — needed to locate its sibling URL-template file alongside it.
func LoadHTTPQueries ¶
func LoadHTTPQueries(projectDir string) ([]LoadedQuery, error)
LoadHTTPQueries scans <projectDir>/queries/** for *.query.json files declaring "type": "HTTP", parsing each into a datatug.QueryDef.
This reads project files directly with encoding/json rather than going through pkg/datatug-core/storage/filestore's generic project-item store, for two reasons: that store's loader does not read a query's sibling text-body file back into QueryDef.Text (only the SAVE path writes it — see store_queries_saver.go; LoadURLTemplate below is this package's own replacement for that missing read), and pulling in the full store abstraction (which also knows about SQL/GraphQL queries, folders, and project-wide caching) for a read-only, HTTP-only scan would add a much larger dependency surface than this package needs.
type Option ¶ added in v0.20.4
type Option func(*openOptions)
Option configures Open. The zero value of every Option's underlying config is production-safe; only AllowInsecureLoopback (see its doc comment) changes behavior, and only when a caller passes it explicitly.
func AllowInsecureLoopback ¶ added in v0.20.4
func AllowInsecureLoopback() Option
AllowInsecureLoopback is a TEST-ONLY Open Option: every dalgo2http.Collection Open builds from projectDir gets Collection.InsecureAllowLoopback set (dal-go/dalgo2http v0.2.0's escape hatch — URLTemplate may then use http://, and the guarded dialer may dial a loopback address, but ONLY when the host is literally loopback; every other blocked address class stays blocked). It exists so this package's own tests, and other packages' tests that drive the same sourceURL -> pkg/dbcopy -> httpsource.Open pipeline in-process (see pkg/dbcopy's BackendRef.OpenForTest and pkg/secureread's Executor.RunStructuredInsecureForTest), can exercise a live-then-snapshot HTTP source against a loopback httptest.Server or an intentionally-unreachable loopback address (e.g. 127.0.0.1:1) — WITHOUT any project descriptor file ever being able to request this itself: the field is set here, in Go code, from an explicit caller opt-in, never from anything LoadHTTPQueries/LoadURLTemplate read off disk. NEVER pass this outside test code.
type Result ¶
type Result struct {
Records []record.Record
Provenance dalgo2http.Provenance
}
Result is what ExecuteQuery returns: the rows a query produced, and the Provenance (live vs snapshot) dalgo2http observed while producing them.
pkg/secureread's Result.Limitations is the natural home for this once it lands (per the stream brief); until then this is the small bridge value a caller — a server endpoint handler, a future secureread integration — reads and forwards into whatever its own result/Limitations shape needs.
func ExecuteQuery ¶
ExecuteQuery runs q against db and returns both its rows and the Provenance dalgo2http observed while running it.
db need not be a database Open returned — ExecuteQuery works against any dal.DB — but Provenance is only ever populated when the backend actually reports one (today, only a dalgo2http-backed db does). Result's zero Provenance value is indistinguishable from "not observed" by design: a caller forwarding it into a Limitations-style note should treat an unpopulated Provenance as "nothing to report", not as "this was live".
Source Files