Documentation
¶
Overview ¶
Package secureread is the policy-enforced read executor for the DataTug agent server (`datatug serve`). It runs structured, DTQL and native-SQL queries against any pkg/dbcopy-supported source through one fixed Session's DALgo access policies, so every read the web UI can trigger goes through the same access-control path regardless of the query shape (Feature core-investigation-loop, REQ:server-acl-all-reads).
Design ¶
A Session is built once, for the whole `datatug serve` process lifetime, from the same --as/--role/--group/policy flags `datatug query run` already accepts (REQ:principal-selection): the server never re-binds its principal per request. An Executor is bound to that Session and reused across requests; each call names only the source URL and the query.
- RunStructured executes a dal.Query (typically built with dal.NewQueryBuilder) through pkg/accesspolicies.Run: row conditions are AND-ed into the query, field allow-lists redact each row, and a query that explicitly references a field no policy allows is refused with ErrAccessDenied rather than silently emptied (AC hidden-column-refused).
- RunDTQL deserializes a DTQL-YAML document (dal-go/dalgo/dtql) and runs it exactly like RunStructured (REQ:dtql-query-type).
- RunNativeSQL executes raw SQL text read-only. DALgo cannot rewrite text it does not parse, so per REQ:opaque-sql-limitation (assumption A2 in the hub Feature) only the source-level allow/deny — an access.OpaqueQueryScope rule — is enforced, never row conditions or field allow-lists; the session is pinned read-only at the SQLite engine level (PRAGMA query_only) and the Result is always stamped with a LimitationNativeSQL entry. Only sqlite:// sources support native SQL today — see RunNativeSQL's doc comment.
Every Result carries the Limitations a caller must show, never apply silently (REQ:limitation-visible): which policy narrowed the rows, whether rows were filtered, which columns an implicit select hid, and whether row/ column policy was skipped for opaque SQL.
See README.md for the exact HTTP call sites this package is meant to back.
Index ¶
- Constants
- Variables
- func ToContractLimitations(in []Limitation) []apicontract.Limitation
- func WithFederatedProgress(ctx context.Context, observer func(dal.FederatedProgress)) context.Context
- type ColumnStatistics
- type DateBucket
- type DateNumericBucket
- type DateNumericSum
- type Executor
- func (e *Executor) CanReadWholeCollection(ctx context.Context, collection string) error
- func (e *Executor) RunDTQL(ctx context.Context, sourceURL string, dtqlDoc []byte, ...) (Result, error)
- func (e *Executor) RunFederatedDTQL(ctx context.Context, document []byte, sourceURLs map[string]string, ...) (Result, error)
- func (e *Executor) RunNativeSQL(ctx context.Context, sourceURL, sqlText string, args ...dal.QueryArg) (Result, error)
- func (e *Executor) RunSnapshot(ctx context.Context, collection string, recordset apicontract.Recordset) (Result, error)
- func (e *Executor) RunStructured(ctx context.Context, sourceURL string, query dal.Query, ...) (Result, error)
- func (e *Executor) RunStructuredInsecureForTest(ctx context.Context, sourceURL string, query dal.Query, ...) (Result, error)
- func (e *Executor) StreamFederatedDTQL(ctx context.Context, document []byte, sourceURLs map[string]string, ...) (*FederatedStream, error)
- type FederatedStream
- type Limitation
- type LimitationKind
- type RecordSetStatistics
- type Result
- type Row
- type Session
- type SessionOptions
- type StatisticsCollector
- type TypeObservations
- type ValueFrequency
- type ValueKind
Constants ¶
const ( MaxStatisticDistinctValues = 256 // 2,048 covers ordinary Chat result limits (including a full Chinook // invoice history) while still preventing an unbounded time-series map. MaxStatisticDateBuckets = 2048 MaxStatisticDateNumericPairs = 64 )
The caps keep result analysis bounded. An incomplete flag always accompanies a retained lower bound, so callers never mistake it for an exact result.
Variables ¶
var ErrAccessDenied = access.ErrAccessDenied
ErrAccessDenied is access.ErrAccessDenied, re-exported so callers can check a secureread error with errors.Is(err, secureread.ErrAccessDenied) without importing the access package themselves. It is the exact same sentinel value dal-go/dalgo/access uses, not a wrapped copy.
var ErrNativeSQLUnsupported = errors.New("secureread: native SQL is not supported for this source")
ErrNativeSQLUnsupported is returned by RunNativeSQL for a source scheme with no native SQL-text execution surface in this repository yet. See RunNativeSQL's doc comment for the exact schemes this covers today.
var ErrNoPrincipal = errors.New("secureread: no principal named; pass --as, --role or --group, or run Unrestricted")
ErrNoPrincipal is returned by NewSession when the session is secured (NoPolicies is unset) but no principal was named via --as/--role/--group. REQ:principal-selection requires `datatug serve` to fix a principal for the whole session; constructing a secured Session with nobody named would let every request quietly fall through to each policy's own default (usually deny) with no caller-visible identity to blame or audit against. An Unrestricted session (--no-policies) may omit a principal since nothing is enforced.
var ErrOpaqueSQLNotGranted = errors.New("secureread: native SQL execution requires an explicit opaque-query grant (--allow-opaque-sql) or an unrestricted session")
ErrOpaqueSQLNotGranted is returned by RunNativeSQL before any dispatch when the session is neither Unrestricted nor carries an explicit AllowOpaqueSQL grant (REQ:opaque-sql-limitation: "The protected profile MUST refuse opaque native SQL before execution with UNSUPPORTED_PROTECTED_EXECUTION"). Every caller of RunNativeSQL — exec/run_query and the legacy exec/select / exec/execute_commands routes alike, since they share one Executor/Session — is refused the same way.
var ErrSnapshotPolicyUnexpressible = errors.New("snapshot policy evaluation is not safely expressible")
ErrSnapshotPolicyUnexpressible means SQLite cannot represent a recorded value without changing policy-comparison semantics. Callers must deny the read without disclosing the value.
Functions ¶
func ToContractLimitations ¶ added in v0.20.6
func ToContractLimitations(in []Limitation) []apicontract.Limitation
ToContractLimitations folds this package's own internally-split Limitation entries (a standalone "rowsFiltered" marker, a standalone "hiddenColumns" marker, one "policy"/"nativeSql" entry per applied policy — see result.go) into api-contract.md's single-shape Limitation{policy, rowsFiltered, hiddenColumns} list — the SAME shape every policy-enforced read endpoint reports (S101: exec/select gets this treatment too, not just exec/run_query, which is where this folding logic originally lived before moving here so both callers share one implementation).
Phase 1's demo scenarios apply exactly one named policy per request (customers-support, security-matrix, etc.), so folding every applied policy name into ONE combined entry alongside the overall rowsFiltered/ hiddenColumns flags is accurate for them. A project with more than one DISTINCT policy applying different row/column restrictions to the SAME request would see them combined into that one entry too (its rowsFiltered/hiddenColumns already reflect the union). Attributing rowsFiltered/hiddenColumns to the SPECIFIC policy that caused each would need this package's own internal Limitation shape to carry that association, which it does not yet — flagged as a Task 13 ("converge protected execution") follow-up, not fixed here.
func WithFederatedProgress ¶ added in v0.46.0
func WithFederatedProgress(ctx context.Context, observer func(dal.FederatedProgress)) context.Context
WithFederatedProgress installs a request-scoped observer without changing the saved-query API or routing progress through stdout result data.
Types ¶
type ColumnStatistics ¶ added in v0.41.0
type ColumnStatistics struct {
Name string `json:"name"`
NullCount int `json:"nullCount"`
NonNullCount int `json:"nonNullCount"`
Cardinality int `json:"cardinality"`
CardinalityIncomplete bool `json:"cardinalityIncomplete,omitempty"`
Frequencies []ValueFrequency `json:"frequencies,omitempty"`
FrequenciesIncomplete bool `json:"frequenciesIncomplete,omitempty"`
Types TypeObservations `json:"types"`
DateBuckets []DateBucket `json:"dateBuckets,omitempty"`
DateBucketsIncomplete bool `json:"dateBucketsIncomplete,omitempty"`
}
type DateBucket ¶ added in v0.41.0
type DateNumericBucket ¶ added in v0.41.0
type DateNumericSum ¶ added in v0.41.0
type DateNumericSum struct {
DateColumn string `json:"dateColumn"`
NumericColumn string `json:"numericColumn"`
Buckets []DateNumericBucket `json:"buckets,omitempty"`
Incomplete bool `json:"incomplete,omitempty"`
}
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor runs queries against any pkg/dbcopy-supported source URL through one fixed Session's access policies. `datatug serve` constructs one Executor for its whole life (REQ:principal-selection) and calls it once per request, naming only the source the request targets — see README.md for the three HTTP call sites this backs.
func NewExecutor ¶
NewExecutor returns an Executor bound to session.
func (*Executor) CanReadWholeCollection ¶ added in v0.40.0
CanReadWholeCollection is a no-row-read policy preflight for a prospective JOIN target. A JOIN must not turn a target's row predicate or field mask into an unprotected read through its parent relation. The actual JOIN still goes through RunDTQL and its normal secure-read authorization.
func (*Executor) RunDTQL ¶
func (e *Executor) RunDTQL(ctx context.Context, sourceURL string, dtqlDoc []byte, variables map[string]any) (Result, error)
RunDTQL deserializes a DTQL-YAML document (dal-go/dalgo/dtql.Deserialize) and executes it exactly like RunStructured (REQ:dtql-query-type).
func (*Executor) RunFederatedDTQL ¶ added in v0.46.0
func (e *Executor) RunFederatedDTQL(ctx context.Context, document []byte, sourceURLs map[string]string, variables map[string]any) (Result, error)
RunFederatedDTQL runs a named-database query through secured, independent leaf reads. The caller supplies source URLs from its project environment. No joined query is delegated to a database or to an OVDB server.
func (*Executor) RunNativeSQL ¶
func (e *Executor) RunNativeSQL(ctx context.Context, sourceURL, sqlText string, args ...dal.QueryArg) (Result, error)
RunNativeSQL executes raw SQL text read-only against sourceURL and stamps the Result with LimitationNativeSQL. DALgo cannot rewrite text it does not parse, so per REQ:opaque-sql-limitation (assumption A2) only the source-level allow/deny is enforced: the query becomes a dal.TextQuery, whose access.Resource is access.OpaqueQuery(sqlText) — a policy needs an access.OpaqueQueryScope rule to allow it at all, and no policy's row condition or field allow-list is applied to the returned rows regardless of what accesspolicies.Explain reports for the collection-scoped rules.
Before any of that, this process's own operator-level grant is checked first (Task 12, api-contract.md REQ:opaque-sql-limitation: "the support demo has no such grant"): unless the session is Unrestricted or was explicitly started with AllowOpaqueSQL (`datatug serve --allow-opaque-sql`), RunNativeSQL refuses with ErrOpaqueSQLNotGranted before doing anything else — this is what makes every native-SQL caller (exec/run_query and the legacy exec/select/exec/execute_commands routes, which all share one Executor/Session) obey the same boundary, rather than each endpoint needing its own check.
args are the query's own bind values (dal.QueryArg{Name, Value}) — a named arg (Name != "") binds an "@name"/":name"/"$name" placeholder in sqlText; a positional arg (Name == "") binds an ordinary "?" placeholder, in order. They reach the database/sql driver exactly as given: dal-go/dalgo2sql v0.11.7+ converts a named dal.QueryArg to sql.Named(Name, Value) and a positional one to its bare Value (see dal-go/dalgo2sql#177 — a real bind, not string substitution into sqlText, so a value can never be mistaken for SQL syntax).
The SQLite session is pinned read-only at the engine level with PRAGMA query_only on a dedicated, single-connection database handle, so even a multi-statement injection inside sqlText cannot write — this is stronger than the access-denied-by-default posture the ACL check alone gives a syntactically valid but policy-forbidden write attempt.
Only sqlite:// sources support native SQL today: dalgo2ingitdb executes only dal.StructuredQuery ("only StructuredQuery is supported"), and postgres:// is not wired at all (dbcopy.ErrPostgresNotWired). A source without a SQL-text execution surface fails with ErrNativeSQLUnsupported. A future SQL-capable adapter for another scheme should add a read-only dal.DB.RunReadonlyTransaction branch alongside this PRAGMA one, per the brief's "PRAGMA query_only for SQLite; read-only tx elsewhere" design.
func (*Executor) RunSnapshot ¶ added in v0.27.0
func (e *Executor) RunSnapshot(ctx context.Context, collection string, recordset apicontract.Recordset) (Result, error)
RunSnapshot replays recorded typed rows through the session's current access policies. A temporary private SQLite source lets the same accesspolicies.Run path enforce both column projections and expressible row conditions; an unsupported condition fails closed in RunStructured.
func (*Executor) RunStructured ¶
func (e *Executor) RunStructured(ctx context.Context, sourceURL string, query dal.Query, variables map[string]any) (Result, error)
RunStructured executes a structured query (typically built with dal.NewQueryBuilder) against sourceURL through the session's access policies (accesspolicies.Run): row conditions are AND-ed in, field allow-lists redact each row, and a query that explicitly references a field no policy allows is refused with ErrAccessDenied rather than silently emptied. variables resolves the query's own `param` nodes (including $currentUser, bound automatically from the session's principal); pass nil when the query has none.
func (*Executor) RunStructuredInsecureForTest ¶ added in v0.20.4
func (e *Executor) RunStructuredInsecureForTest(ctx context.Context, sourceURL string, query dal.Query, variables map[string]any) (Result, error)
RunStructuredInsecureForTest is RunStructured, except the underlying HTTP(S) source is opened via dbcopy.BackendRef.OpenForTest instead of Open: every dalgo2http.Collection it builds gets Collection.InsecureAllowLoopback set (dal-go/dalgo2http v0.2.0's TEST-ONLY escape hatch — see httpsource.AllowInsecureLoopback's doc comment). This lets a test point a QueryDef's .query.http file at a loopback httptest.Server, or a deliberately-unreachable loopback address (e.g. 127.0.0.1:1, for a fast deterministic live-failure), while still exercising RunStructured's exact real wiring — accesspolicies.Run, the dalgo2http.Recorder context, Result.Provenance — end to end, instead of weakening the test by bypassing RunStructured altogether.
NEVER call this from production code. It exists for this package's own tests (executor_provenance_test.go) and for other packages' tests that drive the same sourceURL -> pkg/dbcopy -> pkg/httpsource pipeline in-process (e.g. apps/datatugapp/commands). A production descriptor file can never request this itself: dalgo2http excludes InsecureAllowLoopback from its YAML/JSON schema, and this method is only reachable by Go code that calls it explicitly.
type FederatedStream ¶ added in v0.46.0
type FederatedStream struct {
Reader dal.RecordsReader
Query dal.StructuredQuery
// contains filtered or unexported fields
}
FederatedStream owns every opened source until Close. Reader is already policy filtered; callers must close the stream even after a read error.
func (*FederatedStream) Close ¶ added in v0.46.0
func (s *FederatedStream) Close() error
func (*FederatedStream) Limitations ¶ added in v0.46.0
func (s *FederatedStream) Limitations() []Limitation
type Limitation ¶
type Limitation struct {
Kind LimitationKind
// Policy names the policy document responsible (its access.Policy.Name());
// "native-sql" for LimitationNativeSQL.
Policy string
// Note is a human-readable explanation, e.g. accesspolicies.Line.String()
// or the opaque-SQL note text.
Note string
// Count is reserved for a future exact filtered-row count (e.g. "3 of 12
// rows hidden"); nil until a caller computes it — Executor does not run
// the extra unrestricted count query needed to fill it in.
Count *int
// Columns holds the hidden field names for LimitationHiddenColumns.
Columns []string
}
Limitation is one policy effect a Result's caller MUST be told about (REQ:limitation-visible) rather than have applied silently. Not every field is set for every Kind: LimitationRowsFiltered carries no Policy; LimitationHiddenColumns carries Columns; LimitationPolicy and LimitationNativeSQL carry Policy and Note.
type LimitationKind ¶
type LimitationKind string
LimitationKind names the shape of one applied limitation, per REQ:limitation-visible ("policy name, rows filtered yes/no, hidden columns") plus REQ:opaque-sql-limitation's native-SQL note.
const ( // LimitationPolicy attributes a row condition or field allow-list to // the policy that applied it, with a human-readable explanation in Note // (accesspolicies.Line.String()). LimitationPolicy LimitationKind = "policy" // LimitationRowsFiltered marks that at least one policy's row condition // narrowed the query (the result may legitimately be empty). LimitationRowsFiltered LimitationKind = "rowsFiltered" // LimitationHiddenColumns lists the queried collection's fields some // policy's field allow-list hid from an implicit (wildcard) select. // Columns holds the hidden field names. LimitationHiddenColumns LimitationKind = "hiddenColumns" // LimitationNativeSQL marks that row and column policies were not // applied because the query was opaque SQL text // (REQ:opaque-sql-limitation) — only source-level allow/deny ran. LimitationNativeSQL LimitationKind = "nativeSql" )
type RecordSetStatistics ¶ added in v0.41.0
type RecordSetStatistics struct {
RowCount int `json:"rowCount"`
Columns []ColumnStatistics `json:"columns"`
DateNumericSums []DateNumericSum `json:"dateNumericSums,omitempty"`
DateNumericSumsIncomplete bool `json:"dateNumericSumsIncomplete,omitempty"`
}
RecordSetStatistics is the renderer-independent, immutable analysis of a Result. Slices are sorted deterministically before being exposed or stored.
func StatisticsForRows ¶ added in v0.41.0
func StatisticsForRows(columns []string, rows []Row) RecordSetStatistics
StatisticsForRows derives statistics from already materialized rows. It is used only for legacy result payloads; live reads update the same accumulator as they load records.
func StatisticsFromRecordset ¶ added in v0.41.0
func StatisticsFromRecordset(recordset apicontract.Recordset) RecordSetStatistics
StatisticsFromRecordset preserves apicontract's tagged source types. It is deliberately used after RunSnapshot's SQLite policy replay instead of inferring date, decimal, or boolean semantics from SQLite driver values.
type Result ¶
type Result struct {
Columns []string
Rows []Row
Statistics RecordSetStatistics
Limitations []Limitation
// Collection is the base collection a structured query actually read.
// Callers use the executor-derived value for evidence provenance and
// current-policy replay; it is empty only for opaque native queries.
Collection string
// Provenance is set only when the source observed one while producing
// this Result — today, only a pkg/httpsource-backed source (opened via
// pkg/dbcopy's http(s):// scheme) ever calls the dalgo2http.Observer
// RunStructured wires into the query's context; every other backend
// (sqlite, ingitdb) leaves it nil. nil means "not observed", never
// "this was live" — mirrors pkg/httpsource.Result's own doc comment on
// this exact ambiguity, and matches the ad-hoc `datatug query run --db
// http://...` path's own ($provenance / "source: ..." line, PR #204)
// treatment of the same nil-vs-live distinction.
Provenance *dalgo2http.Provenance
// SnapshotRecordset is populated only by RunSnapshot. It restores the
// original contract value types after SQLite has applied current policy;
// callers must not infer evidence types from database driver values.
SnapshotRecordset *apicontract.Recordset
}
Result is what every Executor.Run* method returns: the columns and rows a principal is allowed to see, and the limitations that applied to produce them.
type Row ¶
Row is one returned record: its key (native SQL rows have none, so Key is "") and its JSON-shaped data, already policy-redacted by the time it reaches here.
type Session ¶
type Session struct {
// Principal is the caller every query in this session runs as; nil only
// when Unrestricted is set.
Principal *access.Principal
// Policies are the loaded documents every secured query runs through.
Policies []accesspolicies.Loaded
// Unrestricted must be set explicitly (--no-policies) to run with no
// policy enforcement at all.
Unrestricted bool
// AllowOpaqueSQL is the "separate explicit opaque-query grant"
// REQ:opaque-sql-limitation describes: without it (and without
// Unrestricted), RunNativeSQL refuses before dispatch with
// ErrOpaqueSQLNotGranted, for every caller — the appendix's exec/
// run_query AND every legacy execution route (exec/select,
// exec/execute_commands) share this one Executor/Session, so gating it
// here (rather than per-endpoint) is what makes "all legacy routes obey
// the same boundary" (api-contract.md "Security and errors") actually
// true instead of aspirational. Set from `datatug serve
// --allow-opaque-sql` via pkg/api.Capabilities/ConfigureSecureSession.
AllowOpaqueSQL bool
}
Session is the fixed identity and policy set one `datatug serve` process runs under for its whole life (REQ:principal-selection): the principal named by --as/--role/--group, the policies loaded from --policies-dir/--policy, and whether the session runs Unrestricted (--no-policies, deliberately bypassing every policy). Build one with NewSession and hand it to NewExecutor.
func NewSession ¶
func NewSession(o SessionOptions) (Session, error)
NewSession loads the session's policies (accesspolicies.Load) and resolves its principal. A secured session (NoPolicies unset) MUST name a principal via As, Roles or Groups — REQ:principal-selection fixes who `datatug serve` runs as for the whole session, never anonymously — so NewSession returns ErrNoPrincipal rather than silently starting an unidentified secured session. An Unrestricted session may omit a principal entirely.
type SessionOptions ¶
type SessionOptions struct {
// As is the --as principal ID; empty means no ID (roles/groups only, or
// no principal at all).
As string
// Roles are the --role values (repeatable).
Roles []string
// Groups are the --group values (repeatable).
Groups []string
// PoliciesDir is the --policies-dir value; empty means
// $DATATUG_POLICIES_DIR or ~/.datatug/policies (accesspolicies.ResolveDir).
PoliciesDir string
// PolicyFiles are additional --policy documents, applied after the
// directory.
PolicyFiles []string
// NoPolicies runs the session Unrestricted (--no-policies): no policies
// are loaded and no principal is required.
NoPolicies bool
}
SessionOptions mirrors the serve-style flags `datatug query run` already accepts (see apps/datatugapp/commands/cmd_query.go), so `datatug serve` can build a Session from the same --as/--role/--group/--policies-dir/ --policy/--no-policies flags without re-deriving the plumbing.
type StatisticsCollector ¶ added in v0.41.0
type StatisticsCollector struct {
// contains filtered or unexported fields
}
StatisticsCollector lets a codec or reader update analysis as each row is already materialized, avoiding a separate chart-only row pass.
func NewStatisticsCollector ¶ added in v0.41.0
func NewStatisticsCollector() *StatisticsCollector
func (*StatisticsCollector) AddRow ¶ added in v0.41.0
func (c *StatisticsCollector) AddRow(row Row)
func (*StatisticsCollector) Finalize ¶ added in v0.41.0
func (c *StatisticsCollector) Finalize(columns []string) RecordSetStatistics
type TypeObservations ¶ added in v0.41.0
type ValueFrequency ¶ added in v0.41.0
type ValueKind ¶ added in v0.41.0
type ValueKind string
ValueKind records the source value family observed by the analysis layer. It intentionally does not depend on a renderer's data model.
const ( ValueKindNull ValueKind = "null" ValueKindString ValueKind = "string" ValueKindBoolean ValueKind = "boolean" ValueKindNumber ValueKind = "number" ValueKindDecimal ValueKind = "decimal" ValueKindDate ValueKind = "date" ValueKindDatetime ValueKind = "datetime" ValueKindOther ValueKind = "other" )