api

package
v0.48.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 24, 2026 License: Apache-2.0 Imports: 46 Imported by: 0

Documentation

Index

Constants

View Source 
const LocalStoreID = "local"

LocalStoreID is the store id a `datatug serve` session's filestore-backed project store is addressed by. It is the ONLY store kind a local serve session ever configures (ConfigureSecureSession's pathsByID, wired through newDatatugStoreFactory in pkg/server/http_server.go) — never "firestore", which no `datatug serve` session ever registers a real store for. See ResolveStoreID.

Variables

View Source 
var ErrAmbiguousQueryID = errors.New("query id is ambiguous across folders")

ErrAmbiguousQueryID is returned by ResolveQueryID when a bare id (no "/") matches more than one query across different folders — translated to INVALID_REQUEST (400) naming every candidate, never a 500.

View Source 
var ErrAmbiguousStore = errors.New("project is served by more than one store")

ErrAmbiguousStore is returned by ResolveStoreID when a project is served by more than one configured store and the caller supplied no explicit ?storage= to disambiguate. Unreachable with today's serve architecture (ConfiguredStoreIDs never returns more than one entry — a single `datatug serve` process configures exactly one filestore-backed store for the whole union of served projects), kept as an explicit case per the brief's own design so a future multi-store serve session fails loudly instead of silently picking one.

View Source 
var ErrCatalogNotFound = errors.New("catalog not found")

ErrCatalogNotFound is returned by GetCatalogTables when environmentID names no catalog with a catalogID.db.json file under the project's environments/<environmentID>/catalogs/<catalogID>/ directory — the contract-route translation for this is NOT_FOUND (404), the same treatment ErrQueryNotFound already gets (see pkg/server/endpoints/util_error_handling.go), never a raw filesystem error.

View Source 
var ErrLegacyQueryWriteConflict = errors.New("the query kept changing while it was being saved, so nothing was saved")

ErrLegacyQueryWriteConflict is a legacy query write that lost every attempt to another write committing to the same query between its read of the stored query and its own conditional write, so nothing was written. The endpoints answer it with 409.

View Source 
var ErrQueryNotFound = errors.New("query not found")

ErrQueryNotFound is returned by ResolveQueryID when id (bare or folder-qualified) does not match any query file under a project's queries/ tree — the contract-route/legacy-envelope translation for this is NOT_FOUND (404), never a raw filesystem error (S97).

View Source 
var ErrSourceUnavailable = errors.New("resolver: source unavailable")

ErrSourceUnavailable marks a source resolution failure the contract maps to SOURCE_UNAVAILABLE (503): zero eligible/registered sources for the requested (environment, source) pair — see pkg/api/resolver.go's ResolveSource/EligibleTargets and pkg/server/endpoints's newSourceUnavailable.

View Source 
var ErrUnknownStoreID = errors.New("unknown storage id")

ErrUnknownStoreID is returned by ResolveStoreID when an explicit ?storage= (or storeID DTO field) names a store this process has not configured for the given project — including the "firestore" literal pkg/server/endpoints/constants.go used to hardcode as a default, which was never actually configured in a `datatug serve --project` session (S87).

Functions

func AddDbServer 

func AddDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

AddDbServer adds db server to project

func AddRowsToRecordset 

func AddRowsToRecordset(params RecordsetDataRequestParams, _ []RowValues) (numberOfRecords int, err error)

AddRowsToRecordset adds rows to a recordset

func AuthTokenFromHTTPRequest added in v0.19.0 

func AuthTokenFromHTTPRequest(_ *http.Request, _ bool) (*sneatauth.Token, error)

AuthTokenFromHTTPRequest implements sneat-go-core/apicore's GetAuthTokenFromHttpRequest hook for `datatug serve` (REQ:principal-selection). Chosen over routing local-agent endpoints around apicore.Execute (option (b) in the brief) because the fix is a two-line wire-up at serve startup and every endpoint keeps using the same apicore.Execute/VerifyRequest path the rest of the server already relies on for its 400/403/500 conventions (util_error_handling.go).

`datatug serve` has no per-request bearer token: the whole process authenticates once, at startup, into a fixed secureread.Session via --as/--role/--group (see resolveServeSession in apps/datatugapp/commands/cmd_serve.go and api.ConfigureSecureSession). So this hook does not look at r at all — it reports the session's already-resolved principal for every request alike, exactly as if that principal had presented a bearer token on each one.

A nil token — which apicore.VerifyRequest already turns into facade.ErrUnauthorized (401) whenever the endpoint is AuthRequired, without this hook needing to duplicate that check — is returned only when the session both names no principal AND is not Unrestricted: a policy set was loaded (Session.Policies is non-empty, or would be were policies findable) but nobody was identified for it to run as. That is exactly the "anonymous request when a policy set exists" case REQ:server-acl-all-reads requires refused, never treated as an implicit admin.

A Session with Principal == nil AND Unrestricted == true (no policies were found at all, and the caller ran `serve` with none of --as/--role/--group) still yields a token: Unrestricted already means every read this process serves is unenforced regardless of who asks, so there is no principal left to gate on — refusing here would only turn "no policies configured" into a different, spurious 401 with no security benefit. resolveServeSession's own production path never actually produces this combination (it defaults --as to "admin" whenever no principal was named and no policies exist), but secureread.NewSession(SessionOptions{NoPolicies: true}) can build one directly (as test helpers across this package already do), so this hook handles it explicitly rather than assuming that default always ran.

func AuthorizeProjectQueryWrite added in v0.25.0 

func AuthorizeProjectQueryWrite(ctx context.Context, projectID, queryID string, operation access.Operations) error

AuthorizeProjectQueryWrite decides whether this `datatug serve` process may perform operation (Insert, Set, Update or Delete) on the saved query queryID - its canonical, folder-qualified ID - of project projectID. It is the one project-write gate every query write path shares: queries/capture and the legacy queries/create_query, update_query and delete_query routes. It returns nil, or a *accesspolicies.WriteDeniedError that wraps secureread.ErrAccessDenied (access.ErrAccessDenied).

Deny by default, in this order:

  1. An agent with no configured serve session refuses.
  2. Without --allow-writes (Capabilities.AllowWrites) every project write is refused, whoever the principal is - api-contract.md "Security and errors": "Project writes must be refused unless the server has an explicit write capability". The endpoints' requireWriteCapability route gate already refuses; this repeats it so no caller of this package can skip it.
  3. Otherwise accesspolicies.AuthorizeWrite decides for the session's fixed principal under its loaded policies: the --no-policies local owner may write; a secured session needs every loaded policy to grant the write, unconditionally, to its principal - so a read-only principal is refused.

Nothing here writes, and no Git commit or hosted team grant is implied.

func CloseExecutionEvidence added in v0.27.0 

func CloseExecutionEvidence() error

CloseExecutionEvidence releases all routed repository and SQLite handles.

func ConfigureExecutionEvidence added in v0.27.0 

func ConfigureExecutionEvidence(pathsByID map[string]string, configured []incidentstore.ConfiguredStore, options executionstore.Options) error

ConfigureExecutionEvidence wires repository routing for every project the serve process exposes and opens private snapshot sidecars lazily.

func ConfigureSecureSession added in v0.16.0 

func ConfigureSecureSession(session secureread.Session, pathsByID map[string]string, caps Capabilities)

ConfigureSecureSession wires the fixed secureread.Session `datatug serve` builds once for its whole process lifetime (REQ:principal-selection) into every request this process handles, and records the project-id -> filesystem-directory map serve already resolved (pathsByID) so a saved query's SQL/DTQL sidecar file can be located — pkg/datatug-core's LoadQuery does not hydrate QueryDef.Text back from that file yet (that lands with the datatug-core sidecar-read story; see loadQueryDocument).

It also mints this process's securityContextId (api-contract.md "Scope and identity": "Browser state additionally keys scope by ... server-issued securityContextId from agent-info. That opaque ID changes on principal or policy-session changes"). Phase 1's serve process runs one fixed principal/policy set for its whole life (REQ:principal-selection), so this ID is minted once here and never rotates within a process — a fresh ConfigureSecureSession call (a new `datatug serve` invocation) is the only thing that changes it, which is exactly the "principal or policy-session changed" case the contract means STALE_CONTEXT to catch, and it is what every scoped handler validates a request's securityContextId against (see ValidateSecurityContext).

Every request-handling call site MUST go through SecureExecutor rather than open a source or run a query directly, so every read really does pass through the one policy-enforced path (REQ:server-acl-all-reads).

func ConfiguredStoreIDs added in v0.20.2 

func ConfiguredStoreIDs(projectID string) []string

ConfiguredStoreIDs returns the store id(s) this serve session has actually configured for projectID: LocalStoreID when projectID is one of ConfigureSecureSession's served projects, nil otherwise.

func CreateBoard 

func CreateBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

CreateBoard creates board

func CreateFolder 

func CreateFolder(ctx context.Context, request dto.CreateFolder) (folder *datatug.Folder, err error)

CreateFolder creates a new folder for queries

func CreateProject 

func CreateProject(ctx context.Context, request dto.CreateProjectRequest) (*datatug.ProjectSummary, error)

CreateProject create a new DataTug project using requested store.

The request is forwarded to the store unchanged, id included: since datatug-core v0.39.0 dto.CreateProjectRequest carries a caller-supplied ID that addresses the project for the rest of its life, and no layer between the caller and the store may derive, fold or replace it. The id's rules live in dto.CreateProjectRequest.Validate (1-64 characters, lower-case ASCII letters, digits, "-" and "_", starting and ending with a letter or a digit); this function only surfaces that error, and never re-implements it.

func CreateQuery 

func CreateQuery(ctx context.Context, request dto.CreateQuery) (*datatug.QueryDefWithFolderPath, error)

CreateQuery is the legacy queries/create_query write. See saveLegacyQuery.

func DeleteBoard 

func DeleteBoard(ctx context.Context, ref dto.ProjectItemRef) error

DeleteBoard deletes board

func DeleteDbServer 

func DeleteDbServer(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (err error)

DeleteDbServer adds db server to project

func DeleteEntity 

func DeleteEntity(ctx context.Context, ref dto.ProjectItemRef) error

DeleteEntity deletes board

func DeleteFolder 

func DeleteFolder(ctx context.Context, ref dto.ProjectItemRef) error

DeleteFolder deletes queries folder

func DeleteQuery 

func DeleteQuery(ctx context.Context, ref dto.ProjectItemRef) error

DeleteQuery is the legacy queries/delete_query write. ref.ID is the query's folder-qualified id; a leading "~/" names the queries root, as the folderPath "~" does for create_query and update_query, so "~/x" and "x" delete the same root query. Every segment must be safe and a folder must be one this build's store resolves (requireFolderSupport; 400 otherwise), and the serving principal must be authorized to delete it (403 otherwise) before the store is touched.

func ExecutionEvidenceStore added in v0.27.0 

func ExecutionEvidenceStore(projectID string, incident *incidents.IncidentRef) (*executionstore.Store, error)

ExecutionEvidenceStore resolves the configured primary project evidence store. Dedicated and application routes are opened explicitly by services that have a qualified IncidentRef.

func ExecutionEvidenceStoreByID added in v0.27.0 

func ExecutionEvidenceStoreByID(projectID, storeID string) (*executionstore.Store, error)

func GetAllEntities 

func GetAllEntities(ctx context.Context, ref dto.ProjectRef) (entity datatug.Entities, err error)

GetAllEntities returns all entities

func GetBoard 

func GetBoard(ctx context.Context, ref dto.ProjectItemRef) (*datatug.Board, error)

GetBoard returns board by ID

func GetDatasetDefinition 

func GetDatasetDefinition(ctx context.Context, ref dto.ProjectItemRef) (dataset *datatug.RecordsetDefinition, err error)

GetDatasetDefinition returns definition of a dataset by ID

func GetDbServerSummary 

func GetDbServerSummary(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (*datatug.ProjDbServer, error)

GetDbServerSummary returns summary on DB server

func GetEntity 

func GetEntity(ctx context.Context, ref dto.ProjectItemRef) (entity *datatug.Entity, err error)

GetEntity returns board by ID

func GetEnvironmentSummary 

func GetEnvironmentSummary(ctx context.Context, ref dto.ProjectItemRef) (*datatug.EnvironmentSummary, error)

GetEnvironmentSummary returns environment summary

func GetProjectFull 

func GetProjectFull(ctx context.Context, ref dto.ProjectRef) (*datatug.Project, error)

GetProjectFull returns full project metadata

func GetProjectSummary 

func GetProjectSummary(ctx context.Context, ref dto.ProjectRef) (projSummary *datatug.ProjectSummary, err error)

GetProjectSummary returns project summary

func GetProjects 

func GetProjects(ctx context.Context, storeID string) ([]datatug.ProjectBrief, error)

GetProjects return all projects

func GetQuery 

func GetQuery(ctx context.Context, ref dto.ProjectItemRef) (query *datatug.QueryDefWithFolderPath, err error)

GetQuery returns query definition. ref.ID may be bare or folder-qualified (S97's one saved-query id convention): resolved via ResolveQueryID before the store ever sees it, so an unknown id is ErrQueryNotFound and an ambiguous bare id is ErrAmbiguousQueryID — never the store's own raw filesystem error text.

func GetRecordset 

func GetRecordset(_ context.Context, _ dto.ProjectItemRef) (recordset *datatug.Recordset, err error)

GetRecordset saves board

func GetRecordsetsSummary 

func GetRecordsetsSummary(ctx context.Context, ref dto.ProjectRef) (*dto.ProjRecordsetSummary, error)

GetRecordsetsSummary returns board by ID

func GetServerDatabases 

func GetServerDatabases(request dto.GetServerDatabasesRequest) (databases []*datatug.DbCatalog, err error)

GetServerDatabases returns list of databases hosted at a server

func IncidentStoreByID added in v0.28.0 

func IncidentStoreByID(projectID, storeID string) (incidents.APIStore, error)

IncidentStoreByID resolves the incident API provider through the same manager and RepositoryStore already used by immutable execution evidence.

func IncidentView added in v0.28.0 

func IncidentView(ctx context.Context, stored incidents.Incident, events []incidents.Event) (incidents.IncidentView, incidents.ViewPolicy, error)

IncidentView applies the current fixed serve-session policy to a canonical projection. Fact decisions are keyed by full ProjectScope+fact ID; absent or unserved scopes remain absent and therefore fail closed in Core.

func LoadQueryDocument added in v0.20.0 

func LoadQueryDocument(projectID, queryID string, queryType datatug.QueryType) (string, error)

loadQueryDocument reads a saved query's text/document sidecar file directly off disk: "<id>.<QueryFileSuffix>.<lowercase(queryType)>" beside "<id>.query.json", the same convention filestore's saveQuery already writes with (store_queries_saver.go). It exists because fsQueriesStore.LoadQuery does not hydrate QueryDef.Text back from that file (the read-back half of REQ:dtql-query-type's sidecar rule lands with datatug-core PR #302 / the S1b module swap — see this stream's PR body). LoadQueryDocument exports loadQueryDocument for pkg/server/endpoints' exec/run_query rewrite (Task 12), which needs the same sidecar-text read this package's own RunQuery/ExecuteSelect already used.

func PolicyCollectionName added in v0.20.6 

func PolicyCollectionName(name, driver string) string

PolicyCollectionName derives the collection name a structured query's FROM clause should carry for BOTH execution and access-policy matching (dal-go/dalgo's access.SecureReadSession derives its Resource path directly from this same name — there is no separate "policy-only" name in DALgo's own model), given the physical, possibly schema-qualified name a caller supplied (e.g. exec/select's ?from=) and the resolved source's driver.

S101's root cause: a policy `path` names the collection the way the semantic layer does (bare, e.g. "/Customer" — semantic/columns uses collection=Customer with no schema), but a real browser's table-browse request names the schema-qualified physical form the SQL driver itself reports (e.g. "main.Customer" for SQLite) — an EXACT policy path match against "main.Customer" then never matches a "/Customer" rule, denying every request for that table regardless of row content.

Rule (this stream's decision — assumption by the lead session, not a founder ruling; api-contract.md is silent on this): when name is qualified by the driver's own DEFAULT schema (case-insensitively), that qualifier is dropped — safe for execution too, since the default schema is exactly what an unqualified name already resolves to for that driver, so "main.Customer" and "Customer" name the identical SQLite table. A NON-default schema (or a driver with no default-schema convention at all, or an already-bare name) is returned unchanged, so "archive.Customer" keeps its own distinct policy identity from "/Customer" or "/main.Customer" — two same-named tables in different schemas must never share a policy by accident. Never widens access: this only ever narrows a qualified name to its bare form when they are provably the same table; an unmatched path still denies exactly as before.

func ProjectDir added in v0.20.0 

func ProjectDir(projectID string) (string, bool)

ProjectDir exports projectDir for other packages (pkg/server/endpoints) that need a project's on-disk directory to build a ResolvedSource/resolver call without duplicating serve's pathsByID map.

func ProjectStoreFor added in v0.20.0 

func ProjectStoreFor(projectID string) (datatug.ProjectStore, error)

ProjectStoreFor returns the datatug.ProjectStore for projectID, over the same storeID convention RunQuery/ExecuteSelect already use (storage.NewDatatugStore("")), for resolver calls that need one.

func QueryIDIndex added in v0.20.5 

func QueryIDIndex(projectDir string) (map[string]string, error)

QueryIDIndex walks every queries/**/*.query.json file under projectDir, mapping each one's canonical (folder-qualified, "/"-joined) id to its bare (folder-relative) id — a query directly under queries/ (no subfolder) has canonical == bare. A missing queries/ directory is not an error (an empty project tree is valid); it simply produces an empty index.

func RemoveRowsFromRecordset 

func RemoveRowsFromRecordset(params RecordsetDataRequestParams, rows []RowWithIndex) (numberOfRecordsAffected int, err error)

RemoveRowsFromRecordset removes rows from a recordset

func ResolveCatalogPath added in v0.19.0 

func ResolveCatalogPath(projectDir, catalogPath string) (string, error)

ResolveCatalogPath expands a datatug.DbCatalog.Path field into an absolute filesystem path the way this repo's real catalog data actually uses it (see datatug-demo-projects/demo-project-1's environments/*/catalogs/*/*.db.json, S45's dead-layout cleanup, e.g. "~/datatug/dbs/chinook-local.sqlite"):

  • a leading "~" or "~/..." expands to the resolved home directory (github.com/mitchellh/go-homedir)
  • a leading "$HOME" or "${HOME}" expands the same way
  • any other relative path resolves against projectDir (the project's own on-disk location), so it means the same thing regardless of the caller's working directory
  • an already-absolute path (after the above) is returned unchanged

datatug-core's DbCatalogBase.Path has no documented convention for any of these forms; this is this repo's one shared answer, used by this package's own sourceURLFromCatalog and by apps/datatugapp/commands/cmd_query_run_saved.go's equivalent for `datatug query run --project/--query` (S52 found sourceURLFromCatalog building "sqlite://" + catalog.Path with no expansion at all, unopenable against demo-project-1's real catalog data — this closes that gap at its root instead of working around it at each call site).

func ResolveIncidentProject added in v0.28.0 

func ResolveIncidentProject(projectID, environment string) (incidents.ProjectRef, error)

ResolveIncidentProject separates source-project provenance from the incident/evidence store route. A local serve currently exposes one source store per configured project; future adapters can resolve other store kinds without changing IncidentScope.

func ResolveQueryID added in v0.20.5 

func ResolveQueryID(projectDir, id string) (string, error)

ResolveQueryID resolves id to the canonical, folder-qualified query id datatug-core's fsQueriesStore.LoadQuery actually needs to find a query that lives in a subfolder (every query in the demo project does — GET /datatug/queries/get_query?project=...&query=customer-invoices used to 500 with the store's own raw "open .../customer-invoices.query.json: no such file or directory" for exactly this reason).

Canonical-id convention (this stream's decision, marked as an assumption — not a founder ruling; api-contract.md's own Candidate/ExecutionRequest types declare `queryId: string` with no format guidance either way): the folder-qualified id ("<folder>/<bare-id>", "/"-joined, matching fsQueriesStore.LoadQuery's own id-splitting convention) is canonical because it is exactly what the filesystem layout already encodes and is guaranteed unique. A bare id is still accepted on input, but only when it names exactly one query across every folder in the project.

  • id containing "/": trusted as already folder-qualified; resolved (returned as-is) only when a query file exists at that exact path — ErrQueryNotFound otherwise.
  • id with no "/": matched against every query file's bare (folder-relative) id. Exactly one match resolves to its canonical form; zero is ErrQueryNotFound; more than one is ErrAmbiguousQueryID, naming every canonical candidate so the caller can pick one.

func ResolveStoreID added in v0.20.2 

func ResolveStoreID(explicit, projectID string) (string, error)

ResolveStoreID picks the store id a "keep-as-is" GET/mutation route should use for projectID, replacing the previous hardcoded "firestore" default (pkg/server/endpoints/constants.go) that was simply wrong for a `datatug serve --project` session — no Firestore store is ever configured there, so every request silently defaulting to it failed downstream with "no store configured for id=firestore" (S85's finding).

  • explicit != "": honored only when it names a store actually configured for projectID (ErrUnknownStoreID otherwise) — an explicit ?storage= is never silently overridden or ignored.
  • explicit == "": defaults to the one configured store; zero configured stores is ErrUnknownStoreID (nothing serves this project — a distinct concern from "project not found", which callers already validate separately); several is ErrAmbiguousStore (see its own doc comment).

func SaveBoard 

func SaveBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

SaveBoard saves board

func SaveEntity 

func SaveEntity(ctx context.Context, ref dto.ProjectRef, entity *datatug.Entity) error

SaveEntity saves board

func SecureConfiguredProjectIDs added in v0.20.0 

func SecureConfiguredProjectIDs() []string

SecureConfiguredProjectIDs returns every project ID this process is serving (ConfigureSecureSession's pathsByID keys), for agent-info's "projects" array.

func SecureExecutor added in v0.16.0 

func SecureExecutor() (*secureread.Executor, bool)

SecureExecutor returns the Executor ConfigureSecureSession built, and false when serve has not configured one yet (e.g. a handler under test with no ConfigureSecureSession call).

func SecureIncidentActor added in v0.28.0 

func SecureIncidentActor(via string) (incidents.Actor, error)

SecureIncidentActor returns the server-attested event actor. Mutation endpoints never accept actor or reporter identity from callers.

func SecurePolicyFingerprint added in v0.27.0 

func SecurePolicyFingerprint() string

SecurePolicyFingerprint identifies the exact policy session that decided the currently served reads without exposing policy text or filesystem paths.

func SecurePrincipalID added in v0.16.0 

func SecurePrincipalID() string

SecurePrincipalID returns the serving principal's ID for `agent-info` to report (REQ:principal-selection), or "" when the session carries no identified principal (Unrestricted with no --as, or a role/group-only principal).

func SecurePrincipalRolesGroups added in v0.20.0 

func SecurePrincipalRolesGroups() (roles, groups []string)

SecurePrincipalRolesGroups returns the serving principal's Roles/Groups for agent-info's exact envelope (AgentInfoPrincipal.Roles/Groups) — never nil, so JSON encoding always emits "[]" rather than "null" for an unset slice, matching the appendix's "roles:string[]" / "groups:string[]".

func SecureSessionUnrestricted added in v0.20.0 

func SecureSessionUnrestricted() bool

SecureSessionUnrestricted reports whether this process's session runs with no access-policy enforcement at all (--no-policies).

func SecurityContextID added in v0.20.0 

func SecurityContextID() string

SecurityContextID returns the securityContextId agent-info reports and every scoped request must echo back (ValidateSecurityContext).

func UpdateDbSchema 

func UpdateDbSchema(ctx context.Context, projectLoader ProjectLoader, projectID, environment, driver, dbModelID string, dbConnParams dbconnection.Params) (project *datatug.Project, err error)

UpdateDbSchema updates DB schema

func UpdateDbServer 

func UpdateDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

UpdateDbServer adds db server to project

func UpdateQuery 

func UpdateQuery(ctx context.Context, request dto.UpdateQuery) (*datatug.QueryDefWithFolderPath, error)

UpdateQuery is the legacy queries/update_query write. See saveLegacyQuery.

func UpdateRowsInRecordset 

func UpdateRowsInRecordset(params RecordsetDataRequestParams, rows []RowWithIndexAndNewValues) (numberOfRecordsAffected int, err error)

UpdateRowsInRecordset updates rows in a recordset

func ValidateSecurityContext added in v0.20.0 

func ValidateSecurityContext(id string) bool

ValidateSecurityContext reports whether id matches this process's current securityContextId. A caller with an empty configured ID (no ConfigureSecureSession call yet — a handler under test) always fails closed rather than accepting any value.

func WarnMissingSourceFiles added in v0.20.1 

func WarnMissingSourceFiles(ctx context.Context, pathsByID map[string]string)

WarnMissingSourceFiles logs one warning line per environment source (SQL/inGitDB catalog) whose file-backed path does not exist, for every project in pathsByID. `datatug serve --project` used to surface this only as a raw 500 on the first request that happened to touch the missing source (S77's finding: a fresh checkout with no prior `datatug demo` run has no ~/datatug/dbs/chinook-local.sqlite) — this gives the operator the same signal at startup, before the browser does. Best-effort: a project or environment this cannot enumerate is skipped silently rather than failing serve's startup over a diagnostic.

Types

type Capabilities added in v0.20.0 

type Capabilities struct {
	AllowWrites    bool
	AllowOpaqueSQL bool
	// HTTPOffline is `datatug serve --http-offline` (Phase 1 Task 14, item
	// 7): when true, every HTTP-typed saved query's LIVE fetch fails as
	// SOURCE_UNAVAILABLE without ever touching the network (see
	// pkg/httpsource.ContextWithDispatch's offline argument), so a demo can
	// prove "network disabled -> honest failure -> explicit snapshot"
	// deterministically. It is a real operator-facing switch (offline demos),
	// not a test hook. False (the default) leaves live dispatch unaffected.
	HTTPOffline bool
	// ExecTimeout overrides pkg/server/endpoints' default 10-second
	// exec/run_query budget (api-contract.md "Bounded lookups and HTTP":
	// "Server execution has a 10-second default timeout, with a configured
	// upper bound of 30 seconds"). Zero means "use the default"; any value
	// above the 30-second ceiling is clamped down to it — see
	// pkg/server/endpoints/contract_scope.go's execTimeoutFor.
	ExecTimeout time.Duration
	// IncidentStores are trusted shared incident/evidence repository routes.
	// EvidencePrivateDir holds server-private snapshot sidecars.
	IncidentStores     []incidentstore.ConfiguredStore
	EvidencePrivateDir string
	EvidenceByteCap    int
	EvidenceRetention  time.Duration
	SnapshotPolicies   map[string]SnapshotProjectPolicy
}

Capabilities are the operator-controlled flags `datatug serve` fixes for its whole process life, beyond the principal/policy set itself: AllowWrites gates every project-mutation route (create/save/delete project/query/board/entity/recordset-rows) closed by default (api-contract.md "Security and errors": "this read journey must not expose an unauthenticated mutation endpoint as a side effect" — REQ:principal-selection: "Unneeded write endpoints MUST fail closed"). AllowOpaqueSQL gates the "separate explicit opaque-query grant" the appendix's REQ:opaque-sql-limitation describes: without it, a SQL-typed saved query (or any native-SQL execution path) is refused before dispatch with UNSUPPORTED_PROTECTED_EXECUTION, matching "the support demo has no such grant". Both default false: Phase 1's read journey is safe-by-default unless an operator explicitly opts a serve process into either capability.

func GetCapabilities added in v0.20.0 

func GetCapabilities() Capabilities

GetCapabilities returns this process's configured Capabilities.

type CatalogColumn added in v0.35.0 

type CatalogColumn struct {
	Name   string `json:"name"`
	DbType string `json:"dbType,omitempty"`
}

CatalogColumn is the compact, model-facing description of one stored database column. It intentionally excludes environment scan bookkeeping: chat needs enough context to construct a query, not the complete dbmodel.

type CatalogRelation added in v0.35.0 

type CatalogRelation struct {
	Schema  string          `json:"schema"`
	Name    string          `json:"name"`
	DbType  string          `json:"dbType"`
	Columns []CatalogColumn `json:"columns"`
	Issue   string          `json:"-"` // partial explorer load only; never publish raw file errors
}

CatalogRelation describes one stored table or view and its columns.

type CatalogSchema added in v0.35.0 

type CatalogSchema struct {
	Relations []CatalogRelation `json:"relations"`
}

CatalogSchema is the compact stored schema supplied to consumers such as DataTug Chat. DataTug's scanned dbmodel remains the source of truth; this function never introspects the live database independently.

func GetCatalogSchema added in v0.35.0 

func GetCatalogSchema(projectDir, environmentID, catalogID string) (*CatalogSchema, error)

GetCatalogSchema resolves a configured catalog to its scanned dbmodel and loads table/view columns in deterministic order.

func GetCatalogSchemaPartial added in v0.46.3 

func GetCatalogSchemaPartial(projectDir, environmentID, catalogID string) (*CatalogSchema, error)

GetCatalogSchemaPartial preserves healthy relations when one relation's stored column file is missing or malformed. Source-level failures still return an error; affected relations carry a local-only Issue instead.

type CatalogTable added in v0.23.0 

type CatalogTable struct {
	Schema string `json:"schema"`
	Name   string `json:"name"`
	DbType string `json:"dbType,omitempty"`
}

CatalogTable is the minimal {schema, name, dbType} identity datatug-apps' ITableFull (libs/datatug/main/src/lib/models/definition/apis/database.ts) already declares — no column/key detail, which env-db-table.page.ts's own /exec/select-backed row fetch remains the source of. Defined here rather than reusing datatug-core's own datatug.TableModel: that type embeds DBCollectionKey, whose schema/catalog/name fields are unexported with no custom MarshalJSON, so it serializes to "{}" over JSON — a file-storage/lookup-key type, not a wire DTO.

type CatalogTables added in v0.23.0 

type CatalogTables struct {
	Tables []CatalogTable `json:"tables"`
	Views  []CatalogTable `json:"views"`
}

CatalogTables is GET /datatug/catalog-tables's response (Task 17 item A.2, S121): the catalog table/view list datatug-apps' EnvDbPageComponent (the catalog overview page one level above env-db-table.page.ts's own /table/<type> route) never had any way to populate before this — S120's report found nothing in this app fed it (no in-app link even targeted the route, and no endpoint returned this shape). Verified live, against a real `datatug serve` agent, that none of the three existing candidates api-contract.md-adjacent code already calls carries it:

  • dbserver-databases needs a live driver/host/port to introspect an actual DB *connection* (the "add a new server" flow) — unrelated to an already-registered project catalog.
  • projects/project_full's dbModels only ever carry {id, environments[].DbCatalogs[].id}: Project.LoadProject never populates DbModel.Schemas (confirmed against a live response).
  • datatug-core's own DbModelsStore/fsDbModelsStore.LoadDbModel is entirely commented out (storage/filestore/store_dbmodels.go) — the same class of "schemer providers moved out" gap the sibling queries/all_queries route (this same task) restores.

So this reads the project's already-scanned dbmodel files directly, the same direct-filesystem-walk pattern loadModuleQueries/loadModuleEntities (pkg/server/endpoints/semantic_project.go) already use for an identical ProjectStore-incompleteness gap.

func GetCatalogTables added in v0.23.0 

func GetCatalogTables(projectDir, environmentID, catalogID string) (*CatalogTables, error)

GetCatalogTables resolves environmentID+catalogID to their dbModel (via the catalog's own <id>.db.json file) and lists every table/view file under that dbModel's dbmodels/<dbModel>/<schema>/{tables,views}/ tree. Read-only; unlike exec/select or dbserver-databases this touches no live database connection.

type CommandExecutionResult added in v0.19.0 

type CommandExecutionResult struct {
	CommandID           string                `json:"commandId"`
	ElapsedMilliseconds int64                 `json:"elapsed,omitempty"`
	Items               []CommandResponseItem `json:"items"`
}

CommandExecutionResult is one ExecuteCommandsResponse.Commands entry, matching datatug-apps' ICommandResponse (commandId, elapsed?, items).

type CommandResponseItem added in v0.19.0 

type CommandResponseItem struct {
	Type  string      `json:"type"`
	Value interface{} `json:"value,omitempty"`
}

CommandResponseItem is one CommandExecutionResult.Items entry, matching datatug-apps' ICommandResponseItem (type, elapsed?, value?). Value is always a QueryResultResponse for the one command type this endpoint implements (SQL) — the same columns/rows/limitations[] shape exec/select and run_query already return (REQ:limitation-visible), so the web UI's recordset rendering has one wire shape to handle regardless of which endpoint produced it.

type ExecuteCommandRequest added in v0.19.0 

type ExecuteCommandRequest struct {
	ID   string `json:"id,omitempty"`
	Type string `json:"type"`
	Text string `json:"text"`
	Env  string `json:"env"`
	DB   string `json:"db"`
	// NamedParams is decoded so a request that sends it gets a clear
	// "not supported" error (see ExecuteCommands) rather than having the
	// parameters silently ignored — RunNativeSQL executes opaque SQL text
	// with no bind-parameter surface (REQ:opaque-sql-limitation). The web
	// client already avoids this combination itself: a single command with
	// namedParams routes through GET /exec/select instead (agent.service.ts,
	// AgentService.execute).
	NamedParams map[string]any `json:"namedParams,omitempty"`
}

ExecuteCommandRequest is one entry of ExecuteCommandsRequest.Commands, matching datatug-apps' ISqlCommandRequest wire shape exactly (id?, type, text, env, db, namedParams?). Only Type "SQL" is implemented: it is the only command type agent.service.ts's execute() ever actually constructs (IHttpCommand exists as a client-side type but nothing builds one to POST here).

func (ExecuteCommandRequest) Validate added in v0.19.0 

func (v ExecuteCommandRequest) Validate() error

Validate returns an error if the command is not well-formed.

type ExecuteCommandsRequest added in v0.19.0 

type ExecuteCommandsRequest struct {
	ID string `json:"id"`
	// Project is "omitempty": the real client never sends it in the body at
	// all (it is filled in from the `?project=` query parameter before the
	// body is decoded — see executeCommandsHandler), and omitempty keeps a
	// body that does not set it from clobbering that query-derived value
	// back to "" the way a bare `json:"project"` zero-value would on
	// decode. A body that DOES explicitly set "project" still overrides it,
	// same as before.
	Project string `json:"project,omitempty"`
	// Commands are decoded from the request body; StoreID is not part of the
	// body at all and is passed to ExecuteCommands separately (see
	// executeCommandsHandler), matching ExecuteSelect/RunQuery.
	Commands []ExecuteCommandRequest `json:"commands"`
}

ExecuteCommandsRequest is the body of POST /datatug/exec/execute_commands (datatug-apps' agent.service.ts, AgentService.execute): unlike pkg/sqlexecute.Request (a direct db-server connection carrying a raw driver/host ServerRef, still used by GetServerDatabases), every command's Env/DB here are project-relative identifiers resolved exactly the way ExecuteSelect/RunQuery resolve them (resolveSourceURL) — the web client never sends a driver/host ServerRef, and sqlexecute.RequestCommand's own Validate() (which requires one) would reject every request this endpoint actually receives. Project comes from the `?project=` query parameter, mirroring how execute_endpoints.go already reads it — the JSON body itself carries no project field (agent.service.ts's execute() deletes projectId from the body before POSTing it).

func (ExecuteCommandsRequest) Validate added in v0.19.0 

func (v ExecuteCommandsRequest) Validate() error

Validate returns an error if the request is not well-formed.

type ExecuteCommandsResponse added in v0.19.0 

type ExecuteCommandsResponse struct {
	// DurationMilliseconds is the wall-clock time every command in this
	// request took, combined.
	DurationMilliseconds int64                    `json:"duration"`
	Commands             []CommandExecutionResult `json:"commands"`
}

ExecuteCommandsResponse is POST /datatug/exec/execute_commands's response, matching datatug-apps' IExecuteResponse wire shape (duration, commands: ICommandResponse[]).

func ExecuteCommands 

func ExecuteCommands(ctx context.Context, storeID string, request ExecuteCommandsRequest) (ExecuteCommandsResponse, error)

ExecuteCommands runs every command in request through the policy-enforced secureread.Executor (REQ:server-acl-all-reads), the same path api.ExecuteSelect and api.RunQuery already use — see routes.go's executeRoutes / execute_endpoints.go. It replaces the `panic("not implemented yet")` this function used to be: the previous signature took a pkg/sqlexecute.Request, whose RequestCommand embeds a datatug.ServerRef requiring a concrete driver/host, which is not the shape datatug-apps' web client actually sends (see ExecuteCommandsRequest's doc comment) — a real implementation against that old signature would have rejected every real request anyway.

A command's own error (an unsupported source scheme, a policy refusal, a database that can't be resolved) aborts the whole request with that error — matching sqlexecute's original all-or-nothing Response.Commands contract, and (for a secureread.ErrAccessDenied) letting util_error_handling.go's existing ACCESS_DENIED/403 mapping apply exactly as it does for exec/select and run_query, with no separate handling needed here.

type ProjectLoader 

type ProjectLoader interface {
	LoadProjectFile(ctx context.Context) (projectFile datatug.ProjectFile, err error)
	LoadProject(ctx context.Context, o ...datatug.StoreOption) (project *datatug.Project, err error)
}

ProjectLoader defines an interface to load project info

type QueryResultResponse added in v0.16.0 

type QueryResultResponse struct {
	Columns     []string                 `json:"columns"`
	Rows        []map[string]any         `json:"rows"`
	Limitations []apicontract.Limitation `json:"limitations"`
	Provenance  apicontract.Provenance   `json:"provenance"`
}

QueryResultResponse is the JSON shape every policy-enforced read endpoint returns: the columns and rows the principal is allowed to see, plus the SAME limitations/provenance shapes exec/run_query's apicontract.Result returns (S101 — datatug-core v0.27.3 pkg/apicontract's own Limitation and Provenance types, reused directly rather than a separately-shaped DTO). Additive: {columns, rows} are unchanged; a caller that reads only those two fields is unaffected. Limitations is empty (never null, never omitted) for an unrestricted/admin read — "nothing applies" is a real, observable state (AC hidden-column-refused's sibling: "an empty limitation list does not authorize opaque execution" applies here in reverse — an empty list here truly means nothing was restricted).

func ExecuteSelect 

func ExecuteSelect(ctx context.Context, storeID string, request SelectRequest) (QueryResultResponse, error)

ExecuteSelect executes a select through the policy-enforced secureread.Executor (REQ:server-acl-all-reads): a "from" selection runs through RunStructured, raw "sql" text through RunNativeSQL. Every read the web UI can trigger through `datatug serve` MUST come through this one path — see routes.go's executeRoutes / execute_endpoints.go.

type RecordsetDataRequestParams 

type RecordsetDataRequestParams struct {
	RecordsetRequestParams
	Data string `json:"data"`
}

RecordsetDataRequestParams is a set of common request parameters

func (RecordsetDataRequestParams) Validate 

func (v RecordsetDataRequestParams) Validate() error

Validate returns error if not valid

type RecordsetRequestParams 

type RecordsetRequestParams struct {
	Project   string `json:"project"`
	Recordset string `json:"recordset"`
}

RecordsetRequestParams is a set of common request parameters

func (RecordsetRequestParams) Validate 

func (v RecordsetRequestParams) Validate() error

Validate returns error if not valid

type ResolvedSource added in v0.20.0 

type ResolvedSource struct {
	ID    string
	Label string
	Kind  SourceKind
	URL   string
	// Collection is the fixed collection name a caller must query this
	// source through: for SQL/inGitDB-via-catalog it is whatever the caller
	// asks for (any table/collection in that database); for an HTTP source
	// it is fixed to ID itself (one QueryDef = one dalgo2http collection —
	// see pkg/httpsource.BuildCollection, "Name: def.ID").
	Collection string
}

ResolvedSource is one entry of the project's unified source registry: the api-contract.md "Scope and identity" resolver plan task 12 requires — "Semantic discovery and execution use the same resolver" — adapting existing environment/catalog records (SQL, inGitDB) and HTTP QueryDefs into one list, with no second persisted store.

ID is the appendix's SourceRef.source: a STABLE project-local identifier. For a SQL/inGitDB catalog this is the catalog's DbCatalogBase.DbModel (e.g. "chinook") — the identifier datatug-demo-projects/demo-project-1's own EntityField.Mappings already key by (verified against the real project: Customer.ID's mapping is {source:"chinook", ...}, NOT {source:"chinook-local"}), and the one the pre-existing semantic endpoints (pkg/server/endpoints/semantic_*.go) already receive as "source" from the web client — so resolving it against the SAME registry execution now shares is the unification this task exists to do, not a new convention. The catalog's own project-item ID (e.g. "chinook-local", distinct per environment) is accepted too, as a fallback alias, so a caller that already has a concrete catalog ID (the CLI's older --db flag shape) keeps working. See ResolveSource.

For a project-level inGitDB recordset (e.g. "support-notes") ID is the recordset definition's own ID, environment-independent: this demo project's data/ingitdb store is one shared directory, not scoped per environment (verified: recordsets/support-notes.recordset.json declares no environment, and semantic_project.go's existing semanticIngitdbPath already treats it as project-wide).

For an HTTP QueryDef (e.g. "country-facts") ID is the QueryDef's own ID, also environment-independent (an HTTP QueryDef's Targets must be empty — datatug-core's QueryDef.Validate enforces this — so it is not tied to any environment/catalog record at all).

func EligibleTargets added in v0.20.0 

func EligibleTargets(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment string, queryDef *datatug.QueryDef) ([]ResolvedSource, error)

EligibleTargets returns the authorized ResolvedSource options a saved query may run against within environment, per api-contract.md "Scope and identity": an HTTP query's only eligible target is itself (its Targets MUST be empty per datatug-core's own QueryDef.Validate); a SQL/DTQL query with declared Targets is filtered to catalogs whose (Driver, Catalog) matches one of them; a SQL/DTQL query with NO declared Targets (every query in datatug-demo-projects/demo-project-1 today) is eligible against every SQL/inGitDB catalog source in environment — a deliberate, documented engineering default for Phase 1's single-catalog-per-environment demo, not a founder ruling: a project that registers more than one same-model catalog per environment without the query declaring explicit Targets will see every one of them offered, which is the conservative (ask rather than guess) behaviour api-contract.md's "with multiple targets the user selects a source explicitly" already calls for.

func ListSources added in v0.20.0 

func ListSources(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment string) ([]ResolvedSource, error)

ListSources enumerates every source this project's registry can resolve for environment: every SQL/inGitDB catalog registered on that environment's DB servers (keyed by BOTH DbModel and catalog ID — see ResolvedSource's doc), every project-level inGitDB recordset definition, and every project-level HTTP QueryDef. Order is deterministic (by Kind then ID) so a Candidate's target list and an "unknown source" error's implied option set are stable across calls.

func ResolveSource added in v0.20.0 

func ResolveSource(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment, source string) (ResolvedSource, error)

ResolveSource resolves one SourceRef.source (see ResolvedSource's doc for the two accepted forms — DbModel or catalog ID) within environment to a pkg/dbcopy-openable ResolvedSource. It is the ONE place semantic discovery (pkg/server/endpoints/semantic_*.go) and execution (RunQuery/ExecuteSelect/exec/run_query) resolve a source, replacing the two previously-diverging conventions (semantic's broken "<projectDir>/dbs/<source>.sqlite" guess and execution's environment/ catalog walk) — see the PR body's inventory for the specifics.

type RowValues 

type RowValues = map[string]interface{}

RowValues set of named values

type RowWithIndex 

type RowWithIndex struct {
	Index  int                    `json:"index"`
	Values map[string]interface{} `json:"values"`
}

RowWithIndex points to specific row with expected values

func (RowWithIndex) Validate 

func (v RowWithIndex) Validate() error

Validate returns error if not valid

type RowWithIndexAndNewValues 

type RowWithIndexAndNewValues struct {
	RowWithIndex
	NewValues map[string]interface{} `json:"newValues"`
}

RowWithIndexAndNewValues points to specific row with expected values and provides new set of named values

type SelectRequest 

type SelectRequest struct {
	Project     string
	Environment string
	Database    string
	From        string
	SQL         string
	Where       string
	Limit       int
	Columns     []string
}

SelectRequest holds request data for GET /datatug/exec/select.

func (SelectRequest) Validate 

func (v SelectRequest) Validate() error

Validate returns error if not valid

type SnapshotProjectPolicy added in v0.27.0 

type SnapshotProjectPolicy struct {
	Sources map[string]SnapshotSourcePolicy `json:"sources" yaml:"sources"`
}

SnapshotProjectPolicy is trusted server configuration for one project. A source absent from Sources is denied snapshot retention by default.

type SnapshotSourcePolicy added in v0.27.0 

type SnapshotSourcePolicy struct {
	Allow         bool     `json:"allow" yaml:"allow"`
	MaskedColumns []string `json:"maskedColumns,omitempty" yaml:"maskedColumns,omitempty"`
}

SnapshotSourcePolicy explicitly allows snapshot retention for one source and names columns that must be removed before bytes reach private storage.

func SnapshotPolicy added in v0.27.0 

func SnapshotPolicy(projectID, sourceID string) (SnapshotSourcePolicy, bool)

SnapshotPolicy returns an isolated copy of the configured source policy. Missing project/source entries and allow:false all fail closed.

type SourceKind added in v0.20.0 

type SourceKind string

SourceKind names which pkg/dbcopy-openable adapter a ResolvedSource opens through. 

const (
	SourceKindSQL     SourceKind = "sql"
	SourceKindInGitDB SourceKind = "ingitdb"
	SourceKindHTTP    SourceKind = "http"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.