api

package
v0.55.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 5, 2026 License: Apache-2.0 Imports: 53 Imported by: 0

Documentation

Index

Constants

View Source 
const DriverPostgres = "postgres"

DriverPostgres is the driver name a PostgreSQL server and catalog carry in a project.

View Source 
const HostRule = "" /* 137-byte string literal not displayed */

HostRule is the sentence that says what the host of a db server is.

View Source 
const LocalStoreID = "local"

LocalStoreID is the store id a `datatug serve` session's filestore-backed project store is addressed by. It is the ONLY store kind a local serve session ever configures (ConfigureSecureSession's pathsByID, wired through newDatatugStoreFactory in pkg/server/http_server.go) — never "firestore", which no `datatug serve` session ever registers a real store for. See ResolveStoreID.

View Source 
const MaxPathIdentifierLength = 512

MaxPathIdentifierLength is the most bytes an ID made of folders has in all, whatever its parts are: a folder tree of a project is nowhere near it, and a longer ID is a text that no file system opens, which would be quoted whole in an answer and in the log.

View Source 
const PlainNameRule = "must be a plain name: letters, digits, '.', '_' and '-', at most 128 characters, starting with a letter or a digit"

PlainNameRule is the sentence that says what a plain name is. Every refusal made by ValidateIdentifier, and the contract routes' own answer for an environment, carry it, and nothing of the value they refused.

View Source 
const PlainPathRule = "" /* 164-byte string literal not displayed */

PlainPathRule is the sentence that says what an ID made of folders is: the folders and the name of a recordset definition or of a folder, each a plain name, and at most MaxPathIdentifierLength bytes in all.

Variables

View Source 
var DriverRule = "must be one of " + strings.Join(dbServerDrivers, ", ")

DriverRule is the sentence that says which drivers a db server can have.

View Source 
var ErrAmbiguousQueryID = errors.New("query id is ambiguous across folders")

ErrAmbiguousQueryID is returned by ResolveQueryID when a bare id (no "/") matches more than one query across different folders — translated to INVALID_REQUEST (400) naming every candidate, never a 500.

View Source 
var ErrAmbiguousStore = errors.New("project is served by more than one store")

ErrAmbiguousStore is returned by ResolveStoreID when a project is served by more than one configured store and the caller supplied no explicit ?storage= to disambiguate. Unreachable with today's serve architecture (ConfiguredStoreIDs never returns more than one entry — a single `datatug serve` process configures exactly one filestore-backed store for the whole union of served projects), kept as an explicit case per the brief's own design so a future multi-store serve session fails loudly instead of silently picking one.

View Source 
var ErrCatalogNotFound = errors.New("catalog not found")

ErrCatalogNotFound is returned by GetCatalogTables when environmentID names no catalog with a catalogID.db.json file under the project's environments/<environmentID>/catalogs/<catalogID>/ directory — the contract-route translation for this is NOT_FOUND (404), the same treatment ErrQueryNotFound already gets (see pkg/server/endpoints/util_error_handling.go), never a raw filesystem error.

View Source 
var ErrLegacyQueryWriteConflict = errors.New("the query kept changing while it was being saved, so nothing was saved")

ErrLegacyQueryWriteConflict is a legacy query write that lost every attempt to another write committing to the same query between its read of the stored query and its own conditional write, so nothing was written. The endpoints answer it with 409.

View Source 
var ErrQueryNotFound = errors.New("query not found")

ErrQueryNotFound is returned by ResolveQueryID when id (bare or folder-qualified) does not match any query file under a project's queries/ tree — the contract-route/legacy-envelope translation for this is NOT_FOUND (404), never a raw filesystem error (S97).

View Source 
var ErrSourceUnavailable = errors.New("resolver: source unavailable")

ErrSourceUnavailable marks a source resolution failure the contract maps to SOURCE_UNAVAILABLE (503): zero eligible/registered sources for the requested (environment, source) pair — see pkg/api/resolver.go's ResolveSource/EligibleTargets and pkg/server/endpoints's newSourceUnavailable.

View Source 
var ErrUnknownStoreID = errors.New("unknown storage id")

ErrUnknownStoreID is returned by ResolveStoreID when an explicit ?storage= (or storeID DTO field) names a store this process has not configured for the given project — including the "firestore" literal pkg/server/endpoints/constants.go used to hardcode as a default, which was never actually configured in a `datatug serve --project` session (S87).

Functions

func AddDbServer 

func AddDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

AddDbServer adds db server to project

func AddRowsToRecordset 

func AddRowsToRecordset(params RecordsetDataRequestParams, _ []RowValues) (numberOfRecords int, err error)

AddRowsToRecordset adds rows to a recordset

func AuthTokenFromHTTPRequest added in v0.19.0 

func AuthTokenFromHTTPRequest(_ *http.Request, _ bool) (*sneatauth.Token, error)

AuthTokenFromHTTPRequest implements sneat-go-core/apicore's GetAuthTokenFromHttpRequest hook for `datatug serve` (REQ:principal-selection). Chosen over routing local-agent endpoints around apicore.Execute (option (b) in the brief) because the fix is a two-line wire-up at serve startup and every endpoint keeps using the same apicore.Execute/VerifyRequest path the rest of the server already relies on for its 400/403/500 conventions (util_error_handling.go).

`datatug serve` has no per-request bearer token: the whole process authenticates once, at startup, into a fixed secureread.Session via --as/--role/--group (see resolveServeSession in apps/datatugapp/commands/cmd_serve.go and api.ConfigureSecureSession). So this hook does not look at r at all — it reports the session's already-resolved principal for every request alike, exactly as if that principal had presented a bearer token on each one.

A nil token — which apicore.VerifyRequest already turns into facade.ErrUnauthorized (401) whenever the endpoint is AuthRequired, without this hook needing to duplicate that check — is returned only when the session both names no principal AND is not Unrestricted: a policy set was loaded (Session.Policies is non-empty, or would be were policies findable) but nobody was identified for it to run as. That is exactly the "anonymous request when a policy set exists" case REQ:server-acl-all-reads requires refused, never treated as an implicit admin.

A Session with Principal == nil AND Unrestricted == true (no policies were found at all, and the caller ran `serve` with none of --as/--role/--group) still yields a token: Unrestricted already means every read this process serves is unenforced regardless of who asks, so there is no principal left to gate on — refusing here would only turn "no policies configured" into a different, spurious 401 with no security benefit. resolveServeSession's own production path never actually produces this combination (it defaults --as to "admin" whenever no principal was named and no policies exist), but secureread.NewSession(SessionOptions{NoPolicies: true}) can build one directly (as test helpers across this package already do), so this hook handles it explicitly rather than assuming that default always ran.

func AuthorizeProjectQueryWrite added in v0.25.0 

func AuthorizeProjectQueryWrite(ctx context.Context, projectID, queryID string, operation access.Operations) error

AuthorizeProjectQueryWrite decides whether this `datatug serve` process may perform operation (Insert, Set, Update or Delete) on the saved query queryID - its canonical, folder-qualified ID - of project projectID. It is the one project-write gate every query write path shares: queries/capture and the legacy queries/create_query, update_query and delete_query routes. It returns nil, or a *accesspolicies.WriteDeniedError that wraps secureread.ErrAccessDenied (access.ErrAccessDenied).

Deny by default, in this order:

  1. An agent with no configured serve session refuses.
  2. Without --allow-writes (Capabilities.AllowWrites) every project write is refused, whoever the principal is - api-contract.md "Security and errors": "Project writes must be refused unless the server has an explicit write capability". The endpoints' requireWriteCapability route gate already refuses; this repeats it so no caller of this package can skip it.
  3. Otherwise accesspolicies.AuthorizeWrite decides for the session's fixed principal under its loaded policies: the --no-policies local owner may write; a secured session needs every loaded policy to grant the write, unconditionally, to its principal - so a read-only principal is refused.

Nothing here writes, and no Git commit or hosted team grant is implied.

func CheckScanDriverAgainstProject added in v0.55.0 

func CheckScanDriverAgainstProject(projectDir, environment, catalogID, driver string) error

CheckScanDriverAgainstProject is an error when the project already records the catalog catalogID in environment under another driver than driver. A catalog file is kept by environment and id, and not by driver, so a database of one id under two drivers in one environment is one catalog file that two servers list: the scan of the second would put its own catalog file in the place of the first's, leave the id on both servers, and take back the tables of the first as the ones its database no longer has. A SQLite file and a PostgreSQL database each have their own id (--db). The message names both drivers, and says to use another --db. A catalog the project does not record, or records with no driver, or whose catalog file cannot be read, has none to disagree with: the scan writes the file anew.

func CheckScanName added in v0.54.3 

func CheckScanName(flag, value string) error

CheckScanName is an error that names flag (such as "--db") when value cannot be the id of a database, a database model or an environment in a project: each is the name of a folder of the project, so it must be a plain name (letters and digits of any script, ".", "_" and "-", at most 128 characters, starting with a letter or a digit) that is also a folder name on every system a project is opened on (see folderNameProblem). There is one definition of a plain name, the one the routes of serve apply to an environment or a catalog a client sends (ValidateIdentifier, built on dbcopy.IsPlainSourceID), and a scan adds to it only what a folder needs: so a name the scan accepts is one every route accepts, and a project a scan writes can always be browsed. A value that is not a plain name is not in the message: it may be a connection string.

func CheckScanNamesAgainstProject added in v0.54.3 

func CheckScanNamesAgainstProject(projectDir, environment, catalogID, model string) error

CheckScanNamesAgainstProject is an error when the environment, the catalog id or the database model of a scan differs only by case from a name the project already has in the same place: an environment in environments/, a catalog in the folder of the catalogs of the environment (a folder, or a file in the flat place datatug-core also reads), a model in dbmodels/. On a file system that does not tell the case of a name apart (macOS and Windows by default) the two are one folder, and a scan of one database would take the tables of another for its own and remove those it does not find. The names are those the folders list, so the answer does not depend on the file system the project is on; ids that are the same, or that differ by more than case, are not refused. The ids are plain names already (see CheckScanName).

func CloseExecutionEvidence added in v0.27.0 

func CloseExecutionEvidence() error

CloseExecutionEvidence releases all routed repository and SQLite handles.

func ConfigureExecutionEvidence added in v0.27.0 

func ConfigureExecutionEvidence(pathsByID map[string]string, configured []incidentstore.ConfiguredStore, options executionstore.Options) error

ConfigureExecutionEvidence wires repository routing for every project the serve process exposes and opens private snapshot sidecars lazily.

func ConfigureSecureSession added in v0.16.0 

func ConfigureSecureSession(session secureread.Session, pathsByID map[string]string, caps Capabilities)

ConfigureSecureSession wires the fixed secureread.Session `datatug serve` builds once for its whole process lifetime (REQ:principal-selection) into every request this process handles, and records the project-id -> filesystem-directory map serve already resolved (pathsByID) so a saved query's SQL/DTQL sidecar file can be located — pkg/datatug-core's LoadQuery does not hydrate QueryDef.Text back from that file yet (that lands with the datatug-core sidecar-read story; see loadQueryDocument).

It also mints this process's securityContextId (api-contract.md "Scope and identity": "Browser state additionally keys scope by ... server-issued securityContextId from agent-info. That opaque ID changes on principal or policy-session changes"). Phase 1's serve process runs one fixed principal/policy set for its whole life (REQ:principal-selection), so this ID is minted once here and never rotates within a process — a fresh ConfigureSecureSession call (a new `datatug serve` invocation) is the only thing that changes it, which is exactly the "principal or policy-session changed" case the contract means STALE_CONTEXT to catch, and it is what every scoped handler validates a request's securityContextId against (see ValidateSecurityContext).

Every request-handling call site MUST go through SecureExecutor rather than open a source or run a query directly, so every read really does pass through the one policy-enforced path (REQ:server-acl-all-reads).

func ConfiguredStoreIDs added in v0.20.2 

func ConfiguredStoreIDs(projectID string) []string

ConfiguredStoreIDs returns the store id(s) this serve session has actually configured for projectID: LocalStoreID when projectID is one of ConfigureSecureSession's served projects, nil otherwise.

func CreateBoard 

func CreateBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

CreateBoard creates board

func CreateFolder 

func CreateFolder(ctx context.Context, request dto.CreateFolder) (folder *datatug.Folder, err error)

CreateFolder creates a new folder for queries

func CreateProject 

func CreateProject(ctx context.Context, request dto.CreateProjectRequest) (*datatug.ProjectSummary, error)

CreateProject create a new DataTug project using requested store.

The request is forwarded to the store unchanged, id included: since datatug-core v0.39.0 dto.CreateProjectRequest carries a caller-supplied ID that addresses the project for the rest of its life, and no layer between the caller and the store may derive, fold or replace it. The id's rules live in dto.CreateProjectRequest.Validate (1-64 characters, lower-case ASCII letters, digits, "-" and "_", starting and ending with a letter or a digit); this function only surfaces that error, and never re-implements it.

func CreateQuery 

func CreateQuery(ctx context.Context, request dto.CreateQuery) (*datatug.QueryDefWithFolderPath, error)

CreateQuery is the legacy queries/create_query write. See saveLegacyQuery.

func DeleteBoard 

func DeleteBoard(ctx context.Context, ref dto.ProjectItemRef) error

DeleteBoard deletes board

func DeleteDbServer 

func DeleteDbServer(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (err error)

DeleteDbServer adds db server to project

func DeleteEntity 

func DeleteEntity(ctx context.Context, ref dto.ProjectItemRef) error

DeleteEntity deletes board

func DeleteFolder 

func DeleteFolder(ctx context.Context, ref dto.ProjectItemRef) error

DeleteFolder deletes queries folder

func DeleteQuery 

func DeleteQuery(ctx context.Context, ref dto.ProjectItemRef) error

DeleteQuery is the legacy queries/delete_query write. ref.ID is the query's folder-qualified id; a leading "~/" names the queries root, as the folderPath "~" does for create_query and update_query, so "~/x" and "x" delete the same root query. Every segment must be safe and a folder must be one this build's store resolves (requireFolderSupport; 400 otherwise), and the serving principal must be authorized to delete it (403 otherwise) before the store is touched.

func ExecutionEvidenceStore added in v0.27.0 

func ExecutionEvidenceStore(projectID string, incident *incidents.IncidentRef) (*executionstore.Store, error)

ExecutionEvidenceStore resolves the configured primary project evidence store. Dedicated and application routes are opened explicitly by services that have a qualified IncidentRef.

func ExecutionEvidenceStoreByID added in v0.27.0 

func ExecutionEvidenceStoreByID(projectID, storeID string) (*executionstore.Store, error)

func GetAllEntities 

func GetAllEntities(ctx context.Context, ref dto.ProjectRef) (entity datatug.Entities, err error)

GetAllEntities returns all entities

func GetBoard 

func GetBoard(ctx context.Context, ref dto.ProjectItemRef) (*datatug.Board, error)

GetBoard returns board by ID

func GetDatasetDefinition 

func GetDatasetDefinition(ctx context.Context, ref dto.ProjectItemRef) (dataset *datatug.RecordsetDefinition, err error)

GetDatasetDefinition returns definition of a dataset by ID. The ID is the folders and the name of the definition under the project's recordsets, joined into a path by the store, so each must be a plain name before the store is asked.

func GetDbServerSummary 

func GetDbServerSummary(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (*datatug.ProjDbServer, error)

GetDbServerSummary returns summary on DB server

func GetEntity 

func GetEntity(ctx context.Context, ref dto.ProjectItemRef) (entity *datatug.Entity, err error)

GetEntity returns board by ID

func GetEnvironmentSummary 

func GetEnvironmentSummary(ctx context.Context, ref dto.ProjectItemRef) (*datatug.EnvironmentSummary, error)

GetEnvironmentSummary returns environment summary

func GetProjectFull 

func GetProjectFull(ctx context.Context, ref dto.ProjectRef) (*datatug.Project, error)

GetProjectFull returns full project metadata

func GetProjectSummary 

func GetProjectSummary(ctx context.Context, ref dto.ProjectRef) (projSummary *datatug.ProjectSummary, err error)

GetProjectSummary returns project summary

func GetProjects 

func GetProjects(ctx context.Context, storeID string) ([]datatug.ProjectBrief, error)

GetProjects return all projects

func GetQuery 

func GetQuery(ctx context.Context, ref dto.ProjectItemRef) (query *datatug.QueryDefWithFolderPath, err error)

GetQuery returns query definition. ref.ID may be bare or folder-qualified (S97's one saved-query id convention): resolved via ResolveQueryID before the store ever sees it, so an unknown id is ErrQueryNotFound and an ambiguous bare id is ErrAmbiguousQueryID — never the store's own raw filesystem error text.

func GetRecordset 

func GetRecordset(_ context.Context, _ dto.ProjectItemRef) (recordset *datatug.Recordset, err error)

GetRecordset saves board

func GetRecordsetsSummary 

func GetRecordsetsSummary(ctx context.Context, ref dto.ProjectRef) (*dto.ProjRecordsetSummary, error)

GetRecordsetsSummary returns board by ID

func GetServerDatabases 

func GetServerDatabases(request dto.GetServerDatabasesRequest) (databases []*datatug.DbCatalog, err error)

GetServerDatabases returns list of databases hosted at a server

func HostlessRule added in v0.55.1 

func HostlessRule(driver string) string

HostlessRule is the sentence that says a db server of a driver of hostlessDrivers has no host; driver is one of them, a plain name.

func IncidentStoreByID added in v0.28.0 

func IncidentStoreByID(projectID, storeID string) (incidents.APIStore, error)

IncidentStoreByID resolves the incident API provider through the same manager and RepositoryStore already used by immutable execution evidence.

func IncidentView added in v0.28.0 

func IncidentView(ctx context.Context, stored incidents.Incident, events []incidents.Event) (incidents.IncidentView, incidents.ViewPolicy, error)

IncidentView applies the current fixed serve-session policy to a canonical projection. Fact decisions are keyed by full ProjectScope+fact ID; absent or unserved scopes remain absent and therefore fail closed in Core.

func LoadQueryDocument added in v0.20.0 

func LoadQueryDocument(projectID, queryID string, queryType datatug.QueryType) (string, error)

loadQueryDocument reads a saved query's text/document sidecar file directly off disk: "<id>.<QueryFileSuffix>.<lowercase(queryType)>" beside "<id>.query.json", the same convention filestore's saveQuery already writes with (store_queries_saver.go). It exists because fsQueriesStore.LoadQuery does not hydrate QueryDef.Text back from that file (the read-back half of REQ:dtql-query-type's sidecar rule lands with datatug-core PR #302 / the S1b module swap — see this stream's PR body). LoadQueryDocument exports loadQueryDocument for pkg/server/endpoints' exec/run_query rewrite (Task 12), which needs the same sidecar-text read this package's own RunQuery/ExecuteSelect already used.

func LocalSQLiteSourceURL added in v0.54.3 

func LocalSQLiteSourceURL(path string) string

LocalSQLiteSourceURL is the "sqlite://" source URL of the SQLite file at path, the path of a catalog as ResolveCatalogPath returns it: a URL that dbcopy.Parse reads back to that path. A file name may hold "%", "#" and "?", which a URL does not read as part of a path (a file named a#b.db is the file a, and one named a%23b.db is a#b.db), and a control character (a tab, a newline), which dbcopy.Parse refuses in a URL, so each is written as percent-encoded, and then the path starts like one ("./") when it is relative, as the host of a URL cannot be encoded.

func LookupError added in v0.54.1 

func LookupError(format string, cause error, ids ...string) error

LookupError builds the error for a lookup of IDs that a client or a flag supplied (an environment, a database, a catalog). format has one %q for each of ids, in order.

A caller may put anything in an ID field, a whole source string with its password included, and the error of a lookup that failed quotes the path the ID was turned into. So each ID is shown only when it is a plain name (see dbcopy.SourceIDDisplay), and cause, the error of the lookup, is wrapped only when every ID is one: otherwise the message says what was not found with the IDs not shown, and cause is left out (so errors.Is no longer sees it, which is the price of not printing it). A nil cause gives the bare message.

func OpenSchemaScanForTest added in v0.55.0 

func OpenSchemaScanForTest() func(dbcopy.BackendRef, context.Context) (dbcopy.SchemaScanDB, error)

OpenSchemaScanForTest is how a PostgreSQL scan opens its source now: what a test stood in with (SetOpenSchemaScanForTest), or else what the test binary stops the run with, or else the real open. A test of the package that runs the real scan command reads it to prove that its binary has the open that stops the run, not the one that dials.

func PolicyCollectionName added in v0.20.6 

func PolicyCollectionName(name, driver string) string

PolicyCollectionName derives the collection name a structured query's FROM clause should carry for BOTH execution and access-policy matching (dal-go/dalgo's access.SecureReadSession derives its Resource path directly from this same name — there is no separate "policy-only" name in DALgo's own model), given the physical, possibly schema-qualified name a caller supplied (e.g. exec/select's ?from=) and the resolved source's driver.

S101's root cause: a policy `path` names the collection the way the semantic layer does (bare, e.g. "/Customer" — semantic/columns uses collection=Customer with no schema), but a real browser's table-browse request names the schema-qualified physical form the SQL driver itself reports (e.g. "main.Customer" for SQLite) — an EXACT policy path match against "main.Customer" then never matches a "/Customer" rule, denying every request for that table regardless of row content.

Rule (this stream's decision — assumption by the lead session, not a founder ruling; api-contract.md is silent on this): when name is qualified by the driver's own DEFAULT schema (case-insensitively), that qualifier is dropped — safe for execution too, since the default schema is exactly what an unqualified name already resolves to for that driver, so "main.Customer" and "Customer" name the identical SQLite table. A NON-default schema (or a driver with no default-schema convention at all, or an already-bare name) is returned unchanged, so "archive.Customer" keeps its own distinct policy identity from "/Customer" or "/main.Customer" — two same-named tables in different schemas must never share a policy by accident. Never widens access: this only ever narrows a qualified name to its bare form when they are provably the same table; an unmatched path still denies exactly as before.

func ProjectDir added in v0.20.0 

func ProjectDir(projectID string) (string, bool)

ProjectDir exports projectDir for other packages (pkg/server/endpoints) that need a project's on-disk directory to build a ResolvedSource/resolver call without duplicating serve's pathsByID map.

func ProjectStoreFor added in v0.20.0 

func ProjectStoreFor(projectID string) (datatug.ProjectStore, error)

ProjectStoreFor returns the datatug.ProjectStore for projectID, over the same storeID convention RunQuery/ExecuteSelect already use (storage.NewDatatugStore("")), for resolver calls that need one.

func QueryIDIndex added in v0.20.5 

func QueryIDIndex(projectDir string) (map[string]string, error)

QueryIDIndex walks every queries/**/*.query.json file under projectDir, mapping each one's canonical (folder-qualified, "/"-joined) id to its bare (folder-relative) id — a query directly under queries/ (no subfolder) has canonical == bare. A missing queries/ directory is not an error (an empty project tree is valid); it simply produces an empty index.

func RemoveRowsFromRecordset 

func RemoveRowsFromRecordset(params RecordsetDataRequestParams, rows []RowWithIndex) (numberOfRecordsAffected int, err error)

RemoveRowsFromRecordset removes rows from a recordset

func ResolveCatalogPath added in v0.19.0 

func ResolveCatalogPath(projectDir, catalogPath string) (string, error)

ResolveCatalogPath expands a datatug.DbCatalog.Path field into an absolute filesystem path the way this repo's real catalog data actually uses it (see datatug-demo-projects/demo-project-1's environments/*/catalogs/*/*.db.json, S45's dead-layout cleanup, e.g. "~/datatug/dbs/chinook-local.sqlite"):

  • a leading "~" or "~/..." expands to the resolved home directory (github.com/mitchellh/go-homedir)
  • a leading "$HOME" or "${HOME}" expands the same way
  • any other relative path resolves against projectDir (the project's own on-disk location), so it means the same thing regardless of the caller's working directory
  • an already-absolute path (after the above) is returned unchanged

datatug-core's DbCatalogBase.Path has no documented convention for any of these forms; this is this repo's one shared answer, used by this package's own sourceURLFromCatalog and by apps/datatugapp/commands/cmd_query_run_saved.go's equivalent for `datatug query run --project/--query` (S52 found sourceURLFromCatalog building "sqlite://" + catalog.Path with no expansion at all, unopenable against demo-project-1's real catalog data — this closes that gap at its root instead of working around it at each call site).

func ResolveDescriptorPath added in v0.55.0 

func ResolveDescriptorPath(projectDir, catalogPath string) (string, error)

ResolveDescriptorPath is the file that the path of a PostgreSQL catalog names, which is the connection descriptor of the catalog, below projectDir. The descriptor names the environment variable that holds the connection URL, and a project file is not trusted (a cloned project must not be able to point a reader at any file of the machine), so the path must be a path inside the project folder: relative, with no ".." that leaves the folder, not starting with "~" or "$" (which ResolveCatalogPath reads as the home directory), and not a path a link leads out of the folder by. A path that is a URL is refused as ResolveCatalogPath refuses it.

func ResolveIncidentProject added in v0.28.0 

func ResolveIncidentProject(projectID, environment string) (incidents.ProjectRef, error)

ResolveIncidentProject separates source-project provenance from the incident/evidence store route. A local serve currently exposes one source store per configured project; future adapters can resolve other store kinds without changing IncidentScope.

func ResolveQueryID added in v0.20.5 

func ResolveQueryID(projectDir, id string) (string, error)

ResolveQueryID resolves id to the canonical, folder-qualified query id datatug-core's fsQueriesStore.LoadQuery actually needs to find a query that lives in a subfolder (every query in the demo project does — GET /datatug/queries/get_query?project=...&query=customer-invoices used to 500 with the store's own raw "open .../customer-invoices.query.json: no such file or directory" for exactly this reason).

Canonical-id convention (this stream's decision, marked as an assumption — not a founder ruling; api-contract.md's own Candidate/ExecutionRequest types declare `queryId: string` with no format guidance either way): the folder-qualified id ("<folder>/<bare-id>", "/"-joined, matching fsQueriesStore.LoadQuery's own id-splitting convention) is canonical because it is exactly what the filesystem layout already encodes and is guaranteed unique. A bare id is still accepted on input, but only when it names exactly one query across every folder in the project.

  • id containing "/": trusted as already folder-qualified; resolved (returned as-is) only when a query file exists at that exact path — ErrQueryNotFound otherwise.
  • id with no "/": matched against every query file's bare (folder-relative) id. Exactly one match resolves to its canonical form; zero is ErrQueryNotFound; more than one is ErrAmbiguousQueryID, naming every canonical candidate so the caller can pick one.

func ResolveScanDbModel added in v0.54.3 

func ResolveScanDbModel(projectDir, environment, catalogID, flag string) (string, error)

ResolveScanDbModel is the database model a scan of catalog catalogID in environment puts the catalog on, for the project at projectDir, given the value of --dbmodel (flag, "" when it is not given, and a plain name when it is). The environment and the catalog id are plain names too, as they are names of folders of the project.

A catalog the project already records has a model, and a scan keeps it: without the flag the model is the one the catalog file names, and a flag that names another model is an error that names both, because scanning onto it would leave the tables of the first model in the project for good and make a second model of the same database. A catalog the project does not record in this environment yet is put on the model of the flag, or else, as one database is one model in every environment, on the model that the other environments record for a catalog of that id when they all name the same one; when they name different ones the scan is refused, naming them, as no model is the obvious one; and when none records one it is put on the model called as the database. A model the catalog file names that cannot be the name of a folder, or a catalog file that cannot be read, records nothing: the scan writes the file anew.

func ResolveScanDbModelNoted added in v0.55.0 

func ResolveScanDbModelNoted(projectDir, environment, catalogID, flag string) (model, note string, err error)

ResolveScanDbModelNoted is ResolveScanDbModel that also returns a sentence, one line, for a scan that is put on a model the database id did not give, by what the other environments record: a scan with no --dbmodel of a catalog this environment does not record yet, which a person who named no model would not expect to find on a model of another name. It is empty for every other answer, and when the model taken is called as the database (that is what no flag gives anyway).

func ResolveStoreID added in v0.20.2 

func ResolveStoreID(explicit, projectID string) (string, error)

ResolveStoreID picks the store id a "keep-as-is" GET/mutation route should use for projectID, replacing the previous hardcoded "firestore" default (pkg/server/endpoints/constants.go) that was simply wrong for a `datatug serve --project` session — no Firestore store is ever configured there, so every request silently defaulting to it failed downstream with "no store configured for id=firestore" (S85's finding).

  • explicit != "": honored only when it names a store actually configured for projectID (ErrUnknownStoreID otherwise) — an explicit ?storage= is never silently overridden or ignored.
  • explicit == "": defaults to the one configured store; zero configured stores is ErrUnknownStoreID (nothing serves this project — a distinct concern from "project not found", which callers already validate separately); several is ErrAmbiguousStore (see its own doc comment).

func SaveBoard 

func SaveBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

SaveBoard saves board

func SaveEntity 

func SaveEntity(ctx context.Context, ref dto.ProjectRef, entity *datatug.Entity) error

SaveEntity saves board

func SaveScannedProject added in v0.54.2 

func SaveScannedProject(ctx context.Context, store datatug.ProjectStore, projectDir string, project *datatug.Project, scanned ScannedCatalog, warnings io.Writer) error

SaveScannedProject saves project, a project UpdateDbSchema returned, and then writes the files of the scanned catalog that datatug-core does not write: the catalog file and one columns file for each table and view.

The project is saved without the schemas of its database models: a model file is an id and its environments, and the tables are the columns files. A schema, or a table or view, whose name cannot be a folder name, or that differs from another of its kind in the same folder by case only, or whose columns file would have a name longer than a file can have, is left out of the project and named on warnings; the others are still written. Nothing is written when the project is invalid.

A scan of an environment keeps what the scans of the other environments of the same database model wrote (see mergeColumns), and takes back what its own earlier scans wrote of a table or view that the database no longer has (see planRetractions): the folder of one that no environment has is removed, and said on warnings, one line each. The first scan of a catalog in an environment (the project holds no catalog file of it there) has no earlier scan of its own, so it takes nothing back: what is in the model for that environment was written by something else. Nothing is written when a folder that is to be removed is, or is inside, a link.

The environment, the catalog and the database model of the scan are names of folders of the project, and one that is not a plain name (see CheckScanName), or that differs only by case from one the project has (see CheckScanNamesAgainstProject), is refused before anything is read, written or removed: this function removes folders, and it does not take its ids on trust from whoever calls it. A catalog that the project records under another driver in the same environment is refused the same way (see CheckScanDriverAgainstProject).

func ScannedServer added in v0.54.3 

func ScannedServer(driver string, dbConnParams dbconnection.Params) datatug.ServerRef

ScannedServer is the server a scan of driver with dbConnParams reads, as the project records it: for a driver with a host (see serverHasHost) the host and the port of the connection parameters, and for the others the driver only.

func SecureConfiguredProjectIDs added in v0.20.0 

func SecureConfiguredProjectIDs() []string

SecureConfiguredProjectIDs returns every project ID this process is serving (ConfigureSecureSession's pathsByID keys), for agent-info's "projects" array.

func SecureExecutor added in v0.16.0 

func SecureExecutor() (*secureread.Executor, bool)

SecureExecutor returns the Executor ConfigureSecureSession built, and false when serve has not configured one yet (e.g. a handler under test with no ConfigureSecureSession call).

func SecureIncidentActor added in v0.28.0 

func SecureIncidentActor(via string) (incidents.Actor, error)

SecureIncidentActor returns the server-attested event actor. Mutation endpoints never accept actor or reporter identity from callers.

func SecurePolicyFingerprint added in v0.27.0 

func SecurePolicyFingerprint() string

SecurePolicyFingerprint identifies the exact policy session that decided the currently served reads without exposing policy text or filesystem paths.

func SecurePrincipalID added in v0.16.0 

func SecurePrincipalID() string

SecurePrincipalID returns the serving principal's ID for `agent-info` to report (REQ:principal-selection), or "" when the session carries no identified principal (Unrestricted with no --as, or a role/group-only principal).

func SecurePrincipalRolesGroups added in v0.20.0 

func SecurePrincipalRolesGroups() (roles, groups []string)

SecurePrincipalRolesGroups returns the serving principal's Roles/Groups for agent-info's exact envelope (AgentInfoPrincipal.Roles/Groups) — never nil, so JSON encoding always emits "[]" rather than "null" for an unset slice, matching the appendix's "roles:string[]" / "groups:string[]".

func SecureSessionUnrestricted added in v0.20.0 

func SecureSessionUnrestricted() bool

SecureSessionUnrestricted reports whether this process's session runs with no access-policy enforcement at all (--no-policies).

func SecurityContextID added in v0.20.0 

func SecurityContextID() string

SecurityContextID returns the securityContextId agent-info reports and every scoped request must echo back (ValidateSecurityContext).

func SetOpenSchemaScanForTest added in v0.55.0 

func SetOpenSchemaScanForTest(open func(dbcopy.BackendRef, context.Context) (dbcopy.SchemaScanDB, error)) (restore func())

SetOpenSchemaScanForTest replaces how a PostgreSQL scan opens its source, for the tests of a package that runs the real scan command (the command's own package): a test of the scan must never dial a server, and every scan opens through here. It returns the function that puts the previous open back.

func UpdateDbSchema 

func UpdateDbSchema(ctx context.Context, projectLoader ProjectLoader, projectID, environment, driver, dbModelID string, dbConnParams dbconnection.Params) (project *datatug.Project, err error)

UpdateDbSchema updates DB schema

func UpdateDbServer 

func UpdateDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

UpdateDbServer adds db server to project

func UpdateQuery 

func UpdateQuery(ctx context.Context, request dto.UpdateQuery) (*datatug.QueryDefWithFolderPath, error)

UpdateQuery is the legacy queries/update_query write. See saveLegacyQuery.

func UpdateRowsInRecordset 

func UpdateRowsInRecordset(params RecordsetDataRequestParams, rows []RowWithIndexAndNewValues) (numberOfRecordsAffected int, err error)

UpdateRowsInRecordset updates rows in a recordset

func ValidateCatalogIdentifiers added in v0.54.4 

func ValidateCatalogIdentifiers(environmentID, catalogID string) error

ValidateCatalogIdentifiers is the check of the two IDs that GET /datatug/catalog-tables turns into folder names, environment first. Both the route and GetCatalogTables apply it, so the refusal is the same wherever it is made.

func ValidateIdentifier added in v0.54.4 

func ValidateIdentifier(field, id string) error

ValidateIdentifier refuses an environment, catalog or project-item ID that a client sent and that becomes a folder or file name: it returns nil when id is a plain name (see dbcopy.IsPlainSourceID: letters and digits of any script, "." "_" and "-", at most 128 characters, starting with a letter or a digit) and a bad-request error for field otherwise.

A plain name is one path segment whatever the platform: it holds no path separator of either kind, no drive letter, no NUL, no "%" (so a separator that is still percent-encoded after the request's own decoding stays inert) and is never "." or "..". The error names the field and the rule and nothing of id: id may be a whole source string, or the text of an attack, and is not echoed.

func ValidatePathIdentifier added in v0.55.1 

func ValidatePathIdentifier(field, id string) error

ValidatePathIdentifier refuses an ID with folders that a client sent and that becomes a path under the project (a recordset definition, a folder): it returns nil when every "/"-separated part of id is a plain name (see ValidateIdentifier), and a bad-request error for field otherwise. A part that is empty, "." or ".." is not a plain name, so there is no absolute path, no "a//b", no trailing slash and no way out of the folder that holds the item; so is an id of more than MaxPathIdentifierLength bytes in all. The error names the field and the rule and nothing of id.

func ValidateProjDbServerNames added in v0.55.1 

func ValidateProjDbServerNames(server datatug.ProjDbServer) error

ValidateProjDbServerNames refuses a db server that is saved whose driver, host or ID a project could not record: the driver and the host pass their rules (see ValidateServerRef), and an ID that is given is the ID of its reference, which is the file name the server is saved under. It checks names only, so the route that takes a server in its body can run it before the validation the server has of its own, which quotes the driver and the IDs it refuses.

func ValidateProjectIdentifier added in v0.55.1 

func ValidateProjectIdentifier(field, id string) error

ValidateProjectIdentifier refuses a project ID that a client sent unless this process serves that project (the ID is then a key it holds, whatever the project is called) or the ID is a plain name. A project ID is never joined into a path by this package, but it names the project directory the store opens, so an ID that is neither served nor a plain name is refused before a store is asked for it. It does not say that a plain name is served: a route that resolves the store of its project does (see ResolveStoreID), and so does each entry that takes the project from the body of the request itself and opens its store (ExecuteSelect, ExecuteCommands, CreateFolder, CreateQuery, UpdateQuery; see servedProjectDir), with the same answer, before a project store is asked for.

func ValidateSecurityContext added in v0.20.0 

func ValidateSecurityContext(id string) bool

ValidateSecurityContext reports whether id matches this process's current securityContextId. A caller with an empty configured ID (no ConfigureSecureSession call yet — a handler under test) always fails closed rather than accepting any value.

func ValidateServerRef added in v0.55.1 

func ValidateServerRef(server datatug.ServerRef) error

ValidateServerRef refuses a db server whose driver or host a project could not record under the file name of the server (the driver is a folder of the project, and the driver and the host are the ID of the server, which is a file name): the driver must be a plain name that a db server can have, and the host a host, and none for a server of a file or a URL (sqlite3 and the like). An empty driver or host passes, as it is the missing-field answer of datatug.ServerRef.Validate. The error names the field and the rule and nothing of the value: the refusals of datatug.ServerRef.Validate that are left for the routes to give do not quote the driver or the host (see DriverRule and HostlessRule).

func WarnMissingSourceFiles added in v0.20.1 

func WarnMissingSourceFiles(ctx context.Context, pathsByID map[string]string)

WarnMissingSourceFiles logs one warning line per environment source (SQL/inGitDB catalog) whose file-backed path does not exist, for every project in pathsByID. `datatug serve --project` used to surface this only as a raw 500 on the first request that happened to touch the missing source (S77's finding: a fresh checkout with no prior `datatug demo` run has no ~/datatug/dbs/chinook-local.sqlite) — this gives the operator the same signal at startup, before the browser does. Best-effort: a project or environment this cannot enumerate is skipped silently rather than failing serve's startup over a diagnostic.

func WithScanWarnings added in v0.54.3 

func WithScanWarnings(ctx context.Context, w io.Writer) context.Context

WithScanWarnings returns a context whose scan names, on w, what it leaves out of the catalog it reads and why, one line for each: the `datatug scan` command gives it its error stream. Without it the scan says nothing of what it leaves out.

Types

type Capabilities added in v0.20.0 

type Capabilities struct {
	AllowWrites    bool
	AllowOpaqueSQL bool
	// HTTPOffline is `datatug serve --http-offline` (Phase 1 Task 14, item
	// 7): when true, every HTTP-typed saved query's LIVE fetch fails as
	// SOURCE_UNAVAILABLE without ever touching the network (see
	// pkg/httpsource.ContextWithDispatch's offline argument), so a demo can
	// prove "network disabled -> honest failure -> explicit snapshot"
	// deterministically. It is a real operator-facing switch (offline demos),
	// not a test hook. False (the default) leaves live dispatch unaffected.
	HTTPOffline bool
	// ExecTimeout overrides pkg/server/endpoints' default 10-second
	// exec/run_query budget (api-contract.md "Bounded lookups and HTTP":
	// "Server execution has a 10-second default timeout, with a configured
	// upper bound of 30 seconds"). Zero means "use the default"; any value
	// above the 30-second ceiling is clamped down to it — see
	// pkg/server/endpoints/contract_scope.go's execTimeoutFor.
	ExecTimeout time.Duration
	// IncidentStores are trusted shared incident/evidence repository routes.
	// EvidencePrivateDir holds server-private snapshot sidecars.
	IncidentStores     []incidentstore.ConfiguredStore
	EvidencePrivateDir string
	EvidenceByteCap    int
	EvidenceRetention  time.Duration
	SnapshotPolicies   map[string]SnapshotProjectPolicy
}

Capabilities are the operator-controlled flags `datatug serve` fixes for its whole process life, beyond the principal/policy set itself: AllowWrites gates every project-mutation route (create/save/delete project/query/board/entity/recordset-rows) closed by default (api-contract.md "Security and errors": "this read journey must not expose an unauthenticated mutation endpoint as a side effect" — REQ:principal-selection: "Unneeded write endpoints MUST fail closed"). AllowOpaqueSQL gates the "separate explicit opaque-query grant" the appendix's REQ:opaque-sql-limitation describes: without it, a SQL-typed saved query (or any native-SQL execution path) is refused before dispatch with UNSUPPORTED_PROTECTED_EXECUTION, matching "the support demo has no such grant". Both default false: Phase 1's read journey is safe-by-default unless an operator explicitly opts a serve process into either capability.

func GetCapabilities added in v0.20.0 

func GetCapabilities() Capabilities

GetCapabilities returns this process's configured Capabilities.

type CatalogColumn added in v0.35.0 

type CatalogColumn struct {
	Name   string `json:"name"`
	DbType string `json:"dbType,omitempty"`
	// PrimaryKeyPosition is the column's 1-based place in its table's primary
	// key, read from the columns file's pkPosition; 0 when it is not part of
	// the key.
	PrimaryKeyPosition int `json:"pkPosition,omitempty"`
}

CatalogColumn is the compact, model-facing description of one stored database column. It intentionally excludes environment scan bookkeeping: chat needs enough context to construct a query, not the complete dbmodel.

type CatalogRelation added in v0.35.0 

type CatalogRelation struct {
	Schema  string          `json:"schema"`
	Name    string          `json:"name"`
	DbType  string          `json:"dbType"`
	Columns []CatalogColumn `json:"columns"`
	Issue   string          `json:"-"` // partial explorer load only; never publish raw file errors
}

CatalogRelation describes one stored table or view and its columns.

type CatalogSchema added in v0.35.0 

type CatalogSchema struct {
	Relations []CatalogRelation `json:"relations"`
}

CatalogSchema is the compact stored schema supplied to consumers such as DataTug Chat. DataTug's scanned dbmodel remains the source of truth; this function never introspects the live database independently.

func GetCatalogSchema added in v0.35.0 

func GetCatalogSchema(projectDir, environmentID, catalogID string) (*CatalogSchema, error)

GetCatalogSchema resolves a configured catalog to its scanned dbmodel and loads table/view columns in deterministic order.

func GetCatalogSchemaPartial added in v0.46.3 

func GetCatalogSchemaPartial(projectDir, environmentID, catalogID string) (*CatalogSchema, error)

GetCatalogSchemaPartial preserves healthy relations when one relation's stored column file is missing or malformed. Source-level failures still return an error; affected relations carry a local-only Issue instead.

type CatalogTable added in v0.23.0 

type CatalogTable struct {
	Schema string `json:"schema"`
	Name   string `json:"name"`
	DbType string `json:"dbType,omitempty"`
}

CatalogTable is the minimal {schema, name, dbType} identity datatug-apps' ITableFull (libs/datatug/main/src/lib/models/definition/apis/database.ts) already declares — no column/key detail, which env-db-table.page.ts's own /exec/select-backed row fetch remains the source of. Defined here rather than reusing datatug-core's own datatug.TableModel: that type embeds DBCollectionKey, whose schema/catalog/name fields are unexported with no custom MarshalJSON, so it serializes to "{}" over JSON — a file-storage/lookup-key type, not a wire DTO.

type CatalogTables added in v0.23.0 

type CatalogTables struct {
	Tables []CatalogTable `json:"tables"`
	Views  []CatalogTable `json:"views"`
}

CatalogTables is GET /datatug/catalog-tables's response (Task 17 item A.2, S121): the catalog table/view list datatug-apps' EnvDbPageComponent (the catalog overview page one level above env-db-table.page.ts's own /table/<type> route) never had any way to populate before this — S120's report found nothing in this app fed it (no in-app link even targeted the route, and no endpoint returned this shape). Verified live, against a real `datatug serve` agent, that none of the three existing candidates api-contract.md-adjacent code already calls carries it:

  • dbserver-databases needs a live driver/host/port to introspect an actual DB *connection* (the "add a new server" flow) — unrelated to an already-registered project catalog.
  • projects/project_full's dbModels only ever carry {id, environments[].DbCatalogs[].id}: Project.LoadProject never populates DbModel.Schemas (confirmed against a live response).
  • datatug-core's own DbModelsStore/fsDbModelsStore.LoadDbModel is entirely commented out (storage/filestore/store_dbmodels.go) — the same class of "schemer providers moved out" gap the sibling queries/all_queries route (this same task) restores.

So this reads the project's already-scanned dbmodel files directly, the same direct-filesystem-walk pattern loadModuleQueries/loadModuleEntities (pkg/server/endpoints/semantic_project.go) already use for an identical ProjectStore-incompleteness gap.

func GetCatalogTables added in v0.23.0 

func GetCatalogTables(projectDir, environmentID, catalogID string) (*CatalogTables, error)

GetCatalogTables resolves environmentID+catalogID to their dbModel (via the catalog's own <id>.db.json file) and lists every table/view file under that dbModel's dbmodels/<dbModel>/<schema>/{tables,views}/ tree. Read-only; unlike exec/select or dbserver-databases this touches no live database connection.

type CommandExecutionResult added in v0.19.0 

type CommandExecutionResult struct {
	CommandID           string                `json:"commandId"`
	ElapsedMilliseconds int64                 `json:"elapsed,omitempty"`
	Items               []CommandResponseItem `json:"items"`
}

CommandExecutionResult is one ExecuteCommandsResponse.Commands entry, matching datatug-apps' ICommandResponse (commandId, elapsed?, items).

type CommandResponseItem added in v0.19.0 

type CommandResponseItem struct {
	Type  string      `json:"type"`
	Value interface{} `json:"value,omitempty"`
}

CommandResponseItem is one CommandExecutionResult.Items entry, matching datatug-apps' ICommandResponseItem (type, elapsed?, value?). Value is always a QueryResultResponse for the one command type this endpoint implements (SQL) — the same columns/rows/limitations[] shape exec/select and run_query already return (REQ:limitation-visible), so the web UI's recordset rendering has one wire shape to handle regardless of which endpoint produced it.

type ExecuteCommandRequest added in v0.19.0 

type ExecuteCommandRequest struct {
	ID   string `json:"id,omitempty"`
	Type string `json:"type"`
	Text string `json:"text"`
	Env  string `json:"env"`
	DB   string `json:"db"`
	// NamedParams is decoded so a request that sends it gets a clear
	// "not supported" error (see ExecuteCommands) rather than having the
	// parameters silently ignored — RunNativeSQL executes opaque SQL text
	// with no bind-parameter surface (REQ:opaque-sql-limitation). The web
	// client already avoids this combination itself: a single command with
	// namedParams routes through GET /exec/select instead (agent.service.ts,
	// AgentService.execute).
	NamedParams map[string]any `json:"namedParams,omitempty"`
}

ExecuteCommandRequest is one entry of ExecuteCommandsRequest.Commands, matching datatug-apps' ISqlCommandRequest wire shape exactly (id?, type, text, env, db, namedParams?). Only Type "SQL" is implemented: it is the only command type agent.service.ts's execute() ever actually constructs (IHttpCommand exists as a client-side type but nothing builds one to POST here).

func (ExecuteCommandRequest) Validate added in v0.19.0 

func (v ExecuteCommandRequest) Validate() error

Validate returns an error if the command is not well-formed.

type ExecuteCommandsRequest added in v0.19.0 

type ExecuteCommandsRequest struct {
	ID string `json:"id"`
	// Project is "omitempty": the real client never sends it in the body at
	// all (it is filled in from the `?project=` query parameter before the
	// body is decoded — see executeCommandsHandler), and omitempty keeps a
	// body that does not set it from clobbering that query-derived value
	// back to "" the way a bare `json:"project"` zero-value would on
	// decode. A body that DOES explicitly set "project" still overrides it,
	// same as before.
	Project string `json:"project,omitempty"`
	// Commands are decoded from the request body; StoreID is not part of the
	// body at all and is passed to ExecuteCommands separately (see
	// executeCommandsHandler), matching ExecuteSelect/RunQuery.
	Commands []ExecuteCommandRequest `json:"commands"`
}

ExecuteCommandsRequest is the body of POST /datatug/exec/execute_commands (datatug-apps' agent.service.ts, AgentService.execute): unlike pkg/sqlexecute.Request (a direct db-server connection carrying a raw driver/host ServerRef, still used by GetServerDatabases), every command's Env/DB here are project-relative identifiers resolved exactly the way ExecuteSelect/RunQuery resolve them (resolveSourceURL) — the web client never sends a driver/host ServerRef, and sqlexecute.RequestCommand's own Validate() (which requires one) would reject every request this endpoint actually receives. Project comes from the `?project=` query parameter, mirroring how execute_endpoints.go already reads it — the JSON body itself carries no project field (agent.service.ts's execute() deletes projectId from the body before POSTing it).

func (ExecuteCommandsRequest) Validate added in v0.19.0 

func (v ExecuteCommandsRequest) Validate() error

Validate returns an error if the request is not well-formed.

type ExecuteCommandsResponse added in v0.19.0 

type ExecuteCommandsResponse struct {
	// DurationMilliseconds is the wall-clock time every command in this
	// request took, combined.
	DurationMilliseconds int64                    `json:"duration"`
	Commands             []CommandExecutionResult `json:"commands"`
}

ExecuteCommandsResponse is POST /datatug/exec/execute_commands's response, matching datatug-apps' IExecuteResponse wire shape (duration, commands: ICommandResponse[]).

func ExecuteCommands 

func ExecuteCommands(ctx context.Context, storeID string, request ExecuteCommandsRequest) (ExecuteCommandsResponse, error)

ExecuteCommands runs every command in request through the policy-enforced secureread.Executor (REQ:server-acl-all-reads), the same path api.ExecuteSelect and api.RunQuery already use — see routes.go's executeRoutes / execute_endpoints.go. It replaces the `panic("not implemented yet")` this function used to be: the previous signature took a pkg/sqlexecute.Request, whose RequestCommand embeds a datatug.ServerRef requiring a concrete driver/host, which is not the shape datatug-apps' web client actually sends (see ExecuteCommandsRequest's doc comment) — a real implementation against that old signature would have rejected every real request anyway.

A command's own error (an unsupported source scheme, a policy refusal, a database that can't be resolved) aborts the whole request with that error — matching sqlexecute's original all-or-nothing Response.Commands contract, and (for a secureread.ErrAccessDenied) letting util_error_handling.go's existing ACCESS_DENIED/403 mapping apply exactly as it does for exec/select and run_query, with no separate handling needed here.

type PostgresScanParams added in v0.54.0 

type PostgresScanParams struct {
	// contains filtered or unexported fields
}

PostgresScanParams are the connection parameters of a PostgreSQL scan. The connection is an environment variable that holds the whole URL, so these parameters print as the variable's name, never as the URL: no verb of fmt shows the password or the rest of the URL, for the parameters or a pointer to them (String and GoString). Their accessors do carry the host, port and user of the URL, as dbconnection.Params has them; the scan records none of them in a project (ScannedServer), and names what it connects to by Display, which holds no user and no password.

func NewPostgresScanParams added in v0.54.0 

func NewPostgresScanParams(lookupEnv func(string) (string, bool), dsnEnv, environment, catalog string) (*PostgresScanParams, error)

NewPostgresScanParams resolves the environment variable dsnEnv, which must hold a postgres:// URL, and names the catalog of environment that the scan will write. lookupEnv reads the environment.

The environment and the catalog each name a file under the project's connections folder, so each must be a plain name (dbcopy.IsPlainSourceID, the one definition: the routes that read the project apply the same). The host of the URL is named on the log line of the scan, so it must be a host name or an address.

func (*PostgresScanParams) Catalog added in v0.54.0 

func (p *PostgresScanParams) Catalog() string

Catalog implements dbconnection.Params: the catalog ID in the project.

func (*PostgresScanParams) ConnectionString added in v0.54.0 

func (p *PostgresScanParams) ConnectionString() string

ConnectionString implements dbconnection.Params. It is the "env:NAME" source and never the URL: nothing that formats these parameters can print a password.

func (*PostgresScanParams) DSNEnv added in v0.54.0 

func (p *PostgresScanParams) DSNEnv() string

DSNEnv is the name of the environment variable that holds the URL.

func (*PostgresScanParams) Display added in v0.55.0 

func (p *PostgresScanParams) Display() string

Display is what the scan says it connects to: the scheme, host, port and database of the URL (dbcopy.SourceDisplay), never its user, its password or its query string. It is built from parts that each passed a strict check, never from a scrubbed copy of the URL.

func (*PostgresScanParams) Driver added in v0.54.0 

func (*PostgresScanParams) Driver() string

Driver implements dbconnection.Params.

func (PostgresScanParams) GoString added in v0.54.0 

func (p PostgresScanParams) GoString() string

GoString makes %#v print the same text as String, not the fields.

func (*PostgresScanParams) Mode added in v0.54.0 

func (*PostgresScanParams) Mode() dbconnection.Mode

Mode implements dbconnection.Params: a scan only reads.

func (*PostgresScanParams) Path added in v0.54.0 

func (p *PostgresScanParams) Path() string

Path is the project-relative path of the connection descriptor the scan writes, which UpdateDbSchema records as the catalog's path.

func (*PostgresScanParams) Port added in v0.54.0 

func (p *PostgresScanParams) Port() int

Port implements dbconnection.Params: the port of the URL, 0 when it names none.

func (*PostgresScanParams) Server added in v0.54.0 

func (p *PostgresScanParams) Server() string

Server implements dbconnection.Params: the host of the URL.

func (*PostgresScanParams) SourceRef added in v0.54.0 

func (p *PostgresScanParams) SourceRef() dbcopy.BackendRef

SourceRef is the source the scan opens: its Raw is "env:NAME", its Path the real URL, which only the open path may use.

func (PostgresScanParams) String added in v0.54.0 

func (p PostgresScanParams) String() string

String implements dbconnection.Params, with the same text as ConnectionString. It has a value receiver so that the parameters print the same through a pointer and as a value: fmt cannot call a method on the unexported fields that hold the URL, and would print them as they are.

func (*PostgresScanParams) User added in v0.54.0 

func (p *PostgresScanParams) User() string

User implements dbconnection.Params: the user of the URL.

func (*PostgresScanParams) WriteDescriptor added in v0.54.0 

func (p *PostgresScanParams) WriteDescriptor(projectDir string) (undo func() error, err error)

WriteDescriptor writes the connection descriptor into projectDir: a file that names the environment variable and nothing else, so the descriptor holds no password, user or host, and neither does any other file the scan writes (the project's server entry is the driver alone). It replaces a descriptor already there, and does not write one that is as it would be written.

It does not follow a link. A project is not trusted (ResolveDescriptorPath refuses a descriptor that a link leads out of the folder to), so a link at connections, at connections/<env> or at the descriptor itself is refused, naming the path in the project that it is, before anything is read through it or written: the scan would otherwise read a file of the machine into memory, truncate it and write the descriptor there. A folder or a file the project has in a place of the path that is not a plain folder, or not a plain file, is refused the same way.

It returns the function that takes the write back, for a scan whose save fails: the descriptor this write made is removed, with the folders it made, and a descriptor that was there before is put back as it was.

type ProjectLoader 

type ProjectLoader interface {
	LoadProjectFile(ctx context.Context) (projectFile datatug.ProjectFile, err error)
	LoadProject(ctx context.Context, o ...datatug.StoreOption) (project *datatug.Project, err error)
}

ProjectLoader defines an interface to load project info

type QueryResultResponse added in v0.16.0 

type QueryResultResponse struct {
	Columns     []string                 `json:"columns"`
	Rows        []map[string]any         `json:"rows"`
	Limitations []apicontract.Limitation `json:"limitations"`
	Provenance  apicontract.Provenance   `json:"provenance"`
}

QueryResultResponse is the JSON shape every policy-enforced read endpoint returns: the columns and rows the principal is allowed to see, plus the SAME limitations/provenance shapes exec/run_query's apicontract.Result returns (S101 — datatug-core v0.27.3 pkg/apicontract's own Limitation and Provenance types, reused directly rather than a separately-shaped DTO). Additive: {columns, rows} are unchanged; a caller that reads only those two fields is unaffected. Limitations is empty (never null, never omitted) for an unrestricted/admin read — "nothing applies" is a real, observable state (AC hidden-column-refused's sibling: "an empty limitation list does not authorize opaque execution" applies here in reverse — an empty list here truly means nothing was restricted).

func ExecuteSelect 

func ExecuteSelect(ctx context.Context, storeID string, request SelectRequest) (QueryResultResponse, error)

ExecuteSelect executes a select through the policy-enforced secureread.Executor (REQ:server-acl-all-reads): a "from" selection runs through RunStructured, raw "sql" text through RunNativeSQL. Every read the web UI can trigger through `datatug serve` MUST come through this one path — see routes.go's executeRoutes / execute_endpoints.go.

type RecordsetDataRequestParams 

type RecordsetDataRequestParams struct {
	RecordsetRequestParams
	Data string `json:"data"`
}

RecordsetDataRequestParams is a set of common request parameters

func (RecordsetDataRequestParams) Validate 

func (v RecordsetDataRequestParams) Validate() error

Validate returns error if not valid

type RecordsetRequestParams 

type RecordsetRequestParams struct {
	Project   string `json:"project"`
	Recordset string `json:"recordset"`
}

RecordsetRequestParams is a set of common request parameters

func (RecordsetRequestParams) Validate 

func (v RecordsetRequestParams) Validate() error

Validate returns error if not valid

type ResolvedSource added in v0.20.0 

type ResolvedSource struct {
	ID    string
	Label string
	Kind  SourceKind
	URL   string
	// Collection is the fixed collection name a caller must query this
	// source through: for SQL/inGitDB-via-catalog it is whatever the caller
	// asks for (any table/collection in that database); for an HTTP source
	// it is fixed to ID itself (one QueryDef = one dalgo2http collection —
	// see pkg/httpsource.BuildCollection, "Name: def.ID").
	Collection string
}

ResolvedSource is one entry of the project's unified source registry: the api-contract.md "Scope and identity" resolver plan task 12 requires — "Semantic discovery and execution use the same resolver" — adapting existing environment/catalog records (SQL, inGitDB) and HTTP QueryDefs into one list, with no second persisted store.

ID is the appendix's SourceRef.source: a STABLE project-local identifier. For a SQL/inGitDB catalog this is the catalog's DbCatalogBase.DbModel (e.g. "chinook") — the identifier datatug-demo-projects/demo-project-1's own EntityField.Mappings already key by (verified against the real project: Customer.ID's mapping is {source:"chinook", ...}, NOT {source:"chinook-local"}), and the one the pre-existing semantic endpoints (pkg/server/endpoints/semantic_*.go) already receive as "source" from the web client — so resolving it against the SAME registry execution now shares is the unification this task exists to do, not a new convention. The catalog's own project-item ID (e.g. "chinook-local", distinct per environment) is accepted too, as a fallback alias, so a caller that already has a concrete catalog ID (the CLI's older --db flag shape) keeps working. See ResolveSource.

For a project-level inGitDB recordset (e.g. "support-notes") ID is the recordset definition's own ID, environment-independent: this demo project's data/ingitdb store is one shared directory, not scoped per environment (verified: recordsets/support-notes.recordset.json declares no environment, and semantic_project.go's existing semanticIngitdbPath already treats it as project-wide).

For an HTTP QueryDef (e.g. "country-facts") ID is the QueryDef's own ID, also environment-independent (an HTTP QueryDef's Targets must be empty — datatug-core's QueryDef.Validate enforces this — so it is not tied to any environment/catalog record at all).

func EligibleTargets added in v0.20.0 

func EligibleTargets(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment string, queryDef *datatug.QueryDef) ([]ResolvedSource, error)

EligibleTargets returns the authorized ResolvedSource options a saved query may run against within environment, per api-contract.md "Scope and identity": an HTTP query's only eligible target is itself (its Targets MUST be empty per datatug-core's own QueryDef.Validate); a SQL/DTQL query with declared Targets is filtered to catalogs whose (Driver, Catalog) matches one of them; a SQL/DTQL query with NO declared Targets (every query in datatug-demo-projects/demo-project-1 today) is eligible against every SQL/inGitDB catalog source in environment — a deliberate, documented engineering default for Phase 1's single-catalog-per-environment demo, not a founder ruling: a project that registers more than one same-model catalog per environment without the query declaring explicit Targets will see every one of them offered, which is the conservative (ask rather than guess) behaviour api-contract.md's "with multiple targets the user selects a source explicitly" already calls for.

func ListSources added in v0.20.0 

func ListSources(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment string) ([]ResolvedSource, error)

ListSources enumerates every source this project's registry can resolve for environment: every SQL/inGitDB catalog registered on that environment's DB servers (keyed by BOTH DbModel and catalog ID — see ResolvedSource's doc), every project-level inGitDB recordset definition, and every project-level HTTP QueryDef. Order is deterministic (by Kind then ID) so a Candidate's target list and an "unknown source" error's implied option set are stable across calls.

func ResolveSource added in v0.20.0 

func ResolveSource(ctx context.Context, projStore datatug.ProjectStore, projectDir, environment, source string) (ResolvedSource, error)

ResolveSource resolves one SourceRef.source (see ResolvedSource's doc for the two accepted forms — DbModel or catalog ID) within environment to a pkg/dbcopy-openable ResolvedSource. It is the ONE place semantic discovery (pkg/server/endpoints/semantic_*.go) and execution (RunQuery/ExecuteSelect/exec/run_query) resolve a source, replacing the two previously-diverging conventions (semantic's broken "<projectDir>/dbs/<source>.sqlite" guess and execution's environment/ catalog walk) — see the PR body's inventory for the specifics.

type RowValues 

type RowValues = map[string]interface{}

RowValues set of named values

type RowWithIndex 

type RowWithIndex struct {
	Index  int                    `json:"index"`
	Values map[string]interface{} `json:"values"`
}

RowWithIndex points to specific row with expected values

func (RowWithIndex) Validate 

func (v RowWithIndex) Validate() error

Validate returns error if not valid

type RowWithIndexAndNewValues 

type RowWithIndexAndNewValues struct {
	RowWithIndex
	NewValues map[string]interface{} `json:"newValues"`
}

RowWithIndexAndNewValues points to specific row with expected values and provides new set of named values

type ScannedCatalog added in v0.54.2 

type ScannedCatalog struct {
	Driver      string // the driver of the scan, such as "sqlite3"
	Environment string // the environment the scan was run for
	ID          string // the catalog (database) id, as given to --db

	// Server is the server the scan read, as ScannedServer says it. It tells which
	// server holds the catalog when the project has more than one of the driver, such as
	// two SQL Server hosts that each have a database of this id; it is not needed
	// otherwise.
	Server datatug.ServerRef
}

ScannedCatalog names a catalog a scan has just read, which is where its files go in the project.

type SelectRequest 

type SelectRequest struct {
	Project     string
	Environment string
	Database    string
	From        string
	SQL         string
	Where       string
	Limit       int
	Columns     []string
}

SelectRequest holds request data for GET /datatug/exec/select.

func (SelectRequest) Validate 

func (v SelectRequest) Validate() error

Validate returns error if not valid

type SnapshotProjectPolicy added in v0.27.0 

type SnapshotProjectPolicy struct {
	Sources map[string]SnapshotSourcePolicy `json:"sources" yaml:"sources"`
}

SnapshotProjectPolicy is trusted server configuration for one project. A source absent from Sources is denied snapshot retention by default.

type SnapshotSourcePolicy added in v0.27.0 

type SnapshotSourcePolicy struct {
	Allow         bool     `json:"allow" yaml:"allow"`
	MaskedColumns []string `json:"maskedColumns,omitempty" yaml:"maskedColumns,omitempty"`
}

SnapshotSourcePolicy explicitly allows snapshot retention for one source and names columns that must be removed before bytes reach private storage.

func SnapshotPolicy added in v0.27.0 

func SnapshotPolicy(projectID, sourceID string) (SnapshotSourcePolicy, bool)

SnapshotPolicy returns an isolated copy of the configured source policy. Missing project/source entries and allow:false all fail closed.

type SourceKind added in v0.20.0 

type SourceKind string

SourceKind names which pkg/dbcopy-openable adapter a ResolvedSource opens through. 

const (
	SourceKindSQL     SourceKind = "sql"
	SourceKindInGitDB SourceKind = "ingitdb"
	SourceKindHTTP    SourceKind = "http"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.