api

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Index

Constants

View Source 
const DataTugAgentVersion = "0.0.1"

DataTugAgentVersion specifies agent version

Variables

This section is empty.

Functions

func AddDbServer 

func AddDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

AddDbServer adds db server to project

func AddRowsToRecordset 

func AddRowsToRecordset(params RecordsetDataRequestParams, _ []RowValues) (numberOfRecords int, err error)

AddRowsToRecordset adds rows to a recordset

func AuthTokenFromHTTPRequest added in v0.19.0 

func AuthTokenFromHTTPRequest(_ *http.Request, _ bool) (*sneatauth.Token, error)

AuthTokenFromHTTPRequest implements sneat-go-core/apicore's GetAuthTokenFromHttpRequest hook for `datatug serve` (REQ:principal-selection). Chosen over routing local-agent endpoints around apicore.Execute (option (b) in the brief) because the fix is a two-line wire-up at serve startup and every endpoint keeps using the same apicore.Execute/VerifyRequest path the rest of the server already relies on for its 400/403/500 conventions (util_error_handling.go).

`datatug serve` has no per-request bearer token: the whole process authenticates once, at startup, into a fixed secureread.Session via --as/--role/--group (see resolveServeSession in apps/datatugapp/commands/cmd_serve.go and api.ConfigureSecureSession). So this hook does not look at r at all — it reports the session's already-resolved principal for every request alike, exactly as if that principal had presented a bearer token on each one.

A nil token — which apicore.VerifyRequest already turns into facade.ErrUnauthorized (401) whenever the endpoint is AuthRequired, without this hook needing to duplicate that check — is returned only when the session both names no principal AND is not Unrestricted: a policy set was loaded (Session.Policies is non-empty, or would be were policies findable) but nobody was identified for it to run as. That is exactly the "anonymous request when a policy set exists" case REQ:server-acl-all-reads requires refused, never treated as an implicit admin.

A Session with Principal == nil AND Unrestricted == true (no policies were found at all, and the caller ran `serve` with none of --as/--role/--group) still yields a token: Unrestricted already means every read this process serves is unenforced regardless of who asks, so there is no principal left to gate on — refusing here would only turn "no policies configured" into a different, spurious 401 with no security benefit. resolveServeSession's own production path never actually produces this combination (it defaults --as to "admin" whenever no principal was named and no policies exist), but secureread.NewSession(SessionOptions{NoPolicies: true}) can build one directly (as test helpers across this package already do), so this hook handles it explicitly rather than assuming that default always ran.

func ConfigureSecureSession added in v0.16.0 

func ConfigureSecureSession(session secureread.Session, pathsByID map[string]string)

ConfigureSecureSession wires the fixed secureread.Session `datatug serve` builds once for its whole process lifetime (REQ:principal-selection) into every request this process handles, and records the project-id -> filesystem-directory map serve already resolved (pathsByID) so a saved query's SQL/DTQL sidecar file can be located — pkg/datatug-core's LoadQuery does not hydrate QueryDef.Text back from that file yet (that lands with the datatug-core sidecar-read story; see loadQueryDocument).

Every request-handling call site MUST go through SecureExecutor rather than open a source or run a query directly, so every read really does pass through the one policy-enforced path (REQ:server-acl-all-reads).

func CreateBoard 

func CreateBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

CreateBoard creates board

func CreateFolder 

func CreateFolder(ctx context.Context, request dto.CreateFolder) (folder *datatug.Folder, err error)

CreateFolder creates a new folder for queries

func CreateProject 

func CreateProject(ctx context.Context, request dto.CreateProjectRequest) (*datatug.ProjectSummary, error)

CreateProject create a new DataTug project using requested store

func CreateQuery 

func CreateQuery(ctx context.Context, request dto.CreateQuery) (*datatug.QueryDefWithFolderPath, error)

CreateQuery creates a new query

func DeleteBoard 

func DeleteBoard(ctx context.Context, ref dto.ProjectItemRef) error

DeleteBoard deletes board

func DeleteDbServer 

func DeleteDbServer(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (err error)

DeleteDbServer adds db server to project

func DeleteEntity 

func DeleteEntity(ctx context.Context, ref dto.ProjectItemRef) error

DeleteEntity deletes board

func DeleteFolder 

func DeleteFolder(ctx context.Context, ref dto.ProjectItemRef) error

DeleteFolder deletes queries folder

func DeleteQuery 

func DeleteQuery(ctx context.Context, ref dto.ProjectItemRef) error

DeleteQuery deletes query

func GetAllEntities 

func GetAllEntities(ctx context.Context, ref dto.ProjectRef) (entity datatug.Entities, err error)

GetAllEntities returns all entities

func GetBoard 

func GetBoard(ctx context.Context, ref dto.ProjectItemRef) (*datatug.Board, error)

GetBoard returns board by ID

func GetDatasetDefinition 

func GetDatasetDefinition(ctx context.Context, ref dto.ProjectItemRef) (dataset *datatug.RecordsetDefinition, err error)

GetDatasetDefinition returns definition of a dataset by ID

func GetDbServerSummary 

func GetDbServerSummary(ctx context.Context, ref dto.ProjectRef, dbServer datatug.ServerRef) (*datatug.ProjDbServer, error)

GetDbServerSummary returns summary on DB server

func GetEntity 

func GetEntity(ctx context.Context, ref dto.ProjectItemRef) (entity *datatug.Entity, err error)

GetEntity returns board by ID

func GetEnvironmentSummary 

func GetEnvironmentSummary(ctx context.Context, ref dto.ProjectItemRef) (*datatug.EnvironmentSummary, error)

GetEnvironmentSummary returns environment summary

func GetProjectFull 

func GetProjectFull(ctx context.Context, ref dto.ProjectRef) (*datatug.Project, error)

GetProjectFull returns full project metadata

func GetProjectSummary 

func GetProjectSummary(ctx context.Context, ref dto.ProjectRef) (projSummary *datatug.ProjectSummary, err error)

GetProjectSummary returns project summary

func GetProjects 

func GetProjects(ctx context.Context, storeID string) ([]datatug.ProjectBrief, error)

GetProjects return all projects

func GetQuery 

func GetQuery(ctx context.Context, ref dto.ProjectItemRef) (query *datatug.QueryDefWithFolderPath, err error)

GetQuery returns query definition

func GetRecordset 

func GetRecordset(_ context.Context, _ dto.ProjectItemRef) (recordset *datatug.Recordset, err error)

GetRecordset saves board

func GetRecordsetsSummary 

func GetRecordsetsSummary(ctx context.Context, ref dto.ProjectRef) (*dto.ProjRecordsetSummary, error)

GetRecordsetsSummary returns board by ID

func GetServerDatabases 

func GetServerDatabases(request dto.GetServerDatabasesRequest) (databases []*datatug.DbCatalog, err error)

GetServerDatabases returns list of databases hosted at a server

func RemoveRowsFromRecordset 

func RemoveRowsFromRecordset(params RecordsetDataRequestParams, rows []RowWithIndex) (numberOfRecordsAffected int, err error)

RemoveRowsFromRecordset removes rows from a recordset

func ResolveCatalogPath added in v0.19.0 

func ResolveCatalogPath(projectDir, catalogPath string) (string, error)

ResolveCatalogPath expands a datatug.DbCatalog.Path field into an absolute filesystem path the way this repo's real catalog data actually uses it (see datatug-demo-projects/demo-project-1's environments/*/catalogs/*/*.db.json, S45's dead-layout cleanup, e.g. "~/datatug/dbs/chinook-local.sqlite"):

  • a leading "~" or "~/..." expands to the resolved home directory (github.com/mitchellh/go-homedir)
  • a leading "$HOME" or "${HOME}" expands the same way
  • any other relative path resolves against projectDir (the project's own on-disk location), so it means the same thing regardless of the caller's working directory
  • an already-absolute path (after the above) is returned unchanged

datatug-core's DbCatalogBase.Path has no documented convention for any of these forms; this is this repo's one shared answer, used by this package's own sourceURLFromCatalog and by apps/datatugapp/commands/cmd_query_run_saved.go's equivalent for `datatug query run --project/--query` (S52 found sourceURLFromCatalog building "sqlite://" + catalog.Path with no expansion at all, unopenable against demo-project-1's real catalog data — this closes that gap at its root instead of working around it at each call site).

func SaveBoard 

func SaveBoard(ctx context.Context, ref dto.ProjectRef, board datatug.Board) (*datatug.Board, error)

SaveBoard saves board

func SaveEntity 

func SaveEntity(ctx context.Context, ref dto.ProjectRef, entity *datatug.Entity) error

SaveEntity saves board

func SecureExecutor added in v0.16.0 

func SecureExecutor() (*secureread.Executor, bool)

SecureExecutor returns the Executor ConfigureSecureSession built, and false when serve has not configured one yet (e.g. a handler under test with no ConfigureSecureSession call).

func SecurePrincipalID added in v0.16.0 

func SecurePrincipalID() string

SecurePrincipalID returns the serving principal's ID for `agent-info` to report (REQ:principal-selection), or "" when the session carries no identified principal (Unrestricted with no --as, or a role/group-only principal).

func UpdateDbSchema 

func UpdateDbSchema(ctx context.Context, projectLoader ProjectLoader, projectID, environment, driver, dbModelID string, dbConnParams dbconnection.Params) (project *datatug.Project, err error)

UpdateDbSchema updates DB schema

func UpdateDbServer 

func UpdateDbServer(ctx context.Context, ref dto.ProjectRef, projDbServer datatug.ProjDbServer) error

UpdateDbServer adds db server to project

func UpdateQuery 

func UpdateQuery(ctx context.Context, request dto.UpdateQuery) (*datatug.QueryDefWithFolderPath, error)

UpdateQuery updates existing query

func UpdateRowsInRecordset 

func UpdateRowsInRecordset(params RecordsetDataRequestParams, rows []RowWithIndexAndNewValues) (numberOfRecordsAffected int, err error)

UpdateRowsInRecordset updates rows in a recordset

Types

type AgentInfo 

type AgentInfo struct {
	Version       string  `json:"version"`
	UptimeMinutes float64 `json:"uptimeMinutes"`
	// Principal is the serving principal's ID (REQ:principal-selection); ""
	// when the session carries no identified principal (Unrestricted with no
	// --as, or a role/group-only principal). The web UI displays this and
	// MUST NOT be able to change it — no endpoint accepts a principal from
	// the request.
	Principal string `json:"principal,omitempty"`
}

AgentInfo holds agent info

func GetAgentInfo 

func GetAgentInfo() AgentInfo

GetAgentInfo returns agent info

type CommandExecutionResult added in v0.19.0 

type CommandExecutionResult struct {
	CommandID           string                `json:"commandId"`
	ElapsedMilliseconds int64                 `json:"elapsed,omitempty"`
	Items               []CommandResponseItem `json:"items"`
}

CommandExecutionResult is one ExecuteCommandsResponse.Commands entry, matching datatug-apps' ICommandResponse (commandId, elapsed?, items).

type CommandResponseItem added in v0.19.0 

type CommandResponseItem struct {
	Type  string      `json:"type"`
	Value interface{} `json:"value,omitempty"`
}

CommandResponseItem is one CommandExecutionResult.Items entry, matching datatug-apps' ICommandResponseItem (type, elapsed?, value?). Value is always a QueryResultResponse for the one command type this endpoint implements (SQL) — the same columns/rows/limitations[] shape exec/select and run_query already return (REQ:limitation-visible), so the web UI's recordset rendering has one wire shape to handle regardless of which endpoint produced it.

type ExecuteCommandRequest added in v0.19.0 

type ExecuteCommandRequest struct {
	ID   string `json:"id,omitempty"`
	Type string `json:"type"`
	Text string `json:"text"`
	Env  string `json:"env"`
	DB   string `json:"db"`
	// NamedParams is decoded so a request that sends it gets a clear
	// "not supported" error (see ExecuteCommands) rather than having the
	// parameters silently ignored — RunNativeSQL executes opaque SQL text
	// with no bind-parameter surface (REQ:opaque-sql-limitation). The web
	// client already avoids this combination itself: a single command with
	// namedParams routes through GET /exec/select instead (agent.service.ts,
	// AgentService.execute).
	NamedParams map[string]any `json:"namedParams,omitempty"`
}

ExecuteCommandRequest is one entry of ExecuteCommandsRequest.Commands, matching datatug-apps' ISqlCommandRequest wire shape exactly (id?, type, text, env, db, namedParams?). Only Type "SQL" is implemented: it is the only command type agent.service.ts's execute() ever actually constructs (IHttpCommand exists as a client-side type but nothing builds one to POST here).

func (ExecuteCommandRequest) Validate added in v0.19.0 

func (v ExecuteCommandRequest) Validate() error

Validate returns an error if the command is not well-formed.

type ExecuteCommandsRequest added in v0.19.0 

type ExecuteCommandsRequest struct {
	ID string `json:"id"`
	// Project is "omitempty": the real client never sends it in the body at
	// all (it is filled in from the `?project=` query parameter before the
	// body is decoded — see executeCommandsHandler), and omitempty keeps a
	// body that does not set it from clobbering that query-derived value
	// back to "" the way a bare `json:"project"` zero-value would on
	// decode. A body that DOES explicitly set "project" still overrides it,
	// same as before.
	Project string `json:"project,omitempty"`
	// Commands are decoded from the request body; StoreID is not part of the
	// body at all and is passed to ExecuteCommands separately (see
	// executeCommandsHandler), matching ExecuteSelect/RunQuery.
	Commands []ExecuteCommandRequest `json:"commands"`
}

ExecuteCommandsRequest is the body of POST /datatug/exec/execute_commands (datatug-apps' agent.service.ts, AgentService.execute): unlike pkg/sqlexecute.Request (a direct db-server connection carrying a raw driver/host ServerRef, still used by GetServerDatabases), every command's Env/DB here are project-relative identifiers resolved exactly the way ExecuteSelect/RunQuery resolve them (resolveSourceURL) — the web client never sends a driver/host ServerRef, and sqlexecute.RequestCommand's own Validate() (which requires one) would reject every request this endpoint actually receives. Project comes from the `?project=` query parameter, mirroring how execute_endpoints.go already reads it — the JSON body itself carries no project field (agent.service.ts's execute() deletes projectId from the body before POSTing it).

func (ExecuteCommandsRequest) Validate added in v0.19.0 

func (v ExecuteCommandsRequest) Validate() error

Validate returns an error if the request is not well-formed.

type ExecuteCommandsResponse added in v0.19.0 

type ExecuteCommandsResponse struct {
	// DurationMilliseconds is the wall-clock time every command in this
	// request took, combined.
	DurationMilliseconds int64                    `json:"duration"`
	Commands             []CommandExecutionResult `json:"commands"`
}

ExecuteCommandsResponse is POST /datatug/exec/execute_commands's response, matching datatug-apps' IExecuteResponse wire shape (duration, commands: ICommandResponse[]).

func ExecuteCommands 

func ExecuteCommands(ctx context.Context, storeID string, request ExecuteCommandsRequest) (ExecuteCommandsResponse, error)

ExecuteCommands runs every command in request through the policy-enforced secureread.Executor (REQ:server-acl-all-reads), the same path api.ExecuteSelect and api.RunQuery already use — see routes.go's executeRoutes / execute_endpoints.go. It replaces the `panic("not implemented yet")` this function used to be: the previous signature took a pkg/sqlexecute.Request, whose RequestCommand embeds a datatug.ServerRef requiring a concrete driver/host, which is not the shape datatug-apps' web client actually sends (see ExecuteCommandsRequest's doc comment) — a real implementation against that old signature would have rejected every real request anyway.

A command's own error (an unsupported source scheme, a policy refusal, a database that can't be resolved) aborts the whole request with that error — matching sqlexecute's original all-or-nothing Response.Commands contract, and (for a secureread.ErrAccessDenied) letting util_error_handling.go's existing ACCESS_DENIED/403 mapping apply exactly as it does for exec/select and run_query, with no separate handling needed here.

type LimitationDTO added in v0.16.0 

type LimitationDTO struct {
	Kind    string   `json:"kind"`
	Policy  string   `json:"policy,omitempty"`
	Note    string   `json:"note,omitempty"`
	Columns []string `json:"columns,omitempty"`
	Count   *int     `json:"count,omitempty"`
}

LimitationDTO mirrors secureread.Limitation for JSON. Kind is one of "policy", "rowsFiltered", "hiddenColumns" or "nativeSql" (secureread.LimitationKind); which of Policy/Note/Columns/Count is set depends on Kind exactly as documented on secureread.Limitation.

type ProjectLoader 

type ProjectLoader interface {
	LoadProjectFile(ctx context.Context) (projectFile datatug.ProjectFile, err error)
	LoadProject(ctx context.Context, o ...datatug.StoreOption) (project *datatug.Project, err error)
}

ProjectLoader defines an interface to load project info

type ProvenanceDTO added in v0.19.0 

type ProvenanceDTO struct {
	Source     string `json:"source"`
	Collection string `json:"collection"`
	FetchedAt  string `json:"fetchedAt"`
}

ProvenanceDTO mirrors dalgo2http.Provenance for JSON, matching apps/datatugapp/commands/query_output.go's queryProvenance field naming (source/collection/fetchedAt) so a client sees the same shape regardless of which surface (CLI --format json, or this HTTP response) it reads.

type QueryResultResponse added in v0.16.0 

type QueryResultResponse struct {
	Columns     []string         `json:"columns"`
	Rows        []map[string]any `json:"rows"`
	Limitations []LimitationDTO  `json:"limitations,omitempty"`
	// Provenance reports whether the rows came from a live HTTP-source
	// fetch or a recorded fixtures/http/ snapshot (S58 finding 2) — set
	// only when secureread.Result.Provenance was observed (today, only an
	// httpsource-backed query); absent for sqlite/ingitdb results, the same
	// nil-means-"not observed" convention as PR #204's ad-hoc `datatug
	// query run --db http://...` $provenance field.
	Provenance *ProvenanceDTO `json:"provenance,omitempty"`
}

QueryResultResponse is the JSON shape every policy-enforced read endpoint returns: the columns and rows the principal is allowed to see, plus the limitations that were applied producing them (REQ:limitation-visible — "every result MUST carry the policy limitations applied ... rather than have applied silently"). It is the direct serialization of secureread.Result; see LimitationDTO for the per-entry shape.

func ExecuteSelect 

func ExecuteSelect(ctx context.Context, storeID string, request SelectRequest) (QueryResultResponse, error)

ExecuteSelect executes a select through the policy-enforced secureread.Executor (REQ:server-acl-all-reads): a "from" selection runs through RunStructured, raw "sql" text through RunNativeSQL. Every read the web UI can trigger through `datatug serve` MUST come through this one path — see routes.go's executeRoutes / execute_endpoints.go.

type RecordsetDataRequestParams 

type RecordsetDataRequestParams struct {
	RecordsetRequestParams
	Data string `json:"data"`
}

RecordsetDataRequestParams is a set of common request parameters

func (RecordsetDataRequestParams) Validate 

func (v RecordsetDataRequestParams) Validate() error

Validate returns error if not valid

type RecordsetRequestParams 

type RecordsetRequestParams struct {
	Project   string `json:"project"`
	Recordset string `json:"recordset"`
}

RecordsetRequestParams is a set of common request parameters

func (RecordsetRequestParams) Validate 

func (v RecordsetRequestParams) Validate() error

Validate returns error if not valid

type RowValues 

type RowValues = map[string]interface{}

RowValues set of named values

type RowWithIndex 

type RowWithIndex struct {
	Index  int                    `json:"index"`
	Values map[string]interface{} `json:"values"`
}

RowWithIndex points to specific row with expected values

func (RowWithIndex) Validate 

func (v RowWithIndex) Validate() error

Validate returns error if not valid

type RowWithIndexAndNewValues 

type RowWithIndexAndNewValues struct {
	RowWithIndex
	NewValues map[string]interface{} `json:"newValues"`
}

RowWithIndexAndNewValues points to specific row with expected values and provides new set of named values

type RunQueryRequest added in v0.16.0 

type RunQueryRequest struct {
	StoreID     string         `json:"storage"`
	ProjectID   string         `json:"project"`
	Environment string         `json:"environment"`
	Database    string         `json:"database"`
	QueryID     string         `json:"queryId,omitempty"`
	DTQL        string         `json:"dtql,omitempty"`
	Parameters  map[string]any `json:"parameters,omitempty"`
}

RunQueryRequest is the body of POST /datatug/exec/run_query: run a saved query (QueryID, resolved via the project store) or an ad-hoc DTQL document (DTQL, raw DTQL-YAML text) against Environment/Database, with Parameters bound as the query's own `param` nodes (including $currentUser, bound automatically from the session's principal — see pkg/accesspolicies.Run/secureread.Executor.RunDTQL).

func (RunQueryRequest) Validate added in v0.16.0 

func (v RunQueryRequest) Validate() error

Validate returns an error if the request is not well-formed.

type RunQueryResponse added in v0.16.0 

type RunQueryResponse struct {
	QueryResultResponse
	BindingsApplied map[string]any `json:"bindingsApplied,omitempty"`
}

RunQueryResponse is POST /datatug/exec/run_query's response: the recordset and limitations every policy-enforced read carries (QueryResultResponse), plus the parameter values that were actually bound into the run.

func RunQuery added in v0.16.0 

func RunQuery(ctx context.Context, request RunQueryRequest) (RunQueryResponse, error)

RunQuery dispatches a saved or ad-hoc query through the policy-enforced secureread.Executor (REQ:server-acl-all-reads, REQ:dtql-query-type): a saved SQL query runs through RunNativeSQL, a saved or ad-hoc DTQL document through RunDTQL. It is the one implementation POST /datatug/exec/run_query and (indirectly, via the same dispatch shape) saved-query runs elsewhere must funnel through.

type SelectRequest 

type SelectRequest struct {
	Project     string
	Environment string
	Database    string
	From        string
	SQL         string
	Where       string
	Limit       int
	Columns     []string
}

SelectRequest holds request data for GET /datatug/exec/select.

func (SelectRequest) Validate 

func (v SelectRequest) Validate() error

Validate returns error if not valid

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.