queryresults

package
v0.134.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package queryresults keeps the private result files that `datatug query run --result-id <id>` writes: <root>/<id>/data.json (the rows) and <root>/<id>/meta.json (everything a card needs except the rows).

The root is always ~/.datatug/query-results: there is deliberately no environment override, so the shell that ran the query, the person's terminal and the daemon all see the same files. Every directory is 0700 and every file 0600; a symlinked root or result directory, one owned by another user, and a root inside a Git repository are refused. Files are written as O_EXCL temporary files, synced and renamed into place, data.json first and meta.json last, so a result is complete exactly when meta.json exists and data.json has the byte length meta.json records.

Every error this package returns is an *Error carrying a reason code. It names no path and no id, so it is safe to show to a model.

Index

Constants

View Source
const (
	QueryKindFrom  = "from"
	QueryKindTugQL = "tugql"
	QueryKindSaved = "saved"

	// Languages of query.text: what the query editor reads it as.
	LanguageTugQL = "tugql"
	LanguageSQL   = "sql"
	LanguageHTTP  = "http"

	PolicyNone       = "none"
	PolicyDiscovered = "discovered"
	PolicyDir        = "dir"
	PolicyExplicit   = "explicit"
	PolicyProject    = "project"

	UnavailableTooLarge      = "too-large"
	UnavailableTooMany       = "too-many"
	UnavailableNativeSQL     = "native-sql"
	UnavailableNoRelMetadata = "no-relationship-metadata"
	UnavailableUnreadable    = "schema-unreadable"

	// MaxParticipants and MaxRelated bound those lists of meta.json.
	MaxParticipants = 32
	MaxRelated      = 200
)

Query kinds, policy modes and unavailable reasons of meta.json.

View Source
const (
	Version    = 1
	DataFormat = "datatug.query-result.data"
	MetaFormat = "datatug.query-result.meta"

	// DefaultRows, MinRows and MaxRows bound --result-rows.
	DefaultRows = 2000
	MinRows     = 1
	MaxRows     = 10000

	// MaxDataBytes keeps data.json under the mod SDK's 4 MiB read limit.
	MaxDataBytes = 4_000_000
	// MaxCellBytes is where one cell is cut, on a UTF-8 boundary.
	MaxCellBytes = 64 << 10
	// MaxMetaBytes bounds meta.json.
	MaxMetaBytes = 1 << 20
	// MaxQueryBytes bounds query.tugql and query.parameters together.
	MaxQueryBytes = 256 << 10
	// MaxPromptBytes bounds the prompt text; MaxPromptJSONBytes is the room
	// meta.json keeps free for it once escaped.
	MaxPromptBytes     = 8 << 10
	MaxPromptJSONBytes = 6*MaxPromptBytes + 256
	// MaxMetaColumns is how many columns meta.json describes.
	MaxMetaColumns = 400

	// KeyColumn is the record key column, always first.
	KeyColumn = "$key"

	EncodingBinary = "binary-placeholder"
	EncodingJSON   = "json-text"

	TruncatedNone  = "none"
	TruncatedRows  = "rows"
	TruncatedBytes = "bytes"
)

Result file formats and bounds.

View Source
const (
	// DefaultDir is the root below the user's home directory.
	DefaultDir = ".datatug/query-results"
	// Retention is how long a result lives.
	Retention = 7 * 24 * time.Hour
	// MaxResults and MaxStoreBytes bound the store; the oldest results go first.
	MaxResults    = 200
	MaxStoreBytes = 512 << 20
	// IncompleteGrace is how old a directory without meta.json must be before
	// pruning removes it (a write may still be in progress before that).
	IncompleteGrace = time.Hour

	DataFile = "data.json"
	MetaFile = "meta.json"
)

Store bounds and lifetimes.

View Source
const (
	ReasonUnwritable = "results-store-unwritable"
	ReasonInsideGit  = "inside-git"
	ReasonSymlink    = "symlink"
	ReasonOwner      = "owner"
	ReasonIDInUse    = "result-id-in-use"
	ReasonBadID      = "bad-result-id"
	ReasonNotFound   = "not-found"
	ReasonIncomplete = "incomplete"
	ReasonExpired    = "expired"
)

Reason codes. They are the whole message of an *Error.

View Source
const MaxPromptReadBytes = 1 << 20

MaxPromptReadBytes bounds how much raw input NormalizePrompt considers.

View Source
const ReasonVersion = "unsupported-version"

ReasonVersion is a result written in a format version this CLI does not know.

Variables

This section is empty.

Functions

func ConvertValue

func ConvertValue(v any) apicontract.TypedValue

ConvertValue is v as a TypedValue, as data.json writes it.

func DefaultRoot

func DefaultRoot() (string, error)

DefaultRoot is ~/.datatug/query-results.

func FailureLine

func FailureLine(reason string) []byte

FailureLine is the path-free line printed when a --result-id run fails.

func LocateLine

func LocateLine(locator *Locator, reason string) []byte

LocateLine is the answer of `query results locate`: the paths of a complete result, or null with the reason.

func LocatorLine

func LocatorLine(data Data) []byte

Locator is the path-free line `query run --result-id` prints: it holds no id, no path and no row.

func NormalizePrompt added in v0.101.0

func NormalizePrompt(raw []byte) (text string, truncated bool)

NormalizePrompt cleans raw prompt input: invalid UTF-8 becomes U+FFFD, carriage returns, every control character but newline and tab, and the Unicode format characters (U+200B-200D, U+2060, U+FEFF, U+202A-202E, U+2066-2069 and the rest of category Cf) are dropped, surrounding white space is trimmed, and the text is cut to MaxPromptBytes on a rune boundary. truncated says anything was cut by a size bound (the read bound or MaxPromptBytes).

func OrderColumns added in v0.102.1

func OrderColumns(columns, source []string) []string

OrderColumns puts columns in the order of source, a table's or SELECT list's columns: those named in source come first in its order, the others follow in the order given. A name matches case-insensitively; a column named twice in source takes its first place.

func ReasonOf

func ReasonOf(err error) string

ReasonOf is the reason code of err when it is an *Error, else "".

func RecordData

func RecordData(data any) (map[string]any, error)

RecordData is the fields of a record's data with their driver values: a map is used as it is, and any other shape is read through its JSON form with numbers kept as exact text.

func ValidID

func ValidID(id string) bool

ValidID reports whether id follows the result id grammar.

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector keeps the first rows of a result, within the row and byte bounds.

func NewCollector

func NewCollector(limit int) *Collector

NewCollector keeps at most limit rows.

func (*Collector) Add

func (c *Collector) Add(key string, data map[string]any) bool

Add takes one record: its key as text and its fields. It answers false once no further record is wanted; the caller then stops reading, so a source is read for at most limit + 1 records.

func (*Collector) Build

func (c *Collector) Build(id string, columns []string, declared map[string]string) Data

Build encodes data.json for id with the columns given, in that order ($key is put first and marked internal in meta.json). A column's type is its one observed kind; a column of several kinds is a string column with each cell written as its text, and it records the kinds in ObservedTypes. A column with a declared contract type takes it when every value casts to it. Rows are dropped from the end while the file would exceed MaxDataBytes.

func (*Collector) FieldNames

func (c *Collector) FieldNames() [][]string

FieldNames lists, per kept row, the names of its fields.

type Column

type Column struct {
	Name          string        `json:"name"`
	Type          string        `json:"type"`
	DeclaredType  *string       `json:"declaredType"`
	Nullable      *bool         `json:"nullable"`
	PK            *int          `json:"pk"`
	Source        *ColumnSource `json:"source"`
	ForeignKeys   []ColumnFK    `json:"foreignKeys"`
	Encoding      string        `json:"encoding,omitempty"`
	CutCells      int           `json:"cutCells,omitempty"`
	ObservedTypes []string      `json:"observedTypes,omitempty"`
	// Internal marks a column that is bookkeeping, not a field of the result
	// (today only the record key "$key"): a consumer that shows fields hides
	// it. It is additive: absent (false) in results written before it
	// existed, which is why IsInternal also knows the key column by name.
	Internal bool `json:"internal,omitempty"`
}

Column describes one column of data.json. The declared facts come from the database's schema, never from the values; they are null where unknown.

func (Column) IsInternal added in v0.102.1

func (c Column) IsInternal() bool

IsInternal is whether a consumer that shows fields should hide the column: it is marked internal, or it is the record key of a result written before the marker existed.

type ColumnFK added in v0.92.0

type ColumnFK struct {
	RefSchema   string `json:"refSchema"`
	RefRelation string `json:"refRelation"`
	RefColumn   string `json:"refColumn"`
}

ColumnFK is a foreign key a column takes part in.

type ColumnFacts added in v0.92.0

type ColumnFacts struct {
	DeclaredType string
	// ContractType is the declared type in the contract's closed set, or "".
	ContractType string
	Nullable     bool
	PK           int // 1-based position in the primary key, 0 when not in it
	Source       ColumnSource
	ForeignKeys  []ColumnFK
}

ColumnFacts are the schema facts of a column that is a direct reference to one source's field.

type ColumnSource added in v0.92.0

type ColumnSource struct {
	Instance string `json:"instance"`
	Schema   string `json:"schema"`
	Relation string `json:"relation"`
	Column   string `json:"column"`
}

ColumnSource is the source field a column reads.

type DB

type DB struct {
	// Ref is datatug://<project>/<db>?env=<env>, or nil when the database is
	// not a source of a registered project.
	Ref     *string `json:"ref"`
	Project *string `json:"project"`
	Env     *string `json:"env"`
	Source  *string `json:"source"`
	Driver  string  `json:"driver"`
}

DB says which database the query ran on, by its project reference only.

type Data

type Data struct {
	Anonymous bool
	Bytes     []byte
	Columns   []Column
	Rows      Rows
}

Data is a built data.json and what meta.json says about it.

func BuildAnonymous added in v0.132.0

func BuildAnonymous(id string, rs apicontract.Recordset, native, encodings []string, limit int) (Data, error)

BuildAnonymous retains ordered typed cells without introducing a key or using label-based lookups. Size overflow is an error, never cell truncation.

type Error

type Error struct{ Reason string }

Error is a store failure: a reason code, never a path or an id.

func (*Error) Error

func (e *Error) Error() string

type JoinPair added in v0.92.0

type JoinPair struct {
	Left  string `json:"left"`
	Right string `json:"right"`
}

JoinPair is one equality of a join: alias.field on each side.

type Loaded added in v0.93.0

type Loaded struct {
	Anonymous bool
	Meta      Meta
	Recordset apicontract.Recordset
	// MetaJSON and DataJSON are the files as they were read.
	MetaJSON, DataJSON []byte
}

Loaded is a complete result read back from its files.

type Locator

type Locator struct {
	Version int    `json:"version"`
	ID      string `json:"id"`
	Root    string `json:"root"`
	Data    string `json:"data"`
	Meta    string `json:"meta"`
}

Locator is where a complete result's files are.

type Meta

type Meta struct {
	Anonymous  bool        `json:"anonymous,omitempty"`
	Format     string      `json:"format"`
	Version    int         `json:"version"`
	ID         string      `json:"id"`
	CreatedAt  time.Time   `json:"createdAt"`
	ExpiresAt  time.Time   `json:"expiresAt"`
	CLIVersion string      `json:"cliVersion"`
	DataBytes  int64       `json:"dataBytes"`
	DB         DB          `json:"db"`
	Query      Query       `json:"query"`
	Rows       Rows        `json:"rows"`
	Provenance *Provenance `json:"provenance"`
	Columns    []Column    `json:"columns"`
	// Participants are the query's source occurrences and Related the tables
	// that can be joined to them.
	Participants []Participant `json:"participants"`
	Related      []Related     `json:"related"`
	// Unavailable names, per part, why it is missing or cut short.
	Unavailable Unavailable `json:"unavailable"`
	// Prompt is the request that produced the result, set by `query results
	// annotate`; absent until then and in results written before it existed.
	Prompt *Prompt `json:"prompt,omitempty"`
}

Meta is meta.json: everything needed to draw a result except its rows. It holds no row value, connection string, file path, host, user, password, environment variable or policy path.

func NewMeta

func NewMeta(id, cliVersion string, data Data) Meta

NewMeta starts the meta.json of id written now.

func (*Meta) Describe added in v0.92.0

func (m *Meta) Describe(facts map[string]ColumnFacts)

Describe sets the declared facts of each column found in facts.

func (Meta) Encode

func (m Meta) Encode() []byte

Encode applies the bounds of meta.json and encodes it: query.tugql and query.parameters together within MaxQueryBytes (else both are dropped, "too-large"), query.text and query.params together within their own MaxQueryBytes (else just those two are dropped), at most MaxMetaColumns columns, and the whole within MaxMetaBytes (else the column descriptions are dropped).

func (*Meta) SetEditor added in v0.101.0

func (m *Meta) SetEditor(text, language string)

SetEditor fills the editor-ready fields of the query from what the run already holds: db.ref and query.parameters. text is "" when the language has no editable text.

type Participant added in v0.92.0

type Participant struct {
	Instance string           `json:"instance"`
	Schema   string           `json:"schema"`
	Relation string           `json:"relation"`
	Alias    string           `json:"alias"`
	DB       *string          `json:"db,omitempty"`
	Kind     string           `json:"kind"`
	Role     string           `json:"role"`
	Join     *ParticipantJoin `json:"join,omitempty"`
}

Participant is one source occurrence of the query.

type ParticipantJoin added in v0.92.0

type ParticipantJoin struct {
	Type string     `json:"type"`
	On   []JoinPair `json:"on"`
}

ParticipantJoin is how a participant is joined.

type Prompt added in v0.101.0

type Prompt struct {
	Text      string `json:"text"`
	Truncated bool   `json:"truncated"`
	// At is when it was recorded, RFC 3339 in UTC.
	At string `json:"at"`
}

Prompt is the text a person or an agent asked for the result with.

type Provenance

type Provenance struct {
	Mode       string `json:"mode"`
	ObservedAt string `json:"observedAt"`
}

Provenance is whether an HTTP source answered live or from a snapshot.

type Query

type Query struct {
	Kind           string                                 `json:"kind"`
	TugQL          string                                 `json:"tugql,omitempty"`
	From           string                                 `json:"from,omitempty"`
	SavedQueryID   string                                 `json:"savedQueryId,omitempty"`
	SavedQueryType string                                 `json:"savedQueryType,omitempty"`
	Parameters     map[string]apicontract.TypedValueOrSet `json:"parameters,omitempty"`
	Policy         string                                 `json:"policy"`
	// Text, Language, DBRef and Params make the query openable in the query
	// editor: the canonical text the editor accepts, its language ("tugql",
	// "sql" or "http"), the datatug:// reference of the database (absent when
	// the run was not against a project source) and the run's parameters as
	// name/value pairs. They are additive: readers that predate them ignore
	// them and results written before them lack them.
	Text     string       `json:"text,omitempty"`
	Language string       `json:"language,omitempty"`
	DBRef    string       `json:"dbRef,omitempty"`
	Params   []QueryParam `json:"params,omitempty"`
}

Query is what ran: enough to run it again.

type QueryParam added in v0.101.0

type QueryParam struct {
	Name  string                      `json:"name"`
	Value apicontract.TypedValueOrSet `json:"value"`
}

QueryParam is one parameter of a run: its name and typed value (a scalar or a set), as in query.parameters.

type Related struct {
	CandidateID  string         `json:"candidateId"`
	FromInstance string         `json:"fromInstance"`
	Target       RelatedTarget  `json:"target"`
	ConstraintID string         `json:"constraintId"`
	Direction    string         `json:"direction"`
	Cardinality  string         `json:"cardinality"`
	Fields       []RelatedField `json:"fields"`
	Evidence     string         `json:"evidence"`
}

Related is one table that can be joined to a participant by a foreign key.

type RelatedField added in v0.92.0

type RelatedField struct {
	SourceField string `json:"sourceField"`
	TargetField string `json:"targetField"`
}

RelatedField pairs a field of the participant with one of the target.

type RelatedTarget added in v0.92.0

type RelatedTarget struct {
	Schema   string `json:"schema"`
	Relation string `json:"relation"`
	Kind     string `json:"kind"`
}

RelatedTarget is the table a related candidate joins.

type Result

type Result struct {
	// contains filtered or unexported fields
}

Result is a result directory being written.

func (*Result) Discard

func (r *Result) Discard()

Discard removes a result that will not be completed.

func (*Result) Write

func (r *Result) Write(data, meta []byte) error

Write writes data.json, then meta.json; each is an O_EXCL 0600 temporary file, synced and renamed into place.

type Rows

type Rows struct {
	Kept        int    `json:"kept"`
	HasMore     bool   `json:"hasMore"`
	TruncatedBy string `json:"truncatedBy"`
}

Rows says how many rows data.json keeps and whether more exist.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is one results root.

func Open

func Open(root string) (*Store, error)

Open checks root without creating it: it must not lie inside a Git repository, must not be a symbolic link, and, when it exists, must be a directory owned by the user.

func OpenDefault

func OpenDefault() (*Store, error)

OpenDefault opens the store at DefaultRoot.

func (*Store) Annotate added in v0.101.0

func (s *Store) Annotate(id, text string, truncated bool, at time.Time) error

Annotate records prompt as the request that produced result id: it sets meta.json's "prompt" and leaves every other member as it was. The file is replaced atomically (private temporary file, sync, rename), so a reader sees the old or the new meta.json, never a mix. Annotating again overwrites. The result must be complete and unexpired; a failure carries the reason codes of Locate, ReasonVersion, or ReasonUnwritable. Members are rewritten in alphabetical key order with their values unchanged. Nothing locks across processes: two annotations racing leave the last one, and one racing rm or prune can fail with ReasonUnwritable.

func (*Store) Check

func (s *Store) Check() error

Check proves the store can be written: it creates the root when missing, then creates and removes one private temporary file in it.

func (*Store) Create

func (s *Store) Create(id string) (*Result, error)

Create makes <root>/<id> with mkdir, so it fails when the id is in use: a result is never overwritten, and nothing is written through a path that already existed.

func (*Store) Load added in v0.93.0

func (s *Store) Load(id string) (Loaded, error)

Load reads a complete, unexpired result: it locates it with the same checks as Locate, then reads meta.json and data.json, refusing a data.json whose size is not the one meta.json records and a format version it does not know.

func (*Store) Locate

func (s *Store) Locate(id string) (Locator, error)

Locate answers where a complete, unexpired result's files are.

func (*Store) Prune

func (s *Store) Prune(all bool) (int, error)

Prune removes expired results, directories without meta.json older than IncompleteGrace, and then the oldest results while more than MaxResults or MaxStoreBytes remain. With all it removes every result. Only directories whose name follows the id grammar are touched, after the symlink and owner checks; a link is never followed. It answers how many were removed.

func (*Store) Remove

func (s *Store) Remove(id string) error

Remove deletes one result directory; a link is never followed.

func (*Store) Root

func (s *Store) Root() string

Root is the absolute root of the store.

type Unavailable

type Unavailable struct {
	Query string `json:"query,omitempty"`
	// Editor says query.text and query.params were dropped for size, so a
	// consumer can tell dropped from absent; the legacy tugql and parameters
	// are untouched by it.
	Editor        string `json:"editor,omitempty"`
	Columns       string `json:"columns,omitempty"`
	Participants  string `json:"participants,omitempty"`
	Related       string `json:"related,omitempty"`
	DeclaredTypes string `json:"declaredTypes,omitempty"`
}

Unavailable holds a reason code per part of the metadata that is missing.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL