Documentation
¶
Overview ¶
Package queryresults keeps the private result files that `datatug query run --result-id <id>` writes: <root>/<id>/data.json (the rows) and <root>/<id>/meta.json (everything a card needs except the rows).
The root is always ~/.datatug/query-results: there is deliberately no environment override, so the shell that ran the query, the person's terminal and the daemon all see the same files. Every directory is 0700 and every file 0600; a symlinked root or result directory, one owned by another user, and a root inside a Git repository are refused. Files are written as O_EXCL temporary files, synced and renamed into place, data.json first and meta.json last, so a result is complete exactly when meta.json exists and data.json has the byte length meta.json records.
Every error this package returns is an *Error carrying a reason code. It names no path and no id, so it is safe to show to a model.
Index ¶
- Constants
- func ConvertValue(v any) apicontract.TypedValue
- func DefaultRoot() (string, error)
- func FailureLine(reason string) []byte
- func LocateLine(locator *Locator, reason string) []byte
- func LocatorLine(data Data) []byte
- func NormalizePrompt(raw []byte) (text string, truncated bool)
- func ReasonOf(err error) string
- func RecordData(data any) (map[string]any, error)
- func ValidID(id string) bool
- type Collector
- type Column
- type ColumnFK
- type ColumnFacts
- type ColumnSource
- type DB
- type Data
- type Error
- type JoinPair
- type Loaded
- type Locator
- type Meta
- type Participant
- type ParticipantJoin
- type Prompt
- type Provenance
- type Query
- type QueryParam
- type Related
- type RelatedField
- type RelatedTarget
- type Result
- type Rows
- type Store
- func (s *Store) Annotate(id, text string, truncated bool, at time.Time) error
- func (s *Store) Check() error
- func (s *Store) Create(id string) (*Result, error)
- func (s *Store) Load(id string) (Loaded, error)
- func (s *Store) Locate(id string) (Locator, error)
- func (s *Store) Prune(all bool) (int, error)
- func (s *Store) Remove(id string) error
- func (s *Store) Root() string
- type Unavailable
Constants ¶
const ( QueryKindFrom = "from" QueryKindDTQL = "dtql" QueryKindSaved = "saved" // Languages of query.text: what the query editor reads it as. LanguageDTQL = "dtql" LanguageSQL = "sql" LanguageHTTP = "http" PolicyNone = "none" PolicyDiscovered = "discovered" PolicyDir = "dir" PolicyExplicit = "explicit" PolicyProject = "project" // MaxParticipants and MaxRelated bound those lists of meta.json. MaxParticipants = 32 MaxRelated = 200 )
Query kinds, policy modes and unavailable reasons of meta.json.
const ( Version = 1 DataFormat = "datatug.query-result.data" MetaFormat = "datatug.query-result.meta" // DefaultRows, MinRows and MaxRows bound --result-rows. DefaultRows = 2000 MinRows = 1 MaxRows = 10000 // MaxDataBytes keeps data.json under the mod SDK's 4 MiB read limit. MaxDataBytes = 4_000_000 // MaxCellBytes is where one cell is cut, on a UTF-8 boundary. MaxCellBytes = 64 << 10 // MaxMetaBytes bounds meta.json. MaxMetaBytes = 1 << 20 // MaxQueryBytes bounds query.dtql and query.parameters together. MaxQueryBytes = 256 << 10 // MaxPromptBytes bounds the prompt text; MaxPromptJSONBytes is the room // meta.json keeps free for it once escaped. MaxPromptBytes = 8 << 10 MaxPromptJSONBytes = 6*MaxPromptBytes + 256 // MaxMetaColumns is how many columns meta.json describes. MaxMetaColumns = 400 // KeyColumn is the record key column, always first. KeyColumn = "$key" EncodingBinary = "binary-placeholder" EncodingJSON = "json-text" TruncatedNone = "none" TruncatedRows = "rows" TruncatedBytes = "bytes" )
Result file formats and bounds.
const ( // DefaultDir is the root below the user's home directory. DefaultDir = ".datatug/query-results" // Retention is how long a result lives. Retention = 7 * 24 * time.Hour // MaxResults and MaxStoreBytes bound the store; the oldest results go first. MaxResults = 200 MaxStoreBytes = 512 << 20 // IncompleteGrace is how old a directory without meta.json must be before // pruning removes it (a write may still be in progress before that). IncompleteGrace = time.Hour DataFile = "data.json" MetaFile = "meta.json" )
Store bounds and lifetimes.
const ( ReasonUnwritable = "results-store-unwritable" ReasonInsideGit = "inside-git" ReasonSymlink = "symlink" ReasonOwner = "owner" ReasonIDInUse = "result-id-in-use" ReasonBadID = "bad-result-id" ReasonNotFound = "not-found" ReasonIncomplete = "incomplete" ReasonExpired = "expired" )
Reason codes. They are the whole message of an *Error.
const MaxPromptReadBytes = 1 << 20
MaxPromptReadBytes bounds how much raw input NormalizePrompt considers.
const ReasonVersion = "unsupported-version"
ReasonVersion is a result written in a format version this CLI does not know.
Variables ¶
This section is empty.
Functions ¶
func ConvertValue ¶
func ConvertValue(v any) apicontract.TypedValue
ConvertValue is v as a TypedValue, as data.json writes it.
func FailureLine ¶
FailureLine is the path-free line printed when a --result-id run fails.
func LocateLine ¶
LocateLine is the answer of `query results locate`: the paths of a complete result, or null with the reason.
func LocatorLine ¶
Locator is the path-free line `query run --result-id` prints: it holds no id, no path and no row.
func NormalizePrompt ¶ added in v0.101.0
NormalizePrompt cleans raw prompt input: invalid UTF-8 becomes U+FFFD, carriage returns, every control character but newline and tab, and the Unicode format characters (U+200B-200D, U+2060, U+FEFF, U+202A-202E, U+2066-2069 and the rest of category Cf) are dropped, surrounding white space is trimmed, and the text is cut to MaxPromptBytes on a rune boundary. truncated says anything was cut by a size bound (the read bound or MaxPromptBytes).
func RecordData ¶
RecordData is the fields of a record's data with their driver values: a map is used as it is, and any other shape is read through its JSON form with numbers kept as exact text.
Types ¶
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector keeps the first rows of a result, within the row and byte bounds.
func (*Collector) Add ¶
Add takes one record: its key as text and its fields. It answers false once no further record is wanted; the caller then stops reading, so a source is read for at most limit + 1 records.
func (*Collector) Build ¶
Build encodes data.json for id with the columns given ($key is put first). A column's type is its one observed kind; a column of several kinds is a string column with each cell written as its text, and it records the kinds in ObservedTypes. A column with a declared contract type takes it when every value casts to it. Rows are dropped from the end while the file would exceed MaxDataBytes.
func (*Collector) FieldNames ¶
FieldNames lists, per kept row, the names of its fields.
type Column ¶
type Column struct {
Name string `json:"name"`
Type string `json:"type"`
DeclaredType *string `json:"declaredType"`
Nullable *bool `json:"nullable"`
PK *int `json:"pk"`
Source *ColumnSource `json:"source"`
ForeignKeys []ColumnFK `json:"foreignKeys"`
Encoding string `json:"encoding,omitempty"`
CutCells int `json:"cutCells,omitempty"`
ObservedTypes []string `json:"observedTypes,omitempty"`
}
Column describes one column of data.json. The declared facts come from the database's schema, never from the values; they are null where unknown.
type ColumnFK ¶ added in v0.92.0
type ColumnFK struct {
RefSchema string `json:"refSchema"`
RefRelation string `json:"refRelation"`
RefColumn string `json:"refColumn"`
}
ColumnFK is a foreign key a column takes part in.
type ColumnFacts ¶ added in v0.92.0
type ColumnFacts struct {
DeclaredType string
// ContractType is the declared type in the contract's closed set, or "".
ContractType string
Nullable bool
PK int // 1-based position in the primary key, 0 when not in it
Source ColumnSource
ForeignKeys []ColumnFK
}
ColumnFacts are the schema facts of a column that is a direct reference to one source's field.
type ColumnSource ¶ added in v0.92.0
type ColumnSource struct {
Instance string `json:"instance"`
Schema string `json:"schema"`
Relation string `json:"relation"`
Column string `json:"column"`
}
ColumnSource is the source field a column reads.
type DB ¶
type DB struct {
// Ref is datatug://<project>/<db>?env=<env>, or nil when the database is
// not a source of a registered project.
Ref *string `json:"ref"`
Project *string `json:"project"`
Env *string `json:"env"`
Source *string `json:"source"`
Driver string `json:"driver"`
}
DB says which database the query ran on, by its project reference only.
type Error ¶
type Error struct{ Reason string }
Error is a store failure: a reason code, never a path or an id.
type Loaded ¶ added in v0.93.0
type Loaded struct {
Meta Meta
Recordset apicontract.Recordset
// MetaJSON and DataJSON are the files as they were read.
MetaJSON, DataJSON []byte
}
Loaded is a complete result read back from its files.
type Locator ¶
type Locator struct {
Version int `json:"version"`
ID string `json:"id"`
Root string `json:"root"`
Data string `json:"data"`
Meta string `json:"meta"`
}
Locator is where a complete result's files are.
type Meta ¶
type Meta struct {
Format string `json:"format"`
Version int `json:"version"`
ID string `json:"id"`
CreatedAt time.Time `json:"createdAt"`
ExpiresAt time.Time `json:"expiresAt"`
CLIVersion string `json:"cliVersion"`
DataBytes int64 `json:"dataBytes"`
DB DB `json:"db"`
Query Query `json:"query"`
Rows Rows `json:"rows"`
Provenance *Provenance `json:"provenance"`
Columns []Column `json:"columns"`
// Participants are the query's source occurrences and Related the tables
// that can be joined to them.
Participants []Participant `json:"participants"`
Related []Related `json:"related"`
Unavailable Unavailable `json:"unavailable"`
// Prompt is the request that produced the result, set by `query results
// annotate`; absent until then and in results written before it existed.
Prompt *Prompt `json:"prompt,omitempty"`
}
Meta is meta.json: everything needed to draw a result except its rows. It holds no row value, connection string, file path, host, user, password, environment variable or policy path.
func (*Meta) Describe ¶ added in v0.92.0
func (m *Meta) Describe(facts map[string]ColumnFacts)
Describe sets the declared facts of each column found in facts.
func (Meta) Encode ¶
Encode applies the bounds of meta.json and encodes it: query.dtql and query.parameters together within MaxQueryBytes (else both are dropped, "too-large"), query.text and query.params together within their own MaxQueryBytes (else just those two are dropped), at most MaxMetaColumns columns, and the whole within MaxMetaBytes (else the column descriptions are dropped).
type Participant ¶ added in v0.92.0
type Participant struct {
Instance string `json:"instance"`
Schema string `json:"schema"`
Relation string `json:"relation"`
Alias string `json:"alias"`
DB *string `json:"db,omitempty"`
Kind string `json:"kind"`
Role string `json:"role"`
Join *ParticipantJoin `json:"join,omitempty"`
}
Participant is one source occurrence of the query.
type ParticipantJoin ¶ added in v0.92.0
ParticipantJoin is how a participant is joined.
type Prompt ¶ added in v0.101.0
type Prompt struct {
Text string `json:"text"`
Truncated bool `json:"truncated"`
// At is when it was recorded, RFC 3339 in UTC.
At string `json:"at"`
}
Prompt is the text a person or an agent asked for the result with.
type Provenance ¶
Provenance is whether an HTTP source answered live or from a snapshot.
type Query ¶
type Query struct {
Kind string `json:"kind"`
DTQL string `json:"dtql,omitempty"`
From string `json:"from,omitempty"`
SavedQueryID string `json:"savedQueryId,omitempty"`
SavedQueryType string `json:"savedQueryType,omitempty"`
Parameters map[string]apicontract.TypedValueOrSet `json:"parameters,omitempty"`
Policy string `json:"policy"`
// Text, Language, DBRef and Params make the query openable in the query
// editor: the canonical text the editor accepts, its language ("dtql",
// "sql" or "http"), the datatug:// reference of the database (absent when
// the run was not against a project source) and the run's parameters as
// name/value pairs. They are additive: readers that predate them ignore
// them and results written before them lack them.
Text string `json:"text,omitempty"`
Language string `json:"language,omitempty"`
DBRef string `json:"dbRef,omitempty"`
Params []QueryParam `json:"params,omitempty"`
}
Query is what ran: enough to run it again.
type QueryParam ¶ added in v0.101.0
type QueryParam struct {
Name string `json:"name"`
Value apicontract.TypedValueOrSet `json:"value"`
}
QueryParam is one parameter of a run: its name and typed value (a scalar or a set), as in query.parameters.
type Related ¶ added in v0.92.0
type Related struct {
CandidateID string `json:"candidateId"`
FromInstance string `json:"fromInstance"`
Target RelatedTarget `json:"target"`
ConstraintID string `json:"constraintId"`
Direction string `json:"direction"`
Cardinality string `json:"cardinality"`
Fields []RelatedField `json:"fields"`
Evidence string `json:"evidence"`
}
Related is one table that can be joined to a participant by a foreign key.
type RelatedField ¶ added in v0.92.0
type RelatedField struct {
SourceField string `json:"sourceField"`
TargetField string `json:"targetField"`
}
RelatedField pairs a field of the participant with one of the target.
type RelatedTarget ¶ added in v0.92.0
type RelatedTarget struct {
Schema string `json:"schema"`
Relation string `json:"relation"`
Kind string `json:"kind"`
}
RelatedTarget is the table a related candidate joins.
type Result ¶
type Result struct {
// contains filtered or unexported fields
}
Result is a result directory being written.
type Rows ¶
type Rows struct {
Kept int `json:"kept"`
HasMore bool `json:"hasMore"`
TruncatedBy string `json:"truncatedBy"`
}
Rows says how many rows data.json keeps and whether more exist.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is one results root.
func Open ¶
Open checks root without creating it: it must not lie inside a Git repository, must not be a symbolic link, and, when it exists, must be a directory owned by the user.
func (*Store) Annotate ¶ added in v0.101.0
Annotate records prompt as the request that produced result id: it sets meta.json's "prompt" and leaves every other member as it was. The file is replaced atomically (private temporary file, sync, rename), so a reader sees the old or the new meta.json, never a mix. Annotating again overwrites. The result must be complete and unexpired; a failure carries the reason codes of Locate, ReasonVersion, or ReasonUnwritable. Members are rewritten in alphabetical key order with their values unchanged. Nothing locks across processes: two annotations racing leave the last one, and one racing rm or prune can fail with ReasonUnwritable.
func (*Store) Check ¶
Check proves the store can be written: it creates the root when missing, then creates and removes one private temporary file in it.
func (*Store) Create ¶
Create makes <root>/<id> with mkdir, so it fails when the id is in use: a result is never overwritten, and nothing is written through a path that already existed.
func (*Store) Load ¶ added in v0.93.0
Load reads a complete, unexpired result: it locates it with the same checks as Locate, then reads meta.json and data.json, refusing a data.json whose size is not the one meta.json records and a format version it does not know.
func (*Store) Prune ¶
Prune removes expired results, directories without meta.json older than IncompleteGrace, and then the oldest results while more than MaxResults or MaxStoreBytes remain. With all it removes every result. Only directories whose name follows the id grammar are touched, after the symlink and owner checks; a link is never followed. It answers how many were removed.
type Unavailable ¶
type Unavailable struct {
// Editor says query.text and query.params were dropped for size, so a
// consumer can tell dropped from absent; the legacy dtql and parameters
// are untouched by it.
}
Unavailable holds a reason code per part of the metadata that is missing.