queryresults

package
v0.94.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 16, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package queryresults keeps the private result files that `datatug query run --result-id <id>` writes: <root>/<id>/data.json (the rows) and <root>/<id>/meta.json (everything a card needs except the rows).

The root is always ~/.datatug/query-results: there is deliberately no environment override, so the shell that ran the query, the person's terminal and the daemon all see the same files. Every directory is 0700 and every file 0600; a symlinked root or result directory, one owned by another user, and a root inside a Git repository are refused. Files are written as O_EXCL temporary files, synced and renamed into place, data.json first and meta.json last, so a result is complete exactly when meta.json exists and data.json has the byte length meta.json records.

Every error this package returns is an *Error carrying a reason code. It names no path and no id, so it is safe to show to a model.

Index

Constants

View Source 
const (
	QueryKindFrom  = "from"
	QueryKindDTQL  = "dtql"
	QueryKindSaved = "saved"

	PolicyNone       = "none"
	PolicyDiscovered = "discovered"
	PolicyDir        = "dir"
	PolicyExplicit   = "explicit"
	PolicyProject    = "project"

	UnavailableTooLarge      = "too-large"
	UnavailableTooMany       = "too-many"
	UnavailableNativeSQL     = "native-sql"
	UnavailableNoRelMetadata = "no-relationship-metadata"
	UnavailableUnreadable    = "schema-unreadable"

	// MaxParticipants and MaxRelated bound those lists of meta.json.
	MaxParticipants = 32
	MaxRelated      = 200
)

Query kinds, policy modes and unavailable reasons of meta.json.

View Source 
const (
	Version    = 1
	DataFormat = "datatug.query-result.data"
	MetaFormat = "datatug.query-result.meta"

	// DefaultRows, MinRows and MaxRows bound --result-rows.
	DefaultRows = 2000
	MinRows     = 1
	MaxRows     = 10000

	// MaxDataBytes keeps data.json under the mod SDK's 4 MiB read limit.
	MaxDataBytes = 4_000_000
	// MaxCellBytes is where one cell is cut, on a UTF-8 boundary.
	MaxCellBytes = 64 << 10
	// MaxMetaBytes bounds meta.json.
	MaxMetaBytes = 1 << 20
	// MaxQueryBytes bounds query.dtql and query.parameters together.
	MaxQueryBytes = 256 << 10
	// MaxMetaColumns is how many columns meta.json describes.
	MaxMetaColumns = 400

	// KeyColumn is the record key column, always first.
	KeyColumn = "$key"

	EncodingBinary = "binary-placeholder"
	EncodingJSON   = "json-text"

	TruncatedNone  = "none"
	TruncatedRows  = "rows"
	TruncatedBytes = "bytes"
)

Result file formats and bounds.

View Source 
const (
	// DefaultDir is the root below the user's home directory.
	DefaultDir = ".datatug/query-results"
	// Retention is how long a result lives.
	Retention = 7 * 24 * time.Hour
	// MaxResults and MaxStoreBytes bound the store; the oldest results go first.
	MaxResults    = 200
	MaxStoreBytes = 512 << 20
	// IncompleteGrace is how old a directory without meta.json must be before
	// pruning removes it (a write may still be in progress before that).
	IncompleteGrace = time.Hour

	DataFile = "data.json"
	MetaFile = "meta.json"
)

Store bounds and lifetimes.

View Source 
const (
	ReasonUnwritable = "results-store-unwritable"
	ReasonInsideGit  = "inside-git"
	ReasonSymlink    = "symlink"
	ReasonOwner      = "owner"
	ReasonIDInUse    = "result-id-in-use"
	ReasonBadID      = "bad-result-id"
	ReasonNotFound   = "not-found"
	ReasonIncomplete = "incomplete"
	ReasonExpired    = "expired"
)

Reason codes. They are the whole message of an *Error.

View Source 
const ReasonVersion = "unsupported-version"

ReasonVersion is a result written in a format version this CLI does not know.

Variables

This section is empty.

Functions

func ConvertValue 

func ConvertValue(v any) apicontract.TypedValue

ConvertValue is v as a TypedValue, as data.json writes it.

func DefaultRoot 

func DefaultRoot() (string, error)

DefaultRoot is ~/.datatug/query-results.

func FailureLine 

func FailureLine(reason string) []byte

FailureLine is the path-free line printed when a --result-id run fails.

func LocateLine 

func LocateLine(locator *Locator, reason string) []byte

LocateLine is the answer of `query results locate`: the paths of a complete result, or null with the reason.

func LocatorLine 

func LocatorLine(data Data) []byte

Locator is the path-free line `query run --result-id` prints: it holds no id, no path and no row.

func ReasonOf 

func ReasonOf(err error) string

ReasonOf is the reason code of err when it is an *Error, else "".

func RecordData 

func RecordData(data any) (map[string]any, error)

RecordData is the fields of a record's data with their driver values: a map is used as it is, and any other shape is read through its JSON form with numbers kept as exact text.

func ValidID 

func ValidID(id string) bool

ValidID reports whether id follows the result id grammar.

Types

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector keeps the first rows of a result, within the row and byte bounds.

func NewCollector 

func NewCollector(limit int) *Collector

NewCollector keeps at most limit rows.

func (*Collector) Add 

func (c *Collector) Add(key string, data map[string]any) bool

Add takes one record: its key as text and its fields. It answers false once no further record is wanted; the caller then stops reading, so a source is read for at most limit + 1 records.

func (*Collector) Build 

func (c *Collector) Build(id string, columns []string, declared map[string]string) Data

Build encodes data.json for id with the columns given ($key is put first). A column's type is its one observed kind; a column of several kinds is a string column with each cell written as its text, and it records the kinds in ObservedTypes. A column with a declared contract type takes it when every value casts to it. Rows are dropped from the end while the file would exceed MaxDataBytes.

func (*Collector) FieldNames 

func (c *Collector) FieldNames() [][]string

FieldNames lists, per kept row, the names of its fields.

type Column 

type Column struct {
	Name          string        `json:"name"`
	Type          string        `json:"type"`
	DeclaredType  *string       `json:"declaredType"`
	Nullable      *bool         `json:"nullable"`
	PK            *int          `json:"pk"`
	Source        *ColumnSource `json:"source"`
	ForeignKeys   []ColumnFK    `json:"foreignKeys"`
	Encoding      string        `json:"encoding,omitempty"`
	CutCells      int           `json:"cutCells,omitempty"`
	ObservedTypes []string      `json:"observedTypes,omitempty"`
}

Column describes one column of data.json. The declared facts come from the database's schema, never from the values; they are null where unknown.

type ColumnFK added in v0.92.0 

type ColumnFK struct {
	RefSchema   string `json:"refSchema"`
	RefRelation string `json:"refRelation"`
	RefColumn   string `json:"refColumn"`
}

ColumnFK is a foreign key a column takes part in.

type ColumnFacts added in v0.92.0 

type ColumnFacts struct {
	DeclaredType string
	// ContractType is the declared type in the contract's closed set, or "".
	ContractType string
	Nullable     bool
	PK           int // 1-based position in the primary key, 0 when not in it
	Source       ColumnSource
	ForeignKeys  []ColumnFK
}

ColumnFacts are the schema facts of a column that is a direct reference to one source's field.

type ColumnSource added in v0.92.0 

type ColumnSource struct {
	Instance string `json:"instance"`
	Schema   string `json:"schema"`
	Relation string `json:"relation"`
	Column   string `json:"column"`
}

ColumnSource is the source field a column reads.

type DB 

type DB struct {
	// Ref is datatug://<project>/<db>?env=<env>, or nil when the database is
	// not a source of a registered project.
	Ref     *string `json:"ref"`
	Project *string `json:"project"`
	Env     *string `json:"env"`
	Source  *string `json:"source"`
	Driver  string  `json:"driver"`
}

DB says which database the query ran on, by its project reference only.

type Data 

type Data struct {
	Bytes   []byte
	Columns []Column
	Rows    Rows
}

Data is a built data.json and what meta.json says about it.

type Error 

type Error struct{ Reason string }

Error is a store failure: a reason code, never a path or an id.

func (*Error) Error 

func (e *Error) Error() string

type JoinPair added in v0.92.0 

type JoinPair struct {
	Left  string `json:"left"`
	Right string `json:"right"`
}

JoinPair is one equality of a join: alias.field on each side.

type Loaded added in v0.93.0 

type Loaded struct {
	Meta      Meta
	Recordset apicontract.Recordset
	// MetaJSON and DataJSON are the files as they were read.
	MetaJSON, DataJSON []byte
}

Loaded is a complete result read back from its files.

type Locator 

type Locator struct {
	Version int    `json:"version"`
	ID      string `json:"id"`
	Root    string `json:"root"`
	Data    string `json:"data"`
	Meta    string `json:"meta"`
}

Locator is where a complete result's files are.

type Meta 

type Meta struct {
	Format     string      `json:"format"`
	Version    int         `json:"version"`
	ID         string      `json:"id"`
	CreatedAt  time.Time   `json:"createdAt"`
	ExpiresAt  time.Time   `json:"expiresAt"`
	CLIVersion string      `json:"cliVersion"`
	DataBytes  int64       `json:"dataBytes"`
	DB         DB          `json:"db"`
	Query      Query       `json:"query"`
	Rows       Rows        `json:"rows"`
	Provenance *Provenance `json:"provenance"`
	Columns    []Column    `json:"columns"`
	// Participants are the query's source occurrences and Related the tables
	// that can be joined to them.
	Participants []Participant `json:"participants"`
	Related      []Related     `json:"related"`
	// Unavailable names, per part, why it is missing or cut short.
	Unavailable Unavailable `json:"unavailable"`
}

Meta is meta.json: everything needed to draw a result except its rows. It holds no row value, connection string, file path, host, user, password, environment variable or policy path.

func NewMeta 

func NewMeta(id, cliVersion string, data Data) Meta

NewMeta starts the meta.json of id written now.

func (*Meta) Describe added in v0.92.0 

func (m *Meta) Describe(facts map[string]ColumnFacts)

Describe sets the declared facts of each column found in facts.

func (Meta) Encode 

func (m Meta) Encode() []byte

Encode applies the bounds of meta.json and encodes it: query.dtql and query.parameters together within MaxQueryBytes (else both are dropped, "too-large"), at most MaxMetaColumns columns, and the whole within MaxMetaBytes (else the column descriptions are dropped).

type Participant added in v0.92.0 

type Participant struct {
	Instance string           `json:"instance"`
	Schema   string           `json:"schema"`
	Relation string           `json:"relation"`
	Alias    string           `json:"alias"`
	DB       *string          `json:"db,omitempty"`
	Kind     string           `json:"kind"`
	Role     string           `json:"role"`
	Join     *ParticipantJoin `json:"join,omitempty"`
}

Participant is one source occurrence of the query.

type ParticipantJoin added in v0.92.0 

type ParticipantJoin struct {
	Type string     `json:"type"`
	On   []JoinPair `json:"on"`
}

ParticipantJoin is how a participant is joined.

type Provenance 

type Provenance struct {
	Mode       string `json:"mode"`
	ObservedAt string `json:"observedAt"`
}

Provenance is whether an HTTP source answered live or from a snapshot.

type Query 

type Query struct {
	Kind           string                                 `json:"kind"`
	DTQL           string                                 `json:"dtql,omitempty"`
	From           string                                 `json:"from,omitempty"`
	SavedQueryID   string                                 `json:"savedQueryId,omitempty"`
	SavedQueryType string                                 `json:"savedQueryType,omitempty"`
	Parameters     map[string]apicontract.TypedValueOrSet `json:"parameters,omitempty"`
	Policy         string                                 `json:"policy"`
}

Query is what ran: enough to run it again. 

type Related struct {
	CandidateID  string         `json:"candidateId"`
	FromInstance string         `json:"fromInstance"`
	Target       RelatedTarget  `json:"target"`
	ConstraintID string         `json:"constraintId"`
	Direction    string         `json:"direction"`
	Cardinality  string         `json:"cardinality"`
	Fields       []RelatedField `json:"fields"`
	Evidence     string         `json:"evidence"`
}

Related is one table that can be joined to a participant by a foreign key.

type RelatedField added in v0.92.0 

type RelatedField struct {
	SourceField string `json:"sourceField"`
	TargetField string `json:"targetField"`
}

RelatedField pairs a field of the participant with one of the target.

type RelatedTarget added in v0.92.0 

type RelatedTarget struct {
	Schema   string `json:"schema"`
	Relation string `json:"relation"`
	Kind     string `json:"kind"`
}

RelatedTarget is the table a related candidate joins.

type Result 

type Result struct {
	// contains filtered or unexported fields
}

Result is a result directory being written.

func (*Result) Discard 

func (r *Result) Discard()

Discard removes a result that will not be completed.

func (*Result) Write 

func (r *Result) Write(data, meta []byte) error

Write writes data.json, then meta.json; each is an O_EXCL 0600 temporary file, synced and renamed into place.

type Rows 

type Rows struct {
	Kept        int    `json:"kept"`
	HasMore     bool   `json:"hasMore"`
	TruncatedBy string `json:"truncatedBy"`
}

Rows says how many rows data.json keeps and whether more exist.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is one results root.

func Open 

func Open(root string) (*Store, error)

Open checks root without creating it: it must not lie inside a Git repository, must not be a symbolic link, and, when it exists, must be a directory owned by the user.

func OpenDefault 

func OpenDefault() (*Store, error)

OpenDefault opens the store at DefaultRoot.

func (*Store) Check 

func (s *Store) Check() error

Check proves the store can be written: it creates the root when missing, then creates and removes one private temporary file in it.

func (*Store) Create 

func (s *Store) Create(id string) (*Result, error)

Create makes <root>/<id> with mkdir, so it fails when the id is in use: a result is never overwritten, and nothing is written through a path that already existed.

func (*Store) Load added in v0.93.0 

func (s *Store) Load(id string) (Loaded, error)

Load reads a complete, unexpired result: it locates it with the same checks as Locate, then reads meta.json and data.json, refusing a data.json whose size is not the one meta.json records and a format version it does not know.

func (*Store) Locate 

func (s *Store) Locate(id string) (Locator, error)

Locate answers where a complete, unexpired result's files are.

func (*Store) Prune 

func (s *Store) Prune(all bool) (int, error)

Prune removes expired results, directories without meta.json older than IncompleteGrace, and then the oldest results while more than MaxResults or MaxStoreBytes remain. With all it removes every result. Only directories whose name follows the id grammar are touched, after the symlink and owner checks; a link is never followed. It answers how many were removed.

func (*Store) Remove 

func (s *Store) Remove(id string) error

Remove deletes one result directory; a link is never followed.

func (*Store) Root 

func (s *Store) Root() string

Root is the absolute root of the store.

type Unavailable 

type Unavailable struct {
	Query         string `json:"query,omitempty"`
	Columns       string `json:"columns,omitempty"`
	Participants  string `json:"participants,omitempty"`
	Related       string `json:"related,omitempty"`
	DeclaredTypes string `json:"declaredTypes,omitempty"`
}

Unavailable holds a reason code per part of the metadata that is missing.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.