queryresults

package
v0.91.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 16, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package queryresults keeps the private result files that `datatug query run --result-id <id>` writes: <root>/<id>/data.json (the rows) and <root>/<id>/meta.json (everything a card needs except the rows).

The root is always ~/.datatug/query-results: there is deliberately no environment override, so the shell that ran the query, the person's terminal and the daemon all see the same files. Every directory is 0700 and every file 0600; a symlinked root or result directory, one owned by another user, and a root inside a Git repository are refused. Files are written as O_EXCL temporary files, synced and renamed into place, data.json first and meta.json last, so a result is complete exactly when meta.json exists and data.json has the byte length meta.json records.

Every error this package returns is an *Error carrying a reason code. It names no path and no id, so it is safe to show to a model.

Index

Constants

View Source 
const (
	QueryKindFrom  = "from"
	QueryKindDTQL  = "dtql"
	QueryKindSaved = "saved"

	PolicyNone       = "none"
	PolicyDiscovered = "discovered"
	PolicyDir        = "dir"
	PolicyExplicit   = "explicit"
	PolicyProject    = "project"

	UnavailableTooLarge = "too-large"
	UnavailableTooMany  = "too-many"
)

Query kinds, policy modes and unavailable reasons of meta.json.

View Source 
const (
	Version    = 1
	DataFormat = "datatug.query-result.data"
	MetaFormat = "datatug.query-result.meta"

	// DefaultRows, MinRows and MaxRows bound --result-rows.
	DefaultRows = 2000
	MinRows     = 1
	MaxRows     = 10000

	// MaxDataBytes keeps data.json under the mod SDK's 4 MiB read limit.
	MaxDataBytes = 4_000_000
	// MaxCellBytes is where one cell is cut, on a UTF-8 boundary.
	MaxCellBytes = 64 << 10
	// MaxMetaBytes bounds meta.json.
	MaxMetaBytes = 1 << 20
	// MaxQueryBytes bounds query.dtql and query.parameters together.
	MaxQueryBytes = 256 << 10
	// MaxMetaColumns is how many columns meta.json describes.
	MaxMetaColumns = 400

	// KeyColumn is the record key column, always first.
	KeyColumn = "$key"

	EncodingBinary = "binary-placeholder"
	EncodingJSON   = "json-text"

	TruncatedNone  = "none"
	TruncatedRows  = "rows"
	TruncatedBytes = "bytes"
)

Result file formats and bounds.

View Source 
const (
	// DefaultDir is the root below the user's home directory.
	DefaultDir = ".datatug/query-results"
	// Retention is how long a result lives.
	Retention = 7 * 24 * time.Hour
	// MaxResults and MaxStoreBytes bound the store; the oldest results go first.
	MaxResults    = 200
	MaxStoreBytes = 512 << 20
	// IncompleteGrace is how old a directory without meta.json must be before
	// pruning removes it (a write may still be in progress before that).
	IncompleteGrace = time.Hour

	DataFile = "data.json"
	MetaFile = "meta.json"
)

Store bounds and lifetimes.

View Source 
const (
	ReasonUnwritable = "results-store-unwritable"
	ReasonInsideGit  = "inside-git"
	ReasonSymlink    = "symlink"
	ReasonOwner      = "owner"
	ReasonIDInUse    = "result-id-in-use"
	ReasonBadID      = "bad-result-id"
	ReasonNotFound   = "not-found"
	ReasonIncomplete = "incomplete"
	ReasonExpired    = "expired"
)

Reason codes. They are the whole message of an *Error.

Variables

This section is empty.

Functions

func ConvertValue 

func ConvertValue(v any) apicontract.TypedValue

ConvertValue is v as a TypedValue, as data.json writes it.

func DefaultRoot 

func DefaultRoot() (string, error)

DefaultRoot is ~/.datatug/query-results.

func FailureLine 

func FailureLine(reason string) []byte

FailureLine is the path-free line printed when a --result-id run fails.

func LocateLine 

func LocateLine(locator *Locator, reason string) []byte

LocateLine is the answer of `query results locate`: the paths of a complete result, or null with the reason.

func LocatorLine 

func LocatorLine(data Data) []byte

Locator is the path-free line `query run --result-id` prints: it holds no id, no path and no row.

func ReasonOf 

func ReasonOf(err error) string

ReasonOf is the reason code of err when it is an *Error, else "".

func RecordData 

func RecordData(data any) (map[string]any, error)

RecordData is the fields of a record's data with their driver values: a map is used as it is, and any other shape is read through its JSON form with numbers kept as exact text.

func ValidID 

func ValidID(id string) bool

ValidID reports whether id follows the result id grammar.

Types

type Collector 

type Collector struct {
	// contains filtered or unexported fields
}

Collector keeps the first rows of a result, within the row and byte bounds.

func NewCollector 

func NewCollector(limit int) *Collector

NewCollector keeps at most limit rows.

func (*Collector) Add 

func (c *Collector) Add(key string, data map[string]any) bool

Add takes one record: its key as text and its fields. It answers false once no further record is wanted; the caller then stops reading, so a source is read for at most limit + 1 records.

func (*Collector) Build 

func (c *Collector) Build(id string, columns []string) Data

Build encodes data.json for id with the columns given ($key is put first). A column's type is its one observed kind; a column of several kinds is a string column with each cell written as its text, and it records the kinds in ObservedTypes. Rows are dropped from the end while the file would exceed MaxDataBytes.

func (*Collector) FieldNames 

func (c *Collector) FieldNames() [][]string

FieldNames lists, per kept row, the names of its fields.

type Column 

type Column struct {
	Name          string   `json:"name"`
	Type          string   `json:"type"`
	Encoding      string   `json:"encoding,omitempty"`
	CutCells      int      `json:"cutCells,omitempty"`
	ObservedTypes []string `json:"observedTypes,omitempty"`
}

Column describes one column of data.json.

type DB 

type DB struct {
	// Ref is datatug://<project>/<db>?env=<env>, or nil when the database is
	// not a source of a registered project.
	Ref     *string `json:"ref"`
	Project *string `json:"project"`
	Env     *string `json:"env"`
	Source  *string `json:"source"`
	Driver  string  `json:"driver"`
}

DB says which database the query ran on, by its project reference only.

type Data 

type Data struct {
	Bytes   []byte
	Columns []Column
	Rows    Rows
}

Data is a built data.json and what meta.json says about it.

type Error 

type Error struct{ Reason string }

Error is a store failure: a reason code, never a path or an id.

func (*Error) Error 

func (e *Error) Error() string

type Locator 

type Locator struct {
	Version int    `json:"version"`
	ID      string `json:"id"`
	Root    string `json:"root"`
	Data    string `json:"data"`
	Meta    string `json:"meta"`
}

Locator is where a complete result's files are.

type Meta 

type Meta struct {
	Format     string      `json:"format"`
	Version    int         `json:"version"`
	ID         string      `json:"id"`
	CreatedAt  time.Time   `json:"createdAt"`
	ExpiresAt  time.Time   `json:"expiresAt"`
	CLIVersion string      `json:"cliVersion"`
	DataBytes  int64       `json:"dataBytes"`
	DB         DB          `json:"db"`
	Query      Query       `json:"query"`
	Rows       Rows        `json:"rows"`
	Provenance *Provenance `json:"provenance"`
	Columns    []Column    `json:"columns"`
	// Unavailable names, per part, why it is missing or cut short.
	Unavailable Unavailable `json:"unavailable"`
}

Meta is meta.json: everything needed to draw a result except its rows. It holds no row value, connection string, file path, host, user, password, environment variable or policy path.

func NewMeta 

func NewMeta(id, cliVersion string, data Data) Meta

NewMeta starts the meta.json of id written now.

func (Meta) Encode 

func (m Meta) Encode() []byte

Encode applies the bounds of meta.json and encodes it: query.dtql and query.parameters together within MaxQueryBytes (else both are dropped, "too-large"), at most MaxMetaColumns columns, and the whole within MaxMetaBytes (else the column descriptions are dropped).

type Provenance 

type Provenance struct {
	Mode       string `json:"mode"`
	ObservedAt string `json:"observedAt"`
}

Provenance is whether an HTTP source answered live or from a snapshot.

type Query 

type Query struct {
	Kind           string                                 `json:"kind"`
	DTQL           string                                 `json:"dtql,omitempty"`
	From           string                                 `json:"from,omitempty"`
	SavedQueryID   string                                 `json:"savedQueryId,omitempty"`
	SavedQueryType string                                 `json:"savedQueryType,omitempty"`
	Parameters     map[string]apicontract.TypedValueOrSet `json:"parameters,omitempty"`
	Policy         string                                 `json:"policy"`
}

Query is what ran: enough to run it again.

type Result 

type Result struct {
	// contains filtered or unexported fields
}

Result is a result directory being written.

func (*Result) Discard 

func (r *Result) Discard()

Discard removes a result that will not be completed.

func (*Result) Write 

func (r *Result) Write(data, meta []byte) error

Write writes data.json, then meta.json; each is an O_EXCL 0600 temporary file, synced and renamed into place.

type Rows 

type Rows struct {
	Kept        int    `json:"kept"`
	HasMore     bool   `json:"hasMore"`
	TruncatedBy string `json:"truncatedBy"`
}

Rows says how many rows data.json keeps and whether more exist.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store is one results root.

func Open 

func Open(root string) (*Store, error)

Open checks root without creating it: it must not lie inside a Git repository, must not be a symbolic link, and, when it exists, must be a directory owned by the user.

func OpenDefault 

func OpenDefault() (*Store, error)

OpenDefault opens the store at DefaultRoot.

func (*Store) Check 

func (s *Store) Check() error

Check proves the store can be written: it creates the root when missing, then creates and removes one private temporary file in it.

func (*Store) Create 

func (s *Store) Create(id string) (*Result, error)

Create makes <root>/<id> with mkdir, so it fails when the id is in use: a result is never overwritten, and nothing is written through a path that already existed.

func (*Store) Locate 

func (s *Store) Locate(id string) (Locator, error)

Locate answers where a complete, unexpired result's files are.

func (*Store) Prune 

func (s *Store) Prune(all bool) (int, error)

Prune removes expired results, directories without meta.json older than IncompleteGrace, and then the oldest results while more than MaxResults or MaxStoreBytes remain. With all it removes every result. Only directories whose name follows the id grammar are touched, after the symlink and owner checks; a link is never followed. It answers how many were removed.

func (*Store) Remove 

func (s *Store) Remove(id string) error

Remove deletes one result directory; a link is never followed.

func (*Store) Root 

func (s *Store) Root() string

Root is the absolute root of the store.

type Unavailable 

type Unavailable struct {
	Query   string `json:"query,omitempty"`
	Columns string `json:"columns,omitempty"`
}

Unavailable holds a reason code per part of the metadata that is missing.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.