Documentation
¶
Overview ¶
Package toolkitcfg resolves typed per-toolkit connection configuration out of the platform's raw toolkits config (map[string]any decoded from YAML/JSON).
The platform stores toolkit config as toolkits.<kind>.instances.<name>.<key>; these helpers walk that structure, pick the default instance when none is named, and extract the typed DataHub / Trino / S3 settings the providers need. Split out of pkg/platform to keep that package under its size budget (#756); the primitive typed-map accessors live in the sibling cfgmap package.
Index ¶
- Constants
- func AutoEnableKind(toolkits map[string]any, kind string)
- func BlobReadTimeout(configured time.Duration, maxObjectBytes int64) time.Duration
- func InstanceConfig(toolkits map[string]any, kind, instance string) map[string]any
- func KindEnabled(kindMap map[string]any) bool
- func MergeInstance(toolkits map[string]any, kind, name string, cfg map[string]any)
- func MergeStored(toolkits map[string]any, instances []StoredInstance)
- func MissingDefaults(toolkits map[string]any) []string
- func PinDeclaredDefaults(toolkits map[string]any)
- func ResolveDefaultInstance(kindCfg, instances map[string]any) string
- type DataHub
- type DeclaredConnections
- type S3
- type StoredInstance
- type Trino
Constants ¶
const ( DefaultTrinoPort = 8080 DefaultTrinoQueryLimit = 1000 DefaultTrinoMaxLimit = 10000 )
Trino connection defaults, applied when an instance omits the setting.
const DefaultBlobReadTimeout = 30 * time.Second
DefaultBlobReadTimeout is the floor a platform-built S3 client's timeout is never set below. It is the value mcp-s3 fills in for itself, so a deployment storing small objects gets exactly the client it got before.
Variables ¶
This section is empty.
Functions ¶
func AutoEnableKind ¶ added in v1.123.0
AutoEnableKind ensures toolkits[kind] exists with enabled=true so the toolkit loader will instantiate it. Idempotent and non-overriding: if the operator has already declared the kind block (enabled OR disabled), their explicit choice is respected.
Logs at Debug, not Info: this is the platform's documented default behavior, not an exceptional condition that requires operator attention. Operators who want to silence the path entirely can set the kind explicitly in YAML (with either enabled state).
func BlobReadTimeout ¶ added in v1.132.4
BlobReadTimeout is the deadline the platform's own S3 clients carry.
mcp-s3 wraps the request and the io.ReadAll of the body in one deadline and fills in 30 s when a caller sets none, which neither builder of these clients did. On a read of a whole object that is not a timeout but a throughput budget: the same 126 MB CSV registered in 2 s on a healthy path and failed at exactly 30.0 s, 116 MB in, the day the deployment's DNS sent the pod across a WAN link -- so whether a file could be registered turned on which A record the resolver handed out (#1773).
The registrar already bounds what it will read to the deployment's own upload ceiling, so that ceiling is what the deadline is sized from: the largest object the platform will ever ask for, at a throughput floor it would have to fall under to be considered broken. An operator who sets the instance's `timeout` still gets exactly that -- their word is the answer, and until now it reached the s3_* tools and nothing else.
This is arithmetic against a deadline that should not have to bound a transfer at all; the separation belongs in mcp-s3, and is asked for there as txn2/mcp-s3#148.
func InstanceConfig ¶
InstanceConfig retrieves one instance's config map for a toolkit kind. When instance is "" it resolves the default (or first) instance. Returns nil if the kind, its instances map, or the named instance is absent or malformed.
func KindEnabled ¶ added in v1.123.0
KindEnabled reports whether a toolkit kind map has enabled=true. It handles both bool and string values, because environment-variable expansion produces strings.
func MergeInstance ¶ added in v1.123.0
MergeInstance merges one stored connection into the toolkit config map under its kind's instances. It is a no-op when the kind is absent or disabled, and when the kind already carries an instance of that name: file config takes precedence over a connection held in the database.
Instances merged here arrive after Config.Validate has run, so they are not covered by MissingDefaults: a kind can hold several of them with no "default", which is why ResolveDefaultInstance resolves deterministically rather than relying on that refusal.
func MergeStored ¶ added in v1.132.0
func MergeStored(toolkits map[string]any, instances []StoredInstance)
MergeStored folds the saved connections into the toolkit configuration the loader builds from, so a process starts serving them rather than taking each one on when a call first names it.
This is a warm start, not the inventory. What connections exist is what the connection store holds — that is what an enumeration reports and what a call naming a connection this process does not serve resolves against (#1757) — and a process that merged nothing would still answer for all of them, one store read later. What the merge buys is that the common case costs no read, and that a kind declared in the file keeps the meaning the file gave it: PinDeclaredDefaults runs before any stored connection joins an instance map, so a saved connection whose name sorts earlier cannot take over an unqualified lookup a declared instance answers today.
A connection the file already declares under the same name is left alone by MergeInstance: the file is what this process runs on.
func MissingDefaults ¶ added in v1.123.0
MissingDefaults returns one message per toolkit kind that configures more than one instance without a "default" key naming which of them a lookup that omits the instance means. Kinds and the instance names within a message are sorted, so the same config produces the same messages on every run. Callers treat a non-empty result as a config error; Config.Validate does.
Two DataHub catalogs with no default is not a deployment that chose either one: whichever the platform picks binds the semantic provider, the query provider and the managed-resource blob store to a connection the operator never named. Naming the candidates lets them make that choice.
A kind is checked whether or not it is enabled, because the providers read an instance's config through InstanceConfig without consulting the enable flag: a catalog used only for enrichment registers no tools and still has to say which of its instances the enrichment reads.
This covers only the instances a config declares. Connections held in the database merge into the toolkits config after validation; PinDeclaredDefaults keeps them from taking over the lookup a declared instance answers today.
func PinDeclaredDefaults ¶ added in v1.123.0
PinDeclaredDefaults records, for every kind that declares instances without a "default", the instance its config resolves to today. Call it once before merging connections held in the database.
Without it, a kind that declares a single instance and needs no "default" changes meaning when an admin-UI connection whose name sorts earlier joins the same map: the next restart resolves the unqualified lookup to the new connection and moves a provider, or managed-resource blob storage, off the connection the file pointed at. Pinning first is the same rule MergeInstance already follows, that file config outranks a stored connection.
func ResolveDefaultInstance ¶
ResolveDefaultInstance determines which instance a lookup that names none means: the one named by the kind's "default" key, else the lexicographically first instance, else "".
The fallback compares names rather than taking whatever a map range hands back first. Go randomizes map iteration order, so ranging resolved a different instance on every process start: two replicas built from one config disagreed about which connection an unqualified lookup meant, and a restart could point managed-resource blob storage at a different S3 connection than the one existing resources were written through. The multi-connection Trino toolkit picks its own default the same way (pkg/toolkits/trino/toolkit.go).
An empty "default" is treated as absent, so it agrees with MissingDefaults about which configs have named an instance.
Types ¶
type DeclaredConnections ¶ added in v1.123.0
DeclaredConnections records, per toolkit kind, the connection instances the config file declared. It is the only record of which connections the file owns: MergeInstance puts stored connections into the same instances map the file produced, and connbackfill seeds a connection_instances row for every file-configured connection, so neither the merged config nor the store can answer "did the file declare this one" afterwards.
The keys are the instance names, which is the same namespace a connection_instances row's name and a MergeInstance call use.
func Declared ¶ added in v1.123.0
func Declared(toolkits map[string]any) DeclaredConnections
Declared snapshots the instances each kind declares. Call it before MergeInstance merges a stored connection into the same map, and before PinDeclaredDefaults, which reads instances but adds none.
Every kind is captured, including the ones the admin connection API does not manage: a file-declared datahub instance is as much the file's as a trino one.
func (DeclaredConnections) Has ¶ added in v1.123.0
func (d DeclaredConnections) Has(kind, name string) bool
Has reports whether the config file declared name as an instance of kind. The zero value declares nothing, so a caller holding no snapshot treats every connection as database-owned.
type S3 ¶
type S3 struct {
Region string
Endpoint string
AccessKeyID string
SecretKey string
BucketPrefix string
// Timeout is the instance's `timeout` key, and is 0 when the instance sets
// none. It bounds a whole S3 call in the client the platform builds from
// this -- the request AND the read of the response body -- so a caller
// reading a whole object is choosing a throughput budget when it sets one.
// See BlobReadTimeout.
Timeout time.Duration
ConnectionName string
UsePathStyle bool
}
S3 holds extracted S3 configuration.
type StoredInstance ¶ added in v1.132.0
StoredInstance is one saved connection as the warm start reads it.
type Trino ¶
type Trino struct {
Host string
Port int
User string
Password string // #nosec G117 -- Trino connection credential from admin config
Catalog string
Schema string
SSL bool
SSLVerify bool
Timeout time.Duration
DefaultLimit int
MaxLimit int
ReadOnly bool
// ConnectionName is the name a call binds this connection by: the resolved
// `instances:` key, which is what the Trino toolkit routes on. The query
// provider stamps it onto every availability answer, so it is the name an
// agent is told to pass as `connection` — a label the router does not know
// would send the agent to a connection that refuses it (#1396).
ConnectionName string
}
Trino holds extracted Trino configuration.
func TrinoConfig ¶
TrinoConfig extracts Trino configuration for the named instance (or the default instance when instance is ""). Returns nil if not configured.
ConnectionName is the resolved instance name rather than the instance's `connection_name`: Trino routes by instance, so that key is the name a `connection` argument carries and a persona rule matches, and it is what the toolkit reports as its connection. Reading `connection_name` here published a name no call could bind, and reading nothing published an empty one (#1396).
Source Files