Documentation
¶
Overview ¶
Package runconfig owns the direct `tau run --config` YAML contract.
Index ¶
- Constants
- Variables
- func DiscoverDefault(dir string) (string, error)
- func ExperimentRunMetadata(e Experiment) experiment.RunMetadata
- func IsKnownManagedKey(path string) bool
- func JSONSchema() ([]byte, error)
- func ManagedPassthroughPaths() []string
- func MetricsOffloadTerminalDrainTimeout(maxAttempts int, retryBackoff, finalStatusTimeout time.Duration) (time.Duration, error)
- func NormalizeEngine(engine string) (string, error)
- func QualifiedNameSegmentRE() *regexp.Regexp
- func ReferenceMarkdown() string
- func ReservedTauEnvKey(name string) bool
- func ReservedTauEnvKeyError(name string) error
- func ResolveMetricsOffloadDeliveryMode(value string) (string, error)
- func ResolveMetricsOffloadRuntime(value string) (string, error)
- func ValidateContainerWorkingDir(value string) error
- func ValidateLiteralEnvPayloads(env map[string]string) error
- func ValidateMetricsOffloadImage(image string) error
- func ValidateMetricsOffloadRuntimeImage(runtime, image string) error
- type Compute
- type Config
- type Duration
- type Execution
- type Experiment
- type FieldInfo
- type FieldStatus
- type ImageAsset
- type Metrics
- type MetricsOffload
- type NormalizedRDMA
- type Policy
- type Profiler
- type RDMA
- type Resilience
- type Run
- type Runtime
- type Security
- type Source
- type Storage
- type UnknownKey
- type Workflow
Constants ¶
const ( // MaxLiteralEnvValueBytes leaves ample room below Linux's per-string exec // limit and prevents source archives from becoming Kubernetes metadata. MaxLiteralEnvValueBytes = 64 * 1024 // MaxLiteralEnvTotalBytes bounds the control-plane amplification when a Job // is copied into Kueue Workloads and Pods. Secret-backed values are excluded. MaxLiteralEnvTotalBytes = 128 * 1024 // MaxTTLSecondsAfterFinished is Kubernetes' signed int32 ceiling for // batch/v1 Job spec.ttlSecondsAfterFinished. MaxTTLSecondsAfterFinished int64 = 1<<31 - 1 )
const ( MetricsOffloadRuntimeCollectorV1 = "collector-v1" MetricsOffloadDeliveryADXRequired = "adx-required" MetricsOffloadMaxADXAttempts = offloadcontract.MaxADXAttempts MetricsOffloadDefaultADXAttempts = offloadcontract.DefaultADXAttempts MetricsOffloadDefaultADXRetryBackoff = offloadcontract.DefaultADXRetryBackoff MetricsOffloadDefaultADXFinalStatusTimeout = offloadcontract.DefaultADXFinalStatusTimeout MetricsOffloadTerminalDrainGrace = offloadcontract.TerminalDrainGrace MetricsOffloadMaxTerminalDrainTimeout = offloadcontract.MaxTerminalDrainTimeout )
const ( EngineJob = "job" EngineRayJob = "rayjob" )
const DefaultRDMAShmSize = "32Gi"
DefaultRDMAShmSize is the /dev/shm size for RDMA Job workloads when the renderer does not receive an explicit override.
const SecurityModeRestricted = "restricted"
const SourceMountPath = "/tau/source"
Variables ¶
var DefaultFiles = []string{"tau.yaml", "tau.yml", ".tau.yaml"}
DefaultFiles are the run config names Tau looks for when no --config is given, in precedence order.
var TauEnvAllowed = []string{
"TAU_RESUME_FROM",
"TAU_RETRY_ATTEMPT",
"TAU_RETRY_MAX",
"TAU_RETRY_REASON",
}
TauEnvAllowed is every TAU_-prefixed env key a user may set. The retry loop injects these four and researcher code reads them back, so they have to survive a round trip through runtime.env; the rest of the namespace is Tau's. This is the whole rule — see ReservedTauEnvKey.
Functions ¶
func DiscoverDefault ¶
DiscoverDefault returns the path of the first DefaultFiles entry present in dir, or "" when the directory holds no run config. An empty dir means the process working directory.
Both products need this: `tau run` resolves the config it is about to submit, and taugrid-portal reads the same file to default experiment flags so `taugrid-portal experiment track` picks up the project and experiment a repository already declares.
func ExperimentRunMetadata ¶
func ExperimentRunMetadata(e Experiment) experiment.RunMetadata
ExperimentRunMetadata derives run identity from a config's experiment block.
func IsKnownManagedKey ¶
func JSONSchema ¶
func ManagedPassthroughPaths ¶
func ManagedPassthroughPaths() []string
IsKnownManagedKey reports whether a dotted path in a managed workflow manifest is recognized, either because Config models it or because it is an accepted pass-through. Exported so cli/internal/manifest can assert, by reflection over the real manifest type, that this package never warns about a legitimate manifest field. ManagedPassthroughPaths returns the manifest-owned keys this package tolerates without warning. Exported so cli/internal/manifest can assert the list in both directions: that no manifest field is missing from it (a missing entry warns spuriously) and that no entry has outlived the field it was added for (a stale entry silently permits a key that no longer exists).
func MetricsOffloadTerminalDrainTimeout ¶
func MetricsOffloadTerminalDrainTimeout(maxAttempts int, retryBackoff, finalStatusTimeout time.Duration) (time.Duration, error)
MetricsOffloadTerminalDrainTimeout returns the finite workload-side deadline that covers every configured ADX attempt, exponential retry backoff, and a small coordination grace period. Configurations exceeding the platform ceiling are rejected so workload teardown never races a longer collector delivery budget.
func NormalizeEngine ¶
NormalizeEngine returns the canonical workload engine. "ray" remains an accepted compatibility alias, but Tau always submits that path as a RayJob.
func QualifiedNameSegmentRE ¶
QualifiedNameSegmentRE is the exported accessor for template renderers that need the same regex (e.g. Kubernetes label value validation). The pattern is anchored and matches a single segment of a qualified resource name.
func ReferenceMarkdown ¶
func ReferenceMarkdown() string
func ReservedTauEnvKey ¶
ReservedTauEnvKey reports whether name is inside the Tau-owned TAU_ namespace without being one of the permitted keys. It is the single gate for both `tau run --config` validation and the Job and RayJob renderers, so a key rejected at render is rejected at load with the same message.
The namespace test is case-insensitive but the escape hatch is exact-case, and the asymmetry is the point. Environment variables are case-sensitive, so `tau_resume_from` is not the key the retry loop injects: code reading TAU_RESUME_FROM would find nothing, silently, on the retry path. Permitting it as a distinct variable hands the researcher that failure; rejecting it shows them the misspelling, because the error lists the keys in the case that works.
The name is matched as given. A legal env name has no surrounding space, so trimming one off could only let a malformed name read as a permitted key. Leading space is not a bypass either: " TAU_RESUME_FROM" is not a name this predicate claims, and every path that reaches a workload validates names against envspec's C_IDENTIFIER rule, which rejects any surrounding space. Reporting such a name as a TAU_ namespace violation would misdescribe an ordinary typo -- " PATH" is not a Tau key.
func ReservedTauEnvKeyError ¶
ReservedTauEnvKeyError is the rejection message for a reserved TAU_ key. It names the permitted keys rather than only asserting the namespace is owned, so the reader's next action is defined.
func ResolveMetricsOffloadDeliveryMode ¶
ResolveMetricsOffloadDeliveryMode returns the collector sink contract. Empty selects the typed ADX-only required-delivery contract.
func ResolveMetricsOffloadRuntime ¶
ResolveMetricsOffloadRuntime returns the sole supported metrics offload executable contract.
func ValidateContainerWorkingDir ¶
ValidateContainerWorkingDir enforces the path shape Kubernetes expects for a container workingDir without resolving it on the host.
func ValidateLiteralEnvPayloads ¶
ValidateLiteralEnvPayloads rejects content-sized literal environment values. Kubernetes repeats literal values across Job, Workload, and Pod objects, and Linux includes them in the argv+environment budget when starting a process. Secret-backed references are intentionally not passed to this function.
func ValidateMetricsOffloadImage ¶
ValidateMetricsOffloadImage rejects mutable or implicit sidecar image references while allowing either an exact digest or an explicit non-latest release tag.
func ValidateMetricsOffloadRuntimeImage ¶
ValidateMetricsOffloadRuntimeImage validates the pinned image reference for an explicit executable contract. Runtime is authoritative: Tau deliberately does not infer compatibility from an image repository name.
Types ¶
type Compute ¶
type Compute struct {
Workers *int `yaml:"workers"`
GPUs *int `yaml:"gpus"`
GPUsPerWorker *int `yaml:"gpus_per_worker"`
CPUWorkers *int `yaml:"cpu_workers"`
WorkloadKind string `yaml:"workload_kind"`
GPUResourceMode string `yaml:"gpu_resource_mode"`
MIGProfile string `yaml:"mig_profile"`
CPURequest string `yaml:"cpu_request"`
MemoryRequest string `yaml:"memory_request"`
CPULimit string `yaml:"cpu_limit"`
MemoryLimit string `yaml:"memory_limit"`
HeadCPURequest string `yaml:"head_cpu_request"`
HeadMemRequest string `yaml:"head_memory_request"`
HeadCPULimit string `yaml:"head_cpu_limit"`
HeadMemLimit string `yaml:"head_memory_limit"`
WorkerCPUReq string `yaml:"worker_cpu_request"`
WorkerMemReq string `yaml:"worker_memory_request"`
WorkerCPULimit string `yaml:"worker_cpu_limit"`
WorkerMemLimit string `yaml:"worker_memory_limit"`
}
type Config ¶
type Config struct {
SchemaVersion any `yaml:"schema_version"`
Name string `yaml:"name"`
Engine string `yaml:"engine"`
Entrypoint string `yaml:"entrypoint"`
Script string `yaml:"script"`
Image string `yaml:"image"`
Run Run `yaml:"run"`
Workflow Workflow `yaml:"workflow"`
Runtime Runtime `yaml:"runtime"`
Compute Compute `yaml:"compute"`
Policy Policy `yaml:"policy"`
Storage Storage `yaml:"storage"`
Profiler Profiler `yaml:"profiler"`
Metrics Metrics `yaml:"metrics"`
Experiment Experiment `yaml:"experiment"`
Execution Execution `yaml:"execution"`
Resilience Resilience `yaml:"resilience"`
}
Config is the implementation-backed shape for direct `tau run --config` files. SDK-generated managed workflow manifests share some field names, but are intentionally documented and validated separately.
func Load ¶
Load reads and validates a direct run config. SDK-generated managed workflow manifests keep their legacy permissive parse so `tau run` does not start rejecting pass-through manifest data while that contract is clarified.
Callers that can surface diagnostics should prefer LoadWithDiagnostics: this wrapper drops the unknown-key warnings on the floor.
func LoadWithDiagnostics ¶
LoadWithDiagnostics is Load plus non-fatal warnings, currently unknown keys in managed workflow manifests.
func (Config) Hash ¶
Hash returns a stable hash of the validated config contract. Source payloads have their own digest metadata; this hash identifies the complete run config.
func (Config) LooksLikeManagedWorkflow ¶
func (Config) ValidateDirect ¶
func (Config) ValidateExecution ¶
ValidateExecution checks that execution.launcher is valid for the given engine and that engine-specific constraints (nodes, processes_per_node, clear_node_selector) are met.
type Execution ¶
type Execution struct {
Launcher *string `yaml:"launcher"`
ProcessesPerNode *int `yaml:"processes_per_node"`
Nodes *int `yaml:"nodes"`
Metric string `yaml:"metric"`
Mode string `yaml:"mode"`
NumSamples *int `yaml:"num_samples"`
MaxConcurrentTrials *int `yaml:"max_concurrent_trials"`
Configs map[string]any `yaml:"configs"`
AllowNCCLOverride bool `yaml:"allow_nccl_override"`
}
type Experiment ¶
type Experiment struct {
Project string `yaml:"project"`
Name string `yaml:"name"`
Group string `yaml:"group"`
// Title is the pre-v0.1 experiment identity spelling. Keep accepting it so
// existing configs migrate safely, but new configs must use Name.
Title string `yaml:"title"`
}
Experiment names where a run belongs in the identity hierarchy:
workspace -> project -> experiment -> run
Name is the experiment: the set of runs being compared. Group is an arm inside that set (baseline vs ablation), not a level of its own, so two groups under one Name are directly comparable.
Name is optional for input compatibility. A pre-v0.1 Title is normalized first, then Group remains the final fallback for configs whose experiment axis used the group. New configs set Name explicitly.
type FieldStatus ¶
type FieldStatus string
type ImageAsset ¶
type Metrics ¶
type Metrics struct {
History []string `yaml:"history"`
Offload MetricsOffload `yaml:"offload"`
}
func (Metrics) Validate ¶
func (m Metrics) Validate(experimentConfig Experiment) error
type MetricsOffload ¶
type MetricsOffload struct {
Enabled bool `yaml:"enabled"`
Runtime string `yaml:"runtime"`
Image string `yaml:"image"`
Out string `yaml:"out"`
DeliveryMode string `yaml:"delivery_mode"`
ADXClusterURI string `yaml:"adx_cluster_uri"`
ADXDatabase string `yaml:"adx_database"`
ADXTable string `yaml:"adx_table"`
ADXMapping string `yaml:"adx_mapping"`
ADXClientID string `yaml:"adx_client_id"`
ADXMaxAttempts int `yaml:"adx_max_attempts"`
ADXRetryBackoff string `yaml:"adx_retry_backoff"`
ADXFinalStatusTimeout string `yaml:"adx_final_status_timeout"`
}
type NormalizedRDMA ¶
type NormalizedRDMA struct {
Enabled bool
}
NormalizedRDMA is the resolved RDMA configuration. ShmSize is deliberately excluded because the RayJob path manages /dev/shm independently.
func NormalizeRDMA ¶
func NormalizeRDMA(cfg RDMA) NormalizedRDMA
NormalizeRDMA returns the shared renderer input. When RDMA is disabled the zero struct is returned.
type Policy ¶
type Policy struct {
Workspace string `yaml:"workspace"`
Namespace string `yaml:"namespace"`
Profile string `yaml:"profile"`
Queue string `yaml:"queue"`
Team string `yaml:"team"`
Lane string `yaml:"lane"`
GPUClass string `yaml:"gpu_class"`
Mode string `yaml:"mode"`
Topology string `yaml:"topology"`
Shape string `yaml:"shape"`
Priority string `yaml:"priority"`
PriorityTier string `yaml:"priority_tier"`
WorkloadProfileSnapshot string `yaml:"workload_profile_snapshot"`
WorkloadPriorityClass string `yaml:"workload_priority_class"`
PodPriorityClass string `yaml:"pod_priority_class"`
NodeSelector map[string]string `yaml:"node_selector"`
ClearNodeSelector bool `yaml:"clear_node_selector"`
DisableDefaultPriorities *bool `yaml:"disable_default_priorities"`
}
type RDMA ¶
type RDMA struct {
Enabled bool `yaml:"enabled"`
}
RDMA opts workload containers into RDMA topology placement and the memlock capabilities NCCL NET/IB needs for verbs memory registration.
type Resilience ¶
type Run ¶
type Run struct {
Name string `yaml:"name"`
Engine string `yaml:"engine"`
Entrypoint string `yaml:"entrypoint"`
Script string `yaml:"script"`
Image string `yaml:"image"`
MainScript string `yaml:"main_script"`
WorkloadKind string `yaml:"workload_kind"`
SmokePairs *int `yaml:"smoke_pairs"`
Source *Source `yaml:"source"`
TTLSecondsAfterFinished *int64 `yaml:"ttl_seconds_after_finished"`
// WorkingDir ships a whole project directory with the run instead of the
// entrypoint alone, so sibling modules and local packages import on
// workers as well as the driver. Empty keeps single-file behaviour.
WorkingDir string `yaml:"working_dir"`
// WorkingDirExcludes are extra glob patterns to leave out of the shipped
// project archive, on top of the built-in defaults.
WorkingDirExcludes []string `yaml:"working_dir_excludes"`
}
type Runtime ¶
type Runtime struct {
Image string `yaml:"image"`
WorkingDir string `yaml:"working_dir"`
Pip []string `yaml:"pip"`
Env map[string]string `yaml:"env"`
EnvSecret map[string]string `yaml:"env_secret"`
EnvKV map[string]string `yaml:"env_kv"`
Security Security `yaml:"security"`
RDMA RDMA `yaml:"rdma"`
}
func (Runtime) ValidateEnvKV ¶
func (Runtime) ValidateEnvSecrets ¶
func (Runtime) ValidateReservedEnvKeys ¶
type Security ¶
type Security struct {
Mode string `yaml:"mode"`
}
Security is the portable pod-security contract for direct workloads.
type Source ¶
Source stages an immutable source tree from a digest-pinned OCI image for a direct Job. Path is copied into SourceMountPath by an init container. The source image must provide /bin/sh, cp, and chmod.
func (*Source) ValidateEntrypoint ¶
type Storage ¶
type Storage struct {
DataPVC string `yaml:"data_pvc"`
ResultPVC string `yaml:"result_pvc"`
Output string `yaml:"output"`
Publish string `yaml:"publish"`
Volumes []string `yaml:"volumes"`
Mounts []string `yaml:"mounts"`
ImageAssets []ImageAsset `yaml:"image_assets"`
// Checkpoint names the file or directory, relative to the run's
// checkpoint directory, that this run produces as its servable model
// (e.g. "last.safetensors"). Declaring it lets tau index the artifact
// after a successful run so `tau serve deploy --from-finetune` and
// `tau data model` can resolve it by run name instead of requiring the
// researcher to pass an absolute --checkpoint path. Optional; when
// empty, no artifact index is written and nothing else changes.
Checkpoint string `yaml:"checkpoint"`
}
func (Storage) ValidateCheckpoint ¶
ValidateCheckpoint enforces that storage.checkpoint stays inside the run's own artifact directory. The finalizer joins this value onto <durable>/finetunes/<run>/artifacts/, and that PVC is shared by every run in the namespace, so an unchecked value is a write primitive into someone else's run: "../../victim/model.bin" resolves to <durable>/finetunes/victim/model.bin. An absolute value is worse than it looks, because pathlib discards every component before it — joining "/etc/passwd" yields "/etc/passwd", not a path under the run directory.
These are the same rules the manifest path already enforces in validateCheckpointArtifact; this field reached the same finalizer without them.
func (Storage) ValidateImageAssets ¶
type UnknownKey ¶
UnknownKey is one unrecognized key plus the child keys nested under it, which are used to infer what the author meant when the key is a container.
func UnknownKeys ¶
func UnknownKeys(raw []byte) ([]UnknownKey, error)
UnknownKeys returns the dotted paths in a managed workflow manifest that neither Config nor the managed manifest schema recognizes.
type Workflow ¶
type Workflow struct {
File string `yaml:"file"`
Script string `yaml:"script"`
MainScript string `yaml:"main_script"`
WorkloadKind string `yaml:"workload_kind"`
ExtraScripts []string `yaml:"extra_scripts"`
UpstreamCheckpoint string `yaml:"upstream_checkpoint"`
SecretPayload string `yaml:"secret_payload"`
SmokePairs *int `yaml:"smoke_pairs"`
}