ml

Overview ¶

Package ml is model serving and training: deploy a model behind an endpoint, run a training job, tune it with experiments.

The /v1/ml/* and /v1/train/* routes are a thin, tenant-scoped bridge that turns three Kubeflow-family CustomResources into a small REST API. No ML logic is reimplemented here — the operators (kserve, trainer, katib) own reconciliation; this subsystem only translates REST <-> the Kubernetes API and enforces tenant isolation.

Three resources, one CRUD shape each (kserve names are internal/opaque — the user-facing model catalog lives in the hub, never here, so no upstream model identity is ever introduced by this layer):

/v1/ml/models           InferenceService  serving.kserve.io/v1beta1
/v1/train/jobs          TrainJob          trainer.kubeflow.org/v1alpha1
/v1/train/experiments   Experiment        kubeflow.org/v1beta1   (katib)

Plus two leaf surfaces: POST /v1/ml/models/{name}/predict proxies the request body to the model's kserve v2 data plane (/v2/models/{name}/infer at the InferenceService's cluster-internal address), and GET /v1/train/experiments/{name}/trials lists the katib Trials owned by an experiment.

Tenancy: every request is scoped to the gateway-minted org (X-Org-Id / c.Org()) narrowed by the org SUB-SCOPE (X-Project-Id / principal.Project), and lands in a PER-ORG(+PROJECT) Kubernetes namespace: "ml-"<org> for the default project (the backward-compatible single-project shape) and "ml-"<org>"-"<project> for a non-default one. The namespace IS the tenant boundary — a tenant physically cannot name into, list, read, mutate or predict against another org's (or project's) resources because the dynamic client is always pinned to the caller's namespace. Both org and project are validated against strict DNS-label regexes (no lossy sanitize), so the (org, project)->namespace map is injective: two distinct scopes can never fold onto one namespace. Empty org is rejected 403 unless the caller is a gateway-minted admin (bucketed under the literal "ml-admin" namespace).

k8s client: built in-process from the in-cluster service account (rest.InClusterConfig) with a KUBECONFIG fallback for local/dev. It is NOT hung off the shared cloud.Deps: a raw Kubernetes client has none of the in-process/ZAP-RPC duality the Deps inter-subsystem clients model, and it is used by exactly this one subsystem — so it stays self-contained here, the same way provisioning builds its own backend clients. When no kubeconfig is resolvable the subsystem mounts anyway and every endpoint fails closed: mutating routes return 503 and the health routes report status "degraded" with the real init error (never a fake success).