platform

Overview ¶

applylive.go — the ONE version-monotonic deploy mechanic shared by the image-source path (deployImage) and the git build reconciler.

Both paths must write the operator Service CR (applyService) AND advance the app's live pointer (FinalizeLive). FinalizeLive is already an atomic, monotonic CAS so the recorded live version can never regress. The remaining gap (RED LOW-1) is the CR write itself: two concurrent deploys of the SAME app could interleave so that an OLDER deploy's applyService lands AFTER a NEWER one already went live, leaving the live Service CR image lagging the recorded live version. applyLive closes that gap by running supersede-check → applyService → FinalizeLive as one per-app-serialized critical section: an older deploy that loses the race is superseded and never writes its (older) CR.

artifact.go — the ARTIFACT lane of POST /v1/runner: build a project and publish what it produced, when what it produced is not a container image.

/v1/runner could build exactly one shape — a Dockerfile → an image pushed to a registry (runner.go → launchDirectBuild). Everything else a repo can produce — a Go binary, a Rust binary, an npm tarball, a wheel — had no in-cluster build path at all, so the platform's answer to "build my project" was "write a Dockerfile". The declaration for those artifacts ALREADY EXISTED: hanzo.yml's `binaries:` + `bucket:` blocks, which hanzoai/ci's reusable workflow has read since it was written. Only the GitHub lane implemented them. This is the platform-native implementation of the SAME contract — one recipe, two front doors, exactly as `images:` is already built by buildx on a runner and by BuildKit in-cluster.

Shape of one build (a k8s Job on the isolated build namespace + CI pool):

initContainers  one per `binaries:` entry, in the toolchain image that entry
                names, running the recipe against a shallow clone pinned to
                the requested ref. They run SEQUENTIALLY (k8s guarantees it)
                and share /w — so a repo whose Go binary and TS package need
                different toolchains is still ONE build with ONE index.
container       the publisher: hashes every file the recipe left in /w/dist,
                PUTs it to hanzoai/s3, and writes binaries.json last.

The split is the security boundary, not a packaging detail. `run:` is arbitrary shell BY DESIGN (it is a build command, the same trust as a Dockerfile RUN), so it must never see a credential: the initContainers carry no object-store env and no service-account token, and the publisher — which holds the S3 credential — runs only artifactPublishScript, a constant.

Every recipe value reaches the scripts as an ENVIRONMENT VARIABLE expanded in double quotes, never as interpolated script text, so nothing a caller sends can rewrite the script itself.

console.go — the flat, top-level console aggregates that the Hanzo Cloud Console's Environments / Pipelines / Builds / Releases pages render. They are NOT a new data model: every row is DERIVED from the SAME per-org project / application / deployment / build records the /v1/platform surface already owns (platform.go, deploy.go, store.go). One store, one deploy mechanic, four read-only projections:

• environment → a deploy target: the distinct Application.Environment values across the org's apps, each aggregating the apps that target it.
• pipeline → an app's build/deploy configuration + its latest run (one pipeline per application).
• build → a REAL arcd BuildKit build record (platform_builds); the git build step of a deploy. Never fabricated — real records or an honest empty.
• release → a deployment that was actually applied to the cluster (status deploying|live): a released image tag on an app/environment.

All four are GET-only. A pipeline/build/release is CREATED only through the ONE existing write path — POST /v1/platform/projects/{p}/apps (create app) and .../deploy (trigger a build+deploy) — so there is exactly one way to make them, never a duplicate trigger here. Environments are a scope derived from apps, not a standalone record, so there is nothing to create/delete independently either.

Every handler is org-scoped through the SAME validated-principal gate as the rest of the platform surface (s.tenant → requires c.User(); org is the only tenancy key). The response is a `{ "<plural>": [...] }` object — the exact shape the console FE normalizers read (r.environments / r.pipelines / r.builds / r.releases).

deploy.go — the deploy lifecycle for /v1/platform: build (arcd BuildKit) + deploy (operator Service CR) + start/stop + deployment history/logs.

Two source kinds, ONE deploy mechanic (write the operator Service CR into tenant-<org>; the operator reconciles):

• source=image — no build. The CR is applied immediately with the requested image tag; the operator rolls it. This is the fully end-to-end path.
• source=git — a build is required. An in-cluster BuildKit Job is launched (client-go, the arcd model) to build the repo and push the per-tenant image; the deployment lands "building". The build watcher that flips "building"→"live" by applying the CR with the built image is phase 2 (platform-in-cloud §4) — until then the git path stops honestly at "building" with the real Job reference, never a fabricated "live".

Every handler is org-scoped (s.tenant) and every cluster write targets tenant-<org> derived from the validated org — never a request value.

domains.go — customer-self-serve domain management for /v1/platform apps: the default host, org-subtree hosts, and VERIFIED BYO custom domains (`yourco.com`), all rendered into the app's operator Service CR ingress so the operator materializes the Ingress + cert-manager TLS. Two host classes, one render source (the app's DomainsJSON):

• default / org-subtree host (`*.<org>.<sitesHost>`) — STRUCTURAL: the org owns its whole subtree by construction, so these are active the moment they are added; no ownership proof is possible or needed.
• BYO custom host (arbitrary `yourco.com`) — needs an OWNERSHIP proof before it can be rendered, or a tenant could claim a host it does not control (the RED hijack boundary). The customer publishes a per-claim TXT token at `_hanzo-challenge.<host>`; only someone who controls that zone's DNS can, so a matching token proves control (the DNS-01 model). Until verified, the claim is a pending row that is NEVER in the ingress. Global uniqueness (one org per host) is the `platform_domains.host` PRIMARY KEY.

Every handler is org-scoped through s.tenant and mutates only tenant-<org>.

drift.go — the drift verdict the fleet board attaches to every App CR row: the "one and only one way to deploy" made native to the cloud binary. It is the Go port of the standalone Dokploy-based platform's build→deploy lifecycle (pkg/platform/src/services/ci/deploy-executor.ts + services/apps/inventory.ts + db/schema/apps-drift.ts), collapsed into an in-process cloud subsystem exactly like clients/ml is the k8s bridge for the Kubeflow CRDs.

The deploy mechanism is the SAME one the operator already reconciles: a merge-patch of the operator `Service` CR's `.spec.image`. No second deployer is invented; the Hanzo operator owns the rollout. This module only observes the declared/running/latest tags per service (the drift board) and flips the one CR field a deploy changes — the identical contract the Node deploy-executor implemented, now native.

drift.go is the PURE half: it derives the drift verdict for one observed service row and performs no IO. It is a faithful port of `pkg/platform/src/db/schema/apps-drift.ts` so the two implementations can never disagree about what "drift" means (one way to compute drift, period). The cluster reader (fleet.go) owns observing the tags; this file only interprets them.

fleet.go — the platform's view of its OWN service tier.

It mounts /v1/platform/fleet on the unified cloud binary and reads the operator's `hanzo.ai/v1` `App` CustomResource — the one workload kind the fleet runs on:

GET  /v1/platform/fleet            — the fleet drift board: list every operator
                                     App CR across the platform namespaces, read
                                     declared vs running tag + health from the CR
                                     (+ its status), and attach the drift verdict
                                     (drift.go).
GET  /v1/platform/fleet/:app       — one app row by CR name.
POST /v1/platform/fleet/:app/deploy— zero-downtime ROLLING RESTART of the app's
                                     Deployment (the `kubectl rollout restart`
                                     mechanism): re-pulls the DECLARED image,
                                     recreates pods gracefully. It never changes the
                                     declared TAG (that stays a git commit, the one
                                     thing Hanzo CD's selfHeal reconciles), so there
                                     is no drift to revert. This is `hanzo deploy`.

FLEET vs PROJECTS/:project/APPS — two collections, two names. `fleet` is the PLATFORM's own tier (the shared services the platform itself runs on: iam, kms, gateway, …), observed from the operator App CRs. `projects/:project/apps` is a CUSTOMER's apps, owned by platform's own store. They answer different questions, so they carry different names under the one /v1/platform prefix. This file used to be a second top-level product (`/v1/paas`) — the same platform under a second name, which is exactly the duplicate definition the one-way rule forbids.

SECURITY — every route is authorized off ONE IAM identity, exactly like the /v1/runner build path (runner.go): the guard admits a validated principal (principal.Validated) who is a SuperAdmin OR an OrgAdmin, and each handler then CONFINES a non-super caller to its own org's platform namespaces (scopedNamespaces, keyed on principal.Org — never a client header). A SuperAdmin observes/acts on the whole fleet; an OrgAdmin only on the namespaces its org owns; a plain member or an unauthenticated caller is refused 403. So a tenant admin can never observe — or restart — another org's, or a platform, app, and the platform operator drives the board off a plain `hanzo login` with no shared token.

k8s client: built in-process from the in-cluster service account (rest.InClusterConfig) with a KUBECONFIG fallback for local/dev — the identical construction clients/ml uses. When no kubeconfig is resolvable the board mounts anyway and every endpoint fails closed (503 + the real init error; the shared /v1/platform/health route reports "degraded"), never status-theater.

k8s.go — the cluster-facing half of /v1/platform: the ONE deploy path.

A user application is deployed by writing an operator hanzo.ai/v1 `App` CR into the caller's OWN tenant namespace; the Hanzo operator reconciles it into a Deployment + Service + Ingress (+ HPA/PDB) on DOKS. cloud never reimplements a deployer — it writes one CR, exactly like the fleet board reads system CRs, but here every object lives in `tenant-<org>` where the org is the gateway-minted, IAM-VALIDATED tenant (c.Org()), never a value from the request body or path. That derivation is the whole cross-tenant isolation boundary: a caller cannot name another org's namespace because the namespace is not an input.

Builds (git-source apps) launch an in-cluster BuildKit Job (the arcd model, buildkit-job.ts) via client-go — no GitHub builders. When the cluster / CI prerequisites are absent the subsystem fails CLOSED with the real reason (never status-theater), matching the fleet board.

logs.go — REAL deployment logs for /v1/platform (the deploy.go phase-2 gap).

deploymentLogs previously returned only the recorded status timeline + a Job reference, ending with a "(live BuildKit Job logs stream in phase 2)" note. This closes that gap: it streams the ACTUAL pod logs from the cluster — the BuildKit Job's pod for a git build, and the running app's pod for a deployed app — via the typed CoreV1 client's Pods().GetLogs subresource. Everything stays operator- consistent and org-scoped: the app's pods are read from tenant-<org> (the same isolation namespace the Service CR lives in), the build pod from the central build namespace by the deterministic job-name label. It NEVER fabricates output — an unreachable cluster or an absent pod yields the honest recorded timeline instead, exactly as the rest of the subsystem degrades.

orphans.go — reaping apps whose project is gone.

A project belongs to IAM and is deleted there. Platform's apps hang off the project NAME, so a delete on the IAM side leaves this side holding apps under a project that no longer exists: unreachable through the API (requireProject 404s) but still deployed, still holding a Service CR, a KMSSecret and a volume, and still costing the tenant money.

The obvious fix is to cascade from platform's own DELETE route, and that is what this replaces. It only ever worked when the project happened to be deleted THROUGH platform — which is the wrong condition, since the project is not platform's to delete. Reaping on the app side instead makes the outcome independent of who did the deleting and where.

Idempotent and org-scoped like the build reconciler: every write targets tenant-<row.Org>, derived from the row and never from a request. Fail-safe on read: a project store that cannot answer reaps NOTHING, because "IAM is unavailable" and "the project is gone" must never be the same signal.

Package platform is the Hanzo Cloud PaaS control plane: the per-org, user-facing Platform-as-a-Service, mounted natively in the unified cloud binary at /v1/platform (HIP-0106). It is the Go port of the standalone Dokploy (platform.hanzo.ai) tRPC backend — the culmination of "one binary ships all of Hanzo Cloud."

Relationship to the sibling subsystems:

• fleet.go (/v1/platform/fleet) — the ADMIN fleet drift board: observes + deploys SYSTEM Service CRs across the platform namespaces, SuperAdmin only. It answers "what is the fleet running, and roll a tag."
• clients/projects (/v1/projects) — per-org STATIC sites (S3 hosting).
• clients/platform (/v1/platform) — THIS: per-org CONTAINER apps. Users create projects + applications, build them (arcd BuildKit) and deploy them (operator hanzo.ai/v1 Service CR into their OWN tenant-<org> namespace).

All three share the ONE deploy mechanic — write an operator CR, let the operator reconcile — but /v1/platform is per-tenant: every route is scoped to the gateway-minted, IAM-VALIDATED X-Org-Id (c.Org()); the deploy namespace is DERIVED from that org (tenant-<org>), never taken from the request. A tenant can never read, build, or deploy into another org's namespace. That is the red-team bar and it is structural: cross-tenant identifiers are simply not inputs to any handler.

The API is designed-first in Goa (clients/platform/design; `goa gen` emits the OpenAPI 3 contract at clients/platform/design/gen/http/openapi3.*). The runtime handlers below implement that contract natively on zip so the binary keeps ONE router and stays behind the SanitizeIdentity trust boundary.

preview.go — the Vercel-defining release flows for /v1/platform, all built on the ONE deploy mechanic (deploy.go's deployTagCore → applyLive; write the operator Service CR, the operator reconciles). Nothing here re-implements a deployer or a CR writer:

• preview — deploy an already-built image to a PER-BRANCH target that is a first-class Application of its own (slug "<app>-<branch>", its own default host "<app>-<branch>.<org>.<sitesHost>"), isolated from prod by a distinct CR name + host in the SAME tenant-<org> namespace. Returns the preview URL.
• promote — set the PROD app's image to an already-built tag or a prior deployment's exact image (forward: pick an artifact, make it prod).
• rollback — redeploy a prior deployment's image (backward: the previous release, resolved from the deployments store).

Every handler is org-scoped through s.tenant and every cluster write targets tenant-<org> derived from the VALIDATED org — never a request value.

projects.go — the project LIFECYCLE port. Projects are owned by Hanzo IAM (hanzo.id), the ONE source of truth for the org-scoped (owner,name) project resource. Platform REFERENCES that store; it never persists a project row of its own. Applications still live under a project (the platform_apps.project_id column is the project NAME), but create/list/get/delete/exists of the bare project delegate here.

IAM is embedded in the SAME cloud binary (clients/iam mounts the clean iam-v2 zip-natively and opens its orm.DB), so the reference is an IN-PROCESS call into the clean iam's project store (github.com/hanzoai/iam/pkg/store over clients/iam.DB()) — no HTTP hop to /v1/iam, and IAM's canonical model.Project is used verbatim, never cloned into a platform-local struct. This couples platform to the embedded IAM runtime: a cloud deployment that enables "platform" MUST also enable "iam" (both are single-binary co-residents by design), else clients/iam.DB() is nil and a project call fails closed (503). The retired iam-v1 object store is GONE.

projects_canonical.go — the ProjectStore over the CANONICAL IAM.

Split-horizon deployments run IAM as its own service (IAM_URL, e.g. http://iam.hanzo.svc) — the store that mints every token and serves /v1/iam/projects at the edge. Platform used to read projects from cloud's EMBEDDED copy of the iam store instead: a second database for the same noun, so a project created at /v1/iam was invisible to the PaaS and vice versa.

This client reads the canonical store over HTTP, authenticated AS THE ORG: each read presents client_secret_basic for that org's own "<org>-platform-kms" identity — the same credential the KMS sync uses, minted on first need by kmsOrgIdentity — and IAM's authorize admits exactly that identity, read-only, own-org-only (iam internal/authz). One identity per tenant, one grant, both stated once.

When IAM_URL is absent the process IS the IAM (single-binary: the embedded subsystem serves /v1/iam), so the in-process store remains the canonical one and iamProjects is used unchanged. The selector is newProjectStore.

push.go — git-push-to-deploy: turn a push landed on the embedded git server (clients/git) into a build for every app that tracks that repo+branch.

Wiring is inverted so git never imports platform: platform registers buildFromPush as the cloud.PushBuilder in Mount; clients/git calls cloud.OnGitPush after a push lands, which dispatches here. Best-effort by contract — a build-trigger failure never fails the push the client committed.

reconcile.go — the git build→deploy handoff (formerly deploy.go's "phase 2").

deployGit launches an in-cluster BuildKit Job and lands the deployment "building"; it does NOT block the HTTP request waiting for the build. This reconciler is the ONE owner of what happens next: it periodically scans every deployment still "building", checks its Job, and on success applies the operator Service CR with the built image — the SAME applyService the image-source path uses — flipping build→succeeded, deployment→deploying, app→live. The operator then reconciles the rollout.

Why a reconciler and not a per-deploy goroutine: state lives in the store, not in memory, so a cloud restart mid-build RESUMES cleanly (the next tick re-reads "building" rows). It is idempotent (applyService is create-or-update; a row is advanced off "building" once handled) and org-scoped (every write targets tenant-<row.Org>, derived from the row, never a request value).

release.go — native release semantics on the /v1/runner build path.

This is the in-cloud port of .github/workflows/release.yml: cloud self-publishes ghcr.io/hanzoai/cloud with the SAME invariant the workflow exists to enforce —

a git tag v<X.Y.Z> exists  ⇔  an image ghcr.io/hanzoai/cloud:v<X.Y.Z> was
pushed AND booted to "listening" in the smoke test.

The tag is a RECEIPT for a proven image, minted only AFTER a successful push + smoke — never a trigger for a build that might fail. The order is inverted from the old tag-triggers-build design (which left phantom tags with no image behind them → ImagePullBackOff): main push → compute version → build → smoke → tag → notify, so any failure fails BEFORE the tag and leaves no receipt.

The whole pipeline is four injectable seams (releasePlan) run in strict order (run), so the ordering invariant is enforced by construction and unit-tested hermetically, while each concrete step (a k8s Job for build/smoke, a GitHub API call for tag/notify) is wired once in releaseFor.

release.go — the first-party release seam. build.go's RegisterServiceReleaser is the inversion that lets a build-completion path (clients/platform/release.go, or any package-cloud caller) request a rollout with no cloud⇄platform import cycle.

The App CRs in the platform namespaces are declared in universe git (infra/k8s/operator/crs/) and reconciled by Hanzo CD with selfHeal, so a direct spec.image patch is reverted on the next sync. releaseService therefore refuses and names the one way to roll a tag: commit it to the manifest. The clean-semver gate (splitReleaseImage) still validates the request so a caller gets an honest, specific error.

run.go — POST /v1/run, the container-serverless one-shot.

It is the single-call shortcut over the project → app → deploy flow: given an image (and optional port / scale bounds / env), it create-or-updates an image-source Application in the org's default project and writes the operator hanzo.ai/v1 Service CR through the ONE shared writer (k8sClient.applyService / serviceCR), so a run is a first-class Application — listable, stoppable and redeployable via the /v1/platform routes — and re-running the same name UPDATES it in place (idempotent). There is NO parallel Service-CR writer here: this handler reuses deploy.go's machinery (s.tenant, the store, sealSecretEnv, seedDefaultDomain, applyService) end to end.

Every cluster write targets tenant-<org> derived from the VALIDATED org (s.tenant → provisioning.SanitizeOrg), never a request value — the same cross-tenant isolation boundary as the rest of platform.

runner.go — POST /v1/runner: the native, privileged build endpoint.

This is the no-GitHub-builders build trigger that `hanzo build`, the git-push-to-deploy hook, and cloud's own self-release all call. It replaces the old /v1/arcd surface: one native build API on the runner fabric.

It differs from the tenant path (/v1/platform/.../deploy, which FORCES a per-tenant image ref): a /v1/runner build is PRIVILEGED — the caller supplies the output image — so it is gated two ways:

• a shared build-callback token (constant-time), and
• an image-ref allowlist restricted to the org registries we own,

so a leaked token can never push to an arbitrary registry.

secrets.go — KMS-sealed secret env vars for /v1/platform apps (closes the createApp/setEnv 501). A user env var flagged `secret:true` is NEVER stored in platform.db and NEVER rendered inline into the operator Service CR. Instead the ONE secret invariant holds end-to-end: a secret value lives ONLY in KMS and the operator-materialized k8s Secret; cloud's control-plane DB holds a masked placeholder and cloud never handles the plaintext at deploy time.

The path a secret takes:

1. SEAL — cloud seals the value into its embedded KMS (deps.KMS, AES-256-GCM envelope, clients/kms) at the ORG-SCOPED coordinate /orgs/<org>/platform/ <app>/<KEY> (kmsSecretRef). Plaintext is sealed before it touches disk, never logged, never echoed back over the API (toAppView masks it). If KMS is unavailable the write FAILS CLOSED — plaintext never lands in the DB.
2. PROVISION — on app-create/deploy, cloud ensures the tenant's PER-TENANT KMS credential exists (ensureTenantKMSAuth): a machine identity whose token carries owner=<org>, projected into tenant-<org> as the creds Secret the CR references. It is the ONE privileged step; when unavailable it degrades to an honest "pending" (never a shared reader), so no cross-tenant identity is ever created.
3. DECLARE — cloud writes a canonical KMSSecret CR (secrets.lux.network/ v1alpha1) into tenant-<org> pointing the operator at /v1/kms/orgs/<org>/ secrets/platform/<app> (projectSlug=<org>, explicit keys). The operator logs in with the per-tenant creds (/v1/kms/auth/login → IAM), reads each key back from cloud's KMS, and materializes the managed Secret `<app>-env`. Cloud's org-scope guard admits the read ONLY for owner==<org>, so a tenant can read ONLY its own scope. Best-effort: a missing CRD/RBAC/credential degrades to a "pending" status, never a failed deploy.
4. MOUNT — the Service CR renders each secret env as `valueFrom.secretKeyRef` → that Secret (optional, so the pod boots even before the sync lands); the hanzo operator mounts it into the Deployment env. The pod reads the secret from the k8s Secret — cloud is never in the plaintext path at runtime.

Everything is org-scoped: the KMS coordinate, the auth identity, the KMSSecret CR, and the managed Secret all live under the VALIDATED tenant, never a request value — the same cross-tenant boundary as the rest of /v1/platform, enforced at cloud's ONE auth boundary (SanitizeIdentity + the kms org-scope guard).

validate.go — the boundary validators + resource-bound policy for /v1/platform.

Three concerns, one home (DRY), because all three feed the SAME privileged cluster path (a BuildKit Job + operator Service CR in tenant-<org>):

1. build-input validation — repo.url / dockerfile / git-ref are the ONLY free-form strings that flow into the in-cluster build. They are validated here so they can never be misread as a separate buildctl argument, a shell token, a path escape, or a git-context fragment break. Combined with the exec-form argv in launchBuildJob (no `sh -c`), this closes OS command injection AND the `--output`/`--opt` override class (CRIT-1).
2. replica bounds — clampReplicas caps a tenant's requested replica count so a single app cannot request an unbounded Deployment (part of MED-3).
3. namespace resource bounds — a ResourceQuota + LimitRange are applied to every tenant namespace so a tenant's TOTAL footprint is capped in the cluster regardless of what slipped past the per-object clamp (defense in depth for MED-3). Concurrent build fan-out is capped in k8s.go using resourceLimits.maxConcurrentBuilds below.

Every bound is a hard-coded safe default, operator-overridable via env — the values are resolved ONCE at k8sClient construction so a request path never re-reads the environment and the tests are deterministic.