openstack

pkg/providers/internal/openstack

Intention

pkg/providers/internal/openstack is the real cloud provider implementation for OpenStack-backed regions.

It does not merely wrap the OpenStack SDK. It is the package that translates the region service's storage model, tenancy model, metadata conventions, and lifecycle rules into concrete OpenStack operations across identity, compute, image, network, security group, load balancer, quota, and block storage surfaces.

The most important philosophy in this package is the trust and scoping model:

• region-level manager authority is used only to provision and manage users and projects inside a managed Keystone domain
• once that scaffolding exists, most OpenStack operations deliberately context switch into the specific project provisioned for one Unikorn identity
• that OpenStack project then becomes the practical mapping, isolation, and accounting boundary between Unikorn resources and real cloud resources

That model is what makes it realistic for multiple regions or deployments to share one underlying cloud while still limiting blast radius. It is also what makes backchannel accounting and billing integration plausible, because the project scope becomes the place where cloud resource usage can be tied back to a specific Unikorn identity and its descendants.

The most important architectural rule is that this package prefers deterministic lookup against OpenStack over maintaining broad mirrored OpenStack state in Kubernetes. In older designs, dedicated Openstack* CRDs were used to persist more provider-side state locally. That drifted from reality and introduced race conditions. The current direction is:

• service-native CRDs remain the primary control objects
• OpenStack remains the source of truth for cloud-side resources where they can be re-found deterministically
• only the provider state that still cannot be reconstructed safely or sufficiently remains persisted locally, most notably OpenstackIdentity

This package therefore owns the mapping between:

• region-native CRDs and OpenStack resources
• service labels, tags, and metadata conventions
• per-identity delegated cloud credentials
• compensating local mechanisms where OpenStack is not sufficient on its own, such as image caching and provider-network VLAN allocation

Links

• ../../../apis/unikorn/v1alpha1
• ../../types
• ../../allocation/vlan
• ./ADMIN.md

pkg/apis/unikorn/v1alpha1 defines the service-native resources and the remaining persisted provider-state records this package consumes. pkg/providers/types defines the provider-neutral contract this package implements. pkg/providers/allocation/vlan covers the local VLAN allocator used when provider networks need segmentation IDs. ADMIN.md keeps the human operator setup guidance for preparing an OpenStack region.

OpenStack Region Registration

An OpenStack-backed region should be registered with hack/openstack/configure and hack/openstack/register-region, rather than by hand-creating the Keystone domain, project, user, Kubernetes Secret, and Region manifest. That flow keeps operator setup aligned with this package's scoping model: region-level provider credentials are used to manage provider-domain scaffolding and discover region inventory, while workload operations still context-switch into per-identity OpenStack projects.

The full operator procedure lives in ./ADMIN.md.

Invariants And Guard Rails

• This package implements the full types.Provider contract for OpenStack regions.

• Provider construction has an explicit bootstrap/runtime split:

  • bootstrap uses uncached Kubernetes reads to assemble OpenStack service clients before controller-manager caches exist
  • runtime operation switches back to the normal Kubernetes client and refreshes derived OpenStack client state when region configuration or credentials change

• OpenStack access is intentionally scoped through different credential modes:

  • region-level service credentials bootstrap privileged service clients and managed-domain scaffolding
  • per-identity credentials are used for most project-scoped operations
  • some operations deliberately bind privileged credentials to a service principal's project when manager-level powers are required
  • pinned server creation is one of those privileged project-scoped operations: it still targets the identity project, but uses region-level credentials so Nova policy can authorise the requested destination
  • pinned server creation can also enable a transient openstack.compute.placementPreflight check. When enabled, the provider asks OpenStack Placement whether the pinned resource provider has available inventory for the flavor's positive custom resources:* extra spec and any trait:* extra specs, unioned with configured requiredTraits. Required flavor traits are sent as positive Placement required entries; forbidden flavor traits are sent as !TRAIT entries. Empty trait inputs mean no trait filter. A miss yields and lets the controller retry.
  • a miss splits the same way the image rows do: the host, resource class and required traits go to the log, and the condition gets wording we choose. None of that detail is on the API — the resource class and flavor traits are Nova extra specs and requiredTraits is region config — so it would be provider vocabulary on a surface the caller cannot act on. The yield is typed DependencyNotReady, because a pinned host is a dependency and is not the user's own server.

• SSH injection is a create-time server decision. OpenStack receives the identity key name only for the resolved identityKeypair mode; ca and none omit Nova key_name. Image rebuild omits both key_name and user_data, so Nova preserves the stored keypair and create-time user data (including the managed SSH-CA cloud-init baked in at create) and rebuilt guests stay create-equivalent. Updated user data therefore takes effect on server replacement, not rebuild — Nova accepts user_data on rebuild from microversion 2.57, deferred until gophercloud's RebuildOpts carries the field.

• A desired server image change is reconciled with Nova rebuild only once the server has booted at least once, decided from Nova's launched_at (OS-SRV-USG:launched_at) read fresh on the same GetServer — never from the monitor-stamped status.launchedAt/status.provisionedAt latches, which are observations and must not authorize the gate (observation is stimulus, never authorization; keying off them silently dropped image changes whenever the monitor had not yet recorded the first ACTIVE, and the clean-completing reconcile then let the API misreport the change as settled). Before first boot the desired image is a create parameter, not a rebuild target, so a pending image change on a server Nova reports with a zero launched_at defers: the reconcile yields, leaving the resource visibly provisioning and re-checking every 10s until first boot, after which a pass submits the rebuild once the server is quiescent. A never-booted server Nova reports in ERROR is likewise deferred here — the reconcile pass yields without writing a health stamp (the monitor owns observed state), though the yield itself is named on the Available condition — and absorbed by the bounded provider-create delete-and-retry flow, which recreates it from the already-updated spec image. That retry adoption keys off the Healthy=Errored stamp the monitor's poll writes, so it takes effect at worst one poll later. This assumes the cloud exposes OS-SRV-USG:launched_at — the same signal the health monitor mirrors and the create-retry guard keys off. Deliberately not a goal: recreating a never-booted server rather than waiting for it to boot (e.g. a queued baremetal deploy, where recreate would skip a wasted provision) is a future optimization, rejected for now to avoid a third delete/recreate site and the known name-collision race.

• reconcileServerImage decides entirely from the fresh GetServer on the same pass.

  Nova commits the image ref and task_state together. nova/compute/api.py sets task_state = REBUILDING and image_ref = image_href on one object and saves them with expected_task_state=[None] — a database compare-and-swap — before _record_action_start and before the RPC cast to the compute node, all inside the synchronous request. Three consequences the pass rests on: there is no interleaving in which a reader sees the new ref without an active rebuild task; the state is durable before the client receives its 202; and two concurrent rebuilds cannot both commit, the loser raising UnexpectedTaskStateError. Nova's own state check (check_instance_state, whose default requires task_state NULL) refuses a rebuild against a rebuilding server with a 409 before that. All of this is decided in Nova's API layer above the driver, so it holds for Ironic exactly as for libvirt.

  The pass order, deciding only from the fresh read:

  	Condition	Action
  R1	Spec.Image == nil	park
  R2	image ref unreadable	park
  R3	ref == desired, rebuild task active	yield
  R3′	ref == desired, ERROR, launched	park
  R3″	ref == desired, otherwise	done
  R4	ref != desired, launched_at zero	yield
  R4′	ref != desired, any task active	yield
  R4″	ref != desired, quiescent	submit

  Every yielding row returns provisioners.Yield with a Provisioning reason and a message naming the wait, so the row a server is parked on is readable from its Available condition. That matters because the rows are not equivalent — R3 is our own rebuild, R4 is a server that has never booted, R4′ is someone else's operation — and a bare ErrYield renders them identical: the manager only lifts a reason and message off a typed provisioners.Error, so a bare sentinel leaves the generic default with no message at all. Provider vocabulary (Nova status, task_state) stays in the log line; the condition gets only user-safe wording.

  The reason stays Provisioning here, and only the message varies. reason is API surface — it projects to provisioningStatusDetail.reason and callers script against it — so it has to describe the wait accurately. None of these rows is a DependencyNotReady: that means a dependency exists but is not yet provisioned, whereas every row above is waiting on the user's own server to finish something. Naming them DependencyNotReady would send a user looking through their networks and security groups for a fault that is not there. The provider-create gate is the one wait in the server path that does use DependencyNotReady, because another service genuinely satisfies it.

  R2 exists because every server this provider creates is image-booted, so an absent, empty or unparseable ref is abnormal: the pass must not report success over an image change it cannot verify. The park is re-derived per pass, so a later readable ref un-parks it without any spec change.

  serverRebuildInFlight tests Nova's own rebuild_states family (rebuilding, rebuild_block_device_mapping, rebuild_spawning — matched by the rebuild prefix, so a substate a newer Nova adds still reads as in flight rather than as settled over a disk being rewritten — plus the REBUILD status projected from the same family), because on a converged ref the question is specifically whether a rebuild is running — an unrelated task such as a user's reboot must not be reported as one. serverTaskActive tests any task at all, because before submitting the question is only whether Nova would accept, and it refuses while any task holds the server.

  R3′ parks a quiesced ERROR on the desired ref as user-action-required (provisioners.UserActionRequired, so provisioningStatus=error with an actionable, cause-neutral message that advertises the spec-edit remedy), mirroring create-retry's exhausted-attempts park. The ref moves at accept, not on a successful write, so this state means the spec image was never realized and must not read as provisioned (INST-1235: measured on kolla 2025.1, a failed rebuild deploy settles exactly here within seconds on both the sushy and fake-controllable fixtures). The park does not claim to attribute the ERROR to the rebuild — an unrelated host failure after a successful rebuild presents identically — but under either cause the spec is unrealized, so error is the honest report on the provisioning axis; the monitor's health axis carries the ambient view on its own cadence. Deferring the report to the monitor instead was measured to lose short-lived failures entirely: the reconciler settles in one 10s yield cycle while the monitor samples at one-minute cadence. Never-launched servers are excluded from the park (an ERROR before first boot is a failed create, owned by the provisioner's bounded retry machinery).

  A parked server has no requeue, so it un-parks on exactly three paths. The user edits the spec — the remedy the message advertises — and the generation-change wake resumes reconciliation, monitor-independent. Or the provider recovers without a spec change, in which case the un-park rides the monitor's status.observed write firing the observed wake: the park is re-derived per pass, so the woken pass walks to R3″ and reads provisioned again (the same measured path as a foreign recovery, below — within one monitor period). Or the provider server disappears entirely — deleted out-of-band — in which case the monitor's Observe finds no row for it and projectServerState's absent path records an observation (errored cleared, generation stamped, image sticky) before surfacing the not-found error — surfaced, not swallowed, because the create-retry provisioner's confirmed-gone gate depends on UpdateServerState returning ErrResourceNotFound. The monitor persists the recorded observation anyway, firing the same observed wake; the reconciler's own fresh read then routes not-found to the create path and recreates the server.

  The provisioning axis reports spec-realization, not attribution. A rebuild-class operation this provider did not submit — nova evacuate is implemented as a rebuild, presenting task_state=rebuilding with the ref already on the spec image — therefore reads exactly as ours when a pass runs. Measured behaviour: an in-flight foreign rebuild does not move the provisioning axis at all (a foreign operation generates no reconciler wake — no spec change, no observed change — so R3 never runs; only the monitor's phase and health axes read Rebuilding/Unknown for one poll cycle), a failed foreign rebuild parks via the observed-errored wake within one monitor period, and a foreign recovery un-parks the same way. A foreign rebuild onto a different image — succeeded or failed — is auto-reverted: the ref flip changes observed.image, the wake fires, and R4″ resubmits toward the spec image (measured: ~20s from the foreign act to the corrective rebuild). The park is therefore scoped precisely to failures on the converged ref, where the spec image itself is implicated; divergent-ref failures are reconverged instead. Operational consequence: operators must change the spec, never act through Nova — a Nova-side image change is undone within seconds. This is deliberate. During any rebuild-shaped operation the server is not serving its spec — a rebuild is rewriting the root disk, while an evacuation on shared storage rebuilds the instance on another host without touching it — so provisioning is honest; after a failed one the spec image is not running. Both reports are true regardless of who initiated the operation, and no destructive row can fire on a foreign operation (R4″ requires a diverged ref, which a foreign rebuild does not present). The park is re-derived per pass, not latched: when a foreign recovery returns the server to ACTIVE on the converged ref, the next pass walks to R3″ and the server reads provisioned again. What this costs is attribution in the messages — a failed evacuation reports the same cause-neutral park message as a failed rebuild — which is the price of having no persisted intent to consult, and why the message names neither.

  R4 defers to the create-retry path rather than duplicating it, and Nova enforces the same precondition itself (must_have_launched).

  R4″ is the single destructive step. The submission (submitServerRebuild) writes a fixed accepted stamp on a 2xx (Active Rebuilding, Healthy Unknown, matching the monitor's REBUILD mapping so the two writers agree rather than churn) and never derives it from the rebuild response body, whose 202 can still describe the pre-destruction server as ACTIVE and would stamp a just-accepted destructive rebuild as running. Acceptance yields rather than completing: Nova now has a destructive operation in flight, and completing would map to Available=Provisioned and report a server whose root disk is being rewritten as settled. A 409 is pre-acceptance — the server is untouched — so it also yields, silently, for a short retry.

  Nova's accept gate refuses an unresolvable or non-active image, an image whose min_ram/min_disk exceeds the flavor, non-bootable image properties, a locked or wrongly-stated instance, and quota — all before writing any state. status.observed structurally cannot carry the rejection, so the body stays in the log and never reaches the API surface. A synchronous HTTP 400 parks the server as user-action-required (a spec edit resumes it) only when the body carries Nova's image-not-found signature ("Cannot find image for rebuild") — the one refusal that provably cannot self-heal. Nova 400s are not homogeneous: a deactivated image (ImageNotActive) is also a 400, but an operator reactivating it bumps no generation and moves no observed field, so parking it would strand the server; it retries instead and converges on the pass after reactivation. Every other 400 — and a reworded or stripped body — likewise surfaces as the pass's error and retries, failing safe toward the retry loop rather than toward a wrong park. An HTTP 409 yields awaiting quiescence, and every other rejection class surfaces as the pass's error and retries on the yield interval.

  The reconciler writes Active/Healthy when it acts and re-asserts the accepted stamp while a rebuild it submitted is still in flight (R3); on every other waiting row it writes nothing and the monitor owns observed state.

• Nova rebuild retains the server UUID, network ports and IP relationships, attached data volumes, flavor, metadata, and placement, but recreates the root disk. It stays on the same compute host; evacuation is a separate operator workflow.

• OpenstackIdentity is the remaining persisted provider-state anchor. It currently stores the secret-bearing user/project/application-credential and bootstrap state needed to operate on behalf of a region Identity.

• The package relies heavily on deterministic naming and metadata conventions to re-find cloud-side resources. This is a convention-heavy contract, not magic:

  • identity-scoped resources use fixed generated names
  • network lookups rely on deterministic names
  • Cinder volumes use volume-{Region Volume UUID} inside the service principal's project; create, delete, observation, attach, and detach rediscover that exact name
  • server metadata is written deliberately as both a control-plane lookup aid and an in-guest linkage surface exposed through the metadata service
  • legacy camelCase server metadata keys remain frozen for backwards compatibility while newer namespaced keys provide the upgrade path

• Cinder Volume create/delete and observation are project-scoped lifecycle operations:

  • the Region Volume controller resolves the full cloud provider and drives its Volume capability after the service-principal Identity is ready
  • the native Region Volume CRD supplies the requested size and VolumeClassID, which becomes the Cinder volume type
  • create lists by the stable generated name and exact-matches the result before submitting a create, so controller retries adopt an existing volume rather than duplicating it
  • a successful rediscovery stamps VolumeStatus.ProvisionedAt once; if a later lookup confirms the backing volume is gone, the provider refuses to recreate it under the same Region Volume ID
  • an accepted Cinder create is partial progress rather than success: the provider yields after submission and on subsequent rediscovery until the volume reports available; all unrecognized non-error states also yield so an unfamiliar provider state cannot be mistaken for convergence
  • a Cinder status beginning with error returns a typed terminal Available=False, Reason=Errored result with the user-safe message provider volume entered an error state; provider IDs, scheduler failures, host details, and other raw Cinder diagnostics remain operator-only
  • user tags are translated with the package's normal namespaced metadata convention; namespaced system linkage keys are written last so user input cannot override identity, organization, project, region, network, or Volume IDs; Volume metadata does not emit the legacy camelCase compatibility keys retained by older resource types
  • delete uses the same rediscovery path, treats a missing Cinder volume as success, yields after Cinder accepts an asynchronous delete until a later rediscovery confirms absence, and treats an absent or not-yet-project-backed OpenstackIdentity as proof that no provider volume could have been created; this lets controller deletion delegate unconditionally before releasing any Identity allocation
  • observation creates an ephemeral Cinder client in the identity's project, exact-matches the stable name volume-{Region Volume UUID} after Cinder's fuzzy name filter, and fails closed on duplicate exact matches
  • observation uses the exact stable name and identity-scoped project lookup to establish the backing resource. Cinder metadata is not revalidated during observation, so mutable provider metadata drift does not block status updates
  • Cinder's whole-GiB capacity becomes a binary resource.Quantity by multiplying by one GiB. Negative capacity or a value that cannot be represented in bytes as an int64 is ErrConsistency, not a neutral observation. Cinder lifecycle values map directly to the neutral Healthy condition: recognized operational states are healthy, deleting and error states are degraded, and empty or unrecognized values are unknown.
  • no exact match returns ErrResourceNotFound, while Cinder request, decoding, and client-construction failures remain errors. A recognized Cinder error lifecycle and an unknown lifecycle are successful observations, not Go errors.
  • UpdateVolumeState mutates the supplied Region Volume with observed size and health without exposing Cinder or Gophercloud types. It returns provider read errors so the monitor can preserve the last observed status. It does not introduce a mirrored provider-state CRD. The Volume controller owns the durable ProvisionedAt marker, create/delete intent, and generic Available provisioning condition. VolumeClass inventory and Nova attach/detach remain separate capabilities

• Flavor export is a hybrid model: OpenStack discovers the flavor inventory, but region configuration can enrich or override user-facing flavor metadata such as architecture, baremetal status, and GPU semantics. Only public flavors with UUID IDs are exported, because Region's FlavorID request and CRD contracts are UUID-backed; configured selectors further restrict that set. Architecture resolves from per-flavor cpu.architecture, then openstack.defaultArchitecture, then the legacy x86_64 fallback for objects that bypass CRD defaulting. The baremetal flag is also operationally meaningful for live lifecycle (Active condition) reporting: a Nova BUILD server with a baremetal flavor is disambiguated through Ironic so the API can distinguish Queued (waiting on hardware) from Building (provider actively deploying).

• VolumeClass configuration follows the same inventory pattern for block storage. Region configuration under openstack.blockStorage.volumeClasses.selector.ids is a strict allowlist: only Cinder volume type IDs explicitly listed there are eligible for export. Missing volumeClasses configuration, a missing selector, or nil/empty IDs exports no VolumeClasses. Selected classes can be enriched with user-facing metadata such as optional minimum/maximum capacity bounds, a supportedFlavors selector, media, maximum performance caps, and encryption signals. Omitted selectors and omitted or empty selector IDs make the class ineligible for Server attachment; the provider resolves this operator-authored selector to the neutral Flavor ID allowlist and does not infer it from Cinder volume types. Capacity bounds are operator-authored positive whole GiB values; either may be omitted, and when both are present the maximum must be at least the minimum. The provider discovers Cinder volume types and combines them with this Region-authored metadata into provider-neutral VolumeClass values. It does not discover capacity bounds from Cinder. Maximum performance metadata records caps rather than guaranteed reservations. VolumeClass is Region-scoped inventory configuration, not a project-owned resource or lifecycle object. The block-storage service client is cached with the other OpenStack service clients so Cinder volume-type inventory cache survives repeated provider calls and is refreshed only when Region configuration or credentials change. Production Region CRs must contain their curated IDs before this fail-closed behavior is rolled out.

• Existing-volume server attachments are a separate, project-scoped write path from Region-scoped VolumeClass inventory. The provider creates ephemeral compute and block-storage clients from the service principal, rediscovers the Nova server and detailed Cinder volume, and uses Cinder's attachment rows as the normal-path observation before calling Nova's volume-attachment create/delete API. Cinder volume lifecycle observation is a separate capability from this attachment slice. The Cinder name filter is treated as fuzzy and post-filtered against the exact stable name volume-{Volume.Name}; duplicate exact matches fail closed with ErrConsistency. Only the optional guest device name crosses the provider-neutral boundary; Nova and Cinder IDs and Gophercloud objects remain internal.

  Attachment behavior is deliberately asymmetric:

  • attach requires both the server and volume; either missing resource maps to ErrResourceNotFound
  • a Cinder attachment already present on the requested server is successful only when Cinder reports the Volume in-use; it returns that row's observed device without a Nova read. reserved, attaching, and other transitional states yield for another observation; error* states return a terminal provider error
  • an attachment to any other server maps to ErrConflict; Region does not support multi-attach even when the Cinder volume is multiattach-capable
  • when Cinder reports no attachment, attach requests Nova only after Cinder observes the Volume available; transitional states yield. It yields after Nova acceptance. A create 409 Conflict is followed by one Nova attachment read so a concurrent desired request yields for Cinder in-use; an unresolved conflict maps to ErrConflict
  • detach resolves Nova and checks it directly, so an empty Cinder attachment list cannot hide the claimed Nova attachment. Per-Volume finalizers block Server deletion until attachments are removed, so deleting and live Servers use the same active Nova detach path. A Cinder attachment row is a fallback only when its Server ID matches the resolved claimed Nova Server. Foreign or unowned rows are never detached; they conflict or yield until provider state converges
  • an accepted Nova delete yields immediately because detach is asynchronous. A later reconcile succeeds only when Nova no longer reports the claimed attachment and Cinder reports no attachment with status available; a missing backing Volume is idempotent success
  • a Nova delete 400 Bad Request yields while the provider attachment state converges; the claim is retained and Region does not reset Cinder state. A 409 Conflict maps to ErrConflict; other provider failures are preserved
  • detach also no-ops when the backing OpenStack identity was never realized, matching the provider's other teardown contracts

  Region owns these resources through its API. Detach does not attempt administrative repair of foreign attachments, multiattach, or resources changed directly through Kubernetes or OpenStack.

  Attachment intent and observed rows remain on Server.Spec.Volumes and Server.Status.Volumes; this provider slice does not mirror attachments into Volume.Status, claim volumes, or reconcile server controllers. Those belong to the volume provisioner (../../../provisioners/managers/volume), which populates Volume.Status.AttachedAt and owns claim release.

• Image handling is a first-class contract surface here:

  • OpenStack image properties are validated against a schema
  • public images can additionally be signature-verified
  • image properties are translated into provider-neutral OS, package, GPU, ownership, virtualization, and tag metadata
  • an explicit Glance architecture property wins; otherwise image conversion uses openstack.defaultArchitecture, with the same defensive legacy x86_64 fallback as flavor conversion
  • an optional refresh-ahead cache exists because raw image API latency is too expensive to expose directly to every caller

• Quota and role behaviour are not purely discovered from OpenStack defaults. The package assumes and applies a managed-role model, including default role names such as manager, member, and load-balancer_member, unless region configuration overrides parts of that behaviour.

• Network, security group, server, and Volume resources are re-found in OpenStack by deterministic lookup rather than relying on mirrored OpenstackNetwork, OpenstackSecurityGroup, or OpenstackServer CRDs as authoritative state.

• Baremetal server progress uses Ironic as an additional provider truth source only while Nova reports BUILD for a flavor marked baremetal in region configuration. The result feeds setServerActive, which sets the Active condition to Queued (pre-deploy Ironic states: not yet picked up, cleaning, inspecting, etc.) or Building (Ironic actively deploying — including the post-deploy Error state and the transient *Fail states, on the principle that the node is still in the build pipeline as far as the platform is concerned and the failure signal belongs on the Healthy condition rather than on the Active condition. The node lifecycle eventually terminates via delete; splitting "in the pipeline" from "in the pipeline but unhappy" across both the Active and Healthy conditions would just duplicate one concept across two axes). Provisioning status itself is a separate axis (the Available condition), provisioner-owned and the monitor never writes it; setServerActive does, however, latch the monitor-owned status.provisionedAt field from Nova launched_at the first time a server is seen booted (write-once, never cleared, independent of live power state), which the controller's bounded provider-create delete-and-retry guard relies on (so a server that has booted is never destroyed and recreated). The image-rebuild gate does not read this latch: it authorizes from Nova launched_at read fresh each pass. Alongside it, setServerMACAddress records the other monitor-owned field, status.macAddress, from the Nova response once the server is ACTIVE (the port MAC rides inline in addresses, reused from the read the caller already took — no extra call). ACTIVE is required because baremetal Ironic rebinds the port to the real NIC MAC asynchronously; the value is only ever written, never cleared. GetServer resolves by name, which forces a list, and the list response can omit fault entirely on Nova up to 2025.2, so a listed errored server carries no fault detail. GetServer itself does not compensate: the fault is operator log detail, not part of the returned server, so it is fetched by a dedicated by-ID read exactly once, on the observed transition into the errored state, and written to the observation log — status records only the neutral observed.errored marker. Best-effort: a failed fetch logs the transition without the detail rather than failing anything, because the fault is an enrichment and not the reason for the read; healthy servers, and servers already known to be errored, never pay the extra call. setServerObservedStatus records the monitor's status.observed region from whichever read the caller took — Observe's batch for the monitor, GetServer for the create path: generation unconditionally, the image via openstackServerImageID (an unreadable ref preserves the previous value rather than clearing it), and the neutral errored marker when Nova reports ERROR. The marker is gated on Status == "ERROR" and not on Fault being populated, because Nova leaves a stale fault on a recovered server, so keying off the struct would report a cleared failure forever. The logged detail is the fault's code, message, and created timestamp; fault.details is excluded as an admin-only stack trace, and nothing from the fault reaches projected status. The Ironic lookup is filtered by instance_uuid. Because Ironic node ownership and visibility are provider infrastructure concerns rather than tenant workload operations, this lookup uses the Region top-level provider credentials scoped to the service principal's project, matching the package's other privileged client patterns. Deployments must grant those credentials enough Ironic policy visibility to list/detail nodes by instance UUID, for example through a narrow bm-mapper-style role or equivalent admin, service, or system-reader policy that permits baremetal:node:list_all/node-detail visibility. If the privileged client cannot be created or Ironic rejects or fails the lookup, the monitor logs the failure and falls back to the VM default Building Active state so API responses still see a coherent live signal rather than failing the monitor path.

• Some OpenStack list APIs are not safe to treat as exact lookup, notably server, network, Cinder volume, and Octavia load-balancer name filters:

  • name filters behave like prefix or regular-expression matches rather than strict equality
  • this package therefore re-checks exact names after listing to avoid aliasing and false matches

• Server port reconciliation compares the security groups and allowed address pairs returned by Neutron with the desired values as unordered sets before issuing an update. An omitted desired address-pair MAC is compared as the port's current MAC, matching Neutron's materialized response. A converged port is reused without a PUT, while private-IP status projection still runs.

• Provider networks that require VLAN segmentation use the local VLAN allocator because OpenStack does not allocate those IDs for us.

Credential Sessions

A credential's callers arrive in herds. Every New*Client used to be a full AuthenticatedClient — a Keystone password grant plus a service catalog fetch — and a reconcile builds two or three. Meanwhile a pinned server authenticates as the Region admin user scoped to the service principal's project rather than as the tenant service principal, so every server pinned to one identity shares one credential. A restart resync of an identity holding 1157 servers asked for that one credential a couple of thousand times inside a few minutes, and the health monitor asked for it again for every server, every minute.

A Session is the per-credential state that outlives every service client built from it, keyed on the whole credential so a rotated secret can never be served a client authenticated with the old one. The key is a keyed digest under a process-lifetime random key rather than the credential verbatim or a bare hash of it: a credential's input space is small enough that an unkeyed digest is reversible by enumeration, and nothing should hold a secret in a map key that lives as long as the process.

What the key really has to capture is everything determining where the derived service clients point. That is the credential today only because every New*Client passes a bare gophercloud.EndpointOpts{} and takes the first public catalog match. Should anything start selecting an endpoint by region, two OpenStack regions behind one Keystone reached with one admin credential would share both a pinned catalog and a session, and the region would have to join the key. It caches the authenticated ProviderClient and collapses concurrent cold-start logins into one, which is what a controller restart needs: no client is cached yet at the moment the herd arrives.

Cache the ProviderClient, not the derived service clients. It holds the token, carries ReauthFunc and serialises its own reauth internally, so it refreshes in place. Caching derived clients would synchronise token expiry across a burst — everything built in one burst expiring in one burst — and reform the herd an hour later.

Three things follow that are worth knowing.

The service catalog is pinned for the life of the process. gophercloud's v3auth closes EndpointLocator over the catalog from the login that built the client, and ReauthFunc only copies the refreshed token back, not a new catalog. An operator who moves a service's public endpoint in Keystone needs this process restarted; previously each client construction re-fetched the catalog and self-healed within a reconcile. The same applies to a service added to the catalog after startup — enabling Octavia on a live region leaves load-balancer clients failing endpoint lookup until a restart. The operator-facing procedure is in ADMIN.md, because the symptom points away from the cause: calls fail against an endpoint that has already been changed while Keystone itself is healthy, and nothing logs the staleness.

The shared client carries a request timeout, and must. Sharing a client shares gophercloud's reauth serialisation: AuthenticatedHeaders waits for an in-progress refresh on every request, on a channel receive that consults no context. One caller whose refresh hangs would otherwise park every other caller on that credential — reconciles, the monitor, and the synchronous reboot, start, stop and console handlers — below the level at which their own contexts could rescue them. A zero-value http.Client has no timeout, so the bound is set explicitly.

It is set before authenticating, which is not interchangeable with setting it afterwards: gophercloud copies the client by value to build the one it reauthenticates with, so a timeout assigned after the login leaves both the login and every subsequent reauth unbounded — and the reauth is the request every other caller waits behind. Pinned by TestSessionBoundsTheLoginItself.

The bound now applies to every OpenStack call made with a shared client, all of which were previously unbounded. Every call in this package is ordinary request/response and pagination pays the bound per page, so the value is defensible, but a call that legitimately took longer would now fail where it used to succeed.

It bounds a request, not a call. One call can serialise a context-free wait on another caller's in-progress reauth, its own request, a reauth on a 401, and the retry, so the worst case a caller observes is several multiples of it. A cold login against a versionless Keystone endpoint is likewise two bounded requests, discovery and the grant. Do not size an upstream timeout off the single-request figure.

The detaching covers the cold-start login only, because that is the one this package's code owns. gophercloud owns the reauth and runs it under whichever caller's request received the 401, sharing that one result with every caller refreshing at the same moment — so at token expiry a caller that hangs up mid-refresh can hand its cancellation to the others. They error and retry and it self-heals, so this is a spurious failure rather than a stuck credential, but it is a consequence of sharing that did not exist when every caller refreshed its own client.

Callers waiting on someone else's login keep their own contexts and stop waiting when those are done. The login itself runs detached from whichever caller opened it, so a handler whose client hung up cannot fail the healthy reconciles that joined its flight, and a caller that gives up leaves the login to finish and populate the cache rather than wasting it.

Entries are never evicted, and grow by two per OpenstackIdentity the process has ever seen — the service principal's own password and the Region admin scoped to that identity's project — rather than tracking the live identity count. An entry appears when a provider is constructed, whether or not it ever logs in, and rotating the Region admin secret re-keys every identity-scoped session and adds another entry per live identity. Each retains a decoded token response and the service catalog it closes over, which is tens of kilobytes, not nothing. For a long-lived controller against a churning estate that is worth bounding; the cost of an eviction is one login.

What this deliberately does not do

It does not share provider reads. Collapsing GetServer was measured and rejected. Nova's name filter is a regular expression, so a filtered read scans the project and returns one row; dropping the filter would let concurrent callers share one list, but it returns every row in the project to every caller. A bulk create emitted three orders of magnitude more instance records because each create retires the sharing anyway, and sharing the reads would have needed a guard against a caller joining a read opened before its own create and building a second server.

The figures that comparison rested on came from a model whose service times were plausible rather than measured, and a real deployment contradicted them by two orders of magnitude: over a representative estate of around 1500 servers a sequential cycle took tens of minutes, where the model put it at 35 seconds. The model counted requests and rows; it did not price what a loaded Nova charges for a regular expression scan over a project. Do not size anything off the 35s figure — neither read form ever fitted the one-minute poll period.

The saving on the monitor's path was never a shared read but a single read, and ObserveServers is now that: one unfiltered, paginated list of the identity's project per cycle, indexed by name, replacing a read per server with a read per identity — two orders of magnitude fewer Nova calls against the same estate. Observation-only reads are allowed to do this; see the note on projected status in the API package.

It pages explicitly rather than using AllPages, which holds every page as a generic map tree and re-marshals the whole thing through JSON to decode it, so that overhead scales with the project rather than the page.

Paging bounds the overhead, not the peak: the decoded slice is the whole project by construction, and ObserveServers retains it for the length of its caller's projection. For a project of around 1500 realistic Nova rows that is about 14MiB live, or 43MiB of heap arena once the per-page decode churn is counted — against roughly 15MiB for a project of 80. That retention is the dominant term. No page size changes that. It is bounded instead by the caller reading one project at a time, and by a project being bounded by the region's physical footprint. Trimming the retained row to the fields the projection actually uses would bound it properly, at the cost of changing a signature the create path also depends on.

Because an unfiltered list is walked by marker, a server created between two pages can be missed and read as absent for that cycle. Harmless here — the absent path stamps an observation and touches no condition, and it self-heals on the next poll — but it is a way to reach that path which a filtered read of one name could not. Nothing destructive keys off it: the create-retry gate goes through UpdateServerState's own fresh read, never the batch.

UpdateServerState keeps its own per-server filtered read, and must. An observation may only refuse an action, never authorise one, so the create path's "is there already a server with exactly this name" question takes a fresh read at decision time rather than consulting a batch someone else took.

Octavia Load Balancers

OpenStack load balancers are reconciled through Octavia in the service principal's project. The region LoadBalancer CRD is still the desired-state root, while Octavia remains the cloud-side source of truth for the realized topology.

The provider reconciles the full topology:

• the Octavia load balancer and VIP
• listeners, pools, members, and optional health monitors
• the optional public floating IP attached to the Octavia-owned VIP port

Cloud-side lookup uses deterministic names:

• load balancer: lb-{loadBalancer}
• listener: lb-{loadBalancer}-{listener}-listener
• pool: lb-{loadBalancer}-{listener}-pool
• health monitor: lb-{loadBalancer}-{listener}-monitor

Those names are not just cosmetic. They are the linkage contract that lets the provider re-find and converge existing Octavia resources without mirrored provider-state CRDs. Octavia list filters are fuzzy in the same way as other OpenStack name filters, so the client always post-filters returned resources by exact name and treats duplicate exact matches as consistency errors.

Octavia provisioning status controls the reconcile outcome:

• ACTIVE allows the provider to continue reconciling the next part of the topology
• PENDING_CREATE, PENDING_UPDATE, and PENDING_DELETE yield the controller so the next pass can observe settled state
• any other state is treated as a consistency error because the provider cannot safely infer a valid next action

Mutable topology is converged in place where Octavia permits it:

• listener allowed CIDRs
• listener default-pool linkage
• TCP listener idle timeouts
• pool members
• health-monitor thresholds
• orphaned listeners, pools, and monitors whose deterministic names are no longer implied by the current spec

Other fields are intentionally blocked before they reach this provider. The handler keeps existing listener protocol and port immutable, and it blocks proxyProtocolV2 drift for an existing listener name because that changes the derived Octavia pool protocol, which Octavia does not allow to be updated in place.

There are a few Octavia-specific constraints worth preserving:

• UDP listeners do not support idle timeouts or Proxy Protocol v2.
• UDP health checks use Octavia's UDP connect monitor type.
• TCP pools use Octavia PROXYV2 only when proxyProtocolV2 is enabled; the load-balancer client pins microversion 2.22 so that protocol is available.
• Floating IP cleanup runs before cascade-deleting the load balancer because the cascade removes the VIP port that otherwise anchors the floating IP lookup.

Caveats

• This package is the convergence point of a large amount of platform policy, provider behaviour, and historical baggage. Its size reflects real behaviour, not just poor code hygiene.

• Deterministic lookup is the preferred direction, but the package still lives in a mixed world:

  • some cloud-side state is derived live from OpenStack
  • some transitional compatibility fields still exist in repo-native CRDs
  • OpenstackIdentity still persists state that the service would ideally stop owning over time

• Deterministic lookup is cleaner than mirrored CRDs, but it is still sensitive to convention drift. Renaming generated resources, changing metadata keys, or casually altering project-scoping assumptions can break the linkage between Unikorn resources, what OpenStack stores, and what users can see from inside provisioned servers.

• OpenstackIdentity should not be treated as permanently special. Its current survival is largely driven by implicit side effects and secret-bearing service-owned state that the architecture should work to remove:

  • ephemeral SSH key generation and download
  • implicit server-group creation
  • persisted service-principal/user/project/application-credential data

• Exposing application credentials to higher layers is current operational reality, not the desired end state. The package's scoping model helps contain blast radius today, while the wider platform works toward removing that exposure entirely.

• If the wider API moves toward explicit SSH certificate authority use, explicit server-group resources, and less implicit provider-side identity scaffolding, deleting OpenstackIdentity becomes more realistic.

• Image metadata translation is powerful but fragile. This package currently depends on OpenStack image properties carrying a large amount of semantic information correctly.

• Image query, get, create, delete, and snapshot flows are tightly coupled to the image cache path. When caching is disabled, large parts of the higher image contract are effectively unavailable rather than merely slower.

• The image query layer still contains its own comment admitting that some logic now operates on generic types and probably should not live here long term.

• Some older assumptions still leak through in status fields and helper paths, especially where compatibility with older API or storage shapes is still being carried.

• Rebuild rests on one environmental fact about the target cloud: that OS-EXT-STS:task_state is actually visible to the region service principal. Its exposure is policy-gated, and an unexposed field decodes indistinguishably from "at rest" — so without it a foreign operation would not be seen to hold the server, and Nova's 409 becomes the only guard. It was visible to a project-scoped admin credential on a kolla 2025.1 all-in-one.

  OS-SRV-USG:launched_at is likewise policy-gated and is now load-bearing for the image-row decisions — the before-first-boot deferral and the never-launched exclusion on the failed-rebuild park — so a cloud hiding it breaks those rows the same way a hidden task_state breaks the in-flight tests.

  Measured on the libvirt driver — the first GET after accept, at 260 ms with 1 s polling, already showed the target ref together with task_state='rebuilding'. The failure side is atomic the same way: an asynchronous failure moved REBUILD→ERROR and cleared task_state within a single observation, so a failed rebuild never presents as a settled one.

  The same contract holds on the Ironic driver (measured on sushy-tools virtual metal): the ref and task_state flip together at accept, the rebuild task states stay visible through the whole redeploy, and a failed redeploy settles ERROR with the ref still on the target — it never reverts, which is what keeps a failed rebuild from re-satisfying the submission row. The node lands in deploy failed, and a rebuild toward another image recovers both server and node without operator action.

  • Do not raise the compute client microversion past 2.92. From 2.93 Nova sets reimage_boot_volume on every rebuild and the Ironic driver refuses the flag outright — volume-backed or not — so every baremetal rebuild fails. An upstream defect, unfixed as of 2025.1 (https://bugs.launchpad.net/nova/+bug/2127017). The client pins 2.90.
  • Ironic node reads require a system-scoped credential — project-scoped admin lists nodes but 404s on node detail — so task_state visibility has an Ironic sibling that needs its own credential arrangement.

• A rebuild toward a nonexistent image is rejected synchronously (HTTP 400, "Cannot find image for rebuild") and touches nothing: the server stays on its image, no wipe, no ERROR. Nova resolves the image before mutating anything, so a bad image ID can never produce a converged-looking ref on a broken server. The controller parks only this signature ("Cannot find image for rebuild", matched in the 400 response body) as user-action-required, since a deleted image can never be re-resolved and the remedy is a spec edit. Other rebuild 400s retry: notably ImageNotActive from a deactivated Glance image, which an operator can clear with an image reactivation that bumps no generation and moves no observed field — a park there would have no recovery path. An HTTP 409 yields awaiting quiescence, and every other rejection class retries. A valid image Nova can resolve but qemu cannot use (measured with a truncated qcow2, which Glance accepts as active) behaves the opposite way: accepted, ref flipped, then an asynchronous failure to ERROR. Its fault carries code: 400 despite the request having been accepted, so a 4xx in a recorded provider error does not imply the request was rejected.

TODO

• Delete the remaining mirror-state OpenStack CRD usage paths entirely: OpenstackNetwork, OpenstackSecurityGroup, and OpenstackServer should not survive as authoritative provider-state patterns.
• Continue shrinking the reasons OpenstackIdentity must exist:
  • remove service-handled private SSH key material in favour of explicit SSH certificate trust
  • stop relying on implicit server-group provisioning
  • move toward explicit API shapes where reconstructable state does not need to be persisted here
• Revisit image-query and image-metadata logic that now operates on provider-neutral types but still lives in this package because of historical coupling.
• Remove remaining compatibility writes and reads that depend on transitional CRD status shapes as those fields disappear from the wider system.

Cross-Package Context

• ../../types defines the neutral provider contract and intermediate types this package must satisfy
• ../../../apis/unikorn/v1alpha1 defines the service-native control objects and the remaining persisted provider-state records this package consumes
• ../../../handler and specific handler packages depend on this package to make region API operations real against OpenStack
• ../../allocation/vlan exists because this package needs a compensating local allocator for provider-network VLAN IDs