Documentation
¶
Overview ¶
Package exporter provides typed HTTP helpers for the data-exporter API exposed by a running DataExport. The API has two endpoints: api/v1/block (block volumes served via http.ServeContent with Range support) and api/v1/files (filesystem volumes: trailing- slash paths return a JSON directory listing; other paths stream file bytes).
Index ¶
- Constants
- Variables
- func BlockURL(baseURL string) (string, error)
- func DataExportName(namespace, group, resource, kind, leafName string, targetUID types.UID) string
- func EnsureDataExport(ctx context.Context, c client.Client, ...) (*deapi.DataExport, error)
- func FilesURL(baseURL string) (string, error)
- func IsTransientDataPlaneError(err error) bool
- func ReleaseDataExport(ctx context.Context, c client.Client, log *slog.Logger, ...) error
- func WaitReady(ctx context.Context, c client.Client, log *slog.Logger, ...) (*deapi.DataExport, error)
- type DataExportAcquisition
- type Doer
- type EnsureOption
- type Export
- type Fetcher
- func (f *Fetcher) GetFile(ctx context.Context, fileURL string) (io.ReadCloser, error)
- func (f *Fetcher) HeadVolume(ctx context.Context, blockURL string) (int64, error)
- func (f *Fetcher) ListDir(ctx context.Context, filesURL string, yield func(Item) error) error
- func (f *Fetcher) RangeGet(ctx context.Context, blockURL string, start, end int64) (io.ReadCloser, error)
- func (f *Fetcher) SourceMD5(ctx context.Context, fileURL string, size int64) (string, error)
- type FetcherOption
- type IdleConnectionCloser
- type Item
- type WaitReadyOption
Constants ¶
const DefaultIdleReadTimeout = 2 * time.Minute
DefaultIdleReadTimeout is the conservative default idle window applied to every data-plane response body a Fetcher hands out or consumes. It bounds how long a single Read may block WITHOUT receiving any bytes; it is not an overall transfer deadline, so a slow-but-flowing stream is never aborted. Override per Fetcher with WithIdleReadTimeout (tests use a short window).
Variables ¶
var ErrContentRangeMismatch = errors.New("server Content-Range does not match requested range")
ErrContentRangeMismatch is returned when a 206 response's Content-Range header does not cover the byte range the caller requested, so the body must not be trusted at the caller's intended offset.
var ErrDataPlaneIdle = errors.New("data-plane read stalled: no bytes within idle timeout")
ErrDataPlaneIdle is reported by a Fetcher-issued response body when no bytes arrive for the configured idle window (see idleReadCloser). The data plane deliberately runs without an overall request deadline — volume transfers are long — so a TCP connection that stops delivering bytes WITHOUT erroring (NAT/LB half-close, wedged exporter pod) would otherwise block a Read (and the whole download) forever. It is wrapped with %w so the chunk/file retry+resume machinery treats a silent stall as an ordinary fetch error.
var ErrExpired = errors.New("DataExport expired")
ErrExpired is returned by WaitReady when the DataExport enters the Expired terminal state and can no longer be used for data transfer.
ErrExportUnauthorized classifies a 401/403 from the data-exporter endpoint. On the public (Ingress) path this is the expected outcome for a certificate-authenticated kubeconfig: Ingress terminates TLS with its own certificate and does not forward the client certificate to the exporter pod, so only bearer-token kubeconfigs authenticate through it.
var ErrPublishForeignOwner = errors.New("DataExport owned by another download run has publish disabled")
ErrPublishForeignOwner is returned when --publish requires enabling spec.publish on a DataExport that ANOTHER live download run owns. Adoption of a foreign CR is read-only by contract, so this run refuses to mutate it and fails fast instead of waiting out the full readiness timeout for a status.publicURL that will never appear.
var ErrTargetRefMismatch = errors.New("existing DataExport targets a different object")
ErrTargetRefMismatch is returned by EnsureDataExport when a same-named DataExport CR already exists but its Spec.TargetRef names a DIFFERENT object than the request. This can occur after a hash collision or when a DataExport was created manually under the deterministic name. Reusing its endpoint would download the wrong object's bytes. EnsureDataExport refuses instead of silently reusing or deleting it.
var ErrTargetUIDMismatch = errors.New("existing DataExport targets a different object UID")
ErrTargetUIDMismatch is returned by EnsureDataExport when an observed DataExport is not stamped for the exact Snapshot CR UID requested by the caller. The targetRef contract does not carry UID, so the client-owned annotation is the authoritative lifecycle discriminator.
var ErrTargetUIDRequired = errors.New("snapshot target UID is required")
ErrTargetUIDRequired is returned when a caller attempts to open a DataExport lifecycle without the exact Snapshot CR UID.
Functions ¶
func BlockURL ¶
BlockURL returns the block-volume endpoint for a DataExport base URL. The block volume is served at api/v1/block.
func DataExportName ¶
DataExportName derives a deterministic DataExport CR name from the canonical namespaced target identity, including the exact Snapshot CR UID. The readable leaf prefix is normalized only for display; the hash covers the original, unnormalized identity so identities that normalize alike remain distinct. The result is a DNS-1123 label no longer than 63 bytes, which also satisfies Kubernetes object-name limits.
func EnsureDataExport ¶
func EnsureDataExport( ctx context.Context, c client.Client, namespace, group, resource, kind, leafName, ttl string, opts ...EnsureOption, ) (*deapi.DataExport, error)
EnsureDataExport idempotently creates a DataExport in namespace targeting the snapshot leaf CR identified by {group, kind, leafName, target UID} with the given TTL (empty → "2h"). Returns the DataExport object (newly created or pre-existing).
group and kind must identify a namespaced snapshot CR (e.g. "snapshot.storage.k8s.io" / "VolumeSnapshot" for a CSI VolumeSnapshot leaf, or the domain group / kind for a domain snapshot CR). The controller routes any such targetRef through its kind-agnostic categorySnapshot path.
Pass WithRunOwner to scope ownership to a single download run: the run stamps its ID on any CR it creates and is warned when it adopts a CR another live run owns (see WithRunOwner and inv #10b). Without it, EnsureDataExport keeps its original ownership-agnostic behavior.
func FilesURL ¶
FilesURL returns the filesystem root-listing endpoint for a DataExport base URL. The trailing slash instructs the server to return a directory listing.
func IsTransientDataPlaneError ¶ added in v0.33.16
IsTransientDataPlaneError reports whether err is a transient transport failure on the volume data plane — one where re-issuing the Range GET from the caller's durable resume offset is the correct response.
It fails CLOSED: anything not explicitly listed is treated as fatal, so a misclassification costs a loud failure, never a silent retry loop that masks a real defect. In particular, sentinels defined outside this package (volume.ErrShortChunkRead, any *os.PathError from local disk I/O) are non-transient by construction, because the default is false.
func ReleaseDataExport ¶
func ReleaseDataExport( ctx context.Context, c client.Client, log *slog.Logger, acquisition *DataExportAcquisition, ) error
ReleaseDataExport deletes only the exact DataExport represented by acquisition. Deterministic name or matching run annotation alone never grants authority. UID, targetRef, target UID annotation, and observed owner must all still match; a foreign owner is never deleted. UID-precondition conflicts and NotFound are idempotent success.
func WaitReady ¶
func WaitReady( ctx context.Context, c client.Client, log *slog.Logger, namespace, deName string, opts ...WaitReadyOption, ) (*deapi.DataExport, error)
WaitReady polls the DataExport named deName until:
- its Ready condition is True and its base URL (status.url, or status.publicURL when WithPublicEndpoint is passed) is populated → returns the DE,
- it is Ready=False with reason Expired → returns a wrapped ErrExpired,
- ctx is cancelled or its deadline is exceeded → returns a wrapped ctx.Err() that includes the last observed DataExport status and an inspection hint.
The poll interval is 3 s. A log line is emitted on the first poll and every logEveryN polls (≈15 s) to avoid spamming output while the export initialises. Callers set a deadline via ctx to bound the wait.
Types ¶
type DataExportAcquisition ¶
type DataExportAcquisition struct {
// contains filtered or unexported fields
}
DataExportAcquisition is operation-scoped evidence that EnsureDataExport successfully acquired one exact DataExport object. Its fields are private so cleanup callers must obtain it from WithAcquisition rather than infer deletion authority from a deterministic name or run annotation.
func (*DataExportAcquisition) Name ¶
func (a *DataExportAcquisition) Name() string
Name returns the acquired DataExport name.
func (*DataExportAcquisition) TargetRef ¶
func (a *DataExportAcquisition) TargetRef() deapi.TargetRefSpec
TargetRef returns the exact targetRef observed when the DataExport was acquired.
func (*DataExportAcquisition) TargetUID ¶
func (a *DataExportAcquisition) TargetUID() types.UID
TargetUID returns the exact Snapshot CR UID bound to the acquisition.
func (*DataExportAcquisition) UID ¶
func (a *DataExportAcquisition) UID() types.UID
UID returns the exact UID observed when the DataExport was acquired.
type Doer ¶
Doer executes a single HTTP request and returns the response. *http.Client and pkg/libsaferequest.SafeClient both satisfy this interface.
type EnsureOption ¶
type EnsureOption func(*ensureOptions)
EnsureOption configures optional behavior of EnsureDataExport.
func WithAcquisition ¶
func WithAcquisition(out **DataExportAcquisition) EnsureOption
WithAcquisition records operation-scoped cleanup evidence when EnsureDataExport successfully returns an exact DataExport. The output remains nil on every pre-acquisition failure, including a targetRef mismatch.
func WithPublish ¶ added in v0.33.16
func WithPublish(publish bool) EnsureOption
WithPublish makes EnsureDataExport request the storage-foundation-published (Ingress) endpoint for this DataExport: spec.publish is set on any CR this call CREATES, and an adopted CR that still has spec.publish=false is upgraded in place (see alignDataExportPublish). Publish is never downgraded back to false.
func WithRunOwner ¶
func WithRunOwner(runID string, log *slog.Logger) EnsureOption
WithRunOwner makes EnsureDataExport stamp runID as the owning run (runOwnerAnnotation) on any DataExport it CREATES, and log an explicit WARN via log when it instead adopts a live CR that a DIFFERENT run already owns. The adopted endpoint is still reused for read-only transfer, but ownership — and therefore the right to delete the CR on release (ReleaseDataExport) — stays with the other run, so neither run tears down the other's in-flight export (inv #10b). runID must be non-empty to take effect; a nil log disables the adoption WARN.
func WithTargetUID ¶
func WithTargetUID(targetUID types.UID) EnsureOption
WithTargetUID binds the DataExport lifecycle to the exact Snapshot CR incarnation. EnsureDataExport rejects calls that omit it.
func WithTerminatingWaitTimeout ¶
func WithTerminatingWaitTimeout(d time.Duration) EnsureOption
WithTerminatingWaitTimeout bounds the wait EnsureDataExport performs when it observes the DataExport already TERMINATING (DeletionTimestamp set) and must wait for it to fully vanish before recreating a fresh one. Without this cap the wait is bounded only by ctx, so a caller that passes a deadline-less ctx (e.g. the raw download-run ctx used by the pipeline's stamp-Ensure) would hang the whole run FOREVER on a wedged finalizer or a downed DataExport controller, with no output — a ctx that merely CAN carry a deadline is not enough (code-style §6). The pipeline derives d from the run's ReadinessTimeout. A non-positive d leaves the wait bounded by ctx alone (the pre-existing behavior for callers that do not opt in).
type Export ¶
type Export struct {
// contains filtered or unexported fields
}
Export holds a resolved DataExport: a ready HTTP endpoint (Fetcher), the VolumeMode reported by the controller, and the internal base URL. Callers release the underlying DataExport CR via ReleaseDataExport + DataExportName using the leaf name they already have, rather than through this value — releasing by deterministic name also covers the case where OpenExport never returned an Export at all (e.g. cancelled while still waiting for Ready).
func NewExport ¶
func NewExport( namespace, deName, volumeMode, baseURL string, fetcher *Fetcher, httpClients ...IdleConnectionCloser, ) *Export
NewExport constructs an Export from pre-built components. Any supplied HTTP clients become owned by the Export and are released by CloseIdleConnections. It is intended for testing and alternative transport implementations that bypass the production DataExport lifecycle.
func OpenExport ¶
func OpenExport( ctx context.Context, log *slog.Logger, c client.Client, namespace, group, resource, kind, leafName, ttl string, sc *transport.Client, opts ...EnsureOption, ) (*Export, error)
OpenExport creates (or re-uses) a DataExport targeting the snapshot leaf identified by {group, kind, leafName}, waits until it is Ready, and returns an Export ready for data transfer.
An isolated copy of sClient is built for the HTTP Fetcher so that CA injection does not mutate the caller's client.
opts are forwarded verbatim to the inner EnsureDataExport. Callers pass WithRunOwner so that if the deterministic de-<leaf> CR has to be RECREATED here (e.g. it vanished between the pipeline's stamp-Ensure and this inner Ensure), the fresh CR is stamped with this run's ownership rather than left unstamped — closing the per-run ownership gap in the vanish window (inv #10b).
func (*Export) CloseIdleConnections ¶
func (e *Export) CloseIdleConnections()
CloseIdleConnections releases every HTTP connection pool owned by the Export.
func (*Export) VolumeMode ¶
VolumeMode returns the volume mode reported by the DataExport controller ("Block" or "Filesystem").
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher wraps a Doer and exposes typed methods for the data-exporter HTTP API.
func NewFetcher ¶
func NewFetcher(doer Doer, opts ...FetcherOption) *Fetcher
NewFetcher creates a Fetcher backed by the given Doer. Unless overridden via WithIdleReadTimeout, response bodies carry an idle-read watchdog with DefaultIdleReadTimeout.
func (*Fetcher) GetFile ¶
GetFile GETs fileURL and returns the response body for streaming. The caller must close the returned ReadCloser.
func (*Fetcher) HeadVolume ¶
HeadVolume issues a HEAD request to blockURL and returns the total content length in bytes.
func (*Fetcher) ListDir ¶
ListDir GETs filesURL (which must end with a trailing slash for directory semantics), requests only the inexpensive stat attributes, and calls yield once per stream-decoded directory entry. It retains only the current item, so a flat directory's memory use is independent of its entry count. Source hashes are fetched separately after each regular file's declared size is known because the producer computes hash.md5 synchronously before emitting that listing item.
func (*Fetcher) RangeGet ¶
func (f *Fetcher) RangeGet(ctx context.Context, blockURL string, start, end int64) (io.ReadCloser, error)
RangeGet issues a GET request with a Range: bytes=start-end header to blockURL and returns the response body. The caller must close the returned ReadCloser. Returns an error unless the server responds with 206 Partial Content AND its Content-Range header confirms the returned body actually covers [start, end]: the block exporter (storage-volume-data-manager images/data-exporter/internal/export_block/ handler.go HandleGetMethod) serves the block device via stdlib http.ServeContent, which always sets Content-Range on a 206 response, so a missing or mismatched header means a misbehaving server/proxy returned bytes from the wrong offset (or the whole object) and must not be trusted at the caller's intended offset.
func (*Fetcher) SourceMD5 ¶
SourceMD5 retrieves the producer-computed plaintext MD5 for one regular file through the filesystem exporter's HEAD attribute contract. The producer must read the complete source file before it can send this response header, so the request uses an overall size-derived deadline rather than the transfer body's progress-based idle watchdog.
The budget assumes the source can be hashed at no less than 1 MiB/s, adds one minute of fixed scheduling slack, floors small or unknown sizes at five minutes, and caps untrusted declared sizes at seven days. It is finite for every int64 size. An empty result means an older exporter returned 200 without the optional hash header.
type FetcherOption ¶
type FetcherOption func(*Fetcher)
FetcherOption customizes a Fetcher at construction time.
func WithIdleReadTimeout ¶
func WithIdleReadTimeout(d time.Duration) FetcherOption
WithIdleReadTimeout sets the idle-read watchdog window for response bodies this Fetcher issues. A value <= 0 disables the watchdog (bodies are returned unwrapped). When unset, DefaultIdleReadTimeout applies.
func WithPublishUnauthorizedHint ¶ added in v0.33.16
func WithPublishUnauthorizedHint() FetcherOption
WithPublishUnauthorizedHint makes this Fetcher append an actionable hint to ErrExportUnauthorized failures. Only the publish path sets it, so the in-cluster path never advertises an Ingress-specific remedy.
func WithSourceHashDoer ¶
func WithSourceHashDoer(doer Doer) FetcherOption
WithSourceHashDoer sets the transport used for source-hash HEAD requests. Production uses a separate transport because source hashing may legitimately take longer than the ordinary data-plane response-header timeout.
type IdleConnectionCloser ¶
type IdleConnectionCloser interface {
CloseIdleConnections()
}
IdleConnectionCloser owns an HTTP connection pool used by an Export.
type Item ¶
type Item struct {
Name string `json:"name"`
Type string `json:"type"`
URI string `json:"uri"`
TargetPath string `json:"targetPath,omitempty"`
Attributes map[string]any `json:"attributes"`
}
Item is one entry returned by the data-exporter filesystem listing API.
type WaitReadyOption ¶ added in v0.33.16
type WaitReadyOption func(*waitReadyOptions)
WaitReadyOption customizes what WaitReady treats as ready.
func WithPublicEndpoint ¶ added in v0.33.16
func WithPublicEndpoint() WaitReadyOption
WithPublicEndpoint makes WaitReady additionally require a non-empty status.publicURL. The controller can flip Ready=True BEFORE it has finished creating the public Service/Ingress and written status.publicURL, so returning on Ready alone would hand the caller an empty base URL.
Named after the endpoint, not "published": in this codebase "published" already means an atomically committed on-disk artifact (archive.PublicationPublished).
Source Files