s3

Overview ¶

Package s3 is Harbor's S3-compatible ArtifactStore driver. It is the operator-controlled-object-store production target — durable, multi-binary-friendly, and the canonical choice for cloud-native deployments. Speaks AWS S3, MinIO, Cloudflare R2, and any other S3-compatible API surface that the configured `Endpoint` and `UsePathStyle` knobs reach.

Built on `github.com/aws/aws-sdk-go-v2`. CGO_ENABLED=0 stays — the SDK is pure Go.

Object-key layout:

<prefix>/<tenant>/<user>/<session>/<namespace>/<id>
<prefix>/<tenant>/<user>/<session>/<namespace>/<id>.meta.json

`<prefix>` is the operator-configured `S3Prefix` (may be empty — then the layout is rooted at the bucket itself). The `<id>.meta.json` sibling carries the `ArtifactRef` JSON (mime, size, sha, scope, namespace, source), and the producing task rides inside that JSON as `Scope.TaskID` — a provenance annotation.

THE KEY IS THE READ KEY, AND ON THIS DRIVER IT HAS TO BE. Reads resolve on the isolation triple, so the object key holds exactly the triple. The task segment this layout used to carry was removed rather than merely ignored, and the reason is concurrency: an object store offers no atomic compare-and-set, so dedup here is a probe followed by a write. With the task in the key, N runs racing IDENTICAL bytes into one session each probe, each miss, and each write a SEPARATE object — so the store would hold N copies of one content-addressed id and `List` would return N rows for it. Keying on the triple makes the racers write the SAME key, so the store converges on one object by construction rather than by winning a probe. The in-memory, filesystem and SQL drivers get the same property from a mutex or a primary key; this driver gets it from the key itself.

Objects written by an earlier build under the old `.../<session>/<task>/<namespace>/<id>` layout stay READABLE and DELETABLE: resolution probes the triple-keyed key first (one HEAD) and falls back to a `ListObjectsV2` scan of the session prefix, whose task-nested matches are returned in ascending task order — the same smallest-task collapse rule the filesystem driver's index rebuild and both SQL migrations apply. Nothing is rewritten in place; a bucket is migrated by being read.

`Delete` removes EVERY copy under the triple — the triple-keyed object plus any task-nested leftover — because a delete that reported success while leaving a copy a later `Get` resolves is the silent degradation CLAUDE.md §13 forbids. That costs `Delete` one session-prefix listing.

Identity-mandatory boundary. Tenant / user / session must be non-empty for Put*, Get, GetRef, Exists, Delete, and PresignGet. Empty `TaskID` is acceptable for session-scoped artifacts (matches the FS / InMem drivers).

404 semantics. `Get`, `GetRef`, `Exists`, `Delete` map S3 404 / `NoSuchKey` / `NotFound` to `(zero-value, false, nil)` — found-false is NOT an error, matching the FS driver's contract. Other errors (network, signature, permission) are wrapped and surfaced.

Dedup. `PutBytes` / `PutText` resolve the triple first (a HEAD of the triple-keyed object, falling back to the session scan for a legacy task-nested one); if the resolved sibling `.meta.json` carries the new bytes' SHA, that existing ref is returned without re-uploading and the FIRST writer's provenance stamp is what comes back. Under a genuine concurrent tie the probe cannot order the writers — an object store has no compare-and-set — so both write the same key and the stored stamp is whichever write landed last. The stored artifact is still one; only the stamp is undetermined, and "first writer" is not a property an unordered pair of writes has.

The cost, stated because it is a per-write cost and not only a migration one: storing a NEW artifact misses the HEAD and therefore pays one session-prefix listing before uploading. Skipping that listing would be cheaper and wrong on a bucket written by an earlier version — the same bytes would be stored a second time under the new key while the task-nested copy remained, so `List` would return two rows for one content-addressed id, which is the divergence keying on the triple exists to remove. The listing is scoped to one session's prefix, and it is the price of one uniform answer instead of two behaviours depending on when the bucket was written.

`Delete`. Resolve-then-delete. The resolved key set carries the `(existed bool)` the other drivers return (S3's DeleteObject reports success regardless of prior existence), and every resolved copy — plus its sibling meta — is removed via per-key `DeleteObject` calls.

Concurrency. The SDK's `*s3.Client` and `*s3.PresignClient` are safe for concurrent use. The driver itself adds only an atomic closed flag; the conformance suite's `Concurrent_PutGet_NoRace` gate (N=128 default) and the supplemental N=32 stress in `concurrent_test.go` prove the contract holds end-to-end.

Presigner capability. The driver implements `artifacts.Presigner` (`PresignGet` only — write-side presigned URLs are an attack surface intentionally not exposed at V1; see plan non-goals). Expiry bounded `[1 minute, 7 days]` — out-of-range returns a clear error.