multisite

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 34 Imported by: 0

README

Ceph RGW multisite

Import github.com/JSYoo5B/ceph-testcontainers-go/rgw/multisite to build an RGW multisite realm across independent disposable clusters, so that a test can check how its S3 application behaves with replication: objects written in one zone appearing in another, a zone outage, a planned metadata master failover, selective sync with sync policies. Ordinary gateway use (users, tenants, quotas) stays in rgw, accounts in rgw/account and placement in rgw/placement. See API purpose and purpose tags.

Order of use

  1. Start fresh clusters with ceph.Run, without gateways or RGW configuration, all in the same network mode.
  2. Build the realm with multisite.Run(ctx, img, multisite.WithConfig(cfg)) for two zones (Source is the metadata master; both accept writes) or multisite.WithTopology(topology) for more zones and zonegroups. img is the gateway image. AddZone, AddZonegroup and RemoveZone change the topology later.
  3. Hand the endpoints to the SUT: each Zone's rgw.Gateway (Source, Destination, Zones()) gives S3Endpoint, Region and the test user's AccessKey / SecretKey. Create users through the master.
  4. Shape replication when the scenario needs it: CreateSyncGroup, then SyncGroup.CreateFlow / CreatePipe / SetStatus and SyncGroup.Apply for zonegroup policy; InterruptZoneLink and ApplyZoneLinkProfile cut or slow the link between zones; PullSourcePeriod / PullDestinationPeriod carry a planned master change.
  5. Observe and verify: WaitSyncReady (metadata and data catch-up), SyncGroup.WaitBucketPolicyReady, SyncGroup.WaitBucketSyncReady, then read the objects through the other zone's endpoint. WatchSync and SyncGroup.WatchBucketSync report lag and catch-up while a fault runs; SyncGroup.WatchBucketPolicy reports the stages of one policy import. rgw/expect evaluates one zone's captured SyncStatus against a stated direction: caught up with, or behind, one source. ZoneAdmin runs radosgw-admin in any zone.
  6. Clean up: terminate the fixture before the clusters; a non-nil *Fixture returned with an error must still be terminated.
ms, err := multisite.Run(ctx, ceph.DefaultImage, multisite.WithConfig(multisite.Config{
	Source: source, Destination: destination,
	SourceZone: "east", DestinationZone: "west",
}))
if ms != nil {
	t.Cleanup(func() {
		ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
		defer cancel()
		if err := ms.Terminate(ctx); err != nil {
			t.Error(err)
		}
	})
}
if err != nil {
	t.Fatal(err)
}
eastEndpoint, err := ms.Source.S3Endpoint(ctx)
if err != nil {
	t.Fatal(err)
}
// Write through east, then wait for metadata in west before writing objects
// into a new bucket, and poll west for the objects.

Watching sync

To see a zone fall behind and catch up while a fault runs, ms.WatchSync(ctx, cfg, "west", "east") polls the same observation as SyncStatus, and group.WatchBucketSync(ctx, cfg, "pipe", "east", "west") the same as BucketSyncStatus. Each sends changes on one channel, 5 seconds apart by default; one read takes over a second. Receive the baseline before you stop a gateway or cut the zone link. A zone event's Ready is what WaitSyncReady waits for, for the selected sources only; a bucket event's Status.CaughtUp is what WaitBucketSyncReady waits for.

Each read takes the fixture's topology lock and releases it before the event is sent, so the test can stop gateways, fault the link or read other status while the watch runs. A failed read, a gateway that cannot be reached and a zone that has not imported the current period are WatchGap events that keep the partial report.

The baseline pins what the watch observes, and these end it instead of reading as progress:

  • a new committed period or realm epoch, which any sync policy or topology commit makes; start a new watch for the new period;
  • for a bucket watch, a bucket recreated under the same name, and the group changed through the fixture or outside it.

group.WatchBucketPolicy(ctx, cfg, "west") reports the stages of the read WaitBucketPolicyReady makes, 2 seconds apart by default: the committed period, the group's policy and the referenced bucket instances imported into the zone, and Status.Imported once all three are. Start it after the policy change; it ends with a nil cause after it has sent the imported status, and an imported baseline is its only event. A referenced bucket that the zone has not imported yet is that stage, BucketsImported false, in a successful read, so the watch starts however long the zone's metadata sync takes; a read that fails is retried within QueryTimeout for the baseline and is a WatchGap later. The watch keeps the committed period, the group's policy, the zone and the bucket instances it captured: a new committed period, a changed group, a group removed after the watch started, a detached zone and a recreated bucket end it.

A bucket group's Remove takes effect without a commit, and the destination imports it as any other change of the bucket's metadata. Call WaitBucketPolicyReady or WatchBucketPolicy on the removed handle to wait for that import: it is complete when the destination's instance of the group's bucket no longer carries the group. The destination must be one the group's pipes selected. Every policy import, a removal included, also needs the destination's instance at an object version (the metadata ver) no older than the master's when the observation began. A destination snapshot from before the group was created has no group either, and only that version tells the two apart. Write objects that the removal must exclude only after that.

Terminate ends a watch with ceph.ErrInvalidRequest, so stop it first. No watch proves that a particular object arrived: compare the objects through the other zone. The lifecycle that every Watch shares (baseline, gaps, overflow and Stop) is in watching state.

Incremental sync timing

On qualified Ceph 20.2.4 gateways, opt in before the realm starts:

multisite.WithConfig(multisite.Config{
    Source: source, Destination: destination,
    Timing: multisite.TimingProfile{IncrementalSync: true},
})

The same Timing field is available in TopologyConfig. It sets rgw_meta_sync_poll_interval and rgw_data_sync_poll_interval to one second, instead of the twenty-second caught-up default, as arguments of the owned gateways, including later zones. Native notifications can wake these loops earlier, so the saving per operation is not fixed. Default fixtures keep native timing. The command-line values take precedence over central and local configuration; the gateway lifetime owns the condition, so termination ends it without a persistent configuration override. A partial fixture returned with an error still needs termination.

This targets already-caught-up incremental polling. Keep metadata readiness and client byte checks; bootstrap, new-bucket initial sync, lease ownership, failure backoff, retries, readiness timeouts and fault windows are separate conditions. See Fixture daemon timing for what every timing profile shares.

Sync lease period

Fixture gateways start with rgw_sync_lease_period 30 instead of the native 120 seconds, so that sync resumes soon after a gateway restart: a killed gateway's sync shard leases stay taken until they expire. A gateway renews each lease every half period and counts it as lost after one period without a renewal, and it renews none while its reads of a source zone that does not answer wait. A zone link cut or a paused source gateway that lasts longer than about 15 seconds can therefore cost a zone's shards their leases, and with them reach the native defect described under known release issues, which leaves the zone behind until its gateway restarts.

A test that keeps a zone from answering for longer selects a period of more than twice that time, counted until the gateway's connections work again (TCP retransmits after a cut can take as long again as the cut):

multisite.WithTopology(multisite.TopologyConfig{
    Zones:  zones,
    Timing: multisite.TimingProfile{SyncLeasePeriod: 5 * time.Minute},
})

The period is a whole number of seconds from 30 to 238609294 (the most the gateway's 32-bit lease arithmetic holds), on every supported release; it is an argument of the owned gateways, including later zones, and ends with them. Sync after a restart of such a gateway can stall for one period. IncrementalSync and SyncLeasePeriod are independent.

Concepts and limits

  • Both zones accept writes and deletes replicate too, so a replica is not a backup. Users and buckets are realm metadata owned by the metadata master: a bucket created through another zone is forwarded to the master, which must be reachable.
  • Replication is asynchronous: allow minutes, especially after a gateway restart (sync leases must expire first). Wait for a new bucket's metadata in every zone before writing objects into it. Resolve S3Endpoint again after any gateway restart.
  • To test a zone outage, stop one gateway, keep working through the other, start it again and wait with WaitSyncReady.
  • A gateway restart in one zone, including placement.Placement.Reload and a stopped zone, interrupts the other zones' data sync with it. On Tentacle 20.2.4 the peer gateway's data-sync thread has intermittently crashed with SIGSEGV (exit 139, not OOM) about one sync backoff after such an outage. This is a crash in RGW itself. The fixture does not restart the crashed gateway or repeat requests, so writes through it fail; check the gateway's state before blaming the restarted zone.
  • The fixture owns its gateways, setup clients and the HTTP bridge between zones; Terminate leaves realm, zone configuration and data in the clusters. Do not edit periods or zones while topology changes or SyncGroup.Apply run. Serialize these period writers with placement.Placement.Apply in the same realm. If a placement update or commit is uncertain, retry the original placement handle successfully before publishing through this fixture.
  • Sync policy: the fixture creates fresh groups only and checks every mutation against the native policy. On Ceph 20.2.4 pipe Priority between overlapping pipes and source-side denial in user mode with an ordinary user have no reliable effect, and S3 PutBucketReplication rejects account-owned requests with NotImplemented. IAM non-root principals and cross-account or cross-tenant principal translation are refused by the helpers.
  • Releases: Tentacle 20.2.4 is the default. On Squid use 19.2.5; 19.2.6 cannot start a secondary zone (see Known release issues). Squid replicates nothing for a user-mode pipe with an account root user, which CreatePipe refuses with ceph.ErrUnsupported.
  • Out of scope: automatic failover, request routing between zones, recovery point guarantees and realistic WAN models. A zone link can still be delayed or made lossy in one direction with ApplyZoneLinkProfile, or cut with InterruptZoneLink.

Reference

Documentation

Overview

Package multisite orchestrates RGW multisite fixtures across independent Ceph clusters: one realm, zones and zonegroups, period transfer, sync status and sync policies.

Run configures a two-zone realm on two existing clusters (WithConfig) or a larger topology (WithTopology). It returns a Fixture that owns the gateways (started with rgw.Start and described by rgw.Gateway), the per-zone setup clients and, in bridge mode, a private HTTP network. The package imports rgw for single-gateway use; rgw does not import this package.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BucketPolicyWatch added in v0.5.0

type BucketPolicyWatch struct {
	// contains filtered or unexported fields
}

BucketPolicyWatch is a running watch of one bucket policy import. It behaves as SyncWatch, except that it ends by itself once it has sent an imported status.

func (*BucketPolicyWatch) Events added in v0.5.0

func (w *BucketPolicyWatch) Events() <-chan BucketPolicyWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchBucketPolicy returns. The channel closes when the watch ends; Stop returns why.

func (*BucketPolicyWatch) Stop added in v0.5.0

func (w *BucketPolicyWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch, as SyncWatch.Stop does: nil also when the watch ended after an imported status. Call it before Terminate.

type BucketPolicyWatchEvent added in v0.5.0

type BucketPolicyWatchEvent struct {
	Sequence          uint64
	Kind              ceph.WatchEventKind
	Started, Finished time.Time
	Status            BucketSyncPolicyStatus
	Err               error
}

BucketPolicyWatchEvent is one event of a BucketPolicyWatch.

Sequence, Kind, Started and Finished are as in SyncWatchEvent. Status is the BucketSyncPolicyStatus of that read; its stages PeriodImported, PolicyImported, BucketsImported and Imported are compared. On a WatchGap Status contains the partial read with Imported false, or the zero value if the query timed out acquiring the topology lock. A referenced bucket that the destination does not know yet is a successful read with BucketsImported false (and PolicyImported false while that bucket is the group's own), not a gap. Two statuses separated by a gap, or by the polling interval, do not establish the stages in between.

type BucketSyncPolicyStatus

type BucketSyncPolicyStatus struct {
	RealmID, PeriodID, ZonegroupID, GroupID, Zone, ZoneID string
	RealmEpoch                                            uint64
	Bucket                                                SyncBucketIdentity
	Buckets                                               []SyncBucketIdentity
	PeriodImported, PolicyImported, BucketsImported       bool
	Imported                                              bool
}

BucketSyncPolicyStatus describes the captured master's configuration and its import into one selected destination. Buckets contains copied, distinct native instances required by pipes selecting that destination. PolicyImported reports that the destination's instance of Bucket carries the captured group, or no such group for a removed handle, at an object version no older than the master's instance had when the observation began. It contains no policy maps, credentials, object names or sync markers.

type BucketSyncStatus

type BucketSyncStatus struct {
	RealmID, PeriodID, ZonegroupID         string
	SourceZone, SourceZoneID, Zone, ZoneID string
	SourceBucket, DestinationBucket        SyncBucketIdentity
	RealmEpoch                             uint64
	State                                  string
	Shards, BehindShards                   int
	CaughtUp                               bool
}

BucketSyncStatus observes the bucket tuple captured by an owned pipe. Native RGW combines comparisons from matching pipes; this does not attest a particular pipe's activation/filter/user mode. State is sanitized: incremental, init, full, stopped, error, or disabled_or_no_sources. Native markers and error text are never returned.

type BucketSyncWatch added in v0.5.0

type BucketSyncWatch struct {
	// contains filtered or unexported fields
}

BucketSyncWatch is a running watch of one bucket checkpoint. It behaves as SyncWatch.

func (*BucketSyncWatch) Events added in v0.5.0

func (w *BucketSyncWatch) Events() <-chan BucketSyncWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchBucketSync returns. The channel closes when the watch ends; Stop returns why.

func (*BucketSyncWatch) Stop added in v0.5.0

func (w *BucketSyncWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch, as SyncWatch.Stop does. Call it before Terminate.

type BucketSyncWatchEvent added in v0.5.0

type BucketSyncWatchEvent struct {
	Sequence          uint64
	Kind              ceph.WatchEventKind
	Started, Finished time.Time
	Status            BucketSyncStatus
	Err               error
}

BucketSyncWatchEvent is one event of a BucketSyncWatch.

Sequence, Kind, Started and Finished are as in SyncWatchEvent. Status is the BucketSyncStatus of that read, compared whole; on a WatchGap it is the partial report of the failed read, with CaughtUp false. Two statuses separated by a gap, or by the polling interval, do not establish the transitions in between.

type Config

type Config struct {
	Source, Destination                           *ceph.Container
	Realm, Zonegroup, SourceZone, DestinationZone string
	// SetupImage explicitly selects the RGW administration runtime.
	SetupImage string
	// Timing selects startup polling and the sync lease period for every
	// gateway this fixture owns.
	Timing TimingProfile
	// contains filtered or unexported fields
}

Config describes a two-zone realm in fresh test clusters. The source is the metadata master; both zones accept object writes. Setup clients use each cluster's RGW image unless SetupImage overrides it.

Realm and the zone names default to generated names and Zonegroup to us-east-1; set the zone names when the test refers to zones by name (sync policies, ZoneAdmin, SyncStatus).

type DataSyncStatus

type DataSyncStatus struct {
	SourceZone, SourceZoneID              string
	Syncing, CaughtUp                     bool
	Shards, IncrementalShards, FullShards int
	BehindShards, RecoveringShards        int
}

DataSyncStatus summarizes the native comparison with one source's log. CaughtUp requires a running sync, complete incremental shards, no recovering shards and the native remote-log comparison. Disabled pipes do not count as caught up merely because an object's absence is expected.

type Fixture

type Fixture struct {
	Source, Destination                      *rgw.Gateway
	RealmID, SourceZoneID, DestinationZoneID string
	// contains filtered or unexported fields
}

Fixture owns one gateway and CLI client per zone: two from Run, and one more for each WithTopology zone, AddZone or AddZonegroup. Bridge mode adds a private HTTP network; host mode uses allocated gateway ports on the Docker host. Source and Destination expose matching ordinary S3 test credentials.

func Run

Run configures native RGW multisite on existing clusters. Exactly one WithConfig or WithTopology among opts selects the realm; the other customizers apply to every gateway, last. Do not replace required networking or commands. A non-nil result returned with an error must still be terminated. The clusters must not already serve standalone RGW traffic. All clusters must use the same network mode. Host mode advertises the configured daemon-host addresses, which must be mutually reachable. img is the gateway image and is required; setup clients use each cluster's RGW image unless SetupImage explicitly overrides it.

With WithTopology the initial master becomes Source and the first remaining zone becomes Destination; Zones returns all sites.

Use fresh clusters from ceph.Run, without gateways or RGW configuration. WithConfig creates one ordinary S3 test user on the master; both gateways expose its keys in AccessKey and SecretKey, and their Region is the zonegroup name. The two-zone setup, and each further WithTopology zone, is bounded by six minutes in addition to ctx. A secondary zone cannot start on Ceph 19.2.6, whose master gateway refuses its realm pull; use 19.2.5 (see the repository's known release issues guide).

Timing.IncrementalSync fixes qualified idle polling for each owned gateway from startup, including subsequently added zones. It is qualified only for Ceph 20.2.4 and preserves the fixture's sync lease duration. Timing.SyncLeasePeriod replaces that duration on every release.

func (*Fixture) AddZone

func (f *Fixture) AddZone(ctx context.Context, image string, config ZoneConfig, opts ...testcontainers.ContainerCustomizer) (*Zone, error)

AddZone joins another fresh storage cluster to the existing realm/zonegroup. It selects the metadata master from the current committed period, verifies every registered zone's identity and endpoint, and refreshes the period in all sites. Existing running gateways restart to load the final period, retaining their listener ports and storage; fenced gateways stay stopped unless an earlier reload of theirs was interrupted. The new gateway starts with its committed endpoint and skips this reload. Running secondaries finish initial metadata full sync before reload. Complete any external master transition first, or keep the previous master fenced. Refresh S3Endpoint afterwards because bridge-mode published ports may change. A non-nil partial result remains owned on error; terminate the fixture if initialization fails rather than adopting an external zone. image selects the new gateway. Its setup client uses the new cluster's rgw image unless the fixture was configured with SetupImage.

func (*Fixture) AddZonegroup

func (f *Fixture) AddZonegroup(ctx context.Context, image string, config ZonegroupConfig, opts ...testcontainers.ContainerCustomizer) (*Zonegroup, error)

AddZonegroup adds a non-master region with one or more fresh independent Ceph clusters. MasterZone selects its local master. The realm metadata master is unchanged. Metadata is shared across regions; object data replicates within each region. Existing S3 endpoint mappings must be refreshed after reload. On error, any newly created groups/zones/containers remain owned for Terminate. image selects the new gateways. Setup clients use each new cluster's rgw image unless the fixture was configured with SetupImage.

func (*Fixture) ApplyZoneLinkProfile

func (f *Fixture) ApplyZoneLinkProfile(ctx context.Context, zoneName, peerZone string, delay fault.NetworkProfile) (*fault.NetworkProfileLease, error)

ApplyZoneLinkProfile applies netem shaping, below RGW, to packets that the gateway of zoneName sends to the gateway of peerZone on the private multisite bridge: to the peer gateway's bridge address and the port of its peer endpoint. Those carry the requests that zoneName makes of peerZone, its sync reads and its notifications. Connections that peerZone opens to zoneName answer from zoneName's own port and stay unselected, so peerZone keeps syncing from zoneName; the zones' local endpoints and other links are untouched. It is fault.ApplyLinkProfile with the opt-in netem helper: delay names the helper and shaping fields, and leaves Plane, Peer, PeerPort and Peers empty, which the fixture fills in. The fixture restores the lease when it terminates; restore it earlier to end the fault.

func (*Fixture) CreateSyncGroup

func (f *Fixture) CreateSyncGroup(ctx context.Context, scope SyncPolicyScope, config SyncGroupConfig) (*SyncGroup, error)

CreateSyncGroup stores a fresh empty group on the current owned metadata master. Zonegroup changes require SyncGroup.Apply; bucket changes are dynamic native metadata updates and need no period commit. A partial unconfirmed handle cannot remove uncertain native configuration. Use Admin inspection or terminate the disposable cluster after an ambiguous creation failure.

func (f *Fixture) InterruptZoneLink(ctx context.Context, zoneName string) (*fault.NetworkInterruption, error)

InterruptZoneLink disconnects a gateway from the private multisite HTTP bridge. Its local Ceph/S3 endpoint stays attached, allowing local operations while cross-zone synchronization is interrupted. Restore reconnects the original peer IP/aliases; realm/period membership is unchanged.

func (*Fixture) PullDestinationPeriod

func (f *Fixture) PullDestinationPeriod(ctx context.Context) error

PullDestinationPeriod imports the source's realm and current committed period into the destination cluster, including the current-period pointer and reflected zonegroups. Use it to recover a fenced destination after a planned failback; promotion and gateway restart remain caller decisions.

It waits for other topology operations within ctx. Calling it on a terminated fixture returns ceph.ErrInvalidRequest.

func (*Fixture) PullSourcePeriod

func (f *Fixture) PullSourcePeriod(ctx context.Context) error

PullSourcePeriod imports the destination's realm and current committed period into the source cluster, using private system-user credentials. Native realm pull updates the realm's current-period pointer and reflects its zonegroups; period pull alone only stores the fetched period. After a metadata master change, fence the former master before calling this method and restart its gateway afterwards. This does not promote a zone or restart a gateway automatically.

It waits for other topology operations within ctx. Calling it on a terminated fixture returns ceph.ErrInvalidRequest.

func (*Fixture) RemoveZone

func (f *Fixture) RemoveZone(ctx context.Context, name string) error

RemoveZone detaches one owned non-master zone from its zonegroup, commits and propagates the new period, then removes that zone's owned gateway and CLI. The realm, zone configuration, users, pools and stored objects remain in its storage cluster; this is not a destructive zone/pool purge. Detached names and clusters cannot be adopted through AddZone. The final zone of a group and any current realm/group master must first be replaced or promoted explicitly. Partial period propagation/container cleanup can be retried with the same name. Terminate also removes any remaining owned containers. Remaining gateway application endpoints may change on bridge-mode reload.

func (*Fixture) SyncStatus

func (f *Fixture) SyncStatus(ctx context.Context, zoneName string) (SyncStatus, error)

SyncStatus observes native metadata/data readiness in an attached owned zone. Native human sync status is required for remote-log comparison even when the CLI is passed --format json. The structured metadata state separately guards period/realm epoch and every shard. This is a point-in-time observation; applications should quiesce writes and still verify their own S3 payloads.

func (*Fixture) Terminate

func (f *Fixture) Terminate(ctx context.Context, opts ...testcontainers.TerminateOption) error

Terminate removes owned gateways, CLI clients and the HTTP bridge. Realm, zone configuration and data remain; neither Ceph cluster is terminated.

func (*Fixture) WaitSyncReady

func (f *Fixture) WaitSyncReady(ctx context.Context, zoneName string, sourceZones ...string) (SyncStatus, error)

WaitSyncReady waits up to four minutes, bounded by the caller's context, for metadata and the explicitly requested source-zone data streams to catch up. No sources means metadata-only readiness. When the wait ends unready it returns the latest complete observation, and its error reports that observation and the latest attempt's error. Native sync leases survive abrupt gateway restarts until expiration, so this method does not change lease TTLs, unlock objects or restart daemons to shorten normal recovery.

Wait for metadata in every zone after creating a bucket and before writing objects into it. Otherwise RGW can lose those objects in another zone (seen with Tentacle 20.2.4): that zone's data sync stores the new bucket's metadata itself, a second data sync shard may sync the bucket before its index exists, and RGW takes the failed index updates for deleted objects; the zone then reports caught up while the objects stay missing until they change again. After an abrupt gateway stop the old process's sync leases must expire (30 seconds for fixture gateways), and sync polls every 20 seconds by default: use deadlines of several minutes.

func (*Fixture) WatchSync added in v0.5.0

func (f *Fixture) WatchSync(ctx context.Context, cfg ceph.WatchConfig, zoneName string, sourceZones ...string) (*SyncWatch, error)

WatchSync polls the sync state of one attached owned zone, the same observation as SyncStatus, and reports its changes. sourceZones selects the data sources the watch compares and Ready covers, as for WaitSyncReady: no sources means metadata only. The default WatchConfig.Interval is 5 seconds.

Before it returns, WatchSync makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read is the baseline: the first event, already buffered, and the identities the watch keeps for its lifetime: the realm, zonegroup and zone IDs and the committed period ID and realm epoch. Receive the baseline before triggering the change to observe.

Each read takes the fixture's topology lock for one bounded observation, as SyncStatus does, and holds it neither while the event is sent nor while the watch waits for the next read, so the test may change the topology, stop gateways or faults while the watch runs. A read compares the metadata status and, for each selected source, whether it is reported and its data status; the other sources and the zone's MetadataMaster flag are reported but not compared, so their changes alone produce no event. A failed read, including a zone that has yet to import the current period or a native sync-status header whose realm, zonegroup or zone differs from the fixture's recorded identity, produces a WatchGap. The next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, Terminate (ceph.ErrInvalidRequest), a full buffer (ceph.ErrWatchOverflow), a zone or selected source that is no longer an attached owned zone of the zonegroup, and a committed period or realm epoch other than the baseline's, as any sync policy or topology commit makes, even while the zone has not imported it yet: watch the new period with a new WatchSync.

Ready reports the native comparison at that read. It does not prove that a particular object arrived or that sync completes under continuous writes: verify the application's objects separately.

func (*Fixture) ZoneAdmin

func (f *Fixture) ZoneAdmin(ctx context.Context, name string, args ...string) ([]byte, error)

ZoneAdmin runs native radosgw-admin in any owned zone, including a partial zone whose CLI client started successfully. Do not edit period/zone state concurrently with AddZone or external metadata master transitions.

It passes the zone's realm, zonegroup and zone but does not add --format. Arguments are checked as rgw.Gateway.Admin checks them: control characters and flags that would change the zone scope or Ceph connection are refused with ceph.ErrInvalidRequest. A failure keeps a *ceph.NativeFailure with the exit code and errno but no reason, because the native output can include secret keys.

func (*Fixture) Zonegroups

func (f *Fixture) Zonegroups() []Zonegroup

Zonegroups returns a sorted snapshot of the owned groups and their attached zones. Detached zones are omitted after their removal period is committed.

func (*Fixture) Zones

func (f *Fixture) Zones() []Zone

Zones returns owned zone descriptors sorted by name, including partial zones returned by a failed AddZone. It does not expose private system credentials.

type MetadataSyncStatus

type MetadataSyncStatus struct {
	State, PeriodID           string
	RealmEpoch                uint64
	Shards, IncrementalShards int
	CaughtUp                  bool
}

MetadataSyncStatus includes only native state/identity/shard counts. It omits metadata keys, markers and credentials from native diagnostic output.

type Option

type Option struct {
	// contains filtered or unexported fields
}

Option carries the realm configuration of a fixture to Run: a two-zone Config (WithConfig) or a larger TopologyConfig (WithTopology). It is not a container customization: its Customize does nothing, so it never reaches a gateway.

func WithConfig

func WithConfig(config Config) Option

WithConfig carries a two-zone realm on two existing clusters to Run.

func WithTopology

func WithTopology(config TopologyConfig) Option

WithTopology carries a realm of at least two zones, in one or several zonegroups, to Run. The zone lists are copied.

func (Option) Customize

Customize implements testcontainers.ContainerCustomizer.

type SyncBucketIdentity

type SyncBucketIdentity struct{ Name, Tenant, ID string }

SyncBucketIdentity fixes a bucket namespace and native instance. Recreating its name cannot silently change the bucket selected by an owned pipe.

type SyncBucketSelector

type SyncBucketSelector struct{ Name, Tenant string }

SyncBucketSelector identifies a preexisting bucket. Its current native instance ID is captured and stored in the pipe, preventing a recreated name from silently becoming a new replication source/destination. Nil selectors retain Ceph's current-bucket wildcard behavior. Same-zone replication is not supported by native RGW multisite.

type SyncFlowConfig

type SyncFlowConfig struct {
	ID                          string
	Zones                       []string
	SourceZone, DestinationZone string
}

SyncFlowConfig selects either a named symmetrical Zones set, or a directional SourceZone/DestinationZone pair. Directional flows have no native ID: leave ID and Zones empty. Zone names must be attached and owned by this fixture in the selected group. Bucket policies may only narrow the flow permitted by their parent zonegroup.

type SyncGroup

type SyncGroup struct {
	// contains filtered or unexported fields
}

SyncGroup is a fresh group created by this fixture. Copies share native readback and lifecycle state. IDs on unrelated existing groups are never adopted. Native edits must not race these methods.

A group is stored on the zonegroup or on one existing bucket. Every mutation compares the native policy before and after the change and refuses unrelated or external changes; if the reply of a confirmed mutation is lost, the next call on the same handle reconciles the pending change first. Group mutations are bounded by two minutes.

func (*SyncGroup) Apply

func (g *SyncGroup) Apply(ctx context.Context) error

Apply publishes only this owned zonegroup policy change, pulls the resulting period into every owned zone and reloads running gateways. Resolve their S3Endpoint again afterwards: a bridge gateway's mapped port can change. Bucket policy is dynamic and this method only checks its owned native snapshot. Unrelated current, locally pending or staging-period edits are refused before period update can overwrite them. Partial publication/reload is not rolled back; retry the same handle after resolving an external failure. Serialize zonegroup publication with Placement.Apply in the same realm. After an uncertain placement update or commit, successfully retry the original placement handle before calling this method.

func (*SyncGroup) BucketSyncStatus

func (g *SyncGroup) BucketSyncStatus(ctx context.Context, pipeID, sourceZone, destinationZone string) (BucketSyncStatus, error)

BucketSyncStatus reads native per-bucket remote-log comparison for an owned unchanged group/pipe and two explicit owned zones. Wildcard bucket selectors require a bucket-scoped group. Concrete bucket instance IDs remain fixed. A caught-up checkpoint does not prove filter semantics or copied payloads; applications should quiesce writes and verify their S3 bytes separately.

On Ceph 19 radosgw-admin ignores --format json for bucket sync status and prints text; this method parses it into the same structure and reports unknown lines as errors.

func (*SyncGroup) CreateFlow

func (g *SyncGroup) CreateFlow(ctx context.Context, config SyncFlowConfig) error

CreateFlow adds a fresh flow to an owned unchanged group. Native directional identity is the exact source/destination pair, not flow-id.

func (*SyncGroup) CreatePipe

func (g *SyncGroup) CreatePipe(ctx context.Context, config SyncPipeConfig) error

CreatePipe adds a fresh identity-checked pipe. Zonegroup policy forms an upper bound for a bucket policy; a stored pipe alone does not prove effective replication. Verify positive/negative writes through the S3 client.

func (SyncGroup) GoString

func (g SyncGroup) GoString() string

GoString formats the sync group like String.

func (*SyncGroup) ID

func (g *SyncGroup) ID() string

ID returns the native sync group ID, or "" for a nil handle.

func (*SyncGroup) Remove

func (g *SyncGroup) Remove(ctx context.Context) error

Remove removes only this confirmed group's unchanged native policy. Bucket/object data remains. Zonegroup removal needs Apply afterwards to publish it; removal is idempotent across handle copies.

func (*SyncGroup) RemoveFlow

func (g *SyncGroup) RemoveFlow(ctx context.Context, config SyncFlowConfig) error

RemoveFlow removes only a flow present in the owned group's exact native snapshot. Symmetrical removal deletes the entire named set; directional removal selects the native pair. It never changes another group's policy.

func (*SyncGroup) RemovePipe

func (g *SyncGroup) RemovePipe(ctx context.Context, id string) error

RemovePipe removes one pipe from an owned group. The pipe must exist in the group's native policy, and the removal must leave the rest of the group unchanged. Objects that were already replicated stay. Like the other policy changes, it confirms the master's stored policy, not its asynchronous import into peers.

func (*SyncGroup) SetPipePrefix

func (g *SyncGroup) SetPipePrefix(ctx context.Context, id, prefix string) error

SetPipePrefix changes only the selected owned pipe's source prefix. Empty prefix removes the filter. It affects future replication selection; historical backfill and deletion of previously copied objects are not promised. For a bucket-scoped group, wait for WaitBucketPolicyReady at the intended destination before writing objects that depend on the new filter. This method confirms the master's stored policy, not its asynchronous import into peers.

func (*SyncGroup) SetStatus

func (g *SyncGroup) SetStatus(ctx context.Context, status SyncGroupStatus) error

SetStatus changes an owned group's allowed/enabled/forbidden state.

After forbidden to enabled, earlier log entries may stay unprocessed until an explicit bucket sync run through ZoneAdmin.

func (SyncGroup) String

func (g SyncGroup) String() string

String names the sync group.

func (*SyncGroup) WaitBucketPolicyReady

func (g *SyncGroup) WaitBucketPolicyReady(ctx context.Context, destinationZone string) (BucketSyncPolicyStatus, error)

WaitBucketPolicyReady waits up to four minutes, bounded by ctx including mutex queue time, for an unchanged owned bucket group's metadata to import into an explicit attached destination selected by its pipes. On a handle removed through the fixture it waits for the removal instead: the destination's instance of the group's bucket no longer carries the group, and Buckets is that bucket alone. The destination must be one the pipes selected when the group was removed. It checks the exact committed period, source-scope group snapshot at an instance object version no older than the master's at admission, and all referenced source/destination bucket instances needed by that destination. Admission reads the master once and returns any error immediately. After admission, missing or older local metadata and transient read failures are retried. Recreated instances and changed master policy are refused. Allowed/forbidden policies can also be observed as imported. WatchBucketPolicy reports the stages of the same read.

This read-only prerequisite does not attest effective discovery hints, data checkpoints, payloads or permissions, or replay objects written before the policy was imported. It does not pull periods, restart gateways or mutate replication state. Use it after creating pipes and before selected writes, and after Remove and before writes that the removal must exclude.

Zonegroup groups, pending mutations and zones outside the pipes' destinations are refused.

func (*SyncGroup) WaitBucketSyncReady

func (g *SyncGroup) WaitBucketSyncReady(ctx context.Context, pipeID, sourceZone, destinationZone string) (BucketSyncStatus, error)

WaitBucketSyncReady waits up to four minutes, bounded by the caller context, for the captured pipe/bucket checkpoint. Disabled, full-sync, missing sources and native per-source errors never satisfy readiness even when CLI exit is 0. It changes no sync state, leases, permissions or object data. When the wait ends without readiness, the returned status is the last one that was read, so its lag survives a later attempt that failed or was canceled.

func (*SyncGroup) WatchBucketPolicy added in v0.5.0

func (g *SyncGroup) WatchBucketPolicy(ctx context.Context, cfg ceph.WatchConfig, destinationZone string) (*BucketPolicyWatch, error)

WatchBucketPolicy polls the import of this owned bucket group's policy into an explicit attached destination selected by its pipes, the same observation as WaitBucketPolicyReady, and reports its stages as they change: the captured committed period, the group's policy and every referenced bucket instance imported, and all of them together as Imported. The default WatchConfig.Interval is 2 seconds.

Under ctx and QueryTimeout, WatchBucketPolicy first admits the request once, as WaitBucketPolicyReady does. An error reading the master during admission returns immediately. After admission, a failed first sample is retried once a second within QueryTimeout, releasing the topology lock between attempts. A fatal error or expiry before the first successful sample starts no watch. A destination that has not imported a referenced bucket yet is not a failed sample: the baseline reports that stage, however long the destination's metadata sync takes. The watch keeps the identities captured at admission: the master's committed period, the group's policy as the fixture confirmed it, the destination zone and the referenced source and destination bucket instances. Start the watch after the policy change, as you would WaitBucketPolicyReady; a baseline that is already imported is the only event. A watch of a handle removed through the fixture reports the import of the removal, as WaitBucketPolicyReady does.

Each read takes the fixture's topology lock for one bounded observation and holds it neither while the event is sent nor between reads, so the test may stop gateways or fault the zone link while the watch runs. A failed read, including a gateway that cannot be reached and a bucket that is deleted but not recreated, produces a WatchGap and the watch continues; the next successful read is WatchResumed. The watch ends with a nil cause after it has sent the first imported status. These end it instead: ctx ending, Stop, Terminate (ceph.ErrInvalidRequest), a full buffer (ceph.ErrWatchOverflow), the group changed through the fixture or a pending change on it, the group removed through the fixture after the watch started, the destination no longer attached, the group's native policy changed on the master, a referenced bucket recreated under the same name, a destination period of another realm, and a committed period on the master other than the captured one, as any sync policy or topology commit makes: watch the new policy with a new WatchBucketPolicy. A destination that has not yet imported the captured period is the first stage, not an error.

Imported is the prerequisite WaitBucketPolicyReady waits for. It does not attest data checkpoints, payloads or permissions: verify the application's objects and the bucket checkpoint separately.

func (*SyncGroup) WatchBucketSync added in v0.5.0

func (g *SyncGroup) WatchBucketSync(ctx context.Context, cfg ceph.WatchConfig, pipeID, sourceZone, destinationZone string) (*BucketSyncWatch, error)

WatchBucketSync polls the bucket checkpoint of one pipe of this owned group between two owned zones, the same observation as BucketSyncStatus, and reports its changes. The default WatchConfig.Interval is 5 seconds.

Before it returns, WatchBucketSync makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read resolves the checkpoint once, as WaitBucketSyncReady does, and the watch keeps it for its lifetime: the group's policy as the fixture confirmed it, the two zones, the source and destination bucket instances, and the committed period ID and realm epoch of the baseline. Receive the baseline before triggering the change to observe.

Each read takes the fixture's topology lock for one bounded observation and holds it neither while the event is sent nor between reads. A read first checks both bucket instances, then reads the checkpoint as BucketSyncStatus does. A failed read, including a bucket that is deleted but not recreated and an endpoint that has not imported the current period, produces a WatchGap and the watch continues; the next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, Terminate (ceph.ErrInvalidRequest), a full buffer (ceph.ErrWatchOverflow), the group removed or changed through the fixture or a pending change on it, the group's native policy changed outside the fixture, a bucket recreated under the same name, and a committed period or realm epoch other than the baseline's, even when that period no longer publishes the pipe or an endpoint has not imported it yet.

CaughtUp reports the native comparison at that read. Native RGW combines comparisons from matching pipes, so it does not attest the selected pipe's filters, and it does not prove that a particular object arrived: verify the application's objects separately.

type SyncGroupConfig

type SyncGroupConfig struct {
	ID     string
	Status SyncGroupStatus
}

SyncGroupConfig selects a new sync group. ID has no spaces or ,/:$ and at most 128 characters; empty generates one. Status is required.

type SyncGroupStatus

type SyncGroupStatus string

SyncGroupStatus separates permission from activation. Allowed permits a bucket policy to enable its subset; Enabled activates the group's pipes; Forbidden takes precedence over other matching enabled groups.

const (
	// SyncAllowed permits replication without activating it; a bucket group
	// may enable a subset.
	SyncAllowed SyncGroupStatus = "allowed"
	// SyncEnabled activates the group's pipes.
	SyncEnabled SyncGroupStatus = "enabled"
	// SyncForbidden blocks replication and takes precedence over other
	// matching enabled groups.
	SyncForbidden SyncGroupStatus = "forbidden"
)

type SyncObjectTag

type SyncObjectTag struct{ Key, Value string }

SyncObjectTag matches an exact key/value. Multiple pairs are ORed and repeated keys with different values are permitted. Native CLI comma parsing cannot represent comma-containing keys/values; '=' is only allowed in values.

Key is 1 to 128 characters and Value up to 256.

type SyncPipeConfig

type SyncPipeConfig struct {
	ID                              string
	SourceZones, DestinationZones   []string
	Prefix                          string
	SourceBucket, DestinationBucket *SyncBucketSelector
	Tags                            []SyncObjectTag
	Priority                        int32
	DestinationOwner, User          *rgw.User
	DestinationStorageClass         string
	// Storage-class translation requires a concrete destination bucket and a
	// confirmed native placement for each concrete DestinationZones entry.
	// The caller must publish/import/reload those placements before traffic.
	DestinationPlacements map[string]*placement.Placement
}

SyncPipeConfig selects owned zone names or a sole "*" on each side. Nil bucket selectors use the current bucket; an empty Prefix selects all keys. Source tags are ORed; when a prefix is also supplied both must match. User mode uses a confirmed ordinary user, or a confirmed account root with concrete source/destination buckets owned by that same account and no DestinationOwner. Principal and selected bucket tenants must match. IAM nonroot, cross-account and cross-tenant principal translation remain unproven and are refused. This helper never creates S3 buckets or grants access.

ID is unique in the group and required. In system mode (no User) concrete source and destination buckets may belong to different tenants when no owner or user translation is requested. RGW preserves source object ACLs, so the destination bucket owner may not be the principal that can read the replicated objects.

On the default image (Ceph 20.2.4) two stored parameters have no reliable effect, so a stored policy and a caught-up checkpoint are no evidence that they work: Priority does not choose between overlapping pipes with the same prefix (the native rebuild order decides which pipe's parameters, such as the destination storage class, apply), and user mode with an ordinary user does not test source-side denial (RGW authorises the replication GET on the system path, so objects replicate even when the principal is denied s3:GetObject). User mode with an account root user honours a destination bucket policy that denies s3:ReplicateObject. On Ceph 19 a user-mode pipe with an account root user replicates nothing, so CreatePipe refuses it with ceph.ErrUnsupported when any zone runs Ceph 19; the *ceph.OperationError names the zone and its version.

type SyncPolicyScope

type SyncPolicyScope struct{ Zonegroup, Bucket, Tenant string }

SyncPolicyScope selects an owned zonegroup, or an existing bucket within that group. An empty Zonegroup selects the original source's group. Bucket policy fixes the native bucket instance ID and rejects a recreated name.

type SyncStatus

type SyncStatus struct {
	RealmID, PeriodID, Zone, ZoneID, ZonegroupID string
	RealmEpoch                                   uint64
	MetadataMaster                               bool
	Metadata                                     MetadataSyncStatus
	DataSources                                  []DataSyncStatus
}

SyncStatus is the metadata and data sync state of one attached zone, as Fixture.SyncStatus reads it.

type SyncWatch added in v0.5.0

type SyncWatch struct {
	// contains filtered or unexported fields
}

SyncWatch is a running watch of one zone's sync state. One goroutine reads the zone and sends events; Events closes after that goroutine has stopped using the fixture. Stop ends the watch and waits for it.

func (*SyncWatch) Events added in v0.5.0

func (w *SyncWatch) Events() <-chan SyncWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchSync returns. The channel closes when the watch ends; Stop returns why.

func (*SyncWatch) Stop added in v0.5.0

func (w *SyncWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchSync context ended, a ceph.ErrWatchOverflow error, or the read error that ended it. Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before Terminate: cancelling the context alone does not wait.

type SyncWatchEvent added in v0.5.0

type SyncWatchEvent struct {
	Sequence          uint64
	Kind              ceph.WatchEventKind
	Started, Finished time.Time
	Status            SyncStatus
	Ready             bool
	Err               error
}

SyncWatchEvent is one event of a SyncWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped. Started and Finished are the local clock around the read, not the time RGW changed state. Status is the SyncStatus of that read; on a WatchGap it is the partial report of the failed read, evidence of what the read got before Err. Ready is the condition WaitSyncReady waits for, for the source zones the watch selected: metadata caught up and every selected source's data caught up. It is false on a gap. Two statuses separated by a gap, or by the polling interval, do not establish the transitions in between.

type TimingProfile added in v0.5.0

type TimingProfile struct {
	// IncrementalSync selects one-second metadata and data incremental sync
	// polling on Ceph 20.2.4. It preserves sync leases and notifications, and
	// does not accelerate initial full synchronization or guarantee an elapsed
	// bound. It is fixed for the lifetime of each owned gateway, including added
	// zones: terminating the fixture ends the condition. It sets no persistent
	// configuration entry and changes no gateway outside the fixture. A native
	// version outside the qualified release returns a qualification error after
	// startup; retain the partial Fixture for termination even on that error.
	IncrementalSync bool
	// SyncLeasePeriod replaces the 30 seconds of rgw_sync_lease_period that
	// the fixture's gateways otherwise start with (the native default is
	// two minutes). A gateway renews each sync shard's lease every half
	// period and counts it as lost after one period without a renewal, and
	// it renews none while its reads of a source zone that does not answer
	// wait. Set it for a test that keeps a zone from answering (a zone link
	// cut, a paused gateway) for longer than half of 30 seconds, to more
	// than twice that time. A killed gateway's leases stay taken until
	// they expire, so sync after a gateway restart can stall for one
	// period. It must be a whole number of seconds, at least 30 and at most
	// 238609294 (the most the gateway's lease arithmetic holds, about seven
	// and a half years). It is fixed for the lifetime of each owned gateway,
	// including added zones, on every release, and sets no persistent
	// configuration entry.
	SyncLeasePeriod time.Duration
}

TimingProfile selects qualified startup conditions for this fixture's gateways. The zero value preserves native default polling.

Example
package main

import (
	"context"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/rgw/multisite"
)

func main() {
	ctx := context.Background()
	source, err := ceph.Run(ctx, ceph.DefaultImage)
	if source != nil {
		defer func() { _ = source.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	destination, err := ceph.Run(ctx, ceph.DefaultImage)
	if destination != nil {
		defer func() { _ = destination.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	link, err := multisite.Run(ctx, ceph.DefaultImage, multisite.WithConfig(multisite.Config{
		Source: source, Destination: destination,
		Timing: multisite.TimingProfile{IncrementalSync: true},
	}))
	if link != nil {
		defer func() {
			cleanup, cancel := context.WithTimeout(context.Background(), time.Minute)
			defer cancel()
			if err := link.Terminate(cleanup); err != nil {
				panic(err)
			}
		}()
	}
	if err != nil {
		panic(err)
	}
	// Prepare metadata readiness barriers before object writes; the native
	// gateways keep their ordinary sync leases and notifications.
}

type TopologyConfig

type TopologyConfig struct {
	Zones                            []ZoneConfig
	Realm, Zonegroup, MetadataMaster string
	SetupImage                       string
	Zonegroups                       []ZonegroupConfig
	MasterZonegroup                  string
	// Timing applies to the initial and subsequently added owned gateways.
	Timing TimingProfile
}

TopologyConfig creates one realm spanning at least two independent zones. Use Zones/Zonegroup/MetadataMaster for a single region, or Zonegroups and MasterZonegroup for several regions with separately selected local masters. MetadataMaster defaults to the first zone; MasterZonegroup to the first group. All clusters use the same network mode and must not serve standalone RGW data. SetupImage optionally overrides every zone's setup client image. Otherwise each client uses its own cluster's RGW image, including added zones.

type Zone

type Zone struct {
	Name, ID, Zonegroup, PeerEndpoint string
	Gateway                           *rgw.Gateway
}

Zone describes one owned multisite zone. PeerEndpoint is reachable by gateways/CLI containers in the fixture's HTTP network or Docker host namespace. Applications obtain their endpoint separately with Gateway.S3Endpoint.

type ZoneConfig

type ZoneConfig struct {
	Name, Zonegroup string
	Cluster         *ceph.Container
}

ZoneConfig assigns one fresh, independent storage cluster to a named zone.

type Zonegroup

type Zonegroup struct {
	Name, ID, MasterZone string
	Zones                []Zone
}

Zonegroup is an owned region snapshot. MasterZone reflects the last topology operation; direct native promotions require a subsequent operation to refresh it. Each zone retains its own storage cluster and daemon endpoint.

type ZonegroupConfig

type ZonegroupConfig struct {
	Name, MasterZone string
	Zones            []ZoneConfig
}

ZonegroupConfig defines a region and its local master zone. MasterZone defaults to the first zone. Object data replicates within each zonegroup; users and bucket metadata share the realm across zonegroups.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL