multisite

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: MIT Imports: 30 Imported by: 0

README

Ceph RGW multisite

Import github.com/JSYoo5B/ceph-testcontainers-go/rgw/multisite to build an RGW multisite realm across independent disposable clusters, so that a test can check how its S3 application behaves with replication: objects written in one zone appearing in another, a zone outage, a planned metadata master failover, selective sync with sync policies. Ordinary gateway use (users, tenants, quotas, placement) stays in rgw. See API purpose and purpose tags.

Order of use

  1. Start fresh clusters with ceph.Run, without gateways or RGW configuration, all in the same network mode.
  2. Build the realm with multisite.Run(ctx, img, multisite.WithConfig(cfg)) for two zones (Source is the metadata master; both accept writes) or multisite.WithTopology(topology) for more zones and zonegroups. img is the gateway image. AddZone, AddZonegroup and RemoveZone change the topology later.
  3. Hand the endpoints to the SUT: each Zone's rgw.Gateway (Source, Destination, Zones()) gives S3Endpoint, Region and the test user's AccessKey / SecretKey. Create users through the master.
  4. Shape replication when the scenario needs it: CreateSyncGroup, then SyncGroup.CreateFlow / CreatePipe / SetStatus and SyncGroup.Apply for zonegroup policy; InterruptZoneLink and ApplyZoneLinkProfile cut or slow the link between zones; PullSourcePeriod / PullDestinationPeriod carry a planned master change.
  5. Observe and verify: WaitSyncReady (metadata and data catch-up), SyncGroup.WaitBucketPolicyReady, SyncGroup.WaitBucketSyncReady, then read the objects through the other zone's endpoint. ZoneAdmin runs radosgw-admin in any zone.
  6. Clean up: terminate the fixture before the clusters; a non-nil *Fixture returned with an error must still be terminated.
ms, err := multisite.Run(ctx, ceph.DefaultImage, multisite.WithConfig(multisite.Config{
	Source: source, Destination: destination,
	SourceZone: "east", DestinationZone: "west",
}))
if ms != nil {
	t.Cleanup(func() {
		ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
		defer cancel()
		if err := ms.Terminate(ctx); err != nil {
			t.Error(err)
		}
	})
}
if err != nil {
	t.Fatal(err)
}
eastEndpoint, err := ms.Source.S3Endpoint(ctx)
if err != nil {
	t.Fatal(err)
}
// Write through east, then wait for metadata in west before writing objects
// into a new bucket, and poll west for the objects.

Concepts and limits

  • Both zones accept writes and deletes replicate too, so a replica is not a backup. Users and buckets are realm metadata owned by the metadata master: a bucket created through another zone is forwarded to the master, which must be reachable.
  • Replication is asynchronous: allow minutes, especially after a gateway restart (sync leases must expire first). Wait for a new bucket's metadata in every zone before writing objects into it. Resolve S3Endpoint again after any gateway restart.
  • To test a zone outage, stop one gateway, keep working through the other, start it again and wait with WaitSyncReady.
  • The fixture owns its gateways, setup clients and the HTTP bridge between zones; Terminate leaves realm, zone configuration and data in the clusters. Do not edit periods or zones while topology changes or SyncGroup.Apply run.
  • Sync policy: the fixture creates fresh groups only and checks every mutation against the native policy. On Ceph 20.2.4 pipe Priority between overlapping pipes and source-side denial in user mode with an ordinary user have no reliable effect, and S3 PutBucketReplication rejects account-owned requests with NotImplemented. IAM non-root principals and cross-account or cross-tenant principal translation are refused by the helpers.
  • Releases: Tentacle 20.2.4 is the default. On Squid use 19.2.5 (19.2.6 cannot set up multisite); Squid replicates nothing for a user-mode pipe with an account root user, which CreatePipe refuses with ceph.ErrUnsupported.
  • Out of scope: automatic failover, request routing between zones, recovery point guarantees and realistic WAN models. A zone link can still be delayed or made lossy in one direction with ApplyZoneLinkProfile, or cut with InterruptZoneLink.

Reference

Documentation

Overview

Package multisite orchestrates RGW multisite fixtures across independent Ceph clusters: one realm, zones and zonegroups, period transfer, sync status and sync policies.

Run configures a two-zone realm on two existing clusters (WithConfig) or a larger topology (WithTopology). It returns a Fixture that owns the gateways (started with rgw.Start and described by rgw.Gateway), the per-zone setup clients and, in bridge mode, a private HTTP network. The package imports rgw for single-gateway use; rgw does not import this package.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BucketSyncPolicyStatus 

type BucketSyncPolicyStatus struct {
	RealmID, PeriodID, ZonegroupID, GroupID, Zone, ZoneID string
	RealmEpoch                                            uint64
	Bucket                                                SyncBucketIdentity
	Buckets                                               []SyncBucketIdentity
	PeriodImported, PolicyImported, BucketsImported       bool
	Imported                                              bool
}

BucketSyncPolicyStatus describes the captured master's configuration and its import into one selected destination. Buckets contains copied, distinct native instances required by pipes selecting that destination. It contains no policy maps, credentials, object names or sync markers.

type BucketSyncStatus 

type BucketSyncStatus struct {
	RealmID, PeriodID, ZonegroupID         string
	SourceZone, SourceZoneID, Zone, ZoneID string
	SourceBucket, DestinationBucket        SyncBucketIdentity
	RealmEpoch                             uint64
	State                                  string
	Shards, BehindShards                   int
	CaughtUp                               bool
}

BucketSyncStatus observes the bucket tuple captured by an owned pipe. Native RGW combines comparisons from matching pipes; this does not attest a particular pipe's activation/filter/user mode. State is sanitized: incremental, init, full, stopped, error, or disabled_or_no_sources. Native markers and error text are never returned.

type Config 

type Config struct {
	Source, Destination                           *ceph.Container
	Realm, Zonegroup, SourceZone, DestinationZone string
	// SetupImage explicitly selects the RGW administration runtime.
	SetupImage string
	// contains filtered or unexported fields
}

Config describes a two-zone realm in fresh test clusters. The source is the metadata master; both zones accept object writes. Setup clients use each cluster's RGW image unless SetupImage overrides it.

Realm and the zone names default to generated names and Zonegroup to us-east-1; set the zone names when the test refers to zones by name (sync policies, ZoneAdmin, SyncStatus).

type DataSyncStatus 

type DataSyncStatus struct {
	SourceZone, SourceZoneID              string
	Syncing, CaughtUp                     bool
	Shards, IncrementalShards, FullShards int
	BehindShards, RecoveringShards        int
}

DataSyncStatus summarizes the native comparison with one source's log. CaughtUp requires a running sync, complete incremental shards, no recovering shards and the native remote-log comparison. Disabled pipes do not count as caught up merely because an object's absence is expected.

type Fixture 

type Fixture struct {
	Source, Destination                      *rgw.Gateway
	RealmID, SourceZoneID, DestinationZoneID string
	// contains filtered or unexported fields
}

Fixture owns one gateway and CLI client per zone: two from Run, and one more for each WithTopology zone, AddZone or AddZonegroup. Bridge mode adds a private HTTP network; host mode uses allocated gateway ports on the Docker host. Source and Destination expose matching ordinary S3 test credentials.

func Run 

func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*Fixture, error)

Run configures native RGW multisite on existing clusters. Exactly one WithConfig or WithTopology among opts selects the realm; the other customizers apply to every gateway, last. Do not replace required networking or commands. A non-nil result returned with an error must still be terminated. The clusters must not already serve standalone RGW traffic. All clusters must use the same network mode. Host mode advertises the configured daemon-host addresses, which must be mutually reachable. img is the gateway image and is required; setup clients use each cluster's RGW image unless SetupImage explicitly overrides it.

With WithTopology the initial master becomes Source and the first remaining zone becomes Destination; Zones returns all sites.

Use fresh clusters from ceph.Run, without gateways or RGW configuration. WithConfig creates one ordinary S3 test user on the master; both gateways expose its keys in AccessKey and SecretKey, and their Region is the zonegroup name. The two-zone setup, and each further WithTopology zone, is bounded by six minutes in addition to ctx. On Ceph 19, 19.2.6 cannot set up multisite: the master gateway rejects realm pull from a secondary with HTTP 403 ("'x-amz-content-sha256' supplied, but not in CanonicalHeaders"), and no setting disables the check; use 19.2.5.

func (*Fixture) AddZone 

func (f *Fixture) AddZone(ctx context.Context, image string, config ZoneConfig, opts ...testcontainers.ContainerCustomizer) (*Zone, error)

AddZone joins another fresh storage cluster to the existing realm/zonegroup. It selects the metadata master from the current committed period, verifies every registered zone's identity and endpoint, and refreshes the period in all sites. Existing running gateways restart to load the final period, retaining their listener ports and storage; fenced gateways stay stopped unless an earlier reload of theirs was interrupted. The new gateway reloads only if its host-mode endpoint changed after startup. Running secondaries finish initial metadata full sync before reload. Complete any external master transition first, or keep the previous master fenced. Refresh S3Endpoint afterwards because bridge-mode published ports may change. A non-nil partial result remains owned on error; terminate the fixture if initialization fails rather than adopting an external zone. image selects the new gateway. Its setup client uses the new cluster's rgw image unless the fixture was configured with SetupImage.

func (*Fixture) AddZonegroup 

func (f *Fixture) AddZonegroup(ctx context.Context, image string, config ZonegroupConfig, opts ...testcontainers.ContainerCustomizer) (*Zonegroup, error)

AddZonegroup adds a non-master region with one or more fresh independent Ceph clusters. MasterZone selects its local master. The realm metadata master is unchanged. Metadata is shared across regions; object data replicates within each region. Existing S3 endpoint mappings must be refreshed after reload. On error, any newly created groups/zones/containers remain owned for Terminate. image selects the new gateways. Setup clients use each new cluster's rgw image unless the fixture was configured with SetupImage.

func (*Fixture) ApplyZoneLinkProfile 

func (f *Fixture) ApplyZoneLinkProfile(ctx context.Context, zoneName, peerZone string, delay fault.NetworkProfile) (*fault.NetworkProfileLease, error)

ApplyZoneLinkProfile applies netem shaping, below RGW, to packets that the gateway of zoneName sends to the gateway of peerZone on the private multisite bridge: to the peer gateway's bridge address and the port of its peer endpoint. Those carry the requests that zoneName makes of peerZone, its sync reads and its notifications. Connections that peerZone opens to zoneName answer from zoneName's own port and stay unselected, so peerZone keeps syncing from zoneName; the zones' local endpoints and other links are untouched. It is fault.ApplyLinkProfile with the opt-in netem helper: delay names the helper and shaping fields, and leaves Plane, Peer, PeerPort and Peers empty, which the fixture fills in. The fixture restores the lease when it terminates; restore it earlier to end the fault.

func (*Fixture) CreateSyncGroup 

func (f *Fixture) CreateSyncGroup(ctx context.Context, scope SyncPolicyScope, config SyncGroupConfig) (*SyncGroup, error)

CreateSyncGroup stores a fresh empty group on the current owned metadata master. Zonegroup changes require SyncGroup.Apply; bucket changes are dynamic native metadata updates and need no period commit. A partial unconfirmed handle cannot remove uncertain native configuration. Use Admin inspection or terminate the disposable cluster after an ambiguous creation failure. 

func (f *Fixture) InterruptZoneLink(ctx context.Context, zoneName string) (*fault.NetworkInterruption, error)

InterruptZoneLink disconnects a gateway from the private multisite HTTP bridge. Its local Ceph/S3 endpoint stays attached, allowing local operations while cross-zone synchronization is interrupted. Restore reconnects the original peer IP/aliases; realm/period membership is unchanged.

func (*Fixture) PullDestinationPeriod 

func (f *Fixture) PullDestinationPeriod(ctx context.Context) error

PullDestinationPeriod imports the source's realm and current committed period into the destination cluster, including the current-period pointer and reflected zonegroups. Use it to recover a fenced destination after a planned failback; promotion and gateway restart remain caller decisions.

func (*Fixture) PullSourcePeriod 

func (f *Fixture) PullSourcePeriod(ctx context.Context) error

PullSourcePeriod imports the destination's realm and current committed period into the source cluster, using private system-user credentials. Native realm pull updates the realm's current-period pointer and reflects its zonegroups; period pull alone only stores the fetched period. After a metadata master change, fence the former master before calling this method and restart its gateway afterwards. This does not promote a zone or restart a gateway automatically.

func (*Fixture) RemoveZone 

func (f *Fixture) RemoveZone(ctx context.Context, name string) error

RemoveZone detaches one owned non-master zone from its zonegroup, commits and propagates the new period, then removes that zone's owned gateway and CLI. The realm, zone configuration, users, pools and stored objects remain in its storage cluster; this is not a destructive zone/pool purge. Detached names and clusters cannot be adopted through AddZone. The final zone of a group and any current realm/group master must first be replaced or promoted explicitly. Partial period propagation/container cleanup can be retried with the same name. Terminate also removes any remaining owned containers. Remaining gateway application endpoints may change on bridge-mode reload.

func (*Fixture) SyncStatus 

func (f *Fixture) SyncStatus(ctx context.Context, zoneName string) (SyncStatus, error)

SyncStatus observes native metadata/data readiness in an attached owned zone. Native human sync status is required for remote-log comparison even when the CLI is passed --format json. The structured metadata state separately guards period/realm epoch and every shard. This is a point-in-time observation; applications should quiesce writes and still verify their own S3 payloads.

func (*Fixture) Terminate 

func (f *Fixture) Terminate(ctx context.Context, opts ...testcontainers.TerminateOption) error

Terminate removes owned gateways, CLI clients and the HTTP bridge. Realm, zone configuration and data remain; neither Ceph cluster is terminated.

func (*Fixture) WaitSyncReady 

func (f *Fixture) WaitSyncReady(ctx context.Context, zoneName string, sourceZones ...string) (SyncStatus, error)

WaitSyncReady waits up to four minutes, bounded by the caller's context, for metadata and the explicitly requested source-zone data streams to catch up. No sources means metadata-only readiness. Native sync leases survive abrupt gateway restarts until expiration, so this method does not change lease TTLs, unlock objects or restart daemons to shorten normal recovery.

Wait for metadata in every zone after creating a bucket and before writing objects into it. Otherwise RGW can lose those objects in another zone (seen with Tentacle 20.2.4): that zone's data sync stores the new bucket's metadata itself, a second data sync shard may sync the bucket before its index exists, and RGW takes the failed index updates for deleted objects; the zone then reports caught up while the objects stay missing until they change again. After an abrupt gateway stop the old process's sync leases must expire (30 seconds for fixture gateways), and sync polls every 20 seconds by default: use deadlines of several minutes.

func (*Fixture) ZoneAdmin 

func (f *Fixture) ZoneAdmin(ctx context.Context, name string, args ...string) ([]byte, error)

ZoneAdmin runs native radosgw-admin in any owned zone, including a partial zone whose CLI client started successfully. Do not edit period/zone state concurrently with AddZone or external metadata master transitions.

func (*Fixture) Zonegroups 

func (f *Fixture) Zonegroups() []Zonegroup

Zonegroups returns a sorted snapshot of the owned groups and their attached zones. Detached zones are omitted after their removal period is committed.

func (*Fixture) Zones 

func (f *Fixture) Zones() []Zone

Zones returns owned zone descriptors sorted by name, including partial zones returned by a failed AddZone. It does not expose private system credentials.

type MetadataSyncStatus 

type MetadataSyncStatus struct {
	State, PeriodID           string
	RealmEpoch                uint64
	Shards, IncrementalShards int
	CaughtUp                  bool
}

MetadataSyncStatus includes only native state/identity/shard counts. It omits metadata keys, markers and credentials from native diagnostic output.

type Option 

type Option struct {
	// contains filtered or unexported fields
}

Option carries the realm configuration of a fixture to Run: a two-zone Config (WithConfig) or a larger TopologyConfig (WithTopology). It is not a container customization: its Customize does nothing, so it never reaches a gateway.

func WithConfig 

func WithConfig(config Config) Option

WithConfig carries a two-zone realm on two existing clusters to Run.

func WithTopology 

func WithTopology(config TopologyConfig) Option

WithTopology carries a realm of at least two zones, in one or several zonegroups, to Run. The zone lists are copied.

func (Option) Customize 

func (Option) Customize(*testcontainers.GenericContainerRequest) error

Customize implements testcontainers.ContainerCustomizer.

type SyncBucketIdentity 

type SyncBucketIdentity struct{ Name, Tenant, ID string }

SyncBucketIdentity fixes a bucket namespace and native instance. Recreating its name cannot silently change the bucket selected by an owned pipe.

type SyncBucketSelector 

type SyncBucketSelector struct{ Name, Tenant string }

SyncBucketSelector identifies a preexisting bucket. Its current native instance ID is captured and stored in the pipe, preventing a recreated name from silently becoming a new replication source/destination. Nil selectors retain Ceph's current-bucket wildcard behavior. Same-zone replication is not supported by native RGW multisite.

type SyncFlowConfig 

type SyncFlowConfig struct {
	ID                          string
	Zones                       []string
	SourceZone, DestinationZone string
}

SyncFlowConfig selects either a named symmetrical Zones set, or a directional SourceZone/DestinationZone pair. Directional flows have no native ID: leave ID and Zones empty. Zone names must be attached and owned by this fixture in the selected group. Bucket policies may only narrow the flow permitted by their parent zonegroup.

type SyncGroup 

type SyncGroup struct {
	// contains filtered or unexported fields
}

SyncGroup is a fresh group created by this fixture. Copies share native readback and lifecycle state. IDs on unrelated existing groups are never adopted. Native edits must not race these methods.

A group is stored on the zonegroup or on one existing bucket. Every mutation compares the native policy before and after the change and refuses unrelated or external changes; if the reply of a confirmed mutation is lost, the next call on the same handle reconciles the pending change first. Group mutations are bounded by two minutes.

func (*SyncGroup) Apply 

func (g *SyncGroup) Apply(ctx context.Context) error

Apply publishes only this owned zonegroup policy change, pulls the resulting period into every owned zone and reloads running gateways. Resolve their S3Endpoint again afterwards: a bridge gateway's mapped port can change. Bucket policy is dynamic and this method only checks its owned native snapshot. Unrelated current, locally pending or staging-period edits are refused before period update can overwrite them. Partial publication/reload is not rolled back; retry the same handle after resolving an external failure.

func (*SyncGroup) BucketSyncStatus 

func (g *SyncGroup) BucketSyncStatus(ctx context.Context, pipeID, sourceZone, destinationZone string) (BucketSyncStatus, error)

BucketSyncStatus reads native per-bucket remote-log comparison for an owned unchanged group/pipe and two explicit owned zones. Wildcard bucket selectors require a bucket-scoped group. Concrete bucket instance IDs remain fixed. A caught-up checkpoint does not prove filter semantics or copied payloads; applications should quiesce writes and verify their S3 bytes separately.

On Ceph 19 radosgw-admin ignores --format json for bucket sync status and prints text; this method parses it into the same structure and reports unknown lines as errors.

func (*SyncGroup) CreateFlow 

func (g *SyncGroup) CreateFlow(ctx context.Context, config SyncFlowConfig) error

CreateFlow adds a fresh flow to an owned unchanged group. Native directional identity is the exact source/destination pair, not flow-id.

func (*SyncGroup) CreatePipe 

func (g *SyncGroup) CreatePipe(ctx context.Context, config SyncPipeConfig) error

CreatePipe adds a fresh identity-checked pipe. Zonegroup policy forms an upper bound for a bucket policy; a stored pipe alone does not prove effective replication. Verify positive/negative writes through the S3 client.

func (SyncGroup) GoString 

func (g SyncGroup) GoString() string

GoString formats the sync group like String.

func (*SyncGroup) ID 

func (g *SyncGroup) ID() string

ID returns the native sync group ID, or "" for a nil handle.

func (*SyncGroup) Remove 

func (g *SyncGroup) Remove(ctx context.Context) error

Remove removes only this confirmed group's unchanged native policy. Bucket/object data remains. Zonegroup removal needs Apply afterwards to publish it; removal is idempotent across handle copies.

func (*SyncGroup) RemoveFlow 

func (g *SyncGroup) RemoveFlow(ctx context.Context, config SyncFlowConfig) error

RemoveFlow removes only a flow present in the owned group's exact native snapshot. Symmetrical removal deletes the entire named set; directional removal selects the native pair. It never changes another group's policy.

func (*SyncGroup) RemovePipe 

func (g *SyncGroup) RemovePipe(ctx context.Context, id string) error

RemovePipe removes one pipe from an owned group. The pipe must exist in the group's native policy, and the removal must leave the rest of the group unchanged. Objects that were already replicated stay. Like the other policy changes, it confirms the master's stored policy, not its asynchronous import into peers.

func (*SyncGroup) SetPipePrefix 

func (g *SyncGroup) SetPipePrefix(ctx context.Context, id, prefix string) error

SetPipePrefix changes only the selected owned pipe's source prefix. Empty prefix removes the filter. It affects future replication selection; historical backfill and deletion of previously copied objects are not promised. For a bucket-scoped group, wait for WaitBucketPolicyReady at the intended destination before writing objects that depend on the new filter. This method confirms the master's stored policy, not its asynchronous import into peers.

func (*SyncGroup) SetStatus 

func (g *SyncGroup) SetStatus(ctx context.Context, status SyncGroupStatus) error

SetStatus changes an owned group's allowed/enabled/forbidden state.

After forbidden to enabled, earlier log entries may stay unprocessed until an explicit bucket sync run through ZoneAdmin.

func (SyncGroup) String 

func (g SyncGroup) String() string

String names the sync group.

func (*SyncGroup) WaitBucketPolicyReady 

func (g *SyncGroup) WaitBucketPolicyReady(ctx context.Context, destinationZone string) (BucketSyncPolicyStatus, error)

WaitBucketPolicyReady waits up to four minutes, bounded by ctx including mutex queue time, for an unchanged owned bucket group's metadata to import into an explicit attached destination selected by its pipes. It checks the exact committed period, source-scope group snapshot, and all referenced source/destination bucket instances needed by that destination. Missing or older local metadata waits; recreated instances and changed master policy are refused. Allowed/forbidden policies can also be observed as imported.

This read-only prerequisite does not attest effective discovery hints, data checkpoints, payloads or permissions, or replay objects written before the policy was imported. It does not pull periods, restart gateways or mutate replication state. Use it after creating pipes and before selected writes.

Zonegroup groups, removed handles, pending mutations and zones outside the pipes' destinations are refused.

func (*SyncGroup) WaitBucketSyncReady 

func (g *SyncGroup) WaitBucketSyncReady(ctx context.Context, pipeID, sourceZone, destinationZone string) (BucketSyncStatus, error)

WaitBucketSyncReady waits up to four minutes, bounded by the caller context, for the captured pipe/bucket checkpoint. Disabled, full-sync, missing sources and native per-source errors never satisfy readiness even when CLI exit is 0. It changes no sync state, leases, permissions or object data. When the wait ends without readiness, the returned status is the last one that was read, so its lag survives a later attempt that failed or was canceled.

type SyncGroupConfig 

type SyncGroupConfig struct {
	ID     string
	Status SyncGroupStatus
}

SyncGroupConfig selects a new sync group. ID has no spaces or ,/:$ and at most 128 characters; empty generates one. Status is required.

type SyncGroupStatus 

type SyncGroupStatus string

SyncGroupStatus separates permission from activation. Allowed permits a bucket policy to enable its subset; Enabled activates the group's pipes; Forbidden takes precedence over other matching enabled groups. 

const (
	// SyncAllowed permits replication without activating it; a bucket group
	// may enable a subset.
	SyncAllowed SyncGroupStatus = "allowed"
	// SyncEnabled activates the group's pipes.
	SyncEnabled SyncGroupStatus = "enabled"
	// SyncForbidden blocks replication and takes precedence over other
	// matching enabled groups.
	SyncForbidden SyncGroupStatus = "forbidden"
)

type SyncObjectTag 

type SyncObjectTag struct{ Key, Value string }

SyncObjectTag matches an exact key/value. Multiple pairs are ORed and repeated keys with different values are permitted. Native CLI comma parsing cannot represent comma-containing keys/values; '=' is only allowed in values.

Key is 1 to 128 characters and Value up to 256.

type SyncPipeConfig 

type SyncPipeConfig struct {
	ID                              string
	SourceZones, DestinationZones   []string
	Prefix                          string
	SourceBucket, DestinationBucket *SyncBucketSelector
	Tags                            []SyncObjectTag
	Priority                        int32
	DestinationOwner, User          *rgw.User
	DestinationStorageClass         string
	// Storage-class translation requires a concrete destination bucket and a
	// confirmed native placement for each concrete DestinationZones entry.
	// The caller must publish/import/reload those placements before traffic.
	DestinationPlacements map[string]*rgw.Placement
}

SyncPipeConfig selects owned zone names or a sole "*" on each side. Nil bucket selectors use the current bucket; an empty Prefix selects all keys. Source tags are ORed; when a prefix is also supplied both must match. User mode uses a confirmed ordinary user, or a confirmed account root with concrete source/destination buckets owned by that same account and no DestinationOwner. Principal and selected bucket tenants must match. IAM nonroot, cross-account and cross-tenant principal translation remain unproven and are refused. This helper never creates S3 buckets or grants access.

ID is unique in the group and required. In system mode (no User) concrete source and destination buckets may belong to different tenants when no owner or user translation is requested. RGW preserves source object ACLs, so the destination bucket owner may not be the principal that can read the replicated objects.

On the default image (Ceph 20.2.4) two stored parameters have no reliable effect, so a stored policy and a caught-up checkpoint are no evidence that they work: Priority does not choose between overlapping pipes with the same prefix (the native rebuild order decides which pipe's parameters, such as the destination storage class, apply), and user mode with an ordinary user does not test source-side denial (RGW authorises the replication GET on the system path, so objects replicate even when the principal is denied s3:GetObject). User mode with an account root user honours a destination bucket policy that denies s3:ReplicateObject. On Ceph 19 a user-mode pipe with an account root user replicates nothing, so CreatePipe refuses it with ceph.ErrUnsupported when any zone runs Ceph 19; the *ceph.OperationError names the zone and its version.

type SyncPolicyScope 

type SyncPolicyScope struct{ Zonegroup, Bucket, Tenant string }

SyncPolicyScope selects an owned zonegroup, or an existing bucket within that group. An empty Zonegroup selects the original source's group. Bucket policy fixes the native bucket instance ID and rejects a recreated name.

type SyncStatus 

type SyncStatus struct {
	RealmID, PeriodID, Zone, ZoneID, ZonegroupID string
	RealmEpoch                                   uint64
	MetadataMaster                               bool
	Metadata                                     MetadataSyncStatus
	DataSources                                  []DataSyncStatus
}

SyncStatus is the metadata and data sync state of one attached zone, as Fixture.SyncStatus reads it.

type TopologyConfig 

type TopologyConfig struct {
	Zones                            []ZoneConfig
	Realm, Zonegroup, MetadataMaster string
	SetupImage                       string
	Zonegroups                       []ZonegroupConfig
	MasterZonegroup                  string
}

TopologyConfig creates one realm spanning at least two independent zones. Use Zones/Zonegroup/MetadataMaster for a single region, or Zonegroups and MasterZonegroup for several regions with separately selected local masters. MetadataMaster defaults to the first zone; MasterZonegroup to the first group. All clusters use the same network mode and must not serve standalone RGW data. SetupImage optionally overrides every zone's setup client image. Otherwise each client uses its own cluster's RGW image, including added zones.

type Zone 

type Zone struct {
	Name, ID, Zonegroup, PeerEndpoint string
	Gateway                           *rgw.Gateway
}

Zone describes one owned multisite zone. PeerEndpoint is reachable by gateways/CLI containers in the fixture's HTTP network or Docker host namespace. Applications obtain their endpoint separately with Gateway.S3Endpoint.

type ZoneConfig 

type ZoneConfig struct {
	Name, Zonegroup string
	Cluster         *ceph.Container
}

ZoneConfig assigns one fresh, independent storage cluster to a named zone.

type Zonegroup 

type Zonegroup struct {
	Name, ID, MasterZone string
	Zones                []Zone
}

Zonegroup is an owned region snapshot. MasterZone reflects the last topology operation; direct native promotions require a subsequent operation to refresh it. Each zone retains its own storage cluster and daemon endpoint.

type ZonegroupConfig 

type ZonegroupConfig struct {
	Name, MasterZone string
	Zones            []ZoneConfig
}

ZonegroupConfig defines a region and its local master zone. MasterZone defaults to the first zone. Object data replicates within each zonegroup; users and bucket metadata share the realm across zonegroups.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.