Documentation
¶
Overview ¶
Package fault injects dedicated faults into an existing Ceph fixture. Cluster-bound operations that return a handle use ceph.Run's lifetime and are restored by its Terminate, in this order: network shaping first, endpoint restoration second, then paused containers are resumed. Standalone link operations (InterruptLink, ApplyLinkProfile) are restored by their caller before it removes their container or bridge. Keep a non-nil handle returned with an error: its restore stays retryable after an uncertain result.
The error categories are those of package ceph; the sentinels of this package match them with errors.Is.
Policy and admission controls that only change how Ceph admits work, such as full ratios, OSD flags and the session blocklist, live in package override even when a test uses them to provoke failures. Object read-error injection is here because it is a purpose-built synthetic fault, although Ceph implements it as a daemon debug switch; scrub, repair and inspection of the damaged PG stay in package ceph.
Index ¶
- Constants
- Variables
- func InjectObjectDataError(ctx context.Context, c *ceph.Container, pool, object string, osdID int) error
- type NetworkDirection
- type NetworkInterruption
- type NetworkPacketFault
- type NetworkPeer
- type NetworkPhase
- type NetworkProfile
- type NetworkProfileLease
- type NetworkProfileStatus
- type NetworkRate
- type PauseLease
Examples ¶
Constants ¶
const ( BitPerSecond NetworkRate = 1 KilobitPerSecond = 1000 * BitPerSecond MegabitPerSecond = 1000 * KilobitPerSecond GigabitPerSecond = 1000 * MegabitPerSecond )
Units of NetworkRate, in powers of 1000 as tc counts them.
const ( MinNetworkRate = 64 * KilobitPerSecond MaxNetworkRate = 10 * GigabitPerSecond )
MinNetworkRate and MaxNetworkRate bound NetworkProfile.Rate.
const DefaultNetworkQueueLimit = 1000
DefaultNetworkQueueLimit is the netem queue of a rate limit whose NetworkProfile.QueueLimit is zero, in packets: tc's default.
const MaxNetworkDelay = 10 * time.Second
MaxNetworkDelay bounds NetworkProfile.Delay.
const MaxNetworkPeers = 16
MaxNetworkPeers bounds NetworkProfile.Peers.
const MaxNetworkQueueLimit = 100000
MaxNetworkQueueLimit bounds NetworkProfile.QueueLimit.
const NetemHelperImage = "ghcr.io/jsyoo5b/ceph-testcontainers-images:extension-netem-v1"
NetemHelperImage is the netem helper that ceph-testcontainers-images publishes for contract 1 of its netem extension, for linux/amd64 and linux/arm64. ApplyNetworkProfile never selects it by itself: pass it, or another image that meets the contract, as NetworkProfile.HelperImage.
Variables ¶
var ( // ErrNetemImageUnsupported reports a helper image that does not meet the // netem contract: a missing or broken tc, ip, sh, sleep or python3, or a // tc without netem, or, for a Rate, a tc that cannot parse netem's rate, // or for a Reorder, Duplicate or Corrupt, a tc that cannot parse that // fault with a correlation. It belongs to ceph.ErrImageCapability. ErrNetemImageUnsupported error = &categoryError{text: "the netem helper image does not meet the netem contract", kind: ceph.ErrImageCapability} // ErrNetemPermissionDenied reports an engine that did not give the helper // NET_ADMIN, or refused a qdisc change for lack of it. It belongs to // ceph.ErrEnvironmentUnavailable. ErrNetemPermissionDenied error = &categoryError{text: "the netem helper has no NET_ADMIN", kind: ceph.ErrEnvironmentUnavailable} // ErrNetemKernelUnsupported reports a kernel without the netem qdisc, // for a delay with peers without the prio qdisc or the u32 classifier, or // for a Rate a netem qdisc that does not keep the rate, or for a Reorder, // Duplicate or Corrupt one that does not keep that fault, its correlation // or, for a Reorder, a gap of 1. It belongs to ceph.ErrEnvironmentUnavailable. ErrNetemKernelUnsupported error = &categoryError{text: "the engine's kernel lacks a qdisc or classifier that the delay needs", kind: ceph.ErrEnvironmentUnavailable} )
Functions ¶
func InjectObjectDataError ¶
func InjectObjectDataError(ctx context.Context, c *ceph.Container, pool, object string, osdID int) error
InjectObjectDataError makes one replica of an object on an owned OSD return a read error to scrub and recovery, through the native injectdataerr debug command. The OSD must be in the object's current acting set of a replicated pool. The command needs bluestore_debug_inject_read_err, which this function enables on that daemon only through its runtime configuration; it is not stored centrally and lasts until the OSD restarts. The setting has no effect on objects without an injected error.
The injection is one-shot and returns no handle: the cluster does not retain it and Terminate does not undo it. It stays in OSD memory until the replica is rewritten, such as by ceph.Container.RepairPG, or the OSD restarts. Client reads served by a healthy primary are unaffected. Use ceph.Container.DeepScrubPG to report the damage and ceph.Observer.PGInconsistencies, ceph.Observer.HealthDetails (OSD_SCRUB_ERRORS, PG_DAMAGED) or ceph.Pool.PGs to observe it. It needs BlueStore OSDs and refuses a cluster that uses OSDMemStore. A failure after the debug setting was enabled leaves it enabled and is not retried or disarmed.
Inject into a non-primary replica to leave scrub evidence: client reads served by the primary are unaffected, a deep scrub reports read_error on that shard, the PG becomes active+clean+inconsistent and health reports OSD_SCRUB_ERRORS and PG_DAMAGED. An error on the primary does not make client reads fail either, and a later deep scrub reports no inconsistency.
The object is addressed in the default namespace. A name containing a slash is refused before any native command, because the native command would reinterpret the part before the first slash as a namespace.
Types ¶
type NetworkDirection ¶
type NetworkDirection int
NetworkDirection is the side of a profile's selected paths that it shapes. The netem helper shapes only what an endpoint sends, so the other side of a path is shaped at the peer's endpoint: for Inbound and BothDirections, every selected peer address must belong to a running container attached to the same bridge, and the lease applies the same profile there, selecting the packets that the peer sends to the target's address of that family. A peer's Port then selects the TCP source port of what the peer sends, so both directions of a conversation with one of the peer's ports, such as a Messenger port of an OSD, are selected by the same NetworkPeer.
const ( // Outbound shapes what the target sends. Outbound NetworkDirection = iota // Inbound shapes what the selected peers send to the target, at each // peer's endpoint; nothing is applied to the target's own endpoint. Inbound // BothDirections shapes both, with one lease on the target's endpoint // and one on each peer's. BothDirections )
func (NetworkDirection) String ¶
func (d NetworkDirection) String() string
String names the direction.
type NetworkInterruption ¶
type NetworkInterruption struct {
ContainerID, NetworkName string
OriginalAddress netip.Addr
// contains filtered or unexported fields
}
NetworkInterruption owns one disconnected Docker endpoint. Restore reconnects its original addresses and aliases without restarting the container or Ceph daemon. It is an endpoint-to-network interruption, not a pairwise packet filter. A non-nil handle returned with an error must be restored: transport failure cannot prove that Docker did not disconnect it. Restore is retryable.
func InterruptLink ¶
func InterruptLink(ctx context.Context, ctr testcontainers.Container, name string) (*NetworkInterruption, error)
InterruptLink disconnects one existing bridge endpoint. The caller owns the container and the returned restoration handle; this function does not stop a daemon, change Ceph policies or remove the network. Fixture-specific helpers also register restoration with their cleanup. Host networking cannot be interrupted through Docker endpoint operations.
func InterruptNetwork ¶
func InterruptNetwork(ctx context.Context, c *ceph.Container, ctr testcontainers.Container, plane ceph.NetworkPlane) (link *NetworkInterruption, err error)
InterruptNetwork disconnects a container from one of this cluster's bridges. The container may be an owned daemon or a caller-owned WithClient container. With a single bridge, both planes select that same endpoint. Host mode is rejected because Docker cannot disconnect a host-network endpoint. It cuts the whole endpoint, not the traffic to selected peers; use ApplyNetworkProfile with a Loss of 100 and Peers for that. The fixture retains the handle and attempts restoration during Terminate. Restore explicitly before destroying caller-owned containers or the cluster.
An interrupted MON stays detached when it is stopped and started: its Start refuses with ceph.ErrInvalidRequest until Restore has reattached the endpoint, because the MON can bind only its monmap address. A cut cluster endpoint can end an OSD process: with three or more OSDs on separate CRUSH hosts its peers report the failed heartbeats, the MON marks it down after osd_heartbeat_grace (20 s), and the OSD exits when it cannot rebind its cluster messenger ("rebind cluster_messenger failed"). Restore does not restart it; call Start on its handle afterwards. With two OSDs one reporter is not enough (mon_osd_min_down_reporters is 2) and the OSD stays up.
Example ¶
package main
import (
"context"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/fault"
)
func main() {
ctx := context.Background()
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithNetworkConfig(ceph.NetworkConfig{}.WithSeparateClusterNetwork()))
if cluster != nil {
defer func() { _ = cluster.Terminate(ctx) }()
}
if err != nil {
panic(err)
}
osds, err := cluster.OSDs(ctx)
if err != nil {
panic(err)
}
cut, err := fault.InterruptNetwork(ctx, cluster, osds[0], ceph.ClusterNetworkPlane)
if cut != nil {
defer func() { _ = cut.Restore(ctx) }()
}
if err != nil {
panic(err)
}
// Restore reconnects the endpoint; it does not restart a daemon that exited.
}
Output:
func (*NetworkInterruption) Restore ¶
func (link *NetworkInterruption) Restore(ctx context.Context) error
Restore reattaches the original endpoint, preserving the IPv4 and, on a dual-stack bridge, the IPv6 address advertised by Ceph. Removed containers need no restoration. An externally reattached endpoint with a different address is refused rather than overwritten, and the handle stays retryable.
type NetworkPacketFault ¶
type NetworkPacketFault struct {
// Percent is the probability that a packet is affected, from more than 0
// to 100. Zero, with a zero Correlation, means no fault.
Percent float64
// Correlation, from 0 to 100, makes each packet's random decision this
// percentage dependent on the previous packet's, so affected packets
// come in runs. It needs a Percent.
Correlation float64
}
NetworkPacketFault is one of netem's packet faults, for NetworkProfile's Reorder, Duplicate and Corrupt: the percentage of the selected packets that it affects, and how much each packet's decision depends on the previous one's.
func (NetworkPacketFault) IsZero ¶
func (f NetworkPacketFault) IsZero() bool
IsZero reports whether the fault is off.
func (NetworkPacketFault) String ¶
func (f NetworkPacketFault) String() string
String formats the fault as tc's arguments do: the percentage, and the correlation when it is not zero.
type NetworkPeer ¶
NetworkPeer selects the packets that a delay's target sends to Addr, an IPv4 address or, for an endpoint that has one, an IPv6 address. With a Port, only TCP packets to that destination port are selected; an IPv6 packet with extension headers before its TCP header is not.
type NetworkPhase ¶
type NetworkPhase int
NetworkPhase is the part of a TCP connection that a profile shapes.
const ( // AllPackets shapes every selected packet. AllPackets NetworkPhase = iota // ConnectionSetup shapes only the selected TCP segments with SYN set: // connection attempts, and the SYN-ACK that answers a peer's attempt. // Established connections keep their packets unshaped, so with a Loss of // 100 every session that exists stays up while every new connection, // reconnect included, fails on that path. Its u32 key sits at the TCP // flags of a header directly after a base IP header, like the port keys. ConnectionSetup )
type NetworkProfile ¶
type NetworkProfile struct {
// HelperImage is the netem helper image: NetemHelperImage, or another image
// that meets contract 1 of the netem extension. Required.
HelperImage string
// Plane selects the bridge whose endpoint of the target is delayed.
Plane ceph.NetworkPlane
// Delay is added to every packet that leaves the target through that
// endpoint: at least 1ms and at most MaxNetworkDelay, in whole
// microseconds. With a Loss, a Rate, a Duplicate or a Corrupt it may be
// zero.
Delay time.Duration
// Jitter varies each packet's delay uniformly within ±Jitter. It is at
// most Delay, in whole microseconds.
Jitter time.Duration
// Peer, when set, limits the delay to the packets that the target sends
// to this address through the endpoint; every other packet leaves
// without delay. It is an IPv4 address, or an IPv6 address for an
// endpoint that has one, such as a daemon's address from
// DaemonAddresses on the same plane. Only packets that leave the target
// are selected, unless Direction also shapes the peer's side.
Peer netip.Addr
// PeerPort, with Peer, limits the delay further to TCP packets to this
// destination port, such as one of the peer's Messenger ports, and on
// the peer's side of an Inbound delay to its packets from that port. Zero
// selects every packet to Peer. An IPv6 packet with extension headers
// before its TCP header is not selected.
PeerPort uint16
// Peers, instead of Peer and PeerPort, limits the delay to the packets
// that the target sends to any of these peers, each with its own
// optional TCP destination port, such as several OSDs of a daemon.
// Every packet that one of them selects gets the same delay and loss;
// every other packet leaves without delay. It holds at most
// MaxNetworkPeers distinct entries, of either family.
Peers []NetworkPeer
// Direction selects which side of the paths to the selected peers is
// shaped: Outbound (the zero value) what the target sends, Inbound what
// the peers send to the target, BothDirections both. Inbound and
// BothDirections need a Peer or Peers; see NetworkDirection.
Direction NetworkDirection
// Phase selects which packets of a selected path are shaped: AllPackets
// (the zero value), or ConnectionSetup, only TCP segments with SYN set.
// ConnectionSetup needs a Peer or Peers; see NetworkPhase.
Phase NetworkPhase
// Loss drops this percentage of the packets that the delay applies to,
// at random and independently, from more than 0 to 100. With 100 every
// selected packet is dropped: with a Peer or Peers, the target's packets
// to those peers, and only those, never arrive. Zero drops nothing.
Loss float64
// Rate, when set, limits the selected packets to this bandwidth, from
// MinNetworkRate to MaxNetworkRate in whole bytes per second (a multiple
// of 8 bit/s), with netem's rate in the same qdisc as the Delay and Loss.
// netem sends each packet only after the previous one has taken its
// length over the rate, so there is no burst: a packet counts with its
// Ethernet header, as the kernel queues it, and a TCP segmentation
// offload packet counts with all of its segments. The packets waiting
// for the rate, and for the Delay, share one queue of QueueLimit
// packets; one that arrives at a full queue is dropped and counted in
// NetworkProfileStatus.Drops. Zero sets no limit. With a Loss, netem
// drops first, then delays, then paces what is left. With Direction
// Outbound it shapes only what the target sends; Inbound and
// BothDirections apply the same Rate at each peer's endpoint. What an application
// achieves through it is less, and is for the caller to measure; the
// status's Rate and counters are configuration and counts, not a
// throughput. The preflight adds and removes a netem qdisc with a rate on
// its loopback and checks what the kernel reports, so a tc or kernel
// without netem's rate is refused before the target changes.
Rate NetworkRate
// QueueLimit, with a Rate, is the netem queue in packets, from 1 to
// MaxNetworkQueueLimit; zero means DefaultNetworkQueueLimit.
QueueLimit int
// Reorder, when set, sends this percentage of the selected packets at
// once, ahead of the packets that wait for the Delay, so they arrive out
// of order. It needs a Delay, which every other selected packet gets, and
// takes no Jitter, which reorders packets by itself, and no Rate, which
// the packets sent at once would bypass. netem's gap is 1: every packet
// may be reordered.
Reorder NetworkPacketFault
// Duplicate, when set, sends a second copy of this percentage of the
// selected packets. iproute2 6.1, the tc of NetemHelperImage, sends a
// duplicate's correlation only together with a delay's correlation, so
// with a Duplicate.Correlation the lease passes the delay with a zero
// correlation (delay 0us 0us 0% without a Delay), the kernel's state
// without one.
Duplicate NetworkPacketFault
// Corrupt, when set, flips one random bit in this percentage of the
// selected packets, within the part of the packet that the kernel holds
// in its first buffer: the headers, and the first payload bytes when they
// are there too. netem first computes a checksum that the device would
// have offloaded, so the receiver's checksum check sees the flipped bit,
// and splits a segmentation offload packet, corrupting one segment.
// netem decides per queued packet, for every fault: a percentage of
// packets is not a percentage of TCP segments.
Corrupt NetworkPacketFault
}
NetworkProfile configures netem shaping for ApplyNetworkProfile or ApplyLinkProfile.
type NetworkProfileLease ¶
type NetworkProfileLease struct {
// ContainerID and NetworkName identify the delayed endpoint.
ContainerID, NetworkName string
// SandboxID is the Docker network sandbox of the target when the delay
// was applied, and Namespace the kernel's name of its network namespace.
SandboxID, Namespace string
// Interface, InterfaceIndex, MACAddress and Address identify the
// delayed interface inside that namespace.
Interface string
InterfaceIndex int
MACAddress string
Address netip.Addr
// HelperImageID is the image of the helper that applied the delay.
HelperImageID string
// Delay and Jitter are the requested values.
Delay, Jitter time.Duration
// Peer and PeerPort are the requested Peer and PeerPort.
Peer netip.Addr
PeerPort uint16
// Peers is the whole selection: Peer and PeerPort as one entry, or the
// requested Peers. Empty means the whole interface.
Peers []NetworkPeer
// Loss is the requested loss percentage.
Loss float64
// Rate is the requested rate, and QueueLimit the queue limit that the
// lease installed with it: the requested one, or
// DefaultNetworkQueueLimit. Both are zero without a Rate.
Rate NetworkRate
QueueLimit int
// Reorder, Duplicate and Corrupt are the requested packet faults.
Reorder, Duplicate, Corrupt NetworkPacketFault
// Direction and Phase are the requested ones.
Direction NetworkDirection
Phase NetworkPhase
// SourcePorts says that each of Peers' ports is the TCP source port of
// what this endpoint sends, not the destination port: it is set on the
// leases in PeerLeases, whose Peers are the target's addresses.
SourcePorts bool
// PeerLeases holds, for Inbound and BothDirections, one lease on each
// selected peer's endpoint, shaping what that peer sends to the target.
// Restore restores them with the rest; each has its own Status. With
// Inbound alone the lease applies nothing to the target's endpoint, and
// its own Status is refused.
PeerLeases []*NetworkProfileLease
// contains filtered or unexported fields
}
NetworkProfileLease owns one netem profile on a container's network endpoint, applied from a helper container in the target's network namespace that alone has NET_ADMIN. Restore removes the delay and the helper. A non-nil lease returned with an error must be restored: the change may have been applied. Restore is retryable.
func ApplyLinkProfile ¶
func ApplyLinkProfile(ctx context.Context, ctr testcontainers.Container, network string, delay NetworkProfile) (*NetworkProfileLease, error)
ApplyLinkProfile applies the same netem profile on another bridge than the cluster's planes, for fixtures that own one, such as the private HTTP bridge between the gateways of an RGW multisite: it delays or drops the packets that ctr sends through its endpoint on the bridge named network. Like InterruptLink, the caller owns ctr, the network and the returned lease, and restores the lease before it removes either: no cluster's Terminate knows it. delay.Plane stays empty; every other field, check and error is that of ApplyNetworkProfile, and a network that is not a bridge is refused.
func ApplyNetworkProfile ¶
func ApplyNetworkProfile(ctx context.Context, c *ceph.Container, target testcontainers.Container, delay NetworkProfile) (*NetworkProfileLease, error)
ApplyNetworkProfile applies netem shaping to the packets that target sends through its endpoint on the plane's bridge, below Ceph. It is an opt-in extension: delay.HelperImage names the netem helper, which runs in the target's network namespace with NET_ADMIN, while the target keeps its own image and capabilities. The target is an owned daemon or a caller-owned container attached to that bridge; host-network clusters, stopped or paused targets and targets that share another container's network namespace are refused. With a Peer or Peers, only the packets to those peers are delayed, through one prio root, one netem child and one u32 filter per peer, all owned by the one lease: to cut a daemon's paths to several peers at once, list them in one delay. A Rate limits the bandwidth of the same packets in the same netem qdisc, alone or with a Delay and a Loss. Reorder, Duplicate and Corrupt add netem's packet faults to the same qdisc: a Reorder with a Delay and without Jitter or a Rate, a Duplicate or a Corrupt with any of the others.
Before it changes anything, it checks the helper image against the netem contract, NET_ADMIN and the kernel's netem, with a Rate netem's rate in tc and the kernel, with a packet fault that fault in tc and the kernel, and with peers its prio qdisc and u32 classifier, in a separate helper without a network, and returns ErrNetemImageUnsupported, ErrNetemPermissionDenied or ErrNetemKernelUnsupported. It refuses an interface whose root qdisc is not the kernel's default and a target that already has a delay on that bridge. A missing HelperImage is ceph.ErrImageCapability, the refusals are ceph.ErrInvalidRequest, and a tc command that fails otherwise is a *ceph.NativeFailure.
The delay lasts until Restore, or until the target stops or leaves the bridge, which removes its interface. Terminate restores delays that are still held; restore a caller-owned target's delay before removing it.
With a Peer or Peers the lease owns a prio root qdisc whose third band leads to the netem child, and one u32 filter per peer that sends its packets there; every priority maps to the first band, so nothing else is delayed. IPv4 peers' filters are in preference 10 and u32 table 800:, with handles 800::1, 800::2 and so on in the given order; IPv6 peers' are in the next preference and table (11 and 801: after IPv4 peers, otherwise 10 and 800:). They are added root first and filters last, so no packet is delayed before the hierarchy is complete.
Direction Inbound or BothDirections also shapes what each selected peer sends to the target: every peer address is resolved to the running container that has it on the bridge, before anything changes, and the same profile is applied to that container's endpoint, selecting the target's address of the peer's family and the peer's port as a TCP source port. One lease per peer container is in the returned lease's PeerLeases, each reserved and restored by Terminate on its own as well; with Inbound alone nothing is applied to the target. Phase ConnectionSetup adds a u32 key on the TCP SYN bit to every filter, so only connection setup is shaped.
TCP absorbs packet faults before Ceph sees them: the receiver reorders segments back, discards duplicates (answering with a D-SACK) and drops a segment whose checksum fails, which the sender retransmits. A Ceph operation that returns the right bytes therefore does not show that a packet was altered; read the receiver's kernel counters for the transport effect.
Example ¶
package main
import (
"context"
"time"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/fault"
)
func main() {
ctx := context.Background()
cluster, err := ceph.Run(ctx, ceph.DefaultImage)
if cluster != nil {
defer func() { _ = cluster.Terminate(ctx) }()
}
if err != nil {
panic(err)
}
osds, err := cluster.OSDs(ctx)
if err != nil {
panic(err)
}
lease, err := fault.ApplyNetworkProfile(ctx, cluster, osds[0], fault.NetworkProfile{
HelperImage: fault.NetemHelperImage, Plane: ceph.PublicNetworkPlane, Delay: 100 * time.Millisecond,
})
if lease != nil {
defer func() { _ = lease.Restore(ctx) }()
}
if err != nil {
panic(err)
}
// Run a bounded native workload while the delay is installed.
}
Output:
func (*NetworkProfileLease) Restore ¶
func (l *NetworkProfileLease) Restore(ctx context.Context) error
Restore removes the delay's qdisc and then the helper, after restoring every lease in PeerLeases. A delay whose interface is gone needs no removal. A root qdisc that someone replaced, or a filter or qdisc that someone added under the lease's root, is refused, not removed, and the lease stays retryable.
func (*NetworkProfileLease) Status ¶
func (l *NetworkProfileLease) Status(ctx context.Context) (NetworkProfileStatus, error)
Status reads the delayed interface's qdiscs through the helper. After the target restarted, stopped, was removed or left the bridge, it reports Gone. With peers it reports Applied only when the root, the netem child and exactly the lease's filters, with the requested address and port keys, are present and nothing else is under the root; Filters holds tc's filter dump with each u32 filter's keys under matches. The counters are the child's, so they count only selected packets.
type NetworkProfileStatus ¶
type NetworkProfileStatus struct {
// Applied means that the delay's own netem qdisc is the root qdisc of the
// interface it was applied to (with peers, that its own hierarchy is,
// with one filter for each peer), and that nothing the lease did not add
// is under that root.
Applied bool
// Gone means that the delayed interface no longer exists: the target
// stopped, restarted, was removed or was disconnected from the network,
// and the delay ended with it.
Gone bool
// Interface is the delayed interface's current name.
Interface string
// RootKind and RootHandle describe the interface's current root qdisc.
RootKind, RootHandle string
// Delay and Jitter are the kernel's values while Applied.
Delay, Jitter time.Duration
// Loss is the kernel's loss percentage while Applied.
Loss float64
// Rate and QueueLimit are the netem qdisc's configured rate and queue
// limit while Applied: the kernel's configuration, not a throughput that
// anything achieved. Rate is zero without a rate limit.
Rate NetworkRate
QueueLimit int
// Reorder, Duplicate and Corrupt are the netem qdisc's configured packet
// faults while Applied, and Gap its reorder gap (1 with a Reorder): the
// kernel's configuration. None of them shows that a packet was affected;
// the receiver's transport counters can.
Reorder, Duplicate, Corrupt NetworkPacketFault
Gap int
// Packets, Bytes and Drops are the netem qdisc's counters while Applied.
// With peers they count only the selected packets, of all peers. Drops
// includes the packets that Loss dropped and those that found the queue
// full.
Packets, Bytes, Drops uint64
// Overlimits and Requeues are the netem qdisc's other counters, and
// Backlog (bytes) and QueueLength (packets) what waits in its queue, all
// while Applied. With a Reorder, Requeues counts the packets that netem
// sent ahead. They are counters, not rates: to know a throughput,
// measure the workload, or compare Bytes over a period you time.
Overlimits, Requeues, Backlog, QueueLength uint64
// Qdiscs is tc's JSON output for the interface.
Qdiscs json.RawMessage
// Filters is tc's egress filter output for the interface, with the keys
// of each u32 filter under "matches".
Filters json.RawMessage
// contains filtered or unexported fields
}
NetworkProfileStatus is what the kernel reports for a profile's interface, read through the helper. It is native state, not the requested NetworkProfile.
type NetworkRate ¶
type NetworkRate uint64
NetworkRate is a bandwidth in bits per second, for NetworkProfile.Rate.
func (NetworkRate) String ¶
func (r NetworkRate) String() string
String formats the rate in bit/s with the largest decimal prefix that divides it.
type PauseLease ¶
type PauseLease struct {
ContainerID string
// contains filtered or unexported fields
}
PauseLease owns one container frozen through Docker's cgroup freezer. Resume thaws it without restarting the process. A non-nil handle returned with an error must be resumed: a transport failure cannot prove that Docker did not freeze the container. Resume is retryable and idempotent.
func Pause ¶
func Pause(ctx context.Context, c *ceph.Container, ctr testcontainers.Container) (*PauseLease, error)
Pause freezes every process of an owned daemon or a caller-owned WithClient container, as with docker pause. Unlike Stop, the process keeps its memory, sockets and Ceph identity, and peers see silence rather than a closed connection: a frozen OSD stops answering client operations and heartbeats until its peers report it down after the heartbeat grace. Hold nodown with override.SetOSDFlag to keep it up and make client operations wait for their own timeouts (rados_osd_op_timeout) or be reported as SLOW_OPS (after osd_op_complaint_time, 30 seconds by default). Host networking is supported.
The container that runs this cluster's control CLI is refused because every fixture command would block on it (in a single-MON cluster that is MON a). A container that is already paused is refused rather than adopted, and only one pause per container is allowed. The cluster retains the handle and resumes it during Terminate, after network faults are restored and before containers are removed; resume explicitly before removing a caller-owned container.
Example ¶
package main
import (
"context"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/fault"
)
func main() {
ctx := context.Background()
cluster, err := ceph.Run(ctx, ceph.DefaultImage)
if cluster != nil {
defer func() { _ = cluster.Terminate(ctx) }()
}
if err != nil {
panic(err)
}
osds, err := cluster.OSDs(ctx)
if err != nil {
panic(err)
}
pause, err := fault.Pause(ctx, cluster, osds[0])
if pause != nil {
defer func() { _ = pause.Resume(ctx) }()
}
if err != nil {
panic(err)
}
// The process stays alive but silent. Resume does not restart it.
}
Output: