blockio

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 18 Imported by: 0

README

Disposable OSD block I/O faults

fault/blockio owns a sparse backing file, loop device, device-mapper mapping and privileged keeper container. It accepts no existing disk, mapping or loop. The backing file lives on the keeper's bounded tmpfs: written pages consume Docker-host memory. This is a fixture for observing a client's behavior under an OSD main-device fault, not a storage benchmark.

Start the backend before the cluster and terminate the cluster before the backend. Register backend cleanup first so reverse-order cleanup preserves the keeper while its OSD consumes the device. Keep a non-nil backend returned with an error and retry its cleanup; do not remove its embedded container directly. The compiled example shows the full sequence: attach an OSD, apply the delay, check its status and restore the device table before teardown.

Choose exactly one positive whole-millisecond Delay (at most MaxDelay) or EIO: true. A successful Apply confirms the installed table; it does not prove a client experienced delay or an I/O error. Status.Requested is the requested profile. Identity and Tables are separately observed native state; PendingStep means completion remains unconfirmed. Inspect that state and retry Restore after an uncertain operation instead of replaying Apply.

Restore restores the recorded linear table. It does not restart an OSD that exited, prove recovery or recover acknowledged bytes. An EIO fault can abort the OSD; clients may observe bounded timeouts rather than an EIO response. Restart the OSD separately, wait for clean state, and verify old and new data with the client under test. A busy or unconfirmed consumer prevents Terminate from releasing the mapping and keeper. Terminate the cluster, retry backend cleanup, and use Status.Released to confirm release.

The blockio helper source and quick-check contract provide the required tools; build an explicit local image from that source. There is no default or automatically published helper tag.

The helper image must contain Python 3, sh, sleep, dmsetup, losetup and blockdev. The Docker host must already expose usable loop/device-mapper controls and register the selected targets. The fixture does not install tools, load host kernel modules or alter daemon configuration. Missing host facilities are ceph.ErrEnvironmentUnavailable; missing image tools are an image capability error. A registered linear target does not imply delay is registered.

The opt-in native recipe uses only public APIs and the native rados CLI. It checks delay against baseline and restored writes when available, separates a missing delay target from successful delay verification, checks EIO against client and OSD evidence, then verifies acknowledged bytes and a new write after restoration and explicit OSD restart. It also checks busy-consumer retention and retryable final release. Select an immutable helper image and compatible Ceph images in the environment:

CEPH_TEST_BLOCKIO_HELPER_IMAGE='<helper image reference>' \
  python3 .github/scripts/tag_scenarios.py optional --batch block_io \
  --directory artifacts/blockio-native

The batch requires CEPH_TEST_BLOCKIO_HELPER_IMAGE. With a role layout, set both CEPH_TEST_ESSENTIAL_IMAGE and CEPH_TEST_CLIENT_IMAGE; otherwise CEPH_TEST_IMAGE selects the aggregate image. An environment-unavailable setup fails the selected batch with its classified cause; it is not a successful native execution. An unavailable delay target is likewise recorded explicitly; it is not a delay validation result.

The keeper and its private tmpfs must stay alive while a chain is retained. A forced helper removal, helper restart or test-process/engine loss is outside the retryable cleanup path; ordinary Docker resource reaping is not a device- mapper or loop cleanup guarantee. Preserve the original cleanup error and the keeper identity rather than forcing removal of a retained backend.

For disposable GitHub-hosted qualification, manually dispatch Block fault probe with public_api=true. Set prepare_kernel=true only when admitting module setup on that disposable runner. The library never loads modules. The job builds the source-pinned helper without publishing it and requires actual Delay and EIO effects, recovery and release on every supported Ceph release. An unavailable target fails this qualification instead of counting as a pass.

Documentation

Overview

Package blockio prepares disposable device-mapper storage for one BlueStore OSD and injects bounded main-device delay or EIO. The main device also carries BlueFS traffic. Restoring its table does not restart or recover the OSD.

Run, AttachOSD, Apply, Restore, Status and Terminate each give their work a two-minute deadline, including waiting for the backend lock where needed. An earlier caller deadline takes precedence.

Index

Examples

Constants

View Source
const (
	// DefaultSize is the logical size of a zero Config.Size.
	DefaultSize int64 = 1 << 30
	// MinSize and MaxSize bound the disposable backing file, in bytes.
	MinSize int64 = 64 << 20
	MaxSize int64 = 4 << 30
	// MaxDelay bounds Profile.Delay.
	MaxDelay = 10 * time.Second
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct{ Size int64 }

Config selects the logical backing size. Size is sector aligned, between MinSize and MaxSize; zero selects DefaultSize. The helper holds the sparse file on its own bounded tmpfs. Written pages consume Docker-host memory.

type Container

type Container struct {
	testcontainers.Container
	// contains filtered or unexported fields
}

Container owns one backing file, loop, mapping and privileged keeper. Keep a non-nil value returned with an error. Terminate retains the keeper and its backing when a consumer or control step may remain active. Do not stop, restart or remove the embedded keeper while the backend is held.

func Run

Run prepares one disposable backend with the explicitly selected helper image. The image supplies sh, sleep, python3, dmsetup, losetup and blockdev; the Docker host grants privileged /dev access and registered linear support. Individual faults check their own registered target before use. No host module is loaded. Customizers reach the keeper; its identity, private tmpfs, command and image must remain intact. Register its CleanupContainer before the cluster's so cluster teardown precedes backing release.

Example
package main

import (
	"context"
	"os"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/fault/blockio"
)

func main() {
	ctx := context.Background()
	// Select an image containing the documented device-mapper helper tools.
	backend, err := blockio.Run(ctx, os.Getenv("CEPH_TEST_BLOCKIO_HELPER_IMAGE"), blockio.WithConfig(blockio.Config{Size: 1 << 30}))
	if backend != nil {
		defer func() { _ = backend.Terminate(ctx) }()
	}
	if err != nil {
		panic(err)
	}
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithNoInitialOSDs().WithPoolDefaults(1, 1)),
		ceph.WithOSDStorage(ceph.OSDStorage{}.WithStartupBenchmark(ceph.OSDBenchmarkSkip)))
	if cluster != nil {
		// This defer runs before backend cleanup: the OSD consumes its device.
		defer func() { _ = cluster.Terminate(ctx) }()
	}
	if err != nil {
		panic(err)
	}
	if _, err := backend.AttachOSD(ctx, cluster); err != nil {
		panic(err)
	}
	if err := backend.Apply(ctx, blockio.Profile{Delay: 250 * time.Millisecond}); err != nil {
		panic(err)
	}
	// Exercise the SUT and observe its I/O independently of the requested table.
	if _, err := backend.Status(ctx); err != nil {
		panic(err)
	}
	if err := backend.Restore(ctx); err != nil {
		panic(err)
	}
	// Restore changes the device table; restarting an exited OSD is separate.
}

func (*Container) Apply

func (b *Container) Apply(ctx context.Context, profile Profile) error

Apply installs one bounded main-device fault. A successful return confirms the requested table is active; it does not prove a client observed it. An error may leave an owned inactive table, suspension or pending kernel step. Inspect Status and retry Restore instead of replaying an uncertain request.

func (*Container) AttachOSD

func (b *Container) AttachOSD(ctx context.Context, cluster *ceph.Container) (*ceph.OSDContainer, error)

AttachOSD creates and starts one OSD on this backend. It accepts only a ready linear backend and allows one attachment attempt, including partial startup. The two-minute operation deadline also covers OSD startup. Increasing the cluster startup timeout cannot extend that deadline. The cluster restores its registered block fault before stopping daemons; this separately owned keeper stays until the caller terminates the backend.

func (*Container) GetContainerID

func (b *Container) GetContainerID() string

GetContainerID implements the container contract for partial startup too.

func (*Container) Restore

func (b *Container) Restore(ctx context.Context) error

Restore returns only this backend's recorded tables to its linear mapping. It settles prior steps before new ones, refuses foreign identity/tables, and remains retryable after cancellation or partial failure. OSD restart and acknowledged-data recovery are separate caller operations.

func (*Container) Status

func (b *Container) Status(ctx context.Context) (Status, error)

Status reads the owned identity and live/inactive tables separately from the last requested profile. Polling may confirm a recorded step, but never dispatches another mutation. A pending creation has no invented identity.

func (*Container) Terminate

func (b *Container) Terminate(ctx context.Context, opts ...testcontainers.TerminateOption) error

Terminate restores the owned table, excludes cluster claims and all device openers/I/O, then releases mapping, loop, file and keeper in that order. Terminate the attached cluster first. A failure retains remaining resources and the same handle for retry; it never forces a busy or foreign mapping.

type Identity

type Identity struct {
	Path, UUID   string
	Major, Minor uint32
	Size         int64
}

Identity identifies the mapping observed for this disposable backend. It is observation only: Run accepts no existing mapping, loop or physical disk.

type Option

type Option struct {
	// contains filtered or unexported fields
}

Option carries Config to Run; its Customize changes no container settings.

func WithConfig

func WithConfig(config Config) Option

WithConfig carries the one optional Config to Run.

func (Option) Customize

Customize implements testcontainers.ContainerCustomizer.

type Profile

type Profile struct {
	Delay time.Duration
	EIO   bool
}

Profile selects exactly one main-device fault. Delay is a positive whole number of milliseconds, at most MaxDelay. EIO makes every sector return an I/O error. It can abort the OSD; client operations may time out instead of returning EIO. A profile never implies independent observation of its effect.

type Status

type Status struct {
	Requested   Profile
	Identity    *Identity
	Tables      *Tables
	PendingStep string
	Released    bool
}

Status keeps requested settings separate from independently read kernel tables. Identity and Tables are nil until observed; PendingStep names an operation whose completion has not been confirmed. Released means that the owned mapping, loop, file and helper have all been released.

type Tables

type Tables struct {
	Live, Inactive                string
	Suspended                     bool
	Openers                       int
	ReadsInFlight, WritesInFlight int64
}

Tables is a native observation of an identified device-mapper node, not an assertion about the client's I/O, daemon health or data recovery.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL