diagnostics

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 16 Imported by: 0

README

Diagnostics

Import github.com/JSYoo5B/ceph-testcontainers-go/ceph/diagnostics alongside ceph to collect read-only evidence from a cluster for a failed test to save or report: a diagnostics report of its containers, networks, log tails and fixed Ceph queries, the crash metadata that its daemons wrote, and the redaction of the secrets that the cluster knows. Collection never starts, stops, reconfigures or adopts a container. A cluster needs nothing from this package to run. See API purpose and purpose tags for where this fits.

Order of use

  1. Start a cluster with ceph.Run.

  2. Run the test. Nothing is recorded in advance.

  3. Collect from a cleanup, with a fresh context of its own:

    Evidence Function Report
    Container state, network addresses, log tails, fixed Ceph queries Collect Report
    Crash metadata files in ceph.CrashDir of the daemon containers CollectCrashMetadata CrashMetadataReport
    Output with the cluster's secrets replaced by [REDACTED] Redact the masked bytes
t.Cleanup(func() {
	if !t.Failed() {
		return
	}
	ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
	defer cancel()
	report, err := diagnostics.Collect(ctx, cluster, diagnostics.Config{})
	if report != nil {
		data, _ := json.MarshalIndent(report, "", "  ")
		t.Logf("ceph diagnostics:\n%s", data)
	}
	if err != nil {
		t.Logf("diagnostics incomplete: %v", err)
	}
})

To collect both reports into a failure archive on one ordered cleanup path, use the finalize package instead.

Concepts and limits

  • Collection is standalone and read-only. It works on a partially started cluster, after Terminate, and on stopped or removed containers. It runs no arbitrary command and reads no file, environment, keyring, auth database or mount source; CollectCrashMetadata reads only crash meta files through Docker's archive API.
  • A partial report is returned together with the aggregate error when some observations fail: keep the report. Truncation alone is not an error. Report.Complete means every observation succeeded untruncated, not that Ceph is healthy.
  • A canceled test context cannot collect: t.Context() has ended when cleanups run.
  • Config and CrashMetadataConfig bound time, bytes, records and concurrency; zero fields select defaults, and invalid limits are refused with an error that matches ceph.ErrInvalidRequest, without a report. Caller-owned clients and mirror daemons are added through AdditionalContainers (Target) and are never adopted.
  • Redaction is best effort: known admin and service credentials, every client key the cluster generated, RedactValues and recognized secret patterns are masked. Log contents and crash metadata can still hold host names, paths and application secrets; review a report before sharing it. An oracle compares the raw bytes, not the output of Redact.
  • No crash records is not proof that no daemon crashed: a container removed earlier took its records along.

Reference

The full contract of every type and function is in go doc: pkg.go.dev or go doc -all github.com/JSYoo5B/ceph-testcontainers-go/ceph/diagnostics. Collection from cleanup is in Contexts and cleanup, the collected queries in Running native commands, and the failure archive layout in Ceph operations.

Documentation

Overview

Package diagnostics collects read-only evidence from a Ceph fixture for a test to save or report: a diagnostics report of the containers, networks, log tails and fixed Ceph queries of a cluster (Collect), the Ceph crash metadata in its daemon containers (CollectCrashMetadata), and the redaction of the secrets that the cluster knows (Redact).

Collection never starts, stops, reconfigures or adopts a container. It works on a partially started or terminated cluster and on stopped or removed containers, and a failed observation stays in the returned report next to the aggregate error. A cluster needs nothing from this package to run; package finalize writes these reports into its failure archive.

The error categories are those of package ceph: invalid limits are refused with an error that matches ceph.ErrInvalidRequest.

Index

Constants

View Source
const (
	// CrashRecordsFound means that the directory was read and held at least
	// one accepted record.
	CrashRecordsFound = CrashOutcome(evidence.CrashRecordsFound)
	// CrashEntriesRejected means that the directory was read, but every
	// metadata file in it was rejected; Omitted says why.
	CrashEntriesRejected = CrashOutcome(evidence.CrashEntriesRejected)
	// CrashDirectoryEmpty means that the directory was read and held no
	// metadata file. It is not proof that the daemon never crashed.
	CrashDirectoryEmpty = CrashOutcome(evidence.CrashDirectoryEmpty)
	// CrashDirectoryAbsent means that the container has no crash directory.
	CrashDirectoryAbsent = CrashOutcome(evidence.CrashDirectoryAbsent)
	// CrashContainerUnavailable means that the container was never created
	// or was removed, with its crash directory.
	CrashContainerUnavailable = CrashOutcome(evidence.CrashContainerUnavailable)
	// CrashReadFailed means that Docker or the container implementation
	// failed, for example without permission or after the context ended.
	CrashReadFailed = CrashOutcome(evidence.CrashReadFailed)
	// CrashArchiveIncomplete means that the archive was malformed, cut off or
	// over a limit; Records holds what was read before.
	CrashArchiveIncomplete = CrashOutcome(evidence.CrashArchiveIncomplete)
)

Variables

This section is empty.

Functions

func Redact

func Redact(ctx context.Context, cluster *ceph.Container, data []byte, values ...string) ([]byte, error)

Redact returns a copy of data in which the secrets that cluster knows, and values, are replaced by [REDACTED], as in Collect: keyring keys, service credentials, every client key the cluster generated, and the private keys, secret assignments and bearer tokens that it recognizes. It is meant for output that a test logs or reports; an oracle compares the raw bytes. Redaction is best effort.

Types

type Artifact

type Artifact struct {
	Kind        string `json:"kind"`
	Name        string `json:"name"`
	Role        string `json:"role,omitempty"`
	ContainerID string `json:"container_id,omitempty"`
	Data        string `json:"data,omitempty"`
	Truncated   bool   `json:"truncated,omitempty"`
	Error       string `json:"error,omitempty"`
}

Artifact retains one observation, including incomplete output or an individual failure. Data is text; inspect observations are JSON when not truncated. Container logs retain the newest bounded text after redaction; Truncated is also set when log collection fails, including before the stream opens. Names and order are stable, but observations are not transactional.

type Config

type Config struct {
	Timeout              time.Duration
	OperationTimeout     time.Duration
	MaxOutputBytes       int
	LogTail              int
	Concurrency          int
	MaxContainers        int
	AdditionalContainers []Target
	RedactValues         []string
}

Config bounds a read-only Collect. Zero fields select a one minute total timeout, ten seconds per operation, 64 KiB per artifact, the last 1000 log lines, four workers and at most 128 containers. Additional containers are caller-owned clients or mirror daemons; collection never adopts them. RedactValues removes additional opaque application secrets. Redaction is best effort: arbitrary log contents still require review before sharing a report.

The limits are 5 minutes total, 1 minute per operation, 1 MiB per artifact, 10000 log lines, 8 workers, 128 containers, 64 additional containers and 128 redaction values of up to 4096 bytes; invalid limits are refused before collection, without a report. Artifact kinds are inventory, container- inspect, container-logs, network-inspect and ceph.

type CrashMetadataConfig

type CrashMetadataConfig struct {
	Timeout              time.Duration
	OperationTimeout     time.Duration
	MaxArchiveBytes      int64
	MaxEntries           int
	MaxRecordBytes       int
	MaxRecords           int
	AdditionalContainers []Target
	RedactValues         []string
}

CrashMetadataConfig bounds CollectCrashMetadata. Zero fields select a one minute total timeout, 20 seconds per container, 16 MiB of archive and 4096 archive entries per container, 64 KiB per metadata file and 64 records per container. AdditionalContainers are caller-owned containers, such as mirror daemons, whose ceph.CrashDir is read too; collection never adopts them. RedactValues removes additional opaque secrets from the collected fields.

The limits are 5 minutes total, 1 minute per container, 256 MiB of archive, 65536 entries, 1 MiB per metadata file, 1024 records per container (later ones are listed in Omitted), 64 additional containers and 128 redaction values of up to 4096 bytes. An additional container is read whatever its Role and Name; the control container, which runs no daemon, is read only when listed here.

type CrashMetadataContainer

type CrashMetadataContainer struct {
	Role        string `json:"role"`
	Name        string `json:"name"`
	ContainerID string `json:"container_id,omitempty"`
	ImageID     string `json:"image_id,omitempty"`
	CrashDir    string `json:"crash_dir"`
	State       string `json:"state,omitempty"`
	ExitCode    int    `json:"exit_code"`
	OOMKilled   bool   `json:"oom_killed,omitempty"`
	StartedAt   string `json:"started_at,omitempty"`
	FinishedAt  string `json:"finished_at,omitempty"`

	Outcome CrashOutcome  `json:"outcome"`
	Error   string        `json:"error,omitempty"`
	Records []CrashRecord `json:"records"`
	// Omitted lists the metadata files that were rejected or not read.
	Omitted []CrashOmission `json:"omitted,omitempty"`
	// IgnoredEntries counts the other files, such as crash logs, whose
	// content was skipped.
	IgnoredEntries int `json:"ignored_entries"`
	// ArchiveBytes counts the archive bytes read.
	ArchiveBytes int64 `json:"archive_bytes"`
}

CrashMetadataContainer is what one container's crash directory held. The state fields are Docker's view at collection, for correlation with the records.

type CrashMetadataReport

type CrashMetadataReport struct {
	SchemaVersion int                      `json:"schema_version"`
	StartedAt     time.Time                `json:"started_at"`
	FinishedAt    time.Time                `json:"finished_at"`
	Closed        bool                     `json:"closed"`
	Complete      bool                     `json:"complete"`
	Containers    []CrashMetadataContainer `json:"containers"`
}

CrashMetadataReport is an independent, JSON-serializable report of the Ceph crash metadata in the cluster's daemon containers. Complete means that every container's crash directory was read in full and every metadata file in it was accepted, or that the directory is absent. It does not mean that every crash in the cluster's history was captured: a removed container takes its records along.

func CollectCrashMetadata

func CollectCrashMetadata(ctx context.Context, cluster *ceph.Container, config CrashMetadataConfig) (*CrashMetadataReport, error)

CollectCrashMetadata reads the Ceph crash metadata files in ceph.CrashDir of the MON, MGR, OSD, MDS and RGW containers and the client toolkit of cluster, and of config.AdditionalContainers, through Docker's archive API. It works on stopped containers and on a partially bootstrapped or terminated cluster, and starts, stops or reconfigures nothing. It reads only regular files named meta in a crash report directory, keeps their known fields and redacts them like Collect; it skips crash logs and every other file, and rejects links, unexpected paths and files that are not JSON objects. Each container's outcome, rejected files and errors stay in the report, which is returned also with an error for failed or incomplete reads. Crash metadata can still hold host names and paths; review a report before sharing it.

Ceph writes a report directory (a meta JSON file, a log file and a done marker) into ceph.CrashDir when a daemon dies on a fatal signal or a failed assertion; only meta files, also in posted/, are read, other entries are counted in IgnoredEntries, and rejected files are listed in Omitted with their reason. A record keeps the known crash fields (crash_id, timestamp, process_name, entity_name, ceph_version, utsname_*, os_*, backtrace, assert_*, io_error*, archived, stack_sig) and names other fields in OmittedFields without values. A failed or incomplete container makes the call return an error with the report and the other containers' records. A container's exit code alone does not tell a crash: a daemon that is PID 1, as here, exits 1 after writing its report. It is not called automatically outside the failure archive of finalize.TerminateOnCleanup.

type CrashOmission

type CrashOmission struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
}

CrashOmission names a metadata file that was not accepted, and why.

type CrashOutcome

type CrashOutcome string

CrashOutcome says what reading one container's crash directory found.

type CrashRecord

type CrashRecord struct {
	CrashID       string         `json:"crash_id"`
	Path          string         `json:"path"`
	Entity        string         `json:"entity,omitempty"`
	Process       string         `json:"process,omitempty"`
	CephVersion   string         `json:"ceph_version,omitempty"`
	Timestamp     string         `json:"timestamp,omitempty"`
	Done          bool           `json:"done"`
	Posted        bool           `json:"posted,omitempty"`
	Fields        map[string]any `json:"fields"`
	OmittedFields []string       `json:"omitted_fields,omitempty"`
}

CrashRecord is one daemon-written crash metadata file. Fields holds the known Ceph crash fields, redacted; OmittedFields names the others, whose values are not kept. Done reports the done marker with which Ceph completes a crash report. Posted means that the report was in the posted directory, where ceph-crash moves the reports that it sent.

type Report

type Report struct {
	SchemaVersion int        `json:"schema_version"`
	StartedAt     time.Time  `json:"started_at"`
	FinishedAt    time.Time  `json:"finished_at"`
	HostNetwork   bool       `json:"host_network"`
	Closed        bool       `json:"closed"`
	Complete      bool       `json:"complete"`
	Artifacts     []Artifact `json:"artifacts"`
}

Report is an independent, JSON-serializable snapshot. Complete means every requested observation succeeded without byte truncation, not that Ceph is healthy or that replication has completed. Logs cover only LogTail.

func Collect

func Collect(ctx context.Context, cluster *ceph.Container, config Config) (*Report, error)

Collect collects, for every MON, MGR, OSD, MDS, RGW, control and client toolkit container that cluster tracks, its container state, network addresses, bounded log tail and fixed Ceph queries. It supports partial bootstrap and stopped/removed containers. Individual failures return both a usable report and an aggregate error; truncation alone does not return an error. A canceled caller needs a fresh context to collect. No daemon is started, stopped or reconfigured, and no file, environment, keyring, auth database, mount source or arbitrary command is collected.

It collects the state and selected metadata of every tracked MON, MGR, OSD, MDS, RGW, control and client toolkit container, network metadata, log tails and fixed Ceph queries (version, status, health, pgs, quorum, monmap, mgrmap, osdmap, fsmap); it never collects files, environment variables, keyrings, the auth database, mount sources or arbitrary commands. The failure archive of finalize.TerminateOnCleanup calls it; otherwise call it before cleanup with its own short context. When some queries fail, the report and an aggregate error are both returned, so save the report first; truncation alone is not an error. A ceph artifact holds the query's stdout followed by its stderr. When ctx ends, every running query is stopped in the control container before the call returns. Known admin and RGW secrets and every client key the cluster generated (also replaced or attempted ones) are masked with RedactValues and named secret patterns. A log window that starts or ends inside a PEM private key masks the visible key material up to its END delimiter or from its BEGIN delimiter.

type Target

type Target struct {
	Role      string
	Name      string
	Container testcontainers.Container
}

Target explicitly adds an external container to a report. Name and Role describe it; Container is only inspected and its logs read.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL