Documentation
¶
Overview ¶
Package diagnostics collects read-only evidence from a Ceph fixture for a test to save or report: a diagnostics report of the containers, networks, log tails and fixed Ceph queries of a cluster (Collect), the Ceph crash metadata in its daemon containers (CollectCrashMetadata), and the redaction of the secrets that the cluster knows (Redact).
Collection never starts, stops, reconfigures or adopts a container. It works on a partially started or terminated cluster and on stopped or removed containers, and a failed observation stays in the returned report next to the aggregate error. A cluster needs nothing from this package to run; package finalize writes these reports into its failure archive.
The error categories are those of package ceph: invalid limits are refused with an error that matches ceph.ErrInvalidRequest.
Index ¶
Constants ¶
const ( // CrashRecordsFound means that the directory was read and held at least // one accepted record. CrashRecordsFound = CrashOutcome(evidence.CrashRecordsFound) // CrashEntriesRejected means that the directory was read, but every // metadata file in it was rejected; Omitted says why. CrashEntriesRejected = CrashOutcome(evidence.CrashEntriesRejected) // CrashDirectoryEmpty means that the directory was read and held no // metadata file. It is not proof that the daemon never crashed. CrashDirectoryEmpty = CrashOutcome(evidence.CrashDirectoryEmpty) // CrashDirectoryAbsent means that the container has no crash directory. CrashDirectoryAbsent = CrashOutcome(evidence.CrashDirectoryAbsent) // or was removed, with its crash directory. CrashContainerUnavailable = CrashOutcome(evidence.CrashContainerUnavailable) // CrashReadFailed means that Docker or the container implementation // failed, for example without permission or after the context ended. CrashReadFailed = CrashOutcome(evidence.CrashReadFailed) // CrashArchiveIncomplete means that the archive was malformed, cut off or // over a limit; Records holds what was read before. CrashArchiveIncomplete = CrashOutcome(evidence.CrashArchiveIncomplete) )
Variables ¶
This section is empty.
Functions ¶
func Redact ¶
func Redact(ctx context.Context, cluster *ceph.Container, data []byte, values ...string) ([]byte, error)
Redact returns a copy of data in which the secrets that cluster knows, and values, are replaced by [REDACTED], as in Collect: keyring keys, service credentials, every client key the cluster generated, and the private keys, secret assignments and bearer tokens that it recognizes. It is meant for output that a test logs or reports; an oracle compares the raw bytes. Redaction is best effort.
Types ¶
type Artifact ¶
type Artifact struct {
Kind string `json:"kind"`
Name string `json:"name"`
Role string `json:"role,omitempty"`
ContainerID string `json:"container_id,omitempty"`
Data string `json:"data,omitempty"`
Truncated bool `json:"truncated,omitempty"`
Error string `json:"error,omitempty"`
}
Artifact retains one observation, including incomplete output or an individual failure. Data is text; inspect observations are JSON when not truncated. Container logs retain the newest bounded text after redaction; Truncated is also set when log collection fails, including before the stream opens. Names and order are stable, but observations are not transactional.
type Config ¶
type Config struct {
Timeout time.Duration
OperationTimeout time.Duration
MaxOutputBytes int
LogTail int
Concurrency int
MaxContainers int
AdditionalContainers []Target
RedactValues []string
}
Config bounds a read-only Collect. Zero fields select a one minute total timeout, ten seconds per operation, 64 KiB per artifact, the last 1000 log lines, four workers and at most 128 containers. Additional containers are caller-owned clients or mirror daemons; collection never adopts them. RedactValues removes additional opaque application secrets. Redaction is best effort: arbitrary log contents still require review before sharing a report.
The limits are 5 minutes total, 1 minute per operation, 1 MiB per artifact, 10000 log lines, 8 workers, 128 containers, 64 additional containers and 128 redaction values of up to 4096 bytes; invalid limits are refused before collection, without a report. Artifact kinds are inventory, container- inspect, container-logs, network-inspect and ceph.
type CrashMetadataConfig ¶
type CrashMetadataConfig struct {
Timeout time.Duration
OperationTimeout time.Duration
MaxArchiveBytes int64
MaxEntries int
MaxRecordBytes int
MaxRecords int
AdditionalContainers []Target
RedactValues []string
}
CrashMetadataConfig bounds CollectCrashMetadata. Zero fields select a one minute total timeout, 20 seconds per container, 16 MiB of archive and 4096 archive entries per container, 64 KiB per metadata file and 64 records per container. AdditionalContainers are caller-owned containers, such as mirror daemons, whose ceph.CrashDir is read too; collection never adopts them. RedactValues removes additional opaque secrets from the collected fields.
The limits are 5 minutes total, 1 minute per container, 256 MiB of archive, 65536 entries, 1 MiB per metadata file, 1024 records per container (later ones are listed in Omitted), 64 additional containers and 128 redaction values of up to 4096 bytes. An additional container is read whatever its Role and Name; the control container, which runs no daemon, is read only when listed here.
type CrashMetadataContainer ¶
type CrashMetadataContainer struct {
Role string `json:"role"`
Name string `json:"name"`
ContainerID string `json:"container_id,omitempty"`
ImageID string `json:"image_id,omitempty"`
CrashDir string `json:"crash_dir"`
State string `json:"state,omitempty"`
ExitCode int `json:"exit_code"`
OOMKilled bool `json:"oom_killed,omitempty"`
StartedAt string `json:"started_at,omitempty"`
FinishedAt string `json:"finished_at,omitempty"`
Outcome CrashOutcome `json:"outcome"`
Error string `json:"error,omitempty"`
Records []CrashRecord `json:"records"`
// Omitted lists the metadata files that were rejected or not read.
Omitted []CrashOmission `json:"omitted,omitempty"`
// IgnoredEntries counts the other files, such as crash logs, whose
// content was skipped.
IgnoredEntries int `json:"ignored_entries"`
// ArchiveBytes counts the archive bytes read.
ArchiveBytes int64 `json:"archive_bytes"`
}
CrashMetadataContainer is what one container's crash directory held. The state fields are Docker's view at collection, for correlation with the records.
type CrashMetadataReport ¶
type CrashMetadataReport struct {
SchemaVersion int `json:"schema_version"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
Closed bool `json:"closed"`
Complete bool `json:"complete"`
Containers []CrashMetadataContainer `json:"containers"`
}
CrashMetadataReport is an independent, JSON-serializable report of the Ceph crash metadata in the cluster's daemon containers. Complete means that every container's crash directory was read in full and every metadata file in it was accepted, or that the directory is absent. It does not mean that every crash in the cluster's history was captured: a removed container takes its records along.
func CollectCrashMetadata ¶
func CollectCrashMetadata(ctx context.Context, cluster *ceph.Container, config CrashMetadataConfig) (*CrashMetadataReport, error)
CollectCrashMetadata reads the Ceph crash metadata files in ceph.CrashDir of the MON, MGR, OSD, MDS and RGW containers and the client toolkit of cluster, and of config.AdditionalContainers, through Docker's archive API. It works on stopped containers and on a partially bootstrapped or terminated cluster, and starts, stops or reconfigures nothing. It reads only regular files named meta in a crash report directory, keeps their known fields and redacts them like Collect; it skips crash logs and every other file, and rejects links, unexpected paths and files that are not JSON objects. Each container's outcome, rejected files and errors stay in the report, which is returned also with an error for failed or incomplete reads. Crash metadata can still hold host names and paths; review a report before sharing it.
Ceph writes a report directory (a meta JSON file, a log file and a done marker) into ceph.CrashDir when a daemon dies on a fatal signal or a failed assertion; only meta files, also in posted/, are read, other entries are counted in IgnoredEntries, and rejected files are listed in Omitted with their reason. A record keeps the known crash fields (crash_id, timestamp, process_name, entity_name, ceph_version, utsname_*, os_*, backtrace, assert_*, io_error*, archived, stack_sig) and names other fields in OmittedFields without values. A failed or incomplete container makes the call return an error with the report and the other containers' records. A container's exit code alone does not tell a crash: a daemon that is PID 1, as here, exits 1 after writing its report. It is not called automatically outside the failure archive of finalize.TerminateOnCleanup.
type CrashOmission ¶
CrashOmission names a metadata file that was not accepted, and why.
type CrashOutcome ¶
type CrashOutcome string
CrashOutcome says what reading one container's crash directory found.
type CrashRecord ¶
type CrashRecord struct {
CrashID string `json:"crash_id"`
Path string `json:"path"`
Entity string `json:"entity,omitempty"`
Process string `json:"process,omitempty"`
CephVersion string `json:"ceph_version,omitempty"`
Timestamp string `json:"timestamp,omitempty"`
Done bool `json:"done"`
Posted bool `json:"posted,omitempty"`
Fields map[string]any `json:"fields"`
OmittedFields []string `json:"omitted_fields,omitempty"`
}
CrashRecord is one daemon-written crash metadata file. Fields holds the known Ceph crash fields, redacted; OmittedFields names the others, whose values are not kept. Done reports the done marker with which Ceph completes a crash report. Posted means that the report was in the posted directory, where ceph-crash moves the reports that it sent.
type Report ¶
type Report struct {
SchemaVersion int `json:"schema_version"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
HostNetwork bool `json:"host_network"`
Closed bool `json:"closed"`
Complete bool `json:"complete"`
Artifacts []Artifact `json:"artifacts"`
}
Report is an independent, JSON-serializable snapshot. Complete means every requested observation succeeded without byte truncation, not that Ceph is healthy or that replication has completed. Logs cover only LogTail.
func Collect ¶
Collect collects, for every MON, MGR, OSD, MDS, RGW, control and client toolkit container that cluster tracks, its container state, network addresses, bounded log tail and fixed Ceph queries. It supports partial bootstrap and stopped/removed containers. Individual failures return both a usable report and an aggregate error; truncation alone does not return an error. A canceled caller needs a fresh context to collect. No daemon is started, stopped or reconfigured, and no file, environment, keyring, auth database, mount source or arbitrary command is collected.
It collects the state and selected metadata of every tracked MON, MGR, OSD, MDS, RGW, control and client toolkit container, network metadata, log tails and fixed Ceph queries (version, status, health, pgs, quorum, monmap, mgrmap, osdmap, fsmap); it never collects files, environment variables, keyrings, the auth database, mount sources or arbitrary commands. The failure archive of finalize.TerminateOnCleanup calls it; otherwise call it before cleanup with its own short context. When some queries fail, the report and an aggregate error are both returned, so save the report first; truncation alone is not an error. A ceph artifact holds the query's stdout followed by its stderr. When ctx ends, every running query is stopped in the control container before the call returns. Known admin and RGW secrets and every client key the cluster generated (also replaced or attempted ones) are masked with RedactValues and named secret patterns. A log window that starts or ends inside a PEM private key masks the visible key material up to its END delimiter or from its BEGIN delimiter.