openstack

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Overview

Package openstack is the OpenStack event collector: it consumes oslo.messaging notifications from AMQP, maps them to Tally events, buffers them in a SQLite outbox, and posts them to the Reporting API. This file parses the collector's environment configuration.

The collector has two modes and each runs its own validation gate. Serving needs the whole pipeline, so it requires the broker, the cloud, the Reporting API and its token, and the buffer path. Dumping notifications maps nothing and posts nothing, so the broker alone is enough and the operator exploring an unknown deployment is not asked for values that mode never reads.

Configuration comes from the environment alone, prefixed TALLY_. Secrets also accept the *_FILE convention, which is how a Kubernetes Secret volume reaches the process without the value appearing in a pod spec.

The normative specification is roadmap/00-conventions.md section 8.

Index

Constants

View Source
const LabelValueLimit = 100

LabelValueLimit is how many distinct event types the series may carry, on the terms internal/core/cardinality bounds them with.

It is 100 here and 128 in internal/reporting/metrics on purpose. Each service bounds what its own traffic can mint, and 100 is several times the mapping table, which is the vocabulary a healthy deployment stays inside. The two numbers are not meant to track each other.

It is exported because the simulator is held to it: a generated month has to stay inside the bound, or the series an operator is told to read fold into event_type="other".

Variables

View Source
var EnvNames = []string{
	envLogLevel,
	envMetricsEnabled,
	envHTTPPort,
	envAMQPURL,
	envAMQPURL + fileSuffix,
	envExchanges,
	envTopics,
	envRequireExchanges,
	envQueueType,
	envCloud,
	envReportingURL,
	envReportingInsecure,
	envToken,
	envToken + fileSuffix,
	envBufferPath,
	envBatchMax,
	envFlushInterval,
	envBufferMaxEvents,
	envPrefetch,
	envUnhealthyThreshold,
	envSummaryInterval,
}

EnvNames is every variable this package reads, including the *_FILE companions of the secrets. Tests blank all of them, so a value in the developer's shell never reaches the code under test.

Functions

func Dump

func Dump(ctx context.Context, cfg Config, out io.Writer, logger *slog.Logger) error

Dump prints the notifications a deployment publishes, one JSON line per delivery, until ctx is done.

It is the check that comes before a rollout. The oslo event types and their payload members differ per OpenStack release, and the exchange and topic names are a deployment's own configuration, so what the mapping table expects is verified against what the broker actually carries before the collector is pointed at it.

The dump consumes through an exclusive, server-named, auto-deleting queue and acknowledges nothing it reads. A topic exchange copies every message to every bound queue, so what the dump prints is a copy: the collector's durable queue, Ceilometer, and a collector running at the same time all keep receiving theirs, and the broker is left as the dump found it.

func MapNotification

func MapNotification(n Notification, cloud string) (event.Event, bool)

MapNotification turns one oslo notification into the Tally event it stands for. The second return is false when the notification is not mapped, either because no table entry claims its type or because the entry's own rule says this particular notification carries nothing to record yet.

Mapping never fails. A payload missing the resource id or the project still produces an event, and the Reporting API dead-letters it with the validation reason it broke. That leaves an operator a record of a notification the mapping did not understand, where dropping it here would have been silent.

Types

type Config

type Config struct {
	// LogLevel is the slog threshold, one of DEBUG, INFO, WARN, or ERROR.
	LogLevel string `env:"TALLY_LOG_LEVEL" envDefault:"INFO"`
	// MetricsEnabled exposes the instrumentation: false makes GET /metrics answer
	// 404. The variable has no OSC infix because roadmap section 8 lists it among
	// the common variables every service reads.
	MetricsEnabled bool `env:"TALLY_METRICS_ENABLED" envDefault:"true"`
	// HTTPPort is the port the probe and metrics endpoints listen on. The
	// collector serves no API of its own.
	HTTPPort int `env:"TALLY_OSC_HTTP_PORT" envDefault:"8080"`
	// AMQPURL is the broker the notifications are consumed from. It carries the
	// broker password, so it supports the *_FILE convention.
	AMQPURL string `env:"TALLY_OSC_AMQP_URL"`
	// Exchanges are the service exchanges the collector binds its queue to. The
	// default covers nova, neutron, cinder and glance: cinder sets no
	// control_exchange of its own and publishes on oslo's default, openstack. A
	// deployment that renamed an exchange through control_exchange lists its own.
	//
	// An exchange the broker does not carry is skipped with a warning and bound
	// once it appears, which is what a fresh cloud needs: glance declares its
	// exchange with its first notification. Octavia publishes on the exchange
	// octavia, and the default leaves it out, because a deployment that runs none
	// would report it missing for as long as the collector runs. A deployment with
	// octavia lists nova,neutron,openstack,glance,octavia.
	Exchanges []string `env:"TALLY_OSC_EXCHANGES" envSeparator:"," envDefault:"nova,neutron,openstack,glance"`
	// Topics are the notification topics bound on each exchange, matching the
	// notification_topics of the services being collected.
	Topics []string `env:"TALLY_OSC_TOPICS" envSeparator:"," envDefault:"notifications.info"`
	// RequireExchanges makes a missing exchange fail the AMQP session instead of
	// being skipped: the collector then consumes nothing until the broker carries
	// every exchange it lists. It is for a deployment that would rather stop than
	// collect from a part of the cloud, and for the simulator stack, whose wait
	// for the collector reads a consumer on the queue as a queue that is bound.
	RequireExchanges bool `env:"TALLY_OSC_REQUIRE_EXCHANGES" envDefault:"false"`
	// QueueType is the type the queue tally-notifications is declared with,
	// quorum or classic. quorum is the default: it declares a replicated queue
	// with the delivery limit disabled, and needs RabbitMQ 4.0 or newer. classic
	// sends no queue type and leaves the choice to the broker: a classic queue on
	// one node, unless the virtual host's default_queue_type is quorum, which
	// creates a quorum queue with the broker's own delivery limit. classic is the
	// setting for a broker older than RabbitMQ 4.0 and for a queue that stays on
	// one node. An existing queue keeps its type, so declaring it with the other
	// one means deleting the queue first.
	QueueType string `env:"TALLY_OSC_QUEUE_TYPE" envDefault:"quorum"`
	// Cloud is the cloud name every emitted event is attributed to. It has no
	// default because a guessed cloud silently books usage to the wrong one.
	Cloud string `env:"TALLY_OSC_CLOUD"`
	// ReportingURL is the base URL of the Reporting API the sender posts to. It
	// must be an absolute https URL, because the ingest token travels on it;
	// ReportingInsecure is what allows a plaintext one.
	ReportingURL string `env:"TALLY_OSC_REPORTING_URL"`
	// ReportingInsecure allows an http Reporting API. It exists for a collector
	// and an API on the same trusted network, and for development; anywhere else
	// it puts the ingest token on the wire in cleartext.
	ReportingInsecure bool `env:"TALLY_OSC_REPORTING_INSECURE" envDefault:"false"`
	// Token authenticates the sender against the Reporting API. Supports the
	// *_FILE convention.
	Token string `env:"TALLY_OSC_TOKEN"`
	// BufferPath is the SQLite file backing the outbox. It belongs on a volume
	// that outlives the container: everything consumed but not yet delivered
	// lives there and nowhere else.
	BufferPath string `env:"TALLY_OSC_BUFFER_PATH"`
	// BatchMax is how many buffered events one POST carries, bounded by what the
	// ingest API accepts.
	BatchMax int `env:"TALLY_OSC_BATCH_MAX" envDefault:"500"`
	// FlushIntervalSeconds is how long the sender waits before posting a batch
	// that has not filled up.
	FlushIntervalSeconds int `env:"TALLY_OSC_FLUSH_INTERVAL_S" envDefault:"5"`
	// BufferMaxEvents is the outbox depth at which the collector stops consuming.
	// The events then wait on the bus instead of being dropped, which is what
	// keeps an unreachable Reporting API from costing usage data.
	BufferMaxEvents int64 `env:"TALLY_OSC_BUFFER_MAX_EVENTS" envDefault:"1000000"`
	// Prefetch is the AMQP QoS bound: how many unacknowledged messages the broker
	// hands out. Acks follow the outbox insert, so this bounds how much work a
	// crash replays.
	Prefetch int `env:"TALLY_OSC_PREFETCH" envDefault:"100"`
	// UnhealthyThresholdSeconds is how long readiness may keep failing before
	// liveness fails too and the orchestrator restarts the pod.
	UnhealthyThresholdSeconds int `env:"TALLY_OSC_UNHEALTHY_THRESHOLD_S" envDefault:"600"`
	// SummaryIntervalSeconds is how often the collector logs its summary line: the
	// notifications consumed, skipped and unparseable and the events delivered
	// since the previous line, with the state of the session and of the outbox.
	// The line is logged whether or not anything happened, so a collector at rest
	// still reports itself. It cannot be turned off; TALLY_LOG_LEVEL=WARN hides it.
	SummaryIntervalSeconds int `env:"TALLY_OSC_SUMMARY_INTERVAL_S" envDefault:"60"`
}

Config is the collector's resolved configuration. Every field is final by the time Load returns: file-backed secrets hold their content, the log level and the queue type are ones this package accepts, and every bounded number is within its bounds.

func Load

func Load() (Config, error)

Load reads the environment, resolves the file-backed secrets, and checks the log level, the queue type and the numeric bounds. It does not check whether the required values are present: which ones are required depends on the mode, which is what ValidateServe and ValidateDump decide.

func (Config) SlogLevel

func (c Config) SlogLevel() slog.Level

SlogLevel is the slog level that LogLevel names. A Config that never went through Load, and so carries an unchecked level, logs at info.

func (Config) ValidateDump

func (c Config) ValidateDump() error

ValidateDump is the notification dump's gate. Dumping prints what the broker delivers: it maps nothing and posts nothing, so it needs neither a cloud nor a destination to send to.

func (Config) ValidateServe

func (c Config) ValidateServe() error

ValidateServe is the collecting service's startup gate. It refuses a configuration the collector cannot honor instead of starting and losing the notifications it consumes.

type Consumer

type Consumer struct {
	// contains filtered or unexported fields
}

Consumer reads oslo notifications off AMQP and buffers the events they map to.

The chain is at-least-once, and the acknowledgement is what makes it so: a delivery is acknowledged only after the mapped event is committed to the outbox, so a crash between the two leaves the notification on the broker and the next connection is handed it again. That redelivery costs nothing, because the event carries the oslo message id as its event_id and the Reporting API deduplicates on it. Every retry along the way rests on that one property: a redelivered notification, a resent batch, and an outbox replayed after a restart all arrive as the same event.

func NewConsumer

func NewConsumer(cfg Config, buffer eventBuffer, m *Metrics, logger *slog.Logger) *Consumer

NewConsumer builds the consumer over cfg's broker, buffering into buffer.

m may be nil, which records nothing, and logger may be nil, which logs through the default logger.

func (*Consumer) Connected

func (c *Consumer) Connected() bool

Connected reports whether a connection and a consumer are established. It is what the readiness probe reads. A consumer paused by backpressure counts as connected: the connection is up and not consuming is the deliberate part. A session that skipped a missing exchange counts as connected too: it consumes from every exchange the broker carries.

func (*Consumer) Run

func (c *Consumer) Run(ctx context.Context) error

Run consumes until ctx is done, and then returns nil, which is the only way it returns. A connection that fails to come up, or one the broker closes, is retried with a growing wait: a broker that is restarting costs the collector a pause rather than the process, and the notifications wait on the bus meanwhile.

An exchange the broker does not carry is skipped and bound once it appears, so a service that has not declared its exchange yet costs the notifications of that service and not the session. A session fails over its exchanges in two cases only: none of the listed ones exists, or Config.RequireExchanges is set and one of them is missing.

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds the collector's instruments and the registry they are registered on: the five counters over the consume and the deliver path that WP 1.12 defines, plus the two gauges that report the outbox. It registers them together with the Go runtime and process collectors, which Handler then serves.

The zero value is not usable; call NewMetrics. A nil *Metrics is usable and records nothing, so a collector built without metrics does not panic on every notification. Passing a nil *Metrics is the supported way to turn the instrumentation off.

The delivered counter counts what an answer says the Reporting API stored, not how many events the batch carried: a resent batch is answered as duplicates, and an item the API dead-lettered was refused. Counting the batch instead would report full throughput while every event in it is being refused, which is the one failure the counter has to make visible.

The totals mirror the five counters as plain numbers for the summary line, which Totals reads.

Every one of the seven series carries the constant labels platform and cloud, the cloud being the one TALLY_OSC_CLOUD names. One process serves one cloud and only the process knows which: the store finds an in-cluster collector through a discovered scrape job, which cannot stamp a label per cloud, so the series name their cloud themselves and two collectors stay apart. The Go runtime and process collectors carry neither, because they report the process rather than the product.

The normative specification is roadmap/01-phase-1-core-platform-openstack.md, WP 1.12.

func NewMetrics

func NewMetrics(reg *prometheus.Registry, cloud string, depth, oldestSeconds func() float64) *Metrics

NewMetrics builds the instruments and registers them on reg, along with the Go runtime and process collectors. reg is also what Handler gathers from. It panics if reg already carries one of these collectors, which makes a second NewMetrics over the same registry a programming error rather than a silent half-registration. It panics on an empty cloud too: the binary refuses an empty TALLY_OSC_CLOUD before it builds the metrics, so an empty cloud here is a caller's mistake rather than a state the process runs into.

depth and oldestSeconds back the two gauges and are read at scrape time, not here. The binary wires them to the outbox's Depth and OldestBufferedSeconds, which is why the buffer needs no recording method of its own: what a gauge reports is a state the outbox already knows, not a sum of events.

func (*Metrics) Consumed

func (m *Metrics) Consumed(eventType string)

Consumed counts one notification that was mapped and buffered. The event type is bounded before it becomes a label value, because it is whatever the broker delivered.

func (*Metrics) Delivered

func (m *Metrics) Delivered(n int)

Delivered counts the n events one 200 answer covered.

func (*Metrics) DeliveryError

func (m *Metrics) DeliveryError()

DeliveryError counts one delivery attempt that failed. A batch the Reporting API refused stays in the outbox and is offered again, so one batch may count here several times before it counts as delivered once.

func (*Metrics) Handler

func (m *Metrics) Handler() http.Handler

Handler serves the registry in the Prometheus exposition format. It is what /metrics is mounted on. Every scrape is answered by the one handler built in NewMetrics, because the in-flight bound above is a bound only while the scrapes share it.

func (*Metrics) Skipped

func (m *Metrics) Skipped(eventType string)

Skipped counts one notification that produced no event, because no mapping table entry covers its type or the entry gated it off. An unmapped type is what mints a series here, so the event type this counter is handed is arbitrary by construction and bounded like the consumed one.

func (*Metrics) Totals added in v0.3.0

func (m *Metrics) Totals() Totals

Totals reads the five counts. A nil *Metrics reports zeros.

func (*Metrics) Unparseable

func (m *Metrics) Unparseable()

Unparseable counts one delivery whose body did not parse.

type Notification

type Notification struct {
	// MessageID is the id the emitting service assigned the notification. It is
	// unique per notification, which makes it the key a redelivery is recognized
	// by.
	MessageID string
	// EventType names what happened, such as "compute.instance.create.end". The
	// namespace is the emitting service's own, not Tally's.
	EventType string
	// Timestamp is when the service recorded the event, always in UTC.
	Timestamp time.Time
	// Payload is the service's description of the resource, decoded as it
	// arrived. Its numbers are json.Number and not float64, so a quantity keeps
	// every digit it was sent with and can still be turned into an exact decimal.
	Payload map[string]any
	// ContextProjectID is the project the request ran in, taken from the request
	// context the service attaches to its notifications.
	ContextProjectID string
	// ContextTenantID is the same project under its older name. Services differ
	// in which of the two they set, so both are read and the caller takes
	// whichever is populated.
	ContextTenantID string
}

Notification is one oslo.messaging notification as an OpenStack service emitted it: what happened, when, in which project, and the service's own description of the resource. It is the raw provider fact, before any mapping into the canonical event schema.

func ParseEnvelope

func ParseEnvelope(body []byte) (Notification, error)

ParseEnvelope decodes a message body into the notification it carries.

The body has two layers. The outer object is the oslo envelope, which holds the notification under "oslo.message" as a string, and that string is the JSON document the emitting service wrote. Both layers are decoded here.

Only the timestamp is required. A notification without a message id, event type, payload, or project comes back as it arrived, because deciding what to do about a gap belongs to the mapping rather than to the decoder. An error means the body is unusable: the consumer acknowledges such a delivery instead of requeueing it, since a body that fails to parse fails again on every redelivery, so this text is all the operator gets to see.

type Outbox

type Outbox struct {
	// contains filtered or unexported fields
}

Outbox is the collector's durable buffer between the consumer and the sender. A mapped event is committed here before its notification is acknowledged, so the broker never drops a message Tally has not written down.

Two goroutines share one handle: the consumer inserts, the sender reads a batch and deletes it once the Reporting API has taken it. WAL mode lets that read run while an insert commits, and the busy timeout absorbs the moments where SQLite still serializes the two.

synchronous is FULL rather than the WAL default NORMAL because the acknowledgement is a promise. NORMAL leaves a commit in the operating system's cache, which a crashing process survives but a power loss does not, and by then the broker has already given the message away. FULL costs an fsync per insert and buys back the guarantee the acknowledgement rests on.

func OpenOutbox

func OpenOutbox(path string) (*Outbox, error)

OpenOutbox opens the buffer at path, creates the file and the schema when they do not exist yet, and picks up the events the file already carries.

path belongs on a volume that outlives the container: between the acknowledgement and the delivery, an event lives here and nowhere else.

func (*Outbox) Batch

func (o *Outbox) Batch(ctx context.Context, limit int) ([]Row, error)

Batch reads up to limit buffered events, oldest first. An empty buffer yields an empty batch and no error, which is the ordinary answer for a sender polling a quiet collector.

func (*Outbox) Close

func (o *Outbox) Close() error

Close releases the handle. Buffered events stay in the file and are picked up by the next start.

func (*Outbox) DeleteBatch

func (o *Outbox) DeleteBatch(ctx context.Context, ids []int64) error

DeleteBatch removes the events one 200 answer covered. It addresses them by id instead of by a bound on id, so the rows the consumer inserted while the batch was in flight stay buffered. An empty list of ids deletes nothing.

It runs after a 200 and only then: a failed POST leaves the batch where it is and the next round offers it again.

func (*Outbox) Depth

func (o *Outbox) Depth() int64

Depth is the number of buffered events. The consumer weighs it against its bound once per message, so it answers from memory and never queries.

func (*Outbox) Insert

func (o *Outbox) Insert(ctx context.Context, eventJSON []byte) error

Insert commits one mapped event. It returns after the row is on disk, which is what makes acknowledging the notification next a safe thing to do.

func (*Outbox) OldestBufferedSeconds

func (o *Outbox) OldestBufferedSeconds() float64

OldestBufferedSeconds is the age of the oldest buffered event, and 0 when the buffer is empty. It backs the gauge that shows how far delivery has fallen behind: a depth that holds steady under load is ordinary, an age that keeps climbing is not.

The oldest event is the one with the smallest id, because the sender deletes in that order. There is no context parameter since Prometheus calls this through a GaugeFunc.

A buffer that cannot be read reports NaN and not 0, because 0 is the value that means drained: the query's one-second bound is shorter than the busy timeout, so what it gives up on is a buffer under the write contention of a backlog, which is exactly when the gauge must not read as healthy. Prometheus stores NaN, and no alert expression matches it.

func (*Outbox) Ping

func (o *Outbox) Ping(ctx context.Context) error

Ping checks that the buffer answers. It is what the probes report on: a collector whose outbox is unusable can neither acknowledge a notification nor deliver one.

It reads the outbox table rather than a constant expression. A constant would prove that the driver answers and nothing else, while what the probe is asked about is whether events can be written down and read back, which a file whose schema this build cannot use answers differently.

type Row

type Row struct {
	// ID orders the buffer and addresses the row in DeleteBatch.
	ID int64
	// EventJSON is the canonical event, byte for byte as Insert received it.
	EventJSON []byte
}

Row is one buffered event: the id the sender deletes it by, and the event document as it was inserted.

type Sender

type Sender struct {
	// contains filtered or unexported fields
}

Sender empties the outbox into the Reporting API. It runs independently of the consumer: the consumer's promise ends at the buffer, and this is what carries the buffered events the rest of the way.

One rule decides a batch's fate, with the single exception named below. A 200 deletes it and nothing else does: every other status, and every error that keeps a status from arriving at all, leaves the batch buffered and offers it again after a wait. Resending costs nothing, because the event carries the oslo message id as its event_id and the API deduplicates on it, so a needless retry is answered as a duplicate. Deleting on anything but a 200 would cost usage nobody can reconstruct, which is why a misconfigured token retries visibly until an operator fixes it rather than draining the buffer into nowhere.

The items a 200 refused are not a failure of the batch. The Reporting API keeps a refused item dead-lettered on its side, so offering it again would only have it refused again: the sender logs each one and deletes the batch with the rest of it.

A 413 is the one status the rule above cannot answer with a plain retry: a body past what the API accepts is past it on every attempt. The batch is halved until it fits, and the size that worked is grown back gradually but never past the smallest size that was refused, so a destination with a smaller body bound than the configured batch settles on one that fits instead of walking the ladder again after every batch.

A single event that is still refused is dropped only when it is past eventMax, which is the bound the consumer buffered it under. Below that bound the 413 describes the destination and not the event, so the event is kept and retried like every other refusal: an ingress with a small body limit would otherwise destroy the whole outbox one event at a time, and those events are gone from SQLite and already acknowledged on the bus.

The retry lives in this loop rather than in the HTTP client. What is retried is the batch and not the request: the wait belongs to the outbox, which holds the events meanwhile, and the attempt that follows reads the buffer again instead of resending a body kept in memory.

func NewSender

func NewSender(outbox eventBatches, reportingURL, token string, batchMax int,
	flushInterval time.Duration, m *Metrics, logger *slog.Logger,
) *Sender

NewSender builds the loop that posts outbox's events to reportingURL, at most batchMax of them per request, waiting flushInterval between the rounds that found nothing to send or did not fill a batch.

m may be nil, which records nothing, and logger may be nil, which logs through the default logger.

func (*Sender) Run

func (s *Sender) Run(ctx context.Context) error

Run delivers until ctx is done, and then returns nil, which is the only way it returns. A failed attempt is waited out with a growing backoff, so a Reporting API that is restarting costs the collector a pause rather than the process, and one that stays down is asked once every five minutes while the events wait in the outbox.

A batch that filled up is followed by the next one without a wait, which is what lets a backlog drain at the speed the API takes it rather than at one batch per flush interval.

type Summary added in v0.3.0

type Summary struct {
	// contains filtered or unexported fields
}

Summary logs the collector's summary line once per interval.

The line carries what the consumer and the sender counted since the previous line, whether the consumer holds a session, and what waits in the outbox. It is logged whether or not anything happened, so a collector at rest keeps reporting itself, and one cut off from the broker, one that cannot deliver and one that receives nothing log different lines.

func NewSummary added in v0.3.0

func NewSummary(interval time.Duration, m *Metrics, connected func() bool,
	depth func() int64, oldestSeconds func() float64, logger *slog.Logger,
) *Summary

NewSummary builds the loop that logs one summary line per interval.

The counts come from m. connected, depth and oldestSeconds are read when a line is logged; the binary wires them to the consumer's Connected and to the outbox's Depth and OldestBufferedSeconds.

m may be nil, which logs the counts as zeros, and logger may be nil, which logs through the default logger.

func (*Summary) Run added in v0.3.0

func (s *Summary) Run(ctx context.Context) error

Run logs a summary line after every interval until ctx is done, and then returns nil, which is the only way it returns. It logs no line when it starts and none when it stops: the first line follows one interval after the start and covers the time since then, and an interval the shutdown cut short gets none.

The five counts are what was recorded since the previous line. The session and the outbox are read at the time of the line.

type Totals added in v0.3.0

type Totals struct {
	Consumed, Skipped, Unparseable, Delivered, DeliveryErrors int64
}

Totals is what the collector has counted since it started, as plain numbers. It is what the summary line is computed from.

Directories

Path Synopsis
Package osmap holds the vocabulary every OpenStack reader in Tally normalizes with: the divisors the reported quantities are converted by, the nova vm_state table, and which protocol a floating IP is an address of.
Package osmap holds the vocabulary every OpenStack reader in Tally normalizes with: the divisors the reported quantities are converted by, the nova vm_state table, and which protocol a floating IP is an address of.
Package simulator publishes one simulated month of oslo.messaging notifications.
Package simulator publishes one simulated month of oslo.messaging notifications.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL