Documentation
¶
Overview ¶
Package simulator publishes one simulated month of oslo.messaging notifications. It renders a seeded month of nova, cinder, neutron, glance, and octavia notifications onto a RabbitMQ broker, or into files when no broker is configured, and a virtual clock decides how much wall time that month costs.
The notifications are the ones a real deployment publishes, so the collector in internal/providers/openstack consumes them unmodified: the simulator sits on the producing side of the bus and nothing on the consuming side knows it is there.
This file parses the simulator's environment configuration. Configuration comes from the environment alone, prefixed TALLY_, and secrets also accept the *_FILE convention; the normative specification is roadmap/00-conventions.md section 8.
This file states what one generated month registers with the project registry of the Reporting API: a row per tenant, a row per Gardener project, and the relation that attributes a tenant's cost to the project running on it. The rows are decided from the month alone, without a network, so what a run would register is a value a test can hold against the oracle.
This file posts what RegistrationsOf decided to the project registry of the Reporting API: a row per project first, the relations between them second. The simulator is a client of that API the way the collector's sender is, so the documents on the wire are mirrored here rather than imported, and a registration carries no retry: an operator who reruns the seed, the period and the cloud registers the same rows again, and the ones an earlier run got through are found in place.
Index ¶
- Constants
- Variables
- func EnsureLocalBroker(url string) error
- func NewCloudAPI(clock *Clock, oracle Oracle) (http.Handler, error)
- func NewControlMux(clock *Clock, progress func() Progress, release func() error) *http.ServeMux
- func NewExporter(month Month, clock *Clock) http.Handler
- func Render(t Transition) ([]byte, error)
- func Replay(ctx context.Context, cfg Config, opts ReplayOptions, publisher *Publisher, ...) error
- func Run(ctx context.Context, cfg Config, opts RunOptions, publisher *Publisher, ...) error
- func TrafficOf(oracle Oracle, seed uint64, interval time.Duration) ([]Sample, []OracleTraffic, error)
- func WriteEvents(path, cloud string, schedule Schedule) (err error)
- func WriteOracle(path string, oracle Oracle) (err error)
- func WriteStream(path string, schedule Schedule) (err error)
- type Clock
- type CompareOptions
- type Config
- type Difference
- type Faults
- type GardenerProject
- type Line
- type Month
- type Oracle
- type OracleCount
- type OracleInterval
- type OracleResource
- type OracleTraffic
- type Progress
- type ProjectKey
- type ProjectRegistration
- type Publisher
- type Pusher
- type Registrar
- type RegistrationReport
- type Registrations
- type RelationRegistration
- type ReplayOptions
- type Report
- type RunOptions
- type Sample
- type SampleKind
- type Schedule
- type Tenant
- type Transition
- type UnpricedType
Constants ¶
const ( FaultPreExisting = "pre-existing" FaultMissingCreate = "missing-create" FaultDuplicates = "duplicates" FaultReordering = "reordering" FaultRefusedShapes = "refused-shapes" FaultHeldBack = "held-back" )
The six fault switches, under the names --faults takes them by. A switch changes what the bus carries and never what the simulated cloud did, so the oracle of a month states the same usage whichever switches are on.
Variables ¶
var EnvNames = []string{
envLogLevel,
envHTTPAddr,
envHTTPPort,
envAMQPURL,
envAMQPURL + fileSuffix,
envCloud,
envReportingURL,
envReportingInsecure,
envAPIToken,
envAPIToken + fileSuffix,
envGardenCloud,
envOTLPURL,
envOTLPUser,
envOTLPPassword,
envOTLPPassword + fileSuffix,
envOTLPInsecure,
envMetricsEnabled,
}
EnvNames is every variable this package reads, including the *_FILE companions of the secrets. Tests blank all of them, so a value in the developer's shell never reaches the code under test.
var FaultNames = []string{ FaultPreExisting, FaultMissingCreate, FaultDuplicates, FaultReordering, FaultRefusedShapes, FaultHeldBack, }
FaultNames holds the six switches in the order everything that lists them uses: the help of the flag, the switches a run logs, and the switches the oracle names per resource.
var ServiceExchanges = []string{
"nova", "openstack", "neutron", "glance", "octavia", "keystone", "designate", "barbican",
}
ServiceExchanges are the exchanges the simulator publishes notifications on, and the ones Connect declares. Seven are a service's own, and openstack is oslo's default, which carries cinder's notifications because cinder sets no control_exchange. The collector's default TALLY_OSC_EXCHANGES lists the first four, and a deployment lists the other four itself (docs/how-to/openstack/connect-the-collector.md, "Bind the exchanges and topics"), so a collector left at its default receives the month without its load balancers and without the keystone, designate, and barbican notifications.
Functions ¶
func EnsureLocalBroker ¶
EnsureLocalBroker refuses a broker that is not on the machine the simulator runs on.
What a run publishes is well-formed billing data. The exchanges, the declare arguments, and the shape of every notification are the ones a real deployment carries, and the wire format holds no cloud, so a collector on the far side of a production broker books a simulated month as real usage under its own TALLY_OSC_CLOUD, whatever TALLY_SIM_CLOUD said. An operator who copies a production TALLY_OSC_AMQP_URL into TALLY_SIM_AMQP_URL to try the simulator against the real broker writes a month of invented usage into the production database: ingestion deduplicates a second run into nothing, and no subcommand of tally-reporting-admin deletes what the first one wrote.
A broker on loopback cannot be that one, so it is dialled without asking. Every other broker is a decision, which is what --allow-remote-broker is: the compose stack passes it because its broker is a container beside the simulator, and an operator on a jump host has to type it.
func NewCloudAPI ¶
NewCloudAPI serves a generated month as the OpenStack API a reconciliation sync reads. The clock is the run's own, so the inventory a listing answers with is the one the simulated month has reached, and a sync started half way through a month sees half a month.
The routes are method-qualified, so a path no route claims and a method no route was registered with are both refused by the mux rather than by a handler that has to guess what was meant.
The one token this API accepts is drawn per call: two runs of the simulator never share one, and a client holding the token of a run that has ended is told so rather than served the month that came after it.
It fails on an oracle whose instances name a flavor this world does not hold. That oracle was written by another build, and a server served without the dimensions of its flavor would be observed sizeless: the sync would book a correction for a resource nobody changed.
func NewControlMux ¶
NewControlMux builds the routes the simulator serves while it publishes: /healthz for the compose health check, /clock to read and change how fast the simulated month runs, and /release to let out the notifications a run with the held-back switch keeps back.
The endpoint changes when the month goes out and nothing else. It cannot alter the month, the notifications, or where they go, which is why it carries no credential: a release publishes the very notifications the run generated, each under the timestamp it always carried. What keeps it from answering anybody is where it is bound: TALLY_SIM_HTTP_ADDR is loopback unless a deployment says otherwise, and the compose stack publishes its port on 127.0.0.1 and puts the container on the kind network, where every pod of a kind cluster on the machine reaches it. Within that reach the worst a caller does is make somebody's demo run at a different speed, or end its hold early. What a page in a browser could send there unasked is refused a step earlier: a request a page makes carries Origin however the page sent it, and one that carries it answers 403 on both routes that change something, without reaching the clock or the hold. Beyond that a release takes a JSON body or none, and a form-encoded one answers 415.
A nil release is a run that holds nothing back, and every request to /release is refused as one.
It takes no logger because the command sets the default one, and the only lines it writes are the factor change and the release.
func NewExporter ¶
NewExporter serves the inventory of one month in the Prometheus exposition format, over the clock the run is paced by. It is what GET /metrics is mounted on for as long as the run publishes.
The registry is private and holds this one collector: no Go collector and no process collector. The endpoint stands in for an exporter of the simulated cloud, and a scrape that also carried the simulator's heap and its file descriptors would state series about the process running the drill under an endpoint that is supposed to speak for a cloud.
func Render ¶
func Render(t Transition) ([]byte, error)
Render turns a transition into the message body an OpenStack service would publish for it.
The body has two layers, and the inner one travels as a string rather than as a nested object. That is what oslo does, and reproducing it is the whole point: the collector decodes the body twice, so a simulator that nested the notification would produce something only the simulator could read.
func Replay ¶
func Replay(ctx context.Context, cfg Config, opts ReplayOptions, publisher *Publisher, logger *slog.Logger) error
Replay publishes a month a run recorded, in the order the file holds it.
The publisher is dialled by the caller, as it is for Run, and for the same reason. A replay has no file mode: the month already exists as a file, so a nil publisher leaves it nothing to do.
It serves the control endpoint the way a run does, and no fake OpenStack API: the recorded file holds the notifications of a month, not the oracle a listing is answered out of.
A cancelled context is a clean stop and a failed publish is an error, the way Run treats them: the recorded message ids are the ones the run published, so a repeated replay is deduplicated by ingestion.
func Run ¶
func Run(ctx context.Context, cfg Config, opts RunOptions, publisher *Publisher, logger *slog.Logger) error
Run generates one month of notifications and publishes it, writes it to files, or both.
The publisher is dialled by the caller rather than here. That is what lets a caller have the service exchanges declared before a collector starts: the collector creates none of them, and one that requires every exchange (TALLY_OSC_REQUIRE_EXCHANGES=true) reconnects until somebody has declared them, so started first it spends the run reconnecting. A nil publisher is file mode, where the month is written out and nothing reaches a bus.
With RegisterProjects the month's tenants, its Gardener projects and the relations between them are registered with the Reporting API first, before anything is written or published.
The files are complete before the first notification is published, so a run that is interrupted halfway still leaves a whole month on disk to replay. There are three of them, and a fourth, held-back.jsonl, when the held-back switch keeps part of the month off the bus.
While the month goes out, the run serves it as a fake OpenStack API as well, on the control endpoint's listener and for exactly as long, the hold included. A reconciliation sync reads the cloud the notifications describe while they are published. File mode serves neither of the two: nothing is in flight there.
The second face of the month goes out beside the notifications: the traffic counters and the inventory gauges are pushed to the OTLP endpoint the configuration names, on the clock the notifications are paced by, and the inventory is served on GET /metrics of the same listener. A run without an OTLP URL pushes nothing, and file mode pushes nothing either, but both write the traffic rows into the oracle. A push that failed is reported once the month has gone out whole, because the month is what the operator asked for.
A cancelled context is a clean stop: what went out stays out, and Run returns nil, so SIGINT and SIGTERM leave exit status 0 the way they do for the collector. A failed publish is an error instead, and what an operator does about it is rerun the same seed, period and cloud: that renders the same message ids, and ingestion deduplicates whatever was already delivered.
func TrafficOf ¶
func TrafficOf(oracle Oracle, seed uint64, interval time.Duration) ([]Sample, []OracleTraffic, error)
TrafficOf places the network traffic of a month: the samples every instance is pushed under, and the rows the oracle records them by.
The samples come back ordered by their instant and then by name and resource id, so a pusher walks them with one cursor instead of indexing them. Their value is the running sum of the steps before them, which is what a cumulative counter reports: the first sample of every series is 0, and the sample at one instant states the bytes accrued before it.
The rows come back ordered by resource id and then by From, because the oracle's instances are sorted by their id and the intervals of one instance by their start. A row states the exact sum of the steps placed inside its interval, so the increment of an instance's last step lands in its last row and in no sample, the way Ceilometer's last poll precedes a delete.
The one refusal is an instance whose workload is none of the three: that is an oracle another build wrote, and a level this one would have to guess at. A byte count nobody stated is worse than no month at all.
func WriteEvents ¶
WriteEvents writes the events the collector is expected to record for the schedule to path, one per line, creating the file or truncating what is already there. The cloud is the one the events are booked to.
It states what a run expects of the collector. notifications.jsonl says what was published, this file says what ingestion has to hold once the month is consumed, and a drill compares the two without mapping the notifications a second time by hand.
A transition the schedule marks billable and the mapping claims nothing for fails the write. The workload decides billable from its own catalog and the collector decides it from its mapping table, so a type added to one and not the other drifts the two apart. Failing here is what catches that drift outside the test suite, on whatever month an operator generates, instead of writing a file that quietly expects fewer events than the month holds.
func WriteOracle ¶
WriteOracle writes the oracle to path as one indented JSON document, creating the file or truncating what is already there. It is the third file a run in file mode leaves beside notifications.jsonl and events.jsonl: notifications.jsonl says what was published, events.jsonl what the collector has to record, and this file what the month was meant to mean.
The oracle of one seed, period and cloud is byte-identical across runs. Its resources and its counts are sorted before they are stated, so what a map was walked in never reaches the file, and two runs of one triple produce two files a diff reports nothing about.
func WriteStream ¶
WriteStream writes the schedule to path as one notification per line, creating the file or truncating what is already there.
It is what lets a generated month outlive the process that generated it: the file holds every notification with the exchange and the routing key it was meant for, so replay puts the same month on a broker again without the generator, and without the seed that produced it.
Types ¶
type Clock ¶
type Clock struct {
// contains filtered or unexported fields
}
Clock maps wall time to the virtual time of the simulated month, so that a month of notifications is emitted in the minutes an operator is willing to wait for it. Everything the simulator paces reads its time here rather than from time.Now, which is what lets the factor be raised or lowered while a run is in flight.
The factor is a float64 because it scales durations and never touches a usage quantity or a price. The forbidigo rules in .golangci.yml that keep floats off the money paths are about those quantities, not about how fast the simulation runs.
func NewClock ¶
NewClock starts a clock at start, running at factor virtual seconds per wall second. now is the wall clock it reads: production passes time.Now, and a test passes a function it advances by hand so that a month-long factor is asserted without waiting for real time.
func (*Clock) SetFactor ¶
SetFactor changes how fast virtual time runs from this instant on. The clock rebases on the virtual now it has reached under the old factor, so a change never moves the simulated month itself: only what comes after it runs at a different speed. Closing changed wakes every SleepUntil, because the wall deadline each of them computed no longer answers to the new factor.
A negative factor is refused rather than clamped: it would run the simulated month backwards, and every consumer of the notifications reads their timestamps as moving forward. NaN is refused beside it, because it compares false against every bound, including this one: it would reach the arithmetic below and turn every virtual instant the clock reports into a duration nothing can wait for.
func (*Clock) SleepUntil ¶
SleepUntil waits until the virtual clock reaches virtual, and reports ctx's error when the context ends first. At factor 0 it returns at once: virtual time does not advance then, so waiting for it would never end, and the run goes as fast as the broker takes the notifications.
The wait is recomputed on every wake because the factor decides what the remaining virtual distance costs in wall time. A change mid-wait invalidates the deadline, which is why the timer and the changed channel are selected on together. A distance that costs more wall time than maxWait is covered in several waits for the same reason.
type CompareOptions ¶
type CompareOptions struct {
// Oracle is the path of the oracle.json a run wrote.
Oracle string
// Export is the directory tally-engine export --format csv --out wrote.
Export string
// Pricing is the path of the pricing model file the run rated with. The
// model is read from the file rather than from the export, because a CSV
// export carries no model: it says which quantity was billed and at which
// amount, and not which dimensions the resource types were held against.
Pricing string
}
CompareOptions names the three files one comparison reads.
func (CompareOptions) Validate ¶
func (o CompareOptions) Validate() error
Validate reports the first member a comparison cannot run without. The three are checked in the order a caller passes them, so an invocation missing all of them is told about its first flag rather than its last.
type Config ¶
type Config struct {
// LogLevel is the slog threshold, one of DEBUG, INFO, WARN, or ERROR. Both
// subcommands read it.
LogLevel string `env:"TALLY_LOG_LEVEL" envDefault:"INFO"`
// HTTPAddr is the address the control endpoint binds. It defaults to loopback
// because the endpoint carries no credential and PUT /clock changes the pace
// of a run: a simulator on a host beside a control plane would otherwise
// answer everybody on the management network. A deployment that means to
// publish the port sets 0.0.0.0, which is what the compose stack does.
HTTPAddr string `env:"TALLY_SIM_HTTP_ADDR" envDefault:"127.0.0.1"`
// HTTPPort is the port the control endpoint listens on. It is served only
// while the simulator publishes, because there is nothing to control once the
// month is over.
HTTPPort int `env:"TALLY_SIM_HTTP_PORT" envDefault:"8080"`
// AMQPURL is the broker the notifications are published to. It carries the
// broker password, so it supports the *_FILE convention. Empty puts run in
// file mode, where the month is written out instead of published.
AMQPURL string `env:"TALLY_SIM_AMQP_URL"`
// Cloud is read by run alone: it is the salt of every generated identifier
// and the cloud of events.jsonl. It has no default because a guessed cloud
// silently books usage to the wrong one.
Cloud string `env:"TALLY_SIM_CLOUD"`
// ReportingURL is the Reporting API the projects are registered with. run
// reads it when --register-projects is on and ignores it otherwise. It must
// be an absolute https URL, because the api token travels on it;
// ReportingInsecure is what allows a plaintext one.
ReportingURL string `env:"TALLY_SIM_REPORTING_URL"`
// ReportingInsecure allows an http Reporting API. It exists for a simulator
// and an API on the same machine, and for development; anywhere else it puts
// an api token of role admin on the wire in cleartext.
ReportingInsecure bool `env:"TALLY_SIM_REPORTING_INSECURE" envDefault:"false"`
// APIToken is an api token of role admin, which POST /api/v1/projects and
// POST /api/v1/projects/{id}/relations demand. It supports the *_FILE
// convention.
APIToken string `env:"TALLY_SIM_API_TOKEN"`
// GardenCloud is the cloud the two Gardener projects are registered under.
// It has no default, for the reason Cloud has none.
GardenCloud string `env:"TALLY_SIM_GARDEN_CLOUD"`
// OTLPURL is the OTLP/HTTP endpoint the traffic and inventory series of a run
// are pushed to. Empty is a run without a push. It has to be absolute and
// carry a host, and to be https unless OTLPInsecure allows a plaintext one,
// because the Basic password travels on it. run reads it alone.
OTLPURL string `env:"TALLY_SIM_OTLP_URL"`
// OTLPUser is the Basic user of the push. The endpoint in front of the
// collector takes Basic auth, so a URL without a user reaches nothing.
OTLPUser string `env:"TALLY_SIM_OTLP_USER"`
// OTLPPassword is the Basic password of the push. It supports the *_FILE
// convention.
OTLPPassword string `env:"TALLY_SIM_OTLP_PASSWORD"`
// OTLPInsecure allows an http endpoint. It exists for a simulator and a
// collector on the same machine, and for development; anywhere else it puts
// the Basic password on the wire in cleartext.
OTLPInsecure bool `env:"TALLY_SIM_OTLP_INSECURE" envDefault:"false"`
// MetricsEnabled serves the inventory on GET /metrics of the control listener
// while a run publishes, where it stands in for the OpenStack database
// exporter a deployment scrapes. False registers no route, and the fake
// OpenStack API then answers that path with its own 404. A Config that never
// went through Load carries false, which is why a test that needs the
// endpoint sets it. The variable has no SIM infix because roadmap section 8
// lists it among the common variables every service reads.
MetricsEnabled bool `env:"TALLY_METRICS_ENABLED" envDefault:"true"`
}
Config is the simulator's resolved configuration. Every field is final by the time Load returns: a file-backed broker URL holds the file's content, and the log level is one this package accepts.
func Load ¶
Load reads the environment, resolves the file-backed secrets, and checks the log level. It does not check whether the required values are present: which ones are required depends on the subcommand and on its switches, which is what ValidateRun, ValidateReplay, ValidateRegistration, and ValidateMetrics decide.
func (Config) ControlAddr ¶
ControlAddr is the address the control endpoint listens on. An empty HTTPAddr, which is a Config that never went through Load, binds loopback rather than every interface: the endpoint carries no credential, so the wider bind is a choice a deployment makes and not one a zero value falls into.
func (Config) SlogLevel ¶
SlogLevel is the slog level that LogLevel names. A Config that never went through Load, and so carries an unchecked level, logs at info.
func (Config) ValidateMetrics ¶
ValidateMetrics is the gate of the push side of the metrics, checked before a run dials the broker. An empty TALLY_SIM_OTLP_URL is a run without a push and passes: the month is generated and published, and the samples it places stay in the process.
A URL that is set is one a whole month is posted to, so what it needs is asked for now rather than at the first flush, an hour into a paced run.
func (Config) ValidateRegistration ¶
ValidateRegistration is the gate of run --register-projects. It asks for the Reporting API, an api token for it, and the cloud the Gardener projects are registered under, none of which a run without the switch reads.
The two clouds have to differ because a cloud is one installation of one platform: a Gardener project registered under the tenants' cloud would key a row of the OpenStack installation, and the relation would then point the project at itself.
func (Config) ValidateReplay ¶
ValidateReplay is the gate of the subcommand that publishes an already generated month. Replaying carries the cloud in the recorded notifications, so the broker is all it needs.
func (Config) ValidateRun ¶
ValidateRun is the gate of the subcommand that generates the month. It asks for the cloud alone: the broker is optional because run --out writes the month to files and never connects to one.
type Difference ¶
type Difference struct {
ResourceType string
ResourceID string
Detail string
More int
// Faults holds the fault switches that touched the oracle's resource. A
// difference beside its switches is one the drill's write-up explains in a
// line, and one without them is a finding about the engine. A resource the
// export books that the oracle does not hold carries none.
Faults []string
}
Difference is one resource the export and the oracle disagree about: the first difference found on it, in the order the intervals run in, and how many further ones it carries. Only the first is spelled out, because a resource billed under the wrong project differs in every interval it has, and a report that printed all of them would bury the resource beside it.
type Faults ¶
type Faults struct {
PreExisting, MissingCreate, Duplicates, Reordering, RefusedShapes, HeldBack bool
}
Faults is the set of switches one run turns on, one field per name in FaultNames. The zero value is every switch off, which is the run nobody passed --faults to.
func ParseFaults ¶
ParseFaults reads the switch names --faults was given. An empty list is every switch off, and a name given twice is the run a name given once is.
pre-existing and missing-create are refused together. Both of them pick the instances they work on from the same set, so a run with both on would have the two switches compete for the same instances.
type GardenerProject ¶
type GardenerProject struct {
// Name is alpha or beta, the external id of the gardener row.
Name string
// TenantID is the ID of the Tenant the shoots run on.
TenantID string
// Shoots holds the shoot names, in the order newGardenerProjects lists them.
Shoots []string
}
GardenerProject is one Gardener project and the tenant its shoots run on.
type Line ¶
type Line struct {
// Exchange is the exchange the notification belongs on, one of nova,
// openstack, neutron, glance, octavia, keystone, designate, and barbican.
// Cinder publishes on openstack.
Exchange string `json:"exchange"`
// RoutingKey is the topic the notification was published under.
RoutingKey string `json:"routing_key"`
// Body is the oslo envelope as Render produced it, kept as raw JSON so a
// replay republishes the very bytes the run generated rather than a
// re-encoding of them.
Body json.RawMessage `json:"body"`
}
Line is one line of notifications.jsonl: the message body a service put on the bus, together with the addressing it was published under. The addressing travels with the body because the body does not carry it: an exchange is what decides which queue a notification reaches, and a replay that guessed it would publish a month no collector receives.
func ReadStream ¶
ReadStream reads the notifications of a stream file back, in the order they stand in the file, which is the order they were published in.
Every body goes through the collector's own envelope parser here rather than at publication time. A file that holds something other than notifications, or a notification without a usable timestamp, is refused before the first message reaches the broker, instead of halfway through a replay that has already published part of a month.
The routing key of a line is read from the line and not assumed. A month recorded against one deployment may have been published under a topic this build no longer defaults to, and a replay has to reach the queue the recording did.
The exchange of a line is held to ServiceExchanges, which are the ones Connect declares. A publish on any other one either fails on a broker that does not carry it, after the lines ahead of it are out, or is confirmed and dropped on a broker that carries it with no queue bound. A month recorded while cinder's notifications went to an exchange named cinder is such a file.
type Month ¶
type Month struct {
// Schedule is what the simulated cloud did: every transition once, sorted by
// its instant. It is what the oracle and WriteEvents are read from.
Schedule Schedule
// Stream is the notifications in publication order, which is what
// notifications.jsonl holds and what the bus carries. The switches of
// faults.go move, repeat, and add to it. With every switch off it is the
// schedule.
Stream Schedule
// Held holds the transitions the held-back switch keeps off the bus until a
// run releases them, in instant order. It is empty unless that switch is on.
Held Schedule
// Tenants is every keystone project of the month, the classic tenants first
// in index order, then the CI tenant, then the Gardener tenants in project
// order. It is built from the world alone: nothing on the bus carries a
// tenant's name, and no stream is consulted, so a month with the registry
// switch off renders exactly the files it rendered before.
Tenants []Tenant
// GardenerProjects is the two Gardener projects and the tenant each runs
// on, in the order newGardenerProjects lists them.
GardenerProjects []GardenerProject
Oracle Oracle
}
Month is one generated month: the transitions the simulated cloud publishes and the oracle of what they were meant to mean.
func GenerateMonth ¶
GenerateMonth builds one month of the simulated cloud's lifecycle transitions, sorted by their instant, together with the oracle of what the month contained.
The month runs on two generators. The shape generator draws everything that decides what happens and when, and it is seeded by the seed alone: the same seed describes the same cloud whichever period or deployment it is run for. The identifier generator draws every id, and it is seeded by the seed together with the cloud and the billing month, so the same seed on another cloud or in another month publishes fresh resources and fresh message ids while the same triple republishes the very notifications ingestion has already deduplicated.
Splitting them is what makes both properties hold at once, and the rule that keeps them apart runs in both directions: an identifier comes from the identifier stream and never from the shape stream, and what happens, when, and therefore how many identifiers a month draws comes from the shape stream and the period's calendar and never from the value of an identifier. The same seed, period, and cloud draw the same identifiers in the same order, and another cloud or another month renames everything while the shape stays the shape stream's.
A third generator names the resources of the noise catalogue (noise.go). It is salted the way the identifier generator is, and it is a stream of its own so that the identifier generator draws the billable month and nothing else, down to the message ids: a noise transition takes its own from the third stream, so a catalogue that grows by one transition renumbers nothing the collector bills.
The fault switches (faults.go) draw from streams of their own, one per switch and seeded by the seed together with the switch's name. None of them draws from the shape stream, so a month with every switch off consumes the three streams above exactly as it would without the switches, and missing-create draws from the pre-existing stream, which is what makes the two pick the same instances for one seed.
type Oracle ¶
type Oracle struct {
Format int `json:"format"`
Cloud string `json:"cloud"`
Seed uint64 `json:"seed"`
PeriodFrom time.Time `json:"period_from"`
PeriodTo time.Time `json:"period_to"`
Resources []OracleResource `json:"resources"`
Counts []OracleCount `json:"counts"`
// Faults holds the fault switches the month ran with, in FaultNames order.
// A month nobody passed --faults to states an empty list.
Faults []string `json:"faults"`
// Traffic holds one row per instance and per interval of that instance,
// ordered by resource id and then by From. Run fills it from the metric
// samples once the month is generated, so buildOracle states an empty list.
//
// The rows lie outside OracleInterval on purpose: foldFacts merges two
// adjacent facts that repeat the state, project and size, and a traffic
// figure inside the interval would make two mergeable intervals compare
// unequal and split that fold.
Traffic []OracleTraffic `json:"traffic"`
}
Oracle is the generator's statement of what a month contained: for every billable resource the intervals of constant state, size and project it intended, clipped to the month, and the count of events it expects the collector to record per project and Tally event type. The size of a load balancer is the listeners and pools the cloud holds, which no notification carries and only a reconciliation sync books.
func ReadOracle ¶
ReadOracle reads back the oracle a run wrote, for a comparison that runs without the generator that produced the month.
The document is decoded under Decoder.UseNumber, so every number of a size comes back as the json.Number it was written from. A quantity that took the detour through a float64 would no longer carry the digits the engine read from the very same notification.
An unknown member fails the read rather than being passed over. An oracle written by another build may state something this one does not know about, and an oracle read in part would report the engine for whatever the part it dropped covered. A member the document lacks is refused for the same reason and needs a check of its own: JSON leaves an absent member at its zero value, so a trimmed document decodes without complaint and states an empty month instead of the one it was written for.
type OracleCount ¶
type OracleCount struct {
ProjectID string `json:"project_id"`
EventType string `json:"event_type"`
Count int `json:"count"`
}
OracleCount is how many events of one Tally event type the month expects a project to have recorded.
type OracleInterval ¶
type OracleInterval struct {
From time.Time `json:"from"`
To time.Time `json:"to"`
State string `json:"state"`
ProjectID string `json:"project_id"`
Size map[string]any `json:"size"`
}
OracleInterval is a half-open span [From, To) over which a resource's state, size and project did not change. Both ends lie inside the month.
type OracleResource ¶
type OracleResource struct {
ResourceType string `json:"resource_type"`
ResourceID string `json:"resource_id"`
Workload string `json:"workload"`
Intervals []OracleInterval `json:"intervals"`
// Faults holds the fault switches that touched this resource, in FaultNames
// order. A resource none of them touched states an empty list.
Faults []string `json:"faults"`
}
OracleResource is one billable resource of the month and the intervals it was meant to be billed over, ordered by their start.
type OracleTraffic ¶
type OracleTraffic struct {
ResourceID string `json:"resource_id"`
From time.Time `json:"from"`
To time.Time `json:"to"`
EgressBytes int64 `json:"egress_bytes"`
IngressBytes int64 `json:"ingress_bytes"`
}
OracleTraffic is the network traffic one instance was given over one of its intervals: the exact sum of the grid steps the generator placed inside it.
type Progress ¶
type Progress struct {
// From and To bound the simulated month, the period the virtual clock runs
// through.
From time.Time
To time.Time
// Published is how many notifications the run has published so far, of Total
// in the whole month.
Published int
Total int
// Held is how many notifications the held-back switch still keeps off the
// bus. It is 0 for a run without the switch, and 0 once a release let them
// out.
Held int
// Holding is whether the run waits for a release: true from the moment the
// last regular notification is on the bus until a release lets the held share
// out. It is the one signal a caller may poll before releasing, because it is
// the very state a release acts on, while the counts reach their hold values
// a moment before the run enters the hold.
Holding bool
}
Progress is how far a run has got: the simulated period it publishes and the notifications it has already put on the bus. The run hands it to the control endpoint through a callback rather than a copy, so a caller reading /clock sees the counts of that moment instead of the counts the endpoint was built with.
type ProjectKey ¶
type ProjectKey struct{ Cloud, ExternalID string }
ProjectKey is the pair the registry keys a project by.
type ProjectRegistration ¶
type ProjectRegistration struct {
Platform string
Key ProjectKey
Name string
Metadata map[string]any
}
ProjectRegistration is one projects row to register.
type Publisher ¶
type Publisher struct {
// contains filtered or unexported fields
}
Publisher is the simulator's side of the bus: one connection and one channel in confirm mode, so every published notification is waited on until the broker has taken responsibility for it. A run that reported a message published while the broker dropped it would leave the collector short of events with nothing to point at.
func Connect ¶
Connect dials the broker, opens the confirming channel, and declares the service exchanges.
The declares are what give the collector something to bind: it creates no exchange and probes the ones it lists (probeExchanges in internal/providers/openstack/osloamqp.go), binding those that exist. On a fresh broker, where no OpenStack service has ever published, a collector running with TALLY_OSC_REQUIRE_EXCHANGES=true reconnects until somebody declares them, and one without it skips them and binds each once it appears.
The arguments are the ones the services use, and the ones declareExchanges in internal/providers/openstack/amqp_integration_test.go uses: a durable topic exchange that is neither auto-deleted nor internal. A declare that differed in any of them would be refused by a broker that already carries the service's exchange.
func (*Publisher) AwaitConsumer ¶
AwaitConsumer waits until queue has at least one consumer, or until timeout has passed. A timeout of zero skips the wait.
The wait is what keeps a run from publishing into nothing: a topic exchange drops every message no queue is bound to, and on a fresh broker the collector's queue does not exist until the collector has connected once and bound it. The consumer count rather than the queue's existence is the signal, because the collector declares, binds, and only then registers its consumer.
A queue that has a consumer is a queue bound to every exchange only for a collector running with TALLY_OSC_REQUIRE_EXCHANGES=true, or one whose exchanges all existed when it connected. A collector without the switch that connected before the exchanges were declared consumes from a queue it has not bound to them yet.
func (*Publisher) Publish ¶
Publish puts one notification on an exchange and returns once the broker has confirmed it. The confirm is waited on per message rather than in batches, which is what lets the run report the count it published and stop at the first message the broker refused.
The message is persistent because the collector's queue is durable: a backlog that piled up there while the collector was stopped is meant to survive a broker restart, and a transient message would be gone with it.
type Pusher ¶
type Pusher struct {
// contains filtered or unexported fields
}
Pusher posts samples to an OTLP/HTTP metrics endpoint under a basic credential.
It never formats the password or the Authorization header into an error or a log line. What a failure states is the url, the status, and the beginning of the body the endpoint refused with, which is what an operator needs and where the credential never is. The url an error carries is the redacted one: a credential a caller put into the URL itself is a password as much as the configured one is.
func NewPusher ¶
NewPusher returns the pusher that posts to url under the credential user and password. Every point it sends carries platform and cloud on top of the labels of its sample: a push has no scrape job to take those two from, and the dashboards filter on both.
A nil httpClient selects the package default, which is the one internal/engine/counters builds for its queries. The retrying transport sits below http.Client.Do, so a push makes one call and sees one result however many attempts it took. The default policy retries connection errors, a 429, which is what the Gateway's BackendTrafficPolicy answers a run that pushes faster than the rate it allows, and a 5xx except 501. A 4xx such as the 401 of a wrong password is returned at once. Tests pass their own client.
func (*Pusher) Push ¶
Push posts the samples in chunks of at most maxDataPoints points and stops at the first chunk that fails, because a batch the endpoint refused is one the ones behind it will be refused with too. An empty batch is no request at all.
A cancelled context comes back as the context's own error, so a caller's errors.Is(err, context.Canceled) holds and the publishing loop reads it as the clean stop a signal asked for rather than as a failed push.
type Registrar ¶
type Registrar struct {
// contains filtered or unexported fields
}
Registrar registers the projects and relations of a month with one Reporting API.
func NewRegistrar ¶
NewRegistrar builds the client that registers with the Reporting API at reportingURL, holding token as its credential. One trailing slash is trimmed off the URL, so a base URL written either way builds the same routes.
logger may be nil, which logs through the default logger.
func (*Registrar) Register ¶
func (r *Registrar) Register(ctx context.Context, regs Registrations) (RegistrationReport, error)
Register posts every project of regs and then every relation between them, and reports how many rows it created and how many the registry already held.
The order is what makes it work twice. A relation needs both of its ends registered, so the rows go first, and a key the registry already holds is answered 409 rather than replaced: that row is looked up by its key and its id is what the relations point at. A relation that is already active is answered 409 too and left as it stands. Rerunning the same seed, period and cloud therefore ends in the registry the first run meant to leave behind, which is what an operator does after a run that failed halfway.
A rerun under another seed, period or cloud is not that: its tenants are new rows, while the Gardener rows are the ones the earlier run registered, so the relations of the two runs would attribute two months to one statement. Such a registration is refused at the relation, naming the route that ends the earlier one and the instant it has to end at, or, when that relation begins no earlier than this month and can therefore not be ended before it, the garden cloud of its own this month needs instead.
The first answer no path can use ends the registration, and the report then counts what got through before it. A set naming a relation whose ends it does not register is refused before the first request, because the rows of a set like that are rows without the relation that gives them their meaning.
type RegistrationReport ¶
type RegistrationReport struct {
ProjectsCreated, ProjectsExisting, RelationsCreated, RelationsExisting int
}
RegistrationReport counts what one Register call found and did.
type Registrations ¶
type Registrations struct {
Projects []ProjectRegistration
Relations []RelationRegistration
}
Registrations is what one month registers: the rows first, the relations between them second.
func RegistrationsOf ¶
func RegistrationsOf(month Month, gardenCloud string) (Registrations, error)
RegistrationsOf returns the rows and relations the month registers: one openstack row per tenant under the cloud the month was rendered for, one gardener row per Gardener project under gardenCloud, and one infrastructure_tenant relation per Gardener project, pointing at the tenant its shoots run on and valid from the start of the month.
The two clouds have to differ because a cloud is one installation of one platform. A Gardener project registered under the tenants' cloud would key a row of the OpenStack installation, and the relation would then point the project at itself.
type RelationRegistration ¶
type RelationRegistration struct {
Source, Target ProjectKey
RelationType string
Metadata map[string]any
ValidFrom time.Time
}
RelationRegistration is one relation to create between two registered rows.
type ReplayOptions ¶
type ReplayOptions struct {
// In is the notifications.jsonl a run wrote.
In string
// Factor is how many virtual seconds pass per wall second. Zero is
// unbounded.
Factor float64
// WaitForCollector is how long to wait for a consumer on the collector's
// queue before the first notification goes out. Zero disables the wait.
WaitForCollector time.Duration
}
ReplayOptions is what a replay needs: the file to publish and the pacing to publish it under. The month itself comes from the file, so there is no seed and no period here.
func (ReplayOptions) Validate ¶
func (o ReplayOptions) Validate() error
Validate checks the pacing options, the way RunOptions.Validate does. The file is not opened here: reading it is the replay's own first step, and a check that opened it would either read the month twice or leave a handle behind.
type Report ¶
type Report struct {
// Compared counts every resource examined: the oracle's resources of a
// priced type whose size a notification states, and the resources the
// export books that the oracle does not hold.
Compared int
// Unpriced counts the oracle's resources per type the model does not price,
// sorted by type.
Unpriced []UnpricedType
// Reconciled counts the oracle's load balancers when the model prices
// them. They are counted rather than compared: the oracle states the size
// the cloud holds, the engine bills what the create and the syncs booked,
// and the two part ways at instants a drill does not control.
Reconciled int
// Skipped counts the rated records of another cloud or another platform,
// which an export of a deployment that bills more than the simulated cloud
// carries beside the ones this oracle is about.
Skipped int
// Differences holds one entry per differing resource, sorted by resource
// type and then by id.
Differences []Difference
// PricingVersion is the version of the model the comparison read, so that a
// report says which prices the unpriced types were unpriced by.
PricingVersion string
// Faults holds the fault switches the month ran with, as its oracle states
// them.
Faults []string
}
Report is what one comparison found.
func Compare ¶
func Compare(opts CompareOptions) (Report, error)
Compare holds the CSV export in a directory against the oracle of the month it was rated from, priced by the model the run rated with.
func (Report) Lines ¶
Lines renders the report as the lines a command prints: the differences first, then the types nothing was compared for, then the records of other clouds, then the switches the month ran with, and last the verdict. Both slices are printed in the order the comparison sorted them into.
A difference on a resource a switch touched names that switch. It stays a difference and counts as one: whether it is the one the switch was turned on for is what the drill's write-up decides.
type RunOptions ¶
type RunOptions struct {
// Period is the billing month to generate, as YYYY-MM.
Period string
// Seed decides the shape of the simulated cloud.
Seed uint64
// Factor is how many virtual seconds pass per wall second. Zero is
// unbounded: the month goes out as fast as the broker takes it.
Factor float64
// Out is the directory notifications.jsonl, events.jsonl and oracle.json are
// written to, and held-back.jsonl beside them when a switch holds something
// back. Empty writes nothing.
Out string
// WaitForCollector is how long to wait for a consumer on the collector's
// queue before the first notification goes out. Zero disables the wait.
WaitForCollector time.Duration
// Faults are the switch names of --faults, which ParseFaults reads. Empty is
// every switch off.
Faults []string
// RegisterProjects registers the month's tenants, its Gardener projects, and
// the infrastructure_tenant relation between each project and its tenant
// with the Reporting API before the first file is written or the first
// notification published. What it needs is the environment, which Config
// checks through ValidateRegistration.
RegisterProjects bool
// MetricsInterval is the grid the traffic counters and the inventory are
// sampled on, counted from the period's start. It is a whole number of
// seconds, at least 30s and at most 24h; the flag defaults to Ceilometer's
// polling interval of 300s.
MetricsInterval time.Duration
}
RunOptions is what a run needs beyond its environment: which month to generate from which seed, how fast to publish it, and where it goes. The flags of the run subcommand map onto it one to one, which is why the errors quote flag names rather than field names.
func (RunOptions) Validate ¶
func (o RunOptions) Validate() (from, to time.Time, err error)
Validate checks the options and returns the month Period names.
It exists so the command refuses a mistyped flag before it dials the broker, which is where a run would otherwise notice. Run calls it again itself, because an exported function cannot rely on its caller having done so.
type Sample ¶
type Sample struct {
Name string
Labels map[string]string
Value int64
At time.Time
Kind SampleKind
}
Sample is one point of one series: the metric name, the labels beyond the ones a scrape job or the pusher adds, the value, and the virtual instant it belongs to. Kind is KindGauge or KindCounter, which decides whether the push encodes it as an OTLP gauge or as a cumulative monotonic sum.
The samples of one series over one interval share one Labels map, and nothing mutates it. A caller that has to carry a label of its own builds a map of its own.
func InventoryAt ¶
InventoryAt is the world of the month as it stands at the virtual instant at: one gauge per live resource of the six families the oracle holds, the routers the schedule left standing, the limits of every project, and the counts the cloud reports about itself.
An instant past the end of the month is answered at that end, the way the fake API answers a listing from a clock that has run past it: every interval of the oracle ends inside the month, and a later instant would report a cloud that lost everything at once.
A size member an interval does not carry reads as zero, which is what sizeDecimal answers with, so one malformed interval costs its own series a value instead of failing a whole scrape. The status of a server is the word nova reports rather than the one the collector books it under.
The samples carry neither a platform nor a cloud label. A real exporter carries neither: both come from the scrape job's static labels, and an endpoint that stated them as well would push the job's own under exported_cloud. A series the table gives no labels of its own carries no map.
type SampleKind ¶
type SampleKind int
SampleKind tells a gauge from a counter.
const ( // KindGauge is a value read at an instant, encoded as an OTLP gauge. KindGauge SampleKind = iota // KindCounter is a cumulative monotonic sum, encoded as an OTLP sum. KindCounter )
type Schedule ¶
type Schedule []Transition
Schedule is one month of transitions in publication order.
func (Schedule) Billable ¶
func (s Schedule) Billable() []Transition
Billable returns the transitions the collector records an event for. It is what a run counts its expected events by: the rest reaches the collector and is skipped there, so counting the whole schedule would overstate the month.
type Tenant ¶
type Tenant struct {
// ID is the keystone project id, the external id of the registry row.
ID string
// Name is what the registry row is called; nothing on the bus carries it.
Name string
// Workload is workloadClassic, workloadGardener, or workloadCI.
Workload string
}
Tenant is one keystone project of the month as the registry keys it.
type Transition ¶
type Transition struct {
// At is the virtual instant the notification is published at. It lies
// inside the generated month.
At time.Time
// EventType is the type the emitting service would use, such as
// "compute.instance.create.end".
EventType string
// Exchange is the notification exchange the type belongs on. A month uses
// eight: nova, openstack, neutron, glance, octavia, keystone, designate, and
// barbican. The second is oslo's default, which cinder publishes on because
// it sets no control_exchange of its own. The collector's default binds
// nova, neutron, openstack, and glance, and a deployment lists the other
// four itself.
Exchange string
// Billable reports whether the collector's mapping records an event for this
// notification. It is false on the image.create that precedes an upload,
// which carries no size yet and is skipped on purpose, and on every
// transition of the noise catalogue (noise.go).
Billable bool
// Workload names which of the three workloads emitted the transition, one of
// the workload constants. Nothing on the wire carries it: it is what a test
// and the later reconciliation oracle tell the workloads of one month apart
// by, since the tenants of a month are otherwise ids alone.
Workload string
// MessageID is the oslo message id, which the Reporting API deduplicates on.
// It is drawn after the month is sorted, so the ids run in the order the
// notifications are published.
MessageID string
// PublisherID names the service instance that emitted the notification.
PublisherID string
// ProjectID is the project the request ran in. It becomes both
// _context_project_id and _context_tenant_id, since services differ in which
// of the two they set and the collector reads either.
ProjectID string
// UserID is the user the request ran as, which becomes _context_user_id.
UserID string
// ResourceID is the resource the transition is about. The rendered payload
// carries it under whatever member the emitting service names it in, so this
// field is what a test holds the mapping's result against.
ResourceID string
// Payload is the service's description of the resource, ready to be
// marshalled as the notification's payload.
Payload map[string]any
// contains filtered or unexported fields
}
Transition is one lifecycle step of the simulated cloud: what happened, when, and everything the notification about it needs. It is the simulator's own form, one level above the wire, so that a month can be sorted, counted, and asserted on before any of it is serialized.
type UnpricedType ¶
UnpricedType is a resource type the pricing model does not price and the number of resources of it the oracle states. They are counted rather than compared: the rating pass skips such a resource, so the export holds no record of it, and a comparison that reported each one missing would report the engine for doing what the model asks.