frame2

package
v0.2.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 23, 2025 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

TODO - frame2.Logf (f,v) - frame2.LogVerbosef (f,v) - frame2.LogLevelf (frame2.Level, f, v) - Shared namespaces - Redo Retry testing, other meta_test with Runner? - Example of generating Runner table from simpler table - Example using test-specific Runner (discouraged?) - Example using inter-test communication via references - Auto Doc() or require Doc - Retry report (logged): len, %, Batalha naval. ▂▃▄...█ if more than 40. Else, individual: ▌▐ █. - Implement all substep/substeps, validator/validators, etc options

Index

Constants

View Source
const (

	// Define the upgrade strategy used by the Upgrade disruptor (possibly
	// other points as well?)
	//
	// Values:
	//
	// CREATION (default): order of skupper init
	// PUB_FIRST
	// PRV_FIRST
	// PUB_ONLY
	// PRV_ONLY
	// LEAVES_FIRST
	// LEAVES_ONLY
	// CORE_FIRST
	// CORE_ONLY
	// EDGES_FIRST
	// EDGES_ONLY
	// INTERIOR_FIRST
	// INTERIOR_ONLY
	//
	// Currently, only CREATION and CREATION:INVERSE are implemented
	//
	// For any of the options, if the value ends with :INVERSE, the order
	// is inverted.  For example, ":INVERSE" or "CREATION:INVERSE" will
	// upgrade the lastly installed skupper site first; the first last.
	//
	// Valid values are of the string type TestUpgradeStrategy
	ENV_UPGRADE_STRATEGY = "SKUPPER_TEST_UPGRADE_STRATEGY"

	// A path to the Skupper binary to be used (the actual file, not just its parent directory)
	ENV_OLD_BIN = "SKUPPER_TEST_OLD_BIN"

	// The version that ENV_OLD_BIN refers to, such as 1.2 or 1.4.0-rc3
	ENV_OLD_VERSION = "SKUPPER_TEST_OLD_VERSION"

	// The expected version of the skupper binary found on the PATH
	ENV_VERSION = "SKUPPER_TEST_VERSION"

	EnvOldConfigSyncImageEnvKey             string = "SKUPPER_TEST_OLD_SKUPPER_CONFIG_SYNC_IMAGE"
	EnvOldConfigSyncPullPolicyEnvKey        string = "SKUPPER_TEST_OLD_SKUPPER_CONFIG_SYNC_IMAGE_PULL_POLICY"
	EnvOldFlowCollectorImageEnvKey          string = "SKUPPER_TEST_OLD_SKUPPER_FLOW_COLLECTOR_IMAGE"
	EnvOldFlowCollectorPullPolicyEnvKey     string = "SKUPPER_TEST_OLD_SKUPPER_FLOW_COLLECTOR_IMAGE_PULL_POLICY"
	EnvOldPrometheusServerImageEnvKey       string = "SKUPPER_TEST_OLD_PROMETHEUS_SERVER_IMAGE"
	EnvOldPrometheusServerPullPolicyEnvKey  string = "SKUPPER_TEST_OLD_PROMETHEUS_SERVER_IMAGE_PULL_POLICY"
	EnvOldRouterImageEnvKey                 string = "SKUPPER_TEST_OLD_QDROUTERD_IMAGE"
	EnvOldRouterPullPolicyEnvKey            string = "SKUPPER_TEST_OLD_QDROUTERD_IMAGE_PULL_POLICY"
	EnvOldServiceControllerImageEnvKey      string = "SKUPPER_TEST_OLD_SKUPPER_SERVICE_CONTROLLER_IMAGE"
	EnvOldServiceControllerPullPolicyEnvKey string = "SKUPPER_TEST_OLD_SKUPPER_SERVICE_CONTROLLER_IMAGE_PULL_POLICY"

	EnvOldSkupperImageRegistryEnvKey    string = "SKUPPER_TEST_OLD_SKUPPER_IMAGE_REGISTRY"
	EnvOldPrometheusImageRegistryEnvKey string = "SKUPPER_TEST_OLD_PROMETHEUS_IMAGE_REGISTRY"

	// Starting with 1.5
	EnvOldOauthProxyImageEnvKey            string = "SKUPPER_TEST_OLD_OAUTH_PROXY_IMAGE"
	EnvOldOauthProxyPullPolicyEnvKey       string = "SKUPPER_TEST_OLD_OAUTH_PROXY_IMAGE_PULL_POLICY"
	EnvOldControllerPodmanImageEnvKey      string = "SKUPPER_TEST_OLD_SKUPPER_CONTROLLER_PODMAN_IMAGE"
	EnvOldControllerPodmanPullPolicyEnvKey string = "SKUPPER_TEST_OLD_SKUPPER_CONTROLLER_PODMAN_IMAGE_PULL_POLICY"
	EnvOldOauthProxyRegistryEnvKey         string = "SKUPPER_TEST_OLD_OAUTH_PROXY_IMAGE_REGISTRY"
)

TODO: Move all skupper-specific variables to a skupper-specific file, on a skupper-specific package

View Source
const (
	// If defined, both stdout and stderr of all issued skupper commands
	// will be shown on the test output, even if they did not fail
	ENV_CLI_VERBOSE_COMMANDS = "SKUPPER_TEST_CLI_VERBOSE_COMMANDS"
)
View Source
const (
	// This sets the 'Allow' parameter of the retry block for the final
	// validations, and needs to be an integer value.  Any validations
	// marked as final will be retried this many times at the end of the
	// test.
	ENV_FINAL_RETRY = "SKUPPER_TEST_FINAL_RETRY"
)
View Source
const EnvFrame2Verbose = "SKUPPER_TEST_FRAME2_VERBOSE"

Variables

View Source
var EnvOldMap = map[string]string{
	EnvOldConfigSyncImageEnvKey:             "SKUPPER_CONFIG_SYNC_IMAGE",
	EnvOldConfigSyncPullPolicyEnvKey:        "SKUPPER_CONFIG_SYNC_IMAGE_PULL_POLICY",
	EnvOldFlowCollectorImageEnvKey:          "SKUPPER_FLOW_COLLECTOR_IMAGE",
	EnvOldFlowCollectorPullPolicyEnvKey:     "SKUPPER_FLOW_COLLECTOR_IMAGE_PULL_POLICY",
	EnvOldPrometheusServerImageEnvKey:       "PROMETHEUS_SERVER_IMAGE",
	EnvOldPrometheusServerPullPolicyEnvKey:  "PROMETHEUS_SERVER_IMAGE_PULL_POLICY",
	EnvOldRouterImageEnvKey:                 "QDROUTERD_IMAGE",
	EnvOldRouterPullPolicyEnvKey:            "QDROUTERD_IMAGE_PULL_POLICY",
	EnvOldServiceControllerImageEnvKey:      "SKUPPER_SERVICE_CONTROLLER_IMAGE",
	EnvOldServiceControllerPullPolicyEnvKey: "SKUPPER_SERVICE_CONTROLLER_IMAGE_PULL_POLICY",

	EnvOldSkupperImageRegistryEnvKey:    "SKUPPER_IMAGE_REGISTRY",
	EnvOldPrometheusImageRegistryEnvKey: "PROMETHEUS_IMAGE_REGISTRY",

	EnvOldOauthProxyImageEnvKey:            "OAUTH_PROXY_IMAGE",
	EnvOldOauthProxyPullPolicyEnvKey:       "OAUTH_PROXY_IMAGE_PULL_POLICY",
	EnvOldControllerPodmanImageEnvKey:      "SKUPPER_CONTROLLER_PODMAN_IMAGE",
	EnvOldControllerPodmanPullPolicyEnvKey: "SKUPPER_CONTROLLER_PODMAN_IMAGE_PULL_POLICY",
	EnvOldOauthProxyRegistryEnvKey:         "OAUTH_PROXY_IMAGE_REGISTRY",
}

final

The map between the variables that indicate the image value for the old version, and the environment variable that actually needs to be set on the environment for that configuration to be effective. Perhaps it would be simpler to just s/SKUPPER_TEST_OLD//?

Functions

func ContextOrDefault

func ContextOrDefault(ctx context.Context) context.Context

If the given context is not nil, return it.

Otherwise, return a default context.

For now, that's a brand new context.Background(), but that might change

func Flag

func Flag()

Right now, this will only add the -H flag for showing the flag's help (flag.Usage())

func GetId

func GetId() string

Returns the string form of the UUID that identifies this test execution

Every execution has a unique ID, that is logged at the very start of the execution.

func GetInt

func GetInt(name string, default_ int) int

Returns the integer value of the named variable; returns the default value if not defined or empty. If there is a value and not an int, panic

TODO: once this file is moved to a package 'env', the function name 'env.GetInt' will make more sense

func GetShortId

func GetShortId() string

Returns the small format of GetId(). It's the three first hex characters of the MD5 form of GetId()

func IsVerboseCommandOutput

func IsVerboseCommandOutput() bool

func OrSetLogger

func OrSetLogger(x FrameLogger, logger *log.Logger)

This checks that its input is a frame2.Logger, then calls OrSetLogger on it.

Its only objective is to keep the code concise (ie, remove all those type checks from the main code)

x should be a pointer to a FrameLogger instance

func SourceRoot

func SourceRoot() string

Returns the root of the source directory

This assumes that this very file is located at the pkg directory, from the source root. Refactoring may require changes to the code

Types

type AlwaysDisruptor

type AlwaysDisruptor interface {
	// This is just a marker; it does nothing
	AlwaysDisruptor()
}

This is just a marker to indicate that the disruptor does not need to be listed on Run.AlwaysDisruptor on the test; just having it on the environment will suffice for it to take effect

type Asserter

type Asserter struct {
	// contains filtered or unexported fields
}

A helper to check a list of conditions and report them back as a block

func (*Asserter) Check

func (a *Asserter) Check(condition bool, message string, params ...any) error

Checks the condition and returns a new error if false.

The error is also saved on the list of errors in the Asserter, and the counters are updated

func (*Asserter) CheckError

func (a *Asserter) CheckError(err error, template string, msgParams ...any) error

Updates the counters and error list, and return its input

func (*Asserter) Error

func (a *Asserter) Error() error

Returns a new error that lists all errors detected by the Asserter, or nil if none.

func (*Asserter) GetErrors

func (a *Asserter) GetErrors() []error

func (*Asserter) GetStats

func (a *Asserter) GetStats() (failures, successes, checks int)

type DefaultMonitor

type DefaultMonitor struct {
	// A list of validators and their names, which cannot conflict
	// with those in ValidatorConfigs
	Validators map[string]Validator

	// The interval between Validator runs.  For ValidatorConfigs, that's
	// the default, if not specified there
	Interval time.Duration

	// Same as Validators, but allow for per-Validator configuration
	ValidatorConfigs []ValidatorConfig

	// TODO Path where the report should be put.  If not set, TODO
	Path string

	Results map[string][]MonitorResult

	Log
	// contains filtered or unexported fields
}

func (*DefaultMonitor) Execute

func (m *DefaultMonitor) Execute() error

This actually only sets up the monitor, internally. The actual execution of the monitor is on Monitor(r).

For any validators configured on this Monitor with an empty Logger configuration, the Monitor will replace it by a no-op Monitor (log.New(io.Discard, "[M]", 0))

func (*DefaultMonitor) Monitor

func (m *DefaultMonitor) Monitor(runner *Run) error

func (*DefaultMonitor) Report

func (d *DefaultMonitor) Report() error

Result() error

func (*DefaultMonitor) Teardown

func (m *DefaultMonitor) Teardown() Executor

type DefaultRunDealer

type DefaultRunDealer struct {
	// Perhaps make this private?
	// For now it is public to break less things
	Runner *Run

	// TODO
	// When implemented, should it really be here, or have another one
	// for it?
	// This needs to be public, so user can change it at will?
	Ctx context.Context
}

DefaultRunDealer is a helper for building frames of more complex behavior

This struct should be embedded into frames that deal with the Context, or that execute other frames via a frame2.Phase.

The framework will automatically update their Runner.

This should be always embedded directly, and never as a reference type.

func (*DefaultRunDealer) GetRunner

func (d *DefaultRunDealer) GetRunner() *Run

func (*DefaultRunDealer) SetRunner

func (d *DefaultRunDealer) SetRunner(parent *Run, kind RunnerType)

type Disruptor

type Disruptor interface {
	DisruptorEnvValue() string
}

TODO Add a new method, Exercised() bool, that reports whether the disruptor was exercised at all, and causes the test to report Skipped if not.

EdgeOnPrivate, for example, should report as not exercised if all namespaces were Public on the test;

Those should probably also be checked for subfinal tests and reset after them; so subtests are checked as well

type DisruptorConfigurer

type DisruptorConfigurer interface {
	Configure(string) error
}

type Execute

type Execute struct {
}

type Executor

type Executor interface {
	Execute() error
}

type Expect

type Expect struct {
	StdOut      []string
	StdErr      []string
	StdOutRe    []regexp.Regexp
	StdErrRe    []regexp.Regexp
	StdOutReNot []regexp.Regexp
	StdErrReNot []regexp.Regexp
}

A way to verify cli commands output

StdOut and StdErr take a slice of plain strings. It will expect that each string comes after the previous one. In other words, the search for the second string from StdOut starts where the match for the first one finished.

If you want to search for one static string instead, where there is nothing in between each segment, just use a single item with one big string

StdOutRe and StdErrRe take a slice of regular expressions. Those do not have the same restriction on one coming after the other. If you want that behavior with regexes, create a single regex with the two expressions you're looking for.

StdOutReNot and StdErrReNot behave like the previous ones, but ensure that the patters are not there in the checked string

func (Expect) Check

func (e Expect) Check(stdout, stderr string) (err error)

Checks all items from the specification.

type FinalizerHook

type FinalizerHook interface {

	// FinalizerHook will be executed at the end of the
	// test, before all other finalizer tasks, such as the
	// re-run of validators marked as final
	PreFinalizerHook(runner *Run) error

	// PostSubFinalizerHook will be executed after the
	// final validators are run.  It can be used, for example,
	// to reset the disruptor, so it starts a new on the
	// next sub test.  On an upgrade disruptor, for example,
	// the next cycle that has a subfinalizer needs to start
	// with the old version again.
	PostSubFinalizerHook(runner *Run) error
}

type FrameLogger

type FrameLogger interface {
	// passes its parameters to a log.Logger's Printf.  Either
	// the one returned by GetLogger, or the default logger if
	// that call returns nil
	Printf(format string, v ...any)
	GetLogger() *log.Logger
	SetLogger(logger *log.Logger)

	// Sets the logger, but only if it is currently nil
	OrSetLogger(logger *log.Logger)
}

type Inspector

type Inspector interface {
	Inspect(step *Step, phase *Phase)
}

Disruptors that implement the Inspector interface will have its Inspect() function called before each step is executed.

The disruptor will then be able to analise whether that step is of interest for it or not, or even change the step's configuration

type Log

type Log struct {
	Logger *log.Logger
}

Unconfigured, this simply forwards any Printf calls to the default logger.

If a log.Logger is set, use it instead.

func (*Log) GetLogger

func (l *Log) GetLogger() *log.Logger

func (*Log) OrSetLogger

func (l *Log) OrSetLogger(logger *log.Logger)

Sets the logger, but only if it is currently nil and the new value is not nil

func (*Log) Printf

func (l *Log) Printf(format string, v ...any)

func (*Log) SetLogger

func (l *Log) SetLogger(logger *log.Logger)

type Monitor

type Monitor interface {
	Executor

	// Starts the monitoring in a different goroutine
	Monitor(*Run) error

	// Result() error
	Report() error
}

type MonitorResult

type MonitorResult struct {
	Timestamp time.Time
	Duration  time.Duration
	Step      *Step
	Id        string
	Result    error
}

type Phase

type Phase struct {
	Name      string
	Doc       string
	Setup     []Step
	Teardown  []Step
	MainSteps []Step

	Runner *Run

	Log
	DefaultRunDealer
	// contains filtered or unexported fields
}

While the Run keeps context accross the test, the phases do the actual work, and they allow the test to be split in multiple pieces.

This allows to: - Put a phase on a loop - Access variables populated in previous phases

func (Phase) Execute

func (p Phase) Execute() error

func (*Phase) GetRunner

func (p *Phase) GetRunner() *Run

TODO Review/remove

func (*Phase) Run

func (p *Phase) Run() error

func (*Phase) RunT

func (p *Phase) RunT(t *testing.T) error

For a Phase that did not define a Run, this will create a Run and set its T accordingly

This is only for the simplest case, when a single phase is required.

If the Phase already had a Runner set, it will fail.

type PostMainSetupHook

type PostMainSetupHook interface {
	PostMainSetupHook(runner *Run) error
}

PostMainSetupHook will be executed right after the setup phase completes, before the main steps.

type Procedure

type Procedure struct {
	Fn func()
}

Calls a function with no args and no return values; use it to cancel contexts, for example

func (*Procedure) Execute

func (p *Procedure) Execute() error

type Retry

type Retry struct {
	Fn      RetryFunction // The thing to be retried
	Options RetryOptions
}

func (Retry) ParallelRun

func (r Retry) ParallelRun() func() []error

Runs the retry in parallel; returns a function that will wait and return the results only when it finished (wait). TODO perhaps give it a context, too

func (Retry) Run

func (r Retry) Run() ([]error, error)

type RetryFunction

type RetryFunction func() (err error)

type RetryOptions

type RetryOptions struct {
	Allow    int           // for initial failures
	Ignore   int           // initial successes
	Ensure   int           // last n tries are successful.  Minimum 1
	Retries  int           // after Allow phase
	Interval time.Duration // if not given, the default is 1s
	Quiet    bool          // if true, no attempt logs

	Min int // TODO Run as normal, but delay report until that number of tries have been done

	Rate float32 // TODO: "Ensure" will not be 100%, but based on this rate.  So, if Ensure is 100

	KeepTrying bool
	Ctx        context.Context
	Timeout    time.Duration
}

Allow accounts for instabilities (for example, a service load balanced on two providers might return a mix of successes and failures while the two providers stabilize). The last success streak in this phase will count to Ensure.

Once past the Allow phase, any errors will cause a failure, unless there are Retries available

Even successes may require additional runs. There are two cases here:

  • If Ensure is set, the test will keep trying on success until the required number of successes are met
  • If Ignore was set, that number of successes will be ignored on the count to Ensure, possibly requiring additional runs until the Ensure target is met

These, however, do not count as Retries. i. e, Retries are only those additional runs that were caused by a failure.

The ignore counts from the first success in the last success streak from the Allow phase, or from the start of the retry phase (if no Allow configured or no success in that phase)

func (RetryOptions) IsEmpty

func (r RetryOptions) IsEmpty() bool

Checks whether any fields on the struct have been set

func (RetryOptions) Max

Returns a new RetryOptions, whose values are the maximum between r and other.

The contexts are 'merged'; if any of them is cancelled, the merged context gets cancelled. Cancelling the merged context has no effect on the other contexts

type Run

type Run struct {
	T                  *testing.T
	Doc                string      // TODO this is currently unused (ie, it's not printed)
	RequiredDisruptors []Disruptor // TODO
	// contains filtered or unexported fields
}

The Run (TODO rename?) keeps context accross the execution of the test. Each phase and each step has its runner, in a tree structure

func (*Run) AllowDisruptors

func (r *Run) AllowDisruptors(list []Disruptor)

List the disruptors that a test accepts, and initialize a disruptor if SKUPPER_TEST_DISRUPTOR set on the environment matches a disruptor from the list.

If any of the values on the environment variable does not match a value on the list, the test will be skipped in this run (ie, a disruptor test was requested, but the test does not allow for it).

If the environment variable is empty, this is a no-op.

Attention when calling with disruptors that use pointer reference methods: define them on the list as a reference to the struct. Otherwise, the pointer reference methods will not be part of the method set, and some interfaces may not match

func (*Run) CancelContext

func (r *Run) CancelContext()

func (*Run) ChildWithT

func (r *Run) ChildWithT(t *testing.T, kind RunnerType) *Run

TODO: make just Child(), which reuses the runner's own T

func (*Run) Finalize

func (r *Run) Finalize()

Run steps that are still part of the test, but must be run at its very end, right before the tear down. Failures here will count as test failure

func (*Run) GetContext

func (r *Run) GetContext() context.Context

GetContext will always return a context. If not defined on the current level, check the parent. If not on the parent, create a new Background context.

When contexts are created on this method, they get scheduled for cancellation on T.Cleanup()

TODO contexts are not expected to change?

func (*Run) GetId

func (r *Run) GetId() string

Return the full ID of the Runner, which includes the ID of its parent

func (*Run) OrDefaultContext

func (r *Run) OrDefaultContext(ctx context.Context) context.Context

Return ctx if not nil. If nil, return the runner's default context

If the runner is not available (call on nil reference), return context.Background

TODO review

func (*Run) Report

func (r *Run) Report()

This will cause all active monitors to report their status on the logs.

It should generally be run as defer r.Report(), right after the Run creation

func (*Run) ReportChildren

func (r *Run) ReportChildren(indent int)

type RunDealer

type RunDealer interface {
	SetRunner(parent *Run, kind RunnerType)
	GetRunner() *Run
}

TODO: add SetContext and GetContext, too?

type RunnerType

type RunnerType int

Each step on a phase runs with its own Runner. These constants identify what type of work is being done by the Runner. The step IDs are derived from their Runner type

const (
	RootRunner RunnerType = iota
	PhaseRunner
	ValidatorRunner
	ModifyRunner
	SetupRunner
	HookRunner
	SubTestRunner
	StepRunner
	TearDownRunner
	MonitorRunner
)

type Step

type Step struct {
	Doc   string
	Name  string
	Level int
	// Whether the step should always print logs
	// Even if false, logs will be done if SKUPPER_TEST_FRAME2_VERBOSE
	Verbose      bool
	PreValidator Validator

	// TODO make this require a pointer, like validator does, to avoid
	// Runner disconnects
	Modify            Executor
	Validator         Validator
	Validators        []Validator
	ValidatorRetry    RetryOptions
	ValidatorFinal    bool // final validators are re-run at the test's end
	ValidatorSubFinal bool // and subfinal, at the end of subtests
	Substep           Stepper
	Substeps          []*Step
	SubstepRetry      RetryOptions
	// A simple way to invert the meaning of the Validator.  Validators
	// are encouraged to provide more specific negative testing behaviors,
	// but this serves for simpler testing.  If set, it inverts the
	// response from the call sent to Retry, so it can be used to wait
	// until an error is returned (but there is no control on which kind
	// of error that will be)
	// TODO: change this by OnError: Fail, Skip, Ignore, Expect?
	// TODO: add option to inspect error message for specific error; perhaps a function (error) bool,
	// function(error)error or function(error)frame2.action (where action is fail, skip, etc)
	ExpectError bool
	// TODO: ExpectIs, ExpectAs; use errors.Is, errors.As against a list of expected errors?
	SkipWhen bool
}

func (*Step) GetStep

func (s *Step) GetStep() *Step

func (Step) GetValidators

func (s Step) GetValidators() []Validator

Returns a list where Step.Validator is the first item, followed by Step.Validators

func (Step) IsVerbose

func (s Step) IsVerbose() bool

func (*Step) IterFrames

func (s *Step) IterFrames(transform TransformFunc) error

IterFrames will run transform() on each of its configured frames (Modify, Validator and Validators[]) and reassign the frame to the return of transform().

It allows disruptors to inspect Executors and Validators in a single go.

func (Step) Logf

func (s Step) Logf(format string, v ...interface{})

type Stepper

type Stepper interface {
	GetStep() *Step
}

type TearDowner

type TearDowner interface {
	Teardown() Executor
}

type TransformFunc

type TransformFunc func(any) (any, error)

type ValidationResultHook

type ValidationResultHook interface {
	ValidationResultHook(runner *Run, step Step, err error) error
}

This hook looks into the validations as all retries have been exhausted, and it allows a disruptor to inspect the error, or transform it into another error, or even clear the error, by returning null.

The interface needs some work. Only err is really required; runner and step may be removed or replaced in the future

type Validator

type Validator interface {
	Validate() error
	FrameLogger
}

type ValidatorConfig

type ValidatorConfig struct {
	Name      string
	Validator Validator
	Interval  time.Duration
}

Directories

Path Synopsis
frames
f2skupper1
Frames (Executor and Validator) for Skupper 1.x
Frames (Executor and Validator) for Skupper 1.x
f2skupper1/tester
The files in this package contains two types of strucs:
The files in this package contains two types of strucs:
test_unit
This package contains simple but full tests encapsulated into callable frame2 executors.
This package contains simple but full tests encapsulated into callable frame2 executors.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL