Documentation
¶
Overview ¶
Package errorlog is the laboratory's own error journal: every component (operator, node-agent, L7 proxy, wg-demux, VPN and gateway pods, the agent) counts its errors by a normalized fingerprint, keeps a few recent messages and publishes the aggregate as a Kubernetes Event that the management agent reads and forwards in MonitoringUpdate.errors. Nothing here may carry tenant or participant data, an address or a secret: every message passes Redact before it is stored, and the fingerprint is made from the further Normalize d text.
Index ¶
- Constants
- func Fingerprint(component, kind, normalized string) string
- func Instance() string
- func Namespace(def string) string
- func Normalize(msg string) string
- func Redact(msg string) string
- func Start(ctx context.Context, agg *Aggregator, cfg *rest.Config, namespace string, ...)
- func WrapLogger(l logr.Logger, agg *Aggregator) logr.Logger
- type Aggregator
- type Group
- type LineWriter
- type Observation
- type Publisher
- type Sink
Constants ¶
const ( // Reason of every journal event. Reason = "ErrorJournal" // Label marks the events of the journal (a cheap server-side selector for the reader). Label = "cybericebox.com/error-journal" )
The journal rides on Kubernetes Events, which every component may already write and the management agent may already read (the API server keeps them for an hour, deduplicates and rate-limits them): no log access and no new kind of permission.
const MaxGroups = 200
MaxGroups bounds the distinct errors one process keeps; past it new ones count into the "overflow" group.
const (
// MaxSampleLen is the longest message kept.
MaxSampleLen = 300
)
const MaxSamples = 3
MaxSamples is how many recent messages a group keeps.
const OverflowKind = "overflow"
OverflowFingerprintKind is the kind of the group that takes the errors past MaxGroups.
Variables ¶
This section is empty.
Functions ¶
func Fingerprint ¶
Fingerprint is the stable id of an error: the component, the kind of the error path and the normalized message.
func Instance ¶
func Instance() string
Instance is the name of this process's pod (its hostname), never an address.
func Namespace ¶
Namespace is where this component publishes: ERROR_JOURNAL_NAMESPACE, else the default.
func Normalize ¶
Normalize is Redact with the parts that vary between occurrences of the same error replaced, so that the occurrences group together: numbers become <n>, long hex runs <id>, generated pod names lose their suffix. The fingerprint is made from it.
func Redact ¶
Redact removes what must not leave the cluster from an error message: key material, tokens, passwords, addresses (IPv4, IPv6, MAC), email addresses, host names, the names of objects and any quoted string (they carry tenant and group names), and shortens it to MaxSampleLen. What is left says what failed, not for whom.
func Start ¶
func Start(ctx context.Context, agg *Aggregator, cfg *rest.Config, namespace string, log logr.Logger)
Start publishes the aggregator's changed groups as Events in namespace until ctx ends. The journal is best effort: a cluster that will not take the events only loses the journal, the component is never stopped for it.
func WrapLogger ¶
func WrapLogger(l logr.Logger, agg *Aggregator) logr.Logger
WrapLogger returns a logger that records its errors in agg and logs as l does.
Types ¶
type Aggregator ¶
type Aggregator struct {
Component string
Instance string
// contains filtered or unexported fields
}
Aggregator counts the errors of one process (one component instance) by fingerprint. It is safe for concurrent use.
func New ¶
func New(component, instance string) *Aggregator
New makes an aggregator for a component instance (a node name or a pod name, never an address).
func Setup ¶
Setup wraps a logger so that its errors are journalled; the returned aggregator is what Start publishes.
func (*Aggregator) Dirty ¶
func (a *Aggregator) Dirty() []Group
Dirty returns the groups that changed since the last call (copies) and forgets the mark.
func (*Aggregator) Record ¶
func (a *Aggregator) Record(kind, msg string)
Record counts one error of the given kind (a short code of the error path: reconcile, ovs, registry, ...).
func (*Aggregator) Snapshot ¶
func (a *Aggregator) Snapshot() []Group
Snapshot returns every group (copies).
type Group ¶
type Group struct {
Fingerprint string
Kind string
Normalized string
// Total counts the occurrences since the process started; the reader takes the difference between two looks.
Total int64
First, Last time.Time
Samples []string
}
Group is one distinct error of one component.
type LineWriter ¶
type LineWriter struct {
// contains filtered or unexported fields
}
LineWriter wraps the writer of the standard library logger: every line that reads like an error is counted, every line is passed on. The agent logs with the standard logger.
func NewLineWriter ¶
func NewLineWriter(agg *Aggregator, next io.Writer) *LineWriter
NewLineWriter counts the error-looking lines written to it into agg and writes all of them to next.
type Observation ¶
type Observation struct {
Component, Instance, Fingerprint, Kind, Normalized string
Total int64
First, Last time.Time
Samples []string
}
Observation is one journal event read back.
func ParseEvent ¶
func ParseEvent(ev *corev1.Event) (Observation, bool)
ParseEvent reads a journal event; false when it is not one (or is damaged).
type Publisher ¶
type Publisher struct {
Client kubernetes.Interface
Namespace string
Agg *Aggregator
Interval time.Duration
Log logr.Logger
}
Publisher writes the changed groups of an aggregator as Events.
type Sink ¶
type Sink struct {
// contains filtered or unexported fields
}
Sink is a logr sink that counts every error-level message into an Aggregator and passes everything on to the real sink. The error path of any controller-runtime based component is then in the journal without touching its code.
func (*Sink) Init ¶
func (s *Sink) Init(info logr.RuntimeInfo)