sre

SRE framework

Framework for golang applications which helps to send metrics, logs and traces into different monitoring tools or vendors.

Features

• Provide plain text, json logs with trace ID (if log entry is based on a span) and source line info
• Provide additional labels and tags for metrics, like: source line, service name and it's version
• Support logging tools (aka logs):
  • Stdout (text, json, template) based on Logrus
  • DataDog based on Logrus over UDP
• Support monitoring tools (aka metrics)
  • Prometheus
  • DataDog
  • Opentelemetry
• Support tracing tools (aka traces)
  • Jaeger
  • DataDog
  • Opentelemetry

Usage

Requirements

• Jaeger works with its Jaeger agent
• DataDog uses DataDog agent for logs, metrics and traces
• Opentelemetry communicates with its Opentelemetry agent

Envs

Set proper GOROOT and PATH variables

export GOROOT="$HOME/go/root/1.16.4"
export PATH="$PATH:$GOROOT/bin"

Go modules

Set go.mod manually

module sre

go 1.16

require github.com/devopsext/sre v0.0.5

Collect go modules

go get

go: finding module for package github.com/devopsext/sre/provider
go: finding module for package github.com/devopsext/sre/common
go: found github.com/devopsext/sre/common in github.com/devopsext/sre v0.0.5
go: found github.com/devopsext/sre/provider in github.com/devopsext/sre v0.0.5

Logs usage

Create logs.go file to test logging functionality

package main

import (
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()

func test() {
  logs.Info("Info message to every log provider...")
  logs.Debug("Debug message to every log provider...")
  logs.Warn("Warn message to every log provider...")
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "template",
    Level:           "info",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage 
  stdout.SetCallerOffset(2)

  // initialize DataDog logger
  datadog := provider.NewDataDogLogger(provider.DataDogLoggerOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "sre-datadog",
      Environment: "stage",
    },
    AgentHost:  "localhost", // set DataDog agent UDP logs host
    AgentPort:  10518, // set DataDog agent UDP logs port
    Level: "info",
  }, logs, stdout)

  // add loggers
  logs.Register(stdout)
  logs.Register(datadog)

  test()
}

Run logs example

go run logs.go

go/sre/logs.go:13 Info message to every log provider...
go/sre/logs.go:14 Debug message to every log provider...
go/sre/logs.go:15 Warn message to every log provider...
...

Metrics usage

Create metrics.go file to test metrics functionality

package main

import (
  "sync"
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()
var metrics = common.NewMetrics()
var mainWG sync.WaitGroup

func test() {
  counter := metrics.Counter("calls", "Calls counter", []string{"time"})
  counter.Inc(time.Now().String())
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "template",
    Level:           "debug",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage
  stdout.SetCallerOffset(2)

  // add Stdout logger
  logs.Register(stdout)

  // initialize Prometheus meter
  prometheus := provider.NewPrometheusMeter(provider.PrometheusOptions{
    URL:    "/metrics",
    Listen: "127.0.0.1:8080",
    Prefix: "sre",
  }, logs, stdout)
  prometheus.Start(&mainWG)

  // initialize DataDog meter
  datadog := provider.NewDataDogMeter(provider.DataDogMeterOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "sre-datadog",
      Environment: "stage",
    },
    AgentHost:  "localhost", // set DataDog agent UDP metrics host
    AgentPort:  10518, // set DataDog agent UDP metrics port
  }, logs, stdout)

  // initialize Opentelemetry meter
  opentelemetry := provider.NewOpentelemetryMeter(provider.OpentelemetryMeterOptions{
    OpentelemetryOptions: provider.OpentelemetryOptions{
      ServiceName: "sre-opentelemetry",
      Environment: "stage",
    },
    AgentHost:   "localhost", // set Opentelemetry agent metrics host
    AgentPort:   4317,        // set Opentelemetry agent metrics port
    Prefix: "sre",
  }, logs, stdout)

  // add meters
  metrics.Register(prometheus)
  metrics.Register(datadog)
  metrics.Register(opentelemetry)

  test()

  mainWG.Wait()
  metrics.Stop() // finalize metrics delivery
}

Run metrcis example

go run metrics.go

sre@v0.0.5/provider/prometheus.go:83 Start prometheus endpoint...
sre@v0.0.5/provider/prometheus.go:93 Prometheus is up. Listening...
sre@v0.0.5/provider/datadog.go:648 DataDog meter is up...

Check Prometheus metrics

curl -sk http://127.0.0.1:8080/metrics | grep sre_

# HELP sre_calls Calls counter
# TYPE sre_calls counter
sre_calls{time="2021-06-17 18:00:30.248990729 +0300 EEST m=+0.002878298"} 1

Traces usage

Create traces.go file to test traces functionality

package main

import (
  "fmt"
  "time"

  "github.com/devopsext/sre/common"
  "github.com/devopsext/sre/provider"
)

var logs = common.NewLogs()
var traces = common.NewTraces()

func test() {

  rootSpan := traces.StartSpan()
  spanCtx := rootSpan.GetContext()
  for i := 0; i < 10; i++ {

    span := traces.StartChildSpan(spanCtx)
    span.SetName(fmt.Sprintf("name-%d", i))

    time.Sleep(time.Duration(100*i) * time.Millisecond)
    logs.SpanDebug(span, "Counter increment %d", i)

    spanCtx = span.GetContext()
    span.Finish()
  }

  // emulate delay of 100 msecs
  time.Sleep(time.Duration(200) * time.Millisecond)

  rootSpan.Finish()

  // wait for a while to delivery all spans to provider
  time.Sleep(time.Duration(3000) * time.Millisecond)
}

func main() {

  // initialize Stdout logger
  stdout := provider.NewStdout(provider.StdoutOptions{
    Format:          "template",
    Level:           "debug",
    Template:        "{{.file}} {{.msg}}",
    TimestampFormat: time.RFC3339Nano,
    TextColors:      true,
  })
  // set caller offset for file:line proper usage
  stdout.SetCallerOffset(2)

  // add Stdout logger
  logs.Register(stdout)

  // initialize Jaeger tracer
  jaeger := provider.NewJaegerTracer(provider.JaegerOptions{
    ServiceName:         "sre-jaeger",
    AgentHost:           "localhost", // set Jaeger agent host
    AgentPort:           6831,        // set Jaeger agent port
    BufferFlushInterval: 0,
    QueueSize:           0,
    Tags:                "key1=value1",
  }, logs, stdout)

  // initialize DataDog tracer
  datadog := provider.NewDataDogTracer(provider.DataDogTracerOptions{
    DataDogOptions: provider.DataDogOptions{
      ServiceName: "sre-datadog",
      Environment: "stage",
    },
    AgentHost: "", // set DataDog agent traces host
    AgentPort: 8126,        // set DataDog agent traces port
  }, logs, stdout)

  // initialize Opentelemetry tracer
  opentelemetry := provider.NewOpentelemetryTracer(provider.OpentelemetryTracerOptions{
    OpentelemetryOptions: provider.OpentelemetryOptions{
      ServiceName: "sre-opentelemetry",
      Environment: "stage",
    },
    AgentHost: "localhost", // set Opentelemetry agent traces host
    AgentPort: 4317,        // set Opentelemetry agnet traces port
  }, logs, stdout)

  // add traces
  traces.Register(jaeger)
  if datadog != nil {
    traces.Register(datadog)
  }
  traces.Register(opentelemetry)

  test()

  traces.Stop() // finalize traces delivery
}

Run traces example

go run traces.go

...
sre@v0.0.5/provider/jaeger.go:444 Jaeger tracer is up...
go/sre/traces.go:66 DataDog tracer is disabled.
sre@v0.0.5/provider/opentelemetry.go:493 Opentelemetry tracer is up...
go/sre/traces.go:24 Counter increment 0
go/sre/traces.go:24 Counter increment 1
go/sre/traces.go:24 Counter increment 2
...

Go to Jaeger UI and there should be seen

For a case of Opentelemetry (Lightstep) screenshoot will be

Framework in other projects

• devopsext/events Kubernetes & Alertmanager events to Telegram, Slack, Workchat and other messengers.