go-service

Go Service

A framework to build services in Go. This came out of us building services over the years and what we consider good practices in building services. Hence it is highly subjective and opinionated.

This framework stands on the shoulder of giants so we don't reinvent the wheel!

Dependency Injection

This framework heavily relies on DI. We have chosen to use Uber FX. So there is great information online to get you up to speed.

CLI

A service has commands that are configured using acmd. Each service has the following commands:

• Server - These are long running apps, e.g. daemons.
• Client - These are short lived apps, e.g. control.

These are configured in the main function.

Configuration

The supported configuration kinds are as follows:

• JSON
• TOML
• YAML

The configuration can be read from multiple sources by specifying a flag called -i. As per the following:

• env:CONFIG - Read from an env variable called CONFIG. The env variable must be a configuration and we expect the format of extension:content, where extension is the supported kinds and content contains the contents of the config that is base64 encoded.
• file:path - Read from the path.
• If all the above fails it will try common locations, such as:
  • The binary location.
  • The user config directory (os.UserConfigDir()) under <serviceName>/.
  • The /etc folder.

The reason for this is that we want to be able to separate how configuration is retrieved.

This is the configuration. We will outline the config required in each section. The following configuration examples will use YAML.

Environment

You can specify the environment of the service.

Paths

In any of the configurations where a path is specified we allow the following:

• env:CONFIG - Read from an environment variable.
• file:path - Read from a file path.
• string - Any arbitrary string.

Environment Configuration

To configure, please specify the following:

environment: development

Compression

We support the following:

• None
• Zstd
• S2
• Snappy

Encoders

We support the following:

• JSON
• TOML
• YAML
• Proto
• GOB

Caching

The framework currently supports the following caching solutions from the awesome Cachego.

Cache Configuration

To configure, please specify the following:

cache:
  kind: redis | sync
  compressor: none | s2 | zstd | snappy
  encoder: json | toml | yaml | proto | gob
  options:
    url: path to url

Feature

The framework supports OpenFeature.

Feature Configuration

To configure, please specify the following:

feature:
  address: localhost:9000
  retry:
    backoff: 100ms
    timeout: 1s
    attempts: 3
  timeout: 10s

Hooks

The framework supports Standard Webhooks.

Hooks Configuration

To configure, please specify the following:

hooks:
  secret: path to secret

ID

The framework supports the generation of ids. The following are supported:

• uuid
• ksuid
• nanoid
• ulid
• xid

ID Configuration

To configure, please specify the following:

id:
  kind: uuid | ksuid | nanoid | ulid | xid

Runtime

We enhance the runtime with the following:

• Automemlimit

SQL

For SQL databases we support the following:

• Postgres

We also support master, slave combinations with the awesome mssqlx.

SQL Configuration

To configure, please specify the following:

sql:
  pg:
    masters:
      -
        url: path to url
    slaves:
      -
        url: path to url
    max_open_conns: 5
    max_idle_conns: 5
    conn_max_lifetime: 1h

Dependencies

Health

The health package is based on go-health. This package allows us to create all sorts of ways to check external and internal systems.

We also provide ways to integrate into container integration systems. So we provide the following endpoints:

• /name/healthz - This allows us to check any external dependency and provide a breakdown of what is not functioning. This should only be used for verification.
• /name/livez: Can be used for k8s liveness.
• /name/readyz: Can be used for k8s readiness.

This is modelled around Kubernetes API health endpoints.

Telemetry

Telemetry is broken down in the following sections:

Logging

For logging we use slog.

Logging Configuration

We have multiple options for logging.

To configure, please specify the following:

telemetry:
  logger:
    kind: json
    level: info

To configure, please specify the following:

telemetry:
  logger:
    kind: text
    level: info

To configure, please specify the following:

telemetry:
  logger:
    kind: otlp
    level: info
    url: http://localhost:3100/loki/api/v1/push
    headers:
      Authorization: path to key

Metrics

For metrics we support the following:

• OpenTelemetry
• Prometheus

Metrics Configuration

Below is the configuration for each system.

To configure, please specify the following:

telemetry:
  metrics:
    kind: prometheus

To configure, please specify the following:

telemetry:
  metrics:
    kind: otlp
    url: http://localhost:9009/otlp/v1/metrics
    headers:
      Authorization: path to key

Trace

For distributed tracing we support the following:

• OpenTelemetry

Trace Configuration

Below is the configuration for each system.

To configure, please specify the following:

telemetry:
  tracer:
    kind: otlp
    url: http://localhost:4318/v1/traces
    headers:
      Authorization: path to key

Telemetry Libraries

• https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime
• https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/host
• https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
• https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
• https://github.com/redis/go-redis/tree/master/extra/redisotel
• https://github.com/XSAM/otelsql

Telemetry Dependencies

Token

The framework allows you to define different token generators and verifiers.

Access

We have support for different access controls using casbin.

JWT

We use the awesome JWT.

Paseto

We use the awesome Paseto.

SSH

We use the awesome SSH.

Limiter

The framework allows you to define a limiter. This will be applied to the different transports.

The different kinds are:

• user-agent
• ip
• token

Time

The framework allows you to use network time services. We use:

• ntp
• nts

Time Configuration

To configure, please specify the following:

time:
  kind: nts
  address: time.cloudflare.com

Transport

The transport layer provides ways to abstract communication for in/out of the service. So we have the following integrations:

• gRPC - The author truly believes in IDLs.
• REST - An abstraction using content negotiation.
• RPC - abstraction using content negotiation.
• MVC - We have a simple framework.
• CloudEvents - A specification for describing event data in a common way.

gRPC

Below is list of the provided interceptors:

• Limiter

REST

Below is list of the provided handlers:

• Limiter

Transport Configuration

To configure, please specify the following:

transport:
  http:
    address: tcp://localhost:8000
    retry:
      backoff: 100ms
      timeout: 1s
      attempts: 3
    timeout: 10s
    options:
      read_timeout: 10s
      write_timeout: 10s
      idle_timeout: 10s
      read_header_timeout: 10s
  grpc:
    address: tcp://localhost:9000
    retry:
      backoff: 100ms
      timeout: 1s
      attempts: 3
    timeout: 10s
    options:
      keepalive_enforcement_policy_ping_min_time: 10s
      keepalive_max_connection_idle: 10s
      keepalive_max_connection_age: 10s
      keepalive_max_connection_age_grace: 10s
      keepalive_ping_time: 10s

If you would like to enable TLS, do the following:

transport:
  http:
    tls:
      cert: path of cert
      key: path of key
  grpc:
    tls:
      cert: path of cert
      key: path of key

If you would like to enable a limiter, do the following:

transport:
  http:
    limiter:
      kind: user-agent
      tokens: 10
      interval: 1s
  grpc:
    limiter:
      kind: user-agent
      tokens: 10
      interval: 1s

Transport Token Access

To configure, please specify the following:

transport:
  http:
    token:
      access:
        policy: path to policy file
  grpc:
    token:
      access:
        policy: path to policy file

The model is based on the following config.

Transport Token JWT

To configure, please specify the following:

transport:
  http:
    token:
      kind: jwt
      jwt:
        iss: issuer
        exp: 1h
        kid: 1234567890
  grpc:
    token:
      kind: jwt
      jwt:
        iss: issuer
        exp: 1h
        kid: 1234567890

Transport Token Paseto

To configure, please specify the following:

transport:
  http:
    token:
      kind: paseto
      paseto:
        iss: issuer
        exp: 1h
  grpc:
    token:
      kind: paseto
      paseto:
        iss: issuer
        exp: 1h

Transport Token SSH

To verify, please specify the following:

transport:
  http:
    token:
      kind: ssh
      ssh:
        keys:
          - name: test
            public: path to key
  grpc:
    token:
      kind: ssh
      ssh:
        keys:
          - name: test
            public: path to key

Transport Dependencies

Cryptography

The crypto package provides sensible defaults for symmetric, asymmetric, hashing and randomness.

We rely on the awesome crypto.

Cryptography Configuration

To configure, please specify the following:

crypto:
  aes:
    key: path to the key
  ed25519:
    public: path to the public
    private: path to the private
  hmac:
    key: path to the key
  rsa:
    public: path to the public
    private: path to the private
  ssh:
    public: path to the public
    private: path to the private

Cryptography Dependencies

Debug

This section outlines all utilities added for your troubleshooting abilities.

statsviz

GET http://localhost:6060/debug/statsviz

Check out statsviz.

pprof

GET http://localhost:6060/debug/pprof/
GET http://localhost:6060/debug/pprof/cmdline
GET http://localhost:6060/debug/pprof/profile
GET http://localhost:6060/debug/pprof/symbol
GET http://localhost:6060/debug/pprof/trace

Check out pprof.

fgprof

GET http://localhost:6060/debug/fgprof?seconds=10

Check out fgprof.

gopsutil

GET http://localhost:6060/debug/psutil

Check out gopsutil.

Debug Configuration

To configure, please specify the following:

debug:
  address: tcp://localhost:6060
  timeout: 10s

If you would like to enable TLS, do the following:

debug:
  tls:
    cert: path of cert
    key: path of key

Development

This section describes how to run and contribute to the project, if you are interested.

Style

We favour what is defined in the Uber Go Style Guide.

Development Dependencies

Please setup the following:

• https://github.com/FiloSottile/mkcert

Setup

To get yourself setup, please run:

git submodule sync
git submodule update --init

mkcert -install
make create-certs

make dep

Development Environment

As we rely on external services these need to be configured:

Starting

Please run:

make start

Stopping

Please run:

make stop

Testing

To be able to test locally, please run:

make specs