substation

package module
v1.0.0-beta.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 5, 2023 License: MIT Imports: 5 Imported by: 0

README

Substation

substation logo

Substation is a cloud-native, event-driven data pipeline toolkit designed for security and observability teams.

Resources

Features

Substation provides three unique capabilities:

  • Deploy modular, serverless data pipelines in minutes
    • Design pipelines based on your unique use cases and requirements
    • Autoscale beyond 100,000 events per second with almost zero maintenance
    • Route data to SIEMs, data lakes, and other log management platforms
  • Inspect, normalize, and enrich event logs in real-time
    • Inspect data before applying transformation functions and routing decisions
    • Normalize data to a common schema for easy analysis and correlation
    • Enrich data with threat, infrastructure, and business context
  • Create custom data processing applications written in Go
    • Build Substation applications that run in any cloud environment or on-prem
    • Use Substation's Go packages to inspect and transform data in your own applications

Getting Started

Substation Explained

Substation transforms event logs like this ...

{
  "ts": 1591367999.305988,
  "uid": "CMdzit1AMNsmfAIiQc",
  "id.orig_h": "192.168.4.76",
  "id.orig_p": 36844,
  "id.resp_h": "192.168.4.1",
  "id.resp_p": 53,
  "proto": "udp",
  "service": "dns",
  "duration": 0.06685185432434082,
  "orig_bytes": 62,
  "resp_bytes": 141,
  "conn_state": "SF",
  "missed_bytes": 0,
  "history": "Dd",
  "orig_pkts": 2,
  "orig_ip_bytes": 118,
  "resp_pkts": 2,
  "resp_ip_bytes": 197
}
{
  "ts": 1591367999.430166,
  "uid": "C5bLoe2Mvxqhawzqqd",
  "id.orig_h": "192.168.4.76",
  "id.orig_p": 46378,
  "id.resp_h": "31.3.245.133",
  "id.resp_p": 80,
  "proto": "tcp",
  "service": "http",
  "duration": 0.25411510467529297,
  "orig_bytes": 77,
  "resp_bytes": 295,
  "conn_state": "SF",
  "missed_bytes": 0,
  "history": "ShADadFf",
  "orig_pkts": 6,
  "orig_ip_bytes": 397,
  "resp_pkts": 4,
  "resp_ip_bytes": 511
}

... into this ...

{
  "event": {
    "original": {
      "ts": 1591367999.305988,
      "uid": "CMdzit1AMNsmfAIiQc",
      "id.orig_h": "192.168.4.76",
      "id.orig_p": 36844,
      "id.resp_h": "192.168.4.1",
      "id.resp_p": 53,
      "proto": "udp",
      "service": "dns",
      "duration": 0.06685185432434082,
      "orig_bytes": 62,
      "resp_bytes": 141,
      "conn_state": "SF",
      "missed_bytes": 0,
      "history": "Dd",
      "orig_pkts": 2,
      "orig_ip_bytes": 118,
      "resp_pkts": 2,
      "resp_ip_bytes": 197
    },
    "hash": "7ed38f773271e700e2d55984a2ba7902be9ec8c2922e52fc7558aeade425c3de",
    "created": "2022-12-30T17:20:41.027457Z",
    "id": "CMdzit1AMNsmfAIiQc",
    "kind": "event",
    "category": [
      "network"
    ],
    "action": "network-connection",
    "outcome": "success",
    "duration": 66851854.32434082
  },
  "@timestamp": "2020-06-05T14:39:59.305988Z",
  "client": {
    "address": "192.168.4.76",
    "ip": "192.168.4.76",
    "port": 36844,
    "packets": 2,
    "bytes": 62
  },
  "server": {
    "address": "192.168.4.1",
    "ip": "192.168.4.1",
    "port": 53,
    "packets": 2,
    "bytes": 141
  },
  "network": {
    "protocol": "udp",
    "bytes": 203,
    "packets": 4,
    "direction": "internal"
  }
}
{
  "event": {
    "original": {
      "ts": 1591367999.430166,
      "uid": "C5bLoe2Mvxqhawzqqd",
      "id.orig_h": "192.168.4.76",
      "id.orig_p": 46378,
      "id.resp_h": "31.3.245.133",
      "id.resp_p": 80,
      "proto": "tcp",
      "service": "http",
      "duration": 0.25411510467529297,
      "orig_bytes": 77,
      "resp_bytes": 295,
      "conn_state": "SF",
      "missed_bytes": 0,
      "history": "ShADadFf",
      "orig_pkts": 6,
      "orig_ip_bytes": 397,
      "resp_pkts": 4,
      "resp_ip_bytes": 511
    },
    "hash": "af70ea0b38e1fb529e230d3eca6badd54cd6a080d7fcb909cac4ee0191bb788f",
    "created": "2022-12-30T17:20:41.027505Z",
    "id": "C5bLoe2Mvxqhawzqqd",
    "kind": "event",
    "category": [
      "network"
    ],
    "action": "network-connection",
    "outcome": "success",
    "duration": 254115104.67529297
  },
  "@timestamp": "2020-06-05T14:39:59.430166Z",
  "client": {
    "address": "192.168.4.76",
    "ip": "192.168.4.76",
    "port": 46378,
    "packets": 6,
    "bytes": 77
  },
  "server": {
    "address": "31.3.245.133",
    "ip": "31.3.245.133",
    "port": 80,
    "packets": 4,
    "bytes": 295,
    "domain": "h31-3-245-133.host.redstation.co.uk",
    "top_level_domain": "co.uk",
    "subdomain": "h31-3-245-133.host",
    "registered_domain": "redstation.co.uk",
    "as": {
      "number": 20860,
      "organization": {
        "name": "Iomart Cloud Services Limited"
      }
    },
    "geo": {
      "continent_name": "Europe",
      "country_name": "United Kingdom",
      "city_name": "Manchester",
      "location": {
        "latitude": 53.5039,
        "longitude": -2.1959
      },
      "accuracy": 1000
    }
  },
  "network": {
    "protocol": "tcp",
    "bytes": 372,
    "packets": 10,
    "direction": "outbound"
  }
}

... using this ...

local sub = import 'substation.libsonnet';

local event = import 'event.libsonnet';
local client = import 'client.libsonnet';
local server = import 'server.libsonnet';
local network = import 'network.libsonnet';
local send = import 'send.libsonnet';

{
  transforms: 
    event.transforms
    + client.transforms
    + server.transforms
    + network.transforms
    + send.transforms
}

... running in any data pipeline like these ...

alt text

Licensing

Substation and its associated code is released under the terms of the MIT License.

Documentation

Overview

Example (CustomSubstation) 
package main

import (
	"context"
	"encoding/json"
	"fmt"

	sub "github.com/brexhq/substation"
)

// Custom applications should embed the Substation configuration and
// add additional configuration options.
type customConfig struct {
	sub.Config

	Auth struct {
		Username string `json:"username"`

		Password string `json:"password"`
	} `json:"auth"`
}

// String returns an example string representation of the custom configuration.
func (c customConfig) String() string {
	return fmt.Sprintf("%s:%s", c.Auth.Username, c.Auth.Password)
}

func main() {
	// Substation applications rely on a context for cancellation and timeouts.
	ctx := context.Background()

	// Define and load the custom configuration. This config includes a username
	// and password for authentication.
	conf := []byte(`
		{
			"transforms":[
				{"type":"object_copy","settings":{"object":{"key":"a","set_key":"c"}}},
				{"type":"send_stdout"}
			],
			"auth":{
				"username":"foo",
				"password":"bar"
			}
		}
	`)

	cfg := customConfig{}
	if err := json.Unmarshal(conf, &cfg); err != nil {
		// Handle error.
		panic(err)
	}

	// Create a new Substation instance from the embedded configuration.
	sub, err := sub.New(ctx, cfg.Config)
	if err != nil {
		// Handle error.
		panic(err)
	}

	// Print the Substation configuration.
	fmt.Println(sub)

	// Print the custom configuration.
	fmt.Println(cfg)

}

Output:

{"transforms":[{"type":"object_copy","settings":{"object":{"key":"a","set_key":"c"}}},{"type":"send_stdout","settings":null}]}
foo:bar

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// Transforms contains a list of data transformatons that are executed.
	Transforms []config.Config `json:"transforms"`
}

Config is the core configuration for the application. Custom applications should embed this and add additional configuration options.

type Substation 

type Substation struct {
	// contains filtered or unexported fields
}

Substation provides access to data transformation functions.

Example 
package main

import (
	"context"
	"encoding/json"
	"fmt"

	sub "github.com/brexhq/substation"
	"github.com/brexhq/substation/message"
	"github.com/brexhq/substation/transform"
)

func main() {
	// Substation applications rely on a context for cancellation and timeouts.
	ctx := context.Background()

	// Define a configuration. For native Substation applications, this is managed by Jsonnet.
	//
	// This example copies an object's value and prints the data to stdout.
	conf := []byte(`
		{
			"transforms":[
				{"type":"object_copy","settings":{"object":{"key":"a","set_key":"c"}}},
				{"type":"send_stdout"}
			]
		}
	`)

	cfg := sub.Config{}
	if err := json.Unmarshal(conf, &cfg); err != nil {
		// Handle error.
		panic(err)
	}

	// Create a new Substation instance.
	station, err := sub.New(ctx, cfg)
	if err != nil {
		// Handle error.
		panic(err)
	}

	// Print the Substation configuration.
	fmt.Println(station)

	// Substation instances process data defined as a Message. Messages can be processed
	// individually or in groups. This example processes multiple messages as a group.
	var msgs []*message.Message

	// Create a data Message and add it to the group.
	b := []byte(`{"a":"b"}`)
	data := message.New().SetData(b)
	msgs = append(msgs, data)

	// Create a control Message and add it to the group. Control messages trigger
	// special behavior in transforms. For example, a control message may be used
	// to flush buffered data or write a file to disk.
	ctrl := message.New(message.AsControl())
	msgs = append(msgs, ctrl)

	for _, msg := range msgs {
		// Transform the group of messages. In this example, results are discarded.
		if _, err := transform.Apply(ctx, station.Transforms(), msg); err != nil {
			// Handle error.
			panic(err)
		}
	}

}

Output:

{"transforms":[{"type":"object_copy","settings":{"object":{"key":"a","set_key":"c"}}},{"type":"send_stdout","settings":null}]}
{"a":"b","c":"b"}

func New 

func New(ctx context.Context, cfg Config) (*Substation, error)

New returns a new Substation instance.

func (*Substation) String 

func (s *Substation) String() string

String returns a JSON representation of the configuration.

func (*Substation) Transforms 

func (s *Substation) Transforms() []transform.Transformer

Transforms returns the configured data transforms.

These are safe to use concurrently.

Source Files

View all Source files

Directories

Path Synopsis
cmd
aws/lambda/autoscaling command
aws/lambda/substation command
aws/lambda/validation command
client/file/substation command
development/benchmark/substation command
Benchmarks the performance of Substation by sending a configurable number of events through the system and reporting the total time taken, the number of events sent, the amount of data sent, and the rate of events and data sent per second.
 Benchmarks the performance of Substation by sending a configurable number of events through the system and reporting the total time taken, the number of events sent, the amount of data sent, and the rate of events and data sent per second.
Package condition provides functions for evaluating data.
 Package condition provides functions for evaluating data.
Package config provides structures for building configurations.
 Package config provides structures for building configurations.
examples
pkg/transform command
internal
aggregate
aws
aws/appconfig
package appconfig provides functions for interacting with AWS AppConfig.
 package appconfig provides functions for interacting with AWS AppConfig.
aws/cloudwatch
aws/dynamodb
aws/firehose
aws/kinesis
aws/lambda
aws/s3manager
package s3manager provides methods and functions for downloading and uploading objects in AWS S3.
 package s3manager provides methods and functions for downloading and uploading objects in AWS S3.
aws/secretsmanager
aws/sns
aws/sqs
base64
bufio
channel
config
package config provides configuration types and functions for Substation.
 package config provides configuration types and functions for Substation.
errors
file
package file provides functions that can be used to retrieve files from local and remote locations.
 package file provides functions that can be used to retrieve files from local and remote locations.
http
kv
log
Package log wraps logrus and provides global logging only debug logging should be used in condition/, process/, and internal/ to reduce the likelihood of corrupting output for apps debug and info logging can be used in cmd/
 Package log wraps logrus and provides global logging only debug logging should be used in condition/, process/, and internal/ to reduce the likelihood of corrupting output for apps debug and info logging can be used in cmd/
media
package media provides capabilities for inspecting the content of data and identifying its media (Multipurpose Internet Mail Extensions, MIME) type.
 package media provides capabilities for inspecting the content of data and identifying its media (Multipurpose Internet Mail Extensions, MIME) type.
metrics
secrets
Package secrets provides functions for retrieving local and remote secrets and interpolating them into configuration files.
 Package secrets provides functions for retrieving local and remote secrets and interpolating them into configuration files.
Package message provides functions for managing data used by conditions and transforms.
 Package message provides functions for managing data used by conditions and transforms.
Package transform provides functions for transforming data.
 Package transform provides functions for transforming data.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.