collector

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 9, 2026 License: BSD-3-Clause, GPL-3.0 Imports: 64 Imported by: 2

Documentation

Overview

Package collector provides a mechanism to collect network packets from a network interface on macOS, linux and windows

Index

Constants

This section is empty.

Variables

View Source
var DefaultConfig = Config{
	Workers:             1000,
	PacketBufferSize:    100,
	WriteUnknownPackets: false,
	Promisc:             false,
	SnapLen:             defaults.SnapLen,
	DPI:                 false,
	BaseLayer:           utils.GetBaseLayer("ethernet"),
	DecodeOptions:       utils.GetDecodeOptions("default"),
	DecoderConfig:       config.DefaultConfig,
	ResolverConfig:      resolvers.DefaultConfig,
	Timeout:             pcap.BlockForever,
	LogErrors:           false,
}

DefaultConfig is a sane example configuration.

View Source
var DefaultConfigDPI = Config{
	Workers:             1000,
	PacketBufferSize:    100,
	WriteUnknownPackets: false,
	Promisc:             false,
	SnapLen:             defaults.SnapLen,
	DPI:                 true,
	BaseLayer:           utils.GetBaseLayer("ethernet"),
	DecodeOptions:       utils.GetDecodeOptions("default"),
	DecoderConfig:       config.DefaultConfig,
	ResolverConfig:      resolvers.DefaultConfig,
	LogErrors:           false,
}

DefaultConfigDPI is a sane example configuration for use with Deep Packet Inspection.

Functions

func IsPcap

func IsPcap(file string) (bool, error)

IsPcap checks whether a file is a PCAP file.

func OpenPCAP added in v0.6.6

func OpenPCAP(file string) (*pcapgo.Reader, *os.File, error)

OpenPCAP opens a Packet Capture file.

func ResetHeaderPrinted added in v0.7.5

func ResetHeaderPrinted()

ResetHeaderPrinted resets the flag that tracks whether the netcap header has been printed. This is useful when starting a new batch of processing or in testing scenarios.

Types

type BatchInfo

type BatchInfo struct {
	Type types.Type
	Chan <-chan []byte
}

BatchInfo contains information about a Batch source.

type Collector

type Collector struct {
	InputFile string
	PrintTime bool
	Bpf       string

	Epochs int
	// contains filtered or unexported fields
}

Collector provides an interface to collect data from PCAP or a network interface. this structure has an optimized field order to avoid excessive padding.

func New

func New(config Config) *Collector

New returns a new Collector instance.

func (*Collector) CloseFileHandleOnShutdown added in v0.6.6

func (c *Collector) CloseFileHandleOnShutdown(f *os.File)

CloseFileHandleOnShutdown allows to register file handles for close on shutdown.

func (*Collector) CollectBPF

func (c *Collector) CollectBPF(path, bpf string) error

CollectBPF open the named PCAP file and sets the specified BPF filter.

func (*Collector) CollectLive

func (c *Collector) CollectLive(i string, bpf string, ctx context.Context) error

CollectLive starts collection of data from the given interface. optionally a BPF can be supplied. this is the linux version that uses the pure go version from pcapgo to fetch packets live.

func (*Collector) CollectPcap

func (c *Collector) CollectPcap(path string) error

CollectPcap implements parallel decoding of incoming packets.

func (*Collector) CollectPcapNG

func (c *Collector) CollectPcapNG(path string) error

CollectPcapNG implements parallel decoding of incoming packets.

func (*Collector) EvaluateRules added in v0.9.0

func (c *Collector) EvaluateRules(record types.AuditRecord)

EvaluateRules evaluates all rules against a record and generates alerts if matched.

func (*Collector) EvaluateRulesWithMetrics added in v0.9.0

func (c *Collector) EvaluateRulesWithMetrics(record types.AuditRecord) int

EvaluateRulesWithMetrics evaluates rules and returns the number of alerts generated. This is used by the FilteringWriter to track metrics.

func (*Collector) FlushAssemblers added in v0.7.6

func (c *Collector) FlushAssemblers()

FlushAssemblers flushes all TCP assemblers to release their pageCaches This is critical for multi-file processing to prevent unbounded memory growth PageCaches grow to handle traffic and NEVER SHRINK, causing memory leaks

func (*Collector) GetAlertCount added in v0.9.0

func (c *Collector) GetAlertCount() int64

GetAlertCount returns the number of alerts generated.

func (*Collector) GetCurrentPacketCount added in v0.9.0

func (c *Collector) GetCurrentPacketCount() int64

GetCurrentPacketCount returns the current packet count (for live statistics)

func (*Collector) GetFilteredCount added in v0.9.0

func (c *Collector) GetFilteredCount() int64

GetFilteredCount returns the number of records filtered out.

func (*Collector) GetNumPackets added in v0.4.0

func (c *Collector) GetNumPackets() int64

GetNumPackets returns the current number of processed packets.

func (*Collector) GetPacketsPerSecond added in v0.9.0

func (c *Collector) GetPacketsPerSecond() int64

GetPacketsPerSecond returns the current packets per second rate (for live statistics)

func (*Collector) GetProfilesCount added in v0.9.0

func (c *Collector) GetProfilesCount() int

GetProfilesCount returns the current number of device profiles (for live statistics)

func (*Collector) GetServicesCount added in v0.9.0

func (c *Collector) GetServicesCount() int

GetServicesCount returns the current number of services (for live statistics)

func (*Collector) GetTotalAuditRecords added in v0.7.6

func (c *Collector) GetTotalAuditRecords() int64

GetTotalAuditRecords returns the total number of audit records generated.

func (*Collector) GetTotalBytesWritten added in v0.7.6

func (c *Collector) GetTotalBytesWritten() int64

GetTotalBytesWritten returns the total bytes written to disk.

func (*Collector) GetTotalPacketCount added in v0.9.0

func (c *Collector) GetTotalPacketCount() int64

GetTotalPacketCount returns the total packet count (for live statistics)

func (*Collector) Init

func (c *Collector) Init() (err error)

Init sets up the collector and starts the configured number of workers must be called prior to usage of the collector instance.

func (*Collector) InitBatching

func (c *Collector) InitBatching(bpf string, in string) ([]BatchInfo, *pcap.Handle, error)

InitBatching initializes batching mode and returns an array of Batchinfos and the pcap handle closing the handle must be done by the caller.

func (*Collector) PrintConfiguration added in v0.4.5

func (c *Collector) PrintConfiguration()

PrintConfiguration dumps the current collector config to stdout.

func (*Collector) ReloadRulesEngine added in v0.9.0

func (c *Collector) ReloadRulesEngine() error

ReloadRulesEngine reloads the rules engine configuration from disk. This is used when rules are updated via the webUI at runtime.

func (*Collector) RenderPacketsPerSecond added in v0.6.6

func (c *Collector) RenderPacketsPerSecond(inputFile string, out string)

RenderPacketsPerSecond will render a html chart for the packet ingestion rate of the collector over time. Do not call while the collector is running, the access to the pps map is not synchronized. If you need runtime metrics, use prometheus.

func (*Collector) SetFilterExpression added in v0.9.0

func (c *Collector) SetFilterExpression(expression string, recordType types.Type) error

SetFilterExpression sets a filter expression for a specific record type.

func (*Collector) SetLogLevel added in v0.9.0

func (c *Collector) SetLogLevel(debug bool)

SetLogLevel updates the log level for all zap loggers at runtime. This allows enabling/disabling debug logging without restarting.

func (*Collector) SetRulesEngine added in v0.9.0

func (c *Collector) SetRulesEngine(engine *rules.Engine)

SetRulesEngine sets the rules engine for alert generation.

func (*Collector) ShouldWriteRecord added in v0.9.0

func (c *Collector) ShouldWriteRecord(record types.AuditRecord) bool

ShouldWriteRecord checks if a record should be written based on the filter. Returns true if the record passes the filter (or no filter is set).

func (*Collector) Stop added in v0.6.6

func (c *Collector) Stop()

Stop will halt packet collection and wait for all processing to finish.

func (*Collector) WrapWritersWithFiltering added in v0.9.0

func (c *Collector) WrapWritersWithFiltering()

WrapWritersWithFiltering wraps all decoder writers with FilteringWriters. This should be called after decoders are initialized and before processing starts.

type Config

type Config struct {

	// Decoder configuration
	DecoderConfig *config.Config

	// Baselayer to start decoding from
	BaseLayer gopacket.LayerType

	// Number of workers to use
	Workers int

	// Size of the input buffer channels for the workers
	PacketBufferSize int

	// Ethernet frame snaplength for live capture
	SnapLen int

	// Can be used to periodically free OS memory
	FreeOSMem int

	// Permissions for output directory
	OutDirPermission os.FileMode

	// Attach in promiscuous mode for live capture
	Promisc bool

	// Controls whether packets that had an unknown layer will get written into a separate file
	WriteUnknownPackets bool

	// Resolver configuration
	ResolverConfig resolvers.Config

	// Decoding options for gopacket
	DecodeOptions gopacket.DecodeOptions

	// Enable deep packet inspection
	DPI bool

	// DPI modules to use (comma-separated: lpi, ndpi, go)
	// If empty and DPI is enabled, all modules will be used
	DPIModules string

	// Use TCP reassembly
	ReassembleConnections bool

	// LogErrors will log verbose packet decoding errors into the errors.log file
	LogErrors bool

	// NoPrompt will disable all human interaction prompts
	NoPrompt bool

	// HTTPShutdownEndpoint will run a HTTP service on localhost:60589
	// sending a GET request there can be used to trigger teardown and audit record flushing
	// which can be used as alternative to using OS signals
	HTTPShutdownEndpoint bool

	// NoSignalHandling disables signal handling in the collector
	// This is useful when running in service mode where the parent process handles signals
	NoSignalHandling bool

	// Timeout for live capture
	// if you set this to 0, the pcap.BlockForever option will be used
	// From the macOS docs on libpcap:
	//   The read timeout is used to arrange that the read not necessarily return
	//   immediately when a packet is seen, but that it wait for some amount of time
	//   to allow more packets to arrive and to read multiple packets from the OS
	//   kernel in one operation.
	Timeout time.Duration

	// Labels is a filesystem path to the labels file on disk
	// that contains the attack mappings
	Labels string

	// Generate scatter chart for the applied labels during labeling.
	Scatter bool

	// ScatterDuration is the interval for data used in the scatter plot.
	ScatterDuration time.Duration

	// LiveFlushInterval is the interval at which audit records are flushed during live capture.
	// This makes data visible while capture is ongoing.
	// If zero, no periodic flushing is performed (records only flushed on shutdown).
	// Recommended: 30s to 60s for most use cases.
	LiveFlushInterval time.Duration
}

Config contains configuration parameters for the Collector instance. this structure has an optimized field order to avoid excessive padding.

type FilteringWriter added in v0.9.0

type FilteringWriter struct {
	// contains filtered or unexported fields
}

FilteringWriter wraps an AuditRecordWriter and applies filtering and rules before writing.

func NewFilteringWriter added in v0.9.0

func NewFilteringWriter(underlying netio.AuditRecordWriter, collector *Collector) *FilteringWriter

NewFilteringWriter creates a new filtering writer that wraps an underlying writer.

func (*FilteringWriter) Close added in v0.9.0

func (fw *FilteringWriter) Close(numRecords int64) (name string, size int64)

Close closes the underlying writer.

func (*FilteringWriter) Flush added in v0.9.0

func (fw *FilteringWriter) Flush() error

Flush flushes the underlying writer's buffer.

func (*FilteringWriter) Write added in v0.9.0

func (fw *FilteringWriter) Write(msg proto.Message) error

Write applies filtering and rules evaluation before writing the record.

func (*FilteringWriter) WriteHeader added in v0.9.0

func (fw *FilteringWriter) WriteHeader(t types.Type) error

WriteHeader writes the header to the underlying writer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL