Documentation
¶
Overview ¶
Package collector provides a mechanism to collect network packets from a network interface on macOS, linux and windows
Index ¶
- Variables
- func IsPcap(file string) (bool, error)
- func OpenPCAP(file string) (*pcapgo.Reader, *os.File, error)
- func ResetHeaderPrinted()
- type BatchInfo
- type Collector
- func (c *Collector) CloseFileHandleOnShutdown(f *os.File)
- func (c *Collector) CollectBPF(path, bpf string) error
- func (c *Collector) CollectLive(i string, bpf string, ctx context.Context) error
- func (c *Collector) CollectPcap(path string) error
- func (c *Collector) CollectPcapNG(path string) error
- func (c *Collector) EvaluateRules(record types.AuditRecord)
- func (c *Collector) EvaluateRulesWithMetrics(record types.AuditRecord) int
- func (c *Collector) FlushAssemblers()
- func (c *Collector) GetAlertCount() int64
- func (c *Collector) GetCurrentPacketCount() int64
- func (c *Collector) GetFilteredCount() int64
- func (c *Collector) GetNumPackets() int64
- func (c *Collector) GetPacketsPerSecond() int64
- func (c *Collector) GetProfilesCount() int
- func (c *Collector) GetServicesCount() int
- func (c *Collector) GetTotalAuditRecords() int64
- func (c *Collector) GetTotalBytesWritten() int64
- func (c *Collector) GetTotalPacketCount() int64
- func (c *Collector) Init() (err error)
- func (c *Collector) InitBatching(bpf string, in string) ([]BatchInfo, *pcap.Handle, error)
- func (c *Collector) PrintConfiguration()
- func (c *Collector) ReloadRulesEngine() error
- func (c *Collector) RenderPacketsPerSecond(inputFile string, out string)
- func (c *Collector) SetFilterExpression(expression string, recordType types.Type) error
- func (c *Collector) SetLogLevel(debug bool)
- func (c *Collector) SetRulesEngine(engine *rules.Engine)
- func (c *Collector) ShouldWriteRecord(record types.AuditRecord) bool
- func (c *Collector) Stop()
- func (c *Collector) WrapWritersWithFiltering()
- type Config
- type FilteringWriter
Constants ¶
This section is empty.
Variables ¶
var DefaultConfig = Config{ Workers: 1000, PacketBufferSize: 100, WriteUnknownPackets: false, Promisc: false, SnapLen: defaults.SnapLen, DPI: false, BaseLayer: utils.GetBaseLayer("ethernet"), DecodeOptions: utils.GetDecodeOptions("default"), DecoderConfig: config.DefaultConfig, ResolverConfig: resolvers.DefaultConfig, Timeout: pcap.BlockForever, LogErrors: false, }
DefaultConfig is a sane example configuration.
var DefaultConfigDPI = Config{ Workers: 1000, PacketBufferSize: 100, WriteUnknownPackets: false, Promisc: false, SnapLen: defaults.SnapLen, DPI: true, BaseLayer: utils.GetBaseLayer("ethernet"), DecodeOptions: utils.GetDecodeOptions("default"), DecoderConfig: config.DefaultConfig, ResolverConfig: resolvers.DefaultConfig, LogErrors: false, }
DefaultConfigDPI is a sane example configuration for use with Deep Packet Inspection.
Functions ¶
func ResetHeaderPrinted ¶ added in v0.7.5
func ResetHeaderPrinted()
ResetHeaderPrinted resets the flag that tracks whether the netcap header has been printed. This is useful when starting a new batch of processing or in testing scenarios.
Types ¶
type Collector ¶
type Collector struct {
InputFile string
PrintTime bool
Bpf string
Epochs int
// contains filtered or unexported fields
}
Collector provides an interface to collect data from PCAP or a network interface. this structure has an optimized field order to avoid excessive padding.
func (*Collector) CloseFileHandleOnShutdown ¶ added in v0.6.6
CloseFileHandleOnShutdown allows to register file handles for close on shutdown.
func (*Collector) CollectBPF ¶
CollectBPF open the named PCAP file and sets the specified BPF filter.
func (*Collector) CollectLive ¶
CollectLive starts collection of data from the given interface. optionally a BPF can be supplied. this is the linux version that uses the pure go version from pcapgo to fetch packets live.
func (*Collector) CollectPcap ¶
CollectPcap implements parallel decoding of incoming packets.
func (*Collector) CollectPcapNG ¶
CollectPcapNG implements parallel decoding of incoming packets.
func (*Collector) EvaluateRules ¶ added in v0.9.0
func (c *Collector) EvaluateRules(record types.AuditRecord)
EvaluateRules evaluates all rules against a record and generates alerts if matched.
func (*Collector) EvaluateRulesWithMetrics ¶ added in v0.9.0
func (c *Collector) EvaluateRulesWithMetrics(record types.AuditRecord) int
EvaluateRulesWithMetrics evaluates rules and returns the number of alerts generated. This is used by the FilteringWriter to track metrics.
func (*Collector) FlushAssemblers ¶ added in v0.7.6
func (c *Collector) FlushAssemblers()
FlushAssemblers flushes all TCP assemblers to release their pageCaches This is critical for multi-file processing to prevent unbounded memory growth PageCaches grow to handle traffic and NEVER SHRINK, causing memory leaks
func (*Collector) GetAlertCount ¶ added in v0.9.0
GetAlertCount returns the number of alerts generated.
func (*Collector) GetCurrentPacketCount ¶ added in v0.9.0
GetCurrentPacketCount returns the current packet count (for live statistics)
func (*Collector) GetFilteredCount ¶ added in v0.9.0
GetFilteredCount returns the number of records filtered out.
func (*Collector) GetNumPackets ¶ added in v0.4.0
GetNumPackets returns the current number of processed packets.
func (*Collector) GetPacketsPerSecond ¶ added in v0.9.0
GetPacketsPerSecond returns the current packets per second rate (for live statistics)
func (*Collector) GetProfilesCount ¶ added in v0.9.0
GetProfilesCount returns the current number of device profiles (for live statistics)
func (*Collector) GetServicesCount ¶ added in v0.9.0
GetServicesCount returns the current number of services (for live statistics)
func (*Collector) GetTotalAuditRecords ¶ added in v0.7.6
GetTotalAuditRecords returns the total number of audit records generated.
func (*Collector) GetTotalBytesWritten ¶ added in v0.7.6
GetTotalBytesWritten returns the total bytes written to disk.
func (*Collector) GetTotalPacketCount ¶ added in v0.9.0
GetTotalPacketCount returns the total packet count (for live statistics)
func (*Collector) Init ¶
Init sets up the collector and starts the configured number of workers must be called prior to usage of the collector instance.
func (*Collector) InitBatching ¶
InitBatching initializes batching mode and returns an array of Batchinfos and the pcap handle closing the handle must be done by the caller.
func (*Collector) PrintConfiguration ¶ added in v0.4.5
func (c *Collector) PrintConfiguration()
PrintConfiguration dumps the current collector config to stdout.
func (*Collector) ReloadRulesEngine ¶ added in v0.9.0
ReloadRulesEngine reloads the rules engine configuration from disk. This is used when rules are updated via the webUI at runtime.
func (*Collector) RenderPacketsPerSecond ¶ added in v0.6.6
RenderPacketsPerSecond will render a html chart for the packet ingestion rate of the collector over time. Do not call while the collector is running, the access to the pps map is not synchronized. If you need runtime metrics, use prometheus.
func (*Collector) SetFilterExpression ¶ added in v0.9.0
SetFilterExpression sets a filter expression for a specific record type.
func (*Collector) SetLogLevel ¶ added in v0.9.0
SetLogLevel updates the log level for all zap loggers at runtime. This allows enabling/disabling debug logging without restarting.
func (*Collector) SetRulesEngine ¶ added in v0.9.0
SetRulesEngine sets the rules engine for alert generation.
func (*Collector) ShouldWriteRecord ¶ added in v0.9.0
func (c *Collector) ShouldWriteRecord(record types.AuditRecord) bool
ShouldWriteRecord checks if a record should be written based on the filter. Returns true if the record passes the filter (or no filter is set).
func (*Collector) Stop ¶ added in v0.6.6
func (c *Collector) Stop()
Stop will halt packet collection and wait for all processing to finish.
func (*Collector) WrapWritersWithFiltering ¶ added in v0.9.0
func (c *Collector) WrapWritersWithFiltering()
WrapWritersWithFiltering wraps all decoder writers with FilteringWriters. This should be called after decoders are initialized and before processing starts.
type Config ¶
type Config struct {
// Decoder configuration
DecoderConfig *config.Config
// Baselayer to start decoding from
BaseLayer gopacket.LayerType
// Number of workers to use
Workers int
// Size of the input buffer channels for the workers
PacketBufferSize int
// Ethernet frame snaplength for live capture
SnapLen int
// Can be used to periodically free OS memory
FreeOSMem int
// Permissions for output directory
OutDirPermission os.FileMode
// Attach in promiscuous mode for live capture
Promisc bool
// Controls whether packets that had an unknown layer will get written into a separate file
WriteUnknownPackets bool
// Resolver configuration
ResolverConfig resolvers.Config
// Decoding options for gopacket
DecodeOptions gopacket.DecodeOptions
// Enable deep packet inspection
DPI bool
// DPI modules to use (comma-separated: lpi, ndpi, go)
// If empty and DPI is enabled, all modules will be used
DPIModules string
// Use TCP reassembly
ReassembleConnections bool
// LogErrors will log verbose packet decoding errors into the errors.log file
LogErrors bool
// NoPrompt will disable all human interaction prompts
NoPrompt bool
// HTTPShutdownEndpoint will run a HTTP service on localhost:60589
// sending a GET request there can be used to trigger teardown and audit record flushing
// which can be used as alternative to using OS signals
HTTPShutdownEndpoint bool
// NoSignalHandling disables signal handling in the collector
// This is useful when running in service mode where the parent process handles signals
NoSignalHandling bool
// Timeout for live capture
// if you set this to 0, the pcap.BlockForever option will be used
// From the macOS docs on libpcap:
// The read timeout is used to arrange that the read not necessarily return
// immediately when a packet is seen, but that it wait for some amount of time
// to allow more packets to arrive and to read multiple packets from the OS
// kernel in one operation.
Timeout time.Duration
// Labels is a filesystem path to the labels file on disk
// that contains the attack mappings
Labels string
// Generate scatter chart for the applied labels during labeling.
Scatter bool
// ScatterDuration is the interval for data used in the scatter plot.
ScatterDuration time.Duration
// LiveFlushInterval is the interval at which audit records are flushed during live capture.
// This makes data visible while capture is ongoing.
// If zero, no periodic flushing is performed (records only flushed on shutdown).
// Recommended: 30s to 60s for most use cases.
LiveFlushInterval time.Duration
}
Config contains configuration parameters for the Collector instance. this structure has an optimized field order to avoid excessive padding.
type FilteringWriter ¶ added in v0.9.0
type FilteringWriter struct {
// contains filtered or unexported fields
}
FilteringWriter wraps an AuditRecordWriter and applies filtering and rules before writing.
func NewFilteringWriter ¶ added in v0.9.0
func NewFilteringWriter(underlying netio.AuditRecordWriter, collector *Collector) *FilteringWriter
NewFilteringWriter creates a new filtering writer that wraps an underlying writer.
func (*FilteringWriter) Close ¶ added in v0.9.0
func (fw *FilteringWriter) Close(numRecords int64) (name string, size int64)
Close closes the underlying writer.
func (*FilteringWriter) Flush ¶ added in v0.9.0
func (fw *FilteringWriter) Flush() error
Flush flushes the underlying writer's buffer.
func (*FilteringWriter) Write ¶ added in v0.9.0
func (fw *FilteringWriter) Write(msg proto.Message) error
Write applies filtering and rules evaluation before writing the record.
func (*FilteringWriter) WriteHeader ¶ added in v0.9.0
func (fw *FilteringWriter) WriteHeader(t types.Type) error
WriteHeader writes the header to the underlying writer.