writers

package
v0.1.47 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2026 License: BSD-2-Clause Imports: 26 Imported by: 0

Documentation

Overview

Legacy Elasticsearch writer: it reproduces the flat, pre-restructure index layout — one main file index plus <index>_creds/_urls/_emails/_phone/_document sibling indices whose documents embed file_id/bucket — kept for datasets that were not migrated to the file/leak/reference model implemented in elastic.go.

Every package-level identifier here carries a Legacy marker so both writers can live in the same package; the formatting helpers (fmtInt, fmtRate, renderKVTable) are shared with elastic.go and intentionally not duplicated.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewElasticClient added in v0.1.47 

func NewElasticClient(uri string, debug bool) (*elk.Client, string, error)

NewElasticWriter returns a new Elasticsearch writer. When debug is true, operational logs are emitted at Info level; otherwise they are emitted at Debug level. NewElasticClient builds a client for the cluster addressed by uri and returns it together with the base index name the URI path carries (defaulting to "intelparser"). It only talks to the cluster to ping it, so it is also the way to reach an index that is being read rather than written -- nothing here creates or alters an index.

Types

type CsvWriter 

type CsvWriter struct {
	FilePath string
	// contains filtered or unexported fields
}

CsvWriter writes CSV files

func NewCsvWriter 

func NewCsvWriter(destination string) (*CsvWriter, error)

NewCsvWriter gets a new CsvWriter

func (*CsvWriter) Write 

func (cw *CsvWriter) Write(result *models.File) error

Write a CSV line

type DbWriter 

type DbWriter struct {
	URI         string
	ControlOnly bool

	ReadOnly bool
	// contains filtered or unexported fields
}

DbWriter is a Database writer

func NewDbWriter 

func NewDbWriter(uri string, debug bool) (*DbWriter, error)

NewDbWriter initialises a database writer

func (*DbWriter) Write 

func (dw *DbWriter) Write(result *models.File) error

Write results to the database

type ElasticLegacyWriter added in v0.1.39 

type ElasticLegacyWriter struct {
	Client *elk.Client
	Index  string
	// contains filtered or unexported fields
}

JsonWriter is a JSON lines writer

func NewElasticLegacyWriter added in v0.1.39 

func NewElasticLegacyWriter(uri string, debug bool) (*ElasticLegacyWriter, error)

NewElasticLegacyWriter returns a new Elasticsearch writer using the legacy index layout. When debug is true, operational logs are emitted at Info level; otherwise they are emitted at Debug level.

func (*ElasticLegacyWriter) CreateDoc added in v0.1.39 

func (ew *ElasticLegacyWriter) CreateDoc(index string, data []byte, doc_id string) error

func (*ElasticLegacyWriter) CreateDocBulk added in v0.1.39 

func (ew *ElasticLegacyWriter) CreateDocBulk(index string, docs map[string][]byte) error

func (*ElasticLegacyWriter) CreateIndex added in v0.1.39 

func (ew *ElasticLegacyWriter) CreateIndex(index string, mapping string) error

func (*ElasticLegacyWriter) Finalize added in v0.1.39 

func (ew *ElasticLegacyWriter) Finalize() error

Finalize renders a human-friendly end-of-run table with the writer's aggregated ingestion statistics. Safe to call after Flush.

func (*ElasticLegacyWriter) Flush added in v0.1.39 

func (ew *ElasticLegacyWriter) Flush() error

Flush closes the queue and waits for all in-flight writes to complete. Must be called once, after producers have stopped invoking Write.

func (*ElasticLegacyWriter) Marshal added in v0.1.39 

func (ew *ElasticLegacyWriter) Marshal(v any) ([]byte, error)

func (*ElasticLegacyWriter) MarshalAppend added in v0.1.39 

func (ew *ElasticLegacyWriter) MarshalAppend(marshalled []byte, new_data map[string]interface{}) ([]byte, error)

func (*ElasticLegacyWriter) Write added in v0.1.39 

func (ew *ElasticLegacyWriter) Write(result *models.File) error

Write enqueues the result for asynchronous ingestion by the worker pool. Errors on the async path are logged and counted; they are not returned here.

type ElasticWriter 

type ElasticWriter struct {
	Client *elk.Client
	Index  string
	// contains filtered or unexported fields
}

JsonWriter is a JSON lines writer

func NewElasticWriter 

func NewElasticWriter(uri string, debug bool) (*ElasticWriter, error)

func (*ElasticWriter) CreateDoc 

func (ew *ElasticWriter) CreateDoc(index string, data []byte, doc_id string) error

func (*ElasticWriter) CreateIndex 

func (ew *ElasticWriter) CreateIndex(index string, mapping string) error

func (*ElasticWriter) Finalize added in v0.1.35 

func (ew *ElasticWriter) Finalize() error

Finalize renders a human-friendly end-of-run table with the writer's aggregated ingestion statistics. Safe to call after Flush.

func (*ElasticWriter) Flush added in v0.1.35 

func (ew *ElasticWriter) Flush() error

Flush closes the queue and waits for all in-flight writes to complete. Must be called once, after producers have stopped invoking Write.

func (*ElasticWriter) Marshal 

func (ew *ElasticWriter) Marshal(v any) ([]byte, error)

func (*ElasticWriter) MarshalAppend 

func (ew *ElasticWriter) MarshalAppend(marshalled []byte, new_data map[string]interface{}) ([]byte, error)

func (*ElasticWriter) Write 

func (ew *ElasticWriter) Write(result *models.File) error

Write enqueues the result for asynchronous ingestion by the worker pool. Errors on the async path are logged and counted; they are not returned here.

type FinalizableWriter added in v0.1.35 

type FinalizableWriter interface {
	Finalize() error
}

FinalizableWriter is optionally implemented by writers that can report a final summary after all writes are flushed. The runner calls Finalize once per writer at the end of a run, after Flush. Used (for example) by the Elastic writer to print a table of server-side statistics.

type FlushableWriter added in v0.1.35 

type FlushableWriter interface {
	Flush() error
}

FlushableWriter is optionally implemented by writers that buffer work asynchronously. Callers should invoke Flush after all producers have stopped, to drain pending writes before shutdown.

type Interceptor added in v0.1.25 

type Interceptor struct {
	// contains filtered or unexported fields
}

func (Interceptor) RoundTrip added in v0.1.25 

func (i Interceptor) RoundTrip(req *http.Request) (*http.Response, error)

type JsonWriter 

type JsonWriter struct {
	FilePath string
}

JsonWriter is a JSON lines writer

func NewJsonWriter 

func NewJsonWriter(destination string) (*JsonWriter, error)

NewJsonWriter return a new Json lines writer

func (*JsonWriter) Write 

func (jw *JsonWriter) Write(result *models.File) error

Write JSON lines to a file

type LegacyInterceptor added in v0.1.39 

type LegacyInterceptor struct {
	// contains filtered or unexported fields
}

func (LegacyInterceptor) RoundTrip added in v0.1.39 

func (i LegacyInterceptor) RoundTrip(req *http.Request) (*http.Response, error)

type MemoryWriter 

type MemoryWriter struct {
	// contains filtered or unexported fields
}

MemoryWriter is a memory-based results queue with a maximum slot count

func NewMemoryWriter 

func NewMemoryWriter(slots int) (*MemoryWriter, error)

NewMemoryWriter initializes a MemoryWriter with the specified number of slots

func (*MemoryWriter) GetAllResults 

func (s *MemoryWriter) GetAllResults() []*models.File

GetAllResults returns a copy of all current results.

func (*MemoryWriter) GetFirst 

func (s *MemoryWriter) GetFirst() *models.File

GetFirst retrieves the oldest result in the MemoryWriter.

func (*MemoryWriter) GetLatest 

func (s *MemoryWriter) GetLatest() *models.File

GetLatest retrieves the most recently added result.

func (*MemoryWriter) Write 

func (s *MemoryWriter) Write(result *models.File) error

Write adds a new result to the MemoryWriter.

type NoneWriter 

type NoneWriter struct {
}

NoneWriter is a None writer

func NewNoneWriter 

func NewNoneWriter() (*NoneWriter, error)

NewNoneWriter initialises a none writer

func (*NoneWriter) Write 

func (s *NoneWriter) Write(result *models.File) error

Write does nothing

type StdoutWriter 

type StdoutWriter struct {
}

StdoutWriter is a Stdout writer

func NewStdoutWriter 

func NewStdoutWriter() (*StdoutWriter, error)

NewStdoutWriter initialises a stdout writer

func (*StdoutWriter) Write 

func (s *StdoutWriter) Write(result *models.File) error

Write results to stdout

type Writer 

type Writer interface {
	Write(*models.File) error
}

Writer is a results writer

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.