Documentation
¶
Overview ¶
Package export provides Writer implementations for exporting scraped items to various formats and destinations.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CSVWriter ¶
type CSVWriter struct {
// contains filtered or unexported fields
}
CSVWriter exports items to a CSV file. It implements the foxhound.Writer interface.
If headers are provided at construction they are used as the column order. If no headers are provided, they are inferred (sorted alphabetically) from the first item written. Missing field values are written as empty strings.
func NewCSV ¶
NewCSV opens path for writing and returns a CSVWriter. Provide explicit headers to fix the column order; omit them to infer from the first item written. Returns an error if the file cannot be created.
type JSONFormat ¶
type JSONFormat int
JSONFormat selects the output format for JSONWriter.
const ( // JSONArray writes a single JSON array containing all items. // The array is opened on creation and closed on Close(). JSONArray JSONFormat = iota // JSONLines writes one JSON object per line (NDJSON / JSON Lines format). JSONLines )
type JSONWriter ¶
type JSONWriter struct {
// contains filtered or unexported fields
}
JSONWriter exports items to a JSON or JSON Lines file. It implements the foxhound.Writer interface.
func NewJSON ¶
func NewJSON(path string, format JSONFormat) (*JSONWriter, error)
NewJSON opens path for writing and returns a JSONWriter. For JSONArray format the opening bracket is written immediately. Returns an error if the file cannot be created.
func (*JSONWriter) Close ¶
func (w *JSONWriter) Close() error
Close finalises the output file. For JSONArray it writes the closing bracket. Always closes the underlying file handle.
type MarkdownFormat ¶
type MarkdownFormat int
MarkdownFormat selects the output style for MarkdownWriter.
const ( // MarkdownTable writes all items as rows in a single pipe-delimited table. // Headers are collected from the first item and written once. MarkdownTable MarkdownFormat = iota // MarkdownList writes each item as a single bullet line. // The first (alphabetically sorted) field is bolded; the rest are // dash-separated. MarkdownList // MarkdownCards writes each item as a level-2 heading (first field) // followed by bold-key bullet points for remaining fields. MarkdownCards )
type MarkdownWriter ¶
type MarkdownWriter struct {
// contains filtered or unexported fields
}
MarkdownWriter exports scraped items to a Markdown file. It implements the foxhound.Writer interface.
func NewMarkdown ¶
func NewMarkdown(path string, format MarkdownFormat) (*MarkdownWriter, error)
NewMarkdown opens path for writing and returns a MarkdownWriter. Returns an error if the file cannot be created.
func (*MarkdownWriter) Close ¶
func (w *MarkdownWriter) Close() error
Close closes the underlying file.
type PostgresOption ¶
type PostgresOption func(*PostgresWriter)
PostgresOption is a functional option for PostgresWriter.
func WithPGBatchSize ¶
func WithPGBatchSize(n int) PostgresOption
WithPGBatchSize sets how many items are buffered before an automatic flush. Values <= 0 are ignored and the default (100) is used.
func WithUpsert ¶
func WithUpsert(keyField string) PostgresOption
WithUpsert configures the writer to use INSERT ... ON CONFLICT DO UPDATE keyed on the JSONB field keyField. When set, a unique index on data->>'<keyField>' is created automatically.
type PostgresWriter ¶
type PostgresWriter struct {
// contains filtered or unexported fields
}
PostgresWriter exports items to a PostgreSQL table as JSONB documents. Each item's Fields map is serialised to a JSONB column called data. The table is created automatically on first use.
PostgresWriter is safe for concurrent use.
func NewPostgres ¶
func NewPostgres(connString, table string, opts ...PostgresOption) (*PostgresWriter, error)
NewPostgres opens a connection to connString, ensures the target table exists, and returns a PostgresWriter ready for use.
connString must be a lib/pq compatible connection string, e.g.:
"postgres://user:pass@localhost/mydb?sslmode=disable"
func (*PostgresWriter) Close ¶
func (w *PostgresWriter) Close() error
Close flushes any remaining buffered items and closes the database connection.
type SQLiteWriter ¶
type SQLiteWriter struct {
// contains filtered or unexported fields
}
SQLiteWriter exports scraped items to a SQLite database. It implements the foxhound.Writer interface.
The table is created automatically from the first item's fields. Subsequent items that contain new fields trigger ALTER TABLE ADD COLUMN so the schema grows incrementally.
func NewSQLite ¶
func NewSQLite(dbPath, table string) (*SQLiteWriter, error)
NewSQLite opens (or creates) the SQLite database at dbPath and returns a SQLiteWriter targeting the given table. Returns an error if the database cannot be opened.
func (*SQLiteWriter) Close ¶
func (w *SQLiteWriter) Close() error
Close closes the underlying database connection.
func (*SQLiteWriter) Flush ¶
func (w *SQLiteWriter) Flush(_ context.Context) error
Flush is a no-op for SQLite (each Write is immediately committed).
type TextFormat ¶
type TextFormat int
TextFormat selects the output style for TextWriter.
const ( // TextLines writes one item per line as space-separated key=value pairs. TextLines TextFormat = iota // TextPretty writes each item as a labelled block surrounded by separators. TextPretty )
type TextWriter ¶
type TextWriter struct {
// contains filtered or unexported fields
}
TextWriter exports scraped items to a plain-text file. It implements the foxhound.Writer interface.
func NewText ¶
func NewText(path string, format TextFormat) (*TextWriter, error)
NewText opens path for writing and returns a TextWriter. Returns an error if the file cannot be created.
type WebhookOption ¶
type WebhookOption func(*WebhookWriter)
WebhookOption is a functional option for NewWebhook.
func WithAuth ¶
func WithAuth(authType, token string) WebhookOption
WithAuth sets the Authorization header sent with every request. authType is the scheme (e.g. "Bearer", "Basic") and token is the credential.
func WithBatchSize ¶
func WithBatchSize(n int) WebhookOption
WithBatchSize sets the number of items accumulated before an automatic flush. Defaults to 1 (flush after every item).
func WithRetry ¶
func WithRetry(n int) WebhookOption
WithRetry sets how many times a failed POST is retried before giving up.
type WebhookWriter ¶
type WebhookWriter struct {
// contains filtered or unexported fields
}
WebhookWriter sends items as JSON POST requests to a webhook URL. Items are buffered and sent in batches when the buffer reaches batchSize, on an explicit Flush, or on Close.
WebhookWriter is safe for concurrent use.
func NewWebhook ¶
func NewWebhook(url string, opts ...WebhookOption) *WebhookWriter
NewWebhook creates a WebhookWriter that POSTs items to url. Default batch size is 1; default retries is 0 (no retries).
func (*WebhookWriter) Close ¶
func (w *WebhookWriter) Close() error
Close flushes any remaining buffered items and releases resources.
type XMLWriter ¶
type XMLWriter struct {
// contains filtered or unexported fields
}
XMLWriter exports scraped items to an XML file. It implements the foxhound.Writer interface.
The output format is:
<?xml version="1.0" encoding="UTF-8"?>
<rootElement>
<itemElement>
<fieldName>value</fieldName>
...
</itemElement>
</rootElement>
func NewXML ¶
NewXML opens path for writing and returns an XMLWriter. rootElement defaults to "items"; itemElement defaults to "item" when blank. Returns an error if the file cannot be created.
Source Files