Documentation
¶
Overview ¶
Package importer brings history over from another tool so day one here isn't a zero dashboard. It holds the format mappers (jsonl, csv, posthog, mixpanel, umami) and the batching senders shared by the two import surfaces: the `smolanalytics import` CLI (batches POSTed to /v1/events) and the MCP import_events tool (batches ingested straight into the running server's store). One parser, one batcher — the two paths cannot drift.
Index ¶
- Variables
- func MapCSV(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapJSONL(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapMixpanel(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapPostHog(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapUmami(r io.Reader, emit EmitFn, skip SkipFn) error
- func MapperFor(format string) (func(io.Reader, EmitFn, SkipFn) error, error)
- type EmitFn
- type HTTPSender
- type IngestSender
- type Sender
- type SkipFn
- type Summary
Constants ¶
This section is empty.
Variables ¶
var BatchSize = 5000
BatchSize is events per batch — half the server's 10k batch cap. A var so tests can shrink it to exercise multi-batch sends.
Functions ¶
func MapCSV ¶
MapCSV reads a generic CSV: header row, a name (or event) column, a distinct_id (or user_id / anonymous_id) column, an optional time (or timestamp) column, and every other column lands as a string property.
func MapJSONL ¶
MapJSONL reads our own export format: one /v1/events-shaped JSON object per line (GET /v1/export?format=jsonl). Ids are kept, so re-importing is idempotent.
func MapMixpanel ¶ added in v0.9.0
MapMixpanel reads Mixpanel's Raw Event Export (JSONL): one object per line shaped
{"event":"Signed up","properties":{"time":1704067200,"distinct_id":"u1","$insert_id":"z",...}}
Unlike our own JSONL, the name/id/time live INSIDE properties and time is a unix stamp, so feeding a Mixpanel export to --format=jsonl silently drops every row (no top-level name). $insert_id becomes the event id, so re-importing the same export is idempotent.
func MapPostHog ¶
MapPostHog reads PostHog's events CSV export (Activity → Export). Properties travel either as one embedded-JSON "properties" column or flattened into "properties.$browser"-style columns — both land as event properties here.
func MapUmami ¶
MapUmami reads Umami's website_event CSV export. Rows without an event_name are pageviews → "$pageview" with url_path as the "path" property (the exact shape our web view reads). session_id becomes distinct_id: Umami keeps no stable cross-session visitor id, so user-level reports treat each session as a user.
Types ¶
type EmitFn ¶
EmitFn ships one mapped event; its error aborts the import (it means a send failed, not a bad row). SkipFn counts a row that couldn't be mapped.
type HTTPSender ¶
type HTTPSender struct {
// contains filtered or unexported fields
}
HTTPSender accumulates events and POSTs them to /v1/events, flushing on count or on approximate body size (the server rejects requests over 4MB). This is the CLI's sender; progress lines go to out.
func NewHTTPSender ¶
func NewHTTPSender(host, key string, out io.Writer) *HTTPSender
func (*HTTPSender) Add ¶
func (s *HTTPSender) Add(e event.Event) (int, error)
Add queues one event, flushing when the batch is full. Returns how many events that flush sent (0 when it only queued).
func (*HTTPSender) Flush ¶
func (s *HTTPSender) Flush() (int, error)
Flush POSTs the queued batch. A rejected batch aborts the import; batches already sent stay stored, so a re-run only avoids duplicates for the jsonl format (ids are preserved there and the server dedupes on id).
type IngestSender ¶
type IngestSender struct {
// contains filtered or unexported fields
}
IngestSender batches events for a direct in-process ingest — the same flush thresholds as the HTTP sender (batch count + approximate encoded size), minus the HTTP hop. The MCP import_events tool uses it to write the server's own store, so both import paths ship identical batches.
func NewIngestSender ¶
func NewIngestSender(ingest func([]event.Event) error) *IngestSender
func (*IngestSender) Flush ¶
func (s *IngestSender) Flush() (int, error)
type Sender ¶
Sender ships mapped events in batches. Add queues one event (flushing when the batch is full) and Flush ships whatever remains; both return how many events that call actually shipped.
type Summary ¶
type Summary struct {
Parsed int // rows mapped to events
Skipped map[string]int // rows dropped, counted per reason
Sent int // events actually shipped (0 on a dry run)
Preview []event.Event // the first 3 mapped events, for dry-run eyeballing
}
Summary is what one import run did — the exact counts, never estimates.
func Run ¶
Run parses src with the format's mapper and ships batches through send. On dryRun it parses and validates only — send is never called. A send error aborts the run (batches already shipped stay shipped); a bad row never does.
func (Summary) SkippedTotal ¶
SkippedTotal sums the per-reason skip counts.
Source Files