store

package
v0.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 7, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrOastNotFound is returned when an OAST session record does not exist.
	ErrOastNotFound = errors.New("oast session not found")
	// ErrOastLabelExists is returned when creating a session with an in-use label.
	ErrOastLabelExists = errors.New("oast label already in use")
)
View Source 
var ErrClosed = errors.New("storage closed")

Functions

func Deserialize 

func Deserialize(data []byte, v any) error

Deserialize decodes msgpack bytes into the provided value.

func Serialize 

func Serialize(v any) ([]byte, error)

Serialize encodes a value to msgpack bytes.

Types

type CrawlFlow added in v0.2.3 

type CrawlFlow struct {
	ID             string        `msgpack:"id"`   // short sectool ID
	SessionID      string        `msgpack:"sid"`  // parent session ID
	URL            string        `msgpack:"url"`  // full URL visited
	Host           string        `msgpack:"host"` // extracted from URL
	Path           string        `msgpack:"path"` // path with query string
	Method         string        `msgpack:"m"`
	FoundOn        string        `msgpack:"fo"` // parent URL where discovered
	Depth          int           `msgpack:"d"`  // crawl depth from seed
	StatusCode     int           `msgpack:"sc"` // HTTP response status
	ContentType    string        `msgpack:"ct"`
	ResponseLength int           `msgpack:"rl"`   // response body length in bytes
	Request        []byte        `msgpack:"req"`  // wire-format bytes
	Response       []byte        `msgpack:"resp"` // wire-format bytes
	Truncated      bool          `msgpack:"tr"`   // response exceeded max_body_bytes
	Duration       time.Duration `msgpack:"dur"`  // request/response round-trip
	DiscoveredAt   time.Time     `msgpack:"da"`
}

CrawlFlow is a single captured request/response from crawling; the canonical crawl flow record shared by the crawler backend and persisted per flow key.

type CrawlSessionData added in v0.2.3 

type CrawlSessionData struct {
	CrawlSessionInfo
	StartedAt       time.Time             `msgpack:"sa"`
	LastActivity    time.Time             `msgpack:"la"`
	FlowIDs         []string              `msgpack:"fids"` // discovery order
	Forms           []protocol.CrawlForm  `msgpack:"fm"`
	Errors          []protocol.CrawlError `msgpack:"er"`
	LastReturnedIdx int                   `msgpack:"lri,omitempty"` // "since last" list cursor
}

CrawlSessionData is the persisted metadata and result index for one crawl session. Flow bodies live under per-flow keys; this record keeps discovery order.

type CrawlSessionInfo added in v0.2.3 

type CrawlSessionInfo struct {
	ID        string    `msgpack:"id"` // short sectool ID
	Label     string    `msgpack:"lb"`
	CreatedAt time.Time `msgpack:"ca"`
	State     string    `msgpack:"st"` // "running", "stopped", "completed", "error"
}

CrawlSessionInfo is the identity and lifecycle metadata for a crawl session.

type CrawlStore added in v0.2.3 

type CrawlStore struct {
	// contains filtered or unexported fields
}

CrawlStore manages crawl sessions and their discovered results with write-through access to a Storage. Reads come from an in-memory mirror that is kept authoritative on every mutation.

func NewCrawlStore added in v0.2.3 

func NewCrawlStore(storage Storage) *CrawlStore

NewCrawlStore creates a new CrawlStore backed by the given storage.

func (*CrawlStore) AppendError added in v0.2.3 

func (s *CrawlStore) AppendError(sessionID string, crawlErr protocol.CrawlError) error

AppendError appends a discovered error to the session, rolling back the mirror on failure.

func (*CrawlStore) AppendFlow added in v0.2.3 

func (s *CrawlStore) AppendFlow(sessionID string, flow *CrawlFlow) error

AppendFlow appends a discovered flow to the session in discovery order. The flow body is persisted under its own key before the session record updates.

func (*CrawlStore) AppendForm added in v0.2.3 

func (s *CrawlStore) AppendForm(sessionID string, form protocol.CrawlForm) error

AppendForm appends a discovered form to the session, rolling back the mirror on failure.

func (*CrawlStore) Close added in v0.2.3 

func (s *CrawlStore) Close(_ context.Context) error

Close closes the underlying storage.

func (*CrawlStore) Count added in v0.2.3 

func (s *CrawlStore) Count() int

Count returns the number of stored sessions.

func (*CrawlStore) Create added in v0.2.3 

func (s *CrawlStore) Create(data *CrawlSessionData) error

Create persists a new session and its label reverse index. Fails if the ID or an equal label is already present. The mirror entry is rolled back if the write fails.

func (*CrawlStore) Delete added in v0.2.3 

func (s *CrawlStore) Delete(sessionID string) error

Delete removes a session and all of its stored results.

func (*CrawlStore) Errors added in v0.2.3 

func (s *CrawlStore) Errors(sessionID string) ([]protocol.CrawlError, bool)

Errors returns the discovered errors for a session.

func (*CrawlStore) Flows added in v0.2.3 

func (s *CrawlStore) Flows(sessionID string) ([]*CrawlFlow, bool)

Flows returns the discovered flows for a session in discovery order. The returned flows are clones; Request and Response bytes are shared with the store and must not be mutated.

func (*CrawlStore) Forms added in v0.2.3 

func (s *CrawlStore) Forms(sessionID string) ([]protocol.CrawlForm, bool)

Forms returns the discovered forms for a session.

func (*CrawlStore) GetFlow added in v0.2.3 

func (s *CrawlStore) GetFlow(flowID string) (*CrawlFlow, bool)

GetFlow returns a flow by its ID across all sessions. Hydrated sessions are answered from the mirror; storage is scanned for the rest.

func (*CrawlStore) GetSession added in v0.2.3 

func (s *CrawlStore) GetSession(identifier string) (*CrawlSessionData, bool)

GetSession resolves a session by ID or label and returns its stored metadata.

func (*CrawlStore) Sessions added in v0.2.3 

func (s *CrawlStore) Sessions() []*CrawlSessionData

Sessions returns all stored sessions in unspecified order.

func (*CrawlStore) UpdateCursor added in v0.2.3 

func (s *CrawlStore) UpdateCursor(sessionID string, idx int) error

UpdateCursor persists a session's "since last" list cursor without touching LastActivity.

func (*CrawlStore) UpdateState added in v0.2.3 

func (s *CrawlStore) UpdateState(sessionID string, state string) error

UpdateState persists a session's state, rolling back the mirror on failure.

type NoteListOptions added in v0.1.6 

type NoteListOptions struct {
	Type     string
	FlowIDs  []string
	Contains string
	AfterID  string
	Limit    int
}

NoteListOptions controls filtering and pagination for note listing.

type NoteMeta added in v0.1.6 

type NoteMeta struct {
	NoteID    string    `msgpack:"nid"`
	Type      string    `msgpack:"t"`
	FlowIDs   []string  `msgpack:"fids"`
	Content   string    `msgpack:"c"`
	CreatedAt time.Time `msgpack:"ca"`
	UpdatedAt time.Time `msgpack:"ua"`
}

NoteMeta holds metadata and content for a saved note.

type NoteStore added in v0.1.6 

type NoteStore struct {
	// contains filtered or unexported fields
}

NoteStore manages notes with thread-safe access and reverse index for flow lookups.

func NewNoteStore added in v0.1.6 

func NewNoteStore(storage Storage) *NoteStore

NewNoteStore creates a new NoteStore backed by the given storage.

func (*NoteStore) Close added in v0.1.6 

func (s *NoteStore) Close(_ context.Context) error

func (*NoteStore) Count added in v0.1.6 

func (s *NoteStore) Count() int

Count returns the number of stored notes.

func (*NoteStore) Create added in v0.1.17 

func (s *NoteStore) Create(note *NoteMeta) error

Create assigns a unique note ID, stores the note, and maintains the reverse flow index. Caller leaves NoteID empty.

func (*NoteStore) Delete added in v0.1.6 

func (s *NoteStore) Delete(noteID string) error

Delete removes a note and cleans up the reverse index.

func (*NoteStore) ForFlowIDs added in v0.1.6 

func (s *NoteStore) ForFlowIDs(flowIDs []string) map[string][]*NoteMeta

ForFlowIDs returns notes associated with the given flow IDs, grouped by flow_id.

func (*NoteStore) Get added in v0.1.6 

func (s *NoteStore) Get(noteID string) (*NoteMeta, bool)

Get retrieves a note by ID.

func (*NoteStore) List added in v0.1.6 

func (s *NoteStore) List(opts NoteListOptions) []*NoteMeta

List returns notes matching the given options, sorted by created_at.

func (*NoteStore) Save added in v0.1.6 

func (s *NoteStore) Save(note *NoteMeta) error

Save upserts a note by its existing NoteID and maintains the reverse flow index. Use Create to add a new note with a generated ID.

func (*NoteStore) SplitReferencedFlows added in v0.1.15 

func (s *NoteStore) SplitReferencedFlows(flowIDs []string) ([]string, []string)

SplitReferencedFlows splits the provided flows into referenced by at least one note (left / first) and unknown or unreferenced (right).

type OastEvent added in v0.2.3 

type OastEvent struct {
	ID        string                 `msgpack:"id"` // short sectool ID
	Time      time.Time              `msgpack:"t"`
	Type      string                 `msgpack:"ty"` // "dns", "http", "smtp", ...
	SourceIP  string                 `msgpack:"src,omitempty"`
	Subdomain string                 `msgpack:"sub,omitempty"` // full subdomain accessed
	Details   map[string]interface{} `msgpack:"det,omitempty"` // protocol-specific details
}

OastEvent is a captured out-of-band interaction; the canonical event record shared by the backend and persisted per sequence key.

type OastSessionData added in v0.2.3 

type OastSessionData struct {
	OastSessionInfo
	Events        []OastEvent `msgpack:"-"` // hydrated view, never serialized
	FirstEventSeq int         `msgpack:"fes"`
	EventCount    int         `msgpack:"ec"`
	DroppedCount  int         `msgpack:"dropped,omitempty"`
	LastPollIdx   int         `msgpack:"lpi,omitempty"` // "since last" poll cursor
}

OastSessionData is the persisted state for one OAST session. Events live under per-event keys; the record tracks the live range so appends never rewrite other events.

type OastSessionInfo added in v0.2.3 

type OastSessionInfo struct {
	ID             string    `msgpack:"id"`  // short sectool ID
	Domain         string    `msgpack:"dom"` // full interactsh domain
	Label          string    `msgpack:"lbl,omitempty"`
	RedirectTarget string    `msgpack:"rt,omitempty"` // URL to 307 redirect to; empty = none
	CreatedAt      time.Time `msgpack:"ca"`
}

OastSessionInfo holds OAST session metadata; the canonical identity shared by the backend and the persisted session record.

type OastStore added in v0.2.3 

type OastStore struct {
	// contains filtered or unexported fields
}

OastStore persists OAST sessions and their events through one Storage.

func NewOastStore added in v0.2.3 

func NewOastStore(storage Storage) *OastStore

NewOastStore creates an OastStore backed by the given storage.

func (*OastStore) AppendEvent added in v0.2.3 

func (s *OastStore) AppendEvent(id string, ev OastEvent, maxEvents int) (bool, error)

AppendEvent appends an event under its own key, trimming the oldest when the live count exceeds maxEvents. Returns true when a trim occurred so callers can shift an ephemeral cursor.

func (*OastStore) Close added in v0.2.3 

func (s *OastStore) Close() error

Close closes the underlying storage.

func (*OastStore) CreateSession added in v0.2.3 

func (s *OastStore) CreateSession(info OastSessionInfo) error

CreateSession persists a new empty session and its reverse indices. Fails if the label is already in use by another session.

func (*OastStore) Delete added in v0.2.3 

func (s *OastStore) Delete(id string) error

Delete removes a session record, its live events, and its reverse indices.

func (*OastStore) FindEvent added in v0.2.3 

func (s *OastStore) FindEvent(eventID string) (*OastEvent, bool)

FindEvent locates an event by ID across all sessions. Events are keyed by session and sequence, not ID, so this scans stored events; intended for rare single-event lookups.

func (*OastStore) Get added in v0.2.3 

func (s *OastStore) Get(id string) (*OastSessionData, bool)

Get returns a stored session with its live events by ID.

func (*OastStore) List added in v0.2.3 

func (s *OastStore) List() []*OastSessionData

List returns every stored session record with events hydrated, in unspecified order.

func (*OastStore) Resolve added in v0.2.3 

func (s *OastStore) Resolve(identifier string) (*OastSessionData, bool)

Resolve finds a session by ID, label, or domain. Events are not hydrated.

func (*OastStore) SessionByLabel added in v0.2.3 

func (s *OastStore) SessionByLabel(label string) (*OastSessionData, bool)

SessionByLabel returns a stored session whose label matches exactly. Events are not hydrated.

func (*OastStore) UpdatePollCursor added in v0.2.3 

func (s *OastStore) UpdatePollCursor(id string, idx int) error

UpdatePollCursor persists a session's "since last" poll cursor.

type Provider added in v0.1.15 

type Provider func(name string) (Storage, error)

Provider allocates a named Storage instance. Backends call this in their constructor and own Close on returned stores. Names are storage addresses: durable implementations must map a given name to the same data across restarts.

func PrefixedProvider added in v0.2.3 

func PrefixedProvider(base Storage) (Provider, func() error)

PrefixedProvider returns a Provider that serves every named store from one shared base Storage, namespacing each under "<name>:". The returned function closes the base; per-store Close calls are no-ops.

type ReplayHistoryEntry 

type ReplayHistoryEntry struct {
	ReplayHistoryMeta
	ReplayHistoryPayload
}

ReplayHistoryEntry stores a replay request/response. The meta and payload halves are embedded so each serializes independently: the meta key keeps summary/list paths from deserializing request/response bodies.

func (*ReplayHistoryEntry) Duration 

func (e *ReplayHistoryEntry) Duration() time.Duration

Duration reports the replay round-trip time, or zero if not yet completed.

type ReplayHistoryMeta 

type ReplayHistoryMeta struct {
	FlowID       string    `msgpack:"fid"`
	Method       string    `msgpack:"m"`
	Host         string    `msgpack:"h"`
	Path         string    `msgpack:"p"`
	Scheme       string    `msgpack:"sc,omitempty"` // "http" or "https"
	Port         int       `msgpack:"po,omitempty"` // original port (0 = infer from scheme)
	Protocol     string    `msgpack:"pr"`
	SourceFlowID string    `msgpack:"sf"`
	CreatedAt    time.Time `msgpack:"ca"`
	CompletedAt  time.Time `msgpack:"cp,omitempty"` // zero until the response is recorded
	RespStatus   int       `msgpack:"rs"`
	RespLen      int       `msgpack:"rl"`
	// Annotations carries sidecar-authored flow metadata; nil for native sends.
	Annotations map[string]any `msgpack:"an,omitempty"`
	// InvokedBy names the sidecar that originated a native send via invoke_adapter.
	InvokedBy string `msgpack:"ib,omitempty"`
	// Adapter names the sidecar that performed a replay; empty for native sends.
	Adapter string `msgpack:"ad,omitempty"`
}

ReplayHistoryMeta holds lightweight metadata for a replay entry. Used by summary/list paths to avoid deserializing full request/response bodies.

type ReplayHistoryPayload 

type ReplayHistoryPayload struct {
	RawRequest      []byte `msgpack:"rq"`
	ModifiedRequest []byte `msgpack:"mq,omitempty"` // post-rule request; nil if no rules applied
	RespHeaders     []byte `msgpack:"rh"`
	RespBody        []byte `msgpack:"rb"`
}

ReplayHistoryPayload holds the heavy request/response data for a replay entry.

type ReplayHistoryStore 

type ReplayHistoryStore struct {
	// contains filtered or unexported fields
}

ReplayHistoryStore manages replay entries with thread-safe access.

func NewReplayHistoryStore 

func NewReplayHistoryStore(storage Storage) *ReplayHistoryStore

NewReplayHistoryStore creates a new ReplayHistoryStore backed by the given storage.

func (*ReplayHistoryStore) Clear 

func (s *ReplayHistoryStore) Clear()

Clear removes all entries.

func (*ReplayHistoryStore) Close 

func (s *ReplayHistoryStore) Close(_ context.Context) error

func (*ReplayHistoryStore) Complete added in v0.2.0 

func (s *ReplayHistoryStore) Complete(flowID string, respHeaders, respBody []byte, status int, completedAt time.Time, annotations map[string]any) bool

Complete attaches a response to an already-stored replay entry: the two-phase form used for deferred and streaming replays. A non-zero completedAt marks the entry completed; annotations merge over existing keys. Returns false when flowID is unknown.

func (*ReplayHistoryStore) Count 

func (s *ReplayHistoryStore) Count() int

Count returns the number of stored replay entries.

func (*ReplayHistoryStore) Delete added in v0.1.15 

func (s *ReplayHistoryStore) Delete(flowIDs []string) int

Delete removes replay entries by flow_id. Idempotent; unknown ids are skipped. Returns the number of entries actually removed.

func (*ReplayHistoryStore) Get 

func (s *ReplayHistoryStore) Get(flowID string) (*ReplayHistoryEntry, bool)

Get retrieves a replay entry by flow_id.

func (*ReplayHistoryStore) LastFlowID added in v0.2.3 

func (s *ReplayHistoryStore) LastFlowID() (string, bool)

LastFlowID returns the persisted proxy_poll "since=last" flow cursor.

func (*ReplayHistoryStore) List 

func (s *ReplayHistoryStore) List() []*ReplayHistoryEntry

List returns all replay entries ordered by creation time.

func (*ReplayHistoryStore) ListMeta 

func (s *ReplayHistoryStore) ListMeta() []ReplayHistoryMeta

ListMeta returns lightweight metadata for all replay entries, ordered by creation time.

func (*ReplayHistoryStore) SetInvokedBy added in v0.2.0 

func (s *ReplayHistoryStore) SetInvokedBy(flowID, invokedBy string) bool

SetInvokedBy records the originating sidecar on an already-stored replay entry. Returns false when flowID is unknown.

func (*ReplayHistoryStore) SetLastFlowID added in v0.2.3 

func (s *ReplayHistoryStore) SetLastFlowID(flowID string) error

SetLastFlowID persists the proxy_poll "since=last" flow cursor.

func (*ReplayHistoryStore) Store 

func (s *ReplayHistoryStore) Store(entry *ReplayHistoryEntry)

Store adds a replay entry. Called after successful replay_send/request_send.

type SpillStoreConfig 

type SpillStoreConfig struct {
	Dir                 string  // optional, auto-created if empty
	FilePrefix          string  // optional prefix for data file name (e.g. "hist" -> "hist.spill.bin")
	MaxHotBytes         int64   // max value bytes in hot cache
	EvictTargetRatio    float64 // evict to this ratio, higher values evict less
	CompactionThreshold int64   // rewrite file when dead bytes exceed this
	ZSTDLevel           int     // compression level (1-21)
}

SpillStoreConfig configures spillStore behavior.

func DefaultSpillStoreConfig 

func DefaultSpillStoreConfig() SpillStoreConfig

DefaultSpillStoreConfig returns config with sensible defaults.

type Storage 

type Storage interface {
	Set(key string, blob []byte) error
	Get(key string) ([]byte, bool, error)
	// Keys returns all keys beginning with prefix, in unspecified order.
	Keys(prefix string) []string
	KeySet() []string
	Size() int
	Delete(key string) error
	DeleteAll() error
	Close() error
}

Storage defines the interface for key-value blob storage.

func MemProvider added in v0.1.15 

func MemProvider(string) (Storage, error)

MemProvider returns a fresh in-memory Storage for every name.

func NewMemStorage 

func NewMemStorage() Storage

NewMemStorage returns an in-memory Storage implementation.

func NewPrefixedStorage added in v0.2.3 

func NewPrefixedStorage(base Storage, prefix string) Storage

NewPrefixedStorage returns a Storage that namespaces every key under prefix within base, isolating stores that share one backing implementation. Close is a no-op; the caller owns the base store's lifetime.

func NewSpillStore 

func NewSpillStore(cfg SpillStoreConfig) (Storage, error)

NewSpillStore creates a new spillStore with the given config.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.