Documentation
¶
Overview ¶
Package traffic implements the request-recording subsystem (docs §10): an embedded go-mitmproxy proxy whose addon writes every target HTTP exchange into PostgreSQL index with content-addressed files for large bodies. Full capture, plaintext (no redaction), target HTTP only.
Index ¶
- Constants
- Variables
- func SeedToolMetas() []actool.CoreTool
- func ValidateProxyURL(raw string) (*url.URL, error)
- type ArchiveExchange
- type ArchiveSnapshot
- type EvidenceExchange
- type ExchangeMeta
- type HostCount
- type HostDeleteStage
- type PageQuery
- type Traffic
- func (t *Traffic) Backend() string
- func (t *Traffic) Blob(hash string) (*os.File, int64, error)
- func (t *Traffic) BlobRange(hash string, offset, length int64) (data []byte, total int64, err error)
- func (t *Traffic) CACertPath() string
- func (t *Traffic) Close() error
- func (t *Traffic) Count() (int, error)
- func (t *Traffic) DB() *sql.DB
- func (t *Traffic) DeleteAll() (deleted int64, reclaimed int64, err error)
- func (t *Traffic) DeleteHost(host string) (int64, error)
- func (t *Traffic) DeleteHostsExact(hosts []string) (int64, error)
- func (t *Traffic) ExportHosts(hosts []string, dir string) (int64, error)
- func (t *Traffic) Get(id string) (req, resp string, err error)
- func (t *Traffic) Hosts() ([]HostCount, error)
- func (t *Traffic) ImportArchive(dir string) (int64, error)
- func (t *Traffic) Page(f PageQuery, page, size int) (rows []ExchangeMeta, total int, err error)
- func (t *Traffic) ProxyAddr() string
- func (t *Traffic) ReadEvidence(ctx context.Context, ids []string, consume func(EvidenceExchange) error) error
- func (t *Traffic) RecoverHostDeleteStages(archiveCommitted func(int64, int64) (bool, error)) error
- func (t *Traffic) Search(host string, page, size int) ([]ExchangeMeta, error)
- func (t *Traffic) SetUpstreamProxy(raw string) error
- func (t *Traffic) StageDeleteHostsExact(hosts []string) (*HostDeleteStage, error)
- func (t *Traffic) StageDeleteHostsExactForArchive(hosts []string, archiveID, taskID int64) (*HostDeleteStage, error)
- func (t *Traffic) Start() error
- func (t *Traffic) Tools() []actool.CoreTool
Constants ¶
const TrafficSearchDescription = "" /* 666-byte string literal not displayed */
TrafficSearchDescription is persisted into the tool catalog for new and upgraded installations. Keep it in the traffic package so the runtime tool and the catalog migration cannot drift apart.
Variables ¶
var ErrClosed = errors.New("traffic recorder is closed")
Functions ¶
func SeedToolMetas ¶
SeedToolMetas returns the traffic tools built on a ZERO receiver, for seeding the tools catalog (metadata only — Name/Description/InputSchema). The handlers close over the nil receiver but are never invoked on this instance, so it is safe.
func ValidateProxyURL ¶
ValidateProxyURL parses and checks a proxy URL (http/https/socks5, optional user:pass), returning the parsed URL. Exposed so callers can validate a global proxy before persisting it even when the traffic proxy itself is disabled.
Types ¶
type ArchiveExchange ¶
type ArchiveExchange struct {
ID string `json:"id"`
TS int64 `json:"ts"`
Host string `json:"host"`
Method string `json:"method"`
URLTemplate string `json:"url_template"`
URL string `json:"url"`
Status int `json:"status"`
ContentType string `json:"content_type"`
ReqLen int `json:"req_len"`
RespLen int `json:"resp_len"`
ReqHead string `json:"req_head"`
ReqBody []byte `json:"req_body,omitempty"`
ReqBlob string `json:"req_blob,omitempty"`
RespHead string `json:"resp_head"`
RespBody []byte `json:"resp_body,omitempty"`
RespBlob string `json:"resp_blob,omitempty"`
}
type ArchiveSnapshot ¶
type ArchiveSnapshot struct {
Version int `json:"version"`
Exchanges []ArchiveExchange `json:"exchanges"`
Blobs []string `json:"blobs"`
}
ArchiveSnapshot is the portable traffic subset embedded in one task archive. Large content-addressed bodies are copied alongside this manifest rather than base64-encoded, preserving deduplication and allowing the tar writer to stream.
type EvidenceExchange ¶
type EvidenceExchange struct {
ID, URL, Method, ContentType string
TS int64
Status int
ReqHead, RespHead string
ReqLen, RespLen int64
ReqHash, RespHash string
Request, Response io.Reader
}
EvidenceExchange exposes complete captured bodies only during the callback. Readers are closed before releasing the writer lock, so host deletion cannot remove a blob between its index lookup and its evidence copy.
type ExchangeMeta ¶
type ExchangeMeta struct {
ID string `json:"id"`
TS int64 `json:"ts"`
Host string `json:"host"`
Method string `json:"method"`
URLTemplate string `json:"url_template"`
URL string `json:"url"`
Status int `json:"status"`
ContentType string `json:"content_type"`
RespLen int `json:"resp_len"`
Path string `json:"path"`
}
ExchangeMeta is one row of the index (returned by Search).
type HostDeleteStage ¶
type HostDeleteStage struct {
// contains filtered or unexported fields
}
HostDeleteStage keeps the traffic delete transaction open so a task deletion can be coordinated with the business transaction: traffic is staged here and only committed once the caller's own transaction has succeeded. The Traffic write lock is held until Commit or Rollback, so no recorder can add rows or a blob reference for a host that is mid-deletion. Rolling back is just a transaction rollback now that nothing is moved on disk.
func (*HostDeleteStage) Commit ¶
func (s *HostDeleteStage) Commit() error
Commit makes the staged deletion permanent. The traffic transaction is committed only after the caller has committed its PostgreSQL task deletion.
func (*HostDeleteStage) Deleted ¶
func (s *HostDeleteStage) Deleted() int64
Deleted reports the number of exchange rows selected by this stage.
func (*HostDeleteStage) Rollback ¶
func (s *HostDeleteStage) Rollback() error
Rollback discards the staged deletion, leaving the traffic store untouched.
type PageQuery ¶
type PageQuery struct {
Host string // host substring
Method string // exact method (case-insensitive)
Query string // broad search box: metadata columns OR captured text
Body string // content box: captured request/response text only (full-text)
Path string // url_template substring — the vulnerability's path
Status string // exact code ("404") or class bucket ("4xx")
RespMin int64 // minimum resp_len, or -1 when unset
RespMax int64 // maximum resp_len, or -1 when unset
Sort string // ts | status | resp_len (default ts)
Order string // asc | desc (default desc)
}
PageQuery bundles the optional filters and sort order for Page. Every filter is combined with AND; an empty string skips that filter, and RespMin/RespMax of -1 skip the response-size bounds. An empty Sort/Order defaults to newest-first by timestamp.
type Traffic ¶
type Traffic struct {
// contains filtered or unexported fields
}
Traffic runs the recording proxy and owns the file tree + index.
func Open ¶
Open opens the legacy SQLite backend for migration tooling and benchmarks. The application uses OpenPostgres; new production captures never use SQLite.
func OpenPostgres ¶
OpenPostgres uses the application's PostgreSQL database and an isolated schema. A legacy SQLite file is imported once, before the recording proxy starts.
func (*Traffic) Backend ¶
Backend reports the active index engine, independently of file/blob storage.
func (*Traffic) Blob ¶
Blob opens a spilled body for streaming; the caller must close the file. Streaming matters here: the driver exposes no incremental BLOB API, so keeping large bodies on disk is what lets them be served without loading them whole.
func (*Traffic) BlobRange ¶
func (t *Traffic) BlobRange(hash string, offset, length int64) (data []byte, total int64, err error)
BlobRange reads at most length bytes of a blob from offset, and reports the blob's total size. Used by the agent tool to page through a large body without pulling all of it into the model's context.
func (*Traffic) CACertPath ¶
CACertPath returns the PEM CA cert clients must trust to verify HTTPS through the MITM proxy (go-mitmproxy writes it here on first start).
func (*Traffic) Close ¶
Close stops accepting proxy traffic and gives HTTP requests a bounded grace period to finish recording. After that no callback may start a file/DB write; admitted recordings and reclamation finish before the database is closed. Other application operations (archive/delete/read) must be drained by the owner before Close, just like HTTP API handlers before Manager.Close.
func (*Traffic) DeleteAll ¶
DeleteAll removes every recorded exchange — the page's clear-everything action. It differs from the host-scoped deletions in two ways. It sweeps host directories without consulting the index, because "clear everything" should leave nothing behind and a directory whose rows are already gone would otherwise survive. And it ends with a full compaction: VACUUM costs what it keeps, so an emptied index is the one moment it is free, and it is also the only way to switch on auto_vacuum for a database created without it.
Evidence already bound to a finding is untouched. Those bodies were copied into the evidence store when they were bound, precisely so that disposable traffic could be cleared without taking proof with it.
Returns the number of exchanges deleted and how many bytes of index the compaction handed back.
func (*Traffic) DeleteHost ¶
DeleteHost removes every recorded exchange whose host contains the given substring — mirroring the UI's host filter, so what you filtered is what gets deleted. All index rows share one database, so deletion is one transaction across the index, bodies, full-text index and blob references. Content-addressed blobs are shared across hosts and so are garbage-collected afterwards rather than deleted by host. Returns rows deleted.
func (*Traffic) DeleteHostsExact ¶
DeleteHostsExact removes recorded exchanges for a set of EXACT hosts (the batch path for the page's multi-select delete): index rows + each host's file tree, then one blob-GC pass. Exact match — unlike DeleteHost's substring — so picking "api.example.com" never sweeps "api.example.com.cn". Returns rows deleted. Duplicate hosts are harmless (idempotent deletes, single tree pass).
func (*Traffic) ExportHosts ¶
ExportHosts writes an exact-host snapshot to dir. It holds the traffic writer lock while reading the index and blobs, so each body and its index row come from one consistent point in time.
func (*Traffic) Get ¶
Get returns the full request/response text of one exchange. Bodies come from the database; rows recorded before bodies moved into SQLite carry a non-empty path and are read from the legacy on-disk tree instead, so history stays readable without being migrated.
func (*Traffic) Hosts ¶
Hosts returns distinct recorded hosts with exchange counts, most recent activity first — powers the page's target picker.
func (*Traffic) ImportArchive ¶
ImportArchive imports only missing exchange IDs. Current hot rows always win, and repeated restore attempts are safe after a partial external failure.
func (*Traffic) Page ¶
Page returns one page of exchange metadata for the traffic list, filtered by the criteria in f (all optional, combined with AND) and ordered by the requested column. Query is the broad search box (metadata columns OR, when the term is long enough for the trigram index, captured request/response text); Body narrows to exchanges whose captured text matches, via that same index. Also returns the total number of rows matching the filter (for the UI's pagination). Bodies are never included in the rows.
func (*Traffic) ProxyAddr ¶
ProxyAddr returns the address workers should set as HTTP(S)_PROXY. A bare ":port" means "bind all interfaces" (legacy default), so map it to loopback.
func (*Traffic) ReadEvidence ¶
func (*Traffic) RecoverHostDeleteStages ¶
RecoverHostDeleteStages resolves filesystem moves left by a process exit. The database rolls open transactions back on restart, while the supplied callback identifies task archives whose PostgreSQL compaction already committed.
func (*Traffic) Search ¶
func (t *Traffic) Search(host string, page, size int) ([]ExchangeMeta, error)
Search returns paged exchange metadata (never bodies).
func (*Traffic) SetUpstreamProxy ¶
SetUpstreamProxy points every captured request at a global egress proxy (http/https/socks5, optional user:pass in the URL). An empty raw string clears it, restoring direct dialing. The change is atomic and takes effect on the next connection — no restart, no proxy rebuild. go-mitmproxy dials all three schemes itself, so socks5 works uniformly here regardless of the target tool.
func (*Traffic) StageDeleteHostsExact ¶
func (t *Traffic) StageDeleteHostsExact(hosts []string) (*HostDeleteStage, error)
StageDeleteHostsExact prepares a reversible exact-host deletion. Callers must finish every successful stage with Commit or Rollback.
func (*Traffic) StageDeleteHostsExactForArchive ¶
func (t *Traffic) StageDeleteHostsExactForArchive(hosts []string, archiveID, taskID int64) (*HostDeleteStage, error)
StageDeleteHostsExactForArchive ties the reversible traffic deletion to a persistent task archive. Startup recovery uses archiveCommitted to decide whether an interrupted stage must be completed or rolled back.
Source Files