Documentation
¶
Overview ¶
Package traffic implements the request-recording subsystem (docs §10): an embedded go-mitmproxy proxy whose addon writes every target HTTP exchange into a human-browsable file tree, with a sidecar SQLite index for paged queries. Full capture, plaintext (no redaction), target HTTP only.
Index ¶
- Constants
- func SeedToolMetas() []actool.CoreTool
- func ValidateProxyURL(raw string) (*url.URL, error)
- type ArchiveExchange
- type ArchiveSnapshot
- type EvidenceExchange
- type ExchangeMeta
- type HostCount
- type HostDeleteStage
- type PageQuery
- type PoolEntry
- type Traffic
- func (t *Traffic) Blob(hash string) (*os.File, int64, error)
- func (t *Traffic) BlobRange(hash string, offset, length int64) (data []byte, total int64, err error)
- func (t *Traffic) CACertPath() string
- func (t *Traffic) Close() error
- func (t *Traffic) Count() (int, error)
- func (t *Traffic) DB() *sql.DB
- func (t *Traffic) DeleteAll() (deleted int64, reclaimed int64, err error)
- func (t *Traffic) DeleteHost(host string) (int64, error)
- func (t *Traffic) DeleteHostsExact(hosts []string) (int64, error)
- func (t *Traffic) ExportHosts(hosts []string, dir string) (int64, error)
- func (t *Traffic) Get(id string) (req, resp string, err error)
- func (t *Traffic) Hosts() ([]HostCount, error)
- func (t *Traffic) ImportArchive(dir string) (int64, error)
- func (t *Traffic) Page(f PageQuery, page, size int) (rows []ExchangeMeta, total int, err error)
- func (t *Traffic) ProxyAddr() string
- func (t *Traffic) ProxyPoolEnabled() bool
- func (t *Traffic) ProxyPoolMarkBad(proxyRaw string, cooldown time.Duration) string
- func (t *Traffic) ProxyPoolReportBad(target string, cooldown time.Duration) string
- func (t *Traffic) ProxyPoolReportFail(proxyRaw string)
- func (t *Traffic) ProxyPoolReportOK(proxyRaw string)
- func (t *Traffic) ProxyPoolStatus() []PoolEntry
- func (t *Traffic) ReadEvidence(ctx context.Context, ids []string, consume func(EvidenceExchange) error) error
- func (t *Traffic) RecoverHostDeleteStages(archiveCommitted func(int64, int64) (bool, error)) error
- func (t *Traffic) Search(host string, page, size int) ([]ExchangeMeta, error)
- func (t *Traffic) SetProxyPool(items []PoolEntry, strategy string) error
- func (t *Traffic) SetUpstreamProxy(raw string) error
- func (t *Traffic) StageDeleteHostsExact(hosts []string) (*HostDeleteStage, error)
- func (t *Traffic) StageDeleteHostsExactForArchive(hosts []string, archiveID, taskID int64) (*HostDeleteStage, error)
- func (t *Traffic) Start() error
- func (t *Traffic) Tools() []actool.CoreTool
Constants ¶
const TrafficSearchDescription = "" /* 666-byte string literal not displayed */
TrafficSearchDescription is persisted into the tool catalog for new and upgraded installations. Keep it in the traffic package so the runtime tool and the catalog migration cannot drift apart.
Variables ¶
This section is empty.
Functions ¶
func SeedToolMetas ¶
SeedToolMetas returns the traffic tools built on a ZERO receiver, for seeding the tools catalog (metadata only — Name/Description/InputSchema). The handlers close over the nil receiver but are never invoked on this instance, so it is safe.
func ValidateProxyURL ¶
ValidateProxyURL parses and checks a proxy URL (http/https/socks5, optional user:pass), returning the parsed URL. Exposed so callers can validate a global proxy before persisting it even when the traffic proxy itself is disabled.
Types ¶
type ArchiveExchange ¶
type ArchiveExchange struct {
ID string `json:"id"`
TS int64 `json:"ts"`
Host string `json:"host"`
Method string `json:"method"`
URLTemplate string `json:"url_template"`
URL string `json:"url"`
Status int `json:"status"`
ContentType string `json:"content_type"`
ReqLen int `json:"req_len"`
RespLen int `json:"resp_len"`
ReqHead string `json:"req_head"`
ReqBody []byte `json:"req_body,omitempty"`
ReqBlob string `json:"req_blob,omitempty"`
RespHead string `json:"resp_head"`
RespBody []byte `json:"resp_body,omitempty"`
RespBlob string `json:"resp_blob,omitempty"`
}
type ArchiveSnapshot ¶
type ArchiveSnapshot struct {
Version int `json:"version"`
Exchanges []ArchiveExchange `json:"exchanges"`
Blobs []string `json:"blobs"`
}
ArchiveSnapshot is the portable traffic subset embedded in one task archive. Large content-addressed bodies are copied alongside this manifest rather than base64-encoded, preserving deduplication and allowing the tar writer to stream.
type EvidenceExchange ¶
type EvidenceExchange struct {
ID, URL, Method, ContentType string
TS int64
Status int
ReqHead, RespHead string
ReqLen, RespLen int64
ReqHash, RespHash string
Request, Response io.Reader
}
EvidenceExchange exposes complete captured bodies only during the callback. Readers are closed before releasing the writer lock, so host deletion cannot remove a blob between its index lookup and its evidence copy.
type ExchangeMeta ¶
type ExchangeMeta struct {
ID string `json:"id"`
TS int64 `json:"ts"`
Host string `json:"host"`
Method string `json:"method"`
URLTemplate string `json:"url_template"`
URL string `json:"url"`
Status int `json:"status"`
ContentType string `json:"content_type"`
RespLen int `json:"resp_len"`
Path string `json:"path"`
}
ExchangeMeta is one row of the index (returned by Search).
type HostDeleteStage ¶
type HostDeleteStage struct {
// contains filtered or unexported fields
}
HostDeleteStage keeps the SQLite delete transaction open so a task deletion can be coordinated with PostgreSQL: the traffic side is staged here and only committed once the caller's own transaction has succeeded. The Traffic write lock is held until Commit or Rollback, so no recorder can add rows or a blob reference for a host that is mid-deletion. Rolling back is just a transaction rollback now that nothing is moved on disk.
func (*HostDeleteStage) Commit ¶
func (s *HostDeleteStage) Commit() error
Commit makes the staged deletion permanent. SQLite is committed only after the caller has committed its PostgreSQL task deletion.
func (*HostDeleteStage) Deleted ¶
func (s *HostDeleteStage) Deleted() int64
Deleted reports the number of exchange rows selected by this stage.
func (*HostDeleteStage) Rollback ¶
func (s *HostDeleteStage) Rollback() error
Rollback discards the staged deletion, leaving the traffic store untouched.
type PageQuery ¶
type PageQuery struct {
Host string // host substring
Method string // exact method (case-insensitive)
Query string // broad search box: metadata columns OR captured text
Body string // content box: captured request/response text only (full-text)
Path string // url_template substring — the vulnerability's path
Status string // exact code ("404") or class bucket ("4xx")
RespMin int64 // minimum resp_len, or -1 when unset
RespMax int64 // maximum resp_len, or -1 when unset
Sort string // ts | status | resp_len (default ts)
Order string // asc | desc (default desc)
}
PageQuery bundles the optional filters and sort order for Page. Every filter is combined with AND; an empty string skips that filter, and RespMin/RespMax of -1 skip the response-size bounds. An empty Sort/Order defaults to newest-first by timestamp.
type PoolEntry ¶ added in v0.3.17
type PoolEntry struct {
URL string `json:"url"` // http/https/socks5,可含 user:pass(输出时掩码)
Name string `json:"name"` // 展示名
Enabled bool `json:"enabled"` // 手动开关
Weight int `json:"weight"` // 轮询权重(round_robin 有效,≥1)
Failures int `json:"failures"` // 连败计数(熔断窗口)
LastOK string `json:"last_ok,omitempty"`
LastFail string `json:"last_fail,omitempty"`
// BreakRemaining 是该上游当前熔断的剩余秒数(0=未熔断)。主动健康检查与
// 手动验证都会读写 last_ok/last_fail/break,页面据此筛选失效代理。
BreakRemaining int `json:"break_remaining,omitempty"`
}
PoolEntry 是池中的一个上游代理。
type Traffic ¶
type Traffic struct {
// contains filtered or unexported fields
}
Traffic runs the recording proxy and owns the file tree + index.
func (*Traffic) Blob ¶
Blob opens a spilled body for streaming; the caller must close the file. Streaming matters here: the driver exposes no incremental BLOB API, so keeping large bodies on disk is what lets them be served without loading them whole.
func (*Traffic) BlobRange ¶
func (t *Traffic) BlobRange(hash string, offset, length int64) (data []byte, total int64, err error)
BlobRange reads at most length bytes of a blob from offset, and reports the blob's total size. Used by the agent tool to page through a large body without pulling all of it into the model's context.
func (*Traffic) CACertPath ¶
CACertPath returns the PEM CA cert clients must trust to verify HTTPS through the MITM proxy (go-mitmproxy writes it here on first start).
func (*Traffic) Close ¶
Close waits for background tree reclamation to finish before closing the index, so shutdown never leaves a goroutine unlinking files out from under a removed data directory. Index-space reclamation is signalled to stop first: it holds a whole minutes-long budget, and finishing it is never worth delaying shutdown for — the next deletion resumes it.
func (*Traffic) DeleteAll ¶
DeleteAll removes every recorded exchange — the page's clear-everything action. It differs from the host-scoped deletions in two ways. It sweeps host directories without consulting the index, because "clear everything" should leave nothing behind and a directory whose rows are already gone would otherwise survive. And it ends with a full compaction: VACUUM costs what it keeps, so an emptied index is the one moment it is free, and it is also the only way to switch on auto_vacuum for a database created without it.
Evidence already bound to a finding is untouched. Those bodies were copied into the evidence store when they were bound, precisely so that disposable traffic could be cleared without taking proof with it.
Returns the number of exchanges deleted and how many bytes of index the compaction handed back.
func (*Traffic) DeleteHost ¶
DeleteHost removes every recorded exchange whose host contains the given substring — mirroring the UI's host filter, so what you filtered is what gets deleted. Everything lives in SQLite, so the deletion is one transaction across the index, bodies, full-text index and blob references. Content-addressed blobs are shared across hosts and so are garbage-collected afterwards rather than deleted by host. Returns rows deleted.
func (*Traffic) DeleteHostsExact ¶
DeleteHostsExact removes recorded exchanges for a set of EXACT hosts (the batch path for the page's multi-select delete): index rows + each host's file tree, then one blob-GC pass. Exact match — unlike DeleteHost's substring — so picking "api.example.com" never sweeps "api.example.com.cn". Returns rows deleted. Duplicate hosts are harmless (idempotent deletes, single tree pass).
func (*Traffic) ExportHosts ¶
ExportHosts writes an exact-host snapshot to dir. It holds the traffic writer lock while reading SQLite and blobs, so each body and its index row come from one consistent point in time.
func (*Traffic) Get ¶
Get returns the full request/response text of one exchange. Bodies come from the database; rows recorded before bodies moved into SQLite carry a non-empty path and are read from the legacy on-disk tree instead, so history stays readable without being migrated.
func (*Traffic) Hosts ¶
Hosts returns distinct recorded hosts with exchange counts, most recent activity first — powers the page's target picker.
func (*Traffic) ImportArchive ¶
ImportArchive imports only missing exchange IDs. Current hot rows always win, and repeated restore attempts are safe after a partial external failure.
func (*Traffic) Page ¶
Page returns one page of exchange metadata for the traffic list, filtered by the criteria in f (all optional, combined with AND) and ordered by the requested column. Query is the broad search box (metadata columns OR, when the term is long enough for the trigram index, captured request/response text); Body narrows to exchanges whose captured text matches, via that same index. Also returns the total number of rows matching the filter (for the UI's pagination). Bodies are never included in the rows.
func (*Traffic) ProxyAddr ¶
ProxyAddr returns the address workers should set as HTTP(S)_PROXY. A bare ":port" means "bind all interfaces" (legacy default), so map it to loopback.
func (*Traffic) ProxyPoolEnabled ¶ added in v0.3.19
ProxyPoolEnabled reports whether the proxy pool is configured and running.
func (*Traffic) ProxyPoolMarkBad ¶ added in v0.3.34
ProxyPoolMarkBad 把某个上游立即熔断 cooldown(主动健康检查 / 手动验证失败时用)。 返回被熔断的掩码 URL,空串表示不在池里。
func (*Traffic) ProxyPoolReportBad ¶ added in v0.3.34
ProxyPoolReportBad 让调用方(Agent)把某个目标「当前正在用的出口」立即熔断。 target 可以是完整 host:port —— MITM 侧按 host 记录了最近一次选中的上游, 所以不需要请求方知道它到底走了哪个代理。返回被熔断的掩码 URL,空串表示 池为空 / 该目标没走过池上游(例如直连或全局单代理)。
func (*Traffic) ProxyPoolReportFail ¶ added in v0.3.34
ProxyPoolReportFail 上报一次失败(只累计连败,不立刻熔断——被动路径的阈值是 5 次)。
func (*Traffic) ProxyPoolReportOK ¶ added in v0.3.34
ProxyPoolReportOK 上报一次成功(清零连败,记录时间)。
func (*Traffic) ProxyPoolStatus ¶ added in v0.3.17
ProxyPoolStatus 导出池内条目(掩码 URL + 熔断计数),供 UI 展示。
func (*Traffic) ReadEvidence ¶
func (*Traffic) RecoverHostDeleteStages ¶
RecoverHostDeleteStages resolves filesystem moves left by a process exit. SQLite rolls open transactions back on restart, while the supplied callback identifies task archives whose PostgreSQL compaction already committed.
func (*Traffic) Search ¶
func (t *Traffic) Search(host string, page, size int) ([]ExchangeMeta, error)
Search returns paged exchange metadata (never bodies).
func (*Traffic) SetProxyPool ¶ added in v0.3.17
SetProxyPool 原子替换代理池(Traffic 门面)。
func (*Traffic) SetUpstreamProxy ¶
SetUpstreamProxy points every captured request at a global egress proxy (http/https/socks5, optional user:pass in the URL). An empty raw string clears it, restoring direct dialing. The change is atomic and takes effect on the next connection — no restart, no proxy rebuild. go-mitmproxy dials all three schemes itself, so socks5 works uniformly here regardless of the target tool.
func (*Traffic) StageDeleteHostsExact ¶
func (t *Traffic) StageDeleteHostsExact(hosts []string) (*HostDeleteStage, error)
StageDeleteHostsExact prepares a reversible exact-host deletion. Callers must finish every successful stage with Commit or Rollback.
func (*Traffic) StageDeleteHostsExactForArchive ¶
func (t *Traffic) StageDeleteHostsExactForArchive(hosts []string, archiveID, taskID int64) (*HostDeleteStage, error)
StageDeleteHostsExactForArchive ties the reversible traffic deletion to a persistent task archive. Startup recovery uses archiveCommitted to decide whether an interrupted stage must be completed or rolled back.
Source Files