Documentation
¶
Index ¶
- Constants
- func BatchBytesWithPrivacyFilter(ctx context.Context, inputs []NamedBlob) ([][]byte, error)
- func Bytes(b []byte) []byte
- func BytesWithPrivacyFilter(ctx context.Context, b []byte) []byte
- func ConfigureCustomRules(cfg CustomRulesConfig)
- func ConfigurePII(cfg PIIConfig)
- func ConfigurePrivacyFilter(cfg OPFConfig)
- func ConfigurePrivacyFilterWithRuntime(cfg OPFConfig, rt opfRuntime)
- func IsKnownOPFCategory(name string) bool
- func JSONLContent(content string) (string, error)
- func JSONLContentWithPrivacyFilter(ctx context.Context, content string) (string, error)
- func OPFBreakerTripped() bool
- func OPFCommand() string
- func OPFEnabled() bool
- func ResetOPFConfigForTest()
- func String(s string) string
- func StringWithPrivacyFilter(ctx context.Context, s string) string
- func SumProseLeafBytes(inputs []NamedBlob) int
- type CustomRulesConfig
- type NamedBlob
- type OPFConfig
- type PIICategory
- type PIIConfig
- type Pack
- type RedactedBytes
- type Rule
- type Sample
- type Span
Constants ¶
const RedactedPlaceholder = "REDACTED"
RedactedPlaceholder is the replacement text used for redacted secrets.
const RedactorsDirName = "redactors"
RedactorsDirName is the .entire subdirectory used for user-defined rule packs.
Variables ¶
This section is empty.
Functions ¶
func BatchBytesWithPrivacyFilter ¶ added in v0.7.8
BatchBytesWithPrivacyFilter redacts N blobs with a single OPF inference call instead of N. Returns redacted bytes in input order (output[i] is the redaction of inputs[i]).
Failure semantics — fail-closed: any error from the OPF runtime returns a non-nil error. Callers running this for privacy-critical operations (e.g. the pre-push rewrite) must abort rather than proceed with partially-redacted content. The per-blob JSONLContentWithPrivacyFilter falls back to 7-layer on batch failure; this batched variant intentionally does not, because the only caller (cross-blob walker) needs an explicit signal that OPF did not finish.
When OPF is unconfigured, disabled, has no enabled categories, or the per-process circuit breaker has tripped, returns 7-layer-only output for every blob with no error. This matches the existing non-batched paths and keeps the caller's hot-path code clean.
func BytesWithPrivacyFilter ¶ added in v0.7.8
BytesWithPrivacyFilter augments Bytes with the OpenAI Privacy Filter for raw (non-JSONL) byte content. Used by checkpoint write paths that handle metadata files which may or may not be JSONL.
func ConfigureCustomRules ¶ added in v0.6.2
func ConfigureCustomRules(cfg CustomRulesConfig)
ConfigureCustomRules compiles user-defined redaction rules and stores the result for use by redact.String(). Sample-validation runs here too, so failures surface the next time any process initializes redaction.
Call once at process startup after loading settings. Thread-safe.
func ConfigurePII ¶ added in v0.5.1
func ConfigurePII(cfg PIIConfig)
ConfigurePII sets the global PII redaction configuration. Pre-compiles patterns so the hot path (String → detectPII) does no compilation. Call once at startup after loading settings. Thread-safe.
func ConfigurePrivacyFilter ¶ added in v0.7.8
func ConfigurePrivacyFilter(cfg OPFConfig)
ConfigurePrivacyFilter sets the global OPF configuration and constructs the default shell-out runtime. Call once at process startup after loading settings. Thread-safe. Subsequent calls replace the previous configuration and reset the circuit breaker (a new config might fix what broke the prior one).
func ConfigurePrivacyFilterWithRuntime ¶ added in v0.7.8
func ConfigurePrivacyFilterWithRuntime(cfg OPFConfig, rt opfRuntime)
ConfigurePrivacyFilterWithRuntime is the test-only variant that takes an explicit runtime instead of constructing one.
func IsKnownOPFCategory ¶ added in v0.7.8
IsKnownOPFCategory reports whether name is one of the OPF native labels the CLI knows how to tag and render. Exported so the settings layer can reject typos at parse time — silent zero-detection of a privacy category would leave users thinking they're protected when they're not.
func JSONLContent ¶
JSONLContent parses each line as JSON to determine which string values need redaction, then performs targeted replacements on the raw JSON bytes. Lines with no secrets are returned unchanged, preserving original formatting.
For multi-line JSON content (e.g., pretty-printed single JSON objects like OpenCode export), the function first attempts to parse the entire content as a single JSON value. This ensures field-aware redaction (which skips ID fields) is used instead of falling back to entropy-based detection on raw text lines, which would corrupt high-entropy identifiers.
func JSONLContentWithPrivacyFilter ¶ added in v0.7.8
JSONLContentWithPrivacyFilter augments JSONLContent with the OpenAI Privacy Filter via batched inference. Walks the content twice: pass 1 collects unique prose-shaped leaves into a single RedactBatch call; pass 2 applies the seven regex layers per leaf plus the cached OPF spans for that leaf. One OPF shell-out covers the whole transcript instead of one per leaf — without batching, a typical 500-leaf transcript would take many minutes per commit.
Falls back to the plain JSONLContent flow when OPF is unconfigured, the breaker is tripped, no categories are enabled, or the batch call errors.
func OPFBreakerTripped ¶ added in v0.7.8
func OPFBreakerTripped() bool
OPFBreakerTripped reports whether the per-process OPF circuit breaker has been tripped — i.e. an OPF invocation failed at some point during this process's lifetime. The pre-push rewrite uses this to detect when OPF silently fell back to 7-layer mid-rewrite and abort before CAS-ing the new ref; otherwise the rewritten commits would carry the Entire-OPF-Applied: true trailer despite containing only 7-layer content, and the next push would skip them.
func OPFCommand ¶ added in v0.7.8
func OPFCommand() string
OPFCommand returns the configured OPF binary command, or the default when OPF is unconfigured. Used by error messages so the user sees the exact command they need to fix.
func OPFEnabled ¶ added in v0.7.8
func OPFEnabled() bool
OPFEnabled reports whether the OpenAI Privacy Filter is configured and turned on for this process. Callers gate pre-push rewrite work on this: when false, the pre-push hook pushes the local 7-layer checkpoint branch verbatim with no extra processing. Independent of the circuit breaker — a tripped breaker still reports Enabled=true because the runtime config didn't change; the rewrite logic itself handles the breaker by short-circuiting per-commit OPF calls.
func ResetOPFConfigForTest ¶ added in v0.7.8
func ResetOPFConfigForTest()
ResetOPFConfigForTest clears OPF configuration and the circuit breaker. Test-only.
func String ¶
String replaces secrets and PII in s using layered detection: 1. Entropy-based: high-entropy alphanumeric sequences (threshold 4.5) 2. Pattern-based: betterleaks regex rules (260+ known secret formats) 3. Credentialed URIs: URLs containing userinfo passwords 4. Database connection strings: JDBC, keyword DSNs, and semicolon strings 5. User-defined custom rules: configured via ConfigureCustomRules 6. Bounded credential key/value pairs: DB_PASSWORD=... 7. PII detection: email, phone, address patterns (only when configured via ConfigurePII) A string is redacted if ANY method flags it.
func StringWithPrivacyFilter ¶ added in v0.7.8
StringWithPrivacyFilter augments String with the OpenAI Privacy Filter. Use only at condensation/export boundaries; per-turn writes must use String to avoid the OPF shell-out cost inside the agent loop.
func SumProseLeafBytes ¶ added in v0.7.8
SumProseLeafBytes returns the cumulative byte size of prose-shaped (has-space) leaves across inputs — the upper bound on what BatchBytesWithPrivacyFilter would send to OPF inference.
Callers use this to enforce a cap before paying the OPF cost: a push with 100MB of mostly-structural JSON has tens of KB of actual leaves; a push with 100MB of dense prose has hundreds of MB. The blob-byte size doesn't tell you which without looking inside.
Returns a CONSERVATIVE UPPER BOUND on what would go to OPF — same has-space gate and JSONL/JSON parse with whole-content fallback as the collector inside BatchBytesWithPrivacyFilter, BUT this function does NOT deduplicate identical leaves across blobs. The actual batch sent to OPF dedups by leaf-text, so a push with many repeated leaves will report higher byte counts here than OPF actually sees. Callers using this for cap enforcement get an over-strict bound, which is safe (false positives possible, false negatives impossible).
Types ¶
type CustomRulesConfig ¶ added in v0.6.2
type CustomRulesConfig struct {
// Inline maps a label (used only in logs/diagnostics) to a Go RE2 regex
// string. Failed compilations are logged via slog.Warn and dropped.
Inline map[string]string
// Packs are pre-parsed rule packs (see LoadPacks). Per-rule regex
// compilation failures are logged and dropped; sample mismatches are
// logged but do not drop the rule.
Packs []*Pack
}
CustomRulesConfig configures inline custom_redactions and parsed rule packs.
type NamedBlob ¶ added in v0.7.8
NamedBlob is one input to BatchBytesWithPrivacyFilter. Name drives redaction shape: a ".jsonl" or ".json" suffix triggers JSON-aware leaf extraction (string values inside the parsed structure); any other suffix treats the whole content as a single leaf.
Content is the raw blob bytes. The blob's redacted output appears at the same index in the function's return slice.
type OPFConfig ¶ added in v0.7.8
type OPFConfig struct {
Enabled bool
Categories map[string]bool
Command string // path or name of the opf binary; "" defaults to "opf"
Timeout int // seconds; 0 defaults to 30
// contains filtered or unexported fields
}
OPFConfig configures the optional OpenAI Privacy Filter detection layer. Defaults are applied by ConfigurePrivacyFilter; callers should pass values straight from settings without local normalization.
type PIICategory ¶ added in v0.5.1
type PIICategory string
PIICategory identifies a category of personally identifiable information.
const ( PIIEmail PIICategory = "email" PIIPhone PIICategory = "phone" PIIAddress PIICategory = "address" )
type PIIConfig ¶ added in v0.5.1
type PIIConfig struct {
// Enabled globally enables/disables PII redaction.
// When false, no PII patterns are checked (secrets still redacted).
Enabled bool
// Categories maps each PII category to whether it is enabled.
// Missing keys default to false (disabled).
Categories map[PIICategory]bool
// CustomPatterns allows teams to define additional regex patterns.
// Each key is a label used in the replacement token (uppercased),
// and each value is a regex pattern string.
// Example: {"employee_id": `EMP-\d{6}`} produces [REDACTED_EMPLOYEE_ID].
CustomPatterns map[string]string
// contains filtered or unexported fields
}
PIIConfig controls which PII categories are detected and redacted.
type Pack ¶ added in v0.6.2
type Pack struct {
Name string `json:"name" yaml:"name"`
Version string `json:"version" yaml:"version"`
Description string `json:"description,omitempty" yaml:"description"`
Rules []Rule `json:"rules" yaml:"rules"`
// contains filtered or unexported fields
}
Pack is a versioned bundle of redaction rules loaded from a single file under .entire/redactors/. Both YAML and JSON encodings are accepted; the schema is identical.
func LoadPacks ¶ added in v0.6.2
LoadPacks discovers and parses all rule packs in dir, including any subdirectories (so the conventional .entire/redactors/local/ path for personal/uncommitted rules is picked up automatically). Files with the extensions .yaml, .yml, and .json are considered packs; other files are ignored. A missing directory is treated as "no packs configured" and returns no error. Per-file parse errors are slog.Warn'd and the file is skipped — never fatal — so one bad file does not silence the rest.
Soft caps: files larger than maxPackFileBytes are skipped with a warning, and discovery stops after maxPackFiles parsed packs. The trust boundary is "user owns repo," so these are runaway-input guards, not security limits.
func ParsePack ¶ added in v0.6.2
ParsePack decodes a single pack file. sourcePath is used both to pick the encoding (YAML by default; JSON only when the extension is .json) and to enforce that the pack's `name` matches the filename stem.
Precondition: sourcePath must be a vetted local file path (the production caller is LoadPacks, which only invokes ParsePack with paths produced by WalkDir under the configured .entire/redactors/ directory). Callers passing arbitrary or remote paths must enforce their own trust model — ParsePack does not sanitize sourcePath beyond reading its extension.
type RedactedBytes ¶ added in v0.5.5
type RedactedBytes struct {
// contains filtered or unexported fields
}
RedactedBytes represents transcript data that has been through secret redaction. Consumers that require pre-redacted input (e.g., compact.Compact, checkpoint stores) accept this type to enforce the contract at compile time.
Produced by JSONLBytes (primary constructor) or trusted wrappers for data previously persisted by checkpoint writers.
func AlreadyRedacted ¶ added in v0.5.5
func AlreadyRedacted(data []byte) RedactedBytes
AlreadyRedacted wraps transcript bytes known to already be redacted by a prior write path. Use this ONLY for trusted sources such as persisted checkpoint transcripts or controlled test fixtures. For fresh transcript input, use JSONLBytes.
func JSONLBytes ¶
func JSONLBytes(b []byte) (RedactedBytes, error)
JSONLBytes redacts secrets in JSONL-formatted byte content and returns the result as RedactedBytes, certifying the output has been through redaction.
func JSONLBytesWithPrivacyFilter ¶ added in v0.7.8
func JSONLBytesWithPrivacyFilter(ctx context.Context, b []byte) (RedactedBytes, error)
JSONLBytesWithPrivacyFilter augments JSONLBytes with the OpenAI Privacy Filter. Use only at condensation/export boundaries; per-turn writes must use JSONLBytes.
func (RedactedBytes) Bytes ¶ added in v0.5.5
func (r RedactedBytes) Bytes() []byte
Bytes returns the underlying byte slice.
func (RedactedBytes) Len ¶ added in v0.5.5
func (r RedactedBytes) Len() int
Len returns the number of bytes in the redacted payload.
type Rule ¶ added in v0.6.2
type Rule struct {
ID string `json:"id" yaml:"id"`
Description string `json:"description,omitempty" yaml:"description"`
Regex string `json:"regex" yaml:"regex"`
Samples []Sample `json:"samples,omitempty" yaml:"samples"`
}
Rule is a single redaction rule within a Pack.
type Sample ¶ added in v0.6.2
type Sample struct {
Input string `json:"input" yaml:"input"`
Redacted bool `json:"redacted" yaml:"redacted"`
}
Sample is a self-test entry for a Rule. The runner asserts whether the rule's regex matching `Input` matches the `Redacted` expectation.
type Span ¶ added in v0.7.8
Span is a redaction region returned by an opfRuntime, with BYTE-offset boundaries against the input text. OPF itself reports character (rune) offsets via its JSON output; the shell-out adapter translates those to byte offsets before returning Spans so callers can slice []byte input directly without re-walking runes.
Source Files