Documentation
¶
Overview ¶
Package dedup ports src/sec_af/agents/dedup.py — the HUNT-phase deduplicator and chain correlator that runs between the hunters and the PROVE phase.
The Python module does three things in sequence:
- fingerprint dedup — collapse findings that share sha256("<file>:<line>:<cwe>")[:16], merging the loser's extra data into the winner (_merge_duplicate);
- a semantic dedup pass — for every same-file/same-CWE pair, ask the LLM `.ai(schema=DuplicateCheck)` whether the two are the same root cause, all pairs in parallel (_deduplicate_with_ai);
- chain correlation — seed candidate chains from a hardcoded CWE-pair table (_fallback_correlate) and hand them, with the finding list, to the harness for expansion (deduplicate_and_correlate).
Everything in step 2 and 3 is best-effort: Python swallows every exception (a failed AI check is "not a duplicate", a failed harness call means "no chains" and falls back to the seeds), and the Go port reproduces that exactly rather than propagating errors.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ComputeFingerprint ¶
func ComputeFingerprint(finding schemas.RawFinding) string
ComputeFingerprint ports src/sec_af/agents/dedup.py compute_fingerprint:
key = f"{finding.file_path}:{finding.start_line}:{finding.cwe_id}"
return hashlib.sha256(key.encode("utf-8")).hexdigest()[:16]
The 16-character prefix is of the lowercase hex digest, i.e. the first 8 bytes of the digest.
func DeduplicateAndCorrelate ¶
func DeduplicateAndCorrelate( ctx context.Context, findings []schemas.RawFinding, recon schemas.ReconResult, app appx.Harnesser, repoPath string, ) (schemas.HuntResult, error)
DeduplicateAndCorrelate ports src/sec_af/agents/dedup.py deduplicate_and_correlate.
async def deduplicate_and_correlate(findings, recon, app, repo_path) -> HuntResult
Python parity notes:
- `recon` is accepted and never read. It is kept in the signature because every call site passes it and the reasoner adapter mirrors the Python argument list.
- The findings are MUTATED in place (fingerprint seeding, merges). Python mutates the caller's pydantic objects; the Go port takes pointers into the caller's slice so the same thing happens.
- The chain-correlation harness call is wrapped in a bare `except Exception: chains = []`, so ANY failure — transport error, timeout, is_error result, unparseable output — silently yields the seed chains. The only error this function can return is a tempdir-creation failure, which in Python happens OUTSIDE the try and therefore propagates.
Types ¶
type Deduplicator ¶
type Deduplicator struct {
// contains filtered or unexported fields
}
Deduplicator ports the src/sec_af/agents/dedup.py Deduplicator class — a thin object binding an app and a repo path so callers can invoke the free function without repeating them. Nothing in the live path uses it; it is part of the module's __all__ and is ported for completeness.
func NewDeduplicator ¶
func NewDeduplicator(app appx.Harnesser, repoPath string) *Deduplicator
NewDeduplicator ports Deduplicator.__init__.
func (*Deduplicator) Run ¶
func (d *Deduplicator) Run(ctx context.Context, findings []schemas.RawFinding, recon schemas.ReconResult) (schemas.HuntResult, error)
Run ports Deduplicator.run.
Source Files