Documentation
¶
Overview ¶
Package recon ports src/sec_af/agents/recon — the RECON phase of the SEC-AF pipeline.
RECON runs five independent harness "mappers" over a repository and folds their flat, pipe-delimited output into the structured schemas the HUNT and PROVE phases consume:
run_architecture_mapper -> schemas.ArchitectureMap run_dependency_auditor -> schemas.DependencyReport run_config_scanner -> schemas.ConfigReport run_data_flow_mapper -> schemas.DataFlowMap (needs the architecture) run_security_context_profiler -> schemas.SecurityContext (needs the architecture)
Python module -> Go file:
agents/recon/architecture.py -> architecture.go
agents/recon/dependencies.py -> dependencies.go
agents/recon/config_scanner.py -> config_scanner.go
agents/recon/data_flow.py -> data_flow.go
agents/recon/security_context.py -> security_context.go
agents/recon/_parsers.py -> parsers.go
agents/recon/__init__.py -> recon.go (run_recon / run_fast_recon /
run_deep_recon), metrics.go (_repo_metrics),
findings.go (extract_recon_findings)
Every mapper follows one shape, so the port does too: build the prompt (template + a literal CONTEXT suffix), make a private temp dir named `secaf-<agent-name>-*`, run the harness with `Cwd` = that temp dir and `ProjectDir` = the repository, extract the `*Raw` model, parse it, and remove the temp dir. The prompt strings are byte-verbatim; the golden fixtures under testdata/golden are produced from the REAL Python builders by go/scripts/gen_golden.py.
Index ¶
- func ArchitectureContextBlock(architecture schemas.ArchitectureMap) string
- func ExtractReconFindings(recon schemas.ReconResult) []schemas.RawFinding
- func ParseArchitectureRaw(raw schemas.ArchitectureMapRaw) schemas.ArchitectureMap
- func ParseConfigReportRaw(raw schemas.ConfigReportRaw) schemas.ConfigReport
- func ParseDataFlowRaw(raw schemas.DataFlowMapRaw) schemas.DataFlowMap
- func ParseDependencyReportRaw(raw schemas.DependencyReportRaw) schemas.DependencyReport
- func ParseSecurityContextRaw(raw schemas.SecurityContextRaw) schemas.SecurityContext
- func QuickDefaults() (schemas.DataFlowMap, schemas.SecurityContext)
- func RepoMetrics(repoPath string) (int, int)
- func RunArchitectureMapper(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ArchitectureMap, error)
- func RunConfigScanner(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ConfigReport, error)
- func RunDataFlowMapper(ctx context.Context, app appx.Harnesser, repoPath string, ...) (schemas.DataFlowMap, error)
- func RunDeepRecon(ctx context.Context, app appx.Harnesser, repoPath string, ...) (schemas.DataFlowMap, schemas.SecurityContext, error)
- func RunDependencyAuditor(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.DependencyReport, error)
- func RunFastRecon(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ReconResult, error)
- func RunRecon(ctx context.Context, app appx.Harnesser, repoPath string, depth string) (schemas.ReconResult, error)
- func RunSecurityContextProfiler(ctx context.Context, app appx.Harnesser, repoPath string, ...) (schemas.SecurityContext, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ArchitectureContextBlock ¶
func ArchitectureContextBlock(architecture schemas.ArchitectureMap) string
ArchitectureContextBlock ports architecture.py architecture_context_block:
def architecture_context_block(architecture: ArchitectureMap) -> str:
return json.dumps(architecture.model_dump(), indent=2)
The result is substituted for `{{ARCHITECTURE_MAP_JSON}}` in the data-flow and security-context prompt templates, so it reaches the LLM verbatim and must be byte-identical to CPython's json.dumps — which Go's encoding/json is NOT (it escapes `<`, `>` and `&`, leaves non-ASCII unescaped where CPython's ensure_ascii=True writes \uXXXX, and renders floats with Go's rules rather than repr()). pyfmt.Dumps is the shared encoder that closes all of those; see DESIGN.md §2b.
Struct fields are emitted in DECLARATION order, which is the order the pydantic class declares them and therefore the order model_dump() inserts them, so no key-order fixup is needed.
Python parity caveat inherited from pyfmt.Dumps: a NIL Go slice renders as `null`, while a pydantic `Field(default_factory=list)` always dumps as `[]`. Every ArchitectureMap that reaches this function came from ParseArchitectureRaw (which builds non-nil slices) or from JSON via ArchitectureMap.UnmarshalJSON (which seeds the `[]` defaults), so the nil case is not reachable through the pipeline — construct with schemas.NewArchitectureMap() rather than a bare literal if you ever call this by hand.
func ExtractReconFindings ¶
func ExtractReconFindings(recon schemas.ReconResult) []schemas.RawFinding
ExtractReconFindings ports extract_recon_findings:
findings.extend(_extract_from_config(recon.config)) findings.extend(_extract_structured_security_items(recon.security_context)) findings.extend(_extract_weak_tls(recon.security_context))
Order matters — the orchestrator prepends this list to the hunt findings, so config secrets lead, then misconfigurations, then weak TLS. Always non-nil so it serializes as `[]`.
func ParseArchitectureRaw ¶
func ParseArchitectureRaw(raw schemas.ArchitectureMapRaw) schemas.ArchitectureMap
ParseArchitectureRaw ports _parsers.parse_architecture_raw.
Row formats (from the ArchitectureMapRaw field descriptions):
modules name | path | language | description entry_points kind | route_or_id | file_path:line | auth_required trust_bounds name | source_zone | target_zone | description services name | type | endpoint | auth_mechanism api_endpoints method | path | handler | file_path:line | auth_required | rate_limited
Python parity: EntryPoint.route is derived, not parsed — the identifier is reused as the route IFF it contains a "/", so "POST /api/login" becomes a route while "migrate" does not. EntryPoint.method, Module.dependencies, TrustBoundary.enforcement and Service.purpose are never populated here; they keep their pydantic defaults.
func ParseConfigReportRaw ¶
func ParseConfigReportRaw(raw schemas.ConfigReportRaw) schemas.ConfigReport
ParseConfigReportRaw ports _parsers.parse_config_report_raw.
Row formats:
secrets type | file_path:line | match_preview | confidence | is_test misconfigs category | file_path:line | key | risk | remediation
Python parity: SecretFinding.confidence falls back to the literal "medium" when the row leaves it empty, while MisconfigFinding.line is `int | None` and becomes nil (not 0) when no positive line was parsed — the two models spell "unknown" differently and the port keeps both spellings. MisconfigFinding.value is never populated here.
Both models mint a fresh uuid4 `id` per row (schemas.NewSecretFinding / NewMisconfigFinding), so repeated parses of identical input are NOT equal — same as Python.
func ParseDataFlowRaw ¶
func ParseDataFlowRaw(raw schemas.DataFlowMapRaw) schemas.DataFlowMap
ParseDataFlowRaw ports _parsers.parse_data_flow_raw.
Row formats:
flows source | sink | sanitized | file1, file2, ... sanitization_points file_path:line | function_name | type | protects_against sinks sink_type | file_path:line | function_name | notes
Python parity: `sanitized=_parse_bool(parts[2]) or False` collapses an UNPARSEABLE value to false, so an ambiguous row is treated as unsanitized — the conservative direction for a security tool. DataFlow.path is never populated here and keeps its `[]` default.
func ParseDependencyReportRaw ¶
func ParseDependencyReportRaw(raw schemas.DependencyReportRaw) schemas.DependencyReport
ParseDependencyReportRaw ports _parsers.parse_dependency_report_raw.
Row formats:
sbom name | version | ecosystem | direct | license known_cves cve_id | package | installed_version | fixed_version | cvss_score | direct | reachable outdated package | current_version | latest_version | direct
Python parity: direct_count / transitive_count are derived ONLY from the sbom rows (`if is_direct: direct_count += 1 else: transitive_count += 1`), so every sbom row lands in exactly one bucket and an unparseable `direct` field counts as transitive. KnownCVE.epss_score is never populated here.
func ParseSecurityContextRaw ¶
func ParseSecurityContextRaw(raw schemas.SecurityContextRaw) schemas.SecurityContext
ParseSecurityContextRaw ports _parsers.parse_security_context_raw.
crypto_usage rows are "algorithm | key_size | mode | usage_context | is_weak". security_signals are unstructured one-liners routed into security_headers / deployment_signals / framework_security by substring match, first-match-wins in that order (headers beat deployment beats the framework catch-all). Order WITHIN each bucket is the model's emission order.
Python parity: key_size uses `_parse_int(parts[1]) if not _is_na(parts[1]) else None`, so a non-numeric-but-not-NA value like "notanint" yields 0 rather than nil — the `_parse_int` default leaks through.
func QuickDefaults ¶
func QuickDefaults() (schemas.DataFlowMap, schemas.SecurityContext)
QuickDefaults ports _quick_defaults — the placeholder pair the QUICK depth profile substitutes for the two expensive architecture-aware mappers.
Python parity: `SecurityContext(auth_model="unknown", auth_details="unknown")` uses the literal "unknown" for BOTH fields. That differs from the `{"auth_model": "unknown", "auth_details": ""}` default that reasoners/phases.py `_recon_model` seeds; do not conflate them.
func RepoMetrics ¶
RepoMetrics ports _repo_metrics:
def _repo_metrics(repo_path: str) -> tuple[int, int]:
root = Path(repo_path)
if not root.exists():
return 0, 0
file_count = 0
line_count = 0
for path in root.rglob("*"):
if any(part in _SKIP_DIRS for part in path.parts):
continue
if not path.is_file():
continue
file_count += 1
if path.suffix.lower() not in _CODE_EXTS:
continue
try:
with path.open("r", encoding="utf-8", errors="ignore") as handle:
for _ in handle:
line_count += 1
except OSError:
continue
return line_count, file_count
Returns (linesOfCode, fileCount) — Python's tuple order, which is the opposite of what the names suggest, so callers must not swap them.
Python parity, in the order the quirks bite:
- The skip test runs over `path.parts`, the components of the FULL path including everything in repo_path itself. A repository that happens to live under a directory named `vendor` or `venv` therefore reports (0, 0). That is reproduced here (rootHasSkippedPart short-circuits), not fixed.
- `is_file()` follows symlinks: a symlink to a regular file IS counted, a broken symlink and a symlink to a directory are not, and neither are sockets/fifos/devices.
- pathlib does not descend THROUGH a symlinked directory, and neither does filepath.WalkDir, so a symlink loop cannot hang either implementation.
- `Path.suffix` is NOT filepath.Ext: a leading dot does not start a suffix (".gitignore" has none), a trailing dot does not end one ("x." has none), and only the last component counts ("a.tar.gz" is ".gz"). pySuffix implements pathlib's rule.
- Line counting is Python TEXT-mode iteration with universal newlines: LF, CRLF and a bare CR each terminate a line, and a final line with no terminator still counts. `errors="ignore"` cannot change the count — UTF-8 continuation bytes are 0x80..0xBF, so a dropped invalid byte can never have been a 0x0A/0x0D.
- Both loops swallow their errors: an unreadable directory contributes nothing (pathlib catches PermissionError), and a file that fails to open stays counted in file_count but adds no lines (`except OSError: continue` runs AFTER the increment).
Deviation: Python's `Path("")` is `Path(".")`, which exists, so an empty repo_path would scan the process working directory. Go's os.Stat("") fails, so this returns (0, 0). No caller passes an empty path — app.py resolves it to an absolute directory first.
func RunArchitectureMapper ¶
func RunArchitectureMapper(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ArchitectureMap, error)
RunArchitectureMapper ports architecture.py run_architecture_mapper.
async def run_architecture_mapper(app, repo_path) -> ArchitectureMap:
prompt = PROMPT_PATH.read_text(...) + "\n\nCONTEXT:\n" + ...
harness_cwd = tempfile.mkdtemp(prefix="secaf-recon-architecture-")
try:
result = await app.harness(prompt=prompt, schema=ArchitectureMapRaw,
cwd=harness_cwd, project_dir=repo_path)
raw = extract_harness_result(result, ArchitectureMapRaw, "Architecture mapper")
return parse_architecture_raw(raw)
finally:
shutil.rmtree(harness_cwd, ignore_errors=True)
Python parity: the harness runs with Cwd set to a PRIVATE scratch directory and ProjectDir set to the repository, so the coding agent explores the repo but writes its JSON output outside it. `shutil.rmtree(..., ignore_errors=True)` maps to a deferred os.RemoveAll whose error is deliberately dropped.
On any failure the zero ArchitectureMap is returned alongside the error; Python raises, so no caller reads the value in that case.
func RunConfigScanner ¶
func RunConfigScanner(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ConfigReport, error)
RunConfigScanner ports config_scanner.py run_config_scanner: harness for a ConfigReportRaw, then parse_config_report_raw.
Structurally identical to RunArchitectureMapper — see its doc comment for the temp-dir / Cwd / ProjectDir contract, which is the same here.
func RunDataFlowMapper ¶
func RunDataFlowMapper(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) (schemas.DataFlowMap, error)
RunDataFlowMapper ports data_flow.py run_data_flow_mapper: substitute the architecture map into the template, harness for a DataFlowMapRaw, then parse_data_flow_raw.
The architecture argument is the ArchitectureMap the architecture mapper produced earlier in the same RECON phase; run_recon only reaches this mapper after that gather has completed.
func RunDeepRecon ¶
func RunDeepRecon(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) ( schemas.DataFlowMap, schemas.SecurityContext, error, )
RunDeepRecon ports run_deep_recon — the second gather, over the two mappers that need the architecture map as input.
data_flows, security_context = await asyncio.gather(
run_data_flow_mapper(app, repo_path, architecture),
run_security_context_profiler(app, repo_path, architecture),
)
Same errgroup contract as runBaseMappers.
func RunDependencyAuditor ¶
func RunDependencyAuditor(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.DependencyReport, error)
RunDependencyAuditor ports dependencies.py run_dependency_auditor: harness for a DependencyReportRaw, then parse_dependency_report_raw.
Structurally identical to RunArchitectureMapper — see its doc comment for the temp-dir / Cwd / ProjectDir contract, which is the same here.
func RunFastRecon ¶
func RunFastRecon(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ReconResult, error)
RunFastRecon ports run_fast_recon — the three cheap mappers plus the QUICK placeholders, used by the orchestrator's streaming path.
Python parity: it differs from RunRecon(depth="quick") in exactly two ways, both of which look like oversights but are reproduced faithfully — `frameworks` is hard-coded to the empty list instead of being derived from the (placeholder) security context, and `recon_duration_seconds` is never set, so it keeps its 0.0 pydantic default.
func RunRecon ¶
func RunRecon(ctx context.Context, app appx.Harnesser, repoPath string, depth string) (schemas.ReconResult, error)
RunRecon ports run_recon — the full RECON phase.
started = time.monotonic()
profile = _normalize_depth(depth)
architecture, dependencies, config = await gather(3 mappers)
if profile == QUICK: data_flows, security_context = _quick_defaults()
else: data_flows, security_context = await gather(2 mappers)
languages = sorted({m.language.lower() for m in architecture.modules if m.language})
frameworks = sorted({s for s in security_context.framework_security if s})
lines_of_code, file_count = _repo_metrics(repo_path)
return ReconResult(..., recon_duration_seconds=time.monotonic() - started)
Python parity:
- depth is normalized LENIENTLY (config.NormalizeDepth): anything that is not quick/standard/thorough silently becomes standard, so only the exact string "quick" (any case) skips the deep mappers.
- languages are lowercased and deduplicated; frameworks keep their original case. Both are sorted, so the Python `set` iteration order — the only nondeterminism in this function — is erased in both runtimes.
- recon_duration_seconds is set HERE and only here; run_fast_recon leaves it at its 0.0 default.
func RunSecurityContextProfiler ¶
func RunSecurityContextProfiler(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) (schemas.SecurityContext, error)
RunSecurityContextProfiler ports security_context.py run_security_context_profiler: substitute the architecture map, harness for a SecurityContextRaw, then parse_security_context_raw (which is where the flat `security_signals` list gets bucketed into headers / deployment / framework).
Types ¶
This section is empty.