recon

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package recon ports src/sec_af/agents/recon — the RECON phase of the SEC-AF pipeline.

RECON runs five independent harness "mappers" over a repository and folds their flat, pipe-delimited output into the structured schemas the HUNT and PROVE phases consume:

run_architecture_mapper        -> schemas.ArchitectureMap
run_dependency_auditor         -> schemas.DependencyReport
run_config_scanner             -> schemas.ConfigReport
run_data_flow_mapper           -> schemas.DataFlowMap        (needs the architecture)
run_security_context_profiler  -> schemas.SecurityContext    (needs the architecture)

Python module -> Go file:

agents/recon/architecture.py       -> architecture.go
agents/recon/dependencies.py       -> dependencies.go
agents/recon/config_scanner.py     -> config_scanner.go
agents/recon/data_flow.py          -> data_flow.go
agents/recon/security_context.py   -> security_context.go
agents/recon/_parsers.py           -> parsers.go
agents/recon/__init__.py           -> recon.go (run_recon / run_fast_recon /
                                     run_deep_recon), metrics.go (_repo_metrics),
                                     findings.go (extract_recon_findings)

Every mapper follows one shape, so the port does too: build the prompt (template + a literal CONTEXT suffix), make a private temp dir named `secaf-<agent-name>-*`, run the harness with `Cwd` = that temp dir and `ProjectDir` = the repository, extract the `*Raw` model, parse it, and remove the temp dir. The prompt strings are byte-verbatim; the golden fixtures under testdata/golden are produced from the REAL Python builders by go/scripts/gen_golden.py.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ArchitectureContextBlock

func ArchitectureContextBlock(architecture schemas.ArchitectureMap) string

ArchitectureContextBlock ports architecture.py architecture_context_block:

def architecture_context_block(architecture: ArchitectureMap) -> str:
    return json.dumps(architecture.model_dump(), indent=2)

The result is substituted for `{{ARCHITECTURE_MAP_JSON}}` in the data-flow and security-context prompt templates, so it reaches the LLM verbatim and must be byte-identical to CPython's json.dumps — which Go's encoding/json is NOT (it escapes `<`, `>` and `&`, leaves non-ASCII unescaped where CPython's ensure_ascii=True writes \uXXXX, and renders floats with Go's rules rather than repr()). pyfmt.Dumps is the shared encoder that closes all of those; see DESIGN.md §2b.

Struct fields are emitted in DECLARATION order, which is the order the pydantic class declares them and therefore the order model_dump() inserts them, so no key-order fixup is needed.

Python parity caveat inherited from pyfmt.Dumps: a NIL Go slice renders as `null`, while a pydantic `Field(default_factory=list)` always dumps as `[]`. Every ArchitectureMap that reaches this function came from ParseArchitectureRaw (which builds non-nil slices) or from JSON via ArchitectureMap.UnmarshalJSON (which seeds the `[]` defaults), so the nil case is not reachable through the pipeline — construct with schemas.NewArchitectureMap() rather than a bare literal if you ever call this by hand.

func ExtractReconFindings

func ExtractReconFindings(recon schemas.ReconResult) []schemas.RawFinding

ExtractReconFindings ports extract_recon_findings:

findings.extend(_extract_from_config(recon.config))
findings.extend(_extract_structured_security_items(recon.security_context))
findings.extend(_extract_weak_tls(recon.security_context))

Order matters — the orchestrator prepends this list to the hunt findings, so config secrets lead, then misconfigurations, then weak TLS. Always non-nil so it serializes as `[]`.

func ParseArchitectureRaw

func ParseArchitectureRaw(raw schemas.ArchitectureMapRaw) schemas.ArchitectureMap

ParseArchitectureRaw ports _parsers.parse_architecture_raw.

Row formats (from the ArchitectureMapRaw field descriptions):

modules        name | path | language | description
entry_points   kind | route_or_id | file_path:line | auth_required
trust_bounds   name | source_zone | target_zone | description
services       name | type | endpoint | auth_mechanism
api_endpoints  method | path | handler | file_path:line | auth_required | rate_limited

Python parity: EntryPoint.route is derived, not parsed — the identifier is reused as the route IFF it contains a "/", so "POST /api/login" becomes a route while "migrate" does not. EntryPoint.method, Module.dependencies, TrustBoundary.enforcement and Service.purpose are never populated here; they keep their pydantic defaults.

func ParseConfigReportRaw

func ParseConfigReportRaw(raw schemas.ConfigReportRaw) schemas.ConfigReport

ParseConfigReportRaw ports _parsers.parse_config_report_raw.

Row formats:

secrets     type | file_path:line | match_preview | confidence | is_test
misconfigs  category | file_path:line | key | risk | remediation

Python parity: SecretFinding.confidence falls back to the literal "medium" when the row leaves it empty, while MisconfigFinding.line is `int | None` and becomes nil (not 0) when no positive line was parsed — the two models spell "unknown" differently and the port keeps both spellings. MisconfigFinding.value is never populated here.

Both models mint a fresh uuid4 `id` per row (schemas.NewSecretFinding / NewMisconfigFinding), so repeated parses of identical input are NOT equal — same as Python.

func ParseDataFlowRaw

func ParseDataFlowRaw(raw schemas.DataFlowMapRaw) schemas.DataFlowMap

ParseDataFlowRaw ports _parsers.parse_data_flow_raw.

Row formats:

flows                source | sink | sanitized | file1, file2, ...
sanitization_points  file_path:line | function_name | type | protects_against
sinks                sink_type | file_path:line | function_name | notes

Python parity: `sanitized=_parse_bool(parts[2]) or False` collapses an UNPARSEABLE value to false, so an ambiguous row is treated as unsanitized — the conservative direction for a security tool. DataFlow.path is never populated here and keeps its `[]` default.

func ParseDependencyReportRaw

func ParseDependencyReportRaw(raw schemas.DependencyReportRaw) schemas.DependencyReport

ParseDependencyReportRaw ports _parsers.parse_dependency_report_raw.

Row formats:

sbom        name | version | ecosystem | direct | license
known_cves  cve_id | package | installed_version | fixed_version | cvss_score | direct | reachable
outdated    package | current_version | latest_version | direct

Python parity: direct_count / transitive_count are derived ONLY from the sbom rows (`if is_direct: direct_count += 1 else: transitive_count += 1`), so every sbom row lands in exactly one bucket and an unparseable `direct` field counts as transitive. KnownCVE.epss_score is never populated here.

func ParseSecurityContextRaw

func ParseSecurityContextRaw(raw schemas.SecurityContextRaw) schemas.SecurityContext

ParseSecurityContextRaw ports _parsers.parse_security_context_raw.

crypto_usage rows are "algorithm | key_size | mode | usage_context | is_weak". security_signals are unstructured one-liners routed into security_headers / deployment_signals / framework_security by substring match, first-match-wins in that order (headers beat deployment beats the framework catch-all). Order WITHIN each bucket is the model's emission order.

Python parity: key_size uses `_parse_int(parts[1]) if not _is_na(parts[1]) else None`, so a non-numeric-but-not-NA value like "notanint" yields 0 rather than nil — the `_parse_int` default leaks through.

func QuickDefaults

func QuickDefaults() (schemas.DataFlowMap, schemas.SecurityContext)

QuickDefaults ports _quick_defaults — the placeholder pair the QUICK depth profile substitutes for the two expensive architecture-aware mappers.

Python parity: `SecurityContext(auth_model="unknown", auth_details="unknown")` uses the literal "unknown" for BOTH fields. That differs from the `{"auth_model": "unknown", "auth_details": ""}` default that reasoners/phases.py `_recon_model` seeds; do not conflate them.

func RepoMetrics

func RepoMetrics(repoPath string) (int, int)

RepoMetrics ports _repo_metrics:

def _repo_metrics(repo_path: str) -> tuple[int, int]:
    root = Path(repo_path)
    if not root.exists():
        return 0, 0
    file_count = 0
    line_count = 0
    for path in root.rglob("*"):
        if any(part in _SKIP_DIRS for part in path.parts):
            continue
        if not path.is_file():
            continue
        file_count += 1
        if path.suffix.lower() not in _CODE_EXTS:
            continue
        try:
            with path.open("r", encoding="utf-8", errors="ignore") as handle:
                for _ in handle:
                    line_count += 1
        except OSError:
            continue
    return line_count, file_count

Returns (linesOfCode, fileCount) — Python's tuple order, which is the opposite of what the names suggest, so callers must not swap them.

Python parity, in the order the quirks bite:

  • The skip test runs over `path.parts`, the components of the FULL path including everything in repo_path itself. A repository that happens to live under a directory named `vendor` or `venv` therefore reports (0, 0). That is reproduced here (rootHasSkippedPart short-circuits), not fixed.
  • `is_file()` follows symlinks: a symlink to a regular file IS counted, a broken symlink and a symlink to a directory are not, and neither are sockets/fifos/devices.
  • pathlib does not descend THROUGH a symlinked directory, and neither does filepath.WalkDir, so a symlink loop cannot hang either implementation.
  • `Path.suffix` is NOT filepath.Ext: a leading dot does not start a suffix (".gitignore" has none), a trailing dot does not end one ("x." has none), and only the last component counts ("a.tar.gz" is ".gz"). pySuffix implements pathlib's rule.
  • Line counting is Python TEXT-mode iteration with universal newlines: LF, CRLF and a bare CR each terminate a line, and a final line with no terminator still counts. `errors="ignore"` cannot change the count — UTF-8 continuation bytes are 0x80..0xBF, so a dropped invalid byte can never have been a 0x0A/0x0D.
  • Both loops swallow their errors: an unreadable directory contributes nothing (pathlib catches PermissionError), and a file that fails to open stays counted in file_count but adds no lines (`except OSError: continue` runs AFTER the increment).

Deviation: Python's `Path("")` is `Path(".")`, which exists, so an empty repo_path would scan the process working directory. Go's os.Stat("") fails, so this returns (0, 0). No caller passes an empty path — app.py resolves it to an absolute directory first.

func RunArchitectureMapper

func RunArchitectureMapper(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ArchitectureMap, error)

RunArchitectureMapper ports architecture.py run_architecture_mapper.

async def run_architecture_mapper(app, repo_path) -> ArchitectureMap:
    prompt = PROMPT_PATH.read_text(...) + "\n\nCONTEXT:\n" + ...
    harness_cwd = tempfile.mkdtemp(prefix="secaf-recon-architecture-")
    try:
        result = await app.harness(prompt=prompt, schema=ArchitectureMapRaw,
                                   cwd=harness_cwd, project_dir=repo_path)
        raw = extract_harness_result(result, ArchitectureMapRaw, "Architecture mapper")
        return parse_architecture_raw(raw)
    finally:
        shutil.rmtree(harness_cwd, ignore_errors=True)

Python parity: the harness runs with Cwd set to a PRIVATE scratch directory and ProjectDir set to the repository, so the coding agent explores the repo but writes its JSON output outside it. `shutil.rmtree(..., ignore_errors=True)` maps to a deferred os.RemoveAll whose error is deliberately dropped.

On any failure the zero ArchitectureMap is returned alongside the error; Python raises, so no caller reads the value in that case.

func RunConfigScanner

func RunConfigScanner(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ConfigReport, error)

RunConfigScanner ports config_scanner.py run_config_scanner: harness for a ConfigReportRaw, then parse_config_report_raw.

Structurally identical to RunArchitectureMapper — see its doc comment for the temp-dir / Cwd / ProjectDir contract, which is the same here.

func RunDataFlowMapper

func RunDataFlowMapper(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) (schemas.DataFlowMap, error)

RunDataFlowMapper ports data_flow.py run_data_flow_mapper: substitute the architecture map into the template, harness for a DataFlowMapRaw, then parse_data_flow_raw.

The architecture argument is the ArchitectureMap the architecture mapper produced earlier in the same RECON phase; run_recon only reaches this mapper after that gather has completed.

func RunDeepRecon

func RunDeepRecon(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) (
	schemas.DataFlowMap, schemas.SecurityContext, error,
)

RunDeepRecon ports run_deep_recon — the second gather, over the two mappers that need the architecture map as input.

data_flows, security_context = await asyncio.gather(
    run_data_flow_mapper(app, repo_path, architecture),
    run_security_context_profiler(app, repo_path, architecture),
)

Same errgroup contract as runBaseMappers.

func RunDependencyAuditor

func RunDependencyAuditor(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.DependencyReport, error)

RunDependencyAuditor ports dependencies.py run_dependency_auditor: harness for a DependencyReportRaw, then parse_dependency_report_raw.

Structurally identical to RunArchitectureMapper — see its doc comment for the temp-dir / Cwd / ProjectDir contract, which is the same here.

func RunFastRecon

func RunFastRecon(ctx context.Context, app appx.Harnesser, repoPath string) (schemas.ReconResult, error)

RunFastRecon ports run_fast_recon — the three cheap mappers plus the QUICK placeholders, used by the orchestrator's streaming path.

Python parity: it differs from RunRecon(depth="quick") in exactly two ways, both of which look like oversights but are reproduced faithfully — `frameworks` is hard-coded to the empty list instead of being derived from the (placeholder) security context, and `recon_duration_seconds` is never set, so it keeps its 0.0 pydantic default.

func RunRecon

func RunRecon(ctx context.Context, app appx.Harnesser, repoPath string, depth string) (schemas.ReconResult, error)

RunRecon ports run_recon — the full RECON phase.

started = time.monotonic()
profile = _normalize_depth(depth)
architecture, dependencies, config = await gather(3 mappers)
if profile == QUICK: data_flows, security_context = _quick_defaults()
else:                data_flows, security_context = await gather(2 mappers)
languages  = sorted({m.language.lower() for m in architecture.modules if m.language})
frameworks = sorted({s for s in security_context.framework_security if s})
lines_of_code, file_count = _repo_metrics(repo_path)
return ReconResult(..., recon_duration_seconds=time.monotonic() - started)

Python parity:

  • depth is normalized LENIENTLY (config.NormalizeDepth): anything that is not quick/standard/thorough silently becomes standard, so only the exact string "quick" (any case) skips the deep mappers.
  • languages are lowercased and deduplicated; frameworks keep their original case. Both are sorted, so the Python `set` iteration order — the only nondeterminism in this function — is erased in both runtimes.
  • recon_duration_seconds is set HERE and only here; run_fast_recon leaves it at its 0.0 default.

func RunSecurityContextProfiler

func RunSecurityContextProfiler(ctx context.Context, app appx.Harnesser, repoPath string, architecture schemas.ArchitectureMap) (schemas.SecurityContext, error)

RunSecurityContextProfiler ports security_context.py run_security_context_profiler: substitute the architecture map, harness for a SecurityContextRaw, then parse_security_context_raw (which is where the flat `security_signals` list gets bucketed into headers / deployment / framework).

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL