crawl

func Crawl ¶

func Crawl(
	ctx context.Context,
	cfg config.Config,
	startKeys []string,
	fetcher fetch.Fetcher,
	sink events.Sink,
) (Summary, error)

Crawl executes a recursive Jira issue crawl starting from startKeys.

It seeds a work queue with startKeys at depth 0, then runs up to cfg.Concurrency workers (minimum 1) that each pull a key from the queue, fetch the issue, parse it, extract outbound references, render Markdown, and write the output to disk.

Error handling policy ¶

• 401 Unauthorized: the crawl is aborted immediately. All in-flight workers finish their current fetch+render+write, then Crawl returns a partial Summary and an error wrapping client.ErrUnauthorized. The caller should map this to exit code 1.
• 403 Forbidden / 404 Not Found: a stub index.md is written for the issue. The crawl continues. The issue is counted in Stubbed.
• Rate limited (429, retries exhausted): the issue is counted in Failed with reason "Rate limited (429) exhausted". No stub is written. The crawl continues.
• Network/transport error (after client retries): a stub is written with reason "Fetch failed: <err>". The crawl continues.
• Parse/render/output error: the issue is counted in Failed with the error message. No stub is written — these are bugs, not expected operational failures, and papering over them with a stub would hide the problem.

Skip-if-exists probe ¶

Before calling fetcher.Fetch, crawl checks whether <outputDir>/<KEY>/ index.md already exists. If it does and cfg.Refetch is false, the issue is counted as Skipped without burning an API call. This probe lives in crawl (not output) because output.ErrAlreadyExists fires only after a fetch has already happened; the pre-fetch short-circuit is a crawl-level optimization.

Graceful shutdown ¶

When ctx is cancelled (SIGINT/SIGTERM from the CLI, a 401 abort, or a time cap), workers finish their in-flight fetch+render+write and exit. Items remaining in the queue are drained and counted as CapLimited.

Concurrency model ¶

Workers are plain goroutines managed with sync.WaitGroup. The work queue is a buffered channel. An atomic "pending" counter tracks the number of items that are either in the queue or being processed by a worker. When pending reaches zero, the queue channel is closed (under c.mu), which causes all workers to exit their range loop.

All sends to the queue channel happen inside enqueue, which is always called with c.mu held. The channel is closed inside closeQueueLocked, which is also always called with c.mu held. This guarantees that close(c.queue) never races with c.queue <- item.

A dedicated "closer" goroutine watches for context cancellation and drains + closes the queue when the context is done. The visited map and summary are protected by c.mu.

func CrawlGraphWithEnrichers ¶ added in v0.2.0

func CrawlGraphWithEnrichers(
	ctx context.Context,
	cfg config.Config,
	startKeys []string,
	fetcher fetch.Fetcher,
	sink events.Sink,
	hier ChildDiscoverer,
	devStatus DevStatusEnricher,
	store output.Store,
	logger *slog.Logger,
) (Summary, graph.Model, error)

CrawlGraphWithEnrichers is the in-memory graph variant of CrawlWithEnrichers. It runs the same crawl with graph collection FORCED ON (independent of cfg.EmitGraph) and SUPPRESSES the post-loop disk write of graph.json / graph.d2, returning the collected graph.Model alongside the Summary instead.

All other observable behavior — events, summary, exit-code mapping of errors — is byte-identical to CrawlWithEnrichers. This is the entry point the gRPC GetGraph handler uses and is also useful to library callers who want the graph in memory without touching the filesystem.

func CrawlWithDiscoverer ¶

func CrawlWithDiscoverer(
	ctx context.Context,
	cfg config.Config,
	startKeys []string,
	fetcher fetch.Fetcher,
	sink events.Sink,
	hier ChildDiscoverer,
) (Summary, error)

CrawlWithDiscoverer is the extended entry point that additionally accepts a ChildDiscoverer for hierarchy expansion.

When hier is nil OR cfg.IncludeChildren is false, hierarchy discovery is skipped and the crawl behaves exactly as the legacy Crawl function. When both are present, every hierarchy-capable issue (per hierarchy.HierarchyCapable) has its JQL children fetched after the fetch+parse+extract sequence completes and the resulting keys are enqueued at depth+1, subject to the usual issue and depth caps.

This entry point is preserved as a thin wrapper over CrawlWithEnrichers for callers that only need hierarchy expansion. Tests that do not care about hierarchy may continue to call Crawl.

func CrawlWithEnrichers ¶

func CrawlWithEnrichers(
	ctx context.Context,
	cfg config.Config,
	startKeys []string,
	fetcher fetch.Fetcher,
	sink events.Sink,
	hier ChildDiscoverer,
	devStatus DevStatusEnricher,
	store output.Store,
	logger *slog.Logger,
) (Summary, error)

CrawlWithEnrichers is the extended entry point that accepts both a ChildDiscoverer for hierarchy expansion and a DevStatusEnricher for Dev Status pull-request, branch, commit, repository, and build enrichment.

hier and devStatus are independent: passing nil for either disables that enrichment (also independently gated by cfg.IncludeChildren and cfg.IncludeDevStatus respectively). The gojira facade constructs both from the shared *client.Client and supplies them here.

The store parameter is additive over the previous signature: it selects the destination for rendered Markdown. Passing nil defaults to an output.FSStore rooted at cfg.OutputDir, preserving the historical on-disk behavior (skip-if-exists vs. refetch semantics continue to be honored). Alternative output.Store implementations can be injected by callers that want to deliver crawl output to a non-filesystem destination (e.g. an in-memory buffer or a future service front-end).

The logger parameter is additive over the previous signature: it is the orchestrator's *slog.Logger sink for the structured span instrumentation (crawl.start / issue.process.start / phase.* / issue.process.end / crawl.end / crawl.measurement). Passing nil defaults to a no-op logger, preserving the prior behavior of emitting nothing for callers that have not yet adopted the observability instrument.

The signature is additive over CrawlWithDiscoverer; existing callers that only need hierarchy expansion are unaffected.