Documentation
¶
Overview ¶
Package ingest handles repository ingestion: cloning remote repos, walking the local filesystem, filtering files by language/size/gitignore rules, and producing a normalized file list for downstream parsing.
Index ¶
- func AcquireCloneRef(dir string)
- func CleanupCloneDir(localPath string) error
- func ContentFilter(focus string, symbols []*parser.Symbol, imports map[string][]string, ...) map[string]bool
- func CountSourceFiles(files []*File) int
- func DetectLanguage(filename string) string
- func IgnoredDirNames() []string
- func IsKeywordFocus(focus string) bool
- func IsRemote(input string) bool
- func IsWordPressPlugin(input string) bool
- func NormalizeSlug(input string) (string, error)
- func NormalizeWPSlug(input string) (string, error)
- func ParseLightweight(ctx context.Context, files []*File) ([]*parser.Symbol, map[string][]string, []parser.CallSite)
- func ReleaseCloneRef(dir string) error
- func RenderTree(files []*File) string
- type CloneOpts
- type CloneResult
- type File
- type IngestOpts
- type IngestResult
- type WPPluginMeta
- type WPPluginOpts
- type WPSearchResponse
- type WPSearchResult
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AcquireCloneRef ¶ added in v1.22.0
func AcquireCloneRef(dir string)
AcquireCloneRef registers an active reader of the clone directory at dir. Every Acquire MUST be balanced by exactly one ReleaseCloneRef. Acquire is a no-op for an empty dir.
func CleanupCloneDir ¶
CleanupCloneDir removes a cloned repository directory from disk.
func ContentFilter ¶
func ContentFilter(focus string, symbols []*parser.Symbol, imports map[string][]string, calls []parser.CallSite) map[string]bool
ContentFilter returns file paths where ANY keyword from focus matches a symbol name, import path, or call site (case-insensitive, OR logic). Returns nil when focus is empty or has no keywords.
func CountSourceFiles ¶ added in v1.22.0
CountSourceFiles returns the number of files with a detected language.
func DetectLanguage ¶
DetectLanguage returns the programming language for the given filename based on its extension. Delegates to parser.DetectLanguageFromPath so there is a single source of truth — the handler registry.
func IgnoredDirNames ¶
func IgnoredDirNames() []string
IgnoredDirNames returns the alphabetically sorted list of directory names that the indexer skips. Used by tool handlers to surface a hint to callers when a symbol search returns zero results — the symbol may live in an excluded path on disk.
func IsKeywordFocus ¶
IsKeywordFocus returns true when focus looks like a keyword query rather than a path or glob pattern. Heuristic: contains multiple keywords separated by spaces or commas.
func IsRemote ¶
IsRemote returns true if the input looks like a GitHub slug or URL rather than a local filesystem path.
func IsWordPressPlugin ¶
IsWordPressPlugin returns true if the input looks like a WP plugin reference.
func NormalizeSlug ¶
NormalizeSlug extracts the canonical "owner/repo" form from any of:
- owner/repo
- github.com/owner/repo[.git]
- gitlab.com/owner/repo[.git]
- https?://github.com/owner/repo[.git]
- https?://gitlab.com/owner/repo[.git]
- git@github.com:owner/repo[.git]
- git@gitlab.com:owner/repo[.git]
Returns an error if the input does not match any recognised form. This is a thin wrapper around slugparse.Parse.
func NormalizeWPSlug ¶
NormalizeWPSlug extracts the plugin slug from a wp: prefix or wordpress.org URL.
func ParseLightweight ¶
func ParseLightweight(ctx context.Context, files []*File) ([]*parser.Symbol, map[string][]string, []parser.CallSite)
ParseLightweight parses files to extract symbols, imports, and call sites without reading function bodies. Used for content-based focus filtering.
func ReleaseCloneRef ¶ added in v1.22.0
ReleaseCloneRef drops one reader reference for dir and removes the directory from disk (via CleanupCloneDir) only when the reference count reaches zero — i.e. no other reader is still using it. It returns the CleanupCloneDir error when it actually removes the dir, and nil otherwise: when other holders remain, when dir is empty, or when dir was never acquired / already released (an unbalanced release is a safe no-op, never a delete of an unheld dir). This is the refcounted replacement for a bare CleanupCloneDir on shared workspace clones.
func RenderTree ¶
RenderTree builds a visual directory tree from a list of files and returns it as a string using box-drawing characters. Output is capped at treeMaxLines lines; a trailing summary line is added when truncated.
Example output:
├── cmd/ │ └── main.go ├── internal/ │ ├── handler.go │ └── config.go └── go.mod
Types ¶
type CloneOpts ¶
type CloneOpts struct {
// Slug is the owner/repo slug used for directory naming.
Slug string
// Ref is the branch, tag, or commit SHA to check out.
// Empty means the default branch.
Ref string
// DestDir is the parent directory where the clone will be placed.
// A subdirectory named after the repo slug will be created inside.
DestDir string
// GithubToken is used for authenticated clones (private repos, higher rate limits).
// Ignored when CloneURL is set (the URL already encodes credentials).
GithubToken string
// CloneURL is a pre-built HTTPS clone URL. When non-empty, it is used
// directly and the GitHub-specific URL building logic is skipped.
// Slug is still required for directory naming.
CloneURL string
// TokenFunc returns a fresh token for authenticated git operations.
// When set, it is called before each refreshClone to obtain a current
// installation token (ghs_, ~1h TTL). Overrides GithubToken for refreshes.
// When nil, refreshClone relies on credentials embedded in .git/config.
TokenFunc func(ctx context.Context) (string, error)
}
CloneOpts controls how a repository is cloned.
type CloneResult ¶
type CloneResult struct {
// LocalPath is the absolute path to the cloned repository root.
LocalPath string
// Ref is the actual ref checked out (may differ if HEAD was resolved)..
Ref string
}
CloneResult contains the result of a successful clone.
func CloneRepo ¶
func CloneRepo(ctx context.Context, opts CloneOpts) (*CloneResult, error)
CloneRepo performs a shallow git clone of the given repository into DestDir. If Ref is specified, it checks out that ref after cloning.
When a cached clone already exists, it is refreshed in-place via git fetch + reset. If the in-place refresh fails (corrupt repo, network blip, missing ref), a fresh clone is performed into a temporary sibling directory which is then atomically swapped into place (see atomicDirectorySwap).
The atomic swap guarantees that concurrent readers (e.g. WalkDir + indexParseParallel) always see either the old snapshot or the new one — never a half-deleted intermediate state that causes ENOENT during file reads.
Disk note: during the swap the workspace directory briefly holds both the old clone and the new tmp clone. If the volume is critically full (< 2× repo size free) the clone step may fail; the tmp directory is cleaned up before the error is returned.
func FetchWPPlugin ¶
func FetchWPPlugin(ctx context.Context, opts WPPluginOpts) (*CloneResult, error)
FetchWPPlugin downloads and extracts a WordPress plugin ZIP into DestDir. Returns a CloneResult with LocalPath pointing to the extracted directory.
type File ¶
type File struct {
// Path is the absolute path to the file on disk.
Path string
// RelPath is the path relative to the repository root.
RelPath string
// Language is the detected programming language (e.g. "go", "python").
Language string
// Size is the file size in bytes.
Size int64
// ModTime is the file modification time.
ModTime time.Time
}
File represents a single source file ready for parsing.
type IngestOpts ¶
type IngestOpts struct {
// Root is the repository root directory (must already be on disk).
Root string
// Focus limits ingestion to files matching this glob pattern, subdirectory
// prefix, or space-separated keywords (matched case-insensitively against
// the relative path). Empty means ingest everything.
Focus string
// Languages limits ingestion to files of these languages.
// Empty means accept all supported languages.
Languages []string
// MaxFileBytes skips files larger than this size. 0 means no limit.
MaxFileBytes int64
// FollowSymlinks controls whether symlinks are followed.
FollowSymlinks bool
// ExcludeTests skips test files (*_test.go) when true.
ExcludeTests bool
}
IngestOpts controls how a repository is ingested.
type IngestResult ¶
type IngestResult struct {
// Files is the ordered list of source files.
Files []*File
// Root is the repository root used for ingestion.
Root string
// TotalBytes is the total size of all ingested files.
TotalBytes int64
// SkippedCount is the number of files skipped (too large, ignored, binary).
SkippedCount int
// SkippedReasons tallies the number of silently-dropped files per reason.
// Reasons: "gitignored", "test_excluded", "language_filter",
// "oversize", "focus_excluded", "unknown_extension".
// A file may be silently excluded (not counted in SkippedCount) for some
// of these reasons — SkippedReasons captures all of them.
SkippedReasons map[string]int
}
IngestResult contains all files found after filtering.
func ContentFallback ¶
func ContentFallback(ctx context.Context, root string, langs []string, maxFileBytes int64, focus string) (*IngestResult, error)
ContentFallback re-ingests a repo without focus filtering, then uses ParseLightweight + ContentFilter to find files matching focus keywords by symbol name, import path, or call site content.
func IngestRepo ¶
func IngestRepo(ctx context.Context, opts IngestOpts) (*IngestResult, error)
IngestRepo walks a local repository directory and returns all source files that match the given options.
It does NOT read file contents — content loading happens at the parse stage.
type WPPluginMeta ¶
type WPPluginMeta struct {
Name string `json:"name"`
Slug string `json:"slug"`
Version string `json:"version"`
DownloadLink string `json:"download_link"`
Author string `json:"author"`
ActiveInstalls int `json:"active_installs"`
}
WPPluginMeta holds metadata returned by the WordPress.org Plugin API.
func FetchWPPluginMeta ¶
func FetchWPPluginMeta(ctx context.Context, slug string) (*WPPluginMeta, error)
FetchWPPluginMeta retrieves plugin metadata from the WordPress.org API.
type WPPluginOpts ¶
WPPluginOpts controls how a WordPress plugin is fetched.
type WPSearchResponse ¶
type WPSearchResponse struct {
Info wpSearchInfo `json:"info"`
Page int `json:"-"`
Pages int `json:"-"`
Plugins []WPSearchResult `json:"plugins"`
}
WPSearchResponse is the top-level response from the query_plugins API.
func SearchWPPlugins ¶
func SearchWPPlugins(ctx context.Context, query string, perPage, page int) (*WPSearchResponse, error)
SearchWPPlugins queries the WordPress.org plugin directory.
type WPSearchResult ¶
type WPSearchResult struct {
Name string `json:"name"`
Slug string `json:"slug"`
Version string `json:"version"`
Author string `json:"author"`
Rating int `json:"rating"`
ActiveInstalls int `json:"active_installs"`
ShortDescription string `json:"short_description"`
DownloadLink string `json:"download_link"`
}
WPSearchResult holds one plugin from a search response.