ingest

package
v1.36.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 16, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package ingest handles repository ingestion: cloning remote repos, walking the local filesystem, filtering files by language/size/gitignore rules, and producing a normalized file list for downstream parsing.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AcquireCloneRef added in v1.22.0

func AcquireCloneRef(dir string)

AcquireCloneRef registers an active reader of the clone directory at dir. Every Acquire MUST be balanced by exactly one ReleaseCloneRef. Acquire is a no-op for an empty dir.

func CleanupCloneDir

func CleanupCloneDir(localPath string) error

CleanupCloneDir removes a cloned repository directory from disk.

func ContentFilter

func ContentFilter(focus string, symbols []*parser.Symbol, imports map[string][]string, calls []parser.CallSite) map[string]bool

ContentFilter returns file paths where ANY keyword from focus matches a symbol name, import path, or call site (case-insensitive, OR logic). Returns nil when focus is empty or has no keywords.

func CountSourceFiles added in v1.22.0

func CountSourceFiles(files []*File) int

CountSourceFiles returns the number of files with a detected language.

func DetectLanguage

func DetectLanguage(filename string) string

DetectLanguage returns the programming language for the given filename based on its extension. Delegates to parser.DetectLanguageFromPath so there is a single source of truth — the handler registry.

func IgnoredDirNames

func IgnoredDirNames() []string

IgnoredDirNames returns the alphabetically sorted list of directory names that the indexer skips. Used by tool handlers to surface a hint to callers when a symbol search returns zero results — the symbol may live in an excluded path on disk.

func IsKeywordFocus

func IsKeywordFocus(focus string) bool

IsKeywordFocus returns true when focus looks like a keyword query rather than a path or glob pattern. Heuristic: contains multiple keywords separated by spaces or commas.

func IsRemote

func IsRemote(input string) bool

IsRemote returns true if the input looks like a GitHub slug or URL rather than a local filesystem path.

func IsWordPressPlugin

func IsWordPressPlugin(input string) bool

IsWordPressPlugin returns true if the input looks like a WP plugin reference.

func NormalizeSlug

func NormalizeSlug(input string) (string, error)

NormalizeSlug extracts the canonical "owner/repo" form from any of:

  • owner/repo
  • github.com/owner/repo[.git]
  • gitlab.com/owner/repo[.git]
  • https?://github.com/owner/repo[.git]
  • https?://gitlab.com/owner/repo[.git]
  • git@github.com:owner/repo[.git]
  • git@gitlab.com:owner/repo[.git]

Returns an error if the input does not match any recognised form. This is a thin wrapper around slugparse.Parse.

func NormalizeWPSlug

func NormalizeWPSlug(input string) (string, error)

NormalizeWPSlug extracts the plugin slug from a wp: prefix or wordpress.org URL.

func ParseLightweight

func ParseLightweight(ctx context.Context, files []*File) ([]*parser.Symbol, map[string][]string, []parser.CallSite)

ParseLightweight parses files to extract symbols, imports, and call sites without reading function bodies. Used for content-based focus filtering.

func ReleaseCloneRef added in v1.22.0

func ReleaseCloneRef(dir string) error

ReleaseCloneRef drops one reader reference for dir and removes the directory from disk (via CleanupCloneDir) only when the reference count reaches zero — i.e. no other reader is still using it. It returns the CleanupCloneDir error when it actually removes the dir, and nil otherwise: when other holders remain, when dir is empty, or when dir was never acquired / already released (an unbalanced release is a safe no-op, never a delete of an unheld dir). This is the refcounted replacement for a bare CleanupCloneDir on shared workspace clones.

func RenderTree

func RenderTree(files []*File) string

RenderTree builds a visual directory tree from a list of files and returns it as a string using box-drawing characters. Output is capped at treeMaxLines lines; a trailing summary line is added when truncated.

Example output:

├── cmd/
│   └── main.go
├── internal/
│   ├── handler.go
│   └── config.go
└── go.mod

Types

type CloneOpts

type CloneOpts struct {
	// Slug is the owner/repo slug used for directory naming.
	Slug string

	// Ref is the branch, tag, or commit SHA to check out.
	// Empty means the default branch.
	Ref string

	// DestDir is the parent directory where the clone will be placed.
	// A subdirectory named after the repo slug will be created inside.
	DestDir string

	// GithubToken is used for authenticated clones (private repos, higher rate limits).
	// Ignored when CloneURL is set (the URL already encodes credentials).
	GithubToken string

	// CloneURL is a pre-built HTTPS clone URL. When non-empty, it is used
	// directly and the GitHub-specific URL building logic is skipped.
	// Slug is still required for directory naming.
	CloneURL string

	// TokenFunc returns a fresh token for authenticated git operations.
	// When set, it is called before each refreshClone to obtain a current
	// installation token (ghs_, ~1h TTL). Overrides GithubToken for refreshes.
	// When nil, refreshClone relies on credentials embedded in .git/config.
	TokenFunc func(ctx context.Context) (string, error)
}

CloneOpts controls how a repository is cloned.

type CloneResult

type CloneResult struct {
	// LocalPath is the absolute path to the cloned repository root.
	LocalPath string

	// Ref is the actual ref checked out (may differ if HEAD was resolved)..
	Ref string
}

CloneResult contains the result of a successful clone.

func CloneRepo

func CloneRepo(ctx context.Context, opts CloneOpts) (*CloneResult, error)

CloneRepo performs a shallow git clone of the given repository into DestDir. If Ref is specified, it checks out that ref after cloning.

When a cached clone already exists, it is refreshed in-place via git fetch + reset. If the in-place refresh fails (corrupt repo, network blip, missing ref), a fresh clone is performed into a temporary sibling directory which is then atomically swapped into place (see atomicDirectorySwap).

The atomic swap guarantees that concurrent readers (e.g. WalkDir + indexParseParallel) always see either the old snapshot or the new one — never a half-deleted intermediate state that causes ENOENT during file reads.

Disk note: during the swap the workspace directory briefly holds both the old clone and the new tmp clone. If the volume is critically full (< 2× repo size free) the clone step may fail; the tmp directory is cleaned up before the error is returned.

func FetchWPPlugin

func FetchWPPlugin(ctx context.Context, opts WPPluginOpts) (*CloneResult, error)

FetchWPPlugin downloads and extracts a WordPress plugin ZIP into DestDir. Returns a CloneResult with LocalPath pointing to the extracted directory.

type File

type File struct {
	// Path is the absolute path to the file on disk.
	Path string

	// RelPath is the path relative to the repository root.
	RelPath string

	// Language is the detected programming language (e.g. "go", "python").
	Language string

	// Size is the file size in bytes.
	Size int64

	// ModTime is the file modification time.
	ModTime time.Time
}

File represents a single source file ready for parsing.

func FilterFiles

func FilterFiles(files []*File, matched map[string]bool) []*File

FilterFiles returns only files whose absolute path is in the matched set.

type IngestOpts

type IngestOpts struct {
	// Root is the repository root directory (must already be on disk).
	Root string

	// Focus limits ingestion to files matching this glob pattern, subdirectory
	// prefix, or space-separated keywords (matched case-insensitively against
	// the relative path). Empty means ingest everything.
	Focus string

	// Languages limits ingestion to files of these languages.
	// Empty means accept all supported languages.
	Languages []string

	// MaxFileBytes skips files larger than this size. 0 means no limit.
	MaxFileBytes int64

	// MaxFiles is the maximum number of files to keep from the walk.
	// 0 means use the default cap (maxWalkFiles).
	MaxFiles int

	// FollowSymlinks controls whether symlinks are followed.
	FollowSymlinks bool

	// ExcludeTests skips test files (*_test.go) when true.
	ExcludeTests bool
}

IngestOpts controls how a repository is ingested.

type IngestResult

type IngestResult struct {
	// Files is the ordered list of source files.
	Files []*File

	// Root is the repository root used for ingestion.
	Root string

	// TotalBytes is the total size of all ingested files.
	TotalBytes int64

	// SkippedCount is the number of files skipped (too large, ignored, binary).
	SkippedCount int

	// SkippedReasons tallies the number of silently-dropped files per reason.
	// Reasons: "gitignored", "test_excluded", "language_filter",
	// "oversize", "focus_excluded", "unknown_extension".
	// A file may be silently excluded (not counted in SkippedCount) for some
	// of these reasons — SkippedReasons captures all of them.
	SkippedReasons map[string]int
}

IngestResult contains all files found after filtering.

func ContentFallback

func ContentFallback(ctx context.Context, opts IngestOpts, focus string) (*IngestResult, error)

ContentFallback re-ingests a repo without focus filtering, then uses ParseLightweight + ContentFilter to find files matching focus keywords by symbol name, import path, or call site content. opts.Focus is ignored so the full repo is walked; the content focus string is supplied separately.

func IngestRepo

func IngestRepo(ctx context.Context, opts IngestOpts) (*IngestResult, error)

IngestRepo walks a local repository directory and returns all source files that match the given options.

It does NOT read file contents — content loading happens at the parse stage.

type WPPluginMeta

type WPPluginMeta struct {
	Name           string `json:"name"`
	Slug           string `json:"slug"`
	Version        string `json:"version"`
	DownloadLink   string `json:"download_link"`
	Author         string `json:"author"`
	ActiveInstalls int    `json:"active_installs"`
}

WPPluginMeta holds metadata returned by the WordPress.org Plugin API.

func FetchWPPluginMeta

func FetchWPPluginMeta(ctx context.Context, slug string) (*WPPluginMeta, error)

FetchWPPluginMeta retrieves plugin metadata from the WordPress.org API.

type WPPluginOpts

type WPPluginOpts struct {
	Slug    string
	Version string // optional; empty = latest
	DestDir string
}

WPPluginOpts controls how a WordPress plugin is fetched.

type WPSearchResponse

type WPSearchResponse struct {
	Info    wpSearchInfo     `json:"info"`
	Page    int              `json:"-"`
	Pages   int              `json:"-"`
	Plugins []WPSearchResult `json:"plugins"`
}

WPSearchResponse is the top-level response from the query_plugins API.

func SearchWPPlugins

func SearchWPPlugins(ctx context.Context, query string, perPage, page int) (*WPSearchResponse, error)

SearchWPPlugins queries the WordPress.org plugin directory.

type WPSearchResult

type WPSearchResult struct {
	Name             string `json:"name"`
	Slug             string `json:"slug"`
	Version          string `json:"version"`
	Author           string `json:"author"`
	Rating           int    `json:"rating"`
	ActiveInstalls   int    `json:"active_installs"`
	ShortDescription string `json:"short_description"`
	DownloadLink     string `json:"download_link"`
}

WPSearchResult holds one plugin from a search response.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL