ingest

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 8, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package ingest handles repository ingestion: cloning remote repos, walking the local filesystem, filtering files by language/size/gitignore rules, and producing a normalized file list for downstream parsing.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CleanupCloneDir

func CleanupCloneDir(localPath string) error

CleanupCloneDir removes a cloned repository directory from disk.

func ContentFilter

func ContentFilter(focus string, symbols []*parser.Symbol, imports map[string][]string, calls []parser.CallSite) map[string]bool

ContentFilter returns file paths where ANY keyword from focus matches a symbol name, import path, or call site (case-insensitive, OR logic). Returns nil when focus is empty or has no keywords.

func DetectLanguage

func DetectLanguage(filename string) string

DetectLanguage returns the programming language for the given filename based on its extension. Delegates to parser.DetectLanguageFromPath so there is a single source of truth — the handler registry.

func IgnoredDirNames

func IgnoredDirNames() []string

IgnoredDirNames returns the alphabetically sorted list of directory names that the indexer skips. Used by tool handlers to surface a hint to callers when a symbol search returns zero results — the symbol may live in an excluded path on disk.

func IsKeywordFocus

func IsKeywordFocus(focus string) bool

IsKeywordFocus returns true when focus looks like a keyword query rather than a path or glob pattern. Heuristic: contains spaces.

func IsRemote

func IsRemote(input string) bool

IsRemote returns true if the input looks like a GitHub slug or URL rather than a local filesystem path.

func IsWordPressPlugin

func IsWordPressPlugin(input string) bool

IsWordPressPlugin returns true if the input looks like a WP plugin reference.

func NormalizeSlug

func NormalizeSlug(input string) (string, error)

NormalizeSlug extracts the canonical "owner/repo" form from any of:

  • owner/repo
  • github.com/owner/repo[.git]
  • gitlab.com/owner/repo[.git]
  • https?://github.com/owner/repo[.git]
  • https?://gitlab.com/owner/repo[.git]
  • git@github.com:owner/repo[.git]
  • git@gitlab.com:owner/repo[.git]

Returns an error if the input does not match any recognised form. This is a thin wrapper around slugparse.Parse.

func NormalizeWPSlug

func NormalizeWPSlug(input string) (string, error)

NormalizeWPSlug extracts the plugin slug from a wp: prefix or wordpress.org URL.

func ParseLightweight

func ParseLightweight(ctx context.Context, files []*File) ([]*parser.Symbol, map[string][]string, []parser.CallSite)

ParseLightweight parses files to extract symbols, imports, and call sites without reading function bodies. Used for content-based focus filtering.

func RenderTree

func RenderTree(files []*File) string

RenderTree builds a visual directory tree from a list of files and returns it as a string using box-drawing characters. Output is capped at treeMaxLines lines; a trailing summary line is added when truncated.

Example output:

├── cmd/
│   └── main.go
├── internal/
│   ├── handler.go
│   └── config.go
└── go.mod

Types

type CloneOpts

type CloneOpts struct {
	// Slug is the owner/repo slug used for directory naming.
	Slug string

	// Ref is the branch, tag, or commit SHA to check out.
	// Empty means the default branch.
	Ref string

	// DestDir is the parent directory where the clone will be placed.
	// A subdirectory named after the repo slug will be created inside.
	DestDir string

	// GithubToken is used for authenticated clones (private repos, higher rate limits).
	// Ignored when CloneURL is set (the URL already encodes credentials).
	GithubToken string

	// CloneURL is a pre-built HTTPS clone URL. When non-empty, it is used
	// directly and the GitHub-specific URL building logic is skipped.
	// Slug is still required for directory naming.
	CloneURL string
}

CloneOpts controls how a repository is cloned.

type CloneResult

type CloneResult struct {
	// LocalPath is the absolute path to the cloned repository root.
	LocalPath string

	// Ref is the actual ref checked out (may differ if HEAD was resolved).
	Ref string
}

CloneResult contains the result of a successful clone.

func CloneRepo

func CloneRepo(ctx context.Context, opts CloneOpts) (*CloneResult, error)

CloneRepo performs a shallow git clone of the given repository into DestDir. If Ref is specified, it checks out that ref after cloning.

func FetchWPPlugin

func FetchWPPlugin(ctx context.Context, opts WPPluginOpts) (*CloneResult, error)

FetchWPPlugin downloads and extracts a WordPress plugin ZIP into DestDir. Returns a CloneResult with LocalPath pointing to the extracted directory.

type File

type File struct {
	// Path is the absolute path to the file on disk.
	Path string

	// RelPath is the path relative to the repository root.
	RelPath string

	// Language is the detected programming language (e.g. "go", "python").
	Language string

	// Size is the file size in bytes.
	Size int64

	// ModTime is the file modification time.
	ModTime time.Time
}

File represents a single source file ready for parsing.

func FilterFiles

func FilterFiles(files []*File, matched map[string]bool) []*File

FilterFiles returns only files whose absolute path is in the matched set.

type IngestOpts

type IngestOpts struct {
	// Root is the repository root directory (must already be on disk).
	Root string

	// Focus limits ingestion to files matching this glob pattern, subdirectory
	// prefix, or space-separated keywords (matched case-insensitively against
	// the relative path). Empty means ingest everything.
	Focus string

	// Languages limits ingestion to files of these languages.
	// Empty means accept all supported languages.
	Languages []string

	// MaxFileBytes skips files larger than this size. 0 means no limit.
	MaxFileBytes int64

	// FollowSymlinks controls whether symlinks are followed.
	FollowSymlinks bool

	// ExcludeTests skips test files (*_test.go) when true.
	ExcludeTests bool
}

IngestOpts controls how a repository is ingested.

type IngestResult

type IngestResult struct {
	// Files is the ordered list of source files.
	Files []*File

	// Root is the repository root used for ingestion.
	Root string

	// TotalBytes is the total size of all ingested files.
	TotalBytes int64

	// SkippedCount is the number of files skipped (too large, ignored, binary).
	SkippedCount int
}

IngestResult contains all files found after filtering.

func ContentFallback

func ContentFallback(ctx context.Context, root string, langs []string, maxFileBytes int64, focus string) (*IngestResult, error)

ContentFallback re-ingests a repo without focus filtering, then uses ParseLightweight + ContentFilter to find files matching focus keywords by symbol name, import path, or call site content.

func IngestRepo

func IngestRepo(ctx context.Context, opts IngestOpts) (*IngestResult, error)

IngestRepo walks a local repository directory and returns all source files that match the given options.

It does NOT read file contents — content loading happens at the parse stage.

type WPPluginMeta

type WPPluginMeta struct {
	Name           string `json:"name"`
	Slug           string `json:"slug"`
	Version        string `json:"version"`
	DownloadLink   string `json:"download_link"`
	Author         string `json:"author"`
	ActiveInstalls int    `json:"active_installs"`
}

WPPluginMeta holds metadata returned by the WordPress.org Plugin API.

func FetchWPPluginMeta

func FetchWPPluginMeta(ctx context.Context, slug string) (*WPPluginMeta, error)

FetchWPPluginMeta retrieves plugin metadata from the WordPress.org API.

type WPPluginOpts

type WPPluginOpts struct {
	Slug    string
	Version string // optional; empty = latest
	DestDir string
}

WPPluginOpts controls how a WordPress plugin is fetched.

type WPSearchResponse

type WPSearchResponse struct {
	Info    wpSearchInfo     `json:"info"`
	Page    int              `json:"-"`
	Pages   int              `json:"-"`
	Plugins []WPSearchResult `json:"plugins"`
}

WPSearchResponse is the top-level response from the query_plugins API.

func SearchWPPlugins

func SearchWPPlugins(ctx context.Context, query string, perPage, page int) (*WPSearchResponse, error)

SearchWPPlugins queries the WordPress.org plugin directory.

type WPSearchResult

type WPSearchResult struct {
	Name             string `json:"name"`
	Slug             string `json:"slug"`
	Version          string `json:"version"`
	Author           string `json:"author"`
	Rating           int    `json:"rating"`
	ActiveInstalls   int    `json:"active_installs"`
	ShortDescription string `json:"short_description"`
	DownloadLink     string `json:"download_link"`
}

WPSearchResult holds one plugin from a search response.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL