cli

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 23, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

Documentation

Overview

Package cli builds the ccrawl command tree on top of the ccrawl library and the any-cli/kit framework. The record-stream commands are kit operations (declared once, exposed as CLI, HTTP, and MCP); the byte-fetch, columnar, and interactive commands are escape-hatch kit.Command commands that share the same run state through the context.

Index

Constants

This section is empty.

Variables

View Source 
var (
	Version = "dev"
	Commit  = "none"
	Date    = "unknown"
)

Build metadata, set via -ldflags at release time.

Functions

func FetchRobotsForHost added in v0.2.0 

func FetchRobotsForHost(ctx context.Context, h *ccrawl.HTTPClient, rc *ccrawl.RobotsCache, host, scheme string) *ccrawl.RobotsEntry

FetchRobotsForHost retrieves (with caching) the robots.txt for a host.

func NewApp added in v0.2.0 

func NewApp() *kit.App

NewApp assembles the kit application: identity, the ccrawl-specific global flags, the client factory that builds the shared engine, the record-stream operations, and the escape-hatch commands.

Types

type App 

type App struct {
	Cfg   ccrawl.Config
	HTTP  *ccrawl.HTTPClient
	Cache *ccrawl.Cache
	Out   *render.Renderer

	Limit      int
	Workers    int
	UseLibrary bool
	LibraryDir string
	// contains filtered or unexported fields
}

App carries the resolved configuration and shared clients for a command run. kit builds one per run through the client factory registered in Root, then hands it to every operation (injected) and every escape-hatch command (fetched from the run context with appFromCtx).

func (*App) AllCrawls 

func (a *App) AllCrawls(ctx context.Context) ([]string, error)

AllCrawls returns the crawl IDs to operate over when -c all/year is given, newest first, otherwise the single resolved crawl.

func (*App) Crawl 

func (a *App) Crawl(ctx context.Context) (string, error)

Crawl resolves the crawl reference once and caches the canonical ID.

func (*App) Library 

func (a *App) Library(ctx context.Context) (ccrawl.Library, error)

Library resolves the crawl ID and returns the dataset library rooted at the configured library dir for that crawl.

type ContentExtractResult added in v0.2.0 

type ContentExtractResult struct {
	URL         string `json:"url" table:"url"`
	CanonURL    string `json:"canon_url,omitempty" table:"canon_url"`
	Title       string `json:"title" table:"title"`
	Description string `json:"description,omitempty" table:"description"`
	Language    string `json:"language" table:"language"`
	WordCount   int    `json:"word_count" table:"word_count"`
	DocID       uint64 `json:"doc_id" table:"doc_id"`
	Snippet     string `json:"snippet" table:"snippet"`
}

ContentExtractResult is the output of content extraction.

type CrawlStatus added in v0.2.0 

type CrawlStatus struct {
	Tier          int   `json:"tier" table:"tier"`
	PagesPerDay   int64 `json:"pages_per_day" table:"pages_per_day"`
	TargetHosts   int64 `json:"target_hosts" table:"target_hosts"`
	IntervalHours int   `json:"interval_hours" table:"interval_hours"`
}

CrawlStatus reports the crawl budget allocation across tiers.

type DegreeRecord added in v0.2.0 

type DegreeRecord struct {
	Host      string `json:"host" kit:"id" table:"host"`
	NodeID    int64  `json:"node_id" table:"node_id"`
	InDegree  int32  `json:"in_degree" table:"in_degree"`
	OutDegree int32  `json:"out_degree" table:"out_degree"`
}

DegreeRecord is one row of per-host in/out degree output.

type FetchRecord added in v0.2.0 

type FetchRecord struct {
	URL         string `json:"url" table:"url"`
	FinalURL    string `json:"final_url,omitempty" table:"final_url"`
	Status      int    `json:"status" table:"status"`
	ContentType string `json:"content_type" table:"content_type"`
	Digest      string `json:"digest" table:"digest"`
	BodySize    int    `json:"body_size" table:"body_size"`
	LinkCount   int    `json:"link_count" table:"link_count"`
	FetchedAt   string `json:"fetched_at" table:"fetched_at"`
}

FetchRecord is the result of crawling one URL.

type IndexBuildResult added in v0.2.0 

type IndexBuildResult struct {
	IndexDir  string `json:"index_dir" table:"index_dir"`
	DocsAdded int    `json:"docs_added" table:"docs_added"`
	Terms     int    `json:"terms" table:"terms"`
}

IndexBuildResult reports the outcome of building an index.

type LinkRecord added in v0.2.0 

type LinkRecord struct {
	URL  string `json:"url" kit:"id" table:"url"`
	Host string `json:"host" table:"host"`
}

LinkRecord is one outbound link extracted from a page.

type QualityReport added in v0.2.0 

type QualityReport struct {
	URL            string  `json:"url" table:"url"`
	WordCount      int     `json:"word_count" table:"word_count"`
	TitleLength    int     `json:"title_length" table:"title_length"`
	HasMainContent bool    `json:"has_main_content" table:"has_main_content"`
	SpamScore      float64 `json:"spam_score" table:"spam_score"`
	IsParked       bool    `json:"is_parked" table:"is_parked"`
}

QualityReport is the output of content quality analysis.

type Row 

type Row = render.Record

Row is one output record: an ordered set of named columns plus the original value rendered by json/jsonl and templates. It is kit's render.Record, so the row builders feed straight into the shared renderer with no per-format code.

type SearchHit added in v0.2.0 

type SearchHit struct {
	DocID   uint64  `json:"doc_id" table:"doc_id"`
	Score   float64 `json:"score" table:"score"`
	URL     string  `json:"url" table:"url"`
	Title   string  `json:"title" table:"title"`
	Snippet string  `json:"snippet" table:"snippet"`
}

SearchHit is one result from index search.

type SeedRecord added in v0.2.0 

type SeedRecord struct {
	Host     string  `json:"host" table:"host"`
	URL      string  `json:"url" kit:"id" table:"url"`
	Tier     int     `json:"tier" table:"tier"`
	Priority float32 `json:"priority" table:"priority"`
}

SeedRecord is one crawl seed URL derived from the host rank table.

type ServeResult added in v0.2.0 

type ServeResult struct {
	Addr     string `json:"addr" table:"addr"`
	IndexDir string `json:"index_dir,omitempty" table:"index_dir"`
}

ServeResult is emitted once when the server starts.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.