Documentation
¶
Overview ¶
Package cli builds the ccrawl command tree on top of the ccrawl library and the any-cli/kit framework. The record-stream commands are kit operations (declared once, exposed as CLI, HTTP, and MCP); the byte-fetch, columnar, and interactive commands are escape-hatch kit.Command commands that share the same run state through the context.
Index ¶
- Variables
- func FetchRobotsForHost(ctx context.Context, h *ccrawl.HTTPClient, rc *ccrawl.RobotsCache, ...) *ccrawl.RobotsEntry
- func NewApp() *kit.App
- type App
- type ContentExtractResult
- type CrawlStatus
- type DegreeRecord
- type FetchRecord
- type IndexBuildResult
- type LinkRecord
- type QualityReport
- type Row
- type SearchHit
- type SeedRecord
- type ServeResult
Constants ¶
This section is empty.
Variables ¶
var ( Version = "dev" Commit = "none" Date = "unknown" )
Build metadata, set via -ldflags at release time.
Functions ¶
func FetchRobotsForHost ¶ added in v0.2.0
func FetchRobotsForHost(ctx context.Context, h *ccrawl.HTTPClient, rc *ccrawl.RobotsCache, host, scheme string) *ccrawl.RobotsEntry
FetchRobotsForHost retrieves (with caching) the robots.txt for a host.
Types ¶
type App ¶
type App struct {
Cfg ccrawl.Config
HTTP *ccrawl.HTTPClient
Cache *ccrawl.Cache
Out *render.Renderer
Limit int
Workers int
UseLibrary bool
LibraryDir string
// contains filtered or unexported fields
}
App carries the resolved configuration and shared clients for a command run. kit builds one per run through the client factory registered in Root, then hands it to every operation (injected) and every escape-hatch command (fetched from the run context with appFromCtx).
func (*App) AllCrawls ¶
AllCrawls returns the crawl IDs to operate over when -c all/year is given, newest first, otherwise the single resolved crawl.
type ContentExtractResult ¶ added in v0.2.0
type ContentExtractResult struct {
URL string `json:"url" table:"url"`
CanonURL string `json:"canon_url,omitempty" table:"canon_url"`
Title string `json:"title" table:"title"`
Description string `json:"description,omitempty" table:"description"`
Language string `json:"language" table:"language"`
WordCount int `json:"word_count" table:"word_count"`
DocID uint64 `json:"doc_id" table:"doc_id"`
Snippet string `json:"snippet" table:"snippet"`
}
ContentExtractResult is the output of content extraction.
type CrawlStatus ¶ added in v0.2.0
type CrawlStatus struct {
Tier int `json:"tier" table:"tier"`
PagesPerDay int64 `json:"pages_per_day" table:"pages_per_day"`
TargetHosts int64 `json:"target_hosts" table:"target_hosts"`
IntervalHours int `json:"interval_hours" table:"interval_hours"`
}
CrawlStatus reports the crawl budget allocation across tiers.
type DegreeRecord ¶ added in v0.2.0
type DegreeRecord struct {
Host string `json:"host" kit:"id" table:"host"`
NodeID int64 `json:"node_id" table:"node_id"`
InDegree int32 `json:"in_degree" table:"in_degree"`
OutDegree int32 `json:"out_degree" table:"out_degree"`
}
DegreeRecord is one row of per-host in/out degree output.
type FetchRecord ¶ added in v0.2.0
type FetchRecord struct {
URL string `json:"url" table:"url"`
FinalURL string `json:"final_url,omitempty" table:"final_url"`
Status int `json:"status" table:"status"`
ContentType string `json:"content_type" table:"content_type"`
Digest string `json:"digest" table:"digest"`
BodySize int `json:"body_size" table:"body_size"`
LinkCount int `json:"link_count" table:"link_count"`
FetchedAt string `json:"fetched_at" table:"fetched_at"`
}
FetchRecord is the result of crawling one URL.
type IndexBuildResult ¶ added in v0.2.0
type IndexBuildResult struct {
IndexDir string `json:"index_dir" table:"index_dir"`
DocsAdded int `json:"docs_added" table:"docs_added"`
Terms int `json:"terms" table:"terms"`
}
IndexBuildResult reports the outcome of building an index.
type LinkRecord ¶ added in v0.2.0
type LinkRecord struct {
URL string `json:"url" kit:"id" table:"url"`
Host string `json:"host" table:"host"`
}
LinkRecord is one outbound link extracted from a page.
type QualityReport ¶ added in v0.2.0
type QualityReport struct {
URL string `json:"url" table:"url"`
WordCount int `json:"word_count" table:"word_count"`
TitleLength int `json:"title_length" table:"title_length"`
HasMainContent bool `json:"has_main_content" table:"has_main_content"`
SpamScore float64 `json:"spam_score" table:"spam_score"`
IsParked bool `json:"is_parked" table:"is_parked"`
}
QualityReport is the output of content quality analysis.
type Row ¶
Row is one output record: an ordered set of named columns plus the original value rendered by json/jsonl and templates. It is kit's render.Record, so the row builders feed straight into the shared renderer with no per-format code.
type SearchHit ¶ added in v0.2.0
type SearchHit struct {
DocID uint64 `json:"doc_id" table:"doc_id"`
Score float64 `json:"score" table:"score"`
URL string `json:"url" table:"url"`
Title string `json:"title" table:"title"`
Snippet string `json:"snippet" table:"snippet"`
}
SearchHit is one result from index search.
type SeedRecord ¶ added in v0.2.0
type SeedRecord struct {
Host string `json:"host" table:"host"`
URL string `json:"url" kit:"id" table:"url"`
Tier int `json:"tier" table:"tier"`
Priority float32 `json:"priority" table:"priority"`
}
SeedRecord is one crawl seed URL derived from the host rank table.
type ServeResult ¶ added in v0.2.0
type ServeResult struct {
Addr string `json:"addr" table:"addr"`
IndexDir string `json:"index_dir,omitempty" table:"index_dir"`
}
ServeResult is emitted once when the server starts.
Source Files