Documentation
¶
Overview ¶
Package cmd holds the cobra root command and all subcommands for the epo-processor binary.
Overview ¶
Each subcommand in this package composes the same five-stage streaming pipeline from github.com/Qubut/epo-processor/internal/pipeline
Subcommands ¶
process — EPO XML → Parquet
Downloads every archive listed in the EPO BDDS product catalogue, unwraps nested tar/gz/zip containers in-stream, parses each <exchange-document> node with github.com/Qubut/epo-processor/internal/parse.ExtractPatentRecord, and writes batched rows to a Parquet file. Checkpointing via bbolt lets interrupted runs resume without reprocessing completed archives.
process-hupd — HUPD .tar → disk
Streams the HUPD all-years .tar from HuggingFace and writes each entry to disk using the same pipeline machinery with pipeline.NoopExtractor and pipeline.NoopSink. Requires at least one retention flag (keep_archive or keep_extracted) or returns an error.
analyze — build the EPO ↔ HUPD linked Parquet dataset
Builds a Parquet dataset where each row is an EPO patent linked to one or more HUPD records via direct citation or the simple family graph. Each row carries the HUPD JSON paths and EPO citation-category codes (X, Y, A, …), making the dataset directly usable for downstream ML tasks. Overlap statistics are printed to stdout as a JSON summary.
The HUPD ID index is held in memory (Feather Arrow scan or parallel JSON walk); the EPO file is streamed exactly once.
Pipeline architecture ¶
All subcommands share the same interface composition:
ArchiveSource → ArchiveOpener → RecordExtractor → Batch → RecordSink
Stages communicate over typed rill channels with backpressure. Cancellation propagates via context.Context. Each stage can be replaced by a Noop implementation without touching any other stage.
Shared wiring (root.go) ¶
- internal/config.LoadConfig merges YAML + env-vars.
- A slog text logger is attached to stderr (console).
- A slog JSON file logger is attached when log.log_dir is set.
- os.Signal listeners for SIGINT / SIGTERM cancel the pipeline context.
Design patterns ¶
Functional Options (pipeline.WithSource, pipeline.WithSink, …) are used to construct every pipeline. Null-Object implementations (pipeline.NoopSink, pipeline.NoopCheckpointer, etc.) eliminate nil-guard branches throughout. The Decorator pattern (pipeline.CheckpointSource) wraps sources with resume logic without modifying the underlying type.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var RootCmd = &cobra.Command{ Use: "epo-processor", Short: "Streaming EPO / HUPD patent processor", PersistentPreRunE: func(cmd *cobra.Command, _ []string) error { var err error cfg, err = config.Load(cfgFile, cmd.Flags(), effectiveBinds(cmd)) if err != nil { return fmt.Errorf("load config: %w", err) } if cfg.Log.LogDir != "" { if err := os.MkdirAll(cfg.Log.LogDir, 0o750); err != nil { return fmt.Errorf("create log directory: %w", err) } } logger, logCtl, logDone, err = applog.New(cfg.Log.LogLevel, cfg.Log.LogDir) if err != nil { return fmt.Errorf("init logger: %w", err) } slog.SetDefault(logger) logging.SetLogger(logger) return nil }, PersistentPostRunE: func(_ *cobra.Command, _ []string) error { if logDone != nil { return logDone() } return nil }, }
RootCmd is the cobra entrypoint for the epo-processor CLI.
var ( // Version is set at build time: // go build -ldflags "-X github.com/Qubut/epo-processor/cmd.Version=v1.0.0" Version = "dev" )
Functions ¶
This section is empty.
Types ¶
type CollisionCitation ¶
type CollisionCitation struct {
// CitedID is the bare US numeric identifier (e.g. "9114971"), obtained
// by stripping the "US" prefix and kind code from the EPO citation.
CitedID string `json:"cited_id" parquet:"cited_id"`
// Categories are the EPO relevance codes for this citation
// (X, Y, A, E, O, T, …) as extracted from <category> nodes.
Categories []string `json:"categories" parquet:"categories,list"`
// Paths are the on-disk paths of the matching HUPD JSON files.
Paths []string `json:"paths" parquet:"paths,list"`
}
CollisionCitation is a HUPD patent cited by an EPO patent with at least one EPO examiner category (X, Y, A, …). CitedID is US-normalised.
type DatasetRecord ¶
type DatasetRecord struct {
// EPOPatentID is the full EPO identifier (country+number+kind, e.g. "EP1234567A1").
EPOPatentID string `json:"epo_patent_id" parquet:"epo_patent_id"`
// EPOHUPDPaths are the HUPD JSON file paths for this EPO patent itself,
// populated only when it appears in the HUPD index.
EPOHUPDPaths []string `json:"epo_hupd_paths" parquet:"epo_hupd_paths,list"`
// CitedHUPD lists every HUPD patent cited in the EPO examination report
// with at least one category annotation. Guaranteed non-empty.
CitedHUPD []CollisionCitation `json:"cited_hupd" parquet:"cited_hupd,list"`
// FamilyHUPD lists HUPD patents that belong to the same simple family
// as the EPO patent but were not directly cited.
FamilyHUPD []HUPDMember `json:"family_hupd" parquet:"family_hupd,list"`
}
DatasetRecord is one row in the EPO↔HUPD dataset (Parquet or JSONL). EPOHUPDPaths is empty when the EPO patent is not itself in HUPD. CitedHUPD is guaranteed non-empty (len ≥ minCollisions).
type HUPDMember ¶
type HUPDMember struct {
// ID is the US-normalised family-member identifier.
ID string `json:"id" parquet:"id"`
// Paths are the on-disk paths of the matching HUPD JSON files.
Paths []string `json:"paths" parquet:"paths,list"`
}
HUPDMember is a HUPD patent found in the EPO patent's family graph.
type Report ¶
type Report struct {
EPOFile string `json:"epo_file"` // path of the EPO input file
HUPDDir string `json:"hupd_dir"` // root of the on-disk HUPD extraction
// EPORecords is the total number of EPO patent records scanned.
EPORecords int `json:"epo_records"`
// HUPDTotal is the number of unique normalised HUPD IDs in the index.
HUPDTotal int `json:"hupd_total"`
// OverlapDirect is the count of HUPD IDs that matched an EPO patent_id directly.
OverlapDirect int `json:"overlap_direct"`
// OverlapFamilyOnly is the count of HUPD IDs reached only via the EPO family graph.
OverlapFamilyOnly int `json:"overlap_family_only"`
// OverlapTotal is OverlapDirect + OverlapFamilyOnly.
OverlapTotal int `json:"overlap_total"`
// HUPDCoveragePct is OverlapTotal / HUPDTotal × 100.
HUPDCoveragePct float64 `json:"hupd_coverage_pct"`
// Dataset fields are populated only when --out is set.
DatasetRows int64 `json:"dataset_rows,omitempty"`
DatasetFile string `json:"dataset_file,omitempty"`
}
Report is the analyze command's structured output, printed as JSON.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Command epo-processor is a zero-copy, resumable streaming ETL for European Patent Office (EPO) bulk data with built-in EPO ↔ HUPD overlap analysis.
|
Command epo-processor is a zero-copy, resumable streaming ETL for European Patent Office (EPO) bulk data with built-in EPO ↔ HUPD overlap analysis. |