README
¶
OpenSearch document parser
opensearch parses OpenSearch search response JSON into
[]*schema.Document. It supports configurable field selectors via ucfg for
content extraction and attaches standard OpenSearch metadata (_id,
_index, _score, _version) plus source hash and ID to each document.
Configuration
type Config struct {
FieldSelectors []string
FieldIgnores []string
SourceIDField string // default "source_id"
SourceHashField string // default "source_hash"
}
Usage
import "github.com/webcenter-fr/eino-ext/components/document/parser/opensearch"
parser, _ := osparser.NewParser(ctx, &osparser.Config{
FieldSelectors: []string{"content", "title"},
})
docs, err := parser.Parse(ctx, reader)
Documentation
¶
Index ¶
Constants ¶
const ( MetaKeyId = "id" MetaKeyIndex = "index" MetaScore = "score" MetaVersion = "version" // DefaultSourceIDField / DefaultSourceHashField are the metadata keys // under which the source record id and the content hash are written when // Config.SourceIDField / Config.SourceHashField are left empty. DefaultSourceIDField = "source_id" DefaultSourceHashField = "source_hash" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// FieldSelectors is a list of field selectors used to extract content
// from each search hit (e.g. "content", "fields.title").
FieldSelectors []string `validate:"omitempty" jsonschema:"description=List of field selectors to extract content"`
// FieldIgnores is the list of fields to exclude from metadata.
FieldIgnores []string `validate:"omitempty" jsonschema:"description=List of fields to ignore"`
// SourceIDField is the metadata key receiving the source record `_id`.
// Defaults to DefaultSourceIDField ("source_id").
SourceIDField string `validate:"omitempty" jsonschema:"description=Metadata key for source record _id,default=source_id"`
// SourceHashField is the metadata key receiving the content hash.
// Defaults to DefaultSourceHashField ("source_hash").
SourceHashField string `validate:"omitempty" jsonschema:"description=Metadata key for content hash,default=source_hash"`
}
Config holds the parser configuration.
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser implements parser.Parser for OpenSearch search results.
func (*Parser) ConvertHit ¶
func (p *Parser) ConvertHit(source map[string]any, hitID, hitIndex string, hitScore *float64, hitVersion *int64) (*schema.Document, error)
ConvertHit converts a raw OpenSearch source map plus its hit metadata into a schema.Document. It applies the configured field selectors, ignores, source id/hash metadata keys, and content hash computation. This is the single conversion path shared by Parse (JSON reader) and direct hit-based callers such as the OpenSearch loader.
source is the _source map. hitID, hitIndex, hitScore, hitVersion correspond to the OpenSearch _id, _index, _score, _version fields.
func (*Parser) Parse ¶
func (p *Parser) Parse(ctx context.Context, reader io.Reader, opts ...parser.Option) ([]*schema.Document, error)
Parse reads OpenSearch search results from the reader and converts each hit into a schema.Document. The search response is expected to be JSON in the format returned by OpenSearch `_search` API.
Source Files