Documentation
¶
Index ¶
- func GetLoaderImplSpecificOptions[T any](base *T, opts ...LoaderOption) *T
- func GetTransformerImplSpecificOptions[T any](base *T, opts ...TransformerOption) *T
- type Loader
- type LoaderCallbackInput
- type LoaderCallbackOutput
- type LoaderOption
- type LoaderOptions
- type Source
- type Transformer
- type TransformerCallbackInput
- type TransformerCallbackOutput
- type TransformerOption
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GetLoaderImplSpecificOptions ¶
func GetLoaderImplSpecificOptions[T any](base *T, opts ...LoaderOption) *T
GetLoaderImplSpecificOptions provides Loader author the ability to extract their own custom options from the unified LoaderOption type. T: the type of the impl specific options struct. This function should be used within the Loader implementation's Load function. It is recommended to provide a base T as the first argument, within which the Loader author can provide default values for the impl specific options. eg.
myOption := &MyOption{
Field1: "default_value",
}
myOption := loader.GetLoaderImplSpecificOptions(myOption, opts...)
func GetTransformerImplSpecificOptions ¶
func GetTransformerImplSpecificOptions[T any](base *T, opts ...TransformerOption) *T
GetTransformerImplSpecificOptions provides Transformer author the ability to extract their own custom options from the unified TransformerOption type. T: the type of the impl specific options struct. This function should be used within the Transformer implementation's Transform function. It is recommended to provide a base T as the first argument, within which the Transformer author can provide default values for the impl specific options. eg.
myOption := &MyOption{
Field1: "default_value",
}
myOption := transformer.GetTransformerImplSpecificOptions(myOption, opts...)
Types ¶
type Loader ¶
type Loader interface {
Load(ctx context.Context, src Source, opts ...LoaderOption) ([]*schema.Document, error)
}
Loader is a document loader.
type LoaderCallbackInput ¶
type LoaderCallbackInput struct {
// Source is the source of the documents.
Source Source
// Extra is the extra information for the callback.
Extra map[string]any
}
LoaderCallbackInput is the input for the loader callback.
func ConvLoaderCallbackInput ¶
func ConvLoaderCallbackInput(src callbacks.CallbackInput) *LoaderCallbackInput
ConvLoaderCallbackInput converts the callback input to the loader callback input.
type LoaderCallbackOutput ¶
type LoaderCallbackOutput struct {
// Source is the source of the documents.
Source Source
// Docs is the documents to be loaded.
Docs []*schema.Document
// Extra is the extra information for the callback.
Extra map[string]any
}
LoaderCallbackOutput is the output for the loader callback.
func ConvLoaderCallbackOutput ¶
func ConvLoaderCallbackOutput(src callbacks.CallbackOutput) *LoaderCallbackOutput
ConvLoaderCallbackOutput converts the callback output to the loader callback output.
type LoaderOption ¶
type LoaderOption struct {
// contains filtered or unexported fields
}
LoaderOption defines call option for Loader component, which is part of the component interface signature. Each Loader implementation could define its own options struct and option funcs within its own package, then wrap the impl specific option funcs into this type, before passing to Load.
func WithParserOptions ¶ added in v0.3.54
func WithParserOptions(opts ...parser.Option) LoaderOption
func WrapLoaderImplSpecificOptFn ¶
func WrapLoaderImplSpecificOptFn[T any](optFn func(*T)) LoaderOption
WrapLoaderImplSpecificOptFn wraps the impl specific option functions into LoaderOption type. T: the type of the impl specific options struct. Loader implementations are required to use this function to convert its own option functions into the unified LoaderOption type. For example, if the Loader impl defines its own options struct:
type customOptions struct {
conf string
}
Then the impl needs to provide an option function as such:
func WithConf(conf string) Option {
return WrapLoaderImplSpecificOptFn(func(o *customOptions) {
o.conf = conf
}
}
type LoaderOptions ¶ added in v0.3.54
func GetLoaderCommonOptions ¶ added in v0.3.54
func GetLoaderCommonOptions(base *LoaderOptions, opts ...LoaderOption) *LoaderOptions
GetLoaderCommonOptions extract loader Options from Option list, optionally providing a base Options with default values.
type Source ¶
type Source struct {
URI string
}
Source is a document source. e.g. https://www.bytedance.com/docx/xxxx, https://xxx.xxx.xxx/xx.pdf. make sure the URI can be reached by service.
type Transformer ¶
type Transformer interface {
Transform(ctx context.Context, src []*schema.Document, opts ...TransformerOption) ([]*schema.Document, error)
}
Transformer is to convert documents, such as split or filter.
type TransformerCallbackInput ¶
type TransformerCallbackInput struct {
// Input is the input documents.
Input []*schema.Document
// Extra is the extra information for the callback.
Extra map[string]any
}
TransformerCallbackInput is the input for the transformer callback.
func ConvTransformerCallbackInput ¶
func ConvTransformerCallbackInput(src callbacks.CallbackInput) *TransformerCallbackInput
ConvTransformerCallbackInput converts the callback input to the transformer callback input.
type TransformerCallbackOutput ¶
type TransformerCallbackOutput struct {
// Output is the output documents.
Output []*schema.Document
// Extra is the extra information for the callback.
Extra map[string]any
}
TransformerCallbackOutput is the output for the transformer callback.
func ConvTransformerCallbackOutput ¶
func ConvTransformerCallbackOutput(src callbacks.CallbackOutput) *TransformerCallbackOutput
ConvTransformerCallbackOutput converts the callback output to the transformer callback output.
type TransformerOption ¶
type TransformerOption struct {
// contains filtered or unexported fields
}
TransformerOption defines call option for Transformer component, which is part of the component interface signature. Each Transformer implementation could define its own options struct and option funcs within its own package, then wrap the impl specific option funcs into this type, before passing to Transform.
func WrapTransformerImplSpecificOptFn ¶
func WrapTransformerImplSpecificOptFn[T any](optFn func(*T)) TransformerOption
WrapTransformerImplSpecificOptFn wraps the impl specific option functions into TransformerOption type. T: the type of the impl specific options struct. Transformer implementations are required to use this function to convert its own option functions into the unified TransformerOption type. For example, if the Transformer impl defines its own options struct:
type customOptions struct {
conf string
}
Then the impl needs to provide an option function as such:
func WithConf(conf string) TransformerOption {
return WrapTransformerImplSpecificOptFn(func(o *customOptions) {
o.conf = conf
}
}
.
Source Files
Directories