Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ReaderDocs = docs.FieldCommon(
"codec", "The way in which the bytes of consumed files are converted into messages, codecs are useful for specifying how large files might be processed in small chunks rather than loading it all in memory. It's possible to consume lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter.", "lines", "delim:\t", "delim:foobar",
).HasAnnotatedOptions(
"auto", "EXPERIMENTAL: Attempts to derive a codec for each file based on information such as the extension. For example, a .tar.gz file would be consumed with the tar-gzip codec. Defaults to all-bytes.",
"all-bytes", "Consume the entire file as a single binary message.",
"csv", "Consume structured rows as comma separated values, the first row must be a header row.",
"csv-gzip", "Consume structured rows as comma separated values from a gzip compressed file, the first row must be a header row.",
"delim:x", "Consume the file in segments divided by a custom delimiter.",
"chunker:x", "Consume the file in chunks of a given number of bytes.",
"lines", "Consume the file in segments divided by linebreaks.",
"tar", "Parse the file as a tar archive, and consume each file of the archive as a message.",
"tar-gzip", "Parse the file as a gzip compressed tar archive, and consume each file of the archive as a message.",
)
ReaderDocs is a static field documentation for input codecs.
var WriterDocs = docs.FieldCommon(
"codec", "The way in which the bytes of messages should be written out into the output file. It's possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter.", "lines", "delim:\t", "delim:foobar",
).HasAnnotatedOptions(
"all-bytes", "Write the message to the file in full. If the file already exists the old content is deleted.",
"append", "Append messages to the file.",
"lines", "Append messages to the file followed by a line break.",
"delim:x", "Append messages to the file followed by a custom delimiter.",
)
WriterDocs is a static field documentation for output codecs.
Functions ¶
func GetWriter ¶
func GetWriter(codec string) (WriterConstructor, WriterConfig, error)
GetWriter returns a constructor that creates write codecs.
Types ¶
type Reader ¶
type Reader interface {
Next(context.Context) (types.Part, ReaderAckFn, error)
Close(context.Context) error
}
Reader is a codec type that reads message parts from a source.
type ReaderAckFn ¶
ReaderAckFn is a function provided to a reader codec that it should call once the underlying io.ReadCloser is fully consumed.
type ReaderConfig ¶
type ReaderConfig struct {
MaxScanTokenSize int
}
ReaderConfig is a general configuration struct that covers all reader codecs.
func NewReaderConfig ¶
func NewReaderConfig() ReaderConfig
NewReaderConfig creates a reader configuration with default values.
type ReaderConstructor ¶
type ReaderConstructor func(string, io.ReadCloser, ReaderAckFn) (Reader, error)
ReaderConstructor creates a reader from a filename, an io.ReadCloser and an ack func which is called by the reader once the io.ReadCloser is finished with. The filename can be empty and is usually ignored, but might be necessary for certain codecs.
func GetReader ¶
func GetReader(codec string, conf ReaderConfig) (ReaderConstructor, error)
GetReader returns a constructor that creates reader codecs.
type Writer ¶
type Writer interface {
Write(context.Context, types.Part) error
Close(context.Context) error
// TODO V4: Remove this, we only have it in place in order to satisfy the
// relatively dodgy empty line at end of batch behaviour.
EndBatch() error
}
Writer is a codec type that reads message parts from a source.
type WriterConfig ¶
WriterConfig contains custom configuration specific to a codec describing how handles should be provided.
type WriterConstructor ¶
type WriterConstructor func(io.WriteCloser) (Writer, error)
WriterConstructor creates a writer from an io.WriteCloser.
Source Files