Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewFile ¶
NewFile creates a source that reads files from the given filesystem. The fs.FS parameter is provided by the client (e.g., lfs.FS, stream.FS). Paths are passed as-is to fs.Open() - the client is responsible for proper formatting.
func NewReaderJSON ¶
NewReaderJSON creates a Source that yields a single document from the given reader with content type application/json. Use this when the reader contains structured JSON data.
Types ¶
type FS ¶
type FS struct {
// contains filtered or unexported fields
}
FS traverses a filesystem directory and yields documents for each file. It uses fs.WalkDir in a background goroutine for progressive file discovery. The goroutine is lazily initialized on the first call to Next(). Directories are skipped - only regular files are yielded as documents.
func NewFS ¶
NewFS creates a source that walks through a filesystem directory progressively. The fs.FS parameter should support fs.ReadDirFS (e.g., from lfs.New or stream.NewFS). The dir parameter specifies the starting directory path. For directories, the path should end with `/` as per stream library conventions.
The walk operation is lazily initialized on the first call to Next(), running in a background goroutine that feeds file paths through a buffered channel. This means:
- No startup delay: NewFS returns immediately
- Directory validation is deferred until Next() is called
- Memory efficient: Only buffer size (100) paths held in memory at a time
- Works with both local filesystem (lfs) and S3 (stream)
type File ¶
type File struct {
// contains filtered or unexported fields
}
File reads one or more files sequentially from a filesystem. A single file is a special case where paths contains one element.
type None ¶
type None struct {
// contains filtered or unexported fields
}
func NewNone ¶
func NewNone() *None
NewNone creates a Source that yields a single document from the given reader.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader wraps an io.Reader as a single-document Source. This is useful for integrating with spool.ForEach or other scenarios where you have an io.Reader and want to use it with Pipeline.
Example with spool:
spool.ForEach(ctx, root, func(ctx, path, r, w) error {
source := source.NewReaderSource(path, r)
sink := sink.NewWriterSink(w)
return pipeline.Run(ctx, source, sink)
})
Source Files