Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type None ¶
type None struct {
// contains filtered or unexported fields
}
func NewNone ¶
func NewNone() *None
NewNone creates a Source that yields a single document from the given reader.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader wraps an io.Reader as a single-document Source. This is useful for integrating with spool.ForEach or other scenarios where you have an io.Reader and want to use it with Pipeline.
Example with spool:
spool.ForEach(ctx, root, func(ctx, path, r, w) error {
source := source.NewReaderSource(path, r)
sink := sink.NewWriterSink(w)
return pipeline.Run(ctx, source, sink)
})
type Storage ¶ added in v0.2.0
type Storage struct {
// contains filtered or unexported fields
}
Storage traverses a filesystem directory and yields documents for each file. It uses fs.WalkDir in a background goroutine for progressive file discovery. The goroutine is lazily initialized on the first call to Next(). Directories are skipped - only regular files are yielded as documents.
func NewStorage ¶ added in v0.2.0
NewStorage creates a source that walks through a filesystem directory progressively. The fs.FS parameter should support fs.ReadDirFS (e.g., from lfs.New or stream.NewStorage). The dir parameter specifies the starting directory path. For directories, the path should end with `/` as per stream library conventions.
The walk operation is lazily initialized on the first call to Next(), running in a background goroutine that feeds file paths through a buffered channel. This means:
- No startup delay: NewStorage returns immediately
- Directory validation is deferred until Next() is called
- Memory efficient: Only buffer size (100) paths held in memory at a time
- Works with both local filesystem (lfs) and S3 (stream)
func (*Storage) Close ¶ added in v0.2.0
Close signals the background walker to stop and drains the path channel to prevent goroutine leaks. It's safe to call Close() even if Next() was never called.
Source Files