Documentation
¶
Overview ¶
Package streamio provides the input and output plumbing shared by commands that move payloads too large to hold in memory.
Everything here returns errors rather than terminating the process, so the decision to exit stays with the command layer. That is the difference from the older helpers in pkg/cli, which call cli.ExitWithError from inside the read and are therefore unusable from anywhere that wants to recover.
Index ¶
- Constants
- Variables
- func Measurable(r io.Reader) bool
- func OpenFile(path string) (io.Reader, func(), error)
- func OpenSeekable(path string) (*os.File, func(), error)
- func PipeReader(in *os.File) (io.Reader, bool, error)
- func Seekable(r io.Reader) (io.Seeker, int64, bool)
- func Spool(r io.Reader) (*os.File, func(), error)
- type OutputFile
Constants ¶
const PipeBufferSize = 1024 * 1024
PipeBufferSize is the window PipeReader buffers over a pipe — generous enough that a typical CLI payload is served from a single read.
Variables ¶
var ErrNoInput = errors.New("no input provided")
ErrNoInput reports that a command was given neither a file argument nor a non-empty pipe. It is distinct from a failure to open a named file, which callers report differently.
var ErrOutputFileFinished = errors.New("streamio: output file already committed or discarded")
ErrOutputFileFinished reports that Commit or Cleanup was called on an OutputFile that had already been committed or discarded.
Functions ¶
func Measurable ¶ added in v0.38.1
Measurable reports whether the SDK can size r without reading it, and so whether the archive gets the compact layout. See handlers.Handler's Encrypt.
func OpenFile ¶ added in v0.38.1
OpenFile opens path for reading, ready to hand to the SDK as-is.
The file stays measurable, except when its stat cannot be trusted: a procfs or sysfs file is a regular file that reports zero bytes and then reads out content. Measuring one declares an empty payload, and the SDK limits its reads to the length it was given, so the encrypt would succeed with nothing in it. Those come back with the Seeker hidden — an archive a few dozen bytes larger beats an archive missing the payload.
cleanup is always non-nil, including on the error return.
func OpenSeekable ¶
OpenSeekable resolves a command's input to something seekable: the named file when one is given, otherwise stdin. It returns ErrNoInput for the same "nothing to read" condition whether the file argument was absent or the pipe was empty.
Only what cannot seek is spooled to disk. A named file or a redirect from one already seeks, and copying it would need a TMPDIR with room for the whole TDF to buy nothing.
The returned cleanup must run on every path, per Spool.
func PipeReader ¶
PipeReader reports whether in carries at least one byte of payload, and returns a reader over it. A terminal, or an empty redirect such as `otdfctl encrypt < /dev/null`, reports false.
A redirect from a regular file — `otdfctl encrypt < payload.txt` — is handed back as the *os.File itself, so the payload stays measurable. Wrapping it would throw that away for nothing: it is the same fd either way. Everything else is a stream, and gets a buffered reader.
func Seekable ¶ added in v0.38.1
Seekable reports r's current position, and whether it could be asked for one.
It probes rather than asserting io.Seeker, because an *os.File on a FIFO, a process substitution, or /dev/stdin on the end of a pipe all satisfy the interface and then return ESPIPE. Seeking to the current position never moves it, so this is safe on a payload about to be read.
func Spool ¶
Spool copies r into a temporary file and rewinds it, giving a seekable view of a stream that has none.
A TDF's manifest sits at the end of the archive, so any command that needs to seek it cannot consume a pipe directly. Spooling trades disk for the memory a whole-payload read would use, and needs a TMPDIR with room for the full TDF — it fails loudly if there isn't one.
The returned cleanup must run on every path. cli.ExitWithError calls os.Exit and skips deferred functions, so deferring it alone is not enough.
Types ¶
type OutputFile ¶
type OutputFile struct {
// contains filtered or unexported fields
}
OutputFile writes to a temporary file alongside the destination and renames it into place only once the write has succeeded, so an interrupted or failed run leaves no partial output where a complete file is expected.
A destination that a rename cannot stand in for — /dev/null, /dev/stdout, a fifo, a symlink the caller means to write through — is opened and written directly instead, matching what os.Create did before. Those destinations give up the no-partial-output guarantee, which is inherent: there is nothing to rename into place.
Note that cli.ExitWithError calls os.Exit, which does not run deferred functions. Cleanup must therefore be called explicitly on every error path, not only via defer.
func NewOutputFile ¶
func NewOutputFile(path string, mode os.FileMode) (*OutputFile, error)
NewOutputFile opens the destination for writing.
For an ordinary destination it creates the temporary file in the destination's own directory. A rename is only atomic within a single filesystem, so the temp file must live beside the destination rather than in a shared temp directory — Commit's os.Rename fails outright (EXDEV) if that invariant is broken.
The temp file is created with mode, and a rename carries that mode onto the destination after the process umask has been applied. A destination written through directly is opened with mode, which the umask likewise applies to, and which has no effect at all on a destination that already exists.
func (*OutputFile) Cleanup ¶
func (o *OutputFile) Cleanup()
Cleanup discards the temporary file. It is a no-op after a successful Commit (or a prior Cleanup), so it is safe to both defer it and call it directly.
A destination being written through directly is only closed, never removed — the file is the caller's, and for /dev/null and friends removing it would do real damage.
func (*OutputFile) Commit ¶
func (o *OutputFile) Commit() error
Commit closes the temporary file and moves it onto the destination path.
It returns ErrOutputFileFinished if called more than once, or after Cleanup — otherwise a second call would re-enter the close/rename-failure path against an already-closed or already-moved file and report a spurious error.
func (*OutputFile) Name ¶
func (o *OutputFile) Name() string
Name reports the path currently being written: the temporary file for an ordinary destination, which is not the destination until Commit succeeds, or the destination itself for one being written through directly.
Source Files