Documentation
¶
Overview ¶
Package exportstream is the plumbing an export streams a body through without holding it whole: a JSON array written item by item into a pipe a storage upload reads from, and a reader that bounds that upload. Extracted from pkg/toolkits/apigateway, whose api_export page walk is its first user, for that package's size budget (#2057).
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrConsumerStopped = errors.New("walk output consumer stopped")
ErrConsumerStopped marks a write that failed because the reader of the stream went away (the storage upload failed or was capped). The reader's own error is the one to report; this one says the writer is not the cause.
Functions ¶
This section is empty.
Types ¶
type CappedReader ¶
type CappedReader struct {
// contains filtered or unexported fields
}
CappedReader bounds a stream at a byte count. Once more than that has been read it returns an error (so the S3 transfer manager aborts the incomplete multipart upload) and Exceeded reports true, which a caller checks to tell an over-cap body from a transient storage error. A cap <= 0 disables it. The over-cap error text is internal; the caller substitutes the sentence its own caller reads.
func NewCappedReader ¶
func NewCappedReader(r io.Reader, limit int64) *CappedReader
NewCappedReader bounds r at limit bytes.
func (*CappedReader) Exceeded ¶
func (c *CappedReader) Exceeded() bool
Exceeded reports whether the stream ran past the cap.
type JSONArrayWriter ¶
type JSONArrayWriter struct {
// contains filtered or unexported fields
}
JSONArrayWriter streams one JSON array: it opens the array on the first write, separates items with commas, and closes it on Close, so memory holds one batch of items at a time however many there are.
func NewJSONArrayWriter ¶
func NewJSONArrayWriter(w io.Writer) *JSONArrayWriter
NewJSONArrayWriter writes the array to w.
func (*JSONArrayWriter) Close ¶
func (a *JSONArrayWriter) Close() error
Close finishes the array. A writer that was handed nothing still writes an empty array.
func (*JSONArrayWriter) Write ¶
func (a *JSONArrayWriter) Write(items []json.RawMessage) error
Write appends items to the array.
Source Files