Documentation
¶
Overview ¶
Package zst implements reading and writing zst storage objects to and from zng row format. The zst storage format consists of a section of column data stored in zng values followed by a section containing a zng record stream comprised of N zng "reassembly records" (one for each zed.TypeRecord or "schema") stored in the zst object, plus an N+1st zng record describing the list of schemas IDs of the original zng rows that were encoded into the zst object.
A zst storage object must be seekable (e.g., a local file or s3 object), so, unlike zng, streaming of zst objects is not supported.
The zst/column package handles reading and writing row data to columns, while the zst package comprises the API used to read and write zst objects.
An Object provides the interface to the underlying storage object. To generate rows or cuts (and in the future more sophisticated traversals and introspection), an Assembly is created from the Object then zng records are read from the assembly, which implements zio.Reader. The Assembly keeps track of where each column is, which is why you need a separate Assembly per scan.
You can have multiple Assembly's referring to one Object as once an object is created, it's state never changes. That said, each assembly will issue reads to the underlying storage object and the read pattern may create performance issues.
Index ¶
- Constants
- Variables
- type Assembler
- type Assembly
- type CutAssembler
- type Object
- type Reader
- func NewCutter(object *Object, fields []string) (*Reader, error)
- func NewCutterFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string, ...) (*Reader, error)
- func NewReader(object *Object) (*Reader, error)
- func NewReaderFromPath(ctx context.Context, zctx *zed.Context, engine storage.Engine, path string) (*Reader, error)
- func NewReaderFromSeeker(zctx *zed.Context, seeker *storage.Seeker) (*Reader, error)
- type Trailer
- type Writer
- type WriterURI
Constants ¶
const ( MagicField = "magic" VersionField = "version" SkewThreshField = "skew_thresh" SegmentThreshField = "segment_thresh" SectionsField = "sections" MagicVal = "zst" VersionVal = 1 TrailerMaxSize = 4096 )
const ( MaxSegmentThresh = column.MaxSegmentThresh MaxSkewThresh = 512 * 1024 * 1024 )
Variables ¶
var ErrBadSchemaID = errors.New("bad schema id in root reassembly column")
var ErrNotZst = errors.New("not a zst object")
Functions ¶
This section is empty.
Types ¶
type Assembler ¶
type Assembler struct {
// contains filtered or unexported fields
}
Assembler implements the zio.Reader and io.Closer. It reads a columnar zst object to generate a stream of zed.Records. It also has methods to read metainformation for test and debugging.
type CutAssembler ¶
type CutAssembler struct {
// contains filtered or unexported fields
}
func NewCutAssembler ¶
type Object ¶
type Object struct {
// contains filtered or unexported fields
}
func NewObjectFromPath ¶
func NewObjectFromSeeker ¶
func (*Object) NewReassemblyReader ¶
func (*Object) NewTrailerReader ¶
type Reader ¶
Reader implements zio.Reader and io.Closer. It reads a columnar zst object to generate a stream of zed.Records. It also has methods to read metainformation for test and debugging.
func NewCutterFromPath ¶
func NewReader ¶
NewReader returns a Reader ready to read a zst object as zed.Records. Close() should be called when done. This embeds a zst.Object.
func NewReaderFromPath ¶
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer implements the zio.Writer interface. A Writer creates a columnar zst object from a stream of zed.Records.
Source Files
Directories