Documentation
¶
Index ¶
- func AppendFile(ctx context.Context, st Store, path string, schemaID int16, entry FileEntry) error
- func AppendFiles(ctx context.Context, st Store, path string, schemaID int16, ...) error
- func ListPaths(m *Manifest, tier string) []string
- func Save(ctx context.Context, st Store, path string, m *Manifest, etag string) (string, error)
- type FSStore
- type FileEntry
- type Manifest
- type PathResolver
- type S3Client
- type S3Store
- type Store
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AppendFile ¶
AppendFile adds a FileEntry to the manifest and saves it. Uses optimistic locking via etag to handle concurrent updates.
func AppendFiles ¶
func AppendFiles(ctx context.Context, st Store, path string, schemaID int16, entries []FileEntry) error
AppendFiles adds multiple FileEntry items to the manifest and saves it. Uses optimistic locking via etag to handle concurrent updates.
Types ¶
type FileEntry ¶
type FileEntry struct {
Tier string `json:"tier"`
Path string `json:"path"`
RowIDMin string `json:"row_id_min"`
RowIDMax string `json:"row_id_max"`
CreatedMin int64 `json:"created_min"`
CreatedMax int64 `json:"created_max"`
SizeBytes int64 `json:"size_bytes"`
RowCount int64 `json:"row_count"`
Checksum string `json:"checksum,omitempty"`
}
FileEntry describes a single parquet file tracked by the manifest. Tier is typically "base" or "delta".
func FilterByTier ¶
FilterBySchema returns files matching the tier (use "" for all tiers).
type Manifest ¶
type Manifest struct {
SchemaID int16 `json:"schema_id"`
Version int64 `json:"version"`
UpdatedAtMs int64 `json:"updated_at_ms"`
Files []FileEntry `json:"files"`
}
Manifest holds per-schema parquet inventory.
type PathResolver ¶
PathResolver builds manifest paths per schema using a template. Defaults to "manifest/{{.SchemaID}}.json" under an optional prefix.
type S3Client ¶ added in v0.0.24
type S3Client interface {
GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error)
}
Click to show internal directories.
Click to hide internal directories.