colgranule

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2026 License: MIT Imports: 23 Imported by: 0

README

Int64 Column Granule Experiment

This package is a non-durable column-store smoke test. It does not publish TreeDB roots, use production TreeDB value-log publication, use collection WAL, or expose public collection APIs.

It currently covers:

  • 8192-row default int64 granules;
  • raw int64 encoding;
  • delta + zigzag varint encoding;
  • double-delta-style int64 encoding;
  • nullable/default-heavy int64, bool bitpack/RLE, and low-cardinality code paths;
  • none, snappy, and lz4 compression;
  • min/max metadata and range-scan skip;
  • sort-key marks and predicate pruning diagnostics;
  • aggregate kernels over encoded granules;
  • non-durable in-memory column parts made from row-aligned granules and independently split column codec blocks;
  • exact serialized in-memory column part images with a manifest, section directory, descriptor bytes, marks, locators, dictionaries, aggregate metadata, and column payload sections;
  • value-log-shaped TCS1 column part assets that wrap serialized images, persist/reopen through ColumnAssetStore, and reconstruct scan-capable parts;
  • a disk-backed ColumnWorkspace that persists workspace manifests, collection manifests, and TCS1 asset refs for reopen/integrity experiments;
  • an experiment ColumnCollectionManifest and ColumnPartSetReader for multipart base/delta/tombstone visibility and latest-row locator lookup;
  • compaction of visible multipart rows back into a new base column part;
  • JSONBench Bluesky fixture loading into int64-derived columns.

Local JSONBench Data

The JSONBench data directory is expected at:

$JSONBENCH_DATA

That path matches the default output from $JSONBENCH_REPO/download_data.sh for the Bluesky data set. The upstream downloader writes larger scales into the same directory: 1m is file_0001.json.gz, 10m is file_0001.json.gz through file_0010.json.gz, 100m through file_0100.json.gz, and 1000m through file_1000.json.gz.

The repository includes a tiny testdata/jsonbench_sample.jsonl fixture for tests. The 129 MiB compressed 1M-row file is intentionally not vendored.

Run the full local 1M-row column summary:

go run ./experiments/colgranule/cmd/jsonbench_colgranule \
  -data $JSONBENCH_DATA \
  -limit 1000000 \
  -rows-per-granule 8192

The loader derives int64 columns from real JSONBench rows, including time_us, line size, row index, string lengths, low-cardinality dictionary codes, boolean presence flags, createdAt milliseconds, and language counts.

The query-oriented derived columns include the JSONBench paths used by the ClickHouse setup and queries: kind, commit.operation, commit.collection, did, and time_us.

Build the raw ClickHouse comparison data and Markdown summary:

go run ./experiments/colgranule/cmd/jsonbench_compare \
  -data $JSONBENCH_DATA \
  -limit 1000000 \
  -rows-per-granule 8192 \
  -attempts 5

This writes:

  • JSONBENCH_COMPARISON_RAW.json: raw ClickHouse result imports, column codec summaries, query-kernel timing attempts, and the compacted TreeDB JSON/BSON remaining-field measurements;
  • JSONBENCH_COMPARISON_REPORT.md: human-readable timing and storage summary.

By default, the comparison command also builds temporary TreeDB collections under artifacts/colgranule_remaining_treedb-*, flushes and compacts each database, and adds the disk footprints to the report. It records two remaining-field shapes: a conservative shape with only top-level time_us removed, and a ClickHouse-aligned shape with the typed JSON paths removed: time_us, kind, did, commit.operation, and commit.collection. Disable this part with -measure-remaining-treedb=false. The command also records a raw TreeDB key/value shape that stores documentID(row) -> original JSON line bytes without collection document encoding.

To iterate on encoded-part build and query kernels without reloading the retained payload collections, reuse a previous raw comparison file:

go run ./experiments/colgranule/cmd/jsonbench_compare \
  -data $JSONBENCH_DATA \
  -limit 1000000 \
  -rows-per-granule 8192 \
  -attempts 5 \
  -measure-remaining-treedb=false \
  -retained-payload-from-json experiments/colgranule/JSONBENCH_COMPARISON_RAW.json

When retained-payload measurements are available, the Markdown report includes a full-dataset estimate that adds the current serialized in-memory column part to the measured TreeDB payload bytes. This is the pre-M2 comparison point for ClickHouse total_size; the encoded part is serialized but still in memory, while the retained payload is measured from compacted TreeDB files.

Serialized Part Image Layout

ColumnPartImage is the pre-file representation that M2 should persist with minimal semantic changes. The image starts with a manifest containing a magic number, image version, part id, row count, manifest byte length, and one section directory entry per following byte section. Section directory entries record kind, category, offset, length, row/granule/block counts, encoding, compression, name, and column name. Accounting exposes the manifest as a separate manifest pseudo-section so descriptor bytes do not hide or double count manifest overhead.

The current canonical section kinds are:

Section Category Purpose
descriptor descriptor Part-level metadata, granule descriptors, column descriptors, block descriptors, and per-block granule reader metadata such as null/default counts and min/max.
sort_key_metadata sort_key_metadata Sort-key columns, direction, and null ordering, distinct from TreeDB logical primary key.
sort_key_marks marks Per-granule sparse mark summaries for sort-key prefixes.
row_locators locators primary_id -> part row, granule, row-in-granule lookup records.
aggregate_metadata aggregate_metadata Admitted exact per-granule aggregate metadata definitions, stats, and entries.
dictionaries dictionaries Part-local dictionary payloads for declared low-cardinality columns when available.
column_data declared_columns Concatenated encoded/compressed column block payload bytes, split by column and addressed by block descriptors.

ParseColumnPartImage validates the manifest and section bounds. ColumnPartFromImage reconstructs a scan-capable read-only part from serialized bytes alone.

TCS1 Column Asset Layout

TCS1 is the M2 value-log-shaped wrapper for a serialized column part image. It is a logical column asset payload, not a separate production column-file lifecycle. The first M2 mode stores one complete ColumnPartImage as a single asset with:

  • TCS1 magic and version;
  • payload kind for serialized part images;
  • format flags, currently strict-zero;
  • payload byte length;
  • part id, row count, and image version copied from the image;
  • payload checksum.

ColumnAssetStore is the experiment seam that models future TreeDB ValuePtr storage without importing TreeDB/internal from this package. The memory store is useful for benchmarks, and the append-segment store proves reopen/readback through a value-log-shaped file id + offset + length + checksum ref. Query kernels still receive a reconstructed ColumnPart, so M2 validates storage plumbing without changing the encoded-block execution interface.

JSONBench query timings now build the serialized image, wrap/store it as a TCS1 asset, read it back, reconstruct the part, and run q1-q5 through the same encoded kernels. The query hot path uses scan-only reconstruction so it does not decode the point-lookup row-locator map or aggregate metadata for scans that do not need them. Full reconstruction with locators and eager metadata remains available for point lookup, metadata kernels, and integrity tests. Split assets for individual blocks, dictionaries, marks, locators, or aggregate metadata should remain benchmark-driven follow-ups; the single-record mode is the baseline.

Disk Workspace, Manifests, and Multipart Parts

ColumnWorkspace is the M3 disk-backed experiment wrapper. It stores TCS1 assets in append segments, writes checksum-protected manifests, and validates referenced assets on reopen. Column-lane files live under an isolated namespace: manifests/ for workspace and collection manifests, assets/segments/ for append-segment assets, assets/indexes/ for future secondary column metadata, prepared/ for staged publish state, quarantine/ for suspect assets, and tmp/ for atomic manifest writes. It is still not a TreeDB root publication path and does not claim durable-at-ack collection semantics.

ColumnCollectionManifest is the M4 collection-lane control-plane model used by the experiment. It records declared columns, logical primary key, sort key, base parts, delta parts, tombstones, retained-payload byte accounting metadata, referenced TCS1 asset records, and compact coverage descriptors for each part ref. Coverage descriptors carry role, generation, compaction level, source parts, optional replacement-delta source row/root generation ranges, primary-id range, sort-key range, row counts, deleted counts, asset refs, and checksums. Collection manifest envelopes are written as compact JSON and checksum the canonical compact manifest payload; reopen can still validate earlier pretty-printed experiment payloads by compacting the raw manifest field before checksum comparison. ColumnPartSetReader opens those refs from the workspace, builds latest-visible row state, exposes LatestLocator, and runs JSONBench q1-q5 over visible rows.

M6 treats declared JSONBench paths as column-owned state and the retained row payload as the non-column remainder. JSONBenchRetainedDocument removes the declared paths from row JSON, JSONBenchDeclaredColumnValuesFromPart reads those values back from an image-backed column part, and RestoreJSONBenchDeclaredColumns must reconstruct the original JSON document under canonical JSON comparison.

Column asset lifecycle planning is manifest/ref based. ColumnAssetRef reachability is computed from active manifests, pending manifests, snapshot-pinned manifests, superseded manifests, prepared assets, and quarantined assets without decoding TCS1 payload bytes or scanning rows. A typed ColumnAssetManager facade owns the experiment refs and models the production pin, zombie, rewrite-debt, quarantine, and deletion gate: reclaimable bytes are not ready to delete until reachability, zombie marking, and pin drain all agree. Compaction reports both pre-publish and post-publish reachability: before the new manifest is published, old active assets remain protected; after publish, old assets are directly reclaimable only when their whole segment has no live refs, otherwise they are rewrite debt. Prepared publish state is recorded in a checksum-protected registry under prepared/, and namespace inventory is a read-only reconciliation input that joins segment files and prepared refs against manifest/ref state before reporting orphan candidates. Publish bookkeeping can also use PlanColumnAssetRefDelta when the changed part refs are known, keeping the accounting path proportional to newly published, superseded, and prepared refs instead of rows or unchanged manifests.

PlanColumnPartSetCompaction is the M6 policy-report seam for future background selection. It reports selected/skipped parts, live bytes, stale bytes, tombstone debt, expected reclaim ratio, sparse-visible-row pressure, read amplification, aggregate-metadata invalidation, and column-asset stale-byte pressure without publishing a new part. The explicit CompactColumnPartSet helper remains the only experiment path that writes the replacement base asset.

Reopen validation has an explicit lazy boundary. The default experiment mode still validates full TCS1 images for integrity tests, while ColumnWorkspaceValidateTCS1Header validates manifest/ref/header metadata without reading every full image payload. Loading the part still verifies the payload checksum and section structure before use.

Adaptive mark sizing is explicit and opt-in through ColumnAdaptiveMarkSizing. When enabled, part build estimates uncompressed declared-column bytes per row and chooses rows per mark from a target byte budget with min/max row clamps. Fixed RowsPerGranule remains the default path for existing gates.

M8A adds a non-durable mutation adapter over the same manifest path. The adapter publishes bulk base batches, row-aligned insert/update delta parts, and delete tombstones, then persists a new collection manifest generation. It accepts complete declared-column vectors produced from collection-shaped mutations; it does not encode physical column diffs as replay source of truth and still does not claim command-WAL durable-at-ack semantics. The restart/replay tests compare logical rows and JSONBench q1-q5 hashes after applying the same mutation sequence into a fresh workspace.

The M8A locator profile keeps the V1 locator shape as one primary id to latest visible row locator per manifest generation. It does not use one locator per column because column parts remain row-aligned across declared columns. The current reader builds an in-memory side index while opening a manifest, and the no-side-index comparison scans part-local locator maps. On an Apple M3, the measured profile was:

Delta parts Side-index locator Part-local scan locator Side-index point value Part-local scan point value
1 ~10-11 ns/op, 0 allocs ~63-65 ns/op, 0 allocs ~58-66 ns/op, 0 allocs ~114-145 ns/op, 0 allocs
8 ~10-12 ns/op, 0 allocs ~237-262 ns/op, 0 allocs ~58-69 ns/op, 0 allocs ~284-317 ns/op, 0 allocs
32 ~10-11 ns/op, 0 allocs ~867-987 ns/op, 0 allocs ~58-75 ns/op, 0 allocs ~912-1100 ns/op, 0 allocs
128 ~10-12 ns/op, 0 allocs ~4.0-4.6 us/op, 0 allocs ~58-66 ns/op, 0 allocs ~4.1-4.2 us/op, 0 allocs

This is enough to defer a root-backed locator for the next production step. The manifest-load side index gives constant-time, zero-allocation lookup and scratch-backed point reconstruction, while the scan path only proves why an index is needed once part counts grow. A production root-backed locator should be added later only when a real update/delete or point-reconstruction workload needs persistent locator state across opens, and it should map primary id -> latest visible row locator for a generation rather than primary id + column.

The multipart query gate should preserve the M2 encoded-part execution shape: avoid projected-row materialization for hot q1-q5 kernels, report visible, superseded, deleted, part, block, decoded-byte, and cache diagnostics, and keep per-query hot allocation bounded. The current synthetic M4 gate is:

GOWORK=off go test ./experiments/colgranule -run '^$' \
  -bench 'BenchmarkJSONBenchEncodedPartQueries|BenchmarkJSONBenchColumnPartSetQueriesM4|BenchmarkJSONBenchColumnPartSetQueriesM4PerQuery' \
  -benchmem -count=5

To compare the historical in-memory colgranule JSONBench kernels with the current durable TreeDB collection column-store physical query path, run:

GOWORK=off go test ./experiments/colgranule -run '^$' \
  -bench '^BenchmarkJSONBenchColumnStoreCompare$' \
  -benchmem -benchtime=3x -count=1

BenchmarkJSONBenchColumnStoreCompare uses a synthetic, filter-degenerate JSONBench fixture where every row is kind=commit, operation=create, and collection=app.bsky.feed.post. This keeps the current TreeDB physical query API comparable to the older JSONBench Q1/Q2/Q3-hour/Q4/Q5 kernels even though TreeDB does not yet expose the experiment's separate filter-column predicates in this query surface. Override row count with TREEDB_JSONBENCH_COMPARE_ROWS=8192 for a smoke run.

The current M5 compaction gate measures both full compaction and phase breakdowns for visible scanning, part rebuild, image serialization, asset publish, and manifest save:

GOWORK=off go test ./experiments/colgranule -run '^$' \
  -bench 'BenchmarkColumnPartSetCompactionM5|BenchmarkColumnPartSetCompactionM5Breakdown' \
  -benchmem -count=5

The local 1M-row multipart benchmark intentionally skips unless JSONBENCH_DATA points to a real 1M-row fixture; the repository fixture is only for tests:

JSONBENCH_DATA=/path/to/file_0001.json.gz \
GOWORK=off go test ./experiments/colgranule -run '^$' \
  -bench 'BenchmarkJSONBenchLocalColumnPartSetQueriesM4PerQuery' \
  -benchmem -count=1

The known q4 caveat is sort order, not row materialization: M2's single-part baseline can early-stop on a globally time-ordered part, while the correct M4 multipart path currently scans visible rows. A future k-way sort-key/mark merge across parts is needed before multipart q4 can early-stop safely. The TreeDB comparison also includes q3_group_hour_count, a production physical reducer shape for dictionary predicates plus collection/hour(time_us) grouping, and explicit aggregate-metadata Top-K q4/q5 variants that report topk_limit/op, topk_candidates/op, and result_shape_ns/op; metadata paths use logical rows/sec with rows_scanned/op=0 and should not be confused with full data-row scans or pruning metadata.

M1D Gates Before Value-Log-Backed M2

Before moving this experiment into durable files, the in-memory representation must stay aligned with the future file format:

  • byte accounting must reconcile to the serialized image byte length, not struct estimates;
  • serialized images must parse back into read-only column parts without relying on builder-owned descriptor or payload structs;
  • the report must show section bytes for manifest/descriptors, declared columns, dictionaries, marks, sort-key metadata, aggregate metadata, and row locators;
  • JSONBench query runners must read column payloads through parsed image-backed parts;
  • retained JSON must remain separate from declared column image bytes in the full-dataset comparison;
  • local 1M JSONBench reports must include granule count, codec block count, part file count, retained-payload file count, build throughput, allocations, and TreeDB/ClickHouse size ratios;
  • BenchmarkJSONBenchPartImageM1D and the optional 1M BenchmarkJSONBenchLocalPartImageM1D benchmark must keep measuring serialization of an existing part, parse/reconstruct from image bytes, and the full build/serialize/parse/reconstruct path;
  • M2 should add TCS1 value-log-shaped persistence, checksums, asset refs, recovery, and lifecycle behavior without changing the logical section model.

Documentation

Index

Constants

View Source 
const (
	DefaultAdaptiveMarkTargetBytes = 1 << 20
	DefaultAdaptiveMarkMinRows     = 512
)
View Source 
const (
	ColumnWorkspaceValidateFullImage  ColumnWorkspaceValidationMode = "full_image"
	ColumnWorkspaceValidateTCS1Header ColumnWorkspaceValidationMode = "tcs1_header"

	ColumnWorkspaceManifestSyncDurable              ColumnWorkspaceManifestSyncMode = "durable"
	ColumnWorkspaceManifestSyncDisabledForBenchmark ColumnWorkspaceManifestSyncMode = "benchmark_no_sync"
)
View Source 
const AggregateMetadataDefinitionVersion uint16 = 1
View Source 
const DefaultJSONBenchDir = "experiments/colgranule/testdata"
View Source 
const DefaultJSONBenchPath = "experiments/colgranule/testdata/jsonbench_sample.jsonl"
View Source 
const (
	DefaultRowsPerGranule = 8192
)

Variables

View Source 
var ErrColumnMutationReplayWorkspaceSyncMode = errors.New("colgranule: column mutation replay workspace sync mode mismatch")

Functions

func ColumnPartFromTCS1Asset 

func ColumnPartFromTCS1Asset(store ColumnAssetStore, ref ColumnAssetRef) (*ColumnPart, TCS1PartRecord, error)

func ColumnPartFromTCS1AssetWithOptions 

func ColumnPartFromTCS1AssetWithOptions(store ColumnAssetStore, ref ColumnAssetRef, opts ColumnPartImageReadOptions) (*ColumnPart, TCS1PartRecord, error)

func DecodeInt64 

func DecodeInt64(dst []int64, g EncodedGranule) ([]int64, error)

func DecodeJSONDocumentPreserveNumbers 

func DecodeJSONDocumentPreserveNumbers(raw []byte, v any) error

func DecodeTCS1ColumnPartImage 

func DecodeTCS1ColumnPartImage(data []byte) (ColumnPartImage, TCS1PartRecord, error)

func EncodeColumnCollectionManifest 

func EncodeColumnCollectionManifest(manifest ColumnCollectionManifest) ([]byte, error)

func EstimateColumnBatchUncompressedBytes 

func EstimateColumnBatchUncompressedBytes(batch ColumnBatch, defs []ColumnDefinition) (int, error)

func EstimateJSONBenchDictionaryBytes 

func EstimateJSONBenchDictionaryBytes(ds JSONBenchDataset) int

func FormatColumnCodecSummary 

func FormatColumnCodecSummary(s ColumnCodecSummary) string

func JSONBenchDeclaredColumnPaths 

func JSONBenchDeclaredColumnPaths() []string

func JSONBenchRawDocumentBytes 

func JSONBenchRawDocumentBytes(ds JSONBenchDataset) int64

func JSONBenchRetainedDocument 

func JSONBenchRetainedDocument(raw []byte) ([]byte, error)

func LoadTCS1ColumnPartImage 

func LoadTCS1ColumnPartImage(store ColumnAssetStore, ref ColumnAssetRef) (ColumnPartImage, TCS1PartRecord, error)

func RangeScanCount 

func RangeScanCount(g EncodedGranule, low, high int64, scratch []int64) (int, []int64, error)

func RestoreJSONBenchDeclaredColumns 

func RestoreJSONBenchDeclaredColumns(doc map[string]any, values JSONBenchDeclaredColumnValues)

func ReverseJSONBenchDictionaries 

func ReverseJSONBenchDictionaries(dictionaries map[string]map[string]int64) map[string]map[int64]string

func ScanOnlyTCS1AssetBackedColumnPart 

func ScanOnlyTCS1AssetBackedColumnPart(part *ColumnPart, dictionaries map[string]map[string]int64, store ColumnAssetStore) (*ColumnPart, ColumnAssetRef, TCS1PartRecord, error)

func StoreTCS1ColumnPartImage 

func StoreTCS1ColumnPartImage(store ColumnAssetStore, image ColumnPartImage) (ColumnAssetRef, TCS1PartRecord, error)

func TCS1AssetBackedColumnPart 

func TCS1AssetBackedColumnPart(part *ColumnPart, dictionaries map[string]map[string]int64, store ColumnAssetStore) (*ColumnPart, ColumnAssetRef, TCS1PartRecord, error)

func TCS1AssetBackedColumnPartWithOptions 

func TCS1AssetBackedColumnPartWithOptions(part *ColumnPart, dictionaries map[string]map[string]int64, store ColumnAssetStore, opts ColumnPartImageReadOptions) (*ColumnPart, ColumnAssetRef, TCS1PartRecord, error)

Types

type AggregateArena 

type AggregateArena struct {
	// contains filtered or unexported fields
}

func (*AggregateArena) ExactDistinctCodes 

func (a *AggregateArena) ExactDistinctCodes(granules []EncodedGranule, cardinality uint32) (int, error)

func (*AggregateArena) ExactDistinctInt64 

func (a *AggregateArena) ExactDistinctInt64(granules []EncodedGranule) (int, error)

func (*AggregateArena) FilteredGroupedCountCodes 

func (a *AggregateArena) FilteredGroupedCountCodes(codeGranules []EncodedGranule, filterGranules []EncodedGranule, filter Int64RangePredicate, cardinality uint32) ([]uint64, PredicateDiagnostics, error)

FilteredGroupedCountCodes returns arena-owned count storage. The returned slice is valid only until the next AggregateArena operation or Reset.

func (*AggregateArena) GroupedCountCodes 

func (a *AggregateArena) GroupedCountCodes(granules []EncodedGranule, cardinality uint32) ([]uint64, error)

GroupedCountCodes returns arena-owned count storage. The returned slice is valid only until the next AggregateArena operation or Reset.

func (*AggregateArena) MinMaxInt64 

func (a *AggregateArena) MinMaxInt64(granules []EncodedGranule, filter *Int64RangePredicate) (int64, int64, bool, error)

func (*AggregateArena) Reset 

func (a *AggregateArena) Reset()

func (*AggregateArena) TimeBucketedCountCodes 

func (a *AggregateArena) TimeBucketedCountCodes(codeGranules []EncodedGranule, timeGranules []EncodedGranule, bucketWidth int64, cardinality uint32) (TimeBucketedCounts, error)

TimeBucketedCountCodes returns a result whose Counts slice is arena-owned. Counts is valid only until the next AggregateArena operation or Reset.

type AggregateMetadata 

type AggregateMetadata struct {
	Definition AggregateMetadataDefinition `json:"definition"`
	Granules   []AggregateMetadataGranule  `json:"granules,omitempty"`
	Stats      AggregateMetadataStats      `json:"stats"`
}

type AggregateMetadataDefinition 

type AggregateMetadataDefinition struct {
	Name           string                       `json:"name"`
	Version        uint16                       `json:"version"`
	Kind           AggregateMetadataKind        `json:"kind"`
	Scope          AggregateMetadataScope       `json:"scope"`
	GroupKeys      []string                     `json:"group_keys"`
	Measures       []AggregateMetadataMeasure   `json:"measures"`
	Predicates     []AggregateMetadataPredicate `json:"predicates,omitempty"`
	MaxBytesPerRow float64                      `json:"max_bytes_per_row"`
}

func JSONBenchPostCreateDidTimeAggregateMetadataDefinition 

func JSONBenchPostCreateDidTimeAggregateMetadataDefinition(ds JSONBenchDataset) (AggregateMetadataDefinition, error)

type AggregateMetadataEntry 

type AggregateMetadataEntry struct {
	Group uint32 `json:"group"`
	Count uint32 `json:"count"`
	Min   int64  `json:"min"`
	Max   int64  `json:"max"`
}

type AggregateMetadataGranule 

type AggregateMetadataGranule struct {
	GranuleOrdinal int                      `json:"granule_ordinal"`
	FirstRow       int                      `json:"first_row"`
	RowCount       int                      `json:"row_count"`
	MatchedRows    int                      `json:"matched_rows"`
	Entries        []AggregateMetadataEntry `json:"entries,omitempty"`
}

type AggregateMetadataKind 

type AggregateMetadataKind string
const (
	AggregateMetadataGroupMinMax AggregateMetadataKind = "group_min_max"
)

type AggregateMetadataMeasure 

type AggregateMetadataMeasure struct {
	Op     AggregateMetadataMeasureOp `json:"op"`
	Column string                     `json:"column,omitempty"`
}

type AggregateMetadataMeasureOp 

type AggregateMetadataMeasureOp string
const (
	AggregateMetadataMeasureCount AggregateMetadataMeasureOp = "count"
	AggregateMetadataMeasureMin   AggregateMetadataMeasureOp = "min"
	AggregateMetadataMeasureMax   AggregateMetadataMeasureOp = "max"
)

type AggregateMetadataPredicate 

type AggregateMetadataPredicate struct {
	Column string                       `json:"column"`
	Op     AggregateMetadataPredicateOp `json:"op"`
	Value  int64                        `json:"value"`
}

type AggregateMetadataPredicateOp 

type AggregateMetadataPredicateOp string
const (
	AggregateMetadataPredicateEq AggregateMetadataPredicateOp = "eq"
)

type AggregateMetadataScope 

type AggregateMetadataScope string
const (
	AggregateMetadataScopeGranule AggregateMetadataScope = "granule"
)

type AggregateMetadataStats 

type AggregateMetadataStats struct {
	Admitted            bool          `json:"admitted"`
	RejectedReason      string        `json:"rejected_reason,omitempty"`
	BuildDuration       time.Duration `json:"build_duration"`
	Granules            int           `json:"granules"`
	GranulesWithRows    int           `json:"granules_with_rows"`
	RowsMatched         int           `json:"rows_matched"`
	Entries             int           `json:"entries"`
	ValueBytes          int           `json:"value_bytes"`
	DescriptorBytes     int           `json:"estimated_descriptor_bytes"`
	TotalBytes          int           `json:"estimated_total_bytes"`
	BytesPerPartRow     float64       `json:"bytes_per_part_row"`
	BytesPerMatchedRow  float64       `json:"bytes_per_matched_row"`
	Compression         string        `json:"compression"`
	AdmissionMaxBytes   float64       `json:"admission_max_bytes_per_row"`
	AdmissionMeasuredBy string        `json:"admission_measured_by"`
}

type CodecReport 

type CodecReport struct {
	Encoding                  Encoding
	RequestedCompression      Compression
	ActualCompression         Compression
	CompressionAttempted      bool
	CompressionKept           bool
	CompressionFallbackReason string
	RawBytes                  int
	StoredBytes               int
	CompressionNanos          int64
}

type ColumnAdaptiveMarkEstimate 

type ColumnAdaptiveMarkEstimate struct {
	Rows           int     `json:"rows"`
	RawBytes       int     `json:"raw_bytes"`
	RawBytesPerRow float64 `json:"raw_bytes_per_row"`
	TargetBytes    int     `json:"target_bytes"`
	MinRows        int     `json:"min_rows"`
	MaxRows        int     `json:"max_rows"`
	RowsPerMark    int     `json:"rows_per_mark"`
	Marks          int     `json:"marks"`
	ClampedByMin   bool    `json:"clamped_by_min,omitempty"`
	ClampedByMax   bool    `json:"clamped_by_max,omitempty"`
}

func EstimateAdaptiveRowsPerMark 

func EstimateAdaptiveRowsPerMark(rows int, rawBytes int, cfg ColumnAdaptiveMarkSizing) (ColumnAdaptiveMarkEstimate, error)

type ColumnAdaptiveMarkSizing 

type ColumnAdaptiveMarkSizing struct {
	Enabled     bool `json:"enabled,omitempty"`
	TargetBytes int  `json:"target_bytes,omitempty"`
	MinRows     int  `json:"min_rows,omitempty"`
	MaxRows     int  `json:"max_rows,omitempty"`
}

func NormalizeColumnAdaptiveMarkSizing 

func NormalizeColumnAdaptiveMarkSizing(cfg ColumnAdaptiveMarkSizing, rowsPerGranule int) (ColumnAdaptiveMarkSizing, error)

type ColumnAssetKind 

type ColumnAssetKind string
const (
	ColumnAssetKindTCS1PartImage ColumnAssetKind = "tcs1_part_image"
)

type ColumnAssetLifecycleState 

type ColumnAssetLifecycleState string
const (
	ColumnAssetStatePrepared              ColumnAssetLifecycleState = "prepared"
	ColumnAssetStateProcessVisible        ColumnAssetLifecycleState = "process_visible"
	ColumnAssetStatePendingPublish        ColumnAssetLifecycleState = "pending_publish"
	ColumnAssetStateRootPublished         ColumnAssetLifecycleState = "root_published"
	ColumnAssetStateRecoveryAuthoritative ColumnAssetLifecycleState = "recovery_authoritative"
	ColumnAssetStateActive                ColumnAssetLifecycleState = "active"
	ColumnAssetStateSuperseded            ColumnAssetLifecycleState = "superseded"
	ColumnAssetStateCleanupSafe           ColumnAssetLifecycleState = "cleanup_safe"
	ColumnAssetStateSnapshotPinned        ColumnAssetLifecycleState = "snapshot_pinned"
	ColumnAssetStateReclaimable           ColumnAssetLifecycleState = "reclaimable"
	ColumnAssetStateDeleting              ColumnAssetLifecycleState = "deleting"
	ColumnAssetStateQuarantined           ColumnAssetLifecycleState = "quarantined"
)

type ColumnAssetManager 

type ColumnAssetManager struct {
	// contains filtered or unexported fields
}

func NewColumnAssetManager 

func NewColumnAssetManager(store ColumnAssetStore) (*ColumnAssetManager, error)

func (*ColumnAssetManager) Close 

func (m *ColumnAssetManager) Close() error

func (*ColumnAssetManager) DebugState 

func (m *ColumnAssetManager) DebugState() ColumnAssetManagerDebugState

func (*ColumnAssetManager) MarkPublishFailed 

func (m *ColumnAssetManager) MarkPublishFailed(prepared []ColumnPreparedAsset, reason string) error

func (*ColumnAssetManager) MarkPublishSucceeded 

func (m *ColumnAssetManager) MarkPublishSucceeded(synced ColumnAssetSyncedPublishClosure) error

func (*ColumnAssetManager) MarkRewriteDebt 

func (m *ColumnAssetManager) MarkRewriteDebt(ref ColumnAssetRef, reason string) error

func (*ColumnAssetManager) MarkZombie 

func (m *ColumnAssetManager) MarkZombie(ref ColumnAssetRef, reason string) error

func (*ColumnAssetManager) Pin 

func (m *ColumnAssetManager) Pin(ref ColumnAssetRef, reason string) (ColumnAssetPin, error)

func (*ColumnAssetManager) PlanReclamation 

func (m *ColumnAssetManager) PlanReclamation(reachability ColumnAssetReachabilityPlan) ColumnAssetManagerReclamationPlan

func (*ColumnAssetManager) PreparePublishClosure 

func (m *ColumnAssetManager) PreparePublishClosure(prepared []ColumnPreparedAsset) (ColumnAssetPublishClosure, error)

func (*ColumnAssetManager) Put 

func (m *ColumnAssetManager) Put(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)

func (*ColumnAssetManager) PutOwned 

func (m *ColumnAssetManager) PutOwned(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)

func (*ColumnAssetManager) Quarantine 

func (m *ColumnAssetManager) Quarantine(ref ColumnAssetRef, reason string) error

func (*ColumnAssetManager) Read 

func (m *ColumnAssetManager) Read(ref ColumnAssetRef) ([]byte, error)

func (*ColumnAssetManager) ReadRange 

func (m *ColumnAssetManager) ReadRange(ref ColumnAssetRef, offset int64, length int) ([]byte, error)

func (*ColumnAssetManager) ReadTo 

func (m *ColumnAssetManager) ReadTo(ref ColumnAssetRef, dst []byte) ([]byte, error)

func (*ColumnAssetManager) Release 

func (m *ColumnAssetManager) Release(pin ColumnAssetPin) error

func (*ColumnAssetManager) SyncPublishClosure 

func (m *ColumnAssetManager) SyncPublishClosure(closure ColumnAssetPublishClosure) (ColumnAssetSyncedPublishClosure, error)

type ColumnAssetManagerDebugState 

type ColumnAssetManagerDebugState struct {
	Quarantine     map[ColumnAssetRef]string
	OperatorLocked map[ColumnAssetRef]struct{}
	PublishFailed  map[ColumnAssetRef]string
	PublishEpoch   uint64
	RefFailedAt    map[ColumnAssetRef]uint64
	SyncedAttempts int
	SyncedRefs     int
}

type ColumnAssetManagerReclamationEntry 

type ColumnAssetManagerReclamationEntry struct {
	Ref              ColumnAssetRef            `json:"ref"`
	State            ColumnAssetLifecycleState `json:"state"`
	Bytes            int                       `json:"bytes"`
	Candidate        bool                      `json:"candidate,omitempty"`
	Pinned           bool                      `json:"pinned,omitempty"`
	Zombie           bool                      `json:"zombie,omitempty"`
	Quarantined      bool                      `json:"quarantined,omitempty"`
	RewriteRequired  bool                      `json:"rewrite_required,omitempty"`
	ReadyToDelete    bool                      `json:"ready_to_delete,omitempty"`
	ManagerReason    string                    `json:"manager_reason,omitempty"`
	ReachableReasons []string                  `json:"reachable_reasons,omitempty"`
}

type ColumnAssetManagerReclamationPlan 

type ColumnAssetManagerReclamationPlan struct {
	Entries            []ColumnAssetManagerReclamationEntry `json:"entries"`
	CandidateBytes     int                                  `json:"candidate_bytes"`
	ReadyToDeleteBytes int                                  `json:"ready_to_delete_bytes"`
	PinnedBytes        int                                  `json:"pinned_bytes"`
	ZombieBytes        int                                  `json:"zombie_bytes"`
	RewriteDebtBytes   int                                  `json:"rewrite_debt_bytes"`
}

type ColumnAssetPin 

type ColumnAssetPin struct {
	Ref    ColumnAssetRef `json:"ref"`
	Reason string         `json:"reason,omitempty"`
}

type ColumnAssetPublishClosure 

type ColumnAssetPublishClosure struct {
	PreparedAssets []ColumnPreparedAsset `json:"prepared_assets,omitempty"`
	RequiredAssets int                   `json:"required_assets"`
	RequiredBytes  int                   `json:"required_bytes"`
	FlushRequired  bool                  `json:"flush_required,omitempty"`
	SyncRequired   bool                  `json:"sync_required,omitempty"`
	// contains filtered or unexported fields
}

ColumnAssetPublishClosure is a single-process durability token returned by PreparePublishClosure. JSON tags are for one-way diagnostics only: a marshal/unmarshal round trip deliberately drops the private prepared identity and is not accepted by SyncPublishClosure.

type ColumnAssetRangeReader 

type ColumnAssetRangeReader interface {
	ReadRange(ref ColumnAssetRef, offset int64, length int) ([]byte, error)
}

type ColumnAssetReachabilityEntry 

type ColumnAssetReachabilityEntry struct {
	Ref            ColumnAssetRef            `json:"ref"`
	State          ColumnAssetLifecycleState `json:"state"`
	Bytes          int                       `json:"bytes"`
	PartID         uint64                    `json:"part_id,omitempty"`
	GenerationID   uint64                    `json:"generation_id,omitempty"`
	DeleteEligible bool                      `json:"delete_eligible,omitempty"`
	Reasons        []string                  `json:"reasons,omitempty"`
}

func ColumnCollectionManifestAssetRefs 

func ColumnCollectionManifestAssetRefs(manifest ColumnCollectionManifest) ([]ColumnAssetReachabilityEntry, error)

type ColumnAssetReachabilityInput 

type ColumnAssetReachabilityInput struct {
	ActiveManifest                 *ColumnCollectionManifest  `json:"active_manifest,omitempty"`
	ProcessVisibleManifests        []ColumnCollectionManifest `json:"process_visible_manifests,omitempty"`
	PendingManifests               []ColumnCollectionManifest `json:"pending_manifests,omitempty"`
	RootPublishedManifests         []ColumnCollectionManifest `json:"root_published_manifests,omitempty"`
	RecoveryAuthoritativeManifests []ColumnCollectionManifest `json:"recovery_authoritative_manifests,omitempty"`
	SnapshotPinnedManifests        []ColumnCollectionManifest `json:"snapshot_pinned_manifests,omitempty"`
	SupersededManifests            []ColumnCollectionManifest `json:"superseded_manifests,omitempty"`
	CleanupSafeManifests           []ColumnCollectionManifest `json:"cleanup_safe_manifests,omitempty"`
	PreparedAssets                 []ColumnPreparedAsset      `json:"prepared_assets,omitempty"`
	QuarantinedAssets              []ColumnPreparedAsset      `json:"quarantined_assets,omitempty"`
}

type ColumnAssetReachabilityPlan 

type ColumnAssetReachabilityPlan struct {
	Entries                []ColumnAssetReachabilityEntry `json:"entries"`
	Stats                  ColumnAssetReachabilityStats   `json:"stats"`
	RetainedBytes          int                            `json:"retained_bytes"`
	PreparedBytes          int                            `json:"prepared_bytes"`
	ProcessVisibleBytes    int                            `json:"process_visible_bytes"`
	PendingBytes           int                            `json:"pending_bytes"`
	RootPublishedBytes     int                            `json:"root_published_bytes"`
	QuarantinedBytes       int                            `json:"quarantined_bytes"`
	SupersededBytes        int                            `json:"superseded_bytes"`
	CleanupSafeBytes       int                            `json:"cleanup_safe_bytes"`
	SnapshotProtectedBytes int                            `json:"snapshot_protected_bytes"`
	RewriteDebtBytes       int                            `json:"rewrite_debt_bytes"`
	ReclaimableBytes       int                            `json:"reclaimable_bytes"`
}

func PlanColumnAssetReachability 

func PlanColumnAssetReachability(input ColumnAssetReachabilityInput) (ColumnAssetReachabilityPlan, error)

func PlanColumnAssetReachabilityFromViews 

func PlanColumnAssetReachabilityFromViews(input ColumnAssetReachabilityViewInput) (ColumnAssetReachabilityPlan, error)

type ColumnAssetReachabilityStats 

type ColumnAssetReachabilityStats struct {
	Manifests                      int `json:"manifests"`
	ActiveManifests                int `json:"active_manifests"`
	ProcessVisibleManifests        int `json:"process_visible_manifests"`
	PendingManifests               int `json:"pending_manifests"`
	RootPublishedManifests         int `json:"root_published_manifests"`
	RecoveryAuthoritativeManifests int `json:"recovery_authoritative_manifests"`
	SnapshotPinnedManifests        int `json:"snapshot_pinned_manifests"`
	SupersededManifests            int `json:"superseded_manifests"`
	CleanupSafeManifests           int `json:"cleanup_safe_manifests"`
	PreparedAssets                 int `json:"prepared_assets"`
	QuarantinedAssets              int `json:"quarantined_assets"`
	AssetRefs                      int `json:"asset_refs"`
	SegmentRefs                    int `json:"segment_refs"`
	MetadataBytesScanned           int `json:"metadata_bytes_scanned"`
	TCS1PayloadBytesDecoded        int `json:"tcs1_payload_bytes_decoded"`
	RowsScanned                    int `json:"rows_scanned"`
	ColumnPayloadBlocksRead        int `json:"column_payload_blocks_read"`
	DirectlyDeletableSegments      int `json:"directly_deletable_segments"`
	MixedLiveDeadSegments          int `json:"mixed_live_dead_segments"`
}

type ColumnAssetReachabilitySummary 

type ColumnAssetReachabilitySummary struct {
	Stats                  ColumnAssetReachabilityStats `json:"stats"`
	RetainedBytes          int                          `json:"retained_bytes"`
	PreparedBytes          int                          `json:"prepared_bytes"`
	ProcessVisibleBytes    int                          `json:"process_visible_bytes"`
	PendingBytes           int                          `json:"pending_bytes"`
	RootPublishedBytes     int                          `json:"root_published_bytes"`
	QuarantinedBytes       int                          `json:"quarantined_bytes"`
	SupersededBytes        int                          `json:"superseded_bytes"`
	CleanupSafeBytes       int                          `json:"cleanup_safe_bytes"`
	SnapshotProtectedBytes int                          `json:"snapshot_protected_bytes"`
	RewriteDebtBytes       int                          `json:"rewrite_debt_bytes"`
	ReclaimableBytes       int                          `json:"reclaimable_bytes"`
}

func PlanColumnAssetReachabilitySummaryFromViews 

func PlanColumnAssetReachabilitySummaryFromViews(input ColumnAssetReachabilityViewInput) (ColumnAssetReachabilitySummary, error)

func PlanColumnAssetReachabilitySummaryFromViewsWithScratch 

func PlanColumnAssetReachabilitySummaryFromViewsWithScratch(input ColumnAssetReachabilityViewInput, scratch *ColumnAssetReachabilitySummaryScratch) (ColumnAssetReachabilitySummary, error)

type ColumnAssetReachabilitySummaryScratch 

type ColumnAssetReachabilitySummaryScratch struct {
	// contains filtered or unexported fields
}

type ColumnAssetReachabilityViewInput 

type ColumnAssetReachabilityViewInput struct {
	ActiveManifest                 *ColumnCollectionManifestView
	ProcessVisibleManifests        []ColumnCollectionManifestView
	PendingManifests               []ColumnCollectionManifestView
	RootPublishedManifests         []ColumnCollectionManifestView
	RecoveryAuthoritativeManifests []ColumnCollectionManifestView
	SnapshotPinnedManifests        []ColumnCollectionManifestView
	SupersededManifests            []ColumnCollectionManifestView
	CleanupSafeManifests           []ColumnCollectionManifestView
	PreparedRegistry               *ColumnPreparedAssetRegistryView
	PreparedAssets                 []ColumnPreparedAsset
	QuarantinedAssets              []ColumnPreparedAsset
}

type ColumnAssetRef 

type ColumnAssetRef struct {
	Kind     ColumnAssetKind `json:"kind"`
	FileID   uint32          `json:"file_id"`
	Offset   int64           `json:"offset"`
	Length   int64           `json:"length"`
	Checksum uint32          `json:"checksum"`
}

type ColumnAssetRefDeltaEntry 

type ColumnAssetRefDeltaEntry struct {
	Kind         ColumnAssetRefDeltaKind `json:"kind"`
	Ref          ColumnAssetRef          `json:"ref"`
	Bytes        int                     `json:"bytes"`
	PartID       uint64                  `json:"part_id,omitempty"`
	GenerationID uint64                  `json:"generation_id,omitempty"`
}

type ColumnAssetRefDeltaInput 

type ColumnAssetRefDeltaInput struct {
	PublishedParts  []ColumnManifestPartRef `json:"published_parts,omitempty"`
	SupersededParts []ColumnManifestPartRef `json:"superseded_parts,omitempty"`
	PreparedAssets  []ColumnPreparedAsset   `json:"prepared_assets,omitempty"`
}

type ColumnAssetRefDeltaKind 

type ColumnAssetRefDeltaKind string
const (
	ColumnAssetRefDeltaPublished  ColumnAssetRefDeltaKind = "published"
	ColumnAssetRefDeltaSuperseded ColumnAssetRefDeltaKind = "superseded"
	ColumnAssetRefDeltaPrepared   ColumnAssetRefDeltaKind = "prepared"
)

type ColumnAssetRefDeltaPlan 

type ColumnAssetRefDeltaPlan struct {
	Entries         []ColumnAssetRefDeltaEntry   `json:"entries"`
	Stats           ColumnAssetReachabilityStats `json:"stats"`
	PublishedBytes  int                          `json:"published_bytes"`
	SupersededBytes int                          `json:"superseded_bytes"`
	PreparedBytes   int                          `json:"prepared_bytes"`
}

func PlanColumnAssetRefDelta 

func PlanColumnAssetRefDelta(input ColumnAssetRefDeltaInput) (ColumnAssetRefDeltaPlan, error)

type ColumnAssetStore 

type ColumnAssetStore interface {
	Put(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)
	// Read may return store-owned bytes. Callers must treat the returned bytes as read-only.
	Read(ref ColumnAssetRef) ([]byte, error)
	// ReadTo appends or writes the asset bytes into dst[:0] and returns caller-owned bytes.
	// The returned slice may alias dst and remains valid until the caller mutates or reuses it.
	ReadTo(ref ColumnAssetRef, dst []byte) ([]byte, error)
}

type ColumnAssetSyncedPublishClosure 

type ColumnAssetSyncedPublishClosure struct {
	// contains filtered or unexported fields
}

type ColumnBatch 

type ColumnBatch struct {
	Rows    int
	Columns map[string][]int64
}

type ColumnBlock 

type ColumnBlock struct {
	Descriptor ColumnBlockDescriptor
	Granule    EncodedGranule
}

type ColumnBlockDescriptor 

type ColumnBlockDescriptor struct {
	FirstRow          int
	RowCount          int
	FirstGranule      int
	LastGranule       int
	Encoding          Encoding
	Compression       Compression
	RawBytes          int
	StoredBytes       int
	CodecBlockOrdinal int
}

type ColumnCodecSummary 

type ColumnCodecSummary struct {
	Column               string
	Rows                 int
	Granules             int
	Encoding             Encoding
	RequestedCompression Compression
	ActualCompressionMix map[Compression]int
	ValueBytes           int
	EncodedRawBytes      int
	StoredBytes          int
	CompressionRow       JSONBenchCompressionReportRow
	EncodeDuration       time.Duration
	DecodeDuration       time.Duration
	RangeScanDuration    time.Duration
	RangeScanMatches     int
}

func SummarizeJSONBenchDataset 

func SummarizeJSONBenchDataset(ds JSONBenchDataset, rowsPerGranule int, configs []Config) ([]ColumnCodecSummary, error)

type ColumnCollectionAttachment 

type ColumnCollectionAttachment struct {
	Model             string   `json:"model"`
	SystemMetadataKey string   `json:"system_metadata_key,omitempty"`
	RowPrimaryRoot    string   `json:"row_primary_root,omitempty"`
	ManifestRootRef   string   `json:"column_manifest_root_ref,omitempty"`
	LocatorRootRef    string   `json:"locator_root_ref,omitempty"`
	SecondaryRoots    []string `json:"secondary_roots,omitempty"`
}

type ColumnCollectionByteAccounting 

type ColumnCollectionByteAccounting struct {
	Parts                     int `json:"parts"`
	BaseParts                 int `json:"base_parts"`
	DeltaParts                int `json:"delta_parts"`
	Rows                      int `json:"rows"`
	VisibleRows               int `json:"visible_rows"`
	Tombstones                int `json:"tombstones"`
	DeclaredColumns           int `json:"declared_columns"`
	DescriptorBytes           int `json:"descriptor_bytes"`
	BaseAssetBytes            int `json:"base_asset_bytes"`
	DeltaAssetBytes           int `json:"delta_asset_bytes"`
	TotalAssetBytes           int `json:"total_asset_bytes"`
	ReclaimableCandidateBytes int `json:"reclaimable_candidate_bytes,omitempty"`
}

type ColumnCollectionManifest 

type ColumnCollectionManifest struct {
	Magic             string                         `json:"magic"`
	Version           uint16                         `json:"version"`
	Collection        string                         `json:"collection"`
	SchemaVersion     uint32                         `json:"schema_version"`
	SchemaMode        ColumnSchemaMode               `json:"schema_mode"`
	LogicalPrimaryKey LogicalPrimaryKey              `json:"logical_primary_key"`
	SortKey           []SortKeyColumn                `json:"sort_key,omitempty"`
	DeclaredColumns   []ColumnDefinition             `json:"declared_columns"`
	ActiveGeneration  uint64                         `json:"active_generation"`
	Attachment        ColumnCollectionAttachment     `json:"attachment"`
	PartSet           ColumnPartSetManifest          `json:"part_set"`
	ByteAccounting    ColumnCollectionByteAccounting `json:"byte_accounting"`
	CreatedUnix       int64                          `json:"created_unix_nano"`
	UpdatedUnix       int64                          `json:"updated_unix_nano"`
}

func DecodeColumnCollectionManifest 

func DecodeColumnCollectionManifest(data []byte) (ColumnCollectionManifest, error)

func NewColumnCollectionManifest 

func NewColumnCollectionManifest(collection string, opts ColumnStoreOptions, baseParts []ColumnManifestPartRef, deltaParts []ColumnManifestPartRef, tombstones []ColumnTombstone) (ColumnCollectionManifest, error)

func (ColumnCollectionManifest) ColumnStoreOptions 

func (m ColumnCollectionManifest) ColumnStoreOptions() ColumnStoreOptions

type ColumnCollectionManifestView 

type ColumnCollectionManifestView struct {
	// contains filtered or unexported fields
}

func DecodeColumnCollectionManifestView 

func DecodeColumnCollectionManifestView(data []byte) (ColumnCollectionManifestView, error)

func (ColumnCollectionManifestView) BodyBytes 

func (v ColumnCollectionManifestView) BodyBytes() int

type ColumnCompressionPolicy 

type ColumnCompressionPolicy struct {
	Default Compression
}

type ColumnDefinition 

type ColumnDefinition struct {
	Name           string
	Type           ColumnType
	Encoding       Encoding
	Compression    Compression
	CompressionSet bool
	Cardinality    uint32
	CodecBlockRows int
}

type ColumnManifestPartRef 

type ColumnManifestPartRef struct {
	Role         ColumnPartRole               `json:"role"`
	GenerationID uint64                       `json:"generation_id"`
	Coverage     ColumnPartCoverageDescriptor `json:"coverage"`
	Part         ColumnWorkspacePartManifest  `json:"part"`
}

func NewColumnManifestPartRef 

func NewColumnManifestPartRef(role ColumnPartRole, generationID uint64, part ColumnWorkspacePartManifest) ColumnManifestPartRef

func NewColumnManifestPartRefWithCoverage 

func NewColumnManifestPartRefWithCoverage(role ColumnPartRole, generationID uint64, part ColumnWorkspacePartManifest, sourceParts []ColumnSourcePartGeneration, compactionLevel uint8) ColumnManifestPartRef

func NewColumnManifestPartRefWithCoverageOptions 

func NewColumnManifestPartRefWithCoverageOptions(role ColumnPartRole, generationID uint64, part ColumnWorkspacePartManifest, opts ColumnPartCoverageOptions) ColumnManifestPartRef

type ColumnMutationAdapter 

type ColumnMutationAdapter struct {
	// contains filtered or unexported fields
}

func NewColumnMutationAdapter 

func NewColumnMutationAdapter(workspace *ColumnWorkspace, opts ColumnMutationAdapterOptions) (*ColumnMutationAdapter, error)

func (*ColumnMutationAdapter) Apply 

func (a *ColumnMutationAdapter) Apply(batch ColumnMutationBatch) (ColumnMutationApplyResult, error)

func (*ColumnMutationAdapter) Manifest 

func (a *ColumnMutationAdapter) Manifest() ColumnCollectionManifest

func (*ColumnMutationAdapter) PublishBaseBatch 

func (a *ColumnMutationAdapter) PublishBaseBatch(batch ColumnBatch, coverage ColumnPartCoverageOptions) (ColumnMutationApplyResult, error)

func (*ColumnMutationAdapter) Reader 

func (a *ColumnMutationAdapter) Reader(opts ColumnPartImageReadOptions) (*ColumnPartSetReader, error)

func (*ColumnMutationAdapter) ReplayProfile 

func (a *ColumnMutationAdapter) ReplayProfile() ColumnMutationReplayProfile

type ColumnMutationAdapterOptions 

type ColumnMutationAdapterOptions struct {
	Collection        string
	StoreOptions      ColumnStoreOptions
	Dictionaries      map[string]map[string]int64
	ReplayProfile     ColumnMutationReplayProfile
	InitialPartID     uint64
	InitialGeneration uint64
}

type ColumnMutationApplyResult 

type ColumnMutationApplyResult struct {
	Manifest     ColumnCollectionManifest
	Part         ColumnWorkspacePartManifest
	GenerationID uint64
	InsertedRows int
	UpdatedRows  int
	DeletedRows  int
}

type ColumnMutationBatch 

type ColumnMutationBatch struct {
	Inserts                 ColumnBatch
	Updates                 ColumnBatch
	Deletes                 []int64
	SourceRowRootGeneration uint64
	SourceRowVersionLower   uint64
	SourceRowVersionUpper   uint64
}

type ColumnMutationReplayDurability 

type ColumnMutationReplayDurability string
const (
	ColumnMutationReplayDurable   ColumnMutationReplayDurability = "durable"
	ColumnMutationReplayWALOnFast ColumnMutationReplayDurability = "wal_on_fast"
	ColumnMutationReplayFast      ColumnMutationReplayDurability = "fast"
)

type ColumnMutationReplayProfile 

type ColumnMutationReplayProfile struct {
	Durability    ColumnMutationReplayDurability
	BenchmarkOnly bool
}

func (ColumnMutationReplayProfile) Label 

func (p ColumnMutationReplayProfile) Label() string

Label returns the normalized profile label and is safe to call before Validate.

func (ColumnMutationReplayProfile) ProductionSupported 

func (p ColumnMutationReplayProfile) ProductionSupported() bool

func (ColumnMutationReplayProfile) Validate 

func (p ColumnMutationReplayProfile) Validate() error

type ColumnPart 

type ColumnPart struct {
	Options           ColumnStoreOptions
	Descriptor        ColumnPartDescriptor
	Columns           map[string]ColumnPartColumn
	Marks             []SortKeyMark
	Locators          map[int64]RowLocator
	AggregateMetadata map[string]AggregateMetadata
}

func BuildColumnDeltaPart 

func BuildColumnDeltaPart(partID uint64, opts ColumnStoreOptions, replacements ColumnBatch) (*ColumnPart, error)

func BuildColumnPart 

func BuildColumnPart(partID uint64, opts ColumnStoreOptions, batch ColumnBatch) (*ColumnPart, error)

func BuildJSONBenchColumnPart 

func BuildJSONBenchColumnPart(ds JSONBenchDataset, rowsPerGranule int) (*ColumnPart, error)

func BuildJSONBenchColumnPartForLayout 

func BuildJSONBenchColumnPartForLayout(ds JSONBenchDataset, rowsPerGranule int, layout JSONBenchColumnPartLayout) (*ColumnPart, error)

func BuildJSONBenchColumnPartWithAggregateMetadataForLayout 

func BuildJSONBenchColumnPartWithAggregateMetadataForLayout(ds JSONBenchDataset, rowsPerGranule int, layout JSONBenchColumnPartLayout) (*ColumnPart, error)

func ColumnPartFromImage 

func ColumnPartFromImage(image ColumnPartImage) (*ColumnPart, error)

func ColumnPartFromImageWithOptions 

func ColumnPartFromImageWithOptions(image ColumnPartImage, opts ColumnPartImageReadOptions) (*ColumnPart, error)

func (*ColumnPart) AggregateMetadataByName 

func (p *ColumnPart) AggregateMetadataByName(name string) (AggregateMetadata, bool)

func (*ColumnPart) ByteAccounting 

func (p *ColumnPart) ByteAccounting() ColumnPartByteAccounting

func (*ColumnPart) ByteAccountingFromImage 

func (p *ColumnPart) ByteAccountingFromImage(image ColumnPartImage) ColumnPartByteAccounting

func (*ColumnPart) EncodedColumnBlocks 

func (p *ColumnPart) EncodedColumnBlocks(name string) ([]EncodedGranule, error)

func (*ColumnPart) LocatePrimaryID 

func (p *ColumnPart) LocatePrimaryID(primaryID int64) (RowLocator, bool)

func (*ColumnPart) NewScanner 

func (p *ColumnPart) NewScanner() *ColumnPartScanner

func (*ColumnPart) WithImagePayloads 

func (p *ColumnPart) WithImagePayloads(image ColumnPartImage) (*ColumnPart, error)

type ColumnPartBuilder 

type ColumnPartBuilder struct {
	// contains filtered or unexported fields
}

func NewColumnPartBuilder 

func NewColumnPartBuilder(opts ColumnStoreOptions) (*ColumnPartBuilder, error)

func (*ColumnPartBuilder) Build 

func (b *ColumnPartBuilder) Build(partID uint64, batch ColumnBatch) (*ColumnPart, error)

type ColumnPartByteAccounting 

type ColumnPartByteAccounting struct {
	Rows                      int                                    `json:"rows"`
	Columns                   int                                    `json:"columns"`
	Granules                  int                                    `json:"granules"`
	CodecBlocks               int                                    `json:"codec_blocks"`
	PhysicalFiles             int                                    `json:"physical_files"`
	SerializedImageBytes      int                                    `json:"serialized_image_bytes,omitempty"`
	SerializedManifestBytes   int                                    `json:"serialized_manifest_bytes,omitempty"`
	LogicalValueBytes         int                                    `json:"logical_value_bytes"`
	EncodedRawBytes           int                                    `json:"encoded_raw_bytes"`
	DeclaredColumnStoredBytes int                                    `json:"declared_column_stored_bytes"`
	DictionaryBytes           int                                    `json:"dictionary_bytes"`
	MarkBytes                 int                                    `json:"mark_bytes"`
	SortKeyMetadataBytes      int                                    `json:"sort_key_metadata_bytes"`
	AggregateMetadataBytes    int                                    `json:"aggregate_metadata_bytes"`
	DescriptorBytes           int                                    `json:"descriptor_bytes"`
	LocatorBytes              int                                    `json:"locator_bytes"`
	TotalStoredBytes          int                                    `json:"total_stored_bytes"`
	BytesPerRow               float64                                `json:"bytes_per_row"`
	RetainedJSONPayload       string                                 `json:"retained_json_payload"`
	ColumnsDetail             []ColumnPartColumnByteAccounting       `json:"columns_detail"`
	CompressionDetail         []ColumnPartCompressionByteAccounting  `json:"compression_detail"`
	SerializedSections        []ColumnPartImageSectionByteAccounting `json:"serialized_sections,omitempty"`
}

func (ColumnPartByteAccounting) CategoryBytes 

func (a ColumnPartByteAccounting) CategoryBytes() int

func (*ColumnPartByteAccounting) RecomputeTotals 

func (a *ColumnPartByteAccounting) RecomputeTotals()

type ColumnPartColumn 

type ColumnPartColumn struct {
	Definition ColumnDefinition
	Blocks     []ColumnBlock
}

type ColumnPartColumnByteAccounting 

type ColumnPartColumnByteAccounting struct {
	Column               string         `json:"column"`
	Type                 ColumnType     `json:"type"`
	Rows                 int            `json:"rows"`
	Blocks               int            `json:"blocks"`
	LogicalValueBytes    int            `json:"logical_value_bytes"`
	EncodedRawBytes      int            `json:"encoded_raw_bytes"`
	StoredBytes          int            `json:"stored_bytes"`
	Encoding             Encoding       `json:"encoding"`
	RequestedCompression Compression    `json:"requested_compression"`
	ActualCompressionMix map[string]int `json:"actual_compression_mix"`
	CompressionAttempted int            `json:"compression_attempted"`
	CompressionKept      int            `json:"compression_kept"`
	CompressionRejected  int            `json:"compression_rejected"`
	FallbackReasons      map[string]int `json:"fallback_reasons,omitempty"`
	CompressionNanos     int64          `json:"compression_nanos"`
}

type ColumnPartColumnDescriptor 

type ColumnPartColumnDescriptor struct {
	Name   string
	Type   ColumnType
	Blocks []ColumnBlockDescriptor
}

type ColumnPartCompressionByteAccounting 

type ColumnPartCompressionByteAccounting struct {
	Column                 string      `json:"column"`
	Substream              string      `json:"substream"`
	Encoding               Encoding    `json:"encoding"`
	RequestedCompression   Compression `json:"requested_compression"`
	ActualCompression      Compression `json:"actual_compression"`
	Blocks                 int         `json:"blocks"`
	CompressionAttempted   int         `json:"compression_attempted"`
	CompressionKept        int         `json:"compression_kept"`
	CompressionRejected    int         `json:"compression_rejected"`
	FallbackReason         string      `json:"fallback_reason,omitempty"`
	EncodedRawBytes        int         `json:"encoded_raw_bytes"`
	StoredBytes            int         `json:"stored_bytes"`
	CompressionNanos       int64       `json:"compression_nanos"`
	StoredToEncodedRawRate float64     `json:"stored_to_encoded_raw_rate"`
}

type ColumnPartCoverageDescriptor 

type ColumnPartCoverageDescriptor struct {
	Role                    ColumnPartRole               `json:"role"`
	GenerationID            uint64                       `json:"generation_id"`
	CompactionLevel         uint8                        `json:"compaction_level,omitempty"`
	SourceParts             []ColumnSourcePartGeneration `json:"source_parts,omitempty"`
	SourceRowRootGeneration uint64                       `json:"source_row_root_generation,omitempty"`
	SourceRowVersionLower   uint64                       `json:"source_row_version_lower,omitempty"`
	SourceRowVersionUpper   uint64                       `json:"source_row_version_upper_exclusive,omitempty"`
	PrimaryIDLower          int64                        `json:"primary_id_lower"`
	PrimaryIDUpperExclusive int64                        `json:"primary_id_upper_exclusive"`
	SortKeyColumns          []string                     `json:"sort_key_columns,omitempty"`
	SortKeyLower            []int64                      `json:"sort_key_lower,omitempty"`
	SortKeyUpperExclusive   []int64                      `json:"sort_key_upper_exclusive,omitempty"`
	SortKeyUpperUnbounded   bool                         `json:"sort_key_upper_unbounded,omitempty"`
	Rows                    int                          `json:"rows"`
	VisibleRows             int                          `json:"visible_rows"`
	DeletedRows             int                          `json:"deleted_rows,omitempty"`
	AssetRefs               []ColumnAssetRef             `json:"asset_refs"`
	Checksums               []uint32                     `json:"checksums"`
}

type ColumnPartCoverageOptions 

type ColumnPartCoverageOptions struct {
	SourceParts             []ColumnSourcePartGeneration
	CompactionLevel         uint8
	SourceRowRootGeneration uint64
	SourceRowVersionLower   uint64
	SourceRowVersionUpper   uint64
}

type ColumnPartDescriptor 

type ColumnPartDescriptor struct {
	Version           uint8
	PartID            uint64
	SchemaVersion     uint32
	RowCount          int
	VisibleRowCount   int
	LogicalPrimaryKey []string
	SortKey           []SortKeyColumn
	Granules          []GranuleDescriptor
	Columns           []ColumnPartColumnDescriptor
}

type ColumnPartImage 

type ColumnPartImage struct {
	Version       uint16                   `json:"version"`
	PartID        uint64                   `json:"part_id"`
	Rows          int                      `json:"rows"`
	ManifestBytes int                      `json:"manifest_bytes"`
	Sections      []ColumnPartImageSection `json:"sections"`
	Bytes         []byte                   `json:"-"`
}

func BuildColumnPartImage 

func BuildColumnPartImage(part *ColumnPart, opts ColumnPartImageOptions) (ColumnPartImage, error)

func ParseColumnPartImage 

func ParseColumnPartImage(data []byte) (ColumnPartImage, error)

func (ColumnPartImage) CategoryBytes 

func (i ColumnPartImage) CategoryBytes(category ColumnPartImageSectionCategory) int

func (ColumnPartImage) Dictionaries 

func (i ColumnPartImage) Dictionaries() (map[string]map[string]int64, error)

func (ColumnPartImage) SectionByteAccounting 

func (i ColumnPartImage) SectionByteAccounting() []ColumnPartImageSectionByteAccounting

func (ColumnPartImage) TotalBytes 

func (i ColumnPartImage) TotalBytes() int

type ColumnPartImageOptions 

type ColumnPartImageOptions struct {
	Dictionaries map[string]map[string]int64
}

type ColumnPartImageReadOptions 

type ColumnPartImageReadOptions struct {
	IncludeRowLocators       bool
	ValidateRowLocators      bool
	IncludeAggregateMetadata bool
}

type ColumnPartImageSection 

type ColumnPartImageSection struct {
	Kind        ColumnPartImageSectionKind     `json:"kind"`
	Category    ColumnPartImageSectionCategory `json:"category"`
	Name        string                         `json:"name,omitempty"`
	Column      string                         `json:"column,omitempty"`
	Offset      int                            `json:"offset"`
	Length      int                            `json:"length"`
	Rows        int                            `json:"rows,omitempty"`
	Granules    int                            `json:"granules,omitempty"`
	Blocks      int                            `json:"blocks,omitempty"`
	Encoding    Encoding                       `json:"encoding,omitempty"`
	Compression Compression                    `json:"compression,omitempty"`
}

type ColumnPartImageSectionByteAccounting 

type ColumnPartImageSectionByteAccounting struct {
	Kind     ColumnPartImageSectionKind     `json:"kind"`
	Category ColumnPartImageSectionCategory `json:"category"`
	Name     string                         `json:"name,omitempty"`
	Column   string                         `json:"column,omitempty"`
	Bytes    int                            `json:"bytes"`
}

type ColumnPartImageSectionCategory 

type ColumnPartImageSectionCategory string
const (
	ColumnPartImageCategoryManifest          ColumnPartImageSectionCategory = "manifest"
	ColumnPartImageCategoryDescriptor        ColumnPartImageSectionCategory = "descriptor"
	ColumnPartImageCategorySortKeyMetadata   ColumnPartImageSectionCategory = "sort_key_metadata"
	ColumnPartImageCategoryMarks             ColumnPartImageSectionCategory = "marks"
	ColumnPartImageCategoryLocators          ColumnPartImageSectionCategory = "locators"
	ColumnPartImageCategoryAggregateMetadata ColumnPartImageSectionCategory = "aggregate_metadata"
	ColumnPartImageCategoryDictionaries      ColumnPartImageSectionCategory = "dictionaries"
	ColumnPartImageCategoryDeclaredColumns   ColumnPartImageSectionCategory = "declared_columns"
)

type ColumnPartImageSectionKind 

type ColumnPartImageSectionKind string
const (
	ColumnPartImageSectionManifest          ColumnPartImageSectionKind = "manifest"
	ColumnPartImageSectionDescriptor        ColumnPartImageSectionKind = "descriptor"
	ColumnPartImageSectionSortKeyMetadata   ColumnPartImageSectionKind = "sort_key_metadata"
	ColumnPartImageSectionSortKeyMarks      ColumnPartImageSectionKind = "sort_key_marks"
	ColumnPartImageSectionRowLocators       ColumnPartImageSectionKind = "row_locators"
	ColumnPartImageSectionAggregateMetadata ColumnPartImageSectionKind = "aggregate_metadata"
	ColumnPartImageSectionDictionaries      ColumnPartImageSectionKind = "dictionaries"
	ColumnPartImageSectionColumnData        ColumnPartImageSectionKind = "column_data"
)

type ColumnPartPolicy 

type ColumnPartPolicy struct {
	RowsPerGranule        int
	DefaultCodecBlockRows int
	AdaptiveMarkSizing    ColumnAdaptiveMarkSizing
}

type ColumnPartRole 

type ColumnPartRole string
const (
	ColumnPartRoleBase  ColumnPartRole = "base"
	ColumnPartRoleDelta ColumnPartRole = "delta"
)

type ColumnPartScanner 

type ColumnPartScanner struct {
	// contains filtered or unexported fields
}

func (*ColumnPartScanner) ScanProjected 

func (s *ColumnPartScanner) ScanProjected(columns []string) (ProjectedScanResult, error)

func (*ColumnPartScanner) ScanProjectedInto 

func (s *ColumnPartScanner) ScanProjectedInto(dst map[string][]int64, columns []string) (ProjectedScanResult, error)

func (*ColumnPartScanner) ValueAt 

func (s *ColumnPartScanner) ValueAt(locator RowLocator, columnName string) (int64, error)

type ColumnPartSetCompactionPlan 

type ColumnPartSetCompactionPlan struct {
	ShouldCompact                bool     `json:"should_compact"`
	Reasons                      []string `json:"reasons,omitempty"`
	SelectedParts                int      `json:"selected_parts"`
	SkippedParts                 int      `json:"skipped_parts"`
	BaseParts                    int      `json:"base_parts"`
	DeltaParts                   int      `json:"delta_parts"`
	Tombstones                   int      `json:"tombstones"`
	ReadAmplificationParts       int      `json:"read_amplification_parts"`
	LiveBytes                    int      `json:"live_bytes"`
	StaleBytes                   int      `json:"stale_bytes"`
	TombstoneDebt                int      `json:"tombstone_debt"`
	ExpectedReclaimPPM           int      `json:"expected_reclaim_ppm"`
	VisibleRowsPPM               int      `json:"visible_rows_ppm"`
	AggregateMetadataInvalid     bool     `json:"aggregate_metadata_invalid"`
	AggregateMetadataRebuilds    bool     `json:"aggregate_metadata_rebuilds"`
	ColumnAssetStaleBytePressure bool     `json:"column_asset_stale_byte_pressure"`
}

func PlanColumnPartSetCompaction 

func PlanColumnPartSetCompaction(manifest ColumnCollectionManifest, stats ColumnPartSetVisibilityStats, policy ColumnPartSetCompactionPolicy) (ColumnPartSetCompactionPlan, error)

type ColumnPartSetCompactionPolicy 

type ColumnPartSetCompactionPolicy struct {
	MaxDeltaParts             int `json:"max_delta_parts,omitempty"`
	MaxDeltaBytes             int `json:"max_delta_bytes,omitempty"`
	MaxTombstones             int `json:"max_tombstones,omitempty"`
	MaxReadAmplificationParts int `json:"max_read_amplification_parts,omitempty"`
	MaxStaleBytes             int `json:"max_stale_bytes,omitempty"`
	MinExpectedReclaimPPM     int `json:"min_expected_reclaim_ppm,omitempty"`
	MinVisibleRowsPPM         int `json:"min_visible_rows_ppm,omitempty"`
}

type ColumnPartSetCompactionResult 

type ColumnPartSetCompactionResult struct {
	Manifest                ColumnCollectionManifest    `json:"manifest"`
	Part                    ColumnWorkspacePartManifest `json:"part"`
	InputRows               int                         `json:"input_rows"`
	VisibleRows             int                         `json:"visible_rows"`
	DroppedRows             int                         `json:"dropped_rows"`
	SupersededRows          int                         `json:"superseded_rows"`
	DeletedRows             int                         `json:"deleted_rows"`
	OldAssetBytes           int                         `json:"old_asset_bytes"`
	NewAssetBytes           int                         `json:"new_asset_bytes"`
	ReclaimableBytes        int                         `json:"reclaimable_bytes"`
	RewriteDebtBytes        int                         `json:"rewrite_debt_bytes"`
	NetBytesReduced         int                         `json:"net_bytes_reduced"`
	SelectionPlan           ColumnPartSetCompactionPlan `json:"selection_plan"`
	PrePublishReachability  ColumnAssetReachabilityPlan `json:"pre_publish_reachability"`
	PostPublishReachability ColumnAssetReachabilityPlan `json:"post_publish_reachability"`
	CompactionUnix          int64                       `json:"compaction_unix_nano"`
}

func CompactColumnPartSet 

func CompactColumnPartSet(workspace *ColumnWorkspace, reader *ColumnPartSetReader, opts ColumnStoreOptions, dictionaries map[string]map[string]int64, newPartID uint64) (ColumnPartSetCompactionResult, error)

type ColumnPartSetManifest 

type ColumnPartSetManifest struct {
	BaseParts  []ColumnManifestPartRef `json:"base_parts,omitempty"`
	DeltaParts []ColumnManifestPartRef `json:"delta_parts,omitempty"`
	Tombstones []ColumnTombstone       `json:"tombstones,omitempty"`
}

type ColumnPartSetPointLookupScratch 

type ColumnPartSetPointLookupScratch struct {
	// contains filtered or unexported fields
}

type ColumnPartSetReader 

type ColumnPartSetReader struct {
	// contains filtered or unexported fields
}

func OpenColumnPartSetReader 

func OpenColumnPartSetReader(workspace *ColumnWorkspace, manifest ColumnCollectionManifest, opts ColumnPartImageReadOptions) (*ColumnPartSetReader, error)

func (*ColumnPartSetReader) CacheStats 

func (r *ColumnPartSetReader) CacheStats() ColumnWorkspaceCacheStats

func (*ColumnPartSetReader) LatestLocator 

func (r *ColumnPartSetReader) LatestLocator(primaryID int64) (RowLocator, bool)

func (*ColumnPartSetReader) Manifest 

func (r *ColumnPartSetReader) Manifest() ColumnCollectionManifest

func (*ColumnPartSetReader) ScanLatestLocator 

func (r *ColumnPartSetReader) ScanLatestLocator(primaryID int64) (RowLocator, bool)

func (*ColumnPartSetReader) ScanProjected 

func (r *ColumnPartSetReader) ScanProjected(columns []string) (ColumnPartSetScanResult, error)

func (*ColumnPartSetReader) ScanProjectedInto 

func (r *ColumnPartSetReader) ScanProjectedInto(dst map[string][]int64, columns []string) (ColumnPartSetScanResult, error)

func (*ColumnPartSetReader) ScanValueAtLatest 

func (r *ColumnPartSetReader) ScanValueAtLatest(primaryID int64, columnName string) (int64, bool, error)

func (*ColumnPartSetReader) ScanValueAtLatestWithScratch 

func (r *ColumnPartSetReader) ScanValueAtLatestWithScratch(primaryID int64, columnName string, scratch *ColumnPartSetPointLookupScratch) (int64, bool, error)

func (*ColumnPartSetReader) ValueAtLatest 

func (r *ColumnPartSetReader) ValueAtLatest(primaryID int64, columnName string) (int64, bool, error)

func (*ColumnPartSetReader) ValueAtLatestWithScratch 

func (r *ColumnPartSetReader) ValueAtLatestWithScratch(primaryID int64, columnName string, scratch *ColumnPartSetPointLookupScratch) (int64, bool, error)

func (*ColumnPartSetReader) VisibilityStats 

func (r *ColumnPartSetReader) VisibilityStats() ColumnPartSetVisibilityStats

type ColumnPartSetScanDiagnostics 

type ColumnPartSetScanDiagnostics struct {
	RowsReturned       int                       `json:"rows_returned"`
	RowsScanned        int                       `json:"rows_scanned"`
	RowsSuperseded     int                       `json:"rows_superseded"`
	RowsDeleted        int                       `json:"rows_deleted"`
	PartsConsidered    int                       `json:"parts_considered"`
	BaseParts          int                       `json:"base_parts"`
	DeltaParts         int                       `json:"delta_parts"`
	Tombstones         int                       `json:"tombstones"`
	ColumnsProjected   []string                  `json:"columns_projected"`
	GranulesConsidered int                       `json:"granules_considered"`
	BlocksDecoded      int                       `json:"blocks_decoded"`
	BytesDecoded       int                       `json:"bytes_decoded"`
	CacheStats         ColumnWorkspaceCacheStats `json:"cache_stats"`
}

type ColumnPartSetScanResult 

type ColumnPartSetScanResult struct {
	Rows        int
	Columns     map[string][]int64
	Diagnostics ColumnPartSetScanDiagnostics
}

type ColumnPartSetVisibilityStats 

type ColumnPartSetVisibilityStats struct {
	Parts          int `json:"parts"`
	BaseParts      int `json:"base_parts"`
	DeltaParts     int `json:"delta_parts"`
	InputRows      int `json:"input_rows"`
	VisibleRows    int `json:"visible_rows"`
	SupersededRows int `json:"superseded_rows"`
	DeletedRows    int `json:"deleted_rows"`
	Tombstones     int `json:"tombstones"`
}

type ColumnPreparedAsset 

type ColumnPreparedAsset struct {
	Ref          ColumnAssetRef `json:"ref"`
	Bytes        int            `json:"bytes"`
	PublishID    uint64         `json:"publish_id,omitempty"`
	GenerationID uint64         `json:"generation_id,omitempty"`
	Reason       string         `json:"reason,omitempty"`
}

type ColumnPreparedAssetRegistry 

type ColumnPreparedAssetRegistry struct {
	Magic        string                `json:"magic"`
	Version      uint16                `json:"version"`
	Collection   string                `json:"collection,omitempty"`
	PublishID    uint64                `json:"publish_id"`
	GenerationID uint64                `json:"generation_id"`
	UpdatedUnix  int64                 `json:"updated_unix_nano"`
	Assets       []ColumnPreparedAsset `json:"assets,omitempty"`
}

type ColumnPreparedAssetRegistryView 

type ColumnPreparedAssetRegistryView struct {
	// contains filtered or unexported fields
}

func DecodeColumnPreparedAssetRegistryView 

func DecodeColumnPreparedAssetRegistryView(data []byte) (ColumnPreparedAssetRegistryView, error)

func (ColumnPreparedAssetRegistryView) BodyBytes 

func (v ColumnPreparedAssetRegistryView) BodyBytes() int

type ColumnSchemaMode 

type ColumnSchemaMode string
const (
	ColumnSchemaFixed ColumnSchemaMode = "fixed"
)

type ColumnSourcePartGeneration 

type ColumnSourcePartGeneration struct {
	PartID       uint64 `json:"part_id"`
	GenerationID uint64 `json:"generation_id"`
}

type ColumnStoreOptions 

type ColumnStoreOptions struct {
	SchemaVersion     uint32
	SchemaMode        ColumnSchemaMode
	Columns           []ColumnDefinition
	LogicalPrimaryKey LogicalPrimaryKey
	SortKey           SortKey
	PartPolicy        ColumnPartPolicy
	Compression       ColumnCompressionPolicy
	AggregateMetadata []AggregateMetadataDefinition
}

func JSONBenchColumnPartOptions 

func JSONBenchColumnPartOptions(ds JSONBenchDataset, rowsPerGranule int) (ColumnStoreOptions, error)

func JSONBenchColumnPartOptionsForLayout 

func JSONBenchColumnPartOptionsForLayout(ds JSONBenchDataset, rowsPerGranule int, layout JSONBenchColumnPartLayout) (ColumnStoreOptions, error)

func JSONBenchColumnPartOptionsWithAggregateMetadataForLayout 

func JSONBenchColumnPartOptionsWithAggregateMetadataForLayout(ds JSONBenchDataset, rowsPerGranule int, layout JSONBenchColumnPartLayout) (ColumnStoreOptions, error)

type ColumnTombstone 

type ColumnTombstone struct {
	PrimaryID     int64  `json:"primary_id"`
	GenerationID  uint64 `json:"generation_id"`
	Reason        string `json:"reason,omitempty"`
	PreparedBytes int    `json:"prepared_bytes,omitempty"`
}

type ColumnType 

type ColumnType string
const (
	ColumnTypeInt64              ColumnType = "int64"
	ColumnTypeLowCardinalityCode ColumnType = "low_cardinality_code"
	ColumnTypeBool               ColumnType = "bool"
)

type ColumnWorkspace 

type ColumnWorkspace struct {
	// contains filtered or unexported fields
}

func OpenColumnWorkspace 

func OpenColumnWorkspace(dir string, opts ColumnWorkspaceOptions) (*ColumnWorkspace, error)

func (*ColumnWorkspace) CacheStats 

func (w *ColumnWorkspace) CacheStats() ColumnWorkspaceCacheStats

func (*ColumnWorkspace) ClearPreparedAssetRegistry 

func (w *ColumnWorkspace) ClearPreparedAssetRegistry() error

func (*ColumnWorkspace) Close 

func (w *ColumnWorkspace) Close() error

func (*ColumnWorkspace) Dir 

func (w *ColumnWorkspace) Dir() string

func (*ColumnWorkspace) InventoryNamespace 

func (w *ColumnWorkspace) InventoryNamespace(manifests ...ColumnCollectionManifest) (ColumnWorkspaceNamespaceInventory, error)

func (*ColumnWorkspace) LoadCollectionManifest 

func (w *ColumnWorkspace) LoadCollectionManifest() (ColumnCollectionManifest, error)

func (*ColumnWorkspace) LoadPart 

func (w *ColumnWorkspace) LoadPart(partID uint64) (ColumnWorkspaceLoadResult, error)

func (*ColumnWorkspace) LoadPartWithOptions 

func (w *ColumnWorkspace) LoadPartWithOptions(partID uint64, opts ColumnPartImageReadOptions) (ColumnWorkspaceLoadResult, error)

func (*ColumnWorkspace) LoadPreparedAssetRegistry 

func (w *ColumnWorkspace) LoadPreparedAssetRegistry() (ColumnPreparedAssetRegistry, error)

func (*ColumnWorkspace) Manifest 

func (w *ColumnWorkspace) Manifest() ColumnWorkspaceManifest

func (*ColumnWorkspace) ManifestSyncMode 

func (w *ColumnWorkspace) ManifestSyncMode() ColumnWorkspaceManifestSyncMode

func (*ColumnWorkspace) Namespace 

func (w *ColumnWorkspace) Namespace() ColumnWorkspaceNamespace

func (*ColumnWorkspace) PublishPart 

func (w *ColumnWorkspace) PublishPart(part *ColumnPart, dictionaries map[string]map[string]int64) (ColumnWorkspacePartManifest, error)

func (*ColumnWorkspace) SaveCollectionManifest 

func (w *ColumnWorkspace) SaveCollectionManifest(manifest ColumnCollectionManifest) error

func (*ColumnWorkspace) SavePreparedAssetRegistry 

func (w *ColumnWorkspace) SavePreparedAssetRegistry(publishID uint64, generationID uint64, assets []ColumnPreparedAsset) error

type ColumnWorkspaceCacheCounter 

type ColumnWorkspaceCacheCounter struct {
	Hits   uint64 `json:"hits"`
	Misses uint64 `json:"misses"`
}

type ColumnWorkspaceCacheStats 

type ColumnWorkspaceCacheStats struct {
	MarkCache       ColumnWorkspaceCacheCounter `json:"mark_cache"`
	CompressedCache ColumnWorkspaceCacheCounter `json:"compressed_block_cache"`
	DecodedCache    ColumnWorkspaceCacheCounter `json:"decoded_block_cache"`
	DictionaryCache ColumnWorkspaceCacheCounter `json:"dictionary_cache"`
}

type ColumnWorkspaceLoadResult 

type ColumnWorkspaceLoadResult struct {
	Part       *ColumnPart
	Manifest   ColumnWorkspacePartManifest
	TCS1       TCS1PartRecord
	CacheState string
	CacheStats ColumnWorkspaceCacheStats
}

type ColumnWorkspaceManifest 

type ColumnWorkspaceManifest struct {
	Magic       string                        `json:"magic"`
	Version     uint16                        `json:"version"`
	Collection  string                        `json:"collection,omitempty"`
	Generation  uint64                        `json:"generation_id"`
	PublishID   uint64                        `json:"publish_id"`
	CreatedUnix int64                         `json:"created_unix_nano"`
	UpdatedUnix int64                         `json:"updated_unix_nano"`
	Parts       []ColumnWorkspacePartManifest `json:"parts,omitempty"`
}

type ColumnWorkspaceManifestSyncMode 

type ColumnWorkspaceManifestSyncMode string

type ColumnWorkspaceNamespace 

type ColumnWorkspaceNamespace struct {
	RootDir       string `json:"root_dir"`
	ManifestDir   string `json:"manifest_dir"`
	AssetDir      string `json:"asset_dir"`
	SegmentDir    string `json:"segment_dir"`
	IndexDir      string `json:"index_dir"`
	PreparedDir   string `json:"prepared_dir"`
	QuarantineDir string `json:"quarantine_dir"`
	TempDir       string `json:"temp_dir"`
}

func ColumnWorkspaceNamespaceForDir 

func ColumnWorkspaceNamespaceForDir(dir string) ColumnWorkspaceNamespace

type ColumnWorkspaceNamespaceInventory 

type ColumnWorkspaceNamespaceInventory struct {
	SegmentFiles            []ColumnWorkspaceSegmentFile `json:"segment_files"`
	PreparedRegistryPresent bool                         `json:"prepared_registry_present"`
	PreparedAssets          []ColumnPreparedAsset        `json:"prepared_assets,omitempty"`
	OrphanPreparedAssets    []ColumnPreparedAsset        `json:"orphan_prepared_assets,omitempty"`
	ReferencedAssets        int                          `json:"referenced_assets"`
}

type ColumnWorkspaceOptions 

type ColumnWorkspaceOptions struct {
	Collection       string
	ValidationMode   ColumnWorkspaceValidationMode
	ManifestSyncMode ColumnWorkspaceManifestSyncMode
	// contains filtered or unexported fields
}

type ColumnWorkspacePartCoverage 

type ColumnWorkspacePartCoverage struct {
	PrimaryIDLower          int64    `json:"primary_id_lower"`
	PrimaryIDUpperExclusive int64    `json:"primary_id_upper_exclusive"`
	SortKeyColumns          []string `json:"sort_key_columns,omitempty"`
	SortKeyLower            []int64  `json:"sort_key_lower,omitempty"`
	SortKeyUpperExclusive   []int64  `json:"sort_key_upper_exclusive,omitempty"`
	SortKeyUpperUnbounded   bool     `json:"sort_key_upper_unbounded,omitempty"`
}

type ColumnWorkspacePartManifest 

type ColumnWorkspacePartManifest struct {
	PartID        uint64                      `json:"part_id"`
	Rows          int                         `json:"rows"`
	VisibleRows   int                         `json:"visible_rows"`
	SchemaVersion uint32                      `json:"schema_version"`
	SortKey       []SortKeyColumn             `json:"sort_key,omitempty"`
	Coverage      ColumnWorkspacePartCoverage `json:"coverage"`
	AssetRef      ColumnAssetRef              `json:"asset_ref"`
	TCS1          TCS1PartRecord              `json:"tcs1"`
	ImageBytes    int                         `json:"image_bytes"`
	ManifestBytes int                         `json:"manifest_bytes"`
	Sections      int                         `json:"sections"`
	AssetBytes    int                         `json:"asset_bytes"`
	PublishedUnix int64                       `json:"published_unix_nano"`
}

type ColumnWorkspaceSegmentFile 

type ColumnWorkspaceSegmentFile struct {
	FileID uint32 `json:"file_id"`
	Path   string `json:"path"`
	Bytes  int64  `json:"bytes"`
}

type ColumnWorkspaceValidationMode 

type ColumnWorkspaceValidationMode string

type Compression 

type Compression uint8
const (
	CompressionNone Compression = iota
	CompressionSnappy
	CompressionLZ4
	CompressionZSTD
	CompressionZSTDDict
)

func (Compression) String 

func (c Compression) String() string

type CompressionSelection 

type CompressionSelection struct {
	Payload []byte
	Actual  Compression
	Scratch []byte
	Report  CodecReport
}

type Config 

type Config struct {
	Encoding    Encoding
	Compression Compression
}

func DefaultJSONBenchConfigs 

func DefaultJSONBenchConfigs() []Config

type EncodedGranule 

type EncodedGranule struct {
	Rows         int
	NullCount    int
	DefaultCount int
	HasMinMax    bool
	Min          int64
	Max          int64
	Encoding     Encoding
	Compression  Compression
	RawBytes     int
	StoredBytes  int
	PayloadRef   PayloadRef
	CodecReport  CodecReport
	Payload      []byte
}

func EncodeInt64 

func EncodeInt64(dst []byte, values []int64, cfg Config) (EncodedGranule, error)

type Encoding 

type Encoding uint8
const (
	EncodingRawInt64 Encoding = iota + 1
	EncodingDeltaVarint
	EncodingDoubleDeltaVarint
	EncodingNullableInt64
	EncodingBoolBitpackRLE
	EncodingLowCardinalityUint32
)

func (Encoding) String 

func (e Encoding) String() string

type GranuleBuilder 

type GranuleBuilder struct {
	// contains filtered or unexported fields
}

func NewGranuleBuilder 

func NewGranuleBuilder(cfg Config) *GranuleBuilder

func (*GranuleBuilder) BuildBool 

func (b *GranuleBuilder) BuildBool(values []bool) (EncodedGranule, error)

BuildBool returns a granule whose payload aliases builder-owned scratch until the next builder Build* or Reset call.

func (*GranuleBuilder) BuildInt64 

func (b *GranuleBuilder) BuildInt64(values []int64) (EncodedGranule, error)

BuildInt64 returns a granule whose payload aliases builder-owned scratch until the next BuildInt64 or Reset call.

func (*GranuleBuilder) BuildNullableInt64 

func (b *GranuleBuilder) BuildNullableInt64(values []int64, nulls []bool, defaults []bool, defaultValue int64) (EncodedGranule, error)

BuildNullableInt64 returns a granule whose payload aliases builder-owned scratch until the next builder Build* or Reset call.

func (*GranuleBuilder) BuildUint32Codes 

func (b *GranuleBuilder) BuildUint32Codes(codes []uint32, cardinality uint32) (EncodedGranule, error)

BuildUint32Codes returns a granule whose payload aliases builder-owned scratch until the next builder Build* or Reset call.

func (*GranuleBuilder) Reset 

func (b *GranuleBuilder) Reset(cfg Config)

type GranuleDescriptor 

type GranuleDescriptor struct {
	Ordinal          int
	FirstRow         int
	RowCount         int
	VisibleRows      int
	DeletedRows      int
	IDLower          int64
	IDUpperExclusive int64
	MarkOrdinal      int
}

type GranuleReader 

type GranuleReader struct {
	// contains filtered or unexported fields
}

func (*GranuleReader) CountInt64RangeWithDiagnostics 

func (r *GranuleReader) CountInt64RangeWithDiagnostics(granules []EncodedGranule, marks []SortKeyMark, plan PredicatePlan) (int, PredicateDiagnostics, error)

func (*GranuleReader) CountTrueBool 

func (r *GranuleReader) CountTrueBool(g EncodedGranule) (int, error)

func (*GranuleReader) CountUint32Codes 

func (r *GranuleReader) CountUint32Codes(g EncodedGranule, counts []int) ([]int, error)

func (*GranuleReader) DecodeBool 

func (r *GranuleReader) DecodeBool(g EncodedGranule) ([]bool, error)

func (*GranuleReader) DecodeBoolInto 

func (r *GranuleReader) DecodeBoolInto(dst []bool, g EncodedGranule) ([]bool, error)

func (*GranuleReader) DecodeInt64 

func (r *GranuleReader) DecodeInt64(g EncodedGranule) ([]int64, error)

func (*GranuleReader) DecodeInt64Into 

func (r *GranuleReader) DecodeInt64Into(dst []int64, g EncodedGranule) ([]int64, error)

func (*GranuleReader) DecodeNullableInt64 

func (r *GranuleReader) DecodeNullableInt64(g EncodedGranule) ([]int64, []bool, []bool, error)

func (*GranuleReader) DecodeNullableInt64Into 

func (r *GranuleReader) DecodeNullableInt64Into(dst []int64, nulls []bool, defaults []bool, g EncodedGranule) ([]int64, []bool, []bool, error)

func (*GranuleReader) DecodeUint32Codes 

func (r *GranuleReader) DecodeUint32Codes(g EncodedGranule) ([]uint32, error)

func (*GranuleReader) DecodeUint32CodesInto 

func (r *GranuleReader) DecodeUint32CodesInto(dst []uint32, g EncodedGranule) ([]uint32, error)

func (*GranuleReader) RangeScanCountInt64 

func (r *GranuleReader) RangeScanCountInt64(g EncodedGranule, low, high int64) (int, error)

type Int64RangePredicate 

type Int64RangePredicate struct {
	Column string
	Low    int64
	High   int64
}

func (Int64RangePredicate) Empty 

func (p Int64RangePredicate) Empty() bool

type JSONBenchColumnPartLayout 

type JSONBenchColumnPartLayout string
const (
	JSONBenchColumnPartLayoutTimeUS                   JSONBenchColumnPartLayout = "time_us"
	JSONBenchColumnPartLayoutClickHouseFilterUserTime JSONBenchColumnPartLayout = "clickhouse_filter_user_time"
)

type JSONBenchCompressionReportRow 

type JSONBenchCompressionReportRow struct {
	CodecLayoutLabel            string        `json:"codec_layout_label"`
	CompressionPolicyLabel      string        `json:"compression_policy_label"`
	RequestedCompression        string        `json:"requested_compression"`
	ActualCompression           string        `json:"actual_compression"`
	SupportState                string        `json:"support_state"`
	SupportReason               string        `json:"support_reason,omitempty"`
	CompressedBytes             int           `json:"compressed_bytes"`
	CompressedBytesSource       string        `json:"compressed_bytes_source"`
	RawBytes                    int           `json:"raw_bytes"`
	RawBytesSource              string        `json:"raw_bytes_source"`
	DecompressedBytes           int           `json:"decompressed_bytes"`
	DecompressedBytesSource     string        `json:"decompressed_bytes_source"`
	CompressionRatio            float64       `json:"compression_ratio"`
	CompressionRatioSource      string        `json:"compression_ratio_source"`
	CompressionDuration         time.Duration `json:"compression_duration"`
	CompressionDurationSource   string        `json:"compression_duration_source"`
	DecompressionDuration       time.Duration `json:"decompression_duration"`
	DecompressionDurationSource string        `json:"decompression_duration_source"`
	BenchmarkBPerOp             uint64        `json:"benchmark_b_per_op"`
	BenchmarkAllocsPerOp        uint64        `json:"benchmark_allocs_per_op"`
	BenchmarkAllocationSource   string        `json:"benchmark_allocation_source"`
}

type JSONBenchDataset 

type JSONBenchDataset struct {
	Rows         int
	Files        []string
	Columns      map[string][]int64
	Dictionaries map[string]map[string]int64
}

func LoadJSONBenchColumns 

func LoadJSONBenchColumns(path string, limit int) (JSONBenchDataset, error)

func (JSONBenchDataset) ColumnNames 

func (d JSONBenchDataset) ColumnNames() []string

type JSONBenchDeclaredColumnValues 

type JSONBenchDeclaredColumnValues struct {
	TimeUS           int64  `json:"time_us"`
	Did              string `json:"did"`
	Kind             string `json:"kind"`
	CommitOperation  string `json:"commit_operation,omitempty"`
	CommitCollection string `json:"commit_collection,omitempty"`
}

func JSONBenchDeclaredColumnValuesFromPart 

func JSONBenchDeclaredColumnValuesFromPart(scanner *ColumnPartScanner, part *ColumnPart, dictionaries map[string]map[int64]string, primaryID int64) (JSONBenchDeclaredColumnValues, error)

type JSONBenchPartBuildAttempt 

type JSONBenchPartBuildAttempt struct {
	Duration                  time.Duration `json:"duration"`
	AllocatedBytes            uint64        `json:"allocated_bytes"`
	Mallocs                   uint64        `json:"mallocs"`
	TemporaryBytes            uint64        `json:"temporary_bytes_estimate"`
	TotalBytes                int           `json:"total_bytes"`
	EncodedRawBytes           int           `json:"encoded_raw_bytes"`
	DeclaredColumnStoredBytes int           `json:"declared_column_stored_bytes"`
}

type JSONBenchPartBuildReport 

type JSONBenchPartBuildReport struct {
	Layout               JSONBenchColumnPartLayout       `json:"layout"`
	Rows                 int                             `json:"rows"`
	RowsPerGranule       int                             `json:"rows_per_granule"`
	Columns              int                             `json:"columns"`
	Attempts             []JSONBenchPartBuildAttempt     `json:"attempts"`
	Best                 JSONBenchPartBuildAttempt       `json:"best"`
	Accounting           ColumnPartByteAccounting        `json:"accounting"`
	CompressionRows      []JSONBenchCompressionReportRow `json:"compression_rows"`
	RawJSONBytes         int64                           `json:"raw_json_bytes"`
	DictionaryBytes      int                             `json:"dictionary_bytes"`
	RowsPerSecond        float64                         `json:"rows_per_second"`
	EncodedMiBPerSecond  float64                         `json:"encoded_mib_per_second"`
	StoredMiBPerSecond   float64                         `json:"stored_mib_per_second"`
	NanosPerRow          float64                         `json:"nanos_per_row"`
	AllocatedBytesPerOp  uint64                          `json:"allocated_bytes_per_op"`
	AllocsPerOp          uint64                          `json:"allocs_per_op"`
	AllocatedBytesPerRow float64                         `json:"allocated_bytes_per_row"`
	TemporaryBytes       uint64                          `json:"temporary_bytes_estimate"`
	TemporaryBytesPerRow float64                         `json:"temporary_bytes_per_row"`
}

func RunJSONBenchPartBuildReports 

func RunJSONBenchPartBuildReports(ds JSONBenchDataset, rowsPerGranule int, attempts int) ([]JSONBenchPartBuildReport, error)

type JSONBenchPartQueryAttempt 

type JSONBenchPartQueryAttempt struct {
	Cache        string                        `json:"cache"`
	Duration     time.Duration                 `json:"duration"`
	ResultRows   int                           `json:"result_rows"`
	ResultDigest uint64                        `json:"result_digest"`
	Diagnostics  JSONBenchPartQueryDiagnostics `json:"diagnostics"`
}

type JSONBenchPartQueryDiagnostics 

type JSONBenchPartQueryDiagnostics struct {
	RowsScanned                    int                       `json:"rows_scanned"`
	RowsReturned                   int                       `json:"rows_returned,omitempty"`
	RowsSuperseded                 int                       `json:"rows_superseded,omitempty"`
	RowsDeleted                    int                       `json:"rows_deleted,omitempty"`
	PartsConsidered                int                       `json:"parts_considered,omitempty"`
	BaseParts                      int                       `json:"base_parts,omitempty"`
	DeltaParts                     int                       `json:"delta_parts,omitempty"`
	Tombstones                     int                       `json:"tombstones,omitempty"`
	GranulesConsidered             int                       `json:"granules_considered"`
	GranulesSkipped                int                       `json:"granules_skipped"`
	GranulesDecoded                int                       `json:"granules_decoded"`
	BlocksDecoded                  int                       `json:"blocks_decoded"`
	BytesDecoded                   int                       `json:"bytes_decoded"`
	ColumnsProjected               []string                  `json:"columns_projected"`
	AggregateKernel                string                    `json:"aggregate_kernel"`
	CacheState                     string                    `json:"cache_state"`
	SortKey                        []string                  `json:"sort_key"`
	EarlyStopAvailable             bool                      `json:"early_stop_available"`
	AggregateMetadataUsed          bool                      `json:"aggregate_metadata_used"`
	AggregateMetadataName          string                    `json:"aggregate_metadata_name,omitempty"`
	AggregateMetadataRows          int                       `json:"aggregate_metadata_rows,omitempty"`
	AggregateMetadataEntries       int                       `json:"aggregate_metadata_entries,omitempty"`
	AggregateMetadataBytes         int                       `json:"aggregate_metadata_bytes,omitempty"`
	AggregateMetadataBuildDuration time.Duration             `json:"aggregate_metadata_build_duration,omitempty"`
	AggregateMetadataBytesPerRow   float64                   `json:"aggregate_metadata_bytes_per_row,omitempty"`
	AggregateMetadataCompression   string                    `json:"aggregate_metadata_compression,omitempty"`
	PartSetCacheStats              ColumnWorkspaceCacheStats `json:"part_set_cache_stats,omitempty"`
}

type JSONBenchPartQueryTiming 

type JSONBenchPartQueryTiming struct {
	Query        string                        `json:"query"`
	Description  string                        `json:"description"`
	Engine       string                        `json:"engine"`
	Attempts     []JSONBenchPartQueryAttempt   `json:"attempts"`
	Best         time.Duration                 `json:"best"`
	BestCache    string                        `json:"best_cache"`
	ResultRows   int                           `json:"result_rows"`
	ResultDigest uint64                        `json:"result_digest"`
	Diagnostics  JSONBenchPartQueryDiagnostics `json:"diagnostics"`
}

func RunJSONBenchColumnPartSetQueries 

func RunJSONBenchColumnPartSetQueries(reader *ColumnPartSetReader, ds JSONBenchDataset, attempts int) ([]JSONBenchPartQueryTiming, error)

func RunJSONBenchPartAggregateMetadataQueries 

func RunJSONBenchPartAggregateMetadataQueries(ds JSONBenchDataset, rowsPerGranule int, attempts int) ([]JSONBenchPartQueryTiming, error)

func RunJSONBenchPartQ4FairnessQueries 

func RunJSONBenchPartQ4FairnessQueries(ds JSONBenchDataset, rowsPerGranule int, attempts int) ([]JSONBenchPartQueryTiming, error)

func RunJSONBenchPartQueries 

func RunJSONBenchPartQueries(ds JSONBenchDataset, rowsPerGranule int, attempts int) ([]JSONBenchPartQueryTiming, error)

type JSONBenchQueryTiming 

type JSONBenchQueryTiming struct {
	Query        string          `json:"query"`
	Description  string          `json:"description"`
	Attempts     []time.Duration `json:"attempts"`
	Best         time.Duration   `json:"best"`
	ResultRows   int             `json:"result_rows"`
	ResultDigest uint64          `json:"result_digest"`
}

func RunJSONBenchQueries 

func RunJSONBenchQueries(ds JSONBenchDataset, attempts int) ([]JSONBenchQueryTiming, error)

type LogicalPrimaryKey 

type LogicalPrimaryKey struct {
	Columns []string
}

type MemoryColumnAssetStore 

type MemoryColumnAssetStore struct {
	// contains filtered or unexported fields
}

func NewMemoryColumnAssetStore 

func NewMemoryColumnAssetStore() *MemoryColumnAssetStore

func (*MemoryColumnAssetStore) Put 

func (s *MemoryColumnAssetStore) Put(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)

func (*MemoryColumnAssetStore) PutOwned 

func (s *MemoryColumnAssetStore) PutOwned(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)

func (*MemoryColumnAssetStore) Read 

func (s *MemoryColumnAssetStore) Read(ref ColumnAssetRef) ([]byte, error)

func (*MemoryColumnAssetStore) ReadRange 

func (s *MemoryColumnAssetStore) ReadRange(ref ColumnAssetRef, offset int64, length int) ([]byte, error)

func (*MemoryColumnAssetStore) ReadTo 

func (s *MemoryColumnAssetStore) ReadTo(ref ColumnAssetRef, dst []byte) ([]byte, error)

func (*MemoryColumnAssetStore) Reset 

func (s *MemoryColumnAssetStore) Reset()

func (*MemoryColumnAssetStore) Verify 

func (s *MemoryColumnAssetStore) Verify(ref ColumnAssetRef) error

type PartScanDiagnostics 

type PartScanDiagnostics struct {
	RowsScanned        int
	ColumnsProjected   int
	GranulesConsidered int
	GranulesDecoded    int
	BlocksDecoded      int
	BytesDecoded       int
}

type PayloadRef 

type PayloadRef struct {
	Kind   PayloadRefKind
	Offset int64
	Length int
}

type PayloadRefKind 

type PayloadRefKind uint8
const (
	PayloadRefInline PayloadRefKind = iota + 1
)

func (PayloadRefKind) String 

func (k PayloadRefKind) String() string

type PredicateDiagnostics 

type PredicateDiagnostics struct {
	Considered      int
	SkippedByMark   int
	SkippedByMinMax int
	Decoded         int
	Matched         int
}

type PredicatePlan 

type PredicatePlan struct {
	Filter        Int64RangePredicate
	SortKeyRanges []Int64RangePredicate
}

type ProjectedScanResult 

type ProjectedScanResult struct {
	Rows        int
	Columns     map[string][]int64
	Diagnostics PartScanDiagnostics
}

type RowLocator 

type RowLocator struct {
	PrimaryID      int64
	PartID         uint64
	PartRow        int
	GranuleOrdinal int
	RowInGranule   int
}

type SegmentColumnAssetStore 

type SegmentColumnAssetStore struct {
	// contains filtered or unexported fields
}

func OpenSegmentColumnAssetStore 

func OpenSegmentColumnAssetStore(dir string) (*SegmentColumnAssetStore, error)

func (*SegmentColumnAssetStore) Close 

func (s *SegmentColumnAssetStore) Close() error

func (*SegmentColumnAssetStore) Path 

func (s *SegmentColumnAssetStore) Path() string

func (*SegmentColumnAssetStore) Put 

func (s *SegmentColumnAssetStore) Put(kind ColumnAssetKind, payload []byte) (ColumnAssetRef, error)

func (*SegmentColumnAssetStore) Read 

func (s *SegmentColumnAssetStore) Read(ref ColumnAssetRef) ([]byte, error)

func (*SegmentColumnAssetStore) ReadRange 

func (s *SegmentColumnAssetStore) ReadRange(ref ColumnAssetRef, offset int64, length int) ([]byte, error)

func (*SegmentColumnAssetStore) ReadTo 

func (s *SegmentColumnAssetStore) ReadTo(ref ColumnAssetRef, dst []byte) ([]byte, error)

func (*SegmentColumnAssetStore) Sync 

func (s *SegmentColumnAssetStore) Sync() error

func (*SegmentColumnAssetStore) Verify 

func (s *SegmentColumnAssetStore) Verify(ref ColumnAssetRef) (err error)

type SortKey 

type SortKey struct {
	Columns []SortKeyColumn
}

type SortKeyBound 

type SortKeyBound struct {
	Values    []int64
	Exclusive bool
	Unbounded bool
}

type SortKeyColumn 

type SortKeyColumn struct {
	Column    string
	Direction SortKeyDirection
	Nulls     SortKeyNullOrder
}

type SortKeyColumnValues 

type SortKeyColumnValues struct {
	Name   string
	Values []int64
}

type SortKeyDirection 

type SortKeyDirection string
const (
	SortKeyAsc  SortKeyDirection = "asc"
	SortKeyDesc SortKeyDirection = "desc"
)

type SortKeyMark 

type SortKeyMark struct {
	Rows         int
	Columns      []string
	ColumnValues [][]int64
	Prefixes     []SortKeyPrefixSummary
}

func BuildSortKeyMark 

func BuildSortKeyMark(columns []SortKeyColumnValues) (SortKeyMark, error)

func (SortKeyMark) MayContainRanges 

func (m SortKeyMark) MayContainRanges(ranges []Int64RangePredicate) (bool, bool, error)

type SortKeyNullOrder 

type SortKeyNullOrder string
const (
	SortKeyNullsDefault SortKeyNullOrder = ""
	SortKeyNullsFirst   SortKeyNullOrder = "first"
	SortKeyNullsLast    SortKeyNullOrder = "last"
)

type SortKeyPrefixSummary 

type SortKeyPrefixSummary struct {
	Columns        []string
	Lower          SortKeyBound
	UpperExclusive SortKeyBound
}

type TCS1PartRecord 

type TCS1PartRecord struct {
	Version      uint16         `json:"version"`
	Kind         uint16         `json:"kind"`
	Flags        uint32         `json:"flags"`
	PartID       uint64         `json:"part_id"`
	Rows         int            `json:"rows"`
	ImageVersion uint16         `json:"image_version"`
	PayloadBytes int            `json:"payload_bytes"`
	TotalBytes   int            `json:"total_bytes"`
	PayloadCRC32 uint32         `json:"payload_crc32"`
	AssetRef     ColumnAssetRef `json:"asset_ref,omitempty"`
}

func DecodeTCS1ColumnPartHeader 

func DecodeTCS1ColumnPartHeader(header []byte, totalBytes int64) (TCS1PartRecord, error)

func EncodeTCS1ColumnPartImage 

func EncodeTCS1ColumnPartImage(image ColumnPartImage) ([]byte, TCS1PartRecord, error)

func LoadTCS1ColumnPartHeader 

func LoadTCS1ColumnPartHeader(store ColumnAssetStore, ref ColumnAssetRef) (TCS1PartRecord, error)

type TimeBucketedCounts 

type TimeBucketedCounts struct {
	BucketWidth int64
	MinBucket   int64
	Buckets     int
	Cardinality uint32
	Counts      []uint64
}

func (TimeBucketedCounts) Count 

func (c TimeBucketedCounts) Count(bucket int64, code uint32) uint64

Source Files

View all Source files

Directories

Path Synopsis
cmd
jsonbench_colgranule command
jsonbench_compare command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.