Help us shape the future of Go! Take the 2026 Go Developer Survey

source

package
v0.1.0-rc.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 26, 2026 License: MIT Imports: 43 Imported by: 0

Documentation

Overview

Package source decodes replayable objects into bounded, owned Arrow batches.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ParseTimestamp 

func ParseTimestamp(s string, c config.Column) (time.Time, error)

ParseTimestamp parses an instant without consulting the machine's local timezone.

Types

type Dataset 

type Dataset struct {
	// contains filtered or unexported fields
}

func New 

func New(ctx context.Context, cfg *config.Config, store *storage.Store, budget *resource.Budget, opts ...Option) (*Dataset, error)

New discovers the configured objects and freezes the dataset schema, either from input.schema or by inspecting headers and sampling records.

func NewGenerated 

func NewGenerated(schema *arrow.Schema, open func(context.Context) (Reader, error)) *Dataset

func (*Dataset) ApplyStatisticsPlan 

func (d *Dataset) ApplyStatisticsPlan(ctx context.Context, plan *data.StatisticsPlan) error

ApplyStatisticsPlan retains an immutable, complete, source-ordered plan. A prepass has already required ordinary values and cannot later be short-cut. Baselines for fully scanned inputs are established first; the plan is only committed once they succeed, so a failed call leaves no plan applied.

func (*Dataset) Close 

func (d *Dataset) Close() error

func (*Dataset) DecoderPassesPerOpen 

func (d *Dataset) DecoderPassesPerOpen() int

DecoderPassesPerOpen counts complete scans surrounding Parquet decoding. Page verification is fused with decoding; only local replay digests require an additional complete scan.

func (*Dataset) InferenceRowsScanned 

func (d *Dataset) InferenceRowsScanned() uint64

InferenceRowsScanned counts successful decoded row visits during schema inference/discovery, including the lexical drain after the sample limit. Raw identity/hash passes and later readers are not included.

func (*Dataset) Objects 

func (d *Dataset) Objects() []storage.Object

func (*Dataset) Open 

func (d *Dataset) Open(ctx context.Context) (Reader, error)

func (*Dataset) OpenPreflight 

func (d *Dataset) OpenPreflight(ctx context.Context) (Reader, error)

OpenPreflight preserves conversion states for whole-dataset castability gates.

func (*Dataset) ParquetGroups 

func (d *Dataset) ParquetGroups() ([]data.ParquetGroup, error)

ParquetGroups borrows guarded facts whose lifetime is the dataset. It never reads payload and does not confer trust; only a complete checks plan can skip.

func (*Dataset) Passes 

func (d *Dataset) Passes() int

Passes reports source passes spent before the main read (schema inference).

func (*Dataset) Schema 

func (d *Dataset) Schema() *arrow.Schema

func (*Dataset) SchemaIssues 

func (d *Dataset) SchemaIssues() []data.Issue

func (*Dataset) VerifyStatisticsInputs 

func (d *Dataset) VerifyStatisticsInputs(ctx context.Context) error

VerifyStatisticsInputs is mandatory before committing selective results. It does not imply unread payload content, encodings, or row sizes were verified.

type Option 

type Option func(*options)

Option configures New.

func WithLogger 

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger for discovery, inference and Parquet safety scan events. Row values are never logged.

func WithPlan 

func WithPlan() Option

WithPlan limits discovery to metadata/sample work that cannot publish a result. In particular, Parquet range handles skip close-time content digests.

type Reader 

type Reader interface {
	Next(context.Context) (*data.Batch, error)
	Close() error
}

type SchemaAttributes 

type SchemaAttributes struct {
	Type         string  `json:"type"`
	ArrowType    string  `json:"arrow_type,omitempty"`
	PhysicalType string  `json:"physical_type,omitempty"`
	Nullable     *bool   `json:"nullable"`
	Required     *bool   `json:"required"`
	BitWidth     *int    `json:"bit_width"`
	Signed       *bool   `json:"signed"`
	Precision    *int32  `json:"precision"`
	Scale        *int32  `json:"scale"`
	Unit         *string `json:"unit"`
	Timezone     *string `json:"timezone"`
}

SchemaAttributes records the logical domain without inventing absent constraints. A nil attribute means the expectation does not constrain it.

type SchemaCheck 

type SchemaCheck struct {
	Code   string `json:"code"`
	Type   string `json:"type"`
	Source string `json:"source,omitempty"`
	Status string `json:"status"`
	Reason string `json:"reason,omitempty"`
}

SchemaCheck explicitly separates metadata outcomes from unevaluated checks.

type SchemaDifference 

type SchemaDifference struct {
	SourceOrdinal     uint64            `json:"source_ordinal"`
	Source            string            `json:"source"`
	Column            string            `json:"column"`
	SourceName        string            `json:"source_name,omitempty"`
	Kind              string            `json:"kind"`
	Expected          *SchemaAttributes `json:"expected"`
	Actual            *SchemaAttributes `json:"actual"`
	Severity          string            `json:"severity"`
	RuleCode          string            `json:"rule_code,omitempty"`
	ExpectationSource string            `json:"expectation_source"`
	Reason            string            `json:"reason"`
	RuleOrdinal       int               `json:"rule_ordinal,omitempty"`
	CheckOrdinal      int               `json:"check_ordinal,omitempty"`
}

SchemaDifference is a policy-relevant mismatch, not a value-validation outcome. Expected/Actual are null when no typed schema exists on that side.

type SchemaField 

type SchemaField struct {
	Name         string            `json:"name"`
	SourceName   string            `json:"source_name,omitempty"`
	Type         string            `json:"type"`
	ArrowType    string            `json:"arrow_type"`
	PhysicalType string            `json:"physical_type,omitempty"`
	Nullable     bool              `json:"nullable"`
	Required     bool              `json:"required,omitempty"`
	Attributes   *SchemaAttributes `json:"attributes"`
}

SchemaField preserves the actual source name separately from its renamed name.

type SchemaObject 

type SchemaObject struct {
	SourceOrdinal uint64        `json:"source_ordinal"`
	URI           string        `json:"uri"`
	DeclaredRows  int64         `json:"declared_rows"`
	Fields        []SchemaField `json:"fields"`
}

SchemaObject is the metadata observed for one pinned input object.

type SchemaReport 

type SchemaReport struct {
	ReportVersion         int                `json:"report_version"`
	RunID                 string             `json:"run_id"`
	Scope                 string             `json:"scope"`
	MetadataOnly          bool               `json:"metadata_only"`
	Validation            string             `json:"validation"`
	Compatible            bool               `json:"compatible"`
	RowsChecked           int64              `json:"rows_checked"`
	ObjectsChecked        int                `json:"objects_checked"`
	DeclaredRows          int64              `json:"declared_rows"`
	Objects               []SchemaObject     `json:"objects"`
	DeclaredSchema        []SchemaField      `json:"declared_schema"`
	SchemaIssues          []data.Issue       `json:"schema_issues"`
	Checks                []SchemaCheck      `json:"checks"`
	NotEvaluated          []SchemaCheck      `json:"not_evaluated"`
	Differences           []SchemaDifference `json:"differences"`
	BaselineSource        string             `json:"baseline_source,omitempty"`
	BaselineSourceOrdinal uint64             `json:"baseline_source_ordinal,omitempty"`
	// contains filtered or unexported fields
}

SchemaReport describes only footer/schema compatibility, never row validation. DeclaredRows is the footer claim, not an observed or validated row count.

func InspectSchema 

func InspectSchema(ctx context.Context, cfg *config.Config, store *storage.Store, budget *resource.Budget) (report *SchemaReport, err error)

InspectSchema reads Parquet footers only. It neither opens a row reader nor compiles checks, loads rule references, validates pages or hashes input data. cfg must already be validated; store and budget remain owned by the caller. The caller must Release the returned report after presentation.

func (*SchemaReport) Release 

func (r *SchemaReport) Release()

Release returns the report's managed reservations. It is idempotent; the report must not be used afterward. No source handles or dataset state remain attached to a completed report.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.