Documentation
¶
Overview ¶
Package types provides core type definitions and data structures for StreamSQL.
This package defines fundamental data types, configuration structures, and interfaces used throughout the StreamSQL stream processing pipeline. It ensures type safety and provides a unified API for data manipulation across components.
Core Features ¶
• Data Types - Core data structures for stream processing • Configuration Management - Centralized configuration structures • Type Safety - Strong typing with validation • Serialization Support - JSON serialization support • Cross-Component Compatibility - Shared types across packages
Configuration Structures ¶
Core configuration types:
type Config struct {
WindowConfig WindowConfig // Window settings
GroupFields []string // GROUP BY fields
SelectFields map[string]aggregator.AggregateType // SELECT aggregations
FieldAlias map[string]string // Field aliases
SimpleFields []string // Non-aggregated fields
FieldExpressions map[string]FieldExpression // Computed expressions
Where string // WHERE clause
Having string // HAVING clause
NeedWindow bool // Window requirement
Distinct bool // DISTINCT flag
Limit int // LIMIT clause
PerformanceConfig PerformanceConfig // Performance settings
}
Window Configuration ¶
Unified configuration for all window types:
type WindowConfig struct {
Type string // Window type
Params map[string]any // Parameters
TsProp string // Timestamp property
TimeUnit time.Duration // Time unit
GroupByKey string // Grouping key
}
// Example configurations
// Tumbling window
windowConfig := WindowConfig{
Type: "tumbling",
Params: map[string]any{
"size": "5s",
},
TsProp: "timestamp",
}
// Sliding window
windowConfig := WindowConfig{
Type: "sliding",
Params: map[string]any{
"size": "30s",
"slide": "10s",
},
TsProp: "timestamp",
}
// Counting window
windowConfig := WindowConfig{
Type: "counting",
Params: map[string]any{
"count": 100,
},
}
// Session window
windowConfig := WindowConfig{
Type: "session",
Params: map[string]any{
"timeout": "5m",
},
GroupByKey: "user_id",
}
Performance Configuration ¶
Comprehensive performance tuning options:
type PerformanceConfig struct {
// Buffer management
BufferSize int // Input buffer size
BatchSize int // Processing batch size
FlushInterval time.Duration // Automatic flush interval
HighWaterMark float64 // Buffer high water mark (0.0-1.0)
LowWaterMark float64 // Buffer low water mark (0.0-1.0)
// Worker pool configuration
WorkerPoolSize int // Number of worker goroutines
MaxWorkers int // Maximum worker limit
WorkerIdleTime time.Duration // Worker idle timeout
// Overflow handling
OverflowStrategy string // "drop", "block", "spill", "compress"
SpillDirectory string // Directory for spill files
CompressionLevel int // Compression level (1-9)
// Memory management
MaxMemoryUsage int64 // Maximum memory usage in bytes
GCInterval time.Duration // Garbage collection interval
MemoryThreshold float64 // Memory usage threshold
// Monitoring
MetricsEnabled bool // Enable metrics collection
MetricsInterval time.Duration // Metrics collection interval
HealthCheckPort int // Health check HTTP port
// Persistence
PersistenceEnabled bool // Enable data persistence
PersistenceType string // "memory", "file", "database"
PersistencePath string // Persistence storage path
RecoveryEnabled bool // Enable automatic recovery
}
Field Management ¶
Advanced field handling and expression support:
type FieldExpression struct {
Field string // Field name
Expression string // Expression
Fields []string // Referenced fields
}
type Projection struct {
SourceType ProjectionSourceType // Source type (field, expression, aggregate)
Source string // Source identifier
Alias string // Output alias
DataType string // Expected data type
}
type ProjectionSourceType string
const (
ProjectionSourceField ProjectionSourceType = "field" // Direct field reference
ProjectionSourceExpression ProjectionSourceType = "expression" // Computed expression
ProjectionSourceAggregate ProjectionSourceType = "aggregate" // Aggregate function
ProjectionSourceConstant ProjectionSourceType = "constant" // Constant value
)
Data Row Representation ¶
Type-safe data row structures for stream processing:
type Row struct {
Data map[string]any // Row data
Timestamp time.Time // Row timestamp
Metadata map[string]any // Additional metadata
GroupKey string // Grouping key for aggregation
WindowID string // Window identifier
}
// Row creation and manipulation
func NewRow(data map[string]any) *Row
func (r *Row) GetValue(field string) any
func (r *Row) SetValue(field string, value any)
func (r *Row) HasField(field string) bool
func (r *Row) Clone() *Row
Time Management ¶
Time-based data structures for window processing:
type TimeSlot struct {
Start time.Time // Slot start time
End time.Time // Slot end time
Duration time.Duration // Slot duration
ID string // Unique slot identifier
}
// Time slot operations
func NewTimeSlot(start time.Time, duration time.Duration) *TimeSlot
func (ts *TimeSlot) Contains(timestamp time.Time) bool
func (ts *TimeSlot) Overlaps(other *TimeSlot) bool
func (ts *TimeSlot) String() string
Configuration Presets ¶
Pre-defined configuration templates for common use cases:
// High Performance Configuration
func NewHighPerformanceConfig() *PerformanceConfig {
return &PerformanceConfig{
BufferSize: 50000,
BatchSize: 1000,
WorkerPoolSize: 8,
FlushInterval: 100 * time.Millisecond,
OverflowStrategy: "spill",
MetricsEnabled: true,
}
}
// Low Latency Configuration
func NewLowLatencyConfig() *PerformanceConfig {
return &PerformanceConfig{
BufferSize: 1000,
BatchSize: 10,
WorkerPoolSize: 2,
FlushInterval: 10 * time.Millisecond,
OverflowStrategy: "drop",
MetricsEnabled: false,
}
}
// Zero Data Loss Configuration
func NewZeroDataLossConfig() *PerformanceConfig {
return &PerformanceConfig{
BufferSize: 10000,
BatchSize: 100,
WorkerPoolSize: 4,
FlushInterval: time.Second,
OverflowStrategy: "block",
PersistenceEnabled: true,
RecoveryEnabled: true,
MetricsEnabled: true,
}
}
Usage Examples ¶
Basic configuration:
config := &Config{
WindowConfig: WindowConfig{
Type: "tumbling",
Params: map[string]any{"size": "5s"},
},
GroupFields: []string{"device_id"},
SelectFields: map[string]aggregator.AggregateType{
"temperature": aggregator.AggregateTypeAvg,
},
NeedWindow: true,
}
Data row operations:
row := NewRow(map[string]any{
"device_id": "sensor001",
"temperature": 25.5,
})
deviceID := row.GetValue("device_id").(string)
row.SetValue("processed", true)
Integration ¶
Integrates with other StreamSQL components:
• Stream Package - Core data types for stream processing • Window Package - WindowConfig for window configurations • Aggregator Package - AggregateType definitions • Condition Package - Data structures for clause evaluation • Functions Package - Type definitions for functions • RSQL Package - Config structures for query execution
Index ¶
- Constants
- func AnalyticSelfTokenN(i int) string
- type AggregationFieldInfo
- type AnalyticCall
- type AnalyticField
- type BufferConfig
- type Config
- type ExpansionConfig
- type FieldExpression
- type JoinConfig
- type JoinOnPair
- type MonitoringConfig
- type OrderByField
- type OverSpec
- type OverflowConfig
- type PerformanceConfig
- type PostAggregationExpression
- type Projection
- type ProjectionSourceType
- type Row
- type RowEvent
- type SortDirection
- type TimeCharacteristic
- type TimeSlot
- type WarningThresholds
- type WhereAnalyticCall
- type WindowConfig
- type WorkerConfig
Constants ¶
const ( // OverflowStrategyBlock blocks when buffer is full OverflowStrategyBlock = "block" // OverflowStrategyDrop drops data when buffer is full OverflowStrategyDrop = "drop" // OverflowStrategyExpand dynamically expands buffer capacity when full, // up to BufferConfig.MaxBufferSize; above the ceiling it drops. Effective // for the data channel (see stream.expandDataChannel). OverflowStrategyExpand = "expand" )
const AnalyticSelfToken = "__analytic_self__"
AnalyticSelfToken 是"表达式包分析函数"回代模板里的占位(如 ts - __analytic_self__): 求值期把分析函数结果写入行的该键,再求整个外层表达式。
Variables ¶
This section is empty.
Functions ¶
func AnalyticSelfTokenN ¶ added in v1.0.3
AnalyticSelfTokenN 返回表达式内第 i 个分析调用在回代模板里的占位标识符。 i==0 沿用 AnalyticSelfToken(单调用向后兼容),i>0 用 __analytic_self_<i>__。 同一表达式含多个分析调用时(如 acc_max(v) - acc_min(v)),每个调用各分配一个占位。
Types ¶
type AggregationFieldInfo ¶ added in v0.10.3
type AggregationFieldInfo struct {
FuncName string `json:"funcName"` // 函数名,如 "first_value"
InputField string `json:"inputField"` // 输入字段,如 "displayNum"
Placeholder string `json:"placeholder"` // 占位符,如 "__first_value_0__"
AggType aggregator.AggregateType `json:"aggType"` // 聚合类型
FullCall string `json:"fullCall"` // 完整函数调用,如 "NTH_VALUE(value, 2)"
}
AggregationFieldInfo holds information about an aggregation function in an expression
type AnalyticCall ¶ added in v1.0.3
type AnalyticCall struct {
FuncName string // 函数名,如 "acc_max"
BareCall string // 完整调用文本(不含 OVER),如 "acc_max(v)"
Args []string // 原始参数表达式片段(未求值),如 ["v"]
}
AnalyticCall 描述"表达式包分析函数"里的单个分析调用(一个字段可含多个,如 acc_max(v) - acc_min(v))。
type AnalyticField ¶ added in v1.0.3
type AnalyticField struct {
FuncName string // 函数名,如 "lag"
Args []string // 原始参数表达式片段(未求值),如 ["temp","1"]
Expression string // 完整调用文本(不含 OVER),如 "lag(temp, 1)"
Alias string // 输出列名(多列函数仅作内部句柄,实际列名由结果决定)
Over *OverSpec // OVER 子句,nil 表示无
// MultiColumn 标记多列动态输出函数(changed_cols):其求值结果为 map[colname]value,
// 投影时按 prefix+colname 扇出为多个输出列。仅 SELECT。
MultiColumn bool
// WrapperExpr 外层算术/表达式回代模板:当字段是"表达式包分析函数"(如 ts - lag(ts))
// 时,分析调用子串被替换为占位 __analytic_self__,得 "ts - __analytic_self__"。
// 求值期先算出分析函数值,代入占位再求整个表达式。空表示纯分析字段。
WrapperExpr string
// Calls 表达式内全部 Analytic 调用(按出现顺序)。纯单调用字段长度为 1;
// 多调用(acc_max(v) - acc_min(v))长度 ≥ 2,每个调用在 WrapperExpr 里对应
// types.AnalyticSelfTokenN(i) 占位。FuncName/Args/Expression 仍保留首个调用值供旧路径。
Calls []AnalyticCall
// InlineAggDisplay 窗口查询里分析函数参数内联聚合的重写映射:隐藏键→显示名。
// 如 changed_cols("t",true,avg(temperature)) 在窗口查询里,avg(temperature) 被提取为
// 隐藏计算字段 __winagg_0__,这里记录 {"__winagg_0__":"avg"},输出列名取显示名(→ tavg)。
InlineAggDisplay map[string]string
}
AnalyticField 描述 SELECT 中的分析函数字段(带可选 OVER)。 走直连路径(EmitSync),由流级状态机逐条求值,不进聚合路径。
type BufferConfig ¶
type BufferConfig struct {
DataChannelSize int `json:"dataChannelSize"` // Data input buffer size
ResultChannelSize int `json:"resultChannelSize"` // Result output buffer size
WindowOutputSize int `json:"windowOutputSize"` // Window output buffer size
EnableDynamicResize bool `json:"enableDynamicResize"` // Enable dynamic buffer resizing
MaxBufferSize int `json:"maxBufferSize"` // Maximum buffer size
UsageThreshold float64 `json:"usageThreshold"` // Buffer usage threshold
}
BufferConfig buffer configuration
type Config ¶
type Config struct {
// SQL processing related configuration
WindowConfig WindowConfig `json:"windowConfig"`
GroupFields []string `json:"groupFields"`
SelectFields map[string]aggregator.AggregateType `json:"selectFields"`
FieldAlias map[string]string `json:"fieldAlias"`
// SelectAlias maps a SELECT item's raw expression to its AS alias (e.g.
// "m.location" -> "loc"). The aggregation path uses it to name output
// columns for grouped non-aggregate columns, matching the direct path
// (where the alias is applied during SimpleField compilation).
SelectAlias map[string]string `json:"selectAlias"`
SimpleFields []string `json:"simpleFields"`
FieldExpressions map[string]FieldExpression `json:"fieldExpressions"`
PostAggExpressions []PostAggregationExpression `json:"postAggExpressions"` // Post-aggregation expressions
FieldOrder []string `json:"fieldOrder"` // Original order of fields in SELECT statement
Where string `json:"where"`
Having string `json:"having"`
// Feature switches
NeedWindow bool `json:"needWindow"`
Distinct bool `json:"distinct"`
// Result control
Limit int `json:"limit"`
Projections []Projection `json:"projections"`
OrderBy []OrderByField `json:"orderBy"` // ORDER BY sort keys, applied per emit batch
// JoinConfigs describes stream-table JOINs (v0.5: metadata enrichment).
// Empty means no JOIN. Each entry references a registered table source by
// name and is resolved at row-processing time.
JoinConfigs []JoinConfig `json:"joinConfigs"`
// SourceAlias is the optional FROM alias (e.g. "s" in "FROM stream AS s").
// When set, stream fields can be qualified as "s.<field>" in SELECT/WHERE.
SourceAlias string `json:"sourceAlias"`
// AnalyticFields 分析函数字段(带可选 OVER)。走直连路径,由
// 流级状态机逐条求值,不进聚合路径。空表示无分析函数。
AnalyticFields []AnalyticField `json:"analyticFields"`
// WhereAnalyticCalls WHERE 中出现的分析函数调用;解析期从 WHERE 文本提取并
// 替换为占位符,求值期在 WHERE 之前算出值注入 dataMap[Placeholder]。
WhereAnalyticCalls []WhereAnalyticCall `json:"whereAnalyticCalls"`
// AnalyticMaxPartitions 每个分析函数字段 PARTITION 状态的分区数上限,超出按
// LRU 淘汰最久未用的分区。≤0 表示用默认值(见 stream.defaultMaxPartitions)。
// 由 WithAnalyticMaxPartitions 注入。高基数分区键(如设备上万)且内存充裕时可调高。
AnalyticMaxPartitions int `json:"analyticMaxPartitions"`
// Logger is the per-instance logger for the stream pipeline. Injected by
// Streamsql.Execute (from WithLogger, else the process default); nil falls
// back to logger.GetDefault() at construction. Immutable after construction.
Logger logger.Logger `json:"-"`
// Performance configuration
PerformanceConfig PerformanceConfig `json:"performanceConfig"`
}
Config stream processing configuration
func NewConfigWithPerformance ¶
func NewConfigWithPerformance(perfConfig PerformanceConfig) Config
NewConfigWithPerformance creates Config with performance configuration
type ExpansionConfig ¶
type ExpansionConfig struct {
GrowthFactor float64 `json:"growthFactor"` // Growth factor
MinIncrement int `json:"minIncrement"` // Minimum expansion increment
TriggerThreshold float64 `json:"triggerThreshold"` // Expansion trigger threshold
ExpansionTimeout time.Duration `json:"expansionTimeout"` // Expansion timeout duration
}
ExpansionConfig expansion configuration
type FieldExpression ¶
type FieldExpression struct {
Field string `json:"field"` // original field name
Expression string `json:"expression"` // complete expression
Fields []string `json:"fields"` // all fields referenced in expression
}
FieldExpression field expression configuration
type JoinConfig ¶ added in v1.0.0
type JoinConfig struct {
Table string // registered table source name
Alias string // table alias; matched columns are namespaced under it. Defaults to Table.
JoinType string // "INNER" (default) or "LEFT"
OnPairs []JoinOnPair // equality predicates linking stream and table fields
}
JoinConfig describes a single stream-table JOIN.
type JoinOnPair ¶ added in v1.0.0
JoinOnPair is one equality of a JOIN ON clause. StreamField is resolved against the incoming stream row, TableField against the matched table row.
type MonitoringConfig ¶
type MonitoringConfig struct {
EnableMonitoring bool `json:"enableMonitoring"` // Enable performance monitoring
StatsUpdateInterval time.Duration `json:"statsUpdateInterval"` // Statistics update interval
EnableDetailedStats bool `json:"enableDetailedStats"` // Enable detailed statistics
WarningThresholds WarningThresholds `json:"warningThresholds"` // Performance warning thresholds
}
MonitoringConfig monitoring configuration
type OrderByField ¶ added in v1.0.0
type OrderByField struct {
Expression string `json:"expression"` // result column name (v0.5: must match an output field)
Direction SortDirection `json:"direction"` // SortAsc (default) or SortDesc
}
OrderByField represents a single ORDER BY sort key.
type OverSpec ¶ added in v1.0.3
type OverSpec struct {
PartitionBy []string // 分区字段,状态按分区独立维护
When string // WHEN 条件表达式;满足才更新状态,否则复用旧值
}
OverSpec 描述分析函数的 OVER 子句。 仅支持 PARTITION BY 和 WHEN,不支持 ORDER BY / ROWS frame(那是 Flink 模型)。
type OverflowConfig ¶
type OverflowConfig struct {
Strategy string `json:"strategy"` // Overflow strategy: "drop", "block", "expand"
BlockTimeout time.Duration `json:"blockTimeout"` // Block timeout duration
AllowDataLoss bool `json:"allowDataLoss"` // Allow data loss
ExpansionConfig ExpansionConfig `json:"expansionConfig"` // Expansion configuration
}
OverflowConfig overflow strategy configuration
type PerformanceConfig ¶
type PerformanceConfig struct {
BufferConfig BufferConfig `json:"bufferConfig"` // buffer configuration
OverflowConfig OverflowConfig `json:"overflowConfig"` // overflow strategy configuration
WorkerConfig WorkerConfig `json:"workerConfig"` // worker pool configuration
MonitoringConfig MonitoringConfig `json:"monitoringConfig"` // monitoring configuration
}
PerformanceConfig performance configuration
func DefaultPerformanceConfig ¶
func DefaultPerformanceConfig() PerformanceConfig
DefaultPerformanceConfig returns default performance configuration Provides balanced performance settings suitable for most scenarios
func HighPerformanceConfig ¶
func HighPerformanceConfig() PerformanceConfig
HighPerformanceConfig returns high performance configuration preset Optimizes throughput performance with large buffers and expansion strategy
func LowLatencyConfig ¶
func LowLatencyConfig() PerformanceConfig
LowLatencyConfig returns low latency configuration preset Optimizes latency performance with smaller buffers and fast response strategy
type PostAggregationExpression ¶ added in v0.10.3
type PostAggregationExpression struct {
OutputField string `json:"outputField"` // 输出字段名
OriginalExpr string `json:"originalExpr"` // 原始表达式
ExpressionTemplate string `json:"expressionTemplate"` // 表达式模板
RequiredFields []AggregationFieldInfo `json:"requiredFields"` // 依赖的聚合字段
}
PostAggregationExpression represents an expression that needs to be evaluated after aggregation
type Projection ¶
type Projection struct {
OutputName string `json:"outputName"` // output field name
SourceType ProjectionSourceType `json:"sourceType"` // data source type
InputName string `json:"inputName"` // input field name
}
Projection projection configuration in SELECT list
type ProjectionSourceType ¶
type ProjectionSourceType int
ProjectionSourceType projection source type
const ( SourceGroupKey ProjectionSourceType = iota SourceAggregateResult SourceWindowProperty // For window_start, window_end )
type SortDirection ¶ added in v1.0.0
type SortDirection string
SortDirection selects ascending or descending order.
const ( SortAsc SortDirection = "ASC" SortDesc SortDirection = "DESC" )
type TimeCharacteristic ¶ added in v0.10.4
type TimeCharacteristic string
TimeCharacteristic represents the time characteristic for window operations
const ( // ProcessingTime uses system clock for window operations // Windows trigger based on when data arrives, not when events occurred ProcessingTime TimeCharacteristic = "ProcessingTime" // EventTime uses event timestamps for window operations // Windows trigger based on event time, requires watermark mechanism EventTime TimeCharacteristic = "EventTime" )
type TimeSlot ¶
func NewTimeSlot ¶
func (*TimeSlot) GetEndTime ¶
func (*TimeSlot) GetStartTime ¶
func (*TimeSlot) WindowStart ¶
type WarningThresholds ¶
type WarningThresholds struct {
DropRateWarning float64 `json:"dropRateWarning"` // Drop rate warning threshold
DropRateCritical float64 `json:"dropRateCritical"` // Drop rate critical threshold
BufferUsageWarning float64 `json:"bufferUsageWarning"` // Buffer usage warning threshold
BufferUsageCritical float64 `json:"bufferUsageCritical"` // Buffer usage critical threshold
}
WarningThresholds performance warning thresholds
type WhereAnalyticCall ¶ added in v1.0.3
type WhereAnalyticCall struct {
Placeholder string // 合成键,如 "__analytic_0__"
FuncName string // 函数名
Args []string // 原始参数片段
Expression string // 完整调用文本(不含 OVER)
Over *OverSpec // OVER 子句,nil 表示无
}
WhereAnalyticCall 描述 WHERE 条件中出现的分析函数调用。 解析期从 WHERE 文本提取,分配占位符;求值期在 WHERE 之前算出值, 注入 dataMap[Placeholder],WHERE 文本中的原始调用已被替换为占位符。
type WindowConfig ¶
type WindowConfig struct {
Type string `json:"type"`
Params []any `json:"params"` // Window function parameters array
TsProp string `json:"tsProp"`
TimeUnit time.Duration `json:"timeUnit"`
TimeCharacteristic TimeCharacteristic `json:"timeCharacteristic"` // Time characteristic: EventTime or ProcessingTime (default: ProcessingTime)
MaxOutOfOrderness time.Duration `json:"maxOutOfOrderness"` // Maximum allowed out-of-orderness for event time (default: 0)
WatermarkInterval time.Duration `json:"watermarkInterval"` // Watermark update interval for event time (default: 200ms)
AllowedLateness time.Duration `json:"allowedLateness"` // Maximum allowed lateness for event time windows (default: 0, meaning no late data accepted after window closes)
IdleTimeout time.Duration `json:"idleTimeout"` // Idle source timeout: when no data arrives within this duration, the watermark advances to (now - maxOutOfOrderness) so idle event-time windows can close. Default 0 disables it. Trade-off: a finite IdleTimeout (e.g. 60s) reaps idle state and closes windows promptly, but events arriving after an idle gap with an event-time behind the advanced watermark are dropped as late; keep IdleTimeout=0 if stale events on resume must not be lost (then idle event-time windows stay open until new data arrives).
CountStateTTL time.Duration `json:"countStateTtl"` // Counting-window keyed state TTL: keys inactive longer than this are reaped (lazy, in the Start goroutine). Default 0 = disabled. Set via SQL STATETTL='24h'.
GroupByKeys []string `json:"groupByKeys"` // Multiple grouping keys for keyed windows
PerformanceConfig PerformanceConfig `json:"performanceConfig"` // Performance configuration
Callback func([]Row) `json:"-"` // Callback function (not serialized)
// Global-window: TriggerCondition is the TRIGGER WHEN predicate string
// (e.g. "COUNT(*) >= 1000"). SelectFields/FieldAlias mirror the SELECT
// aggregation map so the global window maintains running aggregate state per
// group without buffering raw rows. Populated from the parsed SELECT for
// windowType=global only.
TriggerCondition string `json:"triggerCondition,omitempty"`
SelectFields map[string]aggregator.AggregateType `json:"selectFields,omitempty"`
FieldAlias map[string]string `json:"fieldAlias,omitempty"`
}
WindowConfig window configuration
type WorkerConfig ¶
type WorkerConfig struct {
SinkPoolSize int `json:"sinkPoolSize"` // Sink pool size
SinkWorkerCount int `json:"sinkWorkerCount"` // Sink worker count
MaxRetryRoutines int `json:"maxRetryRoutines"` // Maximum retry routines
}
WorkerConfig worker pool configuration