Documentation
¶
Overview ¶
Package clickhouse stores and ranks precomputed vectors with clickhouse-go v2 over native TCP. The host supplies and owns the database/sql pool. The physical table is Space.StorageKey() + "_" + TableName in the bound database. Even equal-width spaces with the same logical TableName use different tables and cannot replace each other's IDs. Reindex legacy unprefixed tables explicitly.
Construction probes native transactions and reads the real four-column table schema: unpartitioned ReplacingMergeTree ordered only by ID, three String columns and Array(Float32), vec_len and vec_finite. The native length constraint owns the width. Dimensions sets the creation width and, if given when opening, must agree with the table. The host must not replace the table while a Store is in use. Metadata is one exact Core JSON string, including null, precise numbers and escaped values. Media documents are unsupported.
Upsert preflights the complete batch before obtaining a driver insert buffer. Commit sends one synchronous native INSERT with implicit_transaction. A rejected input cannot publish earlier rows. Before send all IDs remain unattempted; a lost send acknowledgment leaves all of them unconfirmed. Native FINAL reads choose the current version.
Query runs one native ORDER BY distance, id LIMIT statement, using the selected cosine or L2 distance function, and applies the relevance threshold to validated rows. It does not create an approximate index. An opaque filter returns ErrUnsupportedFilter. Scan is an explicit alternative: its budget counts every source ID, text and raw metadata byte, including nonmatches. Each native page contains up to 512 rows; transport framing is separate. Scan charges records before decoding. Failure returns no partial rows.
DeleteMatching reads and validates the complete bounded snapshot before the first delete. Its ID chunks remain inside that same native transaction. DeleteIDs acts directly on explicit identity. Native COMMIT is the publication boundary; lost acknowledgment makes the selected IDs unconfirmed. Synchronous lightweight deletion hides rows upon success; physical removal follows merges. Cancellation releases native sessions using a separately bounded rollback.
Native experimental transactions require Keeper. All concurrent readers and writers must use transactions to participate in these snapshot guarantees. SDK query settings that could bypass transaction or visibility semantics are replaced. Ordinary context values and cancellation propagate. For live tests set SCOPE_CLICKHOUSE_DSN to an isolated suitable server. See upstream setup: https://github.com/ClickHouse/ClickHouse/blob/v26.8.17.4-lts/tests/config/config.d/transactions.xml.
Index ¶
- Constants
- type DistanceMetric
- type Store
- func (s *Store) CheckDocuments(ctx context.Context, space vectorstore.SpaceID, docs []*document.Document) error
- func (s *Store) CheckQuery(ctx context.Context, space vectorstore.SpaceID, opts vectorstore.SearchOptions) error
- func (s *Store) DeleteIDs(ctx context.Context, ids []string) (err error)
- func (s *Store) DeleteMatching(ctx context.Context, predicate filter.Predicate, budget vectorstore.ScanBudget) (err error)
- func (s *Store) Query(ctx context.Context, request *vectorstore.Query) (response *vectorstore.SearchResponse, err error)
- func (s *Store) Scan(ctx context.Context, request vectorstore.ScanRequest) (found []*document.Document, err error)
- func (s *Store) Upsert(ctx context.Context, request *vectorstore.Batch) (err error)
- type StoreConfig
Constants ¶
const ( DefaultTableName = "vector_store" DefaultIDColumn = "id" DefaultContentColumn = "content" DefaultMetadataColumn = "metadata" DefaultEmbeddingColumn = "embedding" DefaultDistanceMetric = DistanceCosine )
const Provider = "ClickHouse"
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DistanceMetric ¶
type DistanceMetric string
const ( // DistanceCosine uses cosineDistance(a, b) — returns 1 - cosine // similarity, range [0, 2]. DistanceCosine DistanceMetric = "cosine" // DistanceL2 uses L2Distance(a, b) — Euclidean distance, // range [0, ∞). DistanceL2 DistanceMetric = "l2" )
func (DistanceMetric) String ¶
func (d DistanceMetric) String() string
func (DistanceMetric) Valid ¶
func (d DistanceMetric) Valid() bool
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
func NewStore ¶
func NewStore(ctx context.Context, config StoreConfig) (*Store, error)
NewStore verifies native transaction support and the strict current schema. An omitted database is resolved once; every operation uses its qualified table.
func (*Store) CheckDocuments ¶ added in v0.59.0
func (*Store) CheckQuery ¶ added in v0.59.0
func (s *Store) CheckQuery(ctx context.Context, space vectorstore.SpaceID, opts vectorstore.SearchOptions) error
func (*Store) DeleteIDs ¶
DeleteIDs removes the IDs visible in one native snapshot. Empty input is a no-op.
func (*Store) DeleteMatching ¶ added in v0.59.0
func (s *Store) DeleteMatching(ctx context.Context, predicate filter.Predicate, budget vectorstore.ScanBudget) (err error)
DeleteMatching proves membership over the complete bounded snapshot before buffering deletes on the same session. Only native COMMIT publishes them.
func (*Store) Query ¶ added in v0.59.0
func (s *Store) Query(ctx context.Context, request *vectorstore.Query) (response *vectorstore.SearchResponse, err error)
func (*Store) Scan ¶ added in v0.59.0
func (s *Store) Scan(ctx context.Context, request vectorstore.ScanRequest) (found []*document.Document, err error)
Scan returns no partial records. Its grant covers a single native snapshot.
type StoreConfig ¶
type StoreConfig struct {
// DB uses clickhouse-go v2 with the native TCP protocol. The host owns it.
DB *sql.DB
DatabaseName string
TableName string
IDColumn string
ContentColumn string
MetadataColumn string
EmbeddingColumn string
Space vectorstore.SpaceID
Dimensions int
DistanceMetric DistanceMetric
InitializeSchema bool
}
StoreConfig binds the host-owned native ClickHouse database/sql handle. The server must support native transactions; all concurrent writers must use transactions to participate in its snapshots.
func (StoreConfig) Validate ¶
func (s StoreConfig) Validate() error
Source Files