documentservice

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package documentservice exposes TreeDB's pre-alpha Haystack-style document service contract.

The package is the Go implementation seam for the HTTP/JSON API documented in docs/TREEDB_DOCUMENT_SERVICE_API.md. It maps Haystack-style documents (id/content/embedding/meta/score) onto TreeDB collections, supports exact dense-vector scoring with metadata filters, and serves keyword/hybrid retrieval through collection-native SearchText/SearchHybrid APIs. Keyword/hybrid filters and unavailable indexes fail closed; the service does not scan documents as a fallback.

The contract is pre-alpha: request/response schemas may change before TreeDB stabilizes. Exact dense scoring here is the correctness/MVP service path for Python and Haystack clients; hybrid vector sources use TreeDB's declared collection vector index and fail closed when it is unavailable.

Index

Constants

View Source 
const (
	// ContractVersion is returned by health and index metadata responses so
	// clients can pin the pre-alpha schema they implement.
	ContractVersion = "treedb-document-service/v1alpha1"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BenchmarkVectorIndexOptions 

type BenchmarkVectorIndexOptions struct {
	Strategy         collections.VectorIndexStrategy `json:"strategy,omitempty"`
	M                int                             `json:"m,omitempty"`
	EfConstruction   int                             `json:"ef_construction,omitempty"`
	EfSearch         int                             `json:"ef_search,omitempty"`
	QuantizedIndexes []QuantizedIndexInfo            `json:"quantized_indexes,omitempty"`
}

BenchmarkVectorIndexOptions configures the service-owned collection vector index for benchmark lifecycle setups. Omitted fields preserve the legacy create-index defaults used by treedb-client and treedb-haystack.

type BenchmarkVectorQueryMode 

type BenchmarkVectorQueryMode string

BenchmarkVectorQueryMode selects the no-document vector-index benchmark score plane. The legacy /search/vector route does not accept these modes and remains exact dense document scoring. 

const (
	BenchmarkVectorQueryModeExact           BenchmarkVectorQueryMode = "exact"
	BenchmarkVectorQueryModeQuantizedOnly   BenchmarkVectorQueryMode = "quantized_only"
	BenchmarkVectorQueryModeQuantizedRerank BenchmarkVectorQueryMode = "quantized_rerank"
)

type BenchmarkVectorSearchRequest 

type BenchmarkVectorSearchRequest struct {
	ExpectedGeneration        uint64                                 `json:"expected_generation,omitempty"`
	VectorIndexName           string                                 `json:"vector_index_name,omitempty"`
	QueryEmbedding            []float32                              `json:"query_embedding,omitempty"`
	QueryEmbeddingF32LEBase64 string                                 `json:"query_embedding_f32_le_b64,omitempty"`
	TopK                      int                                    `json:"top_k"`
	EfSearch                  int                                    `json:"ef_search,omitempty"`
	QueryMode                 BenchmarkVectorQueryMode               `json:"query_mode,omitempty"`
	QuantizedIndexName        string                                 `json:"quantized_index_name,omitempty"`
	QuantizedRerankCandidates int                                    `json:"quantized_rerank_candidates,omitempty"`
	StatsMode                 collections.VectorIndexSearchStatsMode `json:"stats_mode,omitempty"`
}

BenchmarkVectorSearchRequest runs fail-closed no-document vector-index search through Collection.SearchVectorIndexWithBuffer. Quantized modes require an explicit quantized index name; quantized_rerank may bound exact rerank with QuantizedRerankCandidates (rerank32 is the benchmark baseline when set to 32).

type BenchmarkVectorSearchResponse 

type BenchmarkVectorSearchResponse struct {
	Index                     IndexInfo                                `json:"index"`
	Results                   []BenchmarkVectorSearchResult            `json:"results"`
	Metric                    Metric                                   `json:"metric"`
	VectorIndexName           string                                   `json:"vector_index_name"`
	QueryMode                 BenchmarkVectorQueryMode                 `json:"query_mode"`
	QuantizedIndexName        string                                   `json:"quantized_index_name,omitempty"`
	QuantizedRerankCandidates int                                      `json:"quantized_rerank_candidates,omitempty"`
	NoDocuments               bool                                     `json:"no_documents"`
	Stats                     collections.VectorIndexSearchStats       `json:"stats"`
	Diagnostics               collections.VectorIndexSearchDiagnostics `json:"diagnostics"`
}

type BenchmarkVectorSearchResult 

type BenchmarkVectorSearchResult struct {
	ID      string  `json:"id"`
	Ordinal int     `json:"ordinal"`
	Score   float64 `json:"score"`
}

type CountDocumentsRequest 

type CountDocumentsRequest struct {
	ExpectedGeneration uint64  `json:"expected_generation,omitempty"`
	Filter             *Filter `json:"filter,omitempty"`
}

CountDocumentsRequest counts documents matching Filter. A nil filter counts every service document in the index.

type CountDocumentsResponse 

type CountDocumentsResponse struct {
	Index IndexInfo `json:"index"`
	Count int       `json:"count"`
}

type CreateIndexRequest 

type CreateIndexRequest struct {
	Name               string                       `json:"name"`
	Dimension          int                          `json:"dimension"`
	Metric             Metric                       `json:"metric,omitempty"`
	VectorIndexOptions *BenchmarkVectorIndexOptions `json:"vector_index_options,omitempty"`
}

CreateIndexRequest creates or opens a service index. Existing compatible indexes are returned idempotently; incompatible existing collections fail.

type DeleteDocumentsRequest 

type DeleteDocumentsRequest struct {
	ExpectedGeneration uint64   `json:"expected_generation,omitempty"`
	IDs                []string `json:"ids,omitempty"`
	Filter             *Filter  `json:"filter,omitempty"`
}

DeleteDocumentsRequest deletes either explicit IDs or documents matching a metadata filter. Supplying both IDs and Filter is rejected as ambiguous.

type DeleteDocumentsResponse 

type DeleteDocumentsResponse struct {
	Index   IndexInfo `json:"index"`
	Deleted int       `json:"deleted"`
	IDs     []string  `json:"ids"`
}

type DenseVectorSearchRequest 

type DenseVectorSearchRequest struct {
	ExpectedGeneration uint64    `json:"expected_generation,omitempty"`
	QueryEmbedding     []float32 `json:"query_embedding"`
	TopK               int       `json:"top_k"`
	Filter             *Filter   `json:"filter,omitempty"`
	ReturnEmbedding    bool      `json:"return_embedding,omitempty"`
}

DenseVectorSearchRequest runs exact dense scoring over documents that match Filter. QueryEmbedding must match the index dimension.

type DenseVectorSearchResponse 

type DenseVectorSearchResponse struct {
	Index      IndexInfo  `json:"index"`
	Documents  []Document `json:"documents"`
	Metric     Metric     `json:"metric"`
	Exact      bool       `json:"exact"`
	Candidates int        `json:"candidates"`
}

type Document 

type Document struct {
	ID        string         `json:"id"`
	Content   string         `json:"content,omitempty"`
	Embedding []float32      `json:"embedding,omitempty"`
	Score     *float64       `json:"score,omitempty"`
	Meta      map[string]any `json:"meta,omitempty"`
}

Document is the service's Haystack-compatible document shape.

type Error 

type Error struct {
	Code    ErrorCode `json:"code"`
	Message string    `json:"message"`
	Err     error     `json:"-"`
}

Error is a structured service error. Callers should branch on Code rather than matching message text.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Unwrap 

func (e *Error) Unwrap() error

type ErrorCode 

type ErrorCode string

ErrorCode is the stable, pre-alpha service error vocabulary returned by the HTTP API and exposed by Go service methods. 

const (
	CodeInvalidRequest   ErrorCode = "invalid_request"
	CodeMalformedJSON    ErrorCode = "malformed_json"
	CodeIndexNotFound    ErrorCode = "index_not_found"
	CodeIndexUnavailable ErrorCode = "index_unavailable"
	CodeIndexStale       ErrorCode = "index_stale"
	CodeConflict         ErrorCode = "conflict"
	CodeUnsupported      ErrorCode = "unsupported"
	CodeInternal         ErrorCode = "internal"
)

func ErrorCodeOf 

func ErrorCodeOf(err error) ErrorCode

ErrorCodeOf returns the service code for err. Non-service errors are internal.

type Filter 

type Filter struct {
	Operator   string   `json:"operator"`
	Field      string   `json:"field,omitempty"`
	Value      any      `json:"value,omitempty"`
	Conditions []Filter `json:"conditions,omitempty"`
}

Filter is the supported Haystack-style metadata filter AST.

Boolean nodes use operator AND/OR/NOT and conditions. Leaf nodes use field, operator, and value. Field names may be id, content, meta.<path>, or a metadata path without the meta. prefix.

func (*Filter) Validate 

func (f *Filter) Validate() error

Validate rejects unsupported operators and malformed filter shapes before any document scan runs.

type FilterDocumentsRequest 

type FilterDocumentsRequest struct {
	ExpectedGeneration uint64  `json:"expected_generation,omitempty"`
	Filter             *Filter `json:"filter,omitempty"`
	Limit              int     `json:"limit,omitempty"`
	Offset             int     `json:"offset,omitempty"`
	ReturnEmbedding    bool    `json:"return_embedding,omitempty"`
}

FilterDocumentsRequest lists documents matching Filter in stable document-ID order. Limit=0 returns all matches; Offset must be non-negative.

type FilterDocumentsResponse 

type FilterDocumentsResponse struct {
	Index        IndexInfo  `json:"index"`
	Documents    []Document `json:"documents"`
	MatchedCount int        `json:"matched_count"`
	Truncated    bool       `json:"truncated,omitempty"`
}

type Handler 

type Handler struct {
	Service      *Service
	MaxBodyBytes int64
}

Handler serves the pre-alpha HTTP/JSON document service contract.

func NewHandler 

func NewHandler(service *Service) *Handler

NewHandler returns an HTTP handler for service.

func (*Handler) ServeHTTP 

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

type HybridSearchRequest 

type HybridSearchRequest struct {
	ExpectedGeneration   uint64                          `json:"expected_generation,omitempty"`
	Query                string                          `json:"query,omitempty"`
	QueryEmbedding       []float32                       `json:"query_embedding,omitempty"`
	TopK                 int                             `json:"top_k"`
	TextCandidateLimit   int                             `json:"text_candidate_limit,omitempty"`
	VectorCandidateLimit int                             `json:"vector_candidate_limit,omitempty"`
	CandidateLimit       int                             `json:"candidate_limit,omitempty"`
	EfSearch             int                             `json:"ef_search,omitempty"`
	Fusion               collections.HybridFusionOptions `json:"fusion,omitempty"`
	Filter               *Filter                         `json:"filter,omitempty"`
	ReturnEmbedding      bool                            `json:"return_embedding,omitempty"`
}

HybridSearchRequest runs collection-native text/vector hybrid retrieval. At least one of Query or QueryEmbedding must be supplied. Metadata filters fail closed for now unless the service grows a bounded scalar-index mapping.

type HybridSearchResponse 

type HybridSearchResponse struct {
	Index       IndexInfo                        `json:"index"`
	Documents   []Document                       `json:"documents"`
	TextIndex   string                           `json:"text_index,omitempty"`
	VectorIndex string                           `json:"vector_index,omitempty"`
	Plan        collections.HybridSearchPlan     `json:"plan,omitempty"`
	Snapshot    collections.HybridSearchSnapshot `json:"snapshot,omitempty"`
	Stats       collections.HybridSearchStats    `json:"stats,omitempty"`
}

type IndexCapabilities 

type IndexCapabilities struct {
	DenseVectorSearch       bool `json:"dense_vector_search"`
	ExactDenseScoring       bool `json:"exact_dense_scoring"`
	MetadataFilters         bool `json:"metadata_filters"`
	KeywordSearch           bool `json:"keyword_search"`
	HybridSearch            bool `json:"hybrid_search"`
	KeywordMetadataFilters  bool `json:"keyword_metadata_filters"`
	HybridMetadataFilters   bool `json:"hybrid_metadata_filters"`
	BenchmarkLifecycle      bool `json:"benchmark_lifecycle"`
	VectorIndexMaintenance  bool `json:"vector_index_maintenance"`
	NoDocumentVectorSearch  bool `json:"no_document_vector_search"`
	ColumnGraphVectorSearch bool `json:"column_graph_vector_search"`
	ExactColumnGraphSearch  bool `json:"exact_column_graph_search"`
	QuantizedVectorSearch   bool `json:"quantized_vector_search"`
	QuantizedRerank         bool `json:"quantized_rerank"`
	ScalarU8QuantizedRerank bool `json:"scalar_u8_quantized_rerank"`
	RabitQ1BitExperimental  bool `json:"rabitq_1bit_experimental"`
}

IndexCapabilities describes the supported operations for one service index.

type IndexInfo 

type IndexInfo struct {
	Name                 string                          `json:"name"`
	Dimension            int                             `json:"dimension"`
	Metric               Metric                          `json:"metric"`
	Generation           uint64                          `json:"generation"`
	ContractVersion      string                          `json:"contract_version"`
	EmbeddingField       string                          `json:"embedding_field"`
	VectorIndexName      string                          `json:"vector_index_name"`
	VectorStrategy       collections.VectorIndexStrategy `json:"vector_strategy"`
	VectorM              int                             `json:"vector_m,omitempty"`
	VectorEfConstruction int                             `json:"vector_ef_construction,omitempty"`
	VectorEfSearch       int                             `json:"vector_ef_search,omitempty"`
	QuantizedIndexes     []QuantizedIndexInfo            `json:"quantized_indexes,omitempty"`
	TextField            string                          `json:"text_field"`
	TextIndexName        string                          `json:"text_index_name"`
	DocumentType         string                          `json:"document_type"`
	Capabilities         IndexCapabilities               `json:"capabilities"`
}

IndexInfo is returned by create/open and echoed by operation responses.

type KeywordSearchRequest 

type KeywordSearchRequest struct {
	ExpectedGeneration uint64                         `json:"expected_generation,omitempty"`
	Query              string                         `json:"query"`
	TopK               int                            `json:"top_k"`
	Operator           collections.TextSearchOperator `json:"operator,omitempty"`
	CandidateLimit     int                            `json:"candidate_limit,omitempty"`
	MaxPostingsScanned int                            `json:"max_postings_scanned,omitempty"`
	Filter             *Filter                        `json:"filter,omitempty"`
	ReturnEmbedding    bool                           `json:"return_embedding,omitempty"`
}

KeywordSearchRequest runs ranked lexical search over the service content text index. Metadata filters intentionally fail closed for keyword search in this pre-alpha contract; the service never scans documents as a fallback.

type KeywordSearchResponse 

type KeywordSearchResponse struct {
	Index     IndexInfo          `json:"index"`
	Documents []Document         `json:"documents"`
	TextIndex string             `json:"text_index"`
	Stats     KeywordSearchStats `json:"stats"`
}

type KeywordSearchStats 

type KeywordSearchStats struct {
	QueryTerms                int    `json:"query_terms,omitempty"`
	CandidatesRequested       uint64 `json:"candidates_requested,omitempty"`
	CandidatesReturned        uint64 `json:"candidates_returned,omitempty"`
	PostingsScanned           uint64 `json:"postings_scanned,omitempty"`
	CandidatesScored          uint64 `json:"candidates_scored,omitempty"`
	DocumentsFetched          uint64 `json:"documents_fetched,omitempty"`
	DocumentsMissing          uint64 `json:"documents_missing,omitempty"`
	FullDocumentScanFallbacks uint64 `json:"full_document_scan_fallbacks,omitempty"`
	PostingsScanNanos         uint64 `json:"postings_scan_nanos,omitempty"`
	CandidateScoreNanos       uint64 `json:"candidate_score_nanos,omitempty"`
	DocumentFetchNanos        uint64 `json:"document_fetch_nanos,omitempty"`
	Truncated                 bool   `json:"truncated,omitempty"`
	FailClosed                uint64 `json:"fail_closed,omitempty"`
	FailClosedReason          string `json:"fail_closed_reason,omitempty"`
	Unavailable               bool   `json:"unavailable,omitempty"`
	UnavailableReason         string `json:"unavailable_reason,omitempty"`
}

type Metric 

type Metric string

Metric selects the dense-vector score function. Scores returned by the service are always higher-is-better. 

const (
	MetricCosine       Metric = "cosine"
	MetricL2           Metric = "l2"
	MetricInnerProduct Metric = "inner_product"
)

type OptimizeIndexRequest 

type OptimizeIndexRequest struct {
	ExpectedGeneration uint64 `json:"expected_generation,omitempty"`
	VectorIndexName    string `json:"vector_index_name,omitempty"`
}

OptimizeIndexRequest rebuilds service vector assets after benchmark load.

type OptimizeIndexResponse 

type OptimizeIndexResponse struct {
	Index           IndexInfo                    `json:"index"`
	VectorIndexName string                       `json:"vector_index_name"`
	Status          VectorIndexMaintenanceStatus `json:"status"`
}

type QuantizedIndexInfo 

type QuantizedIndexInfo struct {
	Name                string                                 `json:"name"`
	Codec               string                                 `json:"codec"`
	Version             uint32                                 `json:"version"`
	ScalarU8Calibration *collections.ScalarU8CalibrationConfig `json:"scalar_u8_calibration,omitempty"`
}

QuantizedIndexInfo describes a declared quantized score plane attached to the service vector index.

type ResetIndexRequest 

type ResetIndexRequest struct {
	Dimension          int                          `json:"dimension"`
	Metric             Metric                       `json:"metric,omitempty"`
	DropOld            bool                         `json:"drop_old,omitempty"`
	VectorIndexOptions *BenchmarkVectorIndexOptions `json:"vector_index_options,omitempty"`
}

ResetIndexRequest creates a missing benchmark index or clears an existing compatible non-column_graph index when DropOld is true. Column_graph benchmark reset fails closed for existing indexes; use a fresh data directory or unique index name to preserve the insert-only load boundary required by graph assets.

type ResetIndexResponse 

type ResetIndexResponse struct {
	Index            IndexInfo `json:"index"`
	Created          bool      `json:"created"`
	Reset            bool      `json:"reset"`
	DropOld          bool      `json:"drop_old"`
	DroppedDocuments int       `json:"dropped_documents"`
}

type Service 

type Service struct {
	// contains filtered or unexported fields
}

Service maps the document/search contract onto TreeDB collections.

func New 

func New(manager *collections.CollectionManager) *Service

New returns a document/search service backed by manager.

func (*Service) Close 

func (s *Service) Close() error

Close releases service-owned cached search resources and marks the service unavailable. The underlying collection manager and database remain owned by the caller.

func (*Service) CountDocuments 

func (s *Service) CountDocuments(ctx context.Context, index string, req CountDocumentsRequest) (CountDocumentsResponse, error)

CountDocuments counts documents matching a filter.

func (*Service) CreateIndex 

func (s *Service) CreateIndex(ctx context.Context, req CreateIndexRequest) (IndexInfo, error)

CreateIndex creates or opens a compatible document service index.

func (*Service) DeleteDocuments 

func (s *Service) DeleteDocuments(ctx context.Context, index string, req DeleteDocumentsRequest) (DeleteDocumentsResponse, error)

DeleteDocuments deletes explicit IDs or documents matching a filter.

func (*Service) FilterDocuments 

func (s *Service) FilterDocuments(ctx context.Context, index string, req FilterDocumentsRequest) (FilterDocumentsResponse, error)

FilterDocuments returns documents matching a filter in stable document-ID order.

func (*Service) OpenIndex 

func (s *Service) OpenIndex(ctx context.Context, name string) (IndexInfo, error)

OpenIndex returns metadata for an existing document service index.

func (*Service) OptimizeIndex 

func (s *Service) OptimizeIndex(ctx context.Context, index string, req OptimizeIndexRequest) (OptimizeIndexResponse, error)

OptimizeIndex rebuilds service vector assets after a benchmark load phase.

func (*Service) ResetIndex 

func (s *Service) ResetIndex(ctx context.Context, index string, req ResetIndexRequest) (ResetIndexResponse, error)

ResetIndex creates a missing benchmark index or clears an existing compatible non-column_graph index when drop_old is requested. Column_graph benchmark indexes intentionally fail closed for in-place reset: rebuilding those assets after delete/reinsert tombstones is not the fresh insert-only load boundary VectorDBBench needs. Managed benchmark runs should use a fresh data directory; external shared services should use a unique index name per run.

func (*Service) SearchBenchmarkVector 

func (s *Service) SearchBenchmarkVector(ctx context.Context, index string, req BenchmarkVectorSearchRequest) (BenchmarkVectorSearchResponse, error)

SearchBenchmarkVector runs fail-closed no-document vector-index benchmark search. It never materializes documents and never falls back to exact dense document scanning.

func (*Service) SearchDenseVector 

func (s *Service) SearchDenseVector(ctx context.Context, index string, req DenseVectorSearchRequest) (DenseVectorSearchResponse, error)

SearchDenseVector runs exact dense scoring with optional metadata filters.

func (*Service) SearchHybrid 

func (s *Service) SearchHybrid(ctx context.Context, index string, req HybridSearchRequest) (HybridSearchResponse, error)

SearchHybrid runs collection-native hybrid retrieval with text and/or vector sources.

func (*Service) SearchKeyword 

func (s *Service) SearchKeyword(ctx context.Context, index string, req KeywordSearchRequest) (KeywordSearchResponse, error)

SearchKeyword runs ranked lexical search over the service content text index.

func (*Service) UpsertDocuments 

func (s *Service) UpsertDocuments(ctx context.Context, index string, req UpsertDocumentsRequest) (UpsertDocumentsResponse, error)

UpsertDocuments writes or replaces documents in index.

type UpsertDocumentsRequest 

type UpsertDocumentsRequest struct {
	ExpectedGeneration uint64     `json:"expected_generation,omitempty"`
	Documents          []Document `json:"documents"`
	// DeferVectorIndexRebuild lets benchmark loaders bulk-insert documents and
	// rebuild service vector assets once via OptimizeIndex after the load phase.
	DeferVectorIndexRebuild bool `json:"defer_vector_index_rebuild,omitempty"`
}

UpsertDocumentsRequest writes or replaces service documents.

type UpsertDocumentsResponse 

type UpsertDocumentsResponse struct {
	Index    IndexInfo `json:"index"`
	Upserted int       `json:"upserted"`
	Inserted int       `json:"inserted"`
	Updated  int       `json:"updated"`
	IDs      []string  `json:"ids"`
}

type VectorIndexMaintenanceStatus 

type VectorIndexMaintenanceStatus struct {
	Name             string                          `json:"name"`
	Strategy         collections.VectorIndexStrategy `json:"strategy"`
	State            collections.VectorIndexState    `json:"state"`
	Reason           collections.VectorIndexReason   `json:"reason"`
	Loaded           bool                            `json:"loaded"`
	RebuildNeeded    bool                            `json:"rebuild_needed"`
	RootID           uint64                          `json:"root_id,omitempty"`
	NativeRootLoaded bool                            `json:"native_root_loaded,omitempty"`
	NativeRootBytes  int64                           `json:"native_root_bytes,omitempty"`
	DurationNanos    int64                           `json:"duration_nanos,omitempty"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.