dalgo2typesense

dalgo2typesense

A bounded, read-only native Typesense adapter for DALgo.

It reads documents already stored in an existing Typesense 30.2 collection. It does not write documents, create collections, or require a DALgo envelope. The Typesense document id (a string, including numeric-looking values such as "001") is the DALgo record key.

Status

Adapter checkpoint. The Go CLI and any TypeScript/web UI are separate follow-on deliverables, not part of this module. The module does not link, embed, copy or redistribute the Typesense server or SDK.

Usage

db, err := dalgo2typesense.NewDB(ctx, dalgo2typesense.Config{
    Endpoint:   "https://typesense.example.com:8108",
    Collection: "books",
    Headers: func(ctx context.Context) (map[string]string, error) {
        return map[string]string{"X-TYPESENSE-API-KEY": apiKey}, nil
    },
})

NewDB performs no network I/O. Use dal.As[io.Closer] for Close. The header callback runs once per request, so a rotated scoped key is picked up without constructing a second database. The adapter never logs or stores key values.

Native search

searcher, ok := dal.As[dalgo2typesense.NativeSearcher](db)
result, err := searcher.Search(ctx, dalgo2typesense.SearchRequest{
    Collection: "books",
    Q:          "dune",
    QueryBy:    []string{"title", "author"},
    FilterBy:   "year:>=1965 && genre:=`science fiction`",
    SortBy:     []dalgo2typesense.SortField{{Field: "rank", Descending: true}},
    FacetBy:    []string{"genre"},
    Limit:      20,
})

SearchRequest is typed and bounded. It has no arbitrary option map, raw URL, management endpoint, multi-search, join or RPC surface. FilterBy is validated against a documented subset: joins ($), object filters ({}), control characters, unbalanced backticks and unbalanced parentheses are refused before any request. Facet results are native Typesense facet statistics, not SQL aggregates.

Structured DALgo queries

A single existing root collection, q=*, a positive bounded limit, a bounded offset, top-level scalar AND/OR comparisons, projection and explicit sort are translated. Joins, grouping, HAVING, subqueries, computed columns, nested fields, wildcard projection, unsupported operators and NULL tests are refused before I/O. IS NULL / IS NOT NULL are refused because Typesense does not provide one uniform meaning for absent and explicit-null values.

Structured queries are not text searches. They explicitly send num_typos=0, prefix=false, drop_tokens_threshold=0 and typo_tokens_threshold=0 in addition to q=*. Native Q keeps the server's honest typo, prefix and token-dropping semantics.

Exact structured predicates and honest refusal

A native Typesense filter_by is only a bounded candidate producer. Typesense filtering folds case, accents, punctuation and whitespace, matches array elements for scalar filters, and can return rows that the admitted DALgo predicate does not actually satisfy. The adapter therefore compiles the same condition AST into a small exact verifier and checks every native candidate before exposing it.

• The whole AND/OR tree is evaluated exactly. Scalar Equal and numeric ranges are scalar-only: an array value is not equal to a scalar and is not a numeric range. In intentionally matches an exact array element.
• A missing predicate field never authorizes a row by itself. A true sibling OR branch can still make the whole predicate true.
• If any real candidate fails the exact predicate, the read is refused with ErrSemanticMismatch (which also unwraps to dal.ErrNotSupported) before the first row is exposed. There is no silent row dropping, no scan-around retry, no automatic policy relaxation and no broad whole-collection scan.
• Filtered reads fetch native candidates from offset zero through offset+limit, verify the complete window, and only then apply the requested offset locally. A later-page mismatch or a cancellation during verification cannot leave an apparently successful prefix. The window shape is checked before I/O: a window that cannot fit the configured MaxPages, MaxRowsPerPage, MaxOffset or MaxColumns bounds is refused before any request. Row and response-byte limits, the operation deadline and cancellation are enforced while the window is buffered, so those refusals happen during the read, before any row is exposed. Unfiltered reads keep their lazy native-offset path.
• Predicate helper fields are fetched with the declared projection so the verifier can see them. They count against MaxColumns and are stripped before a projected row is constructed.
• Structured NOT IN is refused before I/O because native negative string folding is not a safe superset. NativeSearcher keeps its native negative filter grammar.

Numeric candidate widening (documented limitation). Typesense 30.2 stores raw JSON numbers but indexes float values through quantised buckets. An actual probe showed strict n:>0.1 omitting stored 0.1000000001 and 0.10000000149, and n:>100000000 returning nothing even though 100000001 is stored. A strict native filter alone therefore cannot be trusted to contain every eligible row. The adapter lowers structured strict > and < candidates to inclusive >= and <=; existing >= and <= are unchanged. The exact verifier still evaluates the original operator and the same compiler wire literal, so an inclusive boundary candidate is refused rather than silently dropped. This means an exact integer boundary (for example year > 1965 when a row has 1965) can refuse the read, and a fractional RHS on an integer field may be rejected by the server. That is an intentional, fail-closed initial limitation, not a bypass. NativeSearcher remains fully engine-native.

Curation and filter safety

Every read path sets:

enable_curations=false
filter_curated_hits=true

Typesense 30.2 defaults are enable_curations=true and filter_curated_hits=false, so a pinned or promoted document can otherwise bypass filter_by. The adapter applies both flags itself and the request types offer no way for a caller to override them. There is no pinned, promoted or native-override bypass.

Permissions

One scoped read-only credential callback is used for every request. A pure documents:search key scoped to one collection can execute native searches and exact-id document reads. Metadata calls made with that key fail explicitly with ErrMetadataForbidden; they are never converted into an empty collection list and never fall back to an admin credential. A key that additionally has collections:get and collections:list can discover the real collection catalog and schema through dbschema.SchemaReader. The adapter never creates collections, keys or documents.

Identity and value fidelity

• The native Typesense id is always a string. "001" and 1 are different.
• JSON numbers are decoded with UseNumber and resolved exactly before any value is exposed. An integral value stays integral even when it is spelled with an exponent or a decimal point: 9007199254740993e0 and 9007199254740993.0 both become exactly 9007199254740993, not the float64-rounded neighbour. Integers that fit int64 become int64; larger non-negative integers that fit uint64 become uint64. An integral value outside those ranges is rejected, never silently substituted. Only non-integral values become finite float64 values with documented IEEE754 semantics.
• A caller-supplied json.Number must use a valid JSON number spelling. Malformed forms such as 1., .1 and 01 are refused rather than normalised.
• Structured numeric comparison reuses the exact wire literal the native filter was built from. A float32 RHS is compared as its shortest float32 wire decimal (for example 0.1), never as the extra binary64 digits of float64(float32(v)). Integral json.Number spellings incl. exponents stay exact int64/uint64; fractional values use the documented finite IEEE754 value. Comparisons use exact rational arithmetic, so adjacent int64/uint64 values above 2^53 stay distinct and are never routed through float64.
• Facet stats keep the exact native JSON class too: an integral min, max, sum or avg is int64/uint64, a fractional one is float64. They stay native Typesense facet statistics, not SQL aggregates.
• Arrays, objects, booleans, strings and null are not stringified.
• A projected DALgo query returns exactly its declared columns in declaration order. The native identity may be fetched as a helper but never leaks into a projected row. A declared but absent field is represented as nil.
• Recordset columns are Column[any] with an explicit any ValueType; no declared affinity coerces a value.
• Missing, null or wrong hits / documents / fields arrays, bad IDs, duplicate IDs within or across pages, mismatched offsets and integer overflow are malformed responses, never an empty successful result.

Pagination and server caps

Typesense accepts at most 250 hits per page. One DALgo query may span several pages under one operation deadline, one byte budget and one page budget. found and out_of are server metadata, distinct from Returned hits. A short or empty page that the server's own found count says is not the end returns ErrPartialResult, so a scoped limit_hits is never presented as a clean EOF. An offset at or beyond found is a valid empty result and is reported as EOF, not as a malformed page.

The adapter also decodes the server's search_cutoff response indicator as a strict boolean. A cutoff makes a native result explicitly Partial with Complete false, even when the cutoff page returned zero hits and found is zero. DALgo readers, Get, Exists and GetMulti refuse a cutoff with ErrPartialResult; it is never mistaken for EOF, a missing document or a full result. search_cutoff_ms is not a strict bound, facets are not bound by it and the adapter does not send it as a hard provider deadline.

No id sort is invented. An explicit application-field sort is the caller's choice; repeatable pages are claimed only for unchanged data with a unique sort value. The adapter does not claim snapshot isolation under concurrent writes.

Configuration bounds

All bounds have modest defaults and hard maxima. Notable defaults:

MaxRowsPerPage is the physical page size of the structured reader and also bounds one native search or one exact-id GetMulti batch. A larger native request is refused before I/O; the configured bound is never silently ignored. MaxRowBytes is enforced by every read path (native search, structured reader and recordset, Get, Exists, GetMulti) before an over-budget document is exposed.

Redirects are refused before a credential can reach another origin. The adapter copies a caller-supplied http.Client and never mutates it. It only closes a transport it created. Error values retain a numeric HTTP status and safe classification, never a URL, query string, body, key or provider error text.

Licensing boundary

The Typesense server is GPL-3.0. This package is an independent MIT-licensed HTTP client. It does not link, embed, copy or redistribute the server or its SDK, and it does not relicense DALgo core or any CLI.

Verification

Unit tests use httptest and cover outbound curation policy, header refresh, redirect refusal, strict response shape, projection, paging, partial results, search_cutoff refusal, offset-at-or-beyond-found EOF, row and page budgets, exact exponent/decimal integer fidelity across readers, recordsets, filters and native facet stats, metadata refusal, cancellation, close and exact-id reads. They also pin the exact structured predicate verifier, filtered-window buffering, numeric candidate widening, float32 wire canonicalisation and adjacent large integers.

The integration suite starts one owned, pinned typesense/typesense:30.2@sha256:610f2d34b1f93d00762869da2c67736775e5798d19a2c8b91b014b8a0cc1e110 container on a random loopback port, seeds native documents and a curation set that promotes a document excluded by a filter, and proves the unprotected server premise and the adapter's red-to-green filter enforcement. It also exercises multi-page reads plus an offset at and beyond the real server's found count, and proves the configured page and row budgets against that server. The same fixture seeds the observed FLOAT boundary values and proves that every strict-eligible raw value appears in the inclusive candidate set, that a boundary candidate refuses before the first row, and that exact integer, string, array, AND/OR, filtered-offset and SecureReadSession residual cases behave as documented.