plugin

Overview ¶

Package plugin is the IVF-PQ vector index integration AND the canonical reference for adding a new vector index algorithm to MatrixOne.

How to add a new vector index algorithm ¶

1. Pick an algo token (e.g. "scann"). Add a constant for it in pkg/catalog/secondary_index_utils.go alongside MoIndexIvfpqAlgo, and a tree.INDEX_TYPE_<X> case in pkg/sql/parsers (only if the algorithm introduces a new CREATE INDEX keyword).

2. Add hidden-table-type constants in pkg/catalog/types.go alongside Ivfpq_TblType_Metadata / Ivfpq_TblType_Storage — one per hidden table the algorithm needs.

3. Copy this directory to pkg/vectorindex/<algo>/plugin/. Rename the inner package names and update the imports. You'll end up with:

   pkg/vectorindex/<algo>/plugin/ ├── plugin.go -- this file: registry entry point ├── runtime/runtime.go -- CatalogHooks (HiddenTableTypes, │ DefaultOptions, ExperimentalFlag) ├── compile/compile.go -- compile.Hooks (CREATE/ALTER/DROP/SYNC) └── plan/ ├── plan.go -- plan.Hooks: thin redirect (~20 LoC) │ whose ApplyForSort / CanApply forward │ to *QueryBuilder methods in pkg/sql/plan ├── schema.go -- BuildSecondaryIndexDefs body │ (hidden-table TableDefs + IndexDefs) └── tablefunc.go -- <algo>_create / <algo>_search FUNCTION_SCAN builders

   Rule of thumb for which sub-package gets the body: lifted code from pkg/sql/compile/<file>.go → compile/; from pkg/sql/plan/<file>.go → plan/; runtime/ is reserved for algorithm-metadata constants that don't belong to a SQL pipeline layer.

4. Implement the three Hooks interfaces: - pkg/indexplugin/catalog.Hooks (4 methods — metadata) - pkg/indexplugin/compile.Hooks (~12 methods — DDL execution) - pkg/indexplugin/plan.Hooks (3 methods — schema + two thin ANN redirects) The Go compiler enforces completeness via the `var _ Hooks = Hooks{}` interface checks in each sub-package.

5. If the algorithm supports ANN `ORDER BY <distfn>(col, v) LIMIT k`, add the body methods to pkg/sql/plan:

   pkg/sql/plan/apply_indices_<algo>.go: func (builder *QueryBuilder) applyIndicesForSortUsing<Algo>(...) func (builder *QueryBuilder) prepare<Algo>IndexContext(...)

   Then wire four redirect methods on *QueryBuilder in pkg/sql/plan/plugin_builder.go (ApplyIndicesForSortUsing<Algo> + CanApply<Algo>) and four matching abstract methods on planplugin.PlanBuilder in pkg/indexplugin/plan/hooks.go. Add the dispatch case at pkg/sql/plan/apply_indices.go.

6. Register: this file's init() calls plugin.Register(New()). To make production binaries and tests pick it up, add ONE blank import line to pkg/indexplugin/all/all.go. That aggregator is the only place that needs editing — pkg/sql/plan and pkg/sql/compile already blank-import pkg/indexplugin/all.

7. End-to-end test: CREATE INDEX, populate, ORDER BY <distfn>(col, v) LIMIT k, ALTER REINDEX, DROP INDEX, DROP TABLE all exercise different hook paths. Add a SQL case under test/distributed/cases/vector/.

Helpers the plugin may use without re-implementing them:

• pkg/indexplugin/plan — schema / tablefunc helper function variables (CreateIndexDef, MakeHiddenColDefByName, ValidateIncludeColumns, DeepCopyColDefList) wired by pkg/sql/plan's init(). Use these from schema.go / tablefunc.go.
• pkg/sql/util.BuildIndexTableName — generate a hidden table name.
• pkg/vectorindex/cache.Cache — runtime in-memory index cache.
• pkg/vectorindex/metric — distance functions, op_type registry.

Helpers the plugin must NOT touch:

• pkg/sql/plan or pkg/sql/compile directly — those packages blank-import the plugin for init() registration, so the cycle would break. Always route through the framework hook interfaces.

What this specific file (plugin.go) does ¶

It is the single registration point. It assembles the three Hooks implementations from the sub-packages into one AlgoPlugin and registers it via init(). If you forget any of the three Hooks the `var _ AlgoPlugin = (*Plugin)(nil)` interface check below fails to compile — that is the safety net the framework provides.