hugr

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2026 License: MIT Imports: 57 Imported by: 3

README

The Hugr query-engine

The Hugr query engine is a part of the Hugr project, which provides a powerful and flexible GraphQL query engine designed to work with various data sources. It supports features like caching, authentication, and schema extensions, making it suitable for building scalable and efficient applications.

See the Hugr documentation for more details.

Features

  • HTTP handlers for GraphQL API (IPC multipart/mixed with Arrow + JSON)
  • GraphiQL UI to execute queries
  • GraphQL schema definition and rule-based compilation
  • Data sources management (DuckDB, PostgreSQL, SQLite, MySQL, S3/Delta/Iceberg)
  • Schema extensions to add sub-queries to existing data sources
  • Schema catalog module with DB-backed metadata, vector embeddings, and LLM summarization
  • MCP (Model Context Protocol) endpoint for AI tool integration
  • Authentication and authorization (API keys, JWT/JWKS, DB-managed keys)
  • Cluster mode with management + worker nodes (heartbeat, schema sync, secret sync)
  • Caching: L1 (in-memory BigCache) and L2 (Redis or Memcached)
  • Query parsing, validation, and parallel execution
  • Result transformation (jq expressions)

Architecture

cmd/dev-server/       Dev server with env-based configuration
cmd/hugr-tools/       CLI utilities (summarize, reindex, schema-info)
pkg/catalog/          Schema catalog: compiler, static/db providers
pkg/cluster/          Cluster mode (management/worker nodes, heartbeat, secret sync)
pkg/data-sources/     Runtime data source management
pkg/planner/          Query planner and SQL generation
pkg/cache/            Two-level caching (L1 + L2)
pkg/auth/             Authentication middleware
pkg/mcp/              MCP server integration
integration-test/     Integration & E2E tests (compiler, DB provider, cluster, MCP, E2E)
docs/                 Internal documentation

Quick Start

Build
# Dev server
CGO_CFLAGS="-O1 -g" go build -tags=duckdb_arrow -o hugr ./cmd/dev-server

# CLI tools (summarize, reindex, schema-info)
CGO_CFLAGS="-O1 -g" go build -tags=duckdb_arrow -o hugr-tools ./cmd/hugr-tools
Install DuckDB extensions
./hugr -install
Run
# Minimal (in-memory)
./hugr

# With persistent CoreDB
CORE_DB_PATH=./core.db ./hugr

# With MCP endpoint + embeddings
CORE_DB_PATH=./core.db MCP_ENABLED=true \
  EMBEDDER_URL='http://localhost:8080/embed?model=text-embedding-3-small&api_key=sk-...' \
  ./hugr
Summarize schema with AI
# Generate descriptions for all schema entities using an LLM
hugr-tools summarize --api-key sk-... --provider openai --model gpt-4o-mini

# Recompute vector embeddings after summarization
hugr-tools reindex --batch-size 100

The server reads configuration from environment variables (or a .env file). See docs/configuration.md for the full reference.

Documentation

Document Description
docs/known-issues.md Known issues and workarounds
docs/configuration.md Dev server configuration reference
docs/hugr-tools.md CLI utilities (summarize, reindex, schema-info)
docs/compiler/01-overview.md Compiler architecture
docs/compiler/02-ddl-operations.md DDL operations
docs/compiler/03-directives.md Directive reference
docs/compiler/04-tables-views.md Tables and views
docs/compiler/05-functions.md Functions
docs/compiler/06-subqueries.md Subqueries and references
docs/compiler/07-extensions.md Extensions and modules

Dependencies

Key packages used:

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	NewClient                = client.NewClient
	WithApiKey               = client.WithApiKey
	WithApiKeyCustomHeader   = client.WithApiKeyCustomHeader
	WithUserRole             = client.WithUserRole
	WithUserRoleCustomHeader = client.WithUserRoleCustomHeader
	WithUserInfo             = client.WithUserInfo
	WithUserInfoCustomHeader = client.WithUserInfoCustomHeader
	WithTimeout              = client.WithTimeout
	WithTransport            = client.WithTransport
	WithToken                = client.WithToken
	WithHttpUrl              = client.WithHttpUrl
	WithJQQueryUrl           = client.WithJQQueryUrl
)
View Source 
var ErrEmptyRequest = errors.New("empty request")
View Source 
var ErrParallelMutationNotSupported = errors.New("parallel mutation queries are not supported")

Functions

This section is empty.

Types

type ChunkProcessFunc added in v0.1.20 

type ChunkProcessFunc func(ctx context.Context, path string, field *ast.Field, rec arrow.RecordBatch) error

type Client added in v0.1.9 

type Client = client.Client

type ClientConfig added in v0.1.9 

type ClientConfig = client.ClientConfig

type Config 

type Config struct {
	DB               db.Config
	AdminUI          bool
	AdminUIFetchPath string
	Debug            bool
	Profiling        bool

	AllowParallel      bool
	MaxParallelQueries int
	MaxDepth           int

	SchemaCacheMaxEntries int           // LRU cache max entries (0 = default 10000)
	SchemaCacheTTL        time.Duration // LRU cache TTL (0 = default 10m)

	MCPEnabled bool // Enable MCP endpoint on /mcp

	CoreDB   *coredb.Source
	Auth     *auth.Config
	Cache    cache.Config
	Embedder EmbedderConfig
	Cluster  cluster.ClusterConfig
}

type EmbedderConfig added in v0.3.0 

type EmbedderConfig struct {
	URL        string // Full URL with query params: model, api_key, api_key_header, timeout
	VectorSize int    // Optional vector size override (default: 0 - no vectorization)
}

type Info added in v0.1.9 

type Info struct {
	AdminUI bool `json:"admin_ui"`
	Debug   bool `json:"debug"`

	AllowParallel      bool `json:"allow_parallel"`
	MaxParallelQueries int  `json:"max_parallel_queries"`
	MaxDepth           int  `json:"max_depth"`

	DuckDB db.Config           `json:"duckdb"`
	CoreDB coredb.Info         `json:"coredb"`
	Auth   []auth.ProviderInfo `json:"auth"`
	Cache  cache.Config        `json:"cache"`
}

type Option added in v0.1.9 

type Option = client.Option

type Service 

type Service struct {
	// contains filtered or unexported fields
}

func New 

func New(config Config) (*Service, error)

func (*Service) AttachRuntimeSource 

func (s *Service) AttachRuntimeSource(ctx context.Context, source sources.RuntimeSource) error

func (*Service) Close 

func (s *Service) Close() error

func (*Service) ClusterSource added in v0.3.0 

func (s *Service) ClusterSource() *cluster.Source

ClusterSource returns the cluster source, or nil if cluster mode is disabled.

func (*Service) Commit 

func (s *Service) Commit(ctx context.Context) error

func (*Service) ContextWithTx 

func (s *Service) ContextWithTx(ctx context.Context) (context.Context, error)

func (*Service) CoreDBVersion added in v0.1.9 

func (s *Service) CoreDBVersion() coredb.Info

func (*Service) DataSourceStatus added in v0.1.9 

func (s *Service) DataSourceStatus(ctx context.Context, name string) (string, error)

func (*Service) DescribeDataSource added in v0.1.14 

func (s *Service) DescribeDataSource(ctx context.Context, name string, self bool) (string, error)

DescribeDataSource returns the formatted schema definition of a data source by its name.

func (*Service) Info added in v0.1.9 

func (s *Service) Info() Info

func (*Service) Init 

func (s *Service) Init(ctx context.Context) (err error)

func (*Service) LoadDataSource added in v0.1.8 

func (s *Service) LoadDataSource(ctx context.Context, name string) error

func (*Service) ProcessOperation added in v0.1.18 

func (s *Service) ProcessOperation(ctx context.Context, provider catalog.Provider, op *catalog.Operation) (map[string]any, map[string]any, error)

func (*Service) ProcessQuery 

func (s *Service) ProcessQuery(ctx context.Context, req types.Request) types.Response

func (*Service) ProcessStreamQuery added in v0.1.20 

func (s *Service) ProcessStreamQuery(ctx context.Context, query string, vars map[string]any) (table db.ArrowTable, finalize func(), err error)

func (*Service) Query 

func (s *Service) Query(ctx context.Context, query string, vars map[string]any) (*types.Response, error)

func (*Service) RegisterDataSource added in v0.1.8 

func (s *Service) RegisterDataSource(ctx context.Context, ds types.DataSource) error

func (*Service) Rollback 

func (s *Service) Rollback(ctx context.Context) error

func (*Service) SchemaProvider added in v0.3.0 

func (s *Service) SchemaProvider() catalog.Provider

func (*Service) ServeHTTP 

func (s *Service) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*Service) UnloadDataSource added in v0.1.8 

func (s *Service) UnloadDataSource(ctx context.Context, name string) error

type StreamMessage added in v0.1.20 

type StreamMessage struct {
	Type           string                 `json:"type"`
	DataObject     string                 `json:"data_object,omitempty"`     // Optional, used for streaming table data
	SelectedFields []string               `json:"selected_fields,omitempty"` // Optional, used for streaming table data
	Query          string                 `json:"query,omitempty"`
	Variables      map[string]interface{} `json:"variables,omitempty"`
	Error          string                 `json:"error,omitempty"`
}

Source Files

View all Source files

Directories

Path Synopsis
Package client provides an IPC client for the Hugr query engine.
 Package client provides an IPC client for the Hugr query engine.
cmd
dev-server command
hugr-tools command
integration-test
compare
internal
http_test_service command
pkg
admin-ui
auth
cache
catalog
catalog/compiler
catalog/compiler/base
catalog/compiler/rules
catalog/db
catalog/db/schema
Package schema provides serialization utilities for storing GraphQL schema metadata in database tables.
 Package schema provides serialization utilities for storing GraphQL schema metadata in database tables.
catalog/sdl
catalog/sources
catalog/static
catalog/types
catalog/validator
catalog/validator/rules
cluster
consts
data-sources
data-sources/sources
data-sources/sources/airport
data-sources/sources/duckdb
data-sources/sources/embedding
data-sources/sources/extension
data-sources/sources/http
data-sources/sources/mssql
data-sources/sources/mysql
data-sources/sources/postgres
data-sources/sources/runtime
data-sources/sources/runtime/core-db
data-sources/sources/runtime/data-sources
data-sources/sources/runtime/gis
data-sources/sources/runtime/meta-info
data-sources/sources/runtime/storage
db
engines
gis
jq
mcp
metadata
perm
planner
queries
types

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.