modusgraph

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 21, 2025 License: Apache-2.0 Imports: 42 Imported by: 1

README

modus

GitHub License chat GitHub Repo stars GitHub commit activity

Docs · Discord

modusGraph is a high-performance, transactional database system. It's designed to be type-first, schema-agnostic, and portable. ModusGraph provides object-oriented APIs that make it simple to build new apps, paired with support for advanced use cases through the Dgraph Query Language (DQL). A dynamic schema allows for natural relations to be expressed in your data with performance that scales with your use case.

modusGraph is available as a Go package for running in-process, providing low-latency reads, writes, and vector searches. We’ve made trade-offs to prioritize speed and simplicity. When runnning in-process, modusGraph internalizes Dgraph's server components, and data is written to a local file-based database. modusGraph also supports remote Dgraph servers, allowing you deploy your apps to any Dgraph cluster simply by changing the connection string.

The modus framework is optimized for apps that require sub-second response times. ModusGraph augments polyglot functions with simple to use data and vector storage. When paired together, you can build a complete AI semantic search or retrieval-augmented generation (RAG) feature with a single framework.

Quickstart

package main

import (
    "context"
    "fmt"
    "time"

    mg "github.com/hypermodeinc/modusgraph"
)

type TestEntity struct {
    Name        string    `json:"name,omitempty" dgraph:"index=exact"`
    Description string    `json:"description,omitempty" dgraph:"index=term"`
    CreatedAt   time.Time `json:"createdAt,omitempty"`

    // UID is a required field for nodes
    UID string `json:"uid,omitempty"`
    // DType is a required field for nodes, will get populated with the struct name
    DType []string `json:"dgraph.type,omitempty"`
}

func main() {
    // Use a file URI to connect to a in-process modusGraph instance, ensure that the directory exists
    uri := "file:///tmp/modusgraph"
    // Assigning a Dgraph URI will connect to a remote Dgraph server
    // uri := "dgraph://localhost:9080"

    client, err := mg.NewClient(uri, mg.WithAutoSchema(true))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    entity := TestEntity{
        Name:        "Test Entity",
        Description: "This is a test entity",
        CreatedAt:   time.Now(),
    }

    ctx := context.Background()
    err = client.Insert(ctx, &entity)

    if err != nil {
        panic(err)
    }
    fmt.Println("Insert successful, entity UID:", entity.UID)

    // Query the entity
    var result TestEntity
    err = client.Get(ctx, &result, entity.UID)
    if err != nil {
        panic(err)
    }
    fmt.Println("Query successful, entity:", result.UID)
}

Limitations

modusGraph has a few limitations to be aware of:

  • Unique constraints in file-based mode: Due to the intricacies of how Dgraph handles unique fields and upserts in its core package, unique field checks and upsert operations are not supported (yet) when using the local (file-based) mode. These operations work properly when using a full Dgraph cluster, but the simplified file-based mode does not support the constraint enforcement mechanisms required for uniqueness guarantees.

  • Schema evolution: While modusGraph supports schema inference through tags, evolving an existing schema with new fields requires careful consideration to avoid data inconsistencies.

CLI Commands and Examples

modusGraph provides several command-line tools and example applications to help you interact with and explore the package. These are organized in the cmd and examples folders:

Commands (cmd folder)

  • cmd/query: A flexible CLI tool for running arbitrary DQL (Dgraph Query Language) queries against a modusGraph database.
    • Reads a query from standard input and prints JSON results.
    • Supports file-based modusGraph storage.
    • Flags: --dir, --pretty, --timeout, -v (verbosity).
    • See cmd/query/README.md for usage and examples.

Examples (examples folder)

  • examples/basic: Demonstrates CRUD operations for a simple Thread entity.

    • Flags: --dir, --addr, --cmd, --author, --name, --uid, --workspace.
    • Supports create, update, delete, get, and list commands.
    • See examples/basic/README.md for details.
  • examples/load: Shows how to load the standard 1million RDF dataset into modusGraph for benchmarking.

    • Downloads, initializes, and loads the dataset into a specified directory.
    • Flags: --dir, --verbosity.
    • See examples/load/README.md for instructions.

You can use these tools as starting points for your own applications or as references for integrating modusGraph into your workflow.

Open Source

The modus framework, including modusGraph, is developed by Hypermode as an open-source project, integral but independent from Hypermode.

We welcome external contributions. See the CONTRIBUTING.md file if you would like to get involved.

Modus and its components are © Hypermode Inc., and licensed under the terms of the Apache License, Version 2.0. See the LICENSE file for a complete copy of the license. If you have any questions about modus licensing, or need an alternate license or other arrangement, please contact us at hello@hypermode.com.

Acknowledgements

modusGraph builds heavily upon packages from the open source projects of Dgraph (graph query processing and transaction management), Badger (data storage), and Ristretto (cache). modusGraph also relies on the dgman repository for much of its functionality. We expect the architecture and implementations of modusGraph and Dgraph to expand in differentiation over time as the projects optimize for different core use cases, while maintaining Dgraph Query Language (DQL) compatibility.

Documentation

Index

Constants

View Source
const (
	// DgraphURIPrefix is the prefix for Dgraph server connections
	DgraphURIPrefix = "dgraph://"

	// FileURIPrefix is the prefix for file-based local connections
	FileURIPrefix = "file://"
)

Variables

View Source
var (
	ErrSingletonOnly = errors.New("only one instance of modusDB can exist in a process")
	ErrEmptyDataDir  = errors.New("data directory is required")
	ErrClosedEngine  = errors.New("modusDB engine is closed")
	ErrNonExistentDB = errors.New("namespace does not exist")
)

Functions

func Create

func Create[T any](ctx context.Context, engine *Engine, object T,
	nsId ...uint64) (uint64, T, error)

func Delete

func Delete[T any, R UniqueField](ctx context.Context, engine *Engine, uniqueField R,
	nsId ...uint64) (uint64, T, error)

func Get

func Get[T any, R UniqueField](ctx context.Context, engine *Engine, uniqueField R,
	nsId ...uint64) (uint64, T, error)

func Query

func Query[T any](ctx context.Context, engine *Engine, queryParams QueryParams,
	nsId ...uint64) ([]uint64, []T, error)

func ResetSingleton

func ResetSingleton()

ResetSingleton resets the singleton state for testing purposes. This should ONLY be called during testing, typically in cleanup functions.

func Upsert

func Upsert[T any](ctx context.Context, engine *Engine, object T,
	nsId ...uint64) (uint64, T, bool, error)

Types

type Client

type Client interface {
	Insert(context.Context, any) error
	Upsert(context.Context, any) error
	Update(context.Context, any) error
	Get(context.Context, any, string) error
	Query(context.Context, any) *dg.Query
	Delete(context.Context, []string) error
	Close()

	UpdateSchema(context.Context, ...any) error
	GetSchema(context.Context) (string, error)
	DropAll(context.Context) error
	DropData(context.Context) error
	// QueryRaw executes a raw Dgraph query with optional query variables.
	// The `query` parameter is the Dgraph query string.
	// The `vars` parameter is a map of variable names to their values, used to parameterize the query.
	QueryRaw(context.Context, string, map[string]string) ([]byte, error)

	DgraphClient() (*dgo.Dgraph, func(), error)
}

Client provides an interface for Dgraph operations

func NewClient

func NewClient(uri string, opts ...ClientOpt) (Client, error)

NewClient creates a new graph database client instance based on the provided URI.

The function supports two URI schemes:

  • dgraph://host:port - Connects to a remote Dgraph instance
  • file:///path/to/db - Creates or opens a local file-based database

Optional configuration can be provided via the opts parameter:

  • WithAutoSchema(bool) - Enable/disable automatic schema creation for inserted objects
  • WithPoolSize(int) - Set the connection pool size for better performance under load
  • WithNamespace(string) - Set the database namespace for multi-tenant installations
  • WithLogger(logr.Logger) - Configure structured logging with custom verbosity levels

The returned Client provides a consistent interface regardless of whether you're connected to a remote Dgraph cluster or a local embedded database. This abstraction helps prevent the connection issues that can occur with raw gRPC/bufconn setups.

For file-based URIs, the client maintains a singleton Engine instance to ensure data consistency across multiple client connections to the same database.

type ClientOpt

type ClientOpt func(*clientOptions)

ClientOpt is a function that configures a client

func WithAutoSchema

func WithAutoSchema(enable bool) ClientOpt

WithAutoSchema enables automatic schema management

func WithLogger

func WithLogger(logger logr.Logger) ClientOpt

WithLogger sets a structured logger for the client

func WithNamespace

func WithNamespace(namespace string) ClientOpt

WithNamespace sets the namespace for the client

func WithPoolSize

func WithPoolSize(size int) ClientOpt

WithPoolSize sets the size of the dgo client connection pool

type Config

type Config struct {
	// contains filtered or unexported fields
}

func NewDefaultConfig

func NewDefaultConfig(dir string) Config

func (Config) WithLimitNormalizeNode

func (cc Config) WithLimitNormalizeNode(d int) Config

func (Config) WithLogger

func (cc Config) WithLogger(logger logr.Logger) Config

WithLogger sets a structured logger for the engine

type ConstrainedField

type ConstrainedField struct {
	Key   string
	Value any
}

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is an instance of modusDB. For now, we only support one instance of modusDB per process.

func NewEngine

func NewEngine(conf Config) (*Engine, error)

NewEngine returns a new modusDB instance.

func (*Engine) Close

func (engine *Engine) Close()

Close closes the modusDB instance.

func (*Engine) CreateNamespace

func (engine *Engine) CreateNamespace() (*Namespace, error)

func (*Engine) DropAll

func (engine *Engine) DropAll(ctx context.Context) error

DropAll drops all the data and schema in the modusDB instance.

func (*Engine) GetClient

func (engine *Engine) GetClient() (*dgo.Dgraph, error)

func (*Engine) GetDefaultNamespace

func (engine *Engine) GetDefaultNamespace() *Namespace

func (*Engine) GetNamespace

func (engine *Engine) GetNamespace(nsID uint64) (*Namespace, error)

func (*Engine) LeaseUIDs

func (ns *Engine) LeaseUIDs(numUIDs uint64) (*pb.AssignedIds, error)

func (*Engine) Load

func (engine *Engine) Load(ctx context.Context, schemaPath, dataPath string) error

func (*Engine) LoadData

func (engine *Engine) LoadData(inCtx context.Context, dataDir string) error

type Filter

type Filter struct {
	Field  string
	String StringPredicate
	Vector VectorPredicate
	And    *Filter
	Or     *Filter
	Not    *Filter
}

type ModusDbOption

type ModusDbOption func(*modusDbOptions)

func WithNamespaceOLD

func WithNamespaceOLD(ns uint64) ModusDbOption

type Namespace

type Namespace struct {
	// contains filtered or unexported fields
}

Namespace is one of the namespaces in modusDB.

func (*Namespace) AlterSchema

func (ns *Namespace) AlterSchema(ctx context.Context, sch string) error

func (*Namespace) DropAll

func (ns *Namespace) DropAll(ctx context.Context) error

DropAll drops all the data and schema in the modusDB instance.

func (*Namespace) DropData

func (ns *Namespace) DropData(ctx context.Context) error

DropData drops all the data in the modusDB instance.

func (*Namespace) ID

func (ns *Namespace) ID() uint64

func (*Namespace) Load

func (n *Namespace) Load(ctx context.Context, schemaPath, dataPath string) error

func (*Namespace) LoadData

func (n *Namespace) LoadData(inCtx context.Context, dataDir string) error

TODO: Add support for CSV file

func (*Namespace) Mutate

func (ns *Namespace) Mutate(ctx context.Context, ms []*api.Mutation) (map[string]uint64, error)

func (*Namespace) Query

func (ns *Namespace) Query(ctx context.Context, query string) (*api.Response, error)

Query performs query or mutation or upsert on the given modusDB instance.

func (*Namespace) QueryWithVars

func (ns *Namespace) QueryWithVars(ctx context.Context, query string, vars map[string]string) (*api.Response, error)

QueryWithVars performs query or mutation or upsert on the given modusDB instance.

type Pagination

type Pagination struct {
	Limit  int64
	Offset int64
	After  string
}

type QueryParams

type QueryParams struct {
	Filter     *Filter
	Pagination *Pagination
	Sorting    *Sorting
}

type Sorting

type Sorting struct {
	OrderAscField  string
	OrderDescField string
	OrderDescFirst bool
}

type StringPredicate

type StringPredicate struct {
	Equals         string
	LessThan       string
	LessOrEqual    string
	GreaterThan    string
	GreaterOrEqual string
	AllOfTerms     []string
	AnyOfTerms     []string
	AllOfText      []string
	AnyOfText      []string
	RegExp         string
}

type UniqueField

type UniqueField interface {
	uint64 | ConstrainedField
}

type VectorPredicate

type VectorPredicate struct {
	SimilarTo []float32
	TopK      int64
}

Directories

Path Synopsis
api
cmd
query command
examples
basic command
load command
readme command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL