datasource

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 8, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

README

Datasource

Kite provides following features to ensure robust and observable interactions with various data sources:

  1. Health Checks

A mechanism for a datasource to self-report its operational status. New datasources require implementing the HealthCheck() method with the signature:

HealthCheck() datasource.Health

This method should return the current health status of the datasource.

  1. Retry Mechanism

Kite attempts to re-establish connections if lost during application runtime. New datasources should be verified for built-in retry mechanisms. If absent, implement a mechanism for automatic reconnection.

  1. Metrics

Datasources should expose relevant metrics for performance monitoring. The specific metrics to be implemented depend on the datasource type. Discussions are required to determine the appropriate metrics for each new datasource.

  1. Logging

Kite supports level-based logging with the PrettyPrint interface. New datasources should implement logging with the following levels:

  • DEBUG: Logs connection attempts with critical details.
  • INFO: Logs successful connection establishment.
  • WARN: Logs connection retrying

Additional logs can be added to enhance debugging and improve user experience.

  1. Tracing

Kite supports tracing for all the datasources, for example for SQL it traces the request using https://github.com/XSAM/otelsql. If any official package or any widely used package is not available we have to implement our own, but in the scope of a different ISSUE.

All logs should include:

  • Timestamp
  • Request ID (Correlation ID)
  • Time taken to execute the query
  • Datasource name (consistent with other logs)

Implementing New Datasources

Kite offers built-in support for popular datasources like SQL (MySQL, PostgreSQL, SQLite), Redis, and Pub/Sub (MQTT, Kafka, Google as a backend). Including additional functionalities within the core Kite binary would increase the application size unnecessarily.

Therefore, Kite utilizes a pluggable approach for new datasources by separating implementation in the following way:

  • Interface Definition:

    Create an interface with required methods within the datasource package. Register the interface with the container (similar to MongoDB in https://github.com/tfogo/mongodb-go-tutorial).

  • Method Registration:

    Create a method in kite.go (similar to the existing one) that accepts the newly defined interface.

  • Separate Repository:

    Develop a separate repository to implement the interface for the new datasource. This approach ensures that the new datasource dependency is only loaded when utilized, minimizing binary size for Kite applications. It also empowers users to create custom implementations beyond the defaults provided by Kite.

Supported Datasources

Datasource Health-Check Logs Metrics Traces As Driver
MySQL ✅ ✅ ✅ ✅
REDIS ✅ ✅ ✅ ✅
PostgreSQL ✅ ✅ ✅ ✅
MongoDB ✅ ✅ ✅ ✅ ✅
SQLite ✅ ✅ ✅ ✅
BadgerDB ✅ ✅ ✅ ✅ ✅
Cassandra ✅ ✅ ✅ ✅ ✅
ClickHouse ✅ ✅ ✅ ✅
FTP ✅ ✅
SFTP ✅ ✅
Solr ✅ ✅ ✅ ✅
DGraph ✅ ✅ ✅ ✅
Azure Event Hubs ✅ ✅ ✅
OpenTSDB ✅ ✅ ✅ ✅
SurrealDB ✅ ✅ ✅ ✅ ✅
ArangoDB ✅ ✅ ✅ ✅ ✅
NATS-KV ✅ ✅ ✅ ✅ ✅
ScyllaDB ✅ ✅ ✅ ✅
Elasticsearch ✅ ✅ ✅ ✅ ✅
Couchbase ✅ ✅ ✅ ✅ ✅

Documentation

Overview

Package datasource contains all the supported data sources in Kite. A datasource refers to any component that provides access to data — such as databases or message queues. Kite comes with built-in support for SQL and Redis data sources out of the box.

Index

Constants

View Source 
const (
	StatusUp   = "UP"
	StatusDown = "DOWN"
)

Variables

View Source 
var (
	ErrFileClosed        = errors.New("File is closed")
	ErrOutOfRange        = errors.New("out of range")
	ErrTooLarge          = errors.New("too large")
	ErrFileNotFound      = os.ErrNotExist
	ErrFileExists        = os.ErrExist
	ErrDestinationExists = os.ErrExist
)

Functions

This section is empty.

Types

type Datasource 

type Datasource interface {
	Register(config config.Config)
}

type ErrorDB 

type ErrorDB struct {
	Err     error
	Message string
}

ErrorDB represents an error specific to database operations.

func (ErrorDB) Error 

func (e ErrorDB) Error() string

func (ErrorDB) StatusCode 

func (ErrorDB) StatusCode() int

func (ErrorDB) WithStack 

func (e ErrorDB) WithStack() ErrorDB

WithStack adds a stack trace to the Error.

type ErrorRecordNotFound 

type ErrorRecordNotFound ErrorDB

ErrorRecordNotFound represents the scenario where no records are found in the DB for the given ID.

func (ErrorRecordNotFound) StatusCode 

func (ErrorRecordNotFound) StatusCode() int

StatusCode implementation on the ErrorRecordNotFound is an aberration since the errors in datasource package should not have anything to do with HTTP status codes.

type File 

type File interface {
	io.Closer
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Writer
	io.WriterAt

	ReadAll() (RowReader, error)
}

File represents a file in the filesystem.

type FileSystem 

type FileSystem interface {
	// Create creates a file in the filesystem, returning the file and an
	// error, if any happens.
	Create(name string) (File, error)

	// Mkdir creates a directory in the filesystem, return an error if any
	// happens.
	Mkdir(name string, perm os.FileMode) error

	// MkdirAll creates a directory path and all parents that does not exist
	// yet.
	MkdirAll(path string, perm os.FileMode) error

	// Open opens a file, returning it or an error, if any happens.
	Open(name string) (File, error)

	// OpenFile opens a file using the given flags and the given mode.
	OpenFile(name string, flag int, perm os.FileMode) (File, error)

	// Remove removes a file identified by name, returning an error, if any
	// happens.
	Remove(name string) error

	// RemoveAll removes a directory path and any children it contains. It
	// does not fail if the path does not exist (return nil).
	RemoveAll(path string) error

	// Rename renames a file.
	Rename(oldname, newname string) error
}

FileSystem : Any simulated or real filesystem should implement this interface.

type FileSystemProvider 

type FileSystemProvider interface {
	FileSystem

	// UseLogger sets the logger for the FileSystem client.
	UseLogger(logger any)

	// UseMetrics sets the metrics for the FileSystem client.
	UseMetrics(metrics any)

	// Connect establishes a connection to FileSystem and registers metrics using the provided configuration when the client was Created.
	Connect()
}

type Health 

type Health struct {
	Status  string         `json:"status,omitempty"`
	Details map[string]any `json:"details,omitempty"`
}

type Logger 

type Logger interface {
	Debug(args ...any)
	Debugf(format string, args ...any)
	Info(args ...any)
	Infof(format string, args ...any)
	Error(args ...any)
	Errorf(format string, args ...any)
	Warn(args ...any)
	Warnf(format string, args ...any)
}

Logger interface is used by datasource packages to log information about query execution. Developer Notes: Note that it's a reduced version of logging.Logger interface. We are not using that package to ensure that datasource package is not dependent on logging package. That way logging package should be easily able to import datasource package and provide a different "pretty" version for different log types defined here while avoiding the cyclical import issue. Idiomatically, interfaces should be defined by packages who are using it; unlike other languages. Also - accept interfaces, return concrete types.

type RowReader 

type RowReader interface {
	Next() bool
	Scan(any) error
}

Source Files

View all Source files

Directories

Path Synopsis
Package file is a generated GoMock package.
 Package file is a generated GoMock package.
Package pubsub provides a foundation for implementing pub/sub clients for various message brokers such as google pub-sub, kafka and MQTT.
 Package pubsub provides a foundation for implementing pub/sub clients for various message brokers such as google pub-sub, kafka and MQTT.
google
Package google provides a client for interacting with Google Cloud Pub/Sub.This package facilitates interaction with Google Cloud Pub/Sub, allowing publishing and subscribing to topics, managing subscriptions, and handling messages.
 Package google provides a client for interacting with Google Cloud Pub/Sub.This package facilitates interaction with Google Cloud Pub/Sub, allowing publishing and subscribing to topics, managing subscriptions, and handling messages.
kafka
Package kafka provides a client for interacting with Apache Kafka message queues.This package facilitates interaction with Apache Kafka, allowing publishing and subscribing to topics, managing consumer groups, and handling messages.
 Package kafka provides a client for interacting with Apache Kafka message queues.This package facilitates interaction with Apache Kafka, allowing publishing and subscribing to topics, managing consumer groups, and handling messages.
mqtt
Package mqtt provides a client for interacting with MQTT message brokers.This package facilitates interaction with MQTT brokers, allowing publishing and subscribing to topics, managing subscriptions, and handling messages.
 Package mqtt provides a client for interacting with MQTT message brokers.This package facilitates interaction with MQTT brokers, allowing publishing and subscribing to topics, managing subscriptions, and handling messages.
Package redis provides a client for interacting with Redis key-value stores.This package allows creating and managing Redis clients, executing Redis commands, and handling connections to Redis databases.
 Package redis provides a client for interacting with Redis key-value stores.This package allows creating and managing Redis clients, executing Redis commands, and handling connections to Redis databases.
sql
Package sql provides functionalities to interact with SQL databases using the database/sql package.This package includes a wrapper around sql.DB and sql.Tx to provide additional features such as query logging, metrics recording, and error handling.
 Package sql provides functionalities to interact with SQL databases using the database/sql package.This package includes a wrapper around sql.DB and sql.Tx to provide additional features such as query logging, metrics recording, and error handling.
qb
Package qb provides SQL query builder utilities for dynamic WHERE/ORDER/GROUP/LIMIT style queries and bulk insert/update/delete statements.
 Package qb provides SQL query builder utilities for dynamic WHERE/ORDER/GROUP/LIMIT style queries and bulk insert/update/delete statements.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.