README
¶
BanyanDB
BanyanDB, as an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data. It's designed to handle observability data generated by observability platform and APM system, like Apache SkyWalking etc.
Introduction
BanyanDB, as an observability database, aims to ingest, analyze and store Metrics, Tracing, and Logging data. It's designed to handle observability data generated by Apache SkyWalking. Before BanyanDB emerges, the Databases that SkyWalking adopted are not ideal for the APM data model, especially for saving tracing and logging data. Consequently, There’s room to improve the performance and resource usage based on the nature of SkyWalking data patterns.
The database research community usually uses RUM Conjecture to describe how a database access data. BanyanDB combines several access methods to build a comprehensive APM database to balance read cost, update cost, and memory overhead.
Contact us
- Submit an issue by selecting the BanyanDB component.
- Mail list: dev@skywalking.apache.org. Mail to dev-subscribe@skywalking.apache.org, follow the reply to subscribe the mail list.
- Send
Request to join SkyWalking slackmail to the mail list(dev@skywalking.apache.org), we will invite you in. - For Chinese speaker, send
[CN] Request to join SkyWalking slackmail to the mail list(dev@skywalking.apache.org), we will invite you in. - X (Twitter): @BanyanDB and @ASFSkyWalking
Documentation
Contributing
For developers who want to contribute to this project, see the Contribution Guide.
License
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
common
Package common defines reusable objects for APIs.
|
Package common defines reusable objects for APIs. |
|
data
Package data contains data transmission topics.
|
Package data contains data transmission topics. |
|
proto/banyandb
Package apiversion contains the version of the API.
|
Package apiversion contains the version of the API. |
|
validate
Package validate provides functions to validate the provided objects.
|
Package validate provides functions to validate the provided objects. |
|
Package banyand implements a executable database server.
|
Package banyand implements a executable database server. |
|
backup
Package backup provides the backup command-line tool.
|
Package backup provides the backup command-line tool. |
|
backup/lifecycle
Package lifecycle provides the lifecycle migration service.
|
Package lifecycle provides the lifecycle migration service. |
|
backup/snapshot
Package snapshot provides the snapshot backup and restore functionality.
|
Package snapshot provides the snapshot backup and restore functionality. |
|
cmd/backup
command
Package main provides main entry for the backup command-line tool.
|
Package main provides main entry for the backup command-line tool. |
|
cmd/dump
command
Package main provides a command-line tool to dump BanyanDB data.
|
Package main provides a command-line tool to dump BanyanDB data. |
|
cmd/lifecycle
command
Package main provides main entry for the backup command-line tool.
|
Package main provides main entry for the backup command-line tool. |
|
cmd/restore
command
Package main provides main entry for the restore command-line tool.
|
Package main provides main entry for the restore command-line tool. |
|
cmd/server
command
Package main implements the executable banyandb server named banyand.
|
Package main implements the executable banyandb server named banyand. |
|
dquery
Package dquery implement the distributed query.
|
Package dquery implement the distributed query. |
|
internal/encoding
Package encoding provides tag value encoding functionality with optimal compression for different data types including int64, float64, and other types using dictionary encoding with fallback to plain encoding with zstd compression.
|
Package encoding provides tag value encoding functionality with optimal compression for different data types including int64, float64, and other types using dictionary encoding with fallback to plain encoding with zstd compression. |
|
internal/sidx
Package sidx provides block structure and operations for organizing elements within parts for efficient storage and retrieval based on user-provided int64 keys.
|
Package sidx provides block structure and operations for organizing elements within parts for efficient storage and retrieval based on user-provided int64 keys. |
|
internal/storage
Package storage implements a time-series-based storage engine.
|
Package storage implements a time-series-based storage engine. |
|
internal/test
Package test provides mock implementations for testing.
|
Package test provides mock implementations for testing. |
|
internal/wqueue
Package wqueue provides a write queue implementation for managing data shards.
|
Package wqueue provides a write queue implementation for managing data shards. |
|
liaison/grpc
Package grpc implements the gRPC services defined by APIs.
|
Package grpc implements the gRPC services defined by APIs. |
|
liaison/http
Package http implements the gRPC gateway.
|
Package http implements the gRPC gateway. |
|
liaison/pkg/auth
Package auth provides configuration management and validation logic for authentication.
|
Package auth provides configuration management and validation logic for authentication. |
|
measure
Package measure implements a time-series-based storage which is consists of a sequence of data points.
|
Package measure implements a time-series-based storage which is consists of a sequence of data points. |
|
metadata
Package metadata implements a Raft-based distributed metadata storage system.
|
Package metadata implements a Raft-based distributed metadata storage system. |
|
metadata/embeddedetcd
Package embeddedetcd implements an embedded etcd server.
|
Package embeddedetcd implements an embedded etcd server. |
|
metadata/embeddedserver
Package embeddedserver implements an embedded meta server.
|
Package embeddedserver implements an embedded meta server. |
|
metadata/schema
Package schema implements CRUD schema.
|
Package schema implements CRUD schema. |
|
observability
Package observability provides metrics, profiling, and etc.
|
Package observability provides metrics, profiling, and etc. |
|
property
Package property provides the property service interface.
|
Package property provides the property service interface. |
|
property/gossip
Package gossip implements the gossip protocol for BanyanDB property repair.
|
Package gossip implements the gossip protocol for BanyanDB property repair. |
|
protector
Package protector provides a set of protectors that stop the query services when the resource usage exceeds the limit.
|
Package protector provides a set of protectors that stop the query services when the resource usage exceeds the limit. |
|
query
Package query implement the query module for liaison and other modules to retrieve data.
|
Package query implement the query module for liaison and other modules to retrieve data. |
|
queue
Package queue implements the data transmission queue.
|
Package queue implements the data transmission queue. |
|
queue/pub
Package pub implements the queue client.
|
Package pub implements the queue client. |
|
queue/sub
Package sub implements the queue server.
|
Package sub implements the queue server. |
|
stream
Package stream implements a time-series-based storage which is consists of a sequence of element.
|
Package stream implements a time-series-based storage which is consists of a sequence of element. |
|
trace
Package trace implements a trace-based storage which consists of trace data.
|
Package trace implements a trace-based storage which consists of trace data. |
|
bydbctl
|
|
|
cmd/cli
command
Package main provides main entry for the command-line toolkit, i.e.
|
Package main provides main entry for the command-line toolkit, i.e. |
|
internal/cmd
Package cmd is an internal package defining cli commands for bydbctl.
|
Package cmd is an internal package defining cli commands for bydbctl. |
|
pkg/file
Package file provides utils to handle files.
|
Package file provides utils to handle files. |
|
pkg
|
|
|
accesslog
Package accesslog provides access log for banyandb.
|
Package accesslog provides access log for banyandb. |
|
auth
Package auth contains authentication helper functions.
|
Package auth contains authentication helper functions. |
|
bus
Package bus implements a message bus which is a common data model and a messaging infrastructure to allow different modules to communicate locally or remotely.
|
Package bus implements a message bus which is a common data model and a messaging infrastructure to allow different modules to communicate locally or remotely. |
|
bydbql
Package bydbql provides BanyanDB Query Language (BydbQL) parsing and translation capabilities.
|
Package bydbql provides BanyanDB Query Language (BydbQL) parsing and translation capabilities. |
|
bytes
Package bytes provides utilities for operating bytes.
|
Package bytes provides utilities for operating bytes. |
|
cgroups
Package cgroups provides a way to get the number of CPUs.
|
Package cgroups provides a way to get the number of CPUs. |
|
cmdsetup
Package cmdsetup implements a real env in which to run tests.
|
Package cmdsetup implements a real env in which to run tests. |
|
compress/zstd
Package zstd provides ZSTD compression and decompression.
|
Package zstd provides ZSTD compression and decompression. |
|
config
Package config implements a configuration system which could load configuration from flags and env vars.
|
Package config implements a configuration system which could load configuration from flags and env vars. |
|
convert
Package convert implements conversions to and from data.
|
Package convert implements conversions to and from data. |
|
encoding
Package encoding implements encoding/decoding data points.
|
Package encoding implements encoding/decoding data points. |
|
filter
Package filter defines and implements data structures and interfaces for skipping index.
|
Package filter defines and implements data structures and interfaces for skipping index. |
|
flow
Package flow implements a streaming calculation framework.
|
Package flow implements a streaming calculation framework. |
|
flow/streaming
Package streaming implement the flow framework to provide the sliding window, top-n aggregation, and etc.
|
Package streaming implement the flow framework to provide the sliding window, top-n aggregation, and etc. |
|
flow/streaming/sources
Package sources implements data sources to sink data into the flow framework.
|
Package sources implements data sources to sink data into the flow framework. |
|
fs
Package fs (file system) is an independent component to operate file and directory.
|
Package fs (file system) is an independent component to operate file and directory. |
|
fs/remote
Package remote provides an interface for interacting with various remote file systems such as S3, GCS, and distributed file systems for backup/restore operations.
|
Package remote provides an interface for interacting with various remote file systems such as S3, GCS, and distributed file systems for backup/restore operations. |
|
fs/remote/aws
Package aws provides an AWS S3 file system implementation.
|
Package aws provides an AWS S3 file system implementation. |
|
fs/remote/azure
Package azure provides an Azure Blob Storage implementation of the remote.FS interface.
|
Package azure provides an Azure Blob Storage implementation of the remote.FS interface. |
|
fs/remote/checksum
Package checksum provides functions for computing checksums algorithms and verifying.
|
Package checksum provides functions for computing checksums algorithms and verifying. |
|
fs/remote/config
Package config provides the configuration for the file system.
|
Package config provides the configuration for the file system. |
|
fs/remote/gcp
Package gcp provides the GCS file system.
|
Package gcp provides the GCS file system. |
|
fs/remote/local
Package local provides a local file system implementation.
|
Package local provides a local file system implementation. |
|
grpchelper
Package grpchelper implements helpers to access gRPC services.
|
Package grpchelper implements helpers to access gRPC services. |
|
healthcheck
Package healthcheck provides a health check client.
|
Package healthcheck provides a health check client. |
|
host
Package host provides information about the host.
|
Package host provides information about the host. |
|
index
Package index implements the index system for searching data.
|
Package index implements the index system for searching data. |
|
index/analyzer
Package analyzer provides analyzers for indexing and searching.
|
Package analyzer provides analyzers for indexing and searching. |
|
index/inverted
Package inverted implements an inverted index repository.
|
Package inverted implements an inverted index repository. |
|
index/posting
Package posting implements a posting list contains a list of document ids.
|
Package posting implements a posting list contains a list of document ids. |
|
index/posting/roaring
Package roaring implements the posting list by a roaring bitmap.
|
Package roaring implements the posting list by a roaring bitmap. |
|
index/testcases
Package testcases implements common helpers for testing inverted and lsm indices.
|
Package testcases implements common helpers for testing inverted and lsm indices. |
|
iter
Package iter implement a generic Iterator.
|
Package iter implement a generic Iterator. |
|
iter/sort
Package sort provides a generic iterator that merges multiple sorted iterators.
|
Package sort provides a generic iterator that merges multiple sorted iterators. |
|
logger
Package logger implements a logging system with a module tag.
|
Package logger implements a logging system with a module tag. |
|
meter
Package meter provides a simple meter system for metrics.
|
Package meter provides a simple meter system for metrics. |
|
meter/native
Package native provides a simple meter system for metrics.
|
Package native provides a simple meter system for metrics. |
|
meter/prom
Package prom provides a prometheus implementation for the meter system.
|
Package prom provides a prometheus implementation for the meter system. |
|
node
Package node provides node selector for liaison.
|
Package node provides node selector for liaison. |
|
partition
Package partition implements a location system to find a shard or index rule.
|
Package partition implements a location system to find a shard or index rule. |
|
pb/v1
Package v1 implements helpers to access data defined by API v1.
|
Package v1 implements helpers to access data defined by API v1. |
|
pool
Package pool provides a pool for reusing objects.
|
Package pool provides a pool for reusing objects. |
|
query
Package query provides the query common interfaces and utilities.
|
Package query provides the query common interfaces and utilities. |
|
query/aggregation
Package aggregation implements aggregation functions to statistic a range of values.
|
Package aggregation implements aggregation functions to statistic a range of values. |
|
query/executor
Package executor defines the specifications accessing underlying data repositories.
|
Package executor defines the specifications accessing underlying data repositories. |
|
query/logical
Package logical implements the executable operations.
|
Package logical implements the executable operations. |
|
query/logical/measure
Package measure implements execution operations for querying measure data.
|
Package measure implements execution operations for querying measure data. |
|
query/logical/stream
Package stream implements execution operations for querying stream data.
|
Package stream implements execution operations for querying stream data. |
|
query/logical/trace
Package trace implements execution operations for querying trace data.
|
Package trace implements execution operations for querying trace data. |
|
query/model
Package model defines the structures and interfaces for query options and results.
|
Package model defines the structures and interfaces for query options and results. |
|
run
Package run implements a lifecycle framework to control modules.
|
Package run implements a lifecycle framework to control modules. |
|
schema
Package schema implements a framework to sync schema info from the metadata repository.
|
Package schema implements a framework to sync schema info from the metadata repository. |
|
signal
Package signal implements a handler to listen to system signals.
|
Package signal implements a handler to listen to system signals. |
|
test
Package test implements helpers for testing.
|
Package test implements helpers for testing. |
|
test/flags
Package flags implements injected flags to run tests.
|
Package flags implements injected flags to run tests. |
|
test/flow
Package flow implements mockers for testing flow.
|
Package flow implements mockers for testing flow. |
|
test/gmatcher
Package gmatcher provides custom Gomega matchers.
|
Package gmatcher provides custom Gomega matchers. |
|
test/helpers
Package helpers implements several helpers for testing.
|
Package helpers implements several helpers for testing. |
|
test/measure
Package measure implements helpers to load schemas for testing.
|
Package measure implements helpers to load schemas for testing. |
|
test/metric
Package metric provides functions for collecting and analyzing metrics for stress testing.
|
Package metric provides functions for collecting and analyzing metrics for stress testing. |
|
test/property
Package property implements helpers to load schemas for testing.
|
Package property implements helpers to load schemas for testing. |
|
test/query
Package query provides functions for analyzing and collecting metrics.
|
Package query provides functions for analyzing and collecting metrics. |
|
test/setup
Package setup implements a real env in which to run tests.
|
Package setup implements a real env in which to run tests. |
|
test/stream
Package stream implements helpers to load schemas for testing.
|
Package stream implements helpers to load schemas for testing. |
|
test/trace
Package trace implements helpers to load schemas for testing.
|
Package trace implements helpers to load schemas for testing. |
|
timestamp
Package timestamp implements several component related to time.
|
Package timestamp implements several component related to time. |
|
tls
Package tls provides common TLS utilities for HTTP and gRPC servers.
|
Package tls provides common TLS utilities for HTTP and gRPC servers. |
|
version
Package version can be used to implement embedding versioning details from git branches and tags into the binary importing this package.
|
Package version can be used to implement embedding versioning details from git branches and tags into the binary importing this package. |
|
watcher
Package watcher provides a watcher to watch the epoch.
|
Package watcher provides a watcher to watch the epoch. |
|
test
|
|
|
cases
Package cases provides some tools to access test data.
|
Package cases provides some tools to access test data. |
|
cases/backup
Package backup_test provides the test cases for the backup command-line tool.
|
Package backup_test provides the test cases for the backup command-line tool. |
|
cases/lifecycle
Package lifecycle_test is the test cases for the lifecycle package.
|
Package lifecycle_test is the test cases for the lifecycle package. |
|
cases/measure
Package measure_test contains integration test cases of the measure.
|
Package measure_test contains integration test cases of the measure. |
|
cases/measure/data
Package data contains integration test cases of the measure.
|
Package data contains integration test cases of the measure. |
|
cases/property
Package property includes test cases for property queries.
|
Package property includes test cases for property queries. |
|
cases/property/data
Package data includes utilities for property test data management and verification.
|
Package data includes utilities for property test data management and verification. |
|
cases/stream
Package stream_test contains integration test cases of the stream.
|
Package stream_test contains integration test cases of the stream. |
|
cases/stream/data
Package data contains integration test cases of the stream.
|
Package data contains integration test cases of the stream. |
|
cases/topn
Package topn_test contains integration test cases of the TopN.
|
Package topn_test contains integration test cases of the TopN. |
|
cases/topn/data
Package data contains integration test cases of the topN.
|
Package data contains integration test cases of the topN. |
|
cases/trace
Package trace_test contains integration test cases of the trace.
|
Package trace_test contains integration test cases of the trace. |
|
cases/trace/data
Package data is used to test the trace service.
|
Package data is used to test the trace service. |
|
integration/distributed/backup
Package backup and recovery tests for different storage types in distributed mode.
|
Package backup and recovery tests for different storage types in distributed mode. |
|
integration/distributed/sync_retry
Package integration_sync_retry_test provides test utilities for sync retry integration tests.
|
Package integration_sync_retry_test provides test utilities for sync retry integration tests. |
|
integration/dockertesthelper
Package dockertesthelper provides utilities and constants for managing Docker containers.
|
Package dockertesthelper provides utilities and constants for managing Docker containers. |
|
integration/standalone
Package standalone is a standalone integration test suite.
|
Package standalone is a standalone integration test suite. |
|
integration/standalone/backup
Package backup Conduct backup and recovery tests for different storage types in standalone mode.
|
Package backup Conduct backup and recovery tests for different storage types in standalone mode. |
|
property_repair
Package propertyrepair package provides utilities for property repair performance testing in BanyanDB.
|
Package propertyrepair package provides utilities for property repair performance testing in BanyanDB. |
|
stress/trace-streaming
Package tracestreaming provides stress testing tools for trace streaming performance.
|
Package tracestreaming provides stress testing tools for trace streaming performance. |
Click to show internal directories.
Click to hide internal directories.