huatuo

module
v0.0.0-...-b125894 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: Apache-2.0

README

Cube Sandbox Logo

HUATUO 华佗

Kernel-wide Insight, Instant Observability, AutoTracing, Continuous Profiling

GitHub Stars Docker Pulls GitHub Issues Apache 2.0 License PRs Welcome CNCF Landscape

Fast startup Hardware-level observability large scale

中文文档 · Quick Start · Documentation ·


What is HUATUO

HUATUO, named after Hua Tuo, a legendary physician in ancient China, is a cloud-native OS observability project open-sourced by DiDi and incubated under the CCF. It provides kernel-level observability for general-purpose cloud-native computing, AI computing, and bare-metal infrastructure services. By integrating Linux kernel dynamic tracing technologies such as kprobes, tracepoints, ftrace, and eBPF, HUATUO provides kernel-wide visibility through finer-grained metrics, automatic runtime context capture in the kernel, and intelligent tracing. Deployed at scale in DiDi's production environment, HUATUO helps troubleshoot system failures and improves the high availability and performance of cloud-native operating systems.

HUATUO is now listed in the CNCF Landscape.

Key Features

  • Kernel-Wide Insight: Leverages eBPF to keep performance overhead below 1%, delivering full-stack, low-level observability insights into Linux kernel subsystems such as memory management, CPU scheduling, networking, and block I/O.
  • Instant Observability: An event-driven runtime context capture mechanism that instruments kernel slow paths. It automatically triggers on events such as page faults, scheduling delays, generating detailed data for immediate analysis.
  • AutoTracing: Employs automated snapshot retention to resolve performance jitter typical in cloud‑native and AI infrastructure environments, addressing issues such as CPU idle drops, CPU system-time spikes, I/O surges, and load average spikes.
  • Continuous Profiling: A comprehensive and continuous performance profiling of the operating system and applications, covering CPU, Memory, I/O, and Locks. This drives business innovation and plays a key role in Chaos, HA and Stability Engineering.
  • Heterogeneous Computing Hardware: Encompassing core system components such as CPUs, memory, PCIe interconnects, network adapters, and storage, alongside specialized AI accelerators including GPUs and NPUs.
  • Ecosystem Integration: Integrates with mainstream open-source observability stacks such as Prometheus, Grafana, Pyroscope, and Elasticsearch, and automatically associates Kubernetes container labels/annotations. This is achieved through zero-instrumentation, kernel-level eBPF programming, ensuring broad compatibility across hardware platforms and Linux distributions.

Big Picture

Ecosystem

Getting Started

  • Quick Run

    To launch the HUATUO service with Docker:

      $ docker run --privileged --pid=host --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest
    

    To pull metrics from another terminal:

      $ curl -s localhost:19704/metrics
    
  • Quick Setup

    To launch the full stack (Elasticsearch, Prometheus, Grafana, and huatuo) using Docker Compose:

      $ docker compose --project-directory ./build/docker up
    

    Once running, access the monitoring dashboard at http://localhost:3000.

  • NOTE

    Do not deploy images with the latest tag to production environments, as this is a development and testing image. Use a formal release image or binary.

Go Client

The public Go client is in client/. Node client files use the node_ prefix.

package main

import (
	"context"
	"encoding/json"

	nodeapi "github.com/ccfos/huatuo/apis/v1/node"
	"github.com/ccfos/huatuo/client"
)

func updateNodeConfig(ctx context.Context, address client.NodeAddress) error {
	nodeClient, err := client.NewNode(&client.NodeConfig{
		BearerToken: "node-token",
	})
	if err != nil {
		return err
	}
	return nodeClient.UpdateConfig(ctx, address, &nodeapi.UpdateConfigRequest{
		Config: map[string]json.RawMessage{
			"Runtime.CPULimitCores": json.RawMessage("1.5"),
		},
	})
}

Kernel Versions

The project supports kernel version 4.18 and later. The following kernel and OS distribution are primarily tested.

HUATUO Kernel OS
1.0.0 4.18.x CentOS 8.x
2.4.0 4.18.x Rocky Linux 8.10
1.0.0 5.4.x OpenCloudOS V8
1.0.0 5.4.x Ubuntu 20.04.6 LTS
1.0.0 5.10.x Anolis OS 8.10
1.0.0 5.10.x openEuler 22.03 LTS-SP4
2.4.0 5.14.x Rocky Linux 9.8
1.0.0 5.15.x Ubuntu 22.04.5 LTS
2.4.0 6.1.x Debian 12 (bookworm)
1.0.0 6.6.x Anolis OS 23.3
1.0.0 6.6.x OpenCloudOS V9
1.0.0 6.6.x openEuler 24.03 LTS-SP4
1.0.0 6.8.x Ubuntu 24.04.4 LTS
2.4.0 6.12.x Debian 13 (trixie)
1.0.0 6.14.x Fedora 42
2.4.0 6.19.x Fedora Linux 44
2.3.0 7.0.x Ubuntu 26.04 LTS

Documentation

For more information, visit https://docs.huatuo.tech

Co-Building

  • ❇️ We welcome all users, developers, companies, and organizations to use Huatuo, report bugs, submit feature requests, share best practices, and help us build a professional and vibrant open-source community.
  • ❤️ Contributors

Contact Us

  • WeChat Group and Official Account:

License

This project is open source under the Apache License 2.0. The BPF code is licensed under the GPL license.

Directories

Path Synopsis
apis
v1
Package v1 provides primitives to interact with the OpenAPI HTTP API.
Package v1 provides primitives to interact with the OpenAPI HTTP API.
v1/node
Package node provides primitives to interact with the OpenAPI HTTP API.
Package node provides primitives to interact with the OpenAPI HTTP API.
v1/server
Package server provides primitives to interact with the OpenAPI HTTP API.
Package server provides primitives to interact with the OpenAPI HTTP API.
build
bpfabi-tool command
openapi/bundle command
pprof-tool command
Package client provides Go clients for Huatuo HTTP APIs.
Package client provides Go clients for Huatuo HTTP APIs.
cmd
dropwatch command
huatuo-bamai command
iotracing command
Select request readers before opening tracing resources.
Select request readers before opening tracing resources.
irqtracing command
perf command
profiler command
sched-blame command
tcpshark command
tcpshark/retransmit
Package retransmit runs TCP retransmit tracing and optional drop correlation.
Package retransmit runs TCP retransmit tracing and optional drop correlation.
core
metrics
Block device health records events and freeze counters.
Block device health records events and freeze counters.
metrics/ascend/hccn
Package hccn collects NPU network metrics via hccn_tool.
Package hccn collects NPU network metrics via hccn_tool.
metrics/ascend/pcie
Package pcie reads PCIe link capability and status from sysfs.
Package pcie reads PCIe link capability and status from sysfs.
internal
blkdevicehealth
Health commands share time and output bounds so event-triggered evidence cannot add unbounded load to an unhealthy storage path.
Health commands share time and output bounds so event-triggered evidence cannot add unbounded load to an unhealthy storage path.
bpf
document
Package document constructs metadata shared by persisted observations.
Package document constructs metadata shared by persisted observations.
dropwatch
Package dropwatch owns BPF resources for software and hardware drop tracing.
Package dropwatch owns BPF resources for software and hardware drop tracing.
executil
Package executil owns one external command and its Linux process group.
Package executil owns one external command and its Linux process group.
httpserver/auth
Package auth provides framework-independent authentication and authorization.
Package auth provides framework-independent authentication and authorization.
httpserver/pprof
Package pprof provides a standalone HTTP server for Go runtime profiles.
Package pprof provides a standalone HTTP server for Go runtime profiles.
job
log
nodeagent/cloudevents
Package cloudevents manages filtered Node event subscriptions and converts tracing documents into the public CloudEvents envelope.
Package cloudevents manages filtered Node event subscriptions and converts tracing documents into the public CloudEvents envelope.
nodeagent/operation
Package operation owns the Node Agent's in-memory execution lifecycle.
Package operation owns the Node Agent's in-memory execution lifecycle.
nodeagent/profiling
Package profiling adapts typed profiling requests to Node operations.
Package profiling adapts typed profiling requests to Node operations.
nodeagent/tracing
Package tracing implements the Node on-demand tracing service boundary.
Package tracing implements the Node on-demand tracing service boundary.
pcapfilter
Package pcapfilter compiles tcpdump filter expressions and injects them into pre-compiled BPF CollectionSpecs using the pwru stub-splice pattern.
Package pcapfilter compiles tcpdump filter expressions and injects them into pre-compiled BPF CollectionSpecs using the pwru stub-splice pattern.
pod
profiling
Package profiling constructs and persists profiling results.
Package profiling constructs and persists profiling results.
profiling/aggregation
Package aggregation deduplicates profiling samples by complete stack and label identity.
Package aggregation deduplicates profiling samples by complete stack and label identity.
profiling/bpfmap
Package bpfmap provides shared stack-map access and native profiler transfer state helpers.
Package bpfmap provides shared stack-map access and native profiler transfer state helpers.
profiling/flamegraph/tui
Package tui renders flamegraph FrameData in an interactive terminal viewer.
Package tui renders flamegraph FrameData in an interactive terminal viewer.
profiling/formatter
Package formatter defines profiling output contracts.
Package formatter defines profiling output contracts.
profiling/formatter/chrometrace
Package chrometrace writes aggregated stacks as Chrome Trace Event JSON.
Package chrometrace writes aggregated stacks as Chrome Trace Event JSON.
profiling/formatter/collapsed
Package collapsed writes folded-stack format (Brendan Gregg): "frame;frame count\n".
Package collapsed writes folded-stack format (Brendan Gregg): "frame;frame count\n".
profiling/formatter/factory
Package factory constructs profiling output formats.
Package factory constructs profiling output formats.
profiling/formatter/flamegraph
Package flamegraph writes an interactive SVG flame graph via the JS-enhanced renderer.
Package flamegraph writes an interactive SVG flame graph via the JS-enhanced renderer.
profiling/formatter/speedscope
Package speedscope encodes aggregated profiles as weighted Speedscope samples.
Package speedscope encodes aggregated profiles as weighted Speedscope samples.
profiling/toolpath
Package toolpath owns the installed layout of external profiling tools.
Package toolpath owns the installed layout of external profiling tools.
profiling/window
Package window validates and converts immutable aggregation windows.
Package window validates and converts immutable aggregation windows.
qdisc
Package qdisc reads Linux queuing discipline statistics.
Package qdisc reads Linux queuing discipline statistics.
randomid
Package randomid creates opaque identifiers for internal execution sessions.
Package randomid creates opaque identifiers for internal execution sessions.
storage/driver
Package driver defines the storage abstraction layer: configuration types, query DSL, the Backend interface, and the backend driver registry.
Package driver defines the storage abstraction layer: configuration types, query DSL, the Backend interface, and the backend driver registry.
storage/elasticsearch
Package elasticsearch implements a storage backend compatible with Elasticsearch v7/v8 and OpenSearch using the official go-elasticsearch/v8 SDK.
Package elasticsearch implements a storage backend compatible with Elasticsearch v7/v8 and OpenSearch using the official go-elasticsearch/v8 SDK.
storage/localfile
Package localfile implements a storage backend that appends records to local files with rotation support.
Package localfile implements a storage backend that appends records to local files with rotation support.
storage/sqlite
Package sqlite implements a storage backend that persists records to SQLite using json_extract-based querying.
Package sqlite implements a storage backend that persists records to SQLite using json_extract-based querying.
timeutil
Package timeutil defines the canonical time format used everywhere in the project: storage records, wire protocols, log lines, file names.
Package timeutil defines the canonical time format used everywhere in the project: storage records, wire protocols, log lines, file names.
toolstream
Package toolstream provides typed event dispatch over a Unix-domain-socket transport.
Package toolstream provides typed event dispatch over a Unix-domain-socket transport.
tracing
Package tracing schedules Node tracers and manages their lifecycle.
Package tracing schedules Node tracers and manages their lifecycle.
utils/fileutil
Package fileutil provides utility functions for file operations.
Package fileutil provides utility functions for file operations.
utils/kernaddr
Package kernaddr encodes and decodes 64-bit kernel pointers as hex strings.
Package kernaddr encodes and decodes 64-bit kernel pointers as hex strings.
version
Package version centralizes CLI build/version metadata across cmd/*.
Package version centralizes CLI build/version metadata across cmd/*.
watch
Package watch provides a generic fan-out pub/sub hub.
Package watch provides a generic fan-out pub/sub hub.
pkg
metric
Package metric provides the metrics collection framework used by HuaTuo to expose runtime observability data via Prometheus.
Package metric provides the metrics collection framework used by HuaTuo to expose runtime observability data via Prometheus.
metric/runtime
Package runtime registers process, Go runtime, and application information metrics.
Package runtime registers process, Go runtime, and application information metrics.
profiling/store
Package store defines the persisted profiling result and its storage access.
Package store defines the persisted profiling result and its storage access.
tracing/store
Package store defines tracing documents and their persistence behavior.
Package store defines tracing documents and their persistence behavior.
types
Package types defines the public data types and sentinel errors shared across HuaTuo components and exposed to external consumers.
Package types defines the public data types and sentinel errors shared across HuaTuo components and exposed to external consumers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL