README
¶
An orchestration engine for background tasks, AI agents, and durable workflows
What is Hatchet?
Hatchet is a platform for orchestrating background tasks, AI agents, and durable workflows at scale. It supports applications written in Python, TypeScript, Go and Ruby, and can be used as a service through Hatchet Cloud or self-hosting. Hatchet provides a full platform for queuing, automatic retries, durability, real-time monitoring, alerting, and logging.
Get started quickly
The fastest way to get started with Hatchet is signing up for Hatchet Cloud to try it out! We recommend this even if you plan on self-hosting, so you can have a look at what a fully-deployed Hatchet platform looks like.
To run Hatchet locally, the fastest path for setup is to install the Hatchet CLI (on MacOS, Linux or WSL) - note that this requires Docker installed locally to work:
curl -fsSL https://install.hatchet.run/install.sh | bash
hatchet --version
hatchet server start
To view full documentation for self-hosting and using cloud, have a look at the docs.
When should I use Hatchet?
You can use Hatchet for running background tasks, AI agents, or other types of long-running workflows. It is designed to be a feature-complete solution for systems where correctness, reliability, horizontal scalability, and observability are essential. From a technical perspective, it differs from other solutions in that it uses Postgres as a durability layer for both the task runtime and the observability system, making it particularly easy to self-host.
For some end-to-end examples of workflows you can build with Hatchet, check out our cookbooks.
Hatchet Features
Background Tasks
- Background tasks: Hatchet supports one-off background tasks defined as simple functions. It supports both fire-and-forget and fire-and-wait tasks with subscriptions
- Retries: flexible and configurable retry policies for tasks, with optional exponential backoff
- Cron jobs and scheduled runs for scheduling tasks at some point in the future
- Task routing based on strict conditions, like worker labels, or more complex, weighted scheduling rules using worker affinity
- Event-based triggering and listeners to build event-driven, highly distributed systems
- Webhook-based triggering for easily triggering Hatchet tasks from upstream data sources
Task orchestration and workflows
- Durable tasks for building fault-tolerant, long-running workflows which can easily recover from failure
- DAGs (directed acyclic graphs) for building data pipelines and simple workflows. See our guide on choosing between durable tasks and DAGs
- Complex pause/resume conditions using durable sleep, event waits, or a combination of both
Scale
- Priority so that critical tasks can run before tasks which aren't latency sensitive, like backfill jobs
- Rate limiting to deal with third-party APIs, or even to enforce per-user rate limits using dynamic rate limits
- Fair scheduling using Hatchet's concurrency policies, which can set a concurrency limit for tasks based on dynamic keys
- Worker slots for ensuring that workers cannot take on more work than they can handle
Monitoring, observability, and management
- Real-time web UI with alerting, monitoring, and logging
- OpenTelemetry (using Hatchet's built-in collector or external destinations)
- Prometheus metrics
- Multi-tenant by default, so a single Hatchet instance can support multiple teams
- Users and roles
Hatchet Cloud features
- Autoscaling and pay-as-you-go plans
- Multi-region deployments
- SSO
- Improved performance for monitoring, logging, and observability
Documentation
The most up-to-date documentation can be found at https://docs.hatchet.run.
Community & Support
- Discord - best for getting in touch with the maintainers and hanging with the community
- Github Issues - used for filing bug reports
- Github Discussions - used for starting in-depth technical discussions that are suited for asynchronous communication
- Email - best for getting Hatchet Cloud support and for help with billing, data deletion, etc.
Hatchet vs...
Hatchet vs Durable Execution Platforms (Temporal, DBOS)
Hatchet's durable tasks feature is a drop-in replacement for Temporal or DBOS workflows. You also get:
- End-to-end observability of durable tasks using OpenTelemetry, monitoring and logging
- Features built for running workflows at scale, such as rate limiting, complex routing, and worker-level slot control
- Multi-tenancy, users and roles supported out of the box
In addition to making durable execution easier to use, Hatchet can also be used as a general-purpose queue, a DAG-based orchestrator, a durable execution engine, or all three, allowing teams to centralize their async and background processing in a single platform.
Hatchet vs Task Queues (Celery, BullMQ)
Traditional task queues like BullMQ and Celery trade off durability for throughput. Tasks persist on the broker (typically Redis or RabbitMQ) while the task is executing, but are not persisted afterwards. This makes it difficult to build complex workflows, as there is no persistent intermediate state. It also makes it difficult to recover and replay tasks which failed and were removed from the queue, resulting in custom admin tooling to work with these libraries at scale.
On the other hand, Hatchet is a durable task queue, meaning it persists the history of all executions (up to a defined retention period), which allows for easy monitoring, debugging and durable task features. Hatchet's durability features add some overhead: while Hatchet has been load-tested up to 10k tasks/second, it consumes more resources than a system built on Redis or RabbitMQ, which can reach much higher throughput.
Hatchet vs DAG-based platforms (Airflow, Prefect, Dagster)
These tools are usually built with data engineers in mind, and aren’t designed to run as part of a high-volume application. They’re usually higher latency and higher cost, with their primary selling point being integrations with common datastores and connectors.
When to use Hatchet: when you'd like to use a DAG-based framework, write your own integrations and functions, and require higher throughput (>100/s)
When to use other DAG-based platforms: when you'd like to use other data stores and connectors that work out of the box
Issues
Please submit any bugs that you encounter via GitHub issues.
I'd Like to Contribute
Please let us know what you're interested in working on in the #contributing channel on Discord. This will help us shape the direction of the project and will make collaboration much easier!
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
v1/server/oas/gen
Package gen provides primitives to interact with the openapi HTTP API.
|
Package gen provides primitives to interact with the openapi HTTP API. |
|
cmd
|
|
|
hatchet-admin
command
|
|
|
hatchet-api
command
|
|
|
hatchet-cli
command
|
|
|
hatchet-cli/cli/internal/mcp
Package mcp implements the local `hatchet mcp` stdio server that lets AI coding agents verify their work against a running Hatchet deployment, plus the grant store that controls which CLI profiles the server may use.
|
Package mcp implements the local `hatchet mcp` stdio server that lets AI coding agents verify their work against a running Hatchet deployment, plus the grant store that controls which CLI profiles the server may use. |
|
hatchet-engine
command
|
|
|
hatchet-lite
command
|
|
|
hatchet-loadtest
command
|
|
|
hatchet-loadtest/go
command
Command hatchet-loadtest-go-worker is a basic, standalone Go SDK worker you can run alongside `cmd/hatchet-loadtest --externalWorker`.
|
Command hatchet-loadtest-go-worker is a basic, standalone Go SDK worker you can run alongside `cmd/hatchet-loadtest --externalWorker`. |
|
hatchet-migrate
command
|
|
|
hatchet-staticfileserver
command
|
|
|
examples
|
|
|
go/affinity-workers
command
|
|
|
go/batch_assign
command
|
|
|
go/bulk-operations
command
|
|
|
go/bulk-operations/cancel
command
|
|
|
go/cancellations
command
|
|
|
go/child-workflows
command
|
|
|
go/concurrency
command
|
|
|
go/conditions
command
|
|
|
go/cron
command
|
|
|
go/dag
command
|
|
|
go/durable
command
|
|
|
go/durable/event
command
|
|
|
go/durable/eviction
command
|
|
|
go/durable/eviction/emit
command
|
|
|
go/durable/eviction/trigger
command
|
|
|
go/durable/sleep
command
|
|
|
go/events
command
|
|
|
go/guides/ai-agents
command
|
|
|
go/guides/batch-processing
command
|
|
|
go/guides/document-processing
command
|
|
|
go/guides/evaluator-optimizer
command
|
|
|
go/guides/event-driven
command
|
|
|
go/guides/human-in-the-loop
command
|
|
|
go/guides/llm-pipelines
command
|
|
|
go/guides/multi-agent
command
|
|
|
go/guides/parallelization
command
|
|
|
go/guides/rag-and-indexing
command
|
|
|
go/guides/routing
command
|
|
|
go/guides/scheduled-jobs
command
|
|
|
go/guides/streaming
command
|
|
|
go/guides/web-scraping
command
|
|
|
go/guides/webhook-processing
command
|
|
|
go/idempotency
command
|
|
|
go/idempotency/trigger
command
|
|
|
go/idempotency/worker
command
|
|
|
go/logs_test
command
|
|
|
go/manual-slot-release
command
|
|
|
go/migration-guides
Package migrationguides holds the Go code rendered into the migration guides.
|
Package migrationguides holds the Go code rendered into the migration guides. |
|
go/on-event
command
|
|
|
go/on-failure
command
|
|
|
go/opentelemetry_instrumentation
command
|
|
|
go/panic-handler
command
|
|
|
go/priority
command
|
|
|
go/rate-limiting
command
|
|
|
go/retries
command
|
|
|
go/retries-concurrency
command
|
|
|
go/run-with-metadata
command
|
|
|
go/runtime-affinity
command
|
|
|
go/sdk-migration
command
|
|
|
go/setup
command
|
|
|
go/simple
command
|
|
|
go/slot-cost
command
|
|
|
go/streaming
command
|
|
|
go/streaming/consumer
command
|
|
|
go/streaming/server
command
|
|
|
go/streaming/worker
command
|
|
|
go/stubs
command
|
|
|
go/timeouts
command
|
|
|
go/webhooks
command
|
|
|
go/with-workflows
command
|
|
|
oas
command
patch-api-resources patches the generated Api.ts file to attach x-resources metadata from the OpenAPI spec to each API method, enabling callers to check resource scope at runtime:
|
patch-api-resources patches the generated Api.ts file to attach x-resources metadata from the OpenAPI spec to each API method, enabling callers to check resource scope at runtime: |
|
internal
|
|
|
integrations/ingestors/sns
Package sns provides helper functions for verifying and processing Amazon AWS SNS HTTP POST payloads.
|
Package sns provides helper functions for verifying and processing Amazon AWS SNS HTTP POST payloads. |
|
operator/claimer
Package claimer hosts the operators a dispatcher claims.
|
Package claimer hosts the operators a dispatcher claims. |
|
operator/hostinproc
Package hostinproc is the in-process operator host: it implements pkg/operator.Host inside the dispatcher process over internal/services/operatorsvc, the same session logic the OperatorService handlers serve to operators that run outside the engine.
|
Package hostinproc is the in-process operator host: it implements pkg/operator.Host inside the dispatcher process over internal/services/operatorsvc, the same session logic the OperatorService handlers serve to operators that run outside the engine. |
|
services/grpcoperator
Package grpcoperator serves v1.OperatorService, the engine API for operators that run outside the engine process (kind = GRPC in v1_operator).
|
Package grpcoperator serves v1.OperatorService, the engine API for operators that run outside the engine process (kind = GRPC in v1_operator). |
|
services/operatorsvc
Package operatorsvc holds the engine-side logic of an operator session: registering the operator and its worker, opening and closing the session that makes the worker live, applying action set deltas, authorizing an operator and its workers, and opening durable invocations.
|
Package operatorsvc holds the engine-side logic of an operator session: registering the operator and its worker, opening and closing the session that makes the worker live, applying action set deltas, authorizing an operator and its workers, and opening durable invocations. |
|
services/operatorsvc/operatorsvctest
Package operatorsvctest holds in-memory doubles for the stores and the dispatcher an operator session talks to.
|
Package operatorsvctest holds in-memory doubles for the stores and the dispatcher an operator session talks to. |
|
services/shared/rpcstream
Package rpcstream holds helpers shared by the engine's streaming RPC handlers.
|
Package rpcstream holds helpers shared by the engine's streaming RPC handlers. |
|
pkg
|
|
|
authmode/keyset/gen
command
Command gen regenerates the embedded auth-disabled API token (keyset/token.jwt), signed with the committed private keyset.
|
Command gen regenerates the embedded auth-disabled API token (keyset/token.jwt), signed with the committed private keyset. |
|
client
Deprecated: This package is part of the legacy v0 workflow definition system.
|
Deprecated: This package is part of the legacy v0 workflow definition system. |
|
client/cloud/rest
Package rest provides primitives to interact with the openapi HTTP API.
|
Package rest provides primitives to interact with the openapi HTTP API. |
|
client/create
Deprecated: This package is part of the legacy v0 workflow definition system.
|
Deprecated: This package is part of the legacy v0 workflow definition system. |
|
client/operatorclient
Package operatorclient connects an out-of-process operator to the engine's OperatorService: one Connect call registers one worker and returns the Session that streams its assigned actions, keeps its action set up to date and reports task progress.
|
Package operatorclient connects an out-of-process operator to the engine's OperatorService: one Connect call registers one worker and returns the Session that streams its assigned actions, keeps its action set up to date and reports task progress. |
|
client/rest
Package rest provides primitives to interact with the openapi HTTP API.
|
Package rest provides primitives to interact with the openapi HTTP API. |
|
client/streaming
Package streaming holds the machinery every long-lived gRPC stream client in this module shares: a stream whose underlying client is replaced on reconnect, the receive loop that drives it with backoff, and the error classification that decides between reconnecting and stopping.
|
Package streaming holds the machinery every long-lived gRPC stream client in this module shares: a stream whose underlying client is replaced on reconnect, the receive loop that drives it with backoff, and the error classification that decides between reconnecting and stopping. |
|
cmdutils
Adapted from: https://github.com/hatchet-dev/hatchet-v1-archived/blob/3c2c13168afa1af68d4baaf5ed02c9d49c5f0323/cmd/cmdutils/interrupt.go
|
Adapted from: https://github.com/hatchet-dev/hatchet-v1-archived/blob/3c2c13168afa1af68d4baaf5ed02c9d49c5f0323/cmd/cmdutils/interrupt.go |
|
config/cli/profilestore
Package profilestore is the shared implementation of the Hatchet CLI's profile store (~/.hatchet/profiles.yaml by default).
|
Package profilestore is the shared implementation of the Hatchet CLI's profile store (~/.hatchet/profiles.yaml by default). |
|
integrations/metrics/prometheus
Package prometheus registers the engine Prometheus collectors (global and tenant-labeled task metrics, pub/sub) and gates tenant series on entitlement.
|
Package prometheus registers the engine Prometheus collectors (global and tenant-labeled task metrics, pub/sub) and gates tenant series on entitlement. |
|
migratediag
Package migratediag provides redacted DSN summaries and uniform migration phase errors for hatchet-migrate and related migration binaries.
|
Package migratediag provides redacted DSN summaries and uniform migration phase errors for hatchet-migrate and related migration binaries. |
|
operator/hostgrpc
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource.
|
Package hostgrpc is the out-of-process operator host: it implements pkg/operator.Host over pkg/client/operatorclient, speaking OperatorService to the engine with a per-tenant API token from a TokenSource. |
|
operator/operatortest
Package operatortest holds a sample contract operator for tests.
|
Package operatortest holds a sample contract operator for tests. |
|
operator/safeclient
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints.
|
Package safeclient provides an SSRF-hardened HTTP client for delivering outbound requests to user-supplied endpoints. |
|
v1
Deprecated: This package is part of the old generics-based v1 Go SDK.
|
Deprecated: This package is part of the old generics-based v1 Go SDK. |
|
v1/factory
Deprecated: This package is part of the legacy v0 workflow definition system.
|
Deprecated: This package is part of the legacy v0 workflow definition system. |
|
v1/features
Deprecated: This package is part of the legacy v0 workflow definition system.
|
Deprecated: This package is part of the legacy v0 workflow definition system. |
|
v1/task
Deprecated: This package is part of the old generics-based v1 Go SDK.
|
Deprecated: This package is part of the old generics-based v1 Go SDK. |
|
v1/worker
Deprecated: This package is part of the old generics-based v1 Go SDK.
|
Deprecated: This package is part of the old generics-based v1 Go SDK. |
|
v1/workflow
Deprecated: This package is part of the old generics-based v1 Go SDK.
|
Deprecated: This package is part of the old generics-based v1 Go SDK. |
|
worker/condition
Deprecated: This package is part of the legacy v0 workflow definition system.
|
Deprecated: This package is part of the legacy v0 workflow definition system. |
|
sdks
|
|
|
go
Define workflows that can declare tasks and be run, scheduled, and so on.
|
Define workflows that can declare tasks and be run, scheduled, and so on. |
|
go/docs/generator
command
Command generator emits the Go SDK reference docs (fumadocs .mdx pages) from the SDK's exported API and doc comments.
|
Command generator emits the Go SDK reference docs (fumadocs .mdx pages) from the SDK's exported API and doc comments. |
|
go/examples/affinity-workers
command
|
|
|
go/examples/batch_assign
command
|
|
|
go/examples/bulk-operations
command
|
|
|
go/examples/cancellations
command
|
|
|
go/examples/child-workflows
command
|
|
|
go/examples/concurrency
command
|
|
|
go/examples/conditions
command
|
|
|
go/examples/cron
command
|
|
|
go/examples/dag
command
|
|
|
go/examples/durable/event
command
|
|
|
go/examples/durable/eviction
command
|
|
|
go/examples/durable/sleep
command
|
|
|
go/examples/events
command
|
|
|
go/examples/idempotency
command
|
|
|
go/examples/logs_test
command
|
|
|
go/examples/manual-slot-release
command
|
|
|
go/examples/migration-guides
Package migrationguides holds the Go code rendered into the migration guides.
|
Package migrationguides holds the Go code rendered into the migration guides. |
|
go/examples/on-event
command
|
|
|
go/examples/on-failure
command
|
|
|
go/examples/panic-handler
command
|
|
|
go/examples/priority
command
|
|
|
go/examples/rate-limiting
command
|
|
|
go/examples/retries
command
|
|
|
go/examples/retries-concurrency
command
|
|
|
go/examples/run-with-metadata
command
|
|
|
go/examples/runtime-affinity
command
|
|
|
go/examples/sdk-migration
command
|
|
|
go/examples/setup
command
|
|
|
go/examples/simple
command
|
|
|
go/examples/slot-cost
command
|
|
|
go/examples/streaming/consumer
command
|
|
|
go/examples/streaming/server
command
|
|
|
go/examples/streaming/worker
command
|
|
|
go/examples/stubs
command
|
|
|
go/examples/timeouts
command
|
|
|
go/examples/webhooks
command
|
|
|
go/examples/with-workflows
command
|
|
|
go/features
package features provides functionality for interacting with hatchet features.
|
package features provides functionality for interacting with hatchet features. |
|
go/internal
Package internal provides internal functionality for the Hatchet Go SDK
|
Package internal provides internal functionality for the Hatchet Go SDK |
|
go/opentelemetry
Package opentelemetry provides OpenTelemetry instrumentation for the Hatchet Go SDK.
|
Package opentelemetry provides OpenTelemetry instrumentation for the Hatchet Go SDK. |
|
sql
|
|
|
atlas-to-goose
command
|
Click to show internal directories.
Click to hide internal directories.