README
¶
xmtpd
⚠️ Experimental: This software is in early development. Expect frequent changes and unresolved issues.
xmtpd (XMTP daemon) is an experimental version of XMTP node software. It is not the node software that currently forms the XMTP network.
After xmtpd meets specific functional requirements, the plan is for it to become the node software that powers the XMTP network.
Some of these requirements include reaching functional parity with the current node software and reliably performing data replication without data loss.
To keep up with and provide feedback about xmtpd development, see the Issues tab in this repo.
[!TIP] Looking for info on how to get started as a registered node operator? See the xmtpd-infrastructure repository, which provides infrastructure-as-code examples and tooling to help node operators deploy and manage xmtpd nodes.
Prerequisites
Quickstart
Install and build dependencies
To install dependencies and build background services needed by the node, run:
dev/up
This starts the following background services in Docker:
-
A Postgres database that is used to store data needed to run the node
-
A Prometheus instance that enables you to view metrics and data about the node, such as resource usage and errors
To stop the background services running in Docker, run:
dev/down
Start the node
To start the xmtpd node, run:
dev/run
Test the node
To run tests against the xmtpd node, run:
dev/test
These tests provide a full suite of unit and integration tests for the xmtpd repo to help ensure and maintain correctness of the code over time and to avoid regressions as the code evolves. You can explore the tests by taking a look at any files with the suffix _test.go.
Monitor the node
To enable metrics you need to start the node with --metrics.enable.
By default, we expose Prometheus metrics on port 8008.
curl http://localhost:8008
About the XMTP testnet
The XMTP testnet environment lives in XMTP Sepolia Chain.
The XMTP testnet nodes operated by Ephemera include:
| DNS name | Location | Public key |
|---|---|---|
| https://grpc.testnet.xmtp.network | US-EAST-2 | 0x03e5442c5d1fe2f02b6b9a1a386383a7766860b40a6079a0223994ffa2ce10512c |
| https://grpc2.testnet.xmtp.network | EU-NORTH-1 | 0x02fc261d43a0153539a4c64c29763cb0e7e377c0eac2910c3d4bedb2235ac70371 |
Deploy a local environment for developing with xmtpd
See Deploy a local environment for developing with xmtpd.
Support a new XMTP testnet node operator
See Support a new XMTP testnet node operator.
Migrate to XMTP node registry V2
xmtpd OpenMetrics catalog
See xmptd OpenMetrics catalog.
Contributing
Please follow the style guide.
Modifying the protobuf schema
Submit and land a PR to https://github.com/xmtp/proto. Then run:
dev/gen/all
Or wait for the nightly automation to finish.
Modify the database schema
Create a new migration by running:
dev/gen/migration {migration-name}
Fill in the migrations in the generated files. If you are unfamiliar with migrations, you may follow this guide. The database is PostgreSQL and the driver is PGX.
Modify database queries
We use sqlc to generate the code for our DB queries. Modify the queries.sql file, and then run:
dev/gen/all
If needed, you may use dev/psql to test more complex SQL syntax. To determine the level of support in sqlc, there is a sqlc playground for experimenting with how the query syntax translates into Go code.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
gateway
command
|
|
|
prune
command
|
|
|
replication
command
|
|
|
xmtpd-cli
command
|
|
|
xmtpd-cli/commands
Package commands implements the CLI commands based on cobra.
|
Package commands implements the CLI commands based on cobra. |
|
xmtpd-cli/options
Package options implements the options for the CLI.
|
Package options implements the options for the CLI. |
|
pkg
|
|
|
api
Package api implements the API server.
|
Package api implements the API server. |
|
api/message
Package message implements the replication API service.
|
Package message implements the replication API service. |
|
api/metadata
Package metadata implements the metadata API service.
|
Package metadata implements the metadata API service. |
|
api/payer
Package payer implements the Payer API service.
|
Package payer implements the Payer API service. |
|
authn
Package authn implements a JWT token factory and verifier.
|
Package authn implements a JWT token factory and verifier. |
|
blockchain
Package blockchain provides infrastructure for interacting with the blockchain.
|
Package blockchain provides infrastructure for interacting with the blockchain. |
|
blockchain/noncemanager
Package noncemanager provides interfaces and implementations for managing transaction nonces in a blockchain environment.
|
Package noncemanager provides interfaces and implementations for managing transaction nonces in a blockchain environment. |
|
blockchain/noncemanager/redis
Package redis provides a Redis-backed implementation of the NonceManager interface.
|
Package redis provides a Redis-backed implementation of the NonceManager interface. |
|
blockchain/noncemanager/sql
Package sql provides a SQL database-backed implementation of the NonceManager interface.
|
Package sql provides a SQL database-backed implementation of the NonceManager interface. |
|
blockchain/oracle
Package oracle provides a blockchain oracle for gas price queries.
|
Package oracle provides a blockchain oracle for gas price queries. |
|
config
Package config implements the configuration options for the xmtpd server.
|
Package config implements the configuration options for the xmtpd server. |
|
constants
Package constants implements the constants for the xmtpd server.
|
Package constants implements the constants for the xmtpd server. |
|
currency
Package currency implements the type PicoDollar and MicroDollar, and its conversions.
|
Package currency implements the type PicoDollar and MicroDollar, and its conversions. |
|
db
Package db provides DB access and advisory-lock helpers for coordinating HA workers.
|
Package db provides DB access and advisory-lock helpers for coordinating HA workers. |
|
debug
Package debug implements the pprof server.
|
Package debug implements the pprof server. |
|
deserializer
Package deserializer implements the deserializer for the MLS messages.
|
Package deserializer implements the deserializer for the MLS messages. |
|
envelopes
Package envelopes implements methods to create and manipulate different types of envelopes.
|
Package envelopes implements methods to create and manipulate different types of envelopes. |
|
fees
Package fees implements the congestion fee calculation.
|
Package fees implements the congestion fee calculation. |
|
gateway
Package gateway implements the gateway service.
|
Package gateway implements the gateway service. |
|
gateway/examples/basic
command
|
|
|
gateway/examples/jwt
command
|
|
|
gateway/examples/rateLimiter
command
|
|
|
indexer
Package indexer implements the Indexer.
|
Package indexer implements the Indexer. |
|
indexer/app_chain
Package appchain implements the AppChain Indexer.
|
Package appchain implements the AppChain Indexer. |
|
indexer/app_chain/contracts
Package contracts implements the GroupMessageBroadcaster and IdentityUpdateBroadcaster contracts.
|
Package contracts implements the GroupMessageBroadcaster and IdentityUpdateBroadcaster contracts. |
|
indexer/common
Package common contains interfaces for the indexer.
|
Package common contains interfaces for the indexer. |
|
indexer/rpc_streamer
Package rpcstreamer implements a log streamer that uses the RPC node to backfill history and then switch to a subscription.
|
Package rpcstreamer implements a log streamer that uses the RPC node to backfill history and then switch to a subscription. |
|
indexer/settlement_chain
Package settlementchain implements the SettlementChain Indexer.
|
Package settlementchain implements the SettlementChain Indexer. |
|
indexer/settlement_chain/contracts
Package contracts implements the payer registry and payer registry manager contracts.
|
Package contracts implements the payer registry and payer registry manager contracts. |
|
integration/builders
Package builders implements a collection of container builders for the integration tests.
|
Package builders implements a collection of container builders for the integration tests. |
|
interceptors/client
Package client implements the authentication interceptors for the client.
|
Package client implements the authentication interceptors for the client. |
|
interceptors/server
Package server implements the server authentication interceptors.
|
Package server implements the server authentication interceptors. |
|
ledger
Package ledger implements the ledger for payer balances.
|
Package ledger implements the ledger for payer balances. |
|
merkle
Package merkle implements the Merkle tree for the payer report.
|
Package merkle implements the Merkle tree for the payer report. |
|
metrics
Package metrics implements the Prometheusmetrics for the XMTPD service.
|
Package metrics implements the Prometheusmetrics for the XMTPD service. |
|
metrics/docs
command
|
|
|
migrator
Package migrator implements a service that migrates data from a source database to a destination database.
|
Package migrator implements a service that migrates data from a source database to a destination database. |
|
misbehavior
Package misbehavior implements the misbehavior reports service.
|
Package misbehavior implements the misbehavior reports service. |
|
mlsvalidate
Package mlsvalidate implements the MLS validation service interface.
|
Package mlsvalidate implements the MLS validation service interface. |
|
payerreport
Package payerreport implements the payer report generator and verifier interfaces.
|
Package payerreport implements the payer report generator and verifier interfaces. |
|
payerreport/workers
Package workers implements the attestation worker for the payer report.
|
Package workers implements the attestation worker for the payer report. |
|
prune
Package prune implements the DB prune executor.
|
Package prune implements the DB prune executor. |
|
ratelimiter
Package ratelimiter provides a rate limiter interface and implementation.
|
Package ratelimiter provides a rate limiter interface and implementation. |
|
registrant
Package registrant implements the registrant for the node.
|
Package registrant implements the registrant for the node. |
|
registry
Package registry implements the node registry interface.
|
Package registry implements the node registry interface. |
|
server
Package server implements the base server that manages all the other services.
|
Package server implements the base server that manages all the other services. |
|
stress
Package stress implements the stress test for the identity updates.
|
Package stress implements the stress test for the identity updates. |
|
sync
Package sync implements the sync server.
|
Package sync implements the sync server. |
|
testutils
Package testutils implements test utils for many different packages.
|
Package testutils implements test utils for many different packages. |
|
testutils/anvil
Package anvil implements the anvil test utils.
|
Package anvil implements the anvil test utils. |
|
testutils/api
Package apiutils implements the api test utils.
|
Package apiutils implements the api test utils. |
|
testutils/envelopes
Package envelopes implements the envelopes test utils.
|
Package envelopes implements the envelopes test utils. |
|
testutils/fees
Package fees implements the fees test utils.
|
Package fees implements the fees test utils. |
|
testutils/flags
Package flags provides control over flags that are set at build time.
|
Package flags provides control over flags that are set at build time. |
|
testutils/network
Package network implements the network test utils.
|
Package network implements the network test utils. |
|
testutils/redis
Package redis implements the redis test utils.
|
Package redis implements the redis test utils. |
|
testutils/registry
Package registry implements the node registry test utils.
|
Package registry implements the node registry test utils. |
|
testutils/server
Package server implements the replication server test utils.
|
Package server implements the replication server test utils. |
|
topic
Package topic implements the topic type.
|
Package topic implements the topic type. |
|
tracing
Package tracing enables [Datadog APM tracing](https://docs.datadoghq.com/tracing/) capabilities, focusing specifically on [Error Tracking](https://docs.datadoghq.com/tracing/error_tracking/)
|
Package tracing enables [Datadog APM tracing](https://docs.datadoghq.com/tracing/) capabilities, focusing specifically on [Error Tracking](https://docs.datadoghq.com/tracing/error_tracking/) |
|
utils
Package utils implements different utilities to be used by production code.
|
Package utils implements different utilities to be used by production code. |
|
utils/retryerrors
Package retryerrors implements the retryable error interface.
|
Package retryerrors implements the retryable error interface. |