webhook

package
v1.17.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 12, 2025 License: Apache-2.0 Imports: 15 Imported by: 0

README

Substreams Webhook Sink Examples

This document provides examples of how to use the webhook sink with the new retry functionality.

Basic Usage

# Basic webhook call with default settings
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name

# With custom start block
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name --start-block 12345

Retry Configuration

The webhook sink now supports configurable retry behavior for handling transient failures:

Default Settings
  • Max Retries: 3 attempts (use -1 for infinite retries)
  • Timeout: 30 seconds per request
  • Max Retry Interval: 30 seconds (exponential backoff cap)
Custom Retry Settings
# Increase retry attempts for unreliable endpoints
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 5

# Shorter timeout for faster failure detection
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-timeout 10s

# Longer maximum retry interval for heavily loaded endpoints
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retry-interval 60s

# Disable retries completely
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 0

# Enable infinite retries (retry until success or context cancellation)
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries -1
Combined Configuration
# Production-ready configuration with aggressive retries
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 5 \
  --webhook-timeout 15s \
  --webhook-max-retry-interval 45s \
  --state-file ./production.cursor

# Production configuration with infinite retries for critical webhooks
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries -1 \
  --webhook-timeout 30s \
  --webhook-max-retry-interval 15s \
  --state-file ./production.cursor

Retry Behavior

What Gets Retried
  • Network errors (connection failures, timeouts)
  • Server errors (5xx HTTP status codes)
  • Temporary service unavailability
What Doesn't Get Retried
  • Client errors (4xx HTTP status codes like 400, 401, 403, 404)
  • Request creation failures (invalid URLs)
  • Context cancellation
Retry Modes
  • Limited retries (default): Retry up to --webhook-max-retries times
  • No retries (--webhook-max-retries 0): Fail immediately on first error
  • Infinite retries (--webhook-max-retries -1): Retry indefinitely until success or context cancellation
Exponential Backoff

The retry mechanism uses exponential backoff with jitter:

  • First retry: ~1 second
  • Second retry: ~2 seconds
  • Third retry: ~4 seconds
  • Maximum interval is capped by --webhook-max-retry-interval
Infinite Retry Behavior

When --webhook-max-retries is set to -1:

  • Webhook calls will retry indefinitely for transient failures
  • Only context cancellation or permanent errors (4xx) will stop retries
  • Useful for critical webhooks that must eventually succeed
  • Consider setting appropriate --webhook-timeout and --webhook-max-retry-interval values
  • Monitor logs for excessive retry patterns that might indicate service issues

State Management

# Custom state file location
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --state-file ./custom/path/state.cursor

# Disable state persistence (start from scratch each time)
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --state-file ""

Error Handling

The webhook sink continues processing even if webhook calls fail after all retries. This ensures that:

  • The substreams processing doesn't get blocked by webhook failures
  • Block processing continues uninterrupted
  • Cursor state is still saved for successful blocks

Monitoring and Logging

The webhook sink provides detailed logging for troubleshooting:

INFO calling webhook block=12345 url=https://api.example.com/webhook
WARN webhook call failed for block 12345: webhook returned server error status 503 for block 12345
INFO calling webhook block=12345 url=https://api.example.com/webhook (retry 1/3)
INFO webhook call successful block=12345 url=https://api.example.com/webhook

Best Practices

  1. Set appropriate timeouts: Use shorter timeouts (5-15s) for real-time processing
  2. Configure retries based on endpoint reliability: More retries for less reliable services
  3. Use infinite retries sparingly: Only for critical webhooks where eventual delivery is essential
  4. Monitor webhook endpoint performance: Adjust settings based on observed behavior
  5. Use state files: Always specify a state file for production deployments
  6. Handle failures gracefully: Implement idempotent webhook handlers that can handle duplicate calls
  7. Context management: When using infinite retries, ensure proper context cancellation for shutdown
Retry Strategy Guidelines

For real-time applications:

--webhook-max-retries 2
--webhook-timeout 5s
--webhook-max-retry-interval 10s

For critical data delivery:

--webhook-max-retries -1
--webhook-timeout 30s
--webhook-max-retry-interval 60s

For testing/development:

--webhook-max-retries 0
--webhook-timeout 10s

Webhook Payload Format

The webhook receives JSON payloads in the following format:

{
  "clock": {
    "timestamp": "2024-02-12T22:23:51.000Z",
    "number": 53448530,
    "id": "f843bc26cea0cbd50b09699546a8a97de6a1727646c17a857c5d8d868fc26142"
  },
  "manifest": {
    "moduleName": "module_name",
    "type": "sf.substreams.ethereum.v1.Events"
  },
  "data": {
    // Your module's output data
  }
}

It is loosely based on the format from https://github.com/pinax-network/substreams-sink-webhook

Payload Structure
  • clock: Contains blockchain timing and identification information
    • timestamp: Block timestamp in RFC3339 format
    • number: Block number
    • id: Block hash/ID
  • manifest: Contains module metadata
    • moduleName: Name of the substreams module that generated this data
    • type: Module output type (automatically strips type.googleapis.com/ prefix)
  • data: Contains the actual output from your substreams module
Example with Real Data
{
  "clock": {
    "timestamp": "2024-02-12T22:23:51.000Z",
    "number": 53448530,
    "id": "f843bc26cea0cbd50b09699546a8a97de6a1727646c17a857c5d8d868fc26142"
  },
  "manifest": {
    "moduleName": "filtered_events",
    "type": "sf.substreams.ethereum.v1.Events"
  },
  "data": {
    "events": [
      {
        "address": "0x1234567890abcdef",
        "topics": ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],
        "data": "0x000000000000000000000000000000000000000000000001158e460913d00000"
      }
    ]
  }
}

"module": "module_name", "block": 12345, "timestamp": "2023-01-01T00:00:00Z", "type": "type.googleapis.com/sf.substreams.ethereum.v1.Events", "payload": { // Your module's output data } }


Make sure your webhook endpoint can handle:
- POST requests with `Content-Type: application/json`
- Potential duplicate calls (implement idempotency)
- Proper HTTP status code responses (2xx for success, 4xx for permanent errors, 5xx for retryable errors)

### Monitoring via Prometheus

* Prometheus metrics are available at `http://localhost:9102` by default. See the --prometheus-addr flag for more details.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var WebhookCallsCounter = sink.Metrics.NewCounter("webhook_calls", "Number of calls made to the webhook")
View Source 
var WebhookSizeBytes = sink.Metrics.NewCounter("webhook_bytes_sent", "Number of bytes sent via webhook")

Functions

This section is empty.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client holds the HTTP client and retry configuration for webhook calls. It implements exponential backoff retry logic for transient failures while avoiding retries for permanent client errors (4xx status codes).

func NewClient 

func NewClient(config Config, logger *zap.Logger) *Client

NewClient creates a new webhook client with retry configuration.

func (*Client) Call 

func (c *Client) Call(ctx context.Context, url string, payload []byte, blockNumber uint64) error

Call makes a webhook call with exponential backoff retry logic. It retries on network errors and server errors (5xx), but not on client errors (4xx). The function respects context cancellation and implements proper error classification.

type Clock 

type Clock struct {
	Timestamp string `json:"timestamp"`
	Number    uint64 `json:"number"`
	ID        string `json:"id"`
}

Clock represents the clock information in the webhook payload

type Config 

type Config struct {
	Timeout     time.Duration // HTTP request timeout for individual calls
	MaxRetries  int           // Maximum number of retry attempts for transient failures (-1 for infinite retries)
	MaxInterval time.Duration // Maximum interval between retries (exponential backoff cap)
}

Config holds configuration for the webhook client

type Manifest 

type Manifest struct {
	ModuleName string `json:"moduleName"`
	Type       string `json:"type"`
}

Manifest represents the manifest information in the webhook payload

type Sink 

type Sink struct {
	// contains filtered or unexported fields
}

Sink represents a webhook sink that sends substream data to HTTP endpoints

func NewSink 

func NewSink(config SinkConfig) (*Sink, error)

NewSink creates a new webhook sink

func (*Sink) PrintStats 

func (s *Sink) PrintStats()

PrintStats prints final statistics

func (*Sink) Run 

func (s *Sink) Run(ctx context.Context) error

Run starts the webhook sink

type SinkConfig 

type SinkConfig struct {
	WebhookURL   string
	StateFile    string
	SinkerConfig *sink.SinkerConfig
	ClientConfig Config
	Logger       *zap.Logger
}

SinkConfig holds configuration for the webhook sink

type WebhookPayload 

type WebhookPayload struct {
	Clock    Clock           `json:"clock"`
	Manifest Manifest        `json:"manifest"`
	Data     json.RawMessage `json:"data"`
}

WebhookPayload represents the payload structure sent to webhook endpoints

func NewWebhookPayload 

func NewWebhookPayload(moduleName string, clock *pbsubstreams.Clock, msgType string, data json.RawMessage) (*WebhookPayload, error)

NewWebhookPayload creates a new webhook payload with the desired format

func (*WebhookPayload) ToJSON 

func (p *WebhookPayload) ToJSON() ([]byte, error)

ToJSON serializes the webhook payload to JSON bytes

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.