webhook

Substreams Webhook Sink Examples

This document provides examples of how to use the webhook sink with the new retry functionality.

Basic Usage

# Basic webhook call with default settings
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name

# With custom start block
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name --start-block 12345

Retry Configuration

The webhook sink now supports configurable retry behavior for handling transient failures:

Default Settings

• Max Retries: 3 attempts (use -1 for infinite retries)
• Timeout: 30 seconds per request
• Max Retry Interval: 30 seconds (exponential backoff cap)

Custom Retry Settings

# Increase retry attempts for unreliable endpoints
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 5

# Shorter timeout for faster failure detection
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-timeout 10s

# Longer maximum retry interval for heavily loaded endpoints
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retry-interval 60s

# Disable retries completely
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 0

# Enable infinite retries (retry until success or context cancellation)
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries -1

Combined Configuration

# Production-ready configuration with aggressive retries
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries 5 \
  --webhook-timeout 15s \
  --webhook-max-retry-interval 45s \
  --state-file ./production.cursor

# Production configuration with infinite retries for critical webhooks
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --webhook-max-retries -1 \
  --webhook-timeout 30s \
  --webhook-max-retry-interval 15s \
  --state-file ./production.cursor

Retry Behavior

What Gets Retried

• Network errors (connection failures, timeouts)
• Server errors (5xx HTTP status codes)
• Temporary service unavailability

What Doesn't Get Retried

• Client errors (4xx HTTP status codes like 400, 401, 403, 404)
• Request creation failures (invalid URLs)
• Context cancellation

Retry Modes

• Limited retries (default): Retry up to --webhook-max-retries times
• No retries (--webhook-max-retries 0): Fail immediately on first error
• Infinite retries (--webhook-max-retries -1): Retry indefinitely until success or context cancellation

Exponential Backoff

The retry mechanism uses exponential backoff with jitter:

• First retry: ~1 second
• Second retry: ~2 seconds
• Third retry: ~4 seconds
• Maximum interval is capped by --webhook-max-retry-interval

Infinite Retry Behavior

When --webhook-max-retries is set to -1:

• Webhook calls will retry indefinitely for transient failures
• Only context cancellation or permanent errors (4xx) will stop retries
• Useful for critical webhooks that must eventually succeed
• Consider setting appropriate --webhook-timeout and --webhook-max-retry-interval values
• Monitor logs for excessive retry patterns that might indicate service issues

State Management

# Custom state file location
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --state-file ./custom/path/state.cursor

# Disable state persistence (start from scratch each time)
substreams sink webhook https://api.example.com/webhook manifest.yaml module_name \
  --state-file ""

Error Handling

The webhook sink continues processing even if webhook calls fail after all retries. This ensures that:

• The substreams processing doesn't get blocked by webhook failures
• Block processing continues uninterrupted
• Cursor state is still saved for successful blocks

Monitoring and Logging

The webhook sink provides detailed logging for troubleshooting:

INFO calling webhook block=12345 url=https://api.example.com/webhook
WARN webhook call failed for block 12345: webhook returned server error status 503 for block 12345
INFO calling webhook block=12345 url=https://api.example.com/webhook (retry 1/3)
INFO webhook call successful block=12345 url=https://api.example.com/webhook

Best Practices

1. Set appropriate timeouts: Use shorter timeouts (5-15s) for real-time processing
2. Configure retries based on endpoint reliability: More retries for less reliable services
3. Use infinite retries sparingly: Only for critical webhooks where eventual delivery is essential
4. Monitor webhook endpoint performance: Adjust settings based on observed behavior
5. Use state files: Always specify a state file for production deployments
6. Handle failures gracefully: Implement idempotent webhook handlers that can handle duplicate calls
7. Context management: When using infinite retries, ensure proper context cancellation for shutdown

Retry Strategy Guidelines

For real-time applications:

--webhook-max-retries 2
--webhook-timeout 5s
--webhook-max-retry-interval 10s

For critical data delivery:

--webhook-max-retries -1
--webhook-timeout 30s
--webhook-max-retry-interval 60s

For testing/development:

--webhook-max-retries 0
--webhook-timeout 10s

Webhook Payload Format

The webhook receives JSON payloads in the following format:

{
  "clock": {
    "timestamp": "2024-02-12T22:23:51.000Z",
    "number": 53448530,
    "id": "f843bc26cea0cbd50b09699546a8a97de6a1727646c17a857c5d8d868fc26142"
  },
  "manifest": {
    "moduleName": "module_name",
    "type": "sf.substreams.ethereum.v1.Events"
  },
  "data": {
    // Your module's output data
  }
}

It is loosely based on the format from https://github.com/pinax-network/substreams-sink-webhook

Payload Structure

• clock: Contains blockchain timing and identification information
  • timestamp: Block timestamp in RFC3339 format
  • number: Block number
  • id: Block hash/ID
• manifest: Contains module metadata
  • moduleName: Name of the substreams module that generated this data
  • type: Module output type (automatically strips type.googleapis.com/ prefix)
• data: Contains the actual output from your substreams module

Example with Real Data

{
  "clock": {
    "timestamp": "2024-02-12T22:23:51.000Z",
    "number": 53448530,
    "id": "f843bc26cea0cbd50b09699546a8a97de6a1727646c17a857c5d8d868fc26142"
  },
  "manifest": {
    "moduleName": "filtered_events",
    "type": "sf.substreams.ethereum.v1.Events"
  },
  "data": {
    "events": [
      {
        "address": "0x1234567890abcdef",
        "topics": ["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],
        "data": "0x000000000000000000000000000000000000000000000001158e460913d00000"
      }
    ]
  }
}

"module": "module_name", "block": 12345, "timestamp": "2023-01-01T00:00:00Z", "type": "type.googleapis.com/sf.substreams.ethereum.v1.Events", "payload": { // Your module's output data } }

Make sure your webhook endpoint can handle:
- POST requests with `Content-Type: application/json`
- Potential duplicate calls (implement idempotency)
- Proper HTTP status code responses (2xx for success, 4xx for permanent errors, 5xx for retryable errors)

### Monitoring via Prometheus

* Prometheus metrics are available at `http://localhost:9102` by default. See the --prometheus-addr flag for more details.