ioswarm

ioSwarm Coordinator

The ioSwarm coordinator runs as a module inside iotex-core on a delegate node. It distributes transaction validation to a swarm of agents over gRPC, compares agent results against on-chain execution (shadow mode), and settles rewards on-chain via the AgentRewardPool contract.

IIP-58: ioSwarm — Distributed AI Execution Layer for IoTeX Delegates

Architecture

Delegate Machine
┌──────────────────────────────────────────────────┐
│  iotex-core (ioswarm-v2.3.5 branch)              │
│  ├── actpool         ← pending transactions      │
│  ├── statedb         ← account/contract state    │
│  └── ioswarm/        ← coordinator module        │
│       ├── Poll actpool for pending txs            │
│       ├── SimulateAccessList → prefetch state     │
│       ├── Build TaskPackages (tx + state)         │
│       ├── Dispatch to agents via gRPC streaming   │
│       ├── Collect results, shadow-compare         │
│       ├── Stream state diffs to L4 agents         │
│       └── On-chain reward settlement              │
│                         │                         │
│  gRPC :14689            │  HTTP :14690            │
└─────────────────────────┼─────────────────────────┘
                          │
          ┌───────────────┼───────────────┐
          │               │               │
       Agent-1         Agent-2         Agent-N
       L3 stateless    L4 stateful     anywhere

Delegate Guide: Enable ioSwarm on Your Node

Any IoTeX delegate can opt-in to ioSwarm with zero consensus changes. Your delegate continues to sign blocks exactly as before — ioSwarm adds a sidecar that distributes execution work to agents and rewards them on-chain.

Step 1 — Pull the ioSwarm-enabled image

docker pull raullen/iotex-core:ioswarm-v16

Or build from source (ioswarm-v2.3.5 branch):

cd iotex-core && git checkout ioswarm-v2.3.5
docker buildx build --platform linux/amd64 -t your-image:tag .

Step 2 — Add ioSwarm config to your config.yaml

ioswarm:
  enabled: true
  grpcPort: 14689                # agent gRPC connections
  swarmApiPort: 14690            # HTTP monitoring API (0 to disable)
  maxAgents: 100
  taskLevel: "L4"                # L1, L2, L3, or L4
  shadowMode: true               # compare agent results vs on-chain (recommended)
  pollIntervalMs: 1000           # actpool poll interval
  masterSecret: "<your-secret>"  # HMAC master secret for agent auth (hex string)
  delegateAddress: ""            # delegate address (auto-detected if empty)
  epochRewardIOTX: 0.5           # IOTX reward per epoch

  # On-chain reward settlement (optional — omit to run without rewards)
  rewardContract: "0x236CBF52125E68Db8fA88b893CcaFB2EE542F2d9"
  rewardSignerKey: "<hex-key>"   # coordinator hot wallet private key (hex, no 0x prefix)

  # State diff for L4 agents
  diffStoreEnabled: true
  diffStorePath: "/var/data/statediffs.db"

  # Reward parameters
  reward:
    delegateCutPct: 10           # delegate keeps 10% of epoch reward
    epochBlocks: 1               # 1 block per epoch (= 10s at IoTeX 10s block time)
    minTasksForReward: 1         # minimum tasks to qualify for reward
    bonusAccuracyPct: 99.5       # accuracy threshold for bonus multiplier
    bonusMultiplier: 1.2         # weight multiplier for high-accuracy agents

Config field reference:

Step 3 — TLS + networking

Important: Do NOT expose port 14689 directly to the public internet. HMAC tokens are sent in gRPC metadata and must be protected by TLS.

Recommended architecture (TLS termination via reverse proxy):

Agent (--tls) → Load Balancer (:443 TCP) → nginx (TLS terminate) → gRPC (:14689 localhost)

# Bind gRPC and HTTP to localhost only — not accessible from outside
docker run -d --name iotex \
  -p 4689:4689 -p 14014:14014 \
  -p 127.0.0.1:14689:14689 -p 127.0.0.1:14690:14690 \
  -v /data:/var/data \
  -v /etc/iotex/config.yaml:/etc/iotex/config_override.yaml \
  raullen/iotex-core:ioswarm-v16

# Use nginx or a cloud load balancer to terminate TLS on :443
# and proxy to localhost:14689 (gRPC) / localhost:14690 (HTTP)
#
# IMPORTANT: Set grpc_read_timeout to at least 3600s for gRPC streaming.
# The default 60s causes GetTasks streams to disconnect with 504 errors.
# Example nginx config:
#   location / {
#       grpc_pass grpc://127.0.0.1:14689;
#       grpc_read_timeout 3600s;
#       grpc_send_timeout 3600s;
#       client_body_timeout 3600s;
#   }

# Verify coordinator is running
curl http://localhost:14690/swarm/status

For local development / testing without TLS, you can bind directly:

docker run -d --name iotex \
  -p 4689:4689 -p 14014:14014 -p 14689:14689 -p 14690:14690 \
  ...
# Agents connect with: --coordinator=host:14689 --tls=false

Step 4 — Deploy AgentRewardPool contract (optional)

Skip this step if you use the shared mainnet contract (0x236CBF52125E68Db8fA88b893CcaFB2EE542F2d9).

To deploy your own:

./ioswarm-agent deploy \
  --private-key=<deployer-key> \
  --coordinator=0x<coordinator-hot-wallet>

Set the returned contract address in rewardContract.

Step 5 — Export L4 snapshot (for L4 agents)

L4 agents need a state snapshot to bootstrap. The snapshotexporter tool reads trie.db via iotex-core's KVStore layer (handles both BoltDB and PebbleDB).

# Build the snapshot exporter (requires CGO for crypto)
docker run --rm --platform linux/amd64 \
  -v $(pwd):/go/apps/iotex-core -w /go/apps/iotex-core \
  golang:1.23.0-alpine sh -c \
  'apk add --no-cache gcc musl-dev linux-headers git && \
   CGO_ENABLED=1 go build -ldflags "-extldflags \"-static\"" \
   -o bin/snapshotexporter ./tools/snapshotexporter/'

# Export Account+Code only (~1 min, ~209 MB — sufficient for L4)
# NOTE: trie.db must NOT be locked by a running iotex-server.
# Stop iotex first, or copy trie.db while stopped.
docker stop iotex
cp /var/data/trie.db /var/data/trie.db.snap
docker start iotex
./bin/snapshotexporter --source /var/data/trie.db.snap --namespaces Account,Code --output acctcode.snap.gz

# Export full state including Contract trie (~1 hour, ~1.4 GB)
./bin/snapshotexporter --source /var/data/trie.db.snap --output baseline.snap.gz

# Host it for agent download (CDN, object store, or file server)

Public snapshot: https://ts.iotex.me — updated daily via cron.

Daily cron setup (automates the above):

# /root/export-snapshot.sh — runs at 04:00 UTC daily
# 1. docker stop iotex → cp trie.db → docker start iotex (~2 min downtime)
# 2. snapshotexporter --namespaces Account,Code (~1 min)
# 3. rsync to snapshot server
# 4. Total: ~4 min, of which ~2 min is delegate downtime

crontab -e
# Add: 0 4 * * * /root/export-snapshot.sh

Why Account+Code only? Contract storage (MPT trie nodes, 80M+ entries) takes ~1 hour to export. L4 agents sync Contract state incrementally via StreamStateDiffs after loading the Account+Code snapshot. The DiffStore retains 10,000 blocks (~27 hours), so a daily snapshot keeps agents within the catch-up window.

Step 6 — Generate agent API keys and onboard agents

Keys are HMAC-SHA256 derived from the master secret:

key = "iosw_" + hex(HMAC-SHA256(masterSecret, agentID))

Share the key + agent ID with each agent operator. They will use them to connect:

ioswarm-agent \
  --coordinator=<your-delegate-ip>:14689 \
  --agent-id=<agent-id> \
  --api-key=iosw_<key> \
  --wallet=<agent-reward-wallet>

See the ioswarm-agent README for full agent setup instructions.

Step 7 — Monitor your swarm

# Public stats (CORS-enabled, no auth — safe to expose to community dashboards)
curl http://localhost:14690/api/stats

# Swarm status (detailed)
curl http://localhost:14690/swarm/status

# Connected agents
curl http://localhost:14690/swarm/agents

# Shadow accuracy
curl http://localhost:14690/swarm/shadow

# Agent leaderboard
curl http://localhost:14690/swarm/leaderboard

Public Stats API

GET /api/stats returns a JSON summary designed for community dashboards and landing pages. It requires no authentication and includes CORS headers (Access-Control-Allow-Origin: *).

{
  "active_agents": 47,
  "agents_by_level": {"L3": 35, "L4": 12},
  "tasks_dispatched": 128473,
  "tasks_received": 128470,
  "shadow_accuracy": 99.87,
  "shadow_compared": 100000,
  "shadow_matched": 99876,
  "current_epoch": 24,
  "epoch_tasks": 4562,
  "epoch_accuracy": 99.83,
  "task_level": "L4",
  "uptime": "24h12m30s",
  "updated_at": 1710340200
}

Response is cached for 30 seconds (Cache-Control: public, max-age=30).

Per-Agent Rewards API

GET /api/rewards?agent=<agent-id> returns reward history for a specific agent. No authentication required, CORS-enabled.

{
  "agent_id": "beta-008",
  "history": [
    {"epoch": 42, "amount_iotx": 0.045, "tasks": 120, "accuracy_pct": 99.8, "rank": 1, "bonus_applied": true},
    {"epoch": 41, "amount_iotx": 0.038, "tasks": 95, "accuracy_pct": 98.5, "rank": 3, "bonus_applied": false}
  ],
  "totals": {"earned_iotx": 0.083, "total_tasks": 215, "avg_accuracy": 99.15}
}

What happens after you enable ioSwarm

1. Zero risk: Your delegate continues signing blocks exactly as before. ioSwarm runs in shadow mode — agent results are compared but never used for block production.
2. Agents connect: They receive transaction validation tasks via gRPC streaming.
3. Rewards flow: Every epoch (~10s), the coordinator calls depositAndSettle() on the AgentRewardPool contract, distributing IOTX to agents proportional to their work.
4. Shadow accuracy: You can monitor agent accuracy in real-time via /swarm/shadow. If all agents disappear, the delegate falls back to running solo.

Validation Levels

L3 State Prefetch: SimulateAccessList

For contract calls, the coordinator runs a read-only EVM simulation (similar to eth_createAccessList) to discover every storage slot the transaction will access. These slots are sent with the task so the agent has complete state for deterministic EVM execution.

Mainnet result: 230+ transactions, 100% shadow accuracy (agent results identical to on-chain execution).

Key implementation details:

• Uses evm.SimulateAndCollectAccessList() in adapter.go
• Requires full blockchain context (bc.Context()) including BaseFee, GetBlockHash, GetBlockTime
• All io1 addresses are converted to 0x hex format before sending to agents (go-ethereum only accepts 20-byte hex addresses)
• BlockContext.Random must be set (even to zero hash) for Shanghai/Cancun opcode support

L4 State Sync

L4 agents maintain a full copy of IoTeX EVM state locally (BoltDB). This allows fully independent transaction validation without relying on coordinator-provided state.

How it works:

1. Bootstrap: Agent loads an IOSWSNAP snapshot (~209 MB for Account+Code, ~1.4 GB for full state including Contract trie nodes)
2. State diffs: After each block commit, coordinator captures the ordered changeset from workingset.SerializeQueue() and streams to agents via StreamStateDiffs gRPC
3. Agent applies: diffs to local BoltDB, maintaining state in sync with the delegate
4. Recovery: On disconnect, agent requests missing diffs from coordinator's DiffStore (last 1000 blocks retained)

Production results (mainnet, March 2026):

• Shadow accuracy: 100% — nonce race false positives are automatically excluded (same-sender tx consumed nonce between dispatch and commit)
• Kill/restart recovery: <200ms from BoltDB state store, no snapshot reload needed
• Memory: ~679 MiB, CPU: ~24.5%
• State store: ~931 MB BoltDB after sync

State data needed (3 of 10+ namespaces):

IOSWSNAP Snapshot Format

Binary format for bootstrapping L4 agents:

header:  magic("IOSWSNAP") + version(uint32) + height(uint64)
entries: [marker(0x01) + ns_len(uint8) + ns + key_len(uint32) + key + val_len(uint32) + val]*
end:     marker(0x00)
trailer: count(uint64) + sha256(32 bytes) + end_magic("SNAPEND\0")

Export tool: ioswarm/cmd/l4baseline reads trie.db (BoltDB) and outputs gzip-compressed IOSWSNAP.

# Stats only
go run ./ioswarm/cmd/l4baseline --source trie.db --stats

# Export Account+Code only (~209 MB compressed)
go run ./ioswarm/cmd/l4baseline --source trie.db --output acctcode.snap.gz --namespaces Account,Code

# Export full baseline (~1.4 GB compressed)
go run ./ioswarm/cmd/l4baseline --source trie.db --output baseline.snap.gz

On-Chain Reward Settlement

Every epoch (configurable, default 1 block = 10s), the coordinator:

1. Calculates each agent's weight: tasks x (accuracy >= 99.5% ? 1.2 : 1.0)
2. Calls depositAndSettle(agents[], weights[]) on the AgentRewardPool contract, sending epochReward x (1 - delegateCut) as IOTX value
3. The contract uses F1 (cumulative reward-per-weight) algorithm for O(1) proportional distribution
4. Agents call claim() at any time to withdraw accumulated rewards

AgentRewardPool v2 contract (mainnet): 0x236CBF52125E68Db8fA88b893CcaFB2EE542F2d9

Source in ioswarm-agent/contracts/.

E2E test results (mainnet, March 2026):

• Single agent payout: PASS
• Multi-agent proportional split: PASS (50/50)
• 10 agents x 5 epochs: PASS (0.225 IOTX each)
• Dynamic join/leave: PASS (contract balance = 0 after all claims)
• MinTasks threshold: PASS (weight=0 gets 0)
• Delegate cut: PASS (10% retained, 90% to agents)

gRPC Protocol

service IOSwarm {
  rpc Register(RegisterRequest) returns (RegisterResponse);
  rpc GetTasks(GetTasksRequest) returns (stream TaskBatch);       // server push
  rpc SubmitResults(BatchResult) returns (SubmitResponse);
  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);
  rpc StreamStateDiffs(StateDiffRequest) returns (stream StateDiffBatch);  // L4
}

Agents connect via Register, authenticate with HMAC key, then open a server-streaming GetTasks call. Results flow back via SubmitResults. Epoch payouts are delivered in HeartbeatResponse.

L4 agents additionally open StreamStateDiffs to receive per-block state mutations. The coordinator's StateDiffBroadcaster fans out diffs from a ring buffer, and DiffStore persists recent diffs for catch-up.

Integration with iotex-core

The coordinator is wired into iotex-core via server/itx/server.go:

// In newServer():
if cfg.IOSwarm.Enabled {
    actPoolAdapter := ioswarm.NewActPoolAdapter(cs.ActPool(), cs.Blockchain())
    stateAdapter := ioswarm.NewStateReaderAdapter(cs.StateFactory(), cs.Blockchain(), cs.Genesis())
    svr.ioswarmCoord = ioswarm.NewCoordinator(cfg.IOSwarm, actPoolAdapter, stateAdapter)
}

// In Start():
if svr.ioswarmCoord != nil {
    svr.ioswarmCoord.Start(ctx)
}

Two interfaces decouple the coordinator from iotex-core internals:

type ActPoolReader interface {
    PendingActions() []*PendingTx
    BlockHeight() uint64
}

type StateReader interface {
    AccountState(addr string) (*pb.AccountSnapshot, error)
    GetCode(addr string) ([]byte, error)
    GetStorageAt(addr, slot string) (string, error)
    SimulateAccessList(from, to string, data []byte, value string, gasLimit uint64) (map[string][]string, error)
}

State diff integration hooks into stateDB.PutBlock():

// In server/itx/server.go:
svr.ioswarmCoord.SetupStateDiffCallback(cs.StateFactory())
// Callback chain: stateDB.Finalize() → diffCallback → coordinator.ReceiveStateDiff()
//   → DiffStore.Append() + StateDiffBroadcaster.Publish()
//   → gRPC StreamStateDiffs to all connected L4 agents

Source Layout

CLI Tools (cmd/)

L5 Roadmap: Agent Block Building (ePBS)

L4 stateful agents are the foundation for L5 block building, following Ethereum's ePBS (EIP-7732) model adapted for IoTeX's DPoS architecture.

Key advantage: IoTeX's deltaStateDigest is a hash of the ordered state change queue (not a Merkle Trie root). Agents need only a flat KV store — no full MPT required. This makes block building feasible on commodity hardware.

For detailed L5 design, see design-epbs-block-building.md.

Related