AgentSkillsCN

hydra-head-troubleshooter

Hydra故障排除：常见问题决策树。将各类症状与对应的修复方案一一对应，并附上验证步骤。

SKILL.md
--- frontmatter
name: hydra-head-troubleshooter
description: "Hydra troubleshooting: decision tree for common issues. Maps symptoms to fixes with verification steps."
allowed-tools:
  - Read
user-invocable: true
context:
  - "!hydra-node --version 2>&1 | head -3"

hydra-head-troubleshooter

When to use

  • •Hydra Head doesn't start or no head observed
  • •Head exists but doesn't make progress
  • •Peers out of sync or disconnected
  • •Log messages: PeerConnected, AckSn, LogicOutcome errors

Operating rules (must follow)

  • •Confirm network and hydra-node version first
  • •Request logs from ALL participants for same time window
  • •Never request key contents (file paths OK)
  • •Output: (1) Root cause (2) Fix steps (3) Verification

Quick diagnostic commands

bash
# Check hydra-node version
hydra-node --version

# Check API health
curl -s localhost:4001/health

# Check peers
curl -s localhost:4001/peers

# Check head status
curl -s localhost:4001/status

# Check metrics (if enabled)
curl -s localhost:6001/metrics | grep hydra

Decision tree

A) "No head is observed from the chain"

Symptoms:

  • •Head never appears in logs
  • •No Init/Commit/Open progression
  • •Client shows no head state

Check 1: Cardano connection

bash
# Verify cardano-node is ready
cardano-cli query tip --testnet-magic 1
# Should show current slot, not error

# Check socket exists
ls -la $CARDANO_NODE_SOCKET_PATH

Fix: Wait for cardano-node sync, verify socket path and network magic

Check 2: Scripts tx id

bash
# Verify you're using correct scripts tx id for network
# Get from hydra-node release notes

# In hydra-node logs, look for:
grep -i "script" hydra-node.log | head -20

Fix: Use correct --hydra-scripts-tx-id for your network

Check 3: Key mismatch

bash
# Verify cardano.sk matches what peers have as your vkey
cardano-cli key verification-key \
  --signing-key-file cardano.sk \
  --verification-key-file check.vkey

# Compare check.vkey with what you distributed

Fix: Re-exchange verification keys with all peers

B) "Head does not make progress"

Symptoms:

  • •Head exists but stuck at Init or Commit
  • •Snapshots not confirmed
  • •Commands hang or timeout

Check 1: Peer connectivity

bash
# In logs, look for PeerConnected
grep "PeerConnected" hydra-node.log

# Check metrics
curl -s localhost:6001/metrics | grep peers_connected

Fix: Verify --peer host:port is correct and ports are reachable

Check 2: Hydra key mismatch

bash
# Look for AckSn issues
grep -E "AckSn|LogicOutcome" hydra-node.log

# Verify hydra keys match
hydra-node verify-keys \
  --hydra-signing-key hydra.sk \
  --hydra-verification-key peer-hydra.vk

Fix: Re-exchange hydra verification keys

Check 3: Version mismatch

bash
# All nodes must run compatible versions
hydra-node --version
# Compare across all participants

Fix: Align hydra-node versions

C) "Head stuck: peer out of sync"

Symptoms:

  • •One node accepts txs, others reject
  • •Snapshots stop being signed
  • •Ledger state diverged

Cause: Different ledger views due to config drift or missed txs

Fix: Snapshot side-loading

bash
# 1. Get latest snapshot from healthy node
curl -s http://healthy-node:4001/snapshot > snapshot.json

# 2. Post to out-of-sync node
curl -X POST http://out-of-sync-node:4001/snapshot \
  -H "Content-Type: application/json" \
  -d @snapshot.json

Verify: Snapshots resume, all parties sign again

D) Mirror nodes / HA issues

Symptoms:

  • •SnapshotAlreadySigned in logs
  • •etcd quorum issues

Reality:

  • •SnapshotAlreadySigned is transient and harmless with mirrors
  • •Keep mirror count < floor(n/2) for quorum stability

Fix: Reduce mirror count or accept transient warnings

Log patterns to grep

PatternMeaning
PeerConnectedPeer connected successfully
PeerDisconnectedPeer lost connection
AckSnSnapshot acknowledgment
LogicOutcomeHead logic result (check for errors)
SnapshotConfirmedSnapshot finalized
HeadIsOpenHead successfully opened
HeadIsClosedHead closed, contestation started

Verification checklist

After any fix:

  • • All nodes same hydra-node version
  • • Scripts tx id correct for network
  • • cardano-node ready (query tip works)
  • • PeerConnected for all peers
  • • AckSn messages flowing
  • • No LogicOutcome errors

Examples

Example: Diagnose stuck head

User request: "My Hydra head won't open—stuck at commit"

Response:

bash
# 1. Check all peers connected
grep "PeerConnected" hydra-node.log | tail -10
# Should see PeerConnected for each peer

# 2. If missing peers, check network
nc -zv peer-host 5001
# Should show "Connection succeeded"

# 3. Check all parties committed
curl -s localhost:4001/status | jq .
# Look for parties and their commit status

# 4. Check for key issues
grep -E "InvalidSignature|WrongParty" hydra-node.log

# Common fix: restart with correct peer addresses
# Verify --peer host:port matches actual peer locations

References

  • •hydra-head (operational guidance)
  • •hydra-head-operator (execution)
  • •reference/sources.md (doc provenance)
  • •hydra.family troubleshooting