Devnet Log Review

Analyze and summarize devnet run results from lean consensus testing.

Quick Start

Run the analysis script:

# From project root (with logs in current directory)
.claude/skills/devnet-log-review/scripts/analyze-logs.sh

# Or specify logs directory
.claude/skills/devnet-log-review/scripts/analyze-logs.sh /path/to/logs

This produces a structured summary with:

• •Error/warning counts per node
• •Block production statistics
• •Consensus progress
• •Proposer assignments

Log File Locations

Analysis Scripts

Usage examples:

# Just count errors/warnings
.claude/skills/devnet-log-review/scripts/count-errors-warnings.sh

# Show errors for specific node
.claude/skills/devnet-log-review/scripts/show-errors.sh -n zeam_0

# Show errors and warnings with limit
.claude/skills/devnet-log-review/scripts/show-errors.sh -w -l 50

Common Investigation Patterns

Tracing Slot-by-Slot Flow

When investigating issues, trace the complete flow for a specific slot using structured logging fields (slot=X).

Note: Logs contain ANSI color codes. Strip them first:

# Strip ANSI codes and grep for a specific slot
sed 's/\x1b\[[0-9;]*m//g' devnet.log | grep -E "slot=3[^0-9]|slot=3$"

# For double-digit slots
sed 's/\x1b\[[0-9;]*m//g' devnet.log | grep -E "slot=12[^0-9]|slot=12$"

Structured logging fields follow key=value format:

• •slot=N - Slot number
• •validator_id=N - Validator index
• •validator=N - Validator index (in gossipsub messages)
• •proposer=N - Block proposer index
• •err=... - Error message

Comparing Clients at Specific Slots

# Extract block hashes for specific slots across all clients
for slot in 1 2 3 4 5; do
  echo "=== Slot $slot ==="
  grep -h "slot=$slot[^0-9]\|@ $slot[^0-9]" *.log | grep -oE "0x[a-f0-9]{8}" | sort -u
done

# Check which client has which head at a specific slot
grep -h "head_slot=18\|Head Slot: 18" *.log

# Compare finalization across clients
grep -h "finalized.*slot\|Finalized block.*@" *.log | tail -20

Finding Validators

Each validator proposes blocks when slot % validator_count == validator_id.

# ethlambda - explicit validator_id in logs
grep "We are the proposer" ethlambda_0.log | head -3
# Output: We are the proposer for this slot slot=5 validator_id=5

# zeam - proposer field in attestation logs
grep "packing proposer attestation" zeam_0.log | head -3
# Output: packing proposer attestation for slot=6 proposer=0

# Generic approach - validator_id = slot % validator_count

Analysis Areas

Fork Analysis

When clients disagree on which blocks are valid, the network splits into forks.

Quick check for forks:

# Compare block hashes at same slot across clients
grep -h "slot=4[^0-9]" *.log | grep -oE "block_root=0x[a-f0-9]{16}" | sort -u

# If you see different hashes → fork exists!

Identifying rejected blocks:

# Check signature verification failures
grep -i "signature.*failed\|invalid signature" *.log | head -20

# ethlambda
grep "Failed to process block" ethlambda_0.log

# qlean
grep "Invalid signatures for block" qlean_0.log

# lantern
grep "signature verification failed" lantern_0.log

See references/FORK_ANALYSIS.md for:

• •Understanding fork types (canonical, orphan, invalid)
• •Tracing parent-child relationships
• •Building fork structure diagrams
• •Determining which validators are on which fork

Finalization Debugging

Finalization should advance every 6-12 slots. If it stalls, investigate:

# Check finalization progress
grep "finalized_slot=" ethlambda_0.log | tail -20

# If finalized_slot stays same for 50+ slots → finalization stalled

Finalization requires >2/3 supermajority:

• •6 validators → need 5 votes minimum
• •9 validators → need 7 votes minimum

See references/FINALIZATION_DEBUG.md for:

• •Common causes of finalization stalls
• •Validator participation calculations
• •Justification chain analysis
• •Step-by-step debugging guide

Error Classification

See references/ERROR_CLASSIFICATION.md for:

• •Critical errors (genesis mismatch, panics, database corruption)
• •Expected/benign messages (TODOs, HandshakeTimedOut to unconfigured nodes)
• •Medium severity issues (encoding mismatches, missing blocks)
• •State transition errors

Client Log Patterns

Different clients have different log formats and key patterns.

See references/CLIENT_LOG_PATTERNS.md for:

• •Log format for each client (zeam, ream, ethlambda, grandine, lantern, qlean)
• •Key log patterns per client
• •Block counting methods
• •ANSI color code handling

Block Proposal Flow (ethlambda)

A healthy block proposal follows this sequence:

1. •We are the proposer for this slot - Node detects it's the proposer
2. •TODO precompute poseidons in parallel + SIMD - XMSS aggregate proof starts
3. •packed_pcs_commit - Proof commitment
4. •Logup data - Logup protocol data
5. •AIR proof{table=poseidon16} / AIR proof{table=poseidon24} - AIR proofs
6. •Published block - Block successfully built and published
7. •Published block to gossipsub - Block broadcast to network

Summary Report Format

Generate concise summaries (20 lines or less) in this structure:

## Devnet Log Summary

**Run:** {N} {client} nodes (`{image}`) | {M} slots ({range})

| Node | Validator | Blocks Proposed | Errors | Warnings | Status |
|------|-----------|-----------------|--------|----------|--------|
| {node_name} | {id} | {count} (slots {list}) | {n} | {n} | {emoji} |

**Issues:**
- {issue 1}
- {issue 2}

**{emoji} {RESULT}** - {one-line explanation}

Status Emoji Guide

Result Line Examples

• •🟢 PASSED - All nodes healthy, consensus achieved
• •🟡 PASSED WITH WARNINGS - Consensus working but minor issues detected
• •🔴 FAILED - Consensus broken: {reason}

Key Rules

1. •Keep summary under 20 lines
2. •Use table for per-node status
3. •Status should reflect whether that node's blocks pass validation (🔴 if not)
4. •End with single-line result with emoji
5. •Don't list "what's working" - focus on issues

Manual Investigation Commands

Use these when scripts don't provide enough detail:

# Find which validators proposed blocks
grep -h "proposer\|We are the proposer" *.log | head -20

# Check peer connections
grep -h "peer connected\|Connection established" *.log | head -20

# Check attestations
grep -i "attestation" *.log | head -50

# Search for specific error patterns
grep -i "genesis mismatch\|panic\|fatal" *.log

# Track attestations to unknown blocks (indicates forks)
grep "Unknown.*block:" ethlambda_0.log | grep -oE "0x[a-f0-9]{64}" | sort | uniq -c | sort -rn

# Check which validators are on invalid fork
grep "rejected vote" lantern_0.log | grep -oE "validator=[0-9]+" | sort | uniq -c

Detailed References

For in-depth analysis, see these specialized guides:

• •FORK_ANALYSIS.md - Comprehensive guide to identifying and analyzing blockchain forks, tracing parent-child relationships, building fork structure diagrams, and determining consensus disagreements
• •FINALIZATION_DEBUG.md - Debugging finalization stalls, validator participation calculations, justification chain analysis, and threshold math
• •CLIENT_LOG_PATTERNS.md - Log formats and key patterns for all clients (zeam, ream, ethlambda, grandine, lantern, qlean), including block counting methods
• •ERROR_CLASSIFICATION.md - Error types, severity levels, expected vs. critical errors, and interoperability issues

Progressive Disclosure

This skill uses progressive disclosure to keep context usage efficient:

1. •Start here (SKILL.md) - Quick start workflow and common patterns
2. •Detailed references (references/*.md) - Deep dives into specific analysis areas
3. •Scripts (scripts/) - Automated analysis tools

Load detailed references only when needed for specific investigations.