Workflow Debugging

Debug agentic workflow runs using gh aw audit and gh aw logs commands.

Core Commands

Audit a Specific Run

Investigate a single workflow run with comprehensive error detection:

gh aw audit <run-id-or-url> --parse -v

Accepts:

• •Numeric run ID: 21005890162
• •GitHub Actions URL: https://github.com/owner/repo/actions/runs/21005890162
• •Job URL: https://github.com/owner/repo/actions/runs/21005890162/job/9876543210
• •Job URL with step: https://github.com/owner/repo/actions/runs/21005890162/job/9876543210#step:7:1

What it does:

• •Downloads artifacts and logs to .github/aw/logs/run-<id>/
• •Detects errors and warnings
• •Analyzes MCP tool usage statistics
• •Generates detailed Markdown report
• •Extracts specific step output (if job URL with step)

Output location: .github/aw/logs/run-<run-id>/

Download Multiple Runs

Analyze patterns across multiple workflow executions:

gh aw logs [workflow] --count <N> --parse

Common options:

• •--count 10 - Download last 10 runs
• •--start-date -1w - Last week's runs
• •--end-date -1d - Until yesterday
• •--engine claude - Filter by engine (claude/codex/copilot)
• •--firewall - Filter runs with firewall enabled
• •--safe-output create-issue - Filter by safe output type
• •--parse - Generate Markdown reports
• •--json - JSON output format

Output location: .github/aw/logs/ (configurable with -o)

Debugging Workflow

Step 1: Audit the Run

Start with the audit command to get a comprehensive overview:

gh aw audit <run-url> --parse -v

Review the generated report for:

• •✅ Success indicators
• •🟡 Warnings
• •❌ Errors
• •Token usage and performance metrics
• •Job status and duration
• •Tool usage statistics

Step 2: Examine Logs

Navigate to the downloaded logs:

cd .github/aw/logs/run-<id>/

Key files:

• •agent-stdio.log - Full agent execution log (search here for errors)
• •aw_info.json - Workflow metadata and configuration
• •workflow-logs/ - GitHub Actions job logs
• •mcp-logs/gateway.md - MCP Gateway status and requests
• •mcp-logs/mcp-gateway.log - Raw MCP Gateway logs
• •sandbox/firewall/logs/access.log - Firewall access logs (if enabled)
• •safe_output.jsonl - Agent's final output (if available)

Step 3: Search for Common Issues

Use the quick scan script for rapid error detection:

python3 scripts/quick_scan.py .github/aw/logs/run-<id>/

Or search manually for specific patterns:

MCP server failures:

grep -E "mcp:.*failed" agent-stdio.log

DNS resolution errors:

grep "dns error.*Name does not resolve" agent-stdio.log

OAuth/authentication issues:

grep "WARN codex_rmcp_client::oauth" agent-stdio.log

Tool availability errors:

grep -i "tool.*not available\|tool.*failed" agent-stdio.log

Firewall blocks:

grep "TCP_DENIED" sandbox/firewall/logs/access.log

Step 4: Check MCP Gateway

Review MCP Gateway logs to verify server connectivity:

cat mcp-logs/gateway.md

Look for:

• •✓ Successfully loaded servers
• •🔍 RPC request/response pairs
• •⚠️ HTTP errors (404, 500, etc.)
• •✓ Tools list responses

Step 5: Analyze Root Cause

Consult the common errors reference for known patterns:

cat references/common_errors.md

This document catalogs:

• •MCP server failures (DNS, OAuth, session)
• •Firewall issues
• •Agent execution errors
• •GitHub Actions problems

Step 6: Document Findings

Create an issue to document the problem:

gh issue create \
  --repo <owner/repo> \
  --title "<concise-issue-title>" \
  --body "<detailed-description>"

Include:

• •Workflow run URL
• •Summary of the issue
• •Evidence from logs (error messages)
• •Root cause analysis
• •Impact assessment
• •Reproduction steps
• •Suggested fixes

Common Patterns

Silent MCP Failures

Symptom: Workflow shows green but agent couldn't use MCP tools

Detection:

gh aw audit <run-id> -v
grep -E "mcp:.*failed" .github/aw/logs/run-<id>/agent-stdio.log

Causes:

• •DNS resolution failure (host.docker.internal)
• •OAuth token issues
• •MCP Gateway not reachable
• •Session not found errors

Reference: See references/common_errors.md for detailed patterns

False Success

Symptom: Workflow completed successfully but didn't produce expected results

Investigation:

1. •Check for MCP server failures (tools unavailable)
2. •Check for firewall blocks (network requests failed)
3. •Review agent output for errors
4. •Verify safe outputs were created

Network Issues

Detection:

grep "TCP_DENIED\|TAG_NONE" sandbox/firewall/logs/access.log

Causes:

• •Domain not in firewall allowlist
• •DNS resolution through proxy failed
• •Network timeout

Tips

Green doesn't mean success: Always audit the logs even if the workflow shows as successful. Many failures are silent.

Use audit first: The audit command provides a comprehensive overview and is faster than manually downloading and examining logs.

Check all MCP servers: If one MCP server fails, check if others also failed—this indicates a systemic issue like DNS or networking.

Firewall logs are crucial: When debugging network issues, always check firewall access logs for blocked domains.

Look for patterns: Use the logs command to download multiple runs and identify patterns across executions.

Reference common errors: Before deep investigation, check references/common_errors.md for known patterns and solutions.

Resources

scripts/quick_scan.py

Rapid error detection script that scans for common issues:

• •MCP server failures
• •DNS resolution errors
• •OAuth/keyring warnings
• •Tool availability errors
• •Firewall blocks
• •MCP Gateway session errors

Usage:

python3 scripts/quick_scan.py <log-directory>

references/common_errors.md

Comprehensive catalog of common error patterns with:

• •Error signatures and patterns
• •Search commands for detection
• •Root cause explanations
• •Impact assessments
• •Known symptoms