Benchmarking with cargo xtask bench

The facet project uses a sophisticated benchmarking system that generates Markdown reports comparing performance across multiple targets.

Quick Reference - Running Specific Benchmarks

# Run specific benchmark by name
cargo bench --bench unified_benchmarks_divan -- flatten_2enums

# Run with Tier-2 diagnostics
FACET_TIER2_DIAG=1 cargo bench --bench unified_benchmarks_divan -- flatten_2enums 2>&1 | grep TIER_DIAG

# Check tier2 statistics (attempts/successes/fallbacks)
cargo bench --bench unified_benchmarks_divan -- flatten_2enums 2>&1 | grep TIER_STATS

# Run all benchmarks matching a pattern
cargo bench --bench unified_benchmarks_divan -- flatten

# Run Tier-2 JIT benchmarks only
cargo bench --bench unified_benchmarks_divan -- "tier2"

# List available benchmarks
cargo bench --bench unified_benchmarks_divan -- --list | grep -v "    " | head -20

⚠️ IMPORTANT: Benchmark .rs files are GENERATED from facet-json/benches/benchmarks.kdl. DO NOT edit unified_benchmarks_*.rs directly - edit benchmarks.kdl instead.

Quick Usage

# Run all benchmarks and generate HTML + Markdown report
cargo xtask bench --index --serve

# Run benchmarks without the full perf index (faster)
cargo xtask bench

# Re-analyze existing benchmark data without re-running
cargo xtask bench --no-run

# Run only specific benchmarks (filter passed to cargo bench)
cargo xtask bench --index booleans

# Just generate reports from latest data
cargo xtask bench --no-run --index --serve

How It Works

The benchmarking system has three main components:

1. Benchmark Definition (benchmarks.kdl)

Benchmarks are defined in facet-json/benches/benchmarks.kdl using KDL syntax:

benchmark name="simple_struct" type="SimpleRecord" category="micro" {
    json "{\"id\": 42, \"name\": \"test\", \"active\": true}"
}

benchmark name="booleans" type="Vec<bool>" category="synthetic" {
    generated "booleans"
}

type_def name="SimpleRecord" {
    code """
#[derive(Debug, PartialEq, Facet, serde::Serialize, serde::Deserialize, Clone)]
struct SimpleRecord {
    id: u64,
    name: String,
    active: bool,
}
"""
}

Categories: micro, synthetic, realistic, other Data sources: json (inline), json_file, json_brotli, generated

2. Benchmark Generation (cargo xtask gen-benchmarks)

Run this after editing benchmarks.kdl:

cargo xtask gen-benchmarks

This generates three files in facet-json/:

• •benches/unified_benchmarks_divan.rs - Wall-clock timing benchmarks
• •benches/unified_benchmarks_gungraun.rs - Instruction count benchmarks
• •tests/generated_benchmark_tests.rs - Test versions for valgrind debugging

Every benchmark gets all 4 targets automatically:

• •serde_json - Baseline (serde_json crate)
• •facet_format_json - facet-format-json without JIT (reflection only)
• •facet_format_jit_t1 - Tier-1 JIT (shape-based, ParseEvent stream)
• •facet_format_jit_t2 - Tier-2 JIT (format-specific, direct byte parsing)

3. Benchmark Execution and Analysis

cargo xtask bench does:

1. •Runs unified_benchmarks_divan (wall-clock times via divan)
2. •Runs unified_benchmarks_gungraun (instruction counts via gungraun + valgrind)
3. •Parses output and combines results
4. •Generates multiple report formats:
   • •bench-reports/run.json - Full structured data (schema: run-v1)
   • •bench-reports/perf/RESULTS.md - Markdown report for LLMs and humans
   • •bench-reports/perf-data.json - Legacy format for perf tracking

The Markdown Report (perf/RESULTS.md)

Located at bench-reports/perf/RESULTS.md, this is the authoritative source for performance analysis:

Structure:

• •Targets table - Definitions of all benchmark targets
• •Benchmark sections - Grouped by category (Micro, Synthetic, Realistic)
• •Per-benchmark tables - Deserialize and Serialize results
  • •Columns: Target, Time (median), Instructions, vs serde_json ratio
  • •Ratios: **0.84×** ✓ (wins), 1.03× (close), 3.12× ⚠ (needs work)
• •Summary - Auto-categorized by performance:
  • •Wins: ≤1.0× vs serde_json
  • •Close: ≤1.5× vs serde_json
  • •Needs Work: >1.5× vs serde_json

Example:

### booleans

**Deserialize:**

| Target | Time (median) | Instructions | vs serde_json |
|--------|---------------|--------------|---------------|
| serde_json | 56.21µs | 1,157,922 | 1.00× |
| format+jit2 | 53.46µs | 972,221 | **0.84×** ✓ |
| format+jit1 | 809.30µs | 7,031,459 | 6.07× ⚠ |
| format | 2.94ms | 23,169,951 | 20.01× ⚠ |

Adding New Benchmarks

1. •

   Edit facet-json/benches/benchmarks.kdl

   kdl

   benchmark name="my_bench" type="MyType" category="synthetic" {
       generated "my_generator"
   }
   
   type_def name="MyType" {
       code """
   #[derive(Debug, Facet, serde::Serialize, serde::Deserialize, Clone)]
   struct MyType {
       field: String,
   }
   """
   }
   

2. •

   If using generated, add generator to tools/benchmark-generator/src/main.rs

   • •Edit generate_json_data() function
   • •Add case for your generator name
3. •

   Regenerate benchmarks

   bash

   cargo xtask gen-benchmarks
   

4. •

   Run benchmarks

   bash

   cargo xtask bench --index --serve
   

Important Flags

--no-run

Skips running benchmarks, uses latest data. Useful for:

• •Regenerating reports after fixing parser bugs
• •Testing report generation changes
• •Quick iterations on report formatting

--index

Generates the full perf.facet.rs index:

• •Clones the facet-rs/perf.facet.rs repo (gh-pages branch)
• •Copies benchmark reports to bench-reports/perf/
• •Generates index.html and supporting files
• •Required for viewing the interactive SPA

--serve

Starts a local server at http://localhost:1999 to view reports. Requires --index.

--push

Pushes generated reports to the perf.facet.rs repo. Use with caution - only for publishing official results.

Debugging Benchmarks with Valgrind

The generated tests in tests/generated_benchmark_tests.rs mirror the benchmarks and can be run under valgrind:

# Run specific benchmark as test under valgrind
cargo nextest run --profile valgrind -p facet-json generated_benchmark_tests::test_booleans --features jit

# Or use the generated test filters
cargo nextest run --profile valgrind -p facet-json test_simple_struct --features jit

This is essential for debugging crashes or memory issues in benchmarks.

Files and Directories

bench-reports/
├── divan-{timestamp}.txt          # Raw divan output
├── gungraun-{timestamp}.txt       # Raw gungraun output
├── run.json                       # Structured results (run-v1 schema)
├── perf-data.json                 # Legacy perf tracking format
└── perf/
    ├── RESULTS.md                 # **MAIN REPORT - READ THIS**
    ├── index.html                 # SPA (generated with --index)
    ├── app.js                     # SPA logic (copied from scripts/)
    └── shared-styles.css          # SPA styles (copied from scripts/)

facet-json/benches/
├── benchmarks.kdl                 # **EDIT THIS to add benchmarks**
├── unified_benchmarks_divan.rs    # Generated (divan)
└── unified_benchmarks_gungraun.rs # Generated (gungraun)

facet-json/tests/
└── generated_benchmark_tests.rs   # Generated (for valgrind)

tools/
├── benchmark-generator/           # KDL → Rust codegen
└── benchmark-analyzer/            # Output parsing + report generation

Don't Edit Generated Files

❌ NEVER edit these files (they're regenerated):

• •unified_benchmarks_divan.rs
• •unified_benchmarks_gungraun.rs
• •generated_benchmark_tests.rs
• •bench-reports/perf/index.html, app.js, shared-styles.css

✅ Edit these instead:

• •facet-json/benches/benchmarks.kdl - Benchmark definitions
• •tools/benchmark-generator/src/main.rs - Generator logic (for generated benchmarks)
• •scripts/app.js, scripts/shared-styles.css - SPA source (not the copies in perf/)

Common Workflows

Quick local benchmark run

cargo xtask bench
# Check bench-reports/perf/RESULTS.md

Full interactive report

cargo xtask bench --index --serve
# Opens http://localhost:1999

After editing benchmarks.kdl

cargo xtask gen-benchmarks
cargo xtask bench

Re-analyze existing data

cargo xtask bench --no-run --index

Benchmark a specific test

cargo xtask bench integers
# Only runs benchmarks matching "integers"

Performance Analysis Tips

1. •

   Focus on the Markdown report first (perf/RESULTS.md)

   • •Easy to grep, parse, and read
   • •Shows all critical metrics in one place
   • •Auto-categorized by performance tier
2. •

   Use instruction counts, not just time

   • •More stable than wall-clock time
   • •Architecture-independent
   • •Appears in "vs serde_json" column when available
3. •

   Look for patterns in the Summary section

   • •"Needs Work" items are optimization targets
   • •"Wins" validate current approach
   • •"Close" items are low-hanging fruit
4. •

   Compare Tier-1 vs Tier-2 JIT

   • •Large gaps = Tier-2 not implemented or buggy
   • •Similar performance = Tier-2 working but not optimized
   • •Tier-2 wins = format-specific optimizations paying off

Troubleshooting

Benchmarks fail to compile

# Regenerate from KDL
cargo xtask gen-benchmarks

Parser errors in output

• •Check bench-reports/divan-*.txt or gungraun-*.txt for malformed output
• •Fix the benchmark code, not the parser (usually)

Missing benchmarks in report

• •Ensure benchmark has category in benchmarks.kdl
• •Check that cargo xtask gen-benchmarks ran successfully
• •Verify benchmark functions are generated (check unified_benchmarks_*.rs)

--index fails

• •Ensure gh CLI is installed and authenticated
• •Check network connection (clones from GitHub)
• •Try --index without --push first

See Also

• •divan docs: https://docs.rs/divan/
• •gungraun: Custom fork with valgrind integration
• •Nextest valgrind profile: .config/nextest.toml
• •Benchmark generator: tools/benchmark-generator/
• •Report analyzer: tools/benchmark-analyzer/