Perf Workflow
Overview
Analyze perf profiles end-to-end: summarize hotspots with perf report, drill down with perf annotate -l, then map to source and explain the exact hot statements and accessed data structures.
Workflow
Step 0: Locate input and binaries
- •By default, use
test/profile_output/index_latest_profile_perfandtest/profile_output/query_latest_profile_perf. - •If the defaults are missing, fall back to any
perf.data(or symlinks like*_latest_profile_perf) and note the binary/shared object shown inperf report. - •If symbols look stripped or source paths are wrong, plan to use
--symfs,--buildid-dir, orperf buildid-cache --add.
Step 1: Summarize hotspots
Run a non-interactive report to get top symbols and call chains:
perf report -i path/to/perf.data --stdio --no-children
Capture the top hot symbols (percent, shared object, symbol) and the relevant call path.
If the user explicitly asks to compare latest vs old, run the same report on both and compare:
- •Percent deltas for top symbols
- •New symbols entering/exiting the top set
- •Any shift between shared objects (e.g.,
postgresvsvector.so)
Step 2: Annotate hotspots with source mapping
O3 can break -s source correlation; use -l to get line-level mapping, then read the assembly around the hot lines.
perf annotate -i path/to/perf.data --stdio -l --symbol <symbol> --percent-limit 1
Prefer --symbol over full-file annotate to keep output focused.
Step 3: Annotate hotspots (comparison flow)
If the user explicitly asks to compare latest vs old, run annotate on the same symbols and compare:
- •Which source lines or basic blocks gained/lost percent
- •Whether the hot block moved to a different inlined path
- •Any difference in the memory access pattern (loads/stores) visible in assembly
Step 4: Map to source and explain data access
- •Use
rgto find the symbol definition. - •Use
nl -ba(or equivalent) to show the hot source lines by number. - •Identify accessed data structures/arrays by matching the assembly loads/stores to source statements.
rg -n "<symbol>" -S . nl -ba path/to/file.c | sed -n 'START,ENDp'
Step 5: Report results
Summarize each hotspot with:
- •Symbol + percent from
perf report - •Source file:line from
perf annotate -l - •The key statement(s) causing cost
- •The data accessed (arrays, structs, pointer-chasing)
- •Any lock/spin or memory access patterns seen in assembly
Practical Notes
- •Use
-lfor O3 builds;-soften mismatches inlined/optimized code. - •If
perf annotateshows only assembly, verify debug symbols and build-id cache. - •When data are in shared objects (e.g.,
vector.so), point to the matching source file in the repo, not just the.so.