Bug Hunting Workflow

Intent

This skill is for finding real bugs in the implementation, not guessing. It forces a discovery phase (build a mental model, reproduce, narrow scope) before making changes.

You must prioritize:

1. •correctness and durability
2. •clear reproduction
3. •minimal fix + regression test

Non-negotiable rules

• •No fix without a reproduction (a failing test, a minimal repro program, or a deterministic HTTP/CLI script).
• •No durability regressions: do not weaken WAL/manifest/atomic rename/fsync ordering.
• •No “drive-by refactors” while bug hunting; keep patches tight and reviewable.
• •After changes, you must run the full quality suite from code-quality.

Discovery phase (required)

0) Establish the boundaries

Before opening files, answer (in notes) these questions:

• •Which surface is affected? CLI, HTTP, embedded API, wasm, or ffi?
• •Is it a correctness issue (wrong results), durability issue (lost/duplicated data), or stability issue (panic/hang)?
• •Does it require a specific feature flag (vectors, zstd, gpu, wasm/IndexedDB, ffi)?
• •Is the failure deterministic or intermittent?

1) Baseline the repo (don’t trust your local state)

Run in workspace root:

• •cargo fmt --all
• •cargo clippy --all --all-features --all-targets -- -D warnings
• •cargo test --all --all-features

If anything fails here, you may be chasing a known failure; fix/resolve that first.

2) Map the architecture you’re about to debug

Create a quick map in notes (1–2 minutes):

• •searchlite-core: schema, indexing, WAL/manifest, segments, query execution, filters/aggs, vectors
• •searchlite-cli: user-facing commands and JSON output formatting
• •searchlite-http: endpoints (/init, /add, /bulk, /delete, /commit, /refresh, /search, /inspect, /stats, /compact), limits/timeouts/concurrency
• •searchlite-wasm: bindings + IndexedDB storage semantics (no fsync)
• •searchlite-ffi: C ABI / safety edges

This map guides where you look first.

3) Reproduce the bug with the smallest possible corpus

Produce a “minimal reproducible case” artifact set:

• •a schema JSON (small)
• •a small NDJSON/doc list (5–50 docs)
• •one request (HTTP JSON or CLI command) that fails

For HTTP:

• •include the server flags/env (bind addr, index path, max-body-bytes, max-concurrency, request-timeout, refresh-on-commit)

For CLI:

• •include exact commands and any flags

Goal: a script a teammate can run in <60 seconds.

4) Turn on observability (don’t add prints yet)

Use existing debugging features first:

• •For search correctness: add explain: true to the request.
• •For performance/odd slowdowns while reproducing: add profile: true.
• •For server behavior: run with RUST_LOG=debug (or more targeted module filters) and capture the relevant lines.

If your issue is “writes not visible,” verify lifecycle:

• •you must commit
• •you may need /refresh in HTTP depending on config

5) Narrow the fault line by bisecting the pipeline

Pick one axis at a time:

Correctness axis

• •Does the doc get ingested successfully?
• •Does it appear in /stats documents?
• •Does /inspect show a new segment after commit?
• •Does the query match in bm25 mode vs wand/bmw mode?

Durability axis

• •Is WAL appended correctly?
• •Is manifest updated atomically?
• •Is WAL truncated only after manifest persistence?
• •After restart, does replay do the expected thing?

HTTP axis

• •Is the request being rejected due to max body / timeouts / concurrency?
• •Is NDJSON streaming partial and rolling back?
• •Are errors correctly surfaced as {"error":{"type","reason"}}?

6) Perform a targeted static sweep of the suspected modules

Once you know the rough area, do a deliberate scan for bug smells:

Search patterns to grep/ripgrep for in the relevant crate:

• •panics: unwrap(), expect(, panic!, unreachable!, todo!
• •suspicious error handling: map_err(|_|, ok()? in non-optional contexts, silent let _ =
• •integer edge cases: casts (as usize), underflow (- 1), unchecked indexing
• •concurrency hazards: Mutex/RwLock lock ordering, .await while holding locks, channels without bounds
• •IO hazards: missing fsync, non-atomic file replace, partial writes not handled, directory fsync omissions
• •schema mismatch hazards: stringly-typed field lookups, analyzer lookup defaults, missing nested path validation
• •vector hazards: dimension mismatch, normalization assumptions, unchecked candidate sizing, NaN propagation

You are not “fixing” in this step—just identifying suspects.

7) Dynamic bug probes by subsystem (choose what fits)

A) WAL / manifest / commit / rollback

If you suspect commit/durability issues:

• •Reproduce a crash-window scenario:
  1. •ingest docs
  2. •start commit
  3. •force termination mid-way (or simulate via test hooks if present)
  4. •restart and observe replay
• •Validate the invariants:
  • •manifest atomic update (rename) and directory sync
  • •WAL truncation happens after manifest sync
  • •rollback cleans new segment artifacts

If you can’t safely crash in tests, create a unit test around writer/commit phases and validate state.

B) Segment lifecycle / compaction / cleanup

If you see duplicates, ghosts, or bloat:

• •Inspect segments via inspect and compare to stats
• •Run compaction and verify:
  • •live docs preserved
  • •deleted/tombstoned dropped
  • •old segment files removed (or expected to remain until cleanup)

C) Query execution modes

If you suspect scoring/pruning bugs:

• •Run the same query with execution=bm25, execution=wand, execution=bmw
• •Results should match for deterministic corpora (unless the contract explicitly says otherwise)
• •If they diverge, you likely have a pruning-bound bug or block-max metadata mismatch

D) Filters/aggs/nested correctness

If filters/aggs are wrong:

• •Verify required fields are fast: true
• •For nested, verify the filter uses a Nested block and that clauses bind to the same object instance
• •Add a repro doc with two nested objects that would catch “cross-object” leakage

E) Vectors (feature vectors)

If vector behavior is wrong:

• •Check dimension mismatch handling
• •Check cosine normalization expectations
• •Confirm ANN parameters (candidate_size, ef_search) don’t overflow or accept nonsensical values
• •Verify hybrid search doesn’t double-count or drop vector scores

Fix phase (required)

1) Write the failing regression test first

• •Prefer a unit test in the most local crate if possible.
• •If the bug is surface-level behavior (HTTP contract, CLI output), add an integration test.
• •Make it deterministic and small.
• •Assert on structure, not strings (parse JSON output).

2) Implement the minimal fix

• •Tight patch, minimal diff.
• •Keep durability semantics intact.
• •Avoid introducing allocations/locks in hot paths unless justified.

3) Prove it

Run:

• •cargo test --all --all-features
• •Any relevant examples (e.g. pruning example if you touched pruning logic)
• •If the bug touched performance-sensitive code, run the relevant bench before/after.

4) Add a “root cause” note

In the PR description or internal notes, include:

• •symptom
• •root cause
• •why the fix is correct
• •what test prevents regressions

Output format (what Codex should produce)

When using this skill, produce:

1. •Repro artifact (schema + docs + request/command)
2. •Fault localization (files + functions involved)
3. •Root cause analysis
4. •Fix plan (minimal changes)
5. •Regression test (where it lives and what it asserts)
6. •Validation commands run (from code-quality)