name: oracle-consistency-auditor description: Audit consistency of repeated long-line oracle outputs and diagnose drift causes across routing, retrieval, and generation settings. Use when validating long-term ZiWei responses, regression testing, or before release quality gates.

Oracle Consistency Auditor

Overview

Evaluate whether repeated outputs for the same long-line query remain stable at conclusion, recommendation direction, and safety reminders.

Input Contract

•test_case_id
•prompt
•profile_summary
•outputs (2+ candidate outputs)
•run_metadata (model, temperature, retrieval scope, routing trace)

Workflow

•Extract each output into 3 layers:

•main_conclusion
•advice_direction
•risk_disclaimer

•Score pairwise consistency with references/scoring-rubric.md.
•Diagnose drift source: routing, retrieval, generation randomness, or memory summary.
•Provide fix plan ranked by impact.

Output Contract

Return:

•consistency_score (0-100)
•layer_scores (conclusion/advice/risk)
•drift_causes (ranked)
•fixes (P0/P1/P2)
•verification_steps

Quality Bar

•Never claim drift without textual evidence.
•Keep evidence snippets short and comparable.
•Recommend minimal effective changes first.

References

•Read references/scoring-rubric.md before scoring.