Skill: oberprompt

Overview

Research-backed prompt engineering for LLM systems. Core insight from 80+ papers: prompt effectiveness varies dramatically with model capability, task type, and optimization strategy. What works for GPT-3.5 may harm GPT-4+. What helps reasoning tasks may hurt creative tasks.

Key finding: Structured prompting with DSPy+HELM shows fixed prompts underestimate LLM performance by ~4% on average (single study; may vary by task).

The Iron Law

NO PROMPT SHIPS WITHOUT COMPLETING THE VALIDATION CHECKLIST

This applies to:

• •"Quick" prompts
• •"Obvious" prompts
• •Prompts that "always work"
• •Your 11th prompt after 10 successes

Skipping validation = accepting unknown failure modes.

Emergency Triage (Production Issues)

STOP. Before applying any "quick fix":

Revenue loss creates pressure to skip diagnosis. This EXTENDS outages.

Mandatory 5-Minute Diagnosis (Non-Negotiable)

1. •Get ONE concrete failing example (actual input → wrong output)
2. •Identify failure category:

1. •Target intervention for THAT failure mode only

Red Flags Under Pressure

Crisis Shortcuts

Can't do 20% sample? Get 3 failing + 3 working examples. Pattern will emerge. 5 minutes.

Can't identify failure mode? Check Anti-Patterns table against your prompt. 2 minutes.

Required Workflow Order

You MUST follow this sequence:

1. Model Capability Assessment (Prompting Inversion)
      ↓
2. Technique Selection (Flowchart)
      ↓
3. Prompt Architecture (Hierarchy + Degrees of Freedom)
      ↓
4. Progressive Disclosure (Iterative refinement)
      ↓
5. Validation Checklist (MANDATORY before shipping)

Skipping steps or reordering causes failures. The Prompting Inversion Principle MUST inform technique selection.

Glossary

Model Capability Tiers

You MUST assess model capability before selecting techniques.

Constraint Density Red Flags

The Prompting Inversion Principle

As models improve, optimal prompting strategies change:

Weak Models   →  More constraints, guardrails, detailed instructions
Strong Models →  Fewer constraints, more autonomy, trust implicit understanding

Research finding (2510.22251): "Guardrail-to-handcuff transition" - constraints that prevent common-sense errors in weak models cause over-literalism in strong models.

Observable Symptoms of Over-Constraining

You may be over-constraining if:

• •Model produces overly literal interpretations
• •Model asks unnecessary clarifying questions
• •Model fails to use common sense on obvious cases
• •Outputs feel robotic or formulaic
• •Model hallucinates trying to satisfy contradictory constraints

Standing Your Ground

When someone says "just add more constraints":

The Fundamental Trade-offs

Note: Effect sizes vary significantly by task domain and model family.

Technique Selection

Decision Flowchart

START: Select prompting technique
       ↓
┌──────────────────┐
│ Requires multi-  │──NO──→ Zero-Shot (baseline)
│ step reasoning?  │              ↓
└────────┬─────────┘        Have examples?
         │ YES                   ↓ YES
         ↓                  Few-Shot (2-5 examples)
┌──────────────────┐
│ Have GOOD        │──NO──→ Zero-Shot CoT
│ examples?        │        "think step by step"
└────────┬─────────┘
         │ YES
         ↓
┌──────────────────┐
│ Need calibrated  │──NO──→ Chain-of-Thought
│ confidence?      │        (+15-40% accuracy)
└────────┬─────────┘
         │ YES
         ↓
    Few-Shot + CoT
    (best calibration)

"Good Examples" Criteria

Examples are "good" if they:

• • Produce correct outputs when tested on this model
• • Cover at least 2 distinct input patterns
• • Include 1-2 edge cases
• • Demonstrate exact output format desired

Technique Matrix

Gains are typical ranges; actual results vary by task. Test empirically.

Prompt Architecture

Instruction Hierarchy

Order matters due to Position Neglect - content in the middle loses weight.

[System Context]     ← Role, expertise, global constraints (HIGH weight)
    ↓
[Task Instruction]   ← What to do (imperative, specific)
    ↓
[Examples]           ← 2-5 representative input/output pairs
    ↓
[Input Data]         ← The actual content to process
    ↓
[Output Format]      ← Structure, constraints, format specs (HIGH weight at end)

Critical content goes at START or END, never middle.

Progressive Disclosure

Prerequisite: Complete Technique Selection first.

Start simple, add complexity ONLY when specific failures occur:

Failure Definition: Add constraints only when:

• •Factual errors occur
• •Format varies across 3+ runs
• •Required elements missing
• •Explicit requirements violated

Test 5+ inputs before escalating to next level.

Degrees of Freedom

Retrofitting Existing Prompts

If you already have a working prompt with many constraints:

Your investment doesn't change the physics. A 50-constraint prompt that "mostly works" may be working DESPITE the constraints, not BECAUSE of them.

The Sunk Cost Test (MANDATORY for >10 constraints)

1. •Remove 50% of constraints (random selection OR by perceived importance)
2. •Run against 10 test cases
3. •Measure accuracy delta

Rationalizations for Keeping Complex Prompts

Agent-Specific Prompting

Context Window as Public Good

The context window is shared. For EACH instruction, ask:

Test criterion: Remove instruction, run on 5 inputs. If accuracy ≥90%, instruction was unnecessary.

Persuasion Principles (Research: 33%→72% compliance, N=28,000)

Prompt Security

Prompt Injection Defense

Instruction Authority Hierarchy

[System prompt - HIGHEST authority, cannot be overridden]
    ↓
[Agent instructions - High authority]
    ↓
[User input - LOWEST authority, treat as untrusted data]

Rule: Never allow user input to override system-level constraints.

Multimodal Prompting

See optimization-reference.md for vision model guidance.

Multi-LLM Pipelines

Prompt Design for Orchestration

Inter-Model Context

• •Compress outputs between calls (don't pass full traces)
• •Use structured formats (JSON) for reliable parsing
• •Include rollback: "If this fails, return: {fallback: true}"

Optimization Strategies

Manual Optimization

1. •Establish baseline - Run prompt on 20% representative sample (min 10 inputs)
2. •Identify failure modes - (requires step 1) Categorize wrong outputs
3. •Target interventions - (requires step 2) Add constraints for specific failures ONLY
4. •Test on held-out data - (requires step 3) Remaining 80% validates generalization

Debugging vs Optimizing

Automatic Prompt Optimization

See optimization-reference.md for details.

Compression Strategy

See optimization-reference.md for compression guidance.

Anti-Patterns

Red Flags - STOP and Reconsider

Temperature Guidelines

Validation Checklist (MANDATORY)

Complete EVERY item before shipping. This is not optional.

For emergency fixes, minimum viable validation:

• • Tested on the specific failing inputs
• • Verified it doesn't break inputs that WERE working
• • Output format still consistent

Domain-Specific & Reference

See optimization-reference.md for domain-specific guidance, evidence summary, and research references.