12-Factor Agents Compliance Analysis

Reference: 12-Factor Agents

Input Parameters

Analysis Framework

Factor 1: Natural Language to Tool Calls

Principle: Convert natural language inputs into structured, deterministic tool calls using schema-validated outputs.

Search Patterns:

# Look for Pydantic schemas
grep -r "class.*BaseModel" --include="*.py"
grep -r "TaskDAG\|TaskResponse\|ToolCall" --include="*.py"

# Look for JSON schema generation
grep -r "model_json_schema\|json_schema" --include="*.py"

# Look for structured output generation
grep -r "output_type\|response_model" --include="*.py"

File Patterns: **/agents/*.py, **/schemas/*.py, **/models/*.py

Compliance Criteria:

Anti-patterns:

• •json.loads(llm_response) without schema validation
• •output.split() or regex parsing of LLM responses
• •dict[str, Any] return types from agents
• •No validation between LLM output and handler execution

Factor 2: Own Your Prompts

Principle: Treat prompts as first-class code you control, version, and iterate on.

Search Patterns:

# Look for embedded prompts
grep -r "SYSTEM_PROMPT\|system_prompt" --include="*.py"
grep -r '""".*You are' --include="*.py"

# Look for template systems
grep -r "jinja\|Jinja\|render_template" --include="*.py"
find . -name "*.jinja2" -o -name "*.j2"

# Look for prompt directories
find . -type d -name "prompts"

File Patterns: **/prompts/**, **/templates/**, **/agents/*.py

Compliance Criteria:

Anti-patterns:

• •f"You are a {role}..." inline in agent methods
• •Prompts mixed with business logic
• •No way to iterate on prompts without code changes
• •No prompt versioning or A/B testing capability

Factor 3: Own Your Context Window

Principle: Control how history, state, and tool results are formatted for the LLM.

Search Patterns:

# Look for context/message management
grep -r "AgentMessage\|ChatMessage\|messages" --include="*.py"
grep -r "context_window\|context_compiler" --include="*.py"

# Look for custom serialization
grep -r "to_xml\|to_context\|serialize" --include="*.py"

# Look for token management
grep -r "token_count\|max_tokens\|truncate" --include="*.py"

File Patterns: **/context/*.py, **/state/*.py, **/core/*.py

Compliance Criteria:

Anti-patterns:

• •Unbounded message accumulation
• •Large artifacts embedded inline (diffs, files)
• •No agent-specific context filtering
• •Same context for all agent types

Factor 4: Tools Are Structured Outputs

Principle: Tools produce schema-validated JSON that triggers deterministic code, not magic function calls.

Search Patterns:

# Look for tool/response schemas
grep -r "class.*Response.*BaseModel" --include="*.py"
grep -r "ToolResult\|ToolOutput" --include="*.py"

# Look for deterministic handlers
grep -r "def handle_\|def execute_" --include="*.py"

# Look for validation layer
grep -r "model_validate\|parse_obj" --include="*.py"

File Patterns: **/tools/*.py, **/handlers/*.py, **/agents/*.py

Compliance Criteria:

Anti-patterns:

• •Tool handlers that directly execute LLM output
• •eval() or exec() on LLM-generated code
• •No separation between decision (LLM) and execution (code)
• •Magic method dispatch based on string matching

Factor 5: Unify Execution State

Principle: Merge execution state (step, retries) with business state (messages, results).

Search Patterns:

# Look for state models
grep -r "ExecutionState\|WorkflowState\|Thread" --include="*.py"

# Look for dual state systems
grep -r "checkpoint\|MemorySaver" --include="*.py"
grep -r "sqlite\|database\|repository" --include="*.py"

# Look for state reconstruction
grep -r "load_state\|restore\|reconstruct" --include="*.py"

File Patterns: **/state/*.py, **/models/*.py, **/database/*.py

Compliance Criteria:

Anti-patterns:

• •Retry count stored separately from task state
• •Error history in logs but not in state
• •LangGraph checkpoints + separate database storage
• •No unified event thread

Factor 6: Launch/Pause/Resume

Principle: Agents support simple APIs for launching, pausing at any point, and resuming.

Search Patterns:

# Look for REST endpoints
grep -r "@router.post\|@app.post" --include="*.py"
grep -r "start_workflow\|pause\|resume" --include="*.py"

# Look for interrupt mechanisms
grep -r "interrupt_before\|interrupt_after" --include="*.py"

# Look for webhook handlers
grep -r "webhook\|callback" --include="*.py"

File Patterns: **/routes/*.py, **/api/*.py, **/orchestrator/*.py

Compliance Criteria:

Anti-patterns:

• •Blocking input() or confirm() calls
• •No way to resume after process restart
• •Approval only at plan level, not per-tool
• •No webhook-based resume from external systems

Factor 7: Contact Humans with Tools

Principle: Human contact is a tool call with question, options, and urgency.

Search Patterns:

# Look for human input mechanisms
grep -r "typer.confirm\|input(\|prompt(" --include="*.py"
grep -r "request_human_input\|human_contact" --include="*.py"

# Look for approval patterns
grep -r "approval\|approve\|reject" --include="*.py"

# Look for structured question formats
grep -r "question.*options\|HumanInputRequest" --include="*.py"

File Patterns: **/agents/*.py, **/tools/*.py, **/orchestrator/*.py

Compliance Criteria:

Anti-patterns:

• •typer.confirm() in agent code
• •Human contact hardcoded at specific graph nodes
• •No way for agents to ask clarifying questions
• •Single response format (yes/no only)

Factor 8: Own Your Control Flow

Principle: Custom control flow, not framework defaults. Full control over routing, retries, compaction.

Search Patterns:

# Look for routing logic
grep -r "add_conditional_edges\|route_\|should_continue" --include="*.py"

# Look for custom loops
grep -r "while True\|for.*in.*range" --include="*.py" | grep -v test

# Look for execution mode control
grep -r "execution_mode\|agentic\|structured" --include="*.py"

File Patterns: **/orchestrator/*.py, **/graph/*.py, **/core/*.py

Compliance Criteria:

Anti-patterns:

• •Single path through graph with no branching
• •No distinction between tool types (all treated same)
• •Framework-default error handling only
• •No rate limiting or resource management

Factor 9: Compact Errors into Context

Principle: Errors in context enable self-healing. Track consecutive errors, escalate after threshold.

Search Patterns:

# Look for error handling
grep -r "except.*Exception\|error_history\|consecutive_errors" --include="*.py"

# Look for retry logic
grep -r "retry\|backoff\|max_attempts" --include="*.py"

# Look for escalation
grep -r "escalate\|human_escalation" --include="*.py"

File Patterns: **/agents/*.py, **/orchestrator/*.py, **/core/*.py

Compliance Criteria:

Anti-patterns:

• •logger.error() without adding to context
• •No retry mechanism (fail immediately)
• •No consecutive error tracking
• •No escalation to humans after repeated failures

Factor 10: Small, Focused Agents

Principle: Each agent has narrow responsibility, 3-10 steps max.

Search Patterns:

# Look for agent classes
grep -r "class.*Agent\|class.*Architect\|class.*Developer" --include="*.py"

# Look for step definitions
grep -r "steps\|tasks" --include="*.py" | head -20

# Count methods per agent
grep -r "async def\|def " agents/*.py 2>/dev/null | wc -l

File Patterns: **/agents/*.py

Compliance Criteria:

Anti-patterns:

• •Single agent with 20+ tools
• •Agent with unbounded step count
• •Mixed responsibilities (planning + execution + review)
• •No step or time limits on agent execution

Factor 11: Trigger from Anywhere

Principle: Workflows triggerable from CLI, REST, WebSocket, Slack, webhooks, etc.

Search Patterns:

# Look for entry points
grep -r "@cli.command\|@router.post\|@app.post" --include="*.py"

# Look for WebSocket support
grep -r "WebSocket\|websocket" --include="*.py"

# Look for external integrations
grep -r "slack\|discord\|webhook" --include="*.py" -i

File Patterns: **/routes/*.py, **/cli/*.py, **/main.py

Compliance Criteria:

Anti-patterns:

• •Only if __name__ == "__main__" entry point
• •No REST API for external systems
• •No event streaming for real-time updates
• •Trigger logic tightly coupled to execution

Factor 12: Stateless Reducer

Principle: Agents as pure functions: (state, input) -> (state, output). No side effects in agent logic.

Search Patterns:

# Look for state mutation patterns
grep -r "\.status = \|\.field = " --include="*.py"

# Look for immutable updates
grep -r "model_copy\|\.copy(\|with_" --include="*.py"

# Look for side effects in agents
grep -r "write_file\|subprocess\|requests\." agents/*.py 2>/dev/null

File Patterns: **/agents/*.py, **/nodes/*.py

Compliance Criteria:

Anti-patterns:

• •state.field = new_value (mutation)
• •File writes inside agent methods
• •HTTP calls inside agent decision logic
• •Shared mutable state between agents

Factor 13: Pre-fetch Context

Principle: Fetch likely-needed data upfront rather than mid-workflow.

Search Patterns:

# Look for context pre-fetching
grep -r "pre_fetch\|prefetch\|fetch_context" --include="*.py"

# Look for RAG/embedding systems
grep -r "embedding\|vector\|semantic_search" --include="*.py"

# Look for related file discovery
grep -r "related_tests\|similar_\|find_relevant" --include="*.py"

File Patterns: **/context/*.py, **/retrieval/*.py, **/rag/*.py

Compliance Criteria:

Anti-patterns:

• •Architect starts with issue only, no codebase context
• •No semantic search for similar past work
• •Related tests/files discovered only during execution
• •No RAG or document retrieval system

Output Format

Executive Summary Table

| Factor | Status | Notes |
|--------|--------|-------|
| 1. Natural Language -> Tool Calls | **Strong/Partial/Weak** | [Key finding] |
| 2. Own Your Prompts | **Strong/Partial/Weak** | [Key finding] |
| ... | ... | ... |
| 13. Pre-fetch Context | **Strong/Partial/Weak** | [Key finding] |

**Overall**: X Strong, Y Partial, Z Weak

Per-Factor Analysis

For each factor, provide:

1. •

   Current Implementation

   • •Evidence with file:line references
   • •Code snippets showing patterns
2. •

   Compliance Level

   • •Strong/Partial/Weak with justification
3. •

   Gaps

   • •What's missing vs. 12-Factor ideal
4. •

   Recommendations

   • •Actionable improvements with code examples

Analysis Workflow

1. •

   Initial Scan

   • •Run search patterns for all factors
   • •Identify key files for each factor
   • •Note any existing compliance documentation
2. •

   Deep Dive (per factor)

   • •Read identified files
   • •Evaluate against compliance criteria
   • •Document evidence with file paths
3. •

   Gap Analysis

   • •Compare current vs. 12-Factor ideal
   • •Identify anti-patterns present
   • •Prioritize by impact
4. •

   Recommendations

   • •Provide actionable improvements
   • •Include before/after code examples
   • •Reference roadmap if exists
5. •

   Summary

   • •Compile executive summary table
   • •Highlight strengths and critical gaps
   • •Suggest priority order for improvements

Quick Reference: Compliance Scoring

When to Use This Skill

• •Evaluating new LLM-powered systems
• •Reviewing agent architecture decisions
• •Auditing production agentic applications
• •Planning improvements to existing agents
• •Comparing frameworks or implementations