Observability Analysis

Query construction and analysis for Prometheus, Loki, and Tempo.

Core Principles

Start with all available metrics then drill down to logs and traces for context.

Progressive Query Construction

• •Start simple → Add filters → Add operations → Optimize
• •Test incrementally to validate each step
• •Adjust based on data characteristics

Multi-Signal Correlation

• •Metrics → Identify anomaly (what/when/how much)
• •Traces → Map request flow (where/which services)
• •Logs → Extract details (why/error messages)
• •Use trace_id, service.name, timestamp for correlation

Token-Efficient Results

## Finding: [One-sentence summary]

**Evidence**: [Specific values/metrics]
**Impact**: [User/business effect]
**Cause**: [Root issue if identified]
**Action**: [Next step]

Target: <500 tokens for complete analysis

Query Patterns

Common starting points (adapt based on context):

# Metrics: Error rate, latency percentiles, traffic patterns
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
histogram_quantile(0.95, sum by (le) (rate(http_duration_bucket[5m])))
sum(rate(http_requests_total[5m])) by (endpoint)

# Logs: Error details, slow operations
{job="service"} |= "error" | json
{job="service"} | json | unwrap duration_ms | duration_ms > threshold

# Traces: Error traces, slow requests, request flow
{status=error && service.name="service"}
{duration > threshold && service.name="service"}
{kind="server" && service.name="service"}

Query Construction Guidelines

Labels: Use specific labels, avoid high cardinality aggregations Time ranges: Match analysis needs (5m for rate, adjust as needed) Aggregations: Filter first, then aggregate for efficiency

Result Interpretation

Extract key information:

• •Magnitude: Absolute values and comparisons
• •Trend: Direction and velocity of change
• •Scope: Affected components/users
• •Timing: When changes occurred

Quantify impact: Convert metrics to business/user impact Prioritize: Focus on severity, scope, and trend

Reference Documentation

Consult references for detailed syntax, patterns, and workflows:

• •references/promql.md - PromQL functions, RED/USE methods, optimization patterns
• •references/logql.md - LogQL parsers, aggregations, pipeline optimization
• •references/traceql.md - TraceQL span filtering, structural queries, performance analysis
• •references/semantic-conventions.md - OpenTelemetry attribute standards and naming
• •references/analysis-patterns.md - Token-efficient templates, output formats, examples
• •references/troubleshooting.md - Investigation workflows, scenario-specific patterns

When to use references:

• •Need specific syntax or advanced query patterns
• •Unfamiliar with query language features
• •Complex troubleshooting scenarios
• •Semantic convention lookups

Behavior

DO:

• •Construct queries progressively and test incrementally
• •Quantify findings with specific numbers and comparisons
• •Present insights in structured, token-efficient format
• •Focus on actionable, high-impact information
• •Lead with conclusions

DON'T:

• •Over-explain investigation process or basic concepts
• •Include unnecessary query variations
• •Generate instrumentation code or alert rules
• •Overwhelm with excessive findings (prioritize top issues)

Success Criteria

Effective analysis provides:

• •Concise findings (<500 tokens for complete analysis)
• •Specific evidence (numbers, comparisons, trends)
• •Clear impact assessment
• •Actionable next steps
• •Structured presentation