Datadog Operations Skill

Comprehensive Datadog automation: query APM/logs/metrics/RUM/database, create monitors/dashboards/synthetics, manage incidents, trigger workflows, analyze costs and LLM usage.

Quick Setup

macOS

bash

./setup.sh

Linux

bash

./setup-linux.sh

Windows (PowerShell as Administrator)

powershell

.\setup-windows.ps1

Platform-Specific Setup

macOS

Install jq (required for JSON processing):

bash

# Homebrew (recommended)
brew install jq

# MacPorts
sudo port install jq

# Direct binary (Apple Silicon)
curl -L -o ~/bin/jq https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-macos-arm64
chmod +x ~/bin/jq

# Direct binary (Intel)
curl -L -o ~/bin/jq https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-macos-amd64
chmod +x ~/bin/jq

Set environment variables in ~/.zshrc:

bash

export DD_API_KEY="your_api_key"
export DD_APP_KEY="your_application_key"
export DD_SITE="datadoghq.com"

Linux

Install dependencies:

bash

# Debian/Ubuntu
sudo apt-get install -y jq curl bc

# Fedora/RHEL
sudo dnf install -y jq curl bc

# Arch
sudo pacman -S jq curl bc

Set environment variables in ~/.bashrc:

bash

export DD_API_KEY="your_api_key"
export DD_APP_KEY="your_application_key"
export DD_SITE="datadoghq.com"

Windows

Option 1: WSL (Recommended)

bash

wsl --install
# Then run Linux setup inside WSL
./setup-linux.sh

Option 2: Git Bash Install Git for Windows, then run ./setup.sh in Git Bash.

Option 3: Native PowerShell

powershell

# Install jq
winget install jqlang.jq
# Or: choco install jq

# Set environment variables
[Environment]::SetEnvironmentVariable("DD_API_KEY", "your_key", "User")
[Environment]::SetEnvironmentVariable("DD_APP_KEY", "your_app_key", "User")

Use Python scripts directly: python python\script_name.py

Required Permissions

Get keys from Datadog → Organization Settings → API Keys / Application Keys

For workflow automation (trigger-workflow): Enable "Actions API Access" on your app key: Datadog → Organization Settings → Application Keys → Click key → Enable "Actions API Access"

Quick Reference

Investigation Scripts

Script	Purpose	Example
`query-apm.sh`	Find slow endpoints	`bash scripts/query-apm.sh --service my-service --duration 1h`
`search-logs.sh`	Search logs for errors	`bash scripts/search-logs.sh --query "status:error" --duration 1h`
`query-security-signals.sh`	Find security threats	`bash scripts/query-security-signals.sh --severity critical`
`query-watchdog.sh`	Anomaly detection	`bash scripts/query-watchdog.sh --service my-service`
`query-metrics.sh`	Fetch metrics data	`bash scripts/query-metrics.sh --metric "system.cpu.user"`
`analyze-usage-cost.sh`	FinOps cost analysis	`bash scripts/analyze-usage-cost.sh --duration 30d`
`analyze-llm.sh`	LLM observability	`bash scripts/analyze-llm.sh --service my-llm-app`
`query-slos.sh`	SLO status	`bash scripts/query-slos.sh --service payment-api`
`query-service-catalog.sh`	Service metadata	`bash scripts/query-service-catalog.sh list`
`query-database.sh`	DB performance	`bash scripts/query-database.sh --host postgres-prod`
`query-rum.sh`	Frontend performance	`bash scripts/query-rum.sh --application abc-123`
`query-kubernetes.sh`	K8s workloads	`bash scripts/query-kubernetes.sh --cluster prod`
`query-containers.sh`	Container metrics	`bash scripts/query-containers.sh --duration 1h`
`query-network.sh`	Network monitoring	`bash scripts/query-network.sh --duration 1h`

Automation Scripts

Script	Purpose	Example
`manage-monitors.sh`	Create/mute monitors	`bash scripts/manage-monitors.sh list`
`create-dashboard.sh`	Generate dashboards	`bash scripts/create-dashboard.sh --service my-service`
`trigger-workflow.sh`	Execute workflows	`bash scripts/trigger-workflow.sh list`
`manage-incidents.sh`	Incident management	`bash scripts/manage-incidents.sh list`
`manage-synthetics.sh`	Synthetic tests	`bash scripts/manage-synthetics.sh list`
`manage-on-call.sh`	On-call scheduling	`bash scripts/manage-on-call.sh list`
`manage-status-pages.sh`	Status pages	`bash scripts/manage-status-pages.sh list`
`verify-setup.sh`	Validate config	`bash scripts/verify-setup.sh`

Workflows

Investigate Production Issue

bash

bash scripts/query-watchdog.sh --service affected-service --duration 24h
bash scripts/query-apm.sh --service affected-service --duration 1h
bash scripts/search-logs.sh --service affected-service --status error --duration 1h

Check Cost & Usage

bash

bash scripts/analyze-usage-cost.sh --duration 30d --product all
bash scripts/analyze-usage-cost.sh --duration 30d | jq '.recommendations[] | select(.priority == "high")'

Monitor LLM Application

bash

bash scripts/analyze-llm.sh --service my-genai-app --duration 24h

Output Format

All scripts return structured JSON to stdout, status messages to stderr:

bash

bash scripts/query-apm.sh --service my-service 2>/dev/null | jq '.summary'

Python Alternative

Python versions available in python/ directory:

bash

python3 -m venv .venv && source .venv/bin/activate
pip install -r python/requirements.txt
python python/query_apm.py --service my-service --json

Go CLI Alternative

Single-binary CLI available in ../dd-skill-test-go/:

bash

# Native on all platforms - no dependencies
datadog-cli apm --service my-service --duration 1h
datadog-cli logs --query "status:error"
datadog-cli monitors list
datadog-cli health

See ../dd-skill-test-go/README.md for installation.

Notes

•Advanced Scripts: Some scripts require Python. Run ./setup.sh first.
•Windows: Use WSL, Git Bash, or Python scripts directly.
•Go CLI: Recommended for Windows - native binary, no dependencies.