Datadog Monitoring

This skill provides access to Datadog for monitoring, logging, and alerting via the Datadog API.

Setup Required

You need to set up API credentials:

•Go to Datadog → Organization Settings → API Keys
•Create or copy an API Key
•Go to Organization Settings → Application Keys
•Create an Application Key

Set these as environment variables (add to your shell profile or .env):

bash

export DD_API_KEY="your-api-key"
export DD_APP_KEY="your-application-key"
export DD_SITE="us3.datadoghq.com"  # Your Datadog site (from browser history: us3)

When to Use

Use this skill when the user:

•Asks about logs, errors, or application behavior
•Wants to check monitor/alert status
•Needs to investigate an incident
•Asks about metrics or performance
•Mentions "Datadog" or monitoring

API Endpoints

Base URL: https://api.$(printenv DD_SITE)/api/v1 or v2

Logs

Search Logs (POST /api/v2/logs/events/search):

bash

curl -s -X POST "https://api.$(printenv DD_SITE)/api/v2/logs/events/search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "query": "service:my-service status:error",
      "from": "now-1h",
      "to": "now"
    },
    "sort": "-timestamp",
    "page": {"limit": 50}
  }'

Common log query filters:

•service:name - Filter by service
•status:error - Filter by log level (error, warn, info, debug)
•@http.status_code:500 - Filter by HTTP status
•host:hostname - Filter by host
•env:production - Filter by environment

Monitors (Alerts)

List All Monitors (GET /api/v1/monitor):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/monitor" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Get Monitor by ID (GET /api/v1/monitor/{id}):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/monitor/{MONITOR_ID}" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Search Monitors:

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/monitor?query=status:Alert" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Metrics

Query Metrics (GET /api/v1/query):

bash

curl -s -G "https://api.$(printenv DD_SITE)/api/v1/query" \
  --data-urlencode "query=avg:system.cpu.user{*}" \
  --data-urlencode "from=$(date -v-1H +%s)" \
  --data-urlencode "to=$(date +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

List Available Metrics (GET /api/v1/metrics):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/metrics?from=$(date -v-1d +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Events

Query Events (GET /api/v1/events):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/events?start=$(date -v-1d +%s)&end=$(date +%s)" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Dashboards

List Dashboards (GET /api/v1/dashboard):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v1/dashboard" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Incidents

List Incidents (GET /api/v2/incidents):

bash

curl -s "https://api.$(printenv DD_SITE)/api/v2/incidents" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)"

Common Workflows

Check for Recent Errors

bash

# Search for error logs in the last hour
curl -s -X POST "https://api.$(printenv DD_SITE)/api/v2/logs/events/search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "query": "status:error",
      "from": "now-1h",
      "to": "now"
    },
    "page": {"limit": 25}
  }' | jq '.data[] | {timestamp: .attributes.timestamp, message: .attributes.message, service: .attributes.service}'

Check Alert Status

bash

# List monitors that are currently alerting
curl -s "https://api.$(printenv DD_SITE)/api/v1/monitor?query=status:Alert" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" | jq '.[] | {name, overall_state, message}'

Investigate a Service

bash

# Get logs for a specific service
curl -s -X POST "https://api.$(printenv DD_SITE)/api/v2/logs/events/search" \
  -H "DD-API-KEY: $(printenv DD_API_KEY)" \
  -H "DD-APPLICATION-KEY: $(printenv DD_APP_KEY)" \
  -H "Content-Type: application/json" \
  -d '{
    "filter": {
      "query": "service:SERVICE_NAME",
      "from": "now-30m",
      "to": "now"
    },
    "page": {"limit": 100}
  }'

Log Query Syntax

Datadog uses a powerful query syntax for logs:

Operator	Example	Description
AND	`service:api status:error`	Both conditions (implicit)
OR	`status:error OR status:warn`	Either condition
NOT	`-status:debug`	Exclude matches
Wildcard	`service:api-*`	Pattern matching
Range	`@duration:>1000`	Numeric comparisons
Exists	`@http.url:*`	Field exists

Time Ranges

For the from and to parameters:

•now - Current time
•now-1h - 1 hour ago
•now-1d - 1 day ago
•now-7d - 1 week ago
•Unix timestamps (seconds)

Notes

•Your Datadog site appears to be us3.datadoghq.com based on browser history
•API rate limits apply - be mindful of query frequency
•Log queries return max 1000 results per request; use pagination for more
•Use jq to parse JSON responses
•Monitor status values: OK, Alert, Warn, No Data