OpenTelemetry Instrumentation Extension

Name: OpenTelemetry Instrumentation Extension
Rating: 92
Author: docker

Automatically extend OpenTelemetry instrumentation for new functionality in the MCP Gateway, following established patterns documented in docs/telemetry/README.md.

When to Use This Skill

Automatically apply when:

•New state-changing operations added (Create, Update, Delete, Push, Pull, Add, Remove, etc.)
•New CLI commands added to cmd/docker-mcp/
•New packages with operations in pkg/
•User mentions "otel", "telemetry", "instrumentation", "metrics", or "tracing"
•Code changes that modify state (database, files, containers, configuration)
•Reviewing code for telemetry coverage

Workflow

Phase 1: Analysis & Suggestion

•
Read project telemetry standards:
- •Read docs/telemetry/README.md "Development Guidelines" section
- •Read pkg/telemetry/telemetry.go to understand existing patterns and metrics
•
Identify scope using git diff:
- •Find new/changed files in pkg/ and cmd/docker-mcp/
- •Identify functions performing state-changing operations
- •Infer domain from package structure (e.g., pkg/foo/ → domain: foo)
•
Categorize findings:
- •Operations in existing domains (use existing metrics)
- •Operations in new domains (need new metrics)
•
Present suggestions to user:
- •List each function needing instrumentation with file:line reference
- •Specify operation type (create, update, delete, etc.)
- •Identify domain and whether metrics exist
- •Show what will be added (instrumentation code, metrics, docs, tests)

Phase 2: Implementation (After User Approval)

Execute in this order:

•Make changes: Instrument operations and add metrics to pkg/telemetry/telemetry.go
•Verify: Build, run with OTEL collector, check docker logs otel-debug output
•Write tests: Add tests to pkg/telemetry/telemetry_test.go
•Update docs: Add metrics/operations to docs/telemetry/README.md
•Run tests: Execute make test
•Final verification: Run ./docs/telemetry/testing/test-telemetry.sh

Key Principles

Follow the project's documented guidelines from docs/telemetry/README.md:

•Use Existing Providers: Get global tracer/meter from OTEL
•Preserve Server Lineage: Include server attribution in all telemetry
•Non-Blocking Operations: Telemetry never blocks or fails operations
•Debug Support: Add logging behind DOCKER_MCP_TELEMETRY_DEBUG
•Follow Naming Conventions: Use mcp.<domain>.<field> pattern
•Deferred Success Tracking: Only set success on clean completion

Where to Find Information

ALWAYS read these files before suggesting changes:

•
docs/telemetry/README.md - Source of truth for:
- •Development guidelines (lines 419-474)
- •Existing metrics and attributes
- •Testing procedures
- •Naming conventions
•
pkg/telemetry/telemetry.go - Understand:
- •Existing metric instruments
- •Recording function patterns
- •Helper functions for spans
- •Init() structure
•
pkg/telemetry/telemetry_test.go - See:
- •Testing patterns
- •How to verify metrics

Instrumentation Pattern

Use the simple defer pattern from cmd/docker-mcp/catalog/create.go:

func OperationName(ctx context.Context, identifier string, ...) error {
    telemetry.Init()
    start := time.Now()
    var success bool
    defer func() {
        duration := time.Since(start)
        telemetry.Record<Domain>Operation(ctx, "operation_name", identifier,
            float64(duration.Milliseconds()), success)
    }()

    // ... operation logic ...

    // Optional: Record resource counts
    telemetry.Record<Domain><Resources>(ctx, identifier, int64(count))

    success = true
    return nil
}

Note: If identifier is generated during execution, the defer captures its final value.

Adding New Domains

When instrumentation is needed for a new domain (e.g., new package pkg/newdomain/):

1. Add Metric Instruments in `pkg/telemetry/telemetry.go`

In the Init() function, add global variables and create metric instruments:

var (
    // ... existing metrics ...

    // New domain metrics
    newdomainOperations        metric.Int64Counter
    newdomainOperationDuration metric.Float64Histogram
    newdomainResources         metric.Int64Gauge  // If managing resources
)

func Init() {
    // ... existing init code ...

    newdomainOperations, _ = meter.Int64Counter(
        "mcp.newdomain.operations",
        metric.WithDescription("New domain operations count"),
    )
    newdomainOperationDuration, _ = meter.Float64Histogram(
        "mcp.newdomain.operation.duration",
        metric.WithDescription("New domain operation duration in milliseconds"),
    )
    newdomainResources, _ = meter.Int64Gauge(
        "mcp.newdomain.resources",
        metric.WithDescription("Number of resources in new domain"),
    )
}

2. Add Recording Functions in `pkg/telemetry/telemetry.go`

After the Init() function, add recording functions:

func RecordNewdomainOperation(ctx context.Context, operation, identifier string, durationMs float64, success bool) {
    if newdomainOperations == nil || newdomainOperationDuration == nil {
        return
    }
    attrs := []attribute.KeyValue{
        attribute.String("mcp.newdomain.operation", operation),
        attribute.String("mcp.newdomain.id", identifier),  // or .name, .ref as appropriate
        attribute.Bool("mcp.newdomain.success", success),
    }
    newdomainOperations.Add(ctx, 1, metric.WithAttributes(attrs...))
    newdomainOperationDuration.Record(ctx, durationMs, metric.WithAttributes(attrs...))
}

// Optional: Add resource counting function if applicable
func RecordNewdomainResources(ctx context.Context, identifier string, count int64) {
    if newdomainResources == nil {
        return
    }
    attrs := []attribute.KeyValue{
        attribute.String("mcp.newdomain.id", identifier),
    }
    newdomainResources.Record(ctx, count, metric.WithAttributes(attrs...))
}

3. Write Tests in `pkg/telemetry/telemetry_test.go`

Add test cases following existing patterns:

func TestRecordNewdomainOperation(t *testing.T) {
    spanRecorder, metricReader := setupTestTelemetry(t)
    Init()
    ctx := context.Background()

    // Test successful operation
    RecordNewdomainOperation(ctx, "create", "test-id", 123.45, true)

    // Collect and verify metrics
    var rm metricdata.ResourceMetrics
    err := metricReader.Collect(ctx, &rm)
    require.NoError(t, err)

    // Find and verify the counter metric
    foundCounter := false
    foundHistogram := false
    for _, sm := range rm.ScopeMetrics {
        for _, m := range sm.Metrics {
            if m.Name == "mcp.newdomain.operations" {
                foundCounter = true
                sum := m.Data.(metricdata.Sum[int64])
                require.Len(t, sum.DataPoints, 1)
                assert.Equal(t, int64(1), sum.DataPoints[0].Value)

                // Verify attributes
                attrs := sum.DataPoints[0].Attributes
                assert.Contains(t, attrs.ToSlice(), attribute.String("mcp.newdomain.operation", "create"))
                assert.Contains(t, attrs.ToSlice(), attribute.String("mcp.newdomain.id", "test-id"))
                assert.Contains(t, attrs.ToSlice(), attribute.Bool("mcp.newdomain.success", true))
            }
            if m.Name == "mcp.newdomain.operation.duration" {
                foundHistogram = true
                histogram := m.Data.(metricdata.Histogram[float64])
                require.Len(t, histogram.DataPoints, 1)
                assert.Equal(t, float64(123.45), histogram.DataPoints[0].Sum)
            }
        }
    }

    assert.True(t, foundCounter, "Counter metric not found")
    assert.True(t, foundHistogram, "Histogram metric not found")
}

4. Update Documentation in `docs/telemetry/README.md`

Add a new section for the domain in the appropriate location. Follow the existing format:

markdown

### New Domain Operations

Operations for managing [description of what this domain does]:
- **`mcp.newdomain.operations`** - New domain operations (create, update, delete, etc.)
- **`mcp.newdomain.operation.duration`** - Duration of new domain operations
- **`mcp.newdomain.resources`** - Gauge showing number of resources in new domain

#### New Domain Attributes
- **`mcp.newdomain.operation`** - Type of operation (create, update, delete, etc.)
- **`mcp.newdomain.id`** - ID of the resource
- **`mcp.newdomain.success`** - Boolean indicating operation success

Verification

After making changes, verify telemetry output:

bash

# Build
make docker-mcp

# Start OTEL collector
docker run --rm -d --name otel-debug \
  -p 4317:4317 -p 4318:4318 \
  -v $(pwd)/docs/telemetry/testing/otel-collector-config.yaml:/config.yaml \
  otel/opentelemetry-collector:latest --config=/config.yaml

# Run with telemetry enabled
export DOCKER_MCP_TELEMETRY_DEBUG=1
export DOCKER_CLI_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
docker mcp [command]

# Check collector output
docker logs otel-debug | grep "mcp.newdomain"

# Cleanup
docker stop otel-debug

Verify the output matches expectations before proceeding to write tests and docs.

Analysis Strategy

When analyzing git diff:

•
Look for operation verbs in function names:
- •Create, Add, Update, Modify, Set, Configure, Register
- •Delete, Remove, Unregister, Clear
- •Push, Pull, Export, Import, Sync
- •Start, Stop, Run, Execute
•
Look for state changes:
- •Database operations (DAO/DB calls)
- •File system operations (create, delete, write)
- •Container operations (start, stop, create, delete)
- •Configuration changes (save, update)
•
Infer domain from file path:
- •pkg/workingset/ → domain: profile
- •pkg/catalog_next/ → domain: catalog_next
- •cmd/docker-mcp/server/ → domain: server
- •Pattern: use logical grouping name
•
Check if telemetry exists:
- •Search pkg/telemetry/telemetry.go for Record<Domain> functions
- •If exists: use existing metrics
- •If not: propose new domain metrics

Implementation Checklist

Follow this sequence:

• Read patterns from docs/telemetry/README.md
• Instrument operations with simple defer pattern
• Add new metrics to pkg/telemetry/telemetry.go (if new domain)
• Verify with docker logs otel-debug - confirm output matches expectations
• Write tests in pkg/telemetry/telemetry_test.go
• Update docs/telemetry/README.md with new metrics
• Run make test - verify tests pass
• Run ./docs/telemetry/testing/test-telemetry.sh - final verification

Important Notes

•Read documentation first
•Follow existing patterns
•Ask for approval before implementing
•Verify early and often with collector logs