langsmith-tracing

Configure LangSmith observability and tracing for LLM applications built with LangChain and LangGraph frameworks.

Overview

LangSmith is the managed observability suite by LangChain that provides:

•Dashboards and alerting for LLM applications
•Human-in-the-loop evaluation capabilities
•Deep LangChain/LangGraph integration
•Run Tree model for nested traces
•MCP connectivity to Claude, VSCode

Capabilities

Core Tracing Setup

•Initialize LangSmith client and API configuration
•Configure project/workspace settings
•Set up trace collection and sampling
•Enable debug logging for agent execution

Integration Patterns

•LangChain chain tracing with automatic instrumentation
•LangGraph workflow state tracking
•Custom span creation for non-LangChain code
•Parent-child trace relationships

Debugging Features

•Fetch execution traces for analysis
•Query run history and metadata
•Export traces for offline analysis
•Compare runs across different versions

Usage

Environment Setup

bash

# Set required environment variables
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=<your-api-key>
export LANGCHAIN_PROJECT=<project-name>

Python Integration

python

from langsmith import Client, traceable
from langchain.callbacks.tracers import LangChainTracer

# Initialize client
client = Client()

# Use @traceable decorator for custom functions
@traceable(name="custom_operation")
def my_function(input_data):
    # Your logic here
    return result

# Initialize tracer for LangChain
tracer = LangChainTracer(project_name="my-project")

# Use with LangChain chains
chain.invoke(input, config={"callbacks": [tracer]})

Trace Retrieval

python

# Fetch traces from LangSmith
runs = client.list_runs(
    project_name="my-project",
    start_time=datetime.now() - timedelta(hours=24),
    execution_order=1,  # Root runs only
    error=False,  # Successful runs only
)

for run in runs:
    print(f"Run ID: {run.id}")
    print(f"Latency: {run.latency_p99}")
    print(f"Tokens: {run.total_tokens}")

Task Definition

When used in a babysitter process, this skill produces:

javascript

const langsmithTracingTask = defineTask({
  name: 'langsmith-tracing-setup',
  description: 'Configure LangSmith tracing for the application',

  inputs: {
    projectName: { type: 'string', required: true },
    apiKeyEnvVar: { type: 'string', default: 'LANGCHAIN_API_KEY' },
    samplingRate: { type: 'number', default: 1.0 },
    enableDebug: { type: 'boolean', default: false }
  },

  outputs: {
    configured: { type: 'boolean' },
    projectUrl: { type: 'string' },
    artifacts: { type: 'array' }
  },

  async run(inputs, taskCtx) {
    return {
      kind: 'skill',
      title: `Configure LangSmith tracing for ${inputs.projectName}`,
      skill: {
        name: 'langsmith-tracing',
        context: {
          projectName: inputs.projectName,
          apiKeyEnvVar: inputs.apiKeyEnvVar,
          samplingRate: inputs.samplingRate,
          enableDebug: inputs.enableDebug,
          instructions: [
            'Verify LangSmith API credentials are available',
            'Create or validate project configuration',
            'Set up tracing instrumentation in codebase',
            'Configure sampling rate and debug settings',
            'Verify traces are being captured correctly'
          ]
        }
      },
      io: {
        inputJsonPath: `tasks/${taskCtx.effectId}/input.json`,
        outputJsonPath: `tasks/${taskCtx.effectId}/result.json`
      }
    };
  }
});

Applicable Processes

•llm-observability-monitoring
•agent-evaluation-framework
•react-agent-implementation
•conversation-quality-testing
•regression-testing-agent

External Dependencies

•LangSmith account and API key
•LangChain Python library
•langsmith Python package

References

Related Skills

•SK-OBS-002 langfuse-integration
•SK-OBS-003 phoenix-arize-setup
•SK-OBS-004 opentelemetry-llm

Related Agents

•AG-OPS-004 observability-engineer
•AG-SAF-004 agent-evaluator