Observability Setup Orchestrator

End-to-end workflow for setting up Databricks observability — Lakehouse Monitoring, AI/BI Dashboards, and SQL Alerts — on top of a completed Gold layer and semantic layer.

Predecessor: semantic-layer-setup skill (Semantic layer should be complete, but Gold tables are the minimum requirement)

Time Estimate: 3-5 hours for initial setup, 30 min per additional table/dashboard

What You'll Create:

1. •Lakehouse Monitors — data quality, drift, and custom business KPIs for Gold tables
2. •AI/BI Dashboards — Lakeview dashboards with monitoring widgets and business metrics
3. •SQL Alerts — config-driven alerting with severity-based routing

Decision Tree

Mandatory Skill Dependencies

CRITICAL: Before generating ANY code for observability, you MUST read and follow the patterns in these common skills. Do NOT generate these patterns from memory.

Monitoring-Domain Dependencies

🔴 Non-Negotiable Defaults

Phased Implementation Workflow

Phase 0: Read Plan (5 minutes)

Before starting implementation, check for a planning manifest that defines what to build.

import yaml
from pathlib import Path

manifest_path = Path("plans/manifests/observability-manifest.yaml")

if manifest_path.exists():
    with open(manifest_path) as f:
        manifest = yaml.safe_load(f)
    
    # Extract implementation checklist from manifest
    monitors = manifest.get('lakehouse_monitors', [])
    dashboards = manifest.get('dashboards', [])
    alerts = manifest.get('alerts', [])
    print(f"Plan: {len(monitors)} monitors, {len(dashboards)} dashboards, {len(alerts)} alerts")
    
    # Each monitor has: table_name, monitor_type, custom_metrics, slicing_exprs
    # Each dashboard has: name, pages, widgets
    # Each alert has: alert_id, severity, query, threshold, schedule
else:
    # Fallback: self-discovery from Gold tables
    print("No manifest found — falling back to Gold table self-discovery")
    # Discover Gold tables from catalog, create one monitor per table

If manifest exists: Use it as the implementation checklist. Every monitor, dashboard, and alert is pre-defined with configuration details. Track completion against the manifest's summary counts.

If manifest doesn't exist: Fall back to self-discovery — inventory Gold tables, create one monitor per table (TimeSeries for facts, Snapshot for dimensions), and generate standard dashboards and alerts. This works but may miss custom business KPIs the planning phase would have defined.

Phase 1: Lakehouse Monitoring (1-2 hours)

MANDATORY: Read each skill below using the Read tool BEFORE writing any code for this phase:

Steps:

1. •Inventory Gold tables that need monitoring (fact tables are highest priority)
2. •Choose monitor type per table (TimeSeries for facts, Snapshot for dimensions)
3. •Define custom business metrics using input_columns=[":table"] for table-level KPIs
4. •Create monitor setup script with graceful degradation (delete-then-create pattern)
5. •Deploy monitors and verify metric tables are populated
6. •Document monitor configuration in Genie Space instructions (if applicable)

Phase 2: AI/BI Dashboards (1-2 hours)

MANDATORY: Read each skill below using the Read tool BEFORE writing any code for this phase:

Steps:

1. •Design dashboard layout: monitoring overview + business metrics sections
2. •Create queries using monitoring profile/drift tables
3. •Build widget configurations with proper number formatting
4. •Set dataset_catalog and dataset_schema for environment portability
5. •Deploy dashboard via Asset Bundle or API
6. •Validate all widgets render correctly

Phase 3: SQL Alerts (1 hour)

MANDATORY: Read each skill below using the Read tool BEFORE writing any code for this phase:

Steps:

1. •Create alert configuration table (Delta table-based, severity-driven)
2. •Define alert rules: threshold, percentage change, anomaly detection
3. •Deploy alerts via Databricks SDK (V2 dict-based or typed classes)
4. •Configure notification destinations per severity level
5. •Set up Quartz cron schedules for alert evaluation
6. •Validate alerts fire correctly with test data

Post-Creation Validation

Common Skill Compliance

• • Names extracted from Gold YAML (not generated) per databricks-expert-agent
• • Asset Bundle YAML follows databricks-asset-bundles patterns
• • Python imports follow databricks-python-imports patterns

Observability Specifics

• • Lakehouse Monitors created for all critical Gold tables
• • Custom business metrics use input_columns=[":table"] syntax
• • Monitor setup uses graceful degradation (delete-then-create)
• • Dashboard uses dataset_catalog/dataset_schema for portability
• • Dashboard widgets align with query columns
• • Alert queries use fully qualified table names (no parameters)
• • Alert severity routing configured (critical → PagerDuty, warning → email)
• • All monitoring jobs use serverless compute

Pipeline Progression

Previous stage: semantic-layer-setup → Metric Views, TVFs, and Genie Spaces should exist

Next stage: After completing observability, proceed to:

• •ml/00-ml-pipeline-setup — Set up ML models, experiments, and batch inference

Related Skills

References

• •Lakehouse Monitoring
• •AI/BI Dashboards
• •SQL Alerts