AgentSkillsCN

lakehouse-pipeline-design

设计Databricks湖仓架构的流水线文档（青铜级、白银级、黄金级，或DLT、Jobs模式），涵盖SLA、数据质量、Unity Catalog治理、监控机制，以及实施检查清单。适用于在Databricks上设计或评审ETL/ELT流水线、DLT流水线、流式数据摄入、CDC，或批量作业时使用。

SKILL.md
--- frontmatter
name: lakehouse-pipeline-design
description: Create a Databricks lakehouse pipeline design doc (bronze/silver/gold, DLT or Jobs), including SLAs, data quality, Unity Catalog governance, monitoring, and an implementation checklist. Use when designing or reviewing ETL/ELT pipelines, DLT pipelines, streaming ingestion, CDC, or batch jobs on Databricks.
license: Proprietary. For internal team use.
compatibility: Designed for Databricks Assistant agent mode in Databricks workspaces.
metadata:
  author: your-team
  version: "1.0"

Lakehouse pipeline design (Databricks)

Use this skill when someone asks for a pipeline design, DLT design, ETL plan, CDC ingestion, or a review of an existing pipeline.

Deliverables

When activated, produce at least:

  1. •A filled design doc based on assets/pipeline-design-doc.md
  2. •A short, actionable implementation checklist (you can reuse references/pipeline-checklist.md)

Optionally (only if asked): a code skeleton (PySpark / SQL / DLT) that matches the design.

Minimal inputs (ask only what’s missing)

Ask up to 3 questions total. Prefer defaults.

  • •Source type: files / DB / API / Kafka / etc.
  • •Mode: batch / streaming / CDC
  • •Target: tables (catalog.schema.*) and consumers (dashboards, ML, downstream jobs)
  • •Volume + SLA: rows/day, latency/freshness SLO, cost constraints
  • •Governance: PII? UC catalogs/schemas, access groups

Design guidance (what to include)

  • •Architecture: bronze → silver → gold; DLT vs Jobs; where to enforce quality
  • •Incremental strategy: watermarking, MERGE for CDC, idempotency
  • •Delta table design: partitioning, ZORDER, OPTIMIZE/VACUUM policy
  • •Quality checks: schema validation, null/unique, freshness, anomaly checks
  • •Observability: metrics, logs, expectations failures, alerts, runbooks
  • •Backfills: replay strategy, how to reprocess safely, versioning
  • •Security: UC permissions, row/column filtering if needed, secrets management
  • •Operational: retries, SLAs, escalation, deployment strategy

Output rules

  • •Put concrete decisions in a “Decisions” section and unknowns in “Open questions”.
  • •If details are missing, keep placeholders like {{...}} and add an “Info needed” section.
  • •Keep the doc concise; link to references/pipeline-checklist.md when you need long checklists.

Examples

User: “Design a DLT pipeline that ingests Salesforce accounts daily and publishes a gold table for dashboards.”
Output: Design doc + checklist + optional DLT skeleton.

User: “Review our existing silver-to-gold job for performance and reliability.”
Output: Review-style design doc: risks, improvements, and prioritized actions.

Edge cases

  • •Streaming sources: include checkpointing, schema evolution handling, and late data policy.
  • •Regulated data: include classification, retention, and UC policy controls.
  • •Multi-tenant tables: call out tenant key, partitioning, and access controls.