Data Engineering Hub
Welcome to the comprehensive data engineering skill suite. This hub organizes all data engineering knowledge into logical, non-overlapping domains.
Skill Map
| Domain | Skills | When to Use |
|---|---|---|
| Core | @data-engineering-core | Polars, DuckDB, PyArrow fundamentals; ETL patterns; error handling; performance optimization |
| Storage | @data-engineering-storage-lakehouse | Delta Lake, Apache Iceberg, Apache Hudi |
@data-engineering-storage-remote-access | fsspec, pyarrow.fs, obstore; cloud access patterns | |
@data-engineering-storage-authentication | AWS, GCP, Azure auth - IAM roles, managed identity, secrets management | |
@data-engineering-storage-formats | Parquet optimizations, Lance, Zarr, Avro, ORC | |
| Orchestration | @data-engineering-orchestration | Prefect, Dagster, dbt, workflow scheduling |
| Streaming | @data-engineering-streaming | Kafka, MQTT, NATS JetStream for real-time data |
| Quality | @data-engineering-quality | Great Expectations, Pandera for data validation |
| Observability | @data-engineering-observability | OpenTelemetry, Prometheus for pipeline monitoring |
| AI/ML | @data-engineering-ai-ml | Embeddings, vector databases, RAG pipelines |
| Best Practices | @data-engineering-best-practices | Medallion architecture, partitioning, file sizing, incremental loads, schema evolution, testing |
| Catalogs | @data-engineering-catalogs | Data catalog systems: Iceberg catalogs, DuckDB multi-source, Amundsen/DataHub/OpenMetadata |
Quick Reference: Core Stack
| Task | Recommended Tool |
|---|---|
| DataFrame operations | Polars (10-50x faster than pandas) |
| SQL analytics | DuckDB (embedded OLAP, zero-copy Arrow integration) |
| Data interchange | PyArrow (Arrow format, zero-copy transfers) |
| Cloud storage access | fsspec (universal), pyarrow.fs (Arrow-native), obstore (high-performance) |
| Lakehouse format | Delta Lake (Spark ecosystem), Iceberg (engine-agnostic), Hudi (streaming CDC) |
| Orchestration | Prefect (Pythonic flows), Dagster (asset-based), dbt (SQL transformations) |
| Validation | Pandera (lightweight), Great Expectations (enterprise) |
Getting Started
New to Data Engineering?
Start with @data-engineering-core to learn the foundational libraries and patterns.
Working with Cloud Storage?
Go to @data-engineering-storage-remote-access for fsspec, pyarrow.fs, and obstore.
Building Data Lakes?
Explore @data-engineering-storage-lakehouse for ACID table formats.
Choosing a Data Catalog?
Check @data-engineering-catalogs for Iceberg catalogs, DuckDB multi-source patterns, and tool comparisons.
Production-Grade Pipelines?
Read @data-engineering-best-practices for medallion architecture, partitioning, schema evolution, and testing strategies.
Orchestrating Pipelines?
Check @data-engineering-orchestration for Prefect, Dagster, and dbt.
Production Monitoring?
See @data-engineering-observability for tracing and metrics.
AI/ML Data Pipelines?
Visit @data-engineering-ai-ml for embeddings, vector databases, and RAG.
Principles
- •Lazy evaluation: Use Polars lazy frames and DuckDB query planning for performance
- •Zero-copy data transfer: Leverage Arrow format for memory efficiency
- •Pushdown optimization: Filter at storage layer to minimize data transfer
- •Type safety: Use explicit schemas and type hints
- •Resilience: Implement retries, circuit breakers, and proper error handling
- •Observability: Instrument pipelines with traces and metrics
- •Security: Never hardcode credentials; use IAM roles and environment variables
Migration from Legacy Skills
This restructured suite replaces the previous split organization (data-engineering-* and remote-filesystems-*). All content has been consolidated to eliminate duplication and clarify ownership.
Legacy skill replacements:
- •
data-engineering-core→@data-engineering-core(plus specific integrations) - •
data-engineering-lakehouse→@data-engineering-storage-lakehouse - •
data-engineering-orchestration→@data-engineering-orchestration - •
data-engineering-streaming→@data-engineering-streaming - •
data-engineering-quality→@data-engineering-quality - •
data-engineering-observability→@data-engineering-observability - •
data-engineering-llm-pipelines→@data-engineering-ai-ml - •
remote-filesystems-*→@data-engineering-storage-remote-accessand integrations
All legacy skills remain functional but are deprecated. New content should be added to the new structure only.