Tech Stack Overview

Language-Specific Conventions

• •Python: See PYTHON.md for detailed conventions and examples

Primary Languages

Frameworks & Libraries

Data Generation (datagen)

• •DuckDB: Local analytical database for historical data
• •Faker: Synthetic data generation
• •Pydantic: Data validation and event schemas
• •azure-eventhub: Event streaming to Azure

Lakehouse

• •Delta Lake: ACID transactions, schema enforcement
• •PySpark: Distributed data processing

Real-Time Analytics

• •Microsoft Fabric Eventhouse: KQL-based analytics
• •Eventstream: Event routing and transformation

Project Architecture

Event Flow:
  datagen (Python)
    → Azure Event Hubs
    → Eventstream
    → KQL Tables + Lakehouse Bronze

Data Layers:
  Bronze (raw JSON)
    → Silver (typed Delta)
    → Gold (aggregated Delta)
    → Semantic Model (Power BI)

Key Dependencies

See datagen/pyproject.toml for Python dependencies.

Core packages:

• •pydantic - Schema validation
• •duckdb - Local analytics
• •faker - Data generation
• •azure-eventhub - Event streaming