Fabric Forecasting

Time-series forecasting pipelines on Fabric — from raw data to trained models with profiling, clustering, and feature engineering.

When to Use

• •Building demand forecasting models (retail, supply chain, finance)
• •Forecasting across many series (products, stores, regions)
• •Classifying time-series patterns (regular, intermittent, lumpy, erratic)
• •Creating feature-engineered datasets for ML models
• •Training and tuning LightGBM, Prophet, or ensemble models on Fabric

Decision Tree

Need time-series forecasting on Fabric?
├─ Have historical data in Lakehouse?
│   ├─ Yes → Start at Phase 1 (Intake & Discovery)
│   └─ No → Use fabric-analytics skill to ingest data first
├─ Know the forecasting scenario?
│   ├─ Clear requirements → Start at Phase 2 (Scenario Interpretation)
│   └─ Need discovery → Start at Phase 1
├─ Have a customization plan?
│   └─ Yes → Start at Phase 4 (Notebook Generation)
├─ Which model?
│   ├─ Many series + external features → LightGBM ✅
│   ├─ Few series + strong seasonality → Prophet ✅
│   ├─ Intermittent demand → Specialized methods (Croston, SBA)
│   └─ Unsure → Profile data first (Phase 1-2), then decide
└─ Not forecasting?
    ├─ Ad-hoc analytics → Use fabric-analytics skill
    └─ Chat-based Q&A → Use fabric-data-agent skill

Pipeline Overview

The forecasting pipeline follows a 5-phase workflow organized as a notebook pipeline:

Phase 1: Intake ─→ Phase 2: Interpret ─→ Phase 3: Plan ─→ Phase 4: Notebooks ─→ Phase 5: Finalize
                                                              │
                                                    ┌────────┼────────┐────────┐────────┐
                                                   NB01    NB02    NB03    NB04    NB05
                                                   Prep   Profile Cluster Feature  Train

Phase 1: Intake & Data Discovery

Goal: Understand the data and user's forecasting scenario.

Checkpoint: Present data summary, confirm column mapping.

Phase 2: Scenario Interpretation

Goal: Translate business requirements into forecasting parameters.

Checkpoint: Confirm scenario parameters with user.

Phase 3: Customization Planning

Goal: Determine which notebook customizations are needed based on the scenario.

Customization Risk Levels

Rule: Low = auto-apply. Medium = explain and confirm. High = detailed proposal + approval.

Checkpoint: Present customization plan, get approval for medium/high risk changes.

Phase 4: Notebook Generation (NB01-NB05)

Five notebooks form the pipeline, each building on the previous output:

Checkpoint: After each notebook, validate output table before proceeding.

Phase 5: Finalization & Delivery

Goal: Package deliverables and deploy to Fabric.

Core Concepts

Time-Series Classification

Profiling classifies each series to guide model selection:

• •CV² = Coefficient of Variation squared (demand variability)
• •ADI = Average Demand Interval (frequency of non-zero demand)

Feature Engineering Patterns

Model Selection Guide

Notebook Conventions

Cell Organization

# Every notebook follows this pattern:
# 1. Title cell (markdown) with scenario name and timestamp
# 2. Import/setup cell
# 3. Configuration cell (all parameters in one place)
# 4. Processing cells with markdown explanations
# 5. Validation cells with data quality checks
# 6. Summary cell with output statistics

Customization Markers

# CUSTOMIZED: Changed lag window from 7 to 14 based on bi-weekly seasonality
lag_features = create_lag_features(df, lags=[7, 14, 21, 28])

# DEFAULT: Using standard clustering parameters
n_clusters = 5

Validation After Each Notebook

# Standard validation pattern
output_df = spark.read.table(f"{scenario}_prepared")
row_count = output_df.count()
null_count = output_df.filter(F.col(target_col).isNull()).count()
date_range = output_df.agg(F.min(date_col), F.max(date_col)).collect()[0]

print(f"✅ Output table: {scenario}_prepared")
print(f"   Rows: {row_count:,}")
print(f"   Nulls in target: {null_count}")
print(f"   Date range: {date_range[0]} to {date_range[1]}")

Livy Session Management

Same rules as fabric-analytics and fabric-data-agent:

1. Check for existing sessions FIRST (reuse idle sessions)
2. Create only if none exist (cold start: 3-6+ minutes)
3. Never close sessions unless explicitly requested
4. Use naming: forecasting-{scenario}-{timestamp}
5. Validate all code via Livy before including in final notebooks

Error Handling

Retry Protocol

Attempt 1 → Execute via Livy
  ↓ (on failure)
Attempt 2 → Diagnose error, apply fix, retry
  ↓ (on failure)
Attempt 3 → Try alternative approach
  ↓ (on failure)
Escalate to user with error details + options:
  A) Suggested fix
  B) Skip this cell and continue
  C) User provides guidance

Common Errors

Output Artifacts

run/{scenario_name}_{YYYYMMDD}/
├── Fabric 01 DataPreparation.ipynb
├── Fabric 02 ProfilingIntermittent.ipynb
├── Fabric 03 Clustering.ipynb
├── Fabric 04 FeatureEngineering.ipynb
├── Fabric 05 TrainTestSelectTune.ipynb
├── completion_report.md
└── requirements.txt (if new dependencies)

Anti-Patterns

• •Skip profiling: Treating all series identically → wrong model for intermittent data
• •Too many lags: 100+ lag features → overfitting, slow training
• •No train/test split: Evaluating on training data → inflated accuracy
• •Ignore data quality: Missing dates, duplicates → biased forecasts
• •Fixed parameters: Using defaults without tuning → suboptimal accuracy
• •No validation checkpoints: Running all notebooks blindly → catching errors too late

Boundaries

Always Do

• •Gather inputs (workspace, lakehouse, table, scenario) before starting
• •Profile data before choosing models
• •Validate output after each notebook
• •Get user approval for medium/high risk customizations
• •Generate all code as reproducible notebooks
• •Use timestamped output folders
• •Document all customization decisions in completion report

Ask First

• •Structural changes (adding/removing notebooks, changing flow)
• •Algorithm changes (swapping LightGBM for another model)
• •New dependencies not in original requirements
• •Skipping entire notebooks or major sections
• •High-risk generative customizations

Never Do

• •Proceed without required inputs
• •Save notebook code that hasn't been validated via Livy
• •Overwrite original template notebooks
• •Hardcode credentials or connection strings
• •Assume column names without schema verification
• •Skip user approval for medium/high risk changes

Reference Index

Asset Templates