Incremental Model Strategy Selector

Overview

Selects and configures optimal incremental model strategies. This skill optimizes data transformation efficiency through proper incremental processing patterns.

Capabilities

• •Incremental strategy selection (append, merge, delete+insert)
• •Partition pruning optimization
• •Unique key configuration
• •On_schema_change handling
• •Full refresh scheduling
• •Lookback window optimization
• •Late-arriving data handling

Input Schema

{
  "modelCharacteristics": {
    "sourceType": "string",
    "updatePattern": "append|update|delete",
    "volumeGB": "number",
    "updateFrequency": "string"
  },
  "platform": "snowflake|bigquery|redshift",
  "existingModel": "object"
}

Output Schema

{
  "strategy": "append|merge|delete+insert",
  "config": "object",
  "partitionStrategy": "object",
  "refreshSchedule": "object",
  "dbtConfig": "object"
}

Target Processes

• •Incremental Model Setup
• •dbt Model Development
• •Pipeline Migration

Usage Guidelines

1. •Analyze source data update patterns
2. •Measure data volume and update frequency
3. •Select strategy based on characteristics
4. •Configure appropriate lookback windows

Best Practices

• •Use append for insert-only sources
• •Use merge for sources with updates
• •Configure partition pruning for large tables
• •Schedule periodic full refreshes for data correction
• •Handle late-arriving data with appropriate lookback