SmolVLM - Local Image Analysis

Analyze images locally using SmolVLM-2B, a state-of-the-art compact vision-language model optimized for Apple Silicon via mlx-vlm.

Quick Usage

Describe an Image

python ~/.claude/skills/smolvlm/scripts/view_image.py /path/to/image.png

Ask a Question About an Image

python ~/.claude/skills/smolvlm/scripts/view_image.py /path/to/image.png "What text is visible?"

Specific Tasks

# Extract text (OCR)
python ~/.claude/skills/smolvlm/scripts/view_image.py screenshot.png "Extract all text"

# UI analysis
python ~/.claude/skills/smolvlm/scripts/view_image.py ui.png "Describe the UI elements"

# Detailed description
python ~/.claude/skills/smolvlm/scripts/view_image.py photo.jpg --detailed

Effective Prompts

General Description

• •"Describe this image" - Basic description
• •"Describe this image in detail, including colors, composition, and any text" - Comprehensive

Text Extraction (OCR)

• •"Extract all visible text from this image"
• •"What text appears in this screenshot?"
• •"Read the text in this document"

UI/Screenshot Analysis

• •"Describe the user interface elements"
• •"What buttons and controls are visible?"
• •"Identify the application and its current state"

Visual Question Answering

• •"How many [objects] are in this image?"
• •"What color is the [object]?"
• •"Is there a [object] in this image?"

Code/Technical

• •"What programming language is shown?"
• •"Describe what this code does"
• •"Identify any errors in this code screenshot"

Model Details

Requirements

• •macOS with Apple Silicon (M1/M2/M3)
• •Python 3.10+
• •mlx-vlm package: uv pip install mlx-vlm --system

Troubleshooting

"Model not found": First run downloads the model (~4GB). Wait for completion.

Out of memory: Close other applications. Model needs ~6GB free RAM.

Slow first inference: Model loading takes 10-15s on first use, subsequent calls are faster.