Auto Skill Generator
Generate skills by researching and crawling authoritative documentation.
Tool: fetch_docs.py
bash
# Search - returns all URLs with snippets python scripts/fetch_docs.py search "Modal GPU Python documentation" # Crawl - with domain/path filtering to stay focused python scripts/fetch_docs.py crawl \ --url https://modal.com/docs/guide/gpu \ --no-external \ --select-paths "/docs/.*" \ --instructions "Focus on GPU setup and code examples" \ --limit 30
Workflow
1. Search for Documentation
bash
python scripts/fetch_docs.py search "{topic} documentation"
Returns JSON with all URLs, titles, scores, and content snippets.
2. Select Best URL
Review search results and select based on:
- •Official docs:
*.com/docs/,docs.*.com,*.readthedocs.io - •Content relevance: Check snippets for API docs, code examples
- •Avoid: Blog posts, changelogs, marketing, glossaries
3. Crawl with Filtering
bash
python scripts/fetch_docs.py crawl \
--url {selected_url} \
--no-external \
--select-paths "/docs/.*" "/guide/.*" \
--instructions "Focus on API methods and code examples"
Core Parameters:
| Parameter | Description |
|---|---|
--url | Required. URL to crawl |
--instructions | Natural language guidance for crawler |
--limit | Total pages (default: 50) |
--max-depth | Link depth (default: 2) |
Domain/Path Filtering (Critical):
| Parameter | Description |
|---|---|
--no-external | Block external domains |
--select-paths | Regex patterns to include (e.g., /docs/.*) |
--exclude-paths | Regex patterns to exclude (e.g., /blog/.*) |
--select-domains | Regex for allowed domains |
--exclude-domains | Regex for blocked domains |
Quality Options:
| Parameter | Description |
|---|---|
--extract-depth | basic (1 credit/5 URLs) or advanced (2 credits/5 URLs) |
--format | markdown or text |
--timeout | Seconds (10-150, default: 150) |
4. Generate Skill File
From crawled content, create:
markdown
---
name: {topic-slug}ß
description: >
{What the skill does}. Use when: {specific triggers}.
---
# {Topic Name}
## Quick Start
## Core API
## Common Patterns
Output
- •Location:
~/.claude/skills/{topic-slug}/SKILL.md - •Extract ALL code blocks from crawled content
- •Keep SKILL.md under 500 lines; split to
references/if longer