Scraping Documentation

Scrape documentation sites and convert to organized markdown files.

Usage

code

/scraping-documentation <url> [options]

Arguments

URL (required):

•Base URL of documentation site (e.g., https://docs.example.com)

Options:

•--output DIR: Output directory (default: ./docs)
•--depth N: How many levels to crawl (default: 2)
•--include PATTERN: Only include URLs matching pattern
•--exclude PATTERN: Exclude URLs matching pattern
•--format FORMAT: Output format (markdown|html|both)
•--index: Generate index file

Examples

bash

# Scrape library documentation
/scraping-documentation https://docs.example.com --output ai_docs/knowledge/example

# Limited depth crawl
/scraping-documentation https://api.example.com/docs --depth 1

# Include only API reference
/scraping-documentation https://docs.example.com --include "/api/*"

# Exclude changelog pages
/scraping-documentation https://docs.example.com --exclude "/changelog/*"

Workflow

Step 1: Discover Pages

python

# Start from base URL
base_url = args.url
discovered = set()
to_crawl = [base_url]

while to_crawl and len(discovered) < max_pages:
    url = to_crawl.pop(0)

    if url in discovered:
        continue

    if not matches_include(url) or matches_exclude(url):
        continue

    # Fetch page
    content = fetch_url(url)

    # Extract links
    links = extract_links(content, base_url)

    # Add to queue (respect depth)
    depth = get_depth(url, base_url)
    if depth < max_depth:
        to_crawl.extend(links)

    discovered.add(url)

Step 2: Fetch and Convert

python

for url in discovered:
    # Fetch content
    html = fetch_url(url)

    # Convert to markdown
    markdown = html_to_markdown(html)

    # Clean up
    markdown = clean_markdown(markdown)

    # Determine output path
    path = url_to_filepath(url, output_dir)

    # Write file
    write_file(path, markdown)

Step 3: HTML to Markdown Conversion

Handle common documentation patterns:

•Code blocks with syntax highlighting
•Tables
•Admonitions/callouts
•Navigation (strip)
•Headers (preserve hierarchy)
•Links (convert to relative)
•Images (download and reference locally)

Step 4: Generate Index

markdown

# {Site Name} Documentation

Scraped from: {base_url}
Date: {timestamp}

## Contents

{for section in sections:}
### {section.title}
{for page in section.pages:}
- [{page.title}]({page.path})

Step 5: Report

markdown

## Scraping Complete

### Summary
- Base URL: {base_url}
- Pages scraped: {count}
- Output directory: {output_dir}
- Total size: {size}

### Files Created
{for file in files:}
- {file.path} ({file.size})

### Structure
{directory tree}

### Next Steps
Add to CLAUDE.md:
```markdown
## Documentation
@{output_dir}/index.md

code


## Output Structure

ai_docs/knowledge/example/ ├── index.md # Table of contents ├── getting-started.md # Converted pages ├── api/ │ ├── index.md │ ├── authentication.md │ └── endpoints.md ├── guides/ │ ├── index.md │ └── quickstart.md └── _assets/ # Downloaded images └── diagram.png

code


## Conversion Rules

### Code Blocks
```html
<pre><code class="language-python">print("hello")</code></pre>

→

markdown

```python
print("hello")
```

Tables

HTML tables → Markdown tables

Callouts

html

<div class="warning">Important note</div>

→

markdown

> ⚠️ **Warning**: Important note

Navigation

Strip navigation, sidebars, footers - keep content only.

Error Handling

Issue	Action
404 page	Skip and log
Rate limited	Back off and retry
Login required	Report and skip
JavaScript rendered	Warn (content may be incomplete)
Large file	Skip with warning

Best Practices

•Respect robots.txt - Check before scraping
•Rate limiting - Don't overload servers
•Attribution - Keep source URL in files
•Updates - Re-run periodically to update
•Selection - Use include/exclude to get relevant content only