Download Docs

Download documentation files from various sources using manifest configurations.

Quick Start

Download from a specific manifest:

bash

.claude/skills/download-docs/scripts/download.sh {manifest-name}

Download from all manifests:

bash

.claude/skills/download-docs/scripts/download.sh

Output location: output/{manifest-name}/

Manifest Format

Manifests are JSON files in scripts/manifests/. Use $schema for IDE validation. Two source types are supported:

GitHub Source (for large doc sets)

Clones repository and extracts markdown files:

json

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "github",
    "repo": "owner/repo-name",
    "branch": "main",
    "path": "docs",
    "extensions": [".md", ".mdx"],
    "exclude": ["**/internal/**"],
    "noDefaultExcludes": false
  }
}

Field	Required	Description
`repo`	Yes	GitHub repository (owner/name)
`branch`	No	Branch to clone (default: "main")
`path`	No	Directory path within repo (default: ".")
`extensions`	No	File extensions to include (default: [".md", ".mdx"])
`exclude`	No	Additional glob patterns to exclude
`noDefaultExcludes`	No	Set true to disable default excludes

Default exclude patterns:

•CHANGELOG*, CONTRIBUTING*, LICENSE*, SECURITY*, CODE_OF_CONDUCT*
•.github/**, /test/, /tests/, /tests/
•/spec/, /fixtures/, /examples/, /node_modules/

URL Source (for individual files)

Downloads specific files from URLs:

json

{
  "$schema": "./manifest.schema.json",
  "_source": { "type": "url" },
  "files": {
    "category-name": {
      "doc-name": "https://example.com/path/to/doc.md"
    }
  }
}

URL Source with HTML Conversion

Downloads HTML and converts to markdown (requires pandoc):

json

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "url",
    "convert": "html"
  },
  "files": {
    "category-name": {
      "doc-name": "https://example.com/docs/page.html"
    }
  }
}

Note: HTML conversion only works for static HTML. SPA sites (React, Angular, etc.) will produce empty output.

Disabling a Manifest

Add _disabled to skip processing:

json

{
  "$schema": "./manifest.schema.json",
  "_source": {
    "type": "url",
    "_disabled": true,
    "_reason": "SPA site - requires Playwright handler"
  },
  "files": { ... }
}

Requirements

•jq - JSON parsing
•git - GitHub source cloning
•curl - URL downloads
•pandoc - HTML conversion (optional)

Adding New Manifests

Create a JSON file in scripts/manifests/ following the format above. The manifest filename (without .json) becomes the output directory name.