Get Web Page (Raw)

When to use

Use this skill when:

• •The user provides a URL to capture as source material
• •You need to store a web page for later distillation
• •You're building up raw materials for the distillation pipeline

Keywords: capture, web page, URL, fetch, raw

Inputs

Required:

• •url (string): The URL of the web page to capture

Optional:

• •title_hint (string): A human-provided hint for the title if extraction fails

Outputs

This skill produces:

1. •A new file in raw/ with the naming pattern: YYYYMMDD-HHMMSSZ--<slug>.md
2. •Metadata returned to the agent:
   • •raw_path: full path to the created file
   • •title: extracted or inferred title
   • •canonical_url: canonical URL if found, otherwise the input URL
   • •captured_at: ISO timestamp (UTC)

Procedure

1. Generate timestamp and slug

• •Get current UTC timestamp: date -u +"%Y%m%d-%H%M%SZ"
• •Create a slug from the page title or URL path:
  • •Use the main topic/title in lowercase
  • •Replace spaces and special chars with hyphens
  • •Keep it concise (typically 3-8 words)
  • •Example: patterns-for-ai-assisted-software-development

2. Fetch the web page content

Primary method: fetch_webpage tool

• •Call fetch_webpage with the URL and a descriptive query about the content
• •This extracts main content, removes navigation/ads, and converts to clean Markdown
• •Captures metadata like title and author when available

Alternative methods (when needed):

• •Playwright MCP: For complex interactive pages, JavaScript-heavy sites, or when you need precise browser control
• •curl + HTML parsing: For simple pages when fetch_webpage isn't available, though requires manual Markdown conversion
• •Browser tools: Built-in browser navigation and snapshot capabilities

Note: Most public articles and blog posts work well with fetch_webpage

3. Build front matter

Create YAML front matter with required and available fields:

---
title: "<extracted from page>"
source_url: "<original URL provided>"
captured_at: "<ISO timestamp from step 1>"
capture_type: web_page
capture_tool: fetch_webpage  # or browser if used
raw_format: markdown
status: captured
author: "<if clearly visible>"  # optional
published_at: "<YYYY-MM-DD if available>"  # optional
tags: ["keyword1", "keyword2"]  # optional, 3-5 relevant tags
---

Tips:

• •Use the page's actual title, not a description
• •Convert published_at to simple date format (YYYY-MM-DD)
• •Tags should be broad topics, not exhaustive keywords

4. Write the file

• •Filename: <timestamp>--<slug>.md (e.g., 20260102-095107Z--patterns-for-ai-assisted-software-development.md)
• •Full path: raw/<filename>
• •Content: front matter + blank line + extracted content
• •Use create_file tool with the full content

5. Confirm to user

Provide a brief confirmation:

• •Link to the created file using relative path
• •One-sentence summary of what the article covers (2-3 key topics)

Examples

Example 1: Dev.to article capture

Input:

url: "https://dev.to/javatarz/patterns-for-ai-assisted-software-development-4ga2"

Actions:

1. •Generate timestamp: 20260102-095107Z
2. •Fetch with fetch_webpage tool
3. •Extract title: "Patterns for AI assisted software development"
4. •Create slug: patterns-for-ai-assisted-software-development
5. •Write to: raw/20260102-095107Z--patterns-for-ai-assisted-software-development.md

Output:

---
title: "Patterns for AI assisted software development"
source_url: "https://dev.to/javatarz/patterns-for-ai-assisted-software-development-4ga2"
captured_at: "2026-01-02T09:51:07Z"
capture_type: web_page
capture_tool: fetch_webpage
raw_format: markdown
status: captured
author: "Karun Japhet"
published_at: "2025-12-27"
tags: ["ai", "programming", "productivity", "patterns"]
---

Edge cases

fetch_webpage fails:

• •Check if authentication is required
• •Try browser-based extraction if needed
• •Inform user if the page cannot be accessed

Missing metadata:

• •If no title is extracted, use the URL path or ask user for a title hint
• •If no author/date visible, omit those fields
• •Minimal front matter is acceptable

Paywalled or restricted content:

• •Inform user about the restriction
• •Capture what's visible if useful (preview text)
• •Consider asking if they have alternative access

References

• •Parent workflow: docs/distillation/distillation-pipeline.md
• •Front matter schema: see "Raw file front matter" in distillation-pipeline.md
• •Next step after capture: use make-distilled skill to process this raw file