Web Search Toolkit

Search the web and extract readable content. Stable CLI with JSON output for agents.

Setup

bash

cd {baseDir}
uv sync  # Install dependencies (once)

Optional: Set BRAVE_API_KEY for better search reliability (ddgs is keyless but flaky).

Commands

Search

bash

{baseDir}/.venv/bin/wstk search "query"                    # Default (10 results)
{baseDir}/.venv/bin/wstk search "query" -n 5 --plain       # URLs only, one per line
{baseDir}/.venv/bin/wstk search "query" --json             # Machine-readable
{baseDir}/.venv/bin/wstk search "query" --time-range w     # Last week
{baseDir}/.venv/bin/wstk search "site:docs.python.org asyncio"  # Site-scoped

Key flags:

•-n, --max-results <N> — Number of results (default: 10)
•--time-range <d|w|m|y> — Filter by recency
•--provider <ddgs|brave_api|auto> — Search provider
•--plain — Output URLs only (for piping)
•--json — Structured output

Pipeline (search → extract)

bash

{baseDir}/.venv/bin/wstk pipeline "python asyncio tutorial" --json
{baseDir}/.venv/bin/wstk pipeline "python asyncio tutorial" --plan --plain

Key flags:

•--top-k <N> — Search results to consider
•--extract-k <N> — Number of results to extract
•--plan — Return candidates without fetching
•--method <http|browser|auto> — Extraction method (default: http)

Extract (fetch + extract readable content)

bash

{baseDir}/.venv/bin/wstk extract https://example.com --plain     # Markdown output
{baseDir}/.venv/bin/wstk extract https://example.com --text      # Plain text
{baseDir}/.venv/bin/wstk extract https://example.com --json      # Full metadata
{baseDir}/.venv/bin/wstk extract ./local-file.html --plain       # From file

Key flags:

•--markdown / --text / --both — Output format
•--strategy <auto|readability|docs> — Extraction strategy
•--max-chars <N> — Truncate output
•--allow-domain <domain> — Restrict to specific domains (safety)

Fetch (raw HTTP, no extraction)

bash

{baseDir}/.venv/bin/wstk fetch https://example.com --json        # Metadata + status
{baseDir}/.venv/bin/wstk fetch https://example.com --plain       # Path to cached body

List providers

bash

{baseDir}/.venv/bin/wstk providers --plain

Decision Guide

•search when you need discovery or candidate URLs.
•pipeline when you want a one-shot search → extract bundle.
•fetch when you need HTTP metadata or the cached body path (no extraction).
•extract when you want readable content from a URL or local HTML.
•render when a page is JS-only or blocked (or use extract --method browser for one-step extraction).

Common Patterns

Search → extract top result:

bash

url=$({baseDir}/.venv/bin/wstk search "python asyncio tutorial" --plain | head -1)
{baseDir}/.venv/bin/wstk extract "$url" --plain --max-chars 8000

Search with JSON for programmatic use:

bash

{baseDir}/.venv/bin/wstk search "openai api reference" --json | jq '.data.results[0].url'

Safe extraction (restrict domains):

bash

{baseDir}/.venv/bin/wstk extract https://docs.python.org/3/library/asyncio.html \
  --allow-domain docs.python.org --plain

Output Formats

•--plain — Stable text for piping (URLs for search, content for extract)
•--json — Structured envelope: { "ok": bool, "data": {...}, "error": {...} }
•Default — Human-readable with colors

Agent Defaults

•Default to --json in agent wrappers; parse ok, error.code, and warnings.
•Surface concise diagnostics by relaying error.message and error.details.reason when ok=false.
•Use --plain only for piping, and add --no-input for non-interactive runs.
•Consider --redact when handling sensitive URLs or content.

Exit Codes

•0 — Success
•1 — Runtime failure (network, provider error)
•2 — Invalid usage
•3 — Not found / empty result
•4 — Blocked / access denied
•5 — Needs JS rendering (page is JS-only)

Global Flags

•--timeout <seconds> — Network timeout
•--no-cache — Disable caching
•--fresh — Bypass cache reads (still writes)
•--quiet — Minimal output
•--verbose — Debug diagnostics to stderr
•--policy <standard|strict|permissive> — Safety defaults

References

•references/troubleshooting.md — 403/JS-only guidance and advanced fetch flags.
•references/providers.md — Provider selection and privacy notes.
•docs/claude-code.md — Claude Code wrapper usage.

When to Use

•Searching for documentation or API references
•Looking up facts or current information
•Extracting content from known URLs
•Any task requiring web search without interactive browsing

Prefer browser-tools when you need: JS interaction, form filling, clicking, or visual inspection.