Feapder Dev

Quick Intake

Ask the user for the minimum information needed to produce a runnable crawler:

• •Target site/app constraints (login, JS rendering, anti-bot, rate limits).
• •Data schema (fields), uniqueness key, and storage target (file/DB/queue).
• •Entry points (seed URLs, keywords, categories) and pagination strategy.
• •Scale constraints (single machine vs distributed; incremental vs full; schedule).
• •Environment constraints (OS, Python version, network/proxy availability).

Workflow Decision Tree

Decide the crawler architecture before writing code:

• •

  Choose crawler type:

  • •Choose AirSpider for small jobs, no distributed scheduling, simple persistence.
  • •Choose Spider for high-volume, distributed crawling, and persistent task queues.
  • •Choose TaskSpider for explicit task tables/queues with retry/continuation semantics.
  • •Choose BatchSpider for periodic batches with batch metadata and separation.
• •

  Choose rendering:

  • •Use HTTP-only requests when pages are static or the API is reachable.
  • •Use browser rendering only when content is JS-rendered or requires complex interaction.
• •

  Choose persistence and flow:

  • •Use Item + pipeline for normalized storage.
  • •Use explicit DB access helpers only when pipeline is insufficient.
• •

  Choose anti-bot strategy:

  • •Add rate limits/backoff/retries first.
  • •Add proxy rotation/user pool only when needed and with observability.

Build Steps

Implement the smallest runnable crawler first, then iterate:

1) Set up environment (prefer uv)

• •Create a project venv with uv and pin a compatible Python version.
• •Install feapder and any runtime dependencies (DB drivers, playwright/selenium if used).
• •Verify import and basic CLI availability before writing spider code.

2) Scaffold a minimal project

• •Create a clean module layout: spiders/, parsers/, items/, pipelines/, settings.py.
• •Start with one spider, one parser, and one item type.

3) Implement a minimal spider + parser

Keep the first version minimal and end-to-end runnable:

from feapder import AirSpider
from feapder.network.request import Request

class DemoSpider(AirSpider):
    def start_requests(self):
        yield Request("https://example.com")

    def parse(self, request, response):
        # extract fields -> yield Item or dict
        yield {"url": response.url}

4) Add Item + pipeline (only after extraction works)

• •Define a stable unique key for de-duplication.
• •Add a pipeline that validates fields, normalizes types, and writes to the chosen sink.

5) Configure settings deliberately

• •Keep settings minimal at first.
• •Turn on logging suitable for debugging.
• •Add retries/timeouts/backoff before adding heavier defenses.

6) Run and debug iteratively

• •Run one seed URL first.
• •Add pagination/expansion only after single-page extraction is correct.
• •Use logs and counters (success/fail/empty) to validate progress.

7) Scale up (when required)

• •Switch to Spider/TaskSpider/BatchSpider only when the workload or scheduling requires it.
• •Add Redis/DB infrastructure as a deliberate dependency and document required services.

Debugging Checklist

Use a checklist-driven approach before changing architecture:

• •Confirm the request layer works: DNS/SSL/proxy, timeouts, status codes, encoding.
• •Confirm selectors/JSON parsing against real responses (save a sample response).
• •Confirm that parse() yields items/requests as expected (no silent drops).
• •Confirm pipeline and settings are loaded (wrong module path is a common cause).
• •Add observability: log key decisions, count produced items, count retries and errors.

References

Read these only when needed to keep context small:

• •references/checklist.md: Common tasks and quick checks for Feapder projects.
• •references/patterns.md: Lightweight patterns for spider types, parsing, and pipelines.
• •references/official-docs/: Local copy of Feapder official docs (Markdown).
• •references/official-docs.md: Short note on using the local docs copy.