Learn and reuse route-specific extraction selectors with Crawlee Python
Learn CSS selectors once per page route, persist them across Crawlee Python runs, and extract validated records with controlled LLM fallback and usage checks.
npx skills add agentskillexchange/skills --skill learn-and-reuse-route-specific-extraction-selectors-with-crawlee-python
Use this workflow when an agent needs to collect the same fields across repeated server-rendered page layouts, and repeated model calls or hand-maintained selectors make the job expensive. Crawlee Python’s PydanticAiSelectorExtractor learns CSS selectors from a representative page, caches them by route, and reuses them on matching pages and later runs. The operator defines the extraction contract, reviews sample records and returns a dataset plus crawl and model-usage evidence.
What this skill actually does
Run the workflow
- Define a Pydantic model for the records and specify authorized seed URLs, allowed link patterns, route labels and a maximum request count. Start with a small sample.
- Configure PydanticAiCrawler with a PydanticAiSelectorExtractor and an explicit PydanticAiDirectExtractor fallback. Supply the chosen model through its supported provider configuration. Set per-model-run UsageLimits on both extractors; use the documented cumulative token checks to stop further processing when the run budget is reached.
- Enqueue matching detail pages with a shared request label. In that route’s handler, call context.extract(Record) and persist record.model_dump() through context.push_data(). The label supplies the default cache bucket; use cache_tag when separate layouts need explicit buckets.
- Review sample values against their source pages. The extractor reuses matching cached selectors without another model call. A new layout can generate another selector variant; failed generation or an unsupported schema uses the configured fallback. Record failures and budget exceptions rather than treating them as valid records.
- Keep the same Crawlee storage for reruns: selector caches persist in its KeyValueStore. Return the extracted records, failed or skipped URLs, schema used and observed model usage. Check sample correctness again when layouts change.
Scope and limits
The entry covers route-specific selector learning, reuse and controlled fallback in a bounded HTTP extraction job. It is useful when the desired output is known and layouts repeat. General browser control, hosted scraping deployment and broad crawler setup are outside the workflow. PydanticAiCrawler fetches HTML over HTTP and does not render JavaScript. Use a browser workflow when required data is client-rendered.
The selector extractor supports scalar fields, scalar lists, item lists and one level of nested items; unsupported structures need the direct extractor. Pydantic validates structure, but operators must still check factual correctness. The upstream API is experimental: pin the release and review changes before upgrading. Per-model-run limits and cumulative checks are operational controls, not a promise that concurrent work can never overshoot a budget.
Setup and references
Use Python 3.10+ in a project virtual environment. Install the verified release with python -m pip install ‘crawlee[pydantic-ai]==1.11.2’. Configure a supported model provider credential through its environment variable, such as OPENAI_API_KEY. Start from the upstream v1.11.2 selector_extractor_example.py, set the authorized seed URLs, allowed URL patterns, request limit and Pydantic schema, and use an environment-backed model configuration instead of the example’s literal key placeholder. Add the documented UsageLimits and cumulative usage checks to both extractor paths. Run the resulting project script with python extract_routes.py; retain its Crawlee storage between runs to reuse cached selectors. This HTTP-only workflow needs no browser installation, MCP server or Apify account.
Follow the release-specific guide, selector-cache example and usage-limit example. Metadata and documentation were checked; package execution and a security review were not performed.