Scrapling is a scraping framework written in Python that keeps requesting, browsing, parsing, whole-site crawling and exporting inside one library. Version 0.5.1, released on 4 November 2026, does two things: it opens the browser session to outside programs so an agent can read motion and drag, and it smooths the road from a whole site to clean Markdown. Anyone building a corpus, or letting an AI read public pages on its own, can spend ten minutes here. It runs on Windows, macOS and Linux.

Wash the page into Markdown before the model sees it

A large model loses on raw HTML. Navigation, scripts, styles and ad slots sit in the same file, and the useful body is wedged in the middle while every token of noise is billed the same. Scrapling does the subtraction at the conversion step: after markdown() runs, scripts, styles and hidden elements are emptied out, and white text hidden by CSS, blocks marked aria-hidden, HTML comments and zero-width characters go with them.

No model sits in that path. It is string processing on the local machine, so ten thousand pages cost nothing in model fees. The saving is not only money. A clean corpus keeps chunking stable downstream, and a retrieved passage does not arrive with half a menu attached.

Whole-site conversion comes as a template. Set the allowed domains, name the content selector, give a page ceiling, and the run leaves a folder of .md files plus a JSONL where every record carries source, title and body, ready to pour into a knowledge base. Documentation sites, blogs and product manuals have the regular structure that makes this the least painful way to build a corpus.

Elements carry a fingerprint, so a redesign spares the selector

The brittle part of any scraper is the selector. A site renames its CSS classes and yesterday's working code returns an empty list, and the log has to be read layer by layer before the cause appears. Adaptive element tracking takes another route: the first lookup records position, attributes and text features as a fingerprint, and after the structure shifts it looks again by similarity for the node that moved but stayed the same thing.

Similar-element matching covers the other common case. A page holds dozens of cards with identical structure and one of them is wanted; find a reference element first, then expand by structural similarity, which holds better than a hand-written nth-child.

One request and a whole site share the same syntax

Fetcher

Plain HTTP requests carrying the TLS fingerprint and headers of a real browser, enough for static pages and public APIs.

Browser scraper

Drives Chromium through Playwright, waits for JavaScript to finish, and can attach to a browser that is already open.

Spider

Concurrent crawling with automatic throttling, progress saved on Ctrl+C and resumed later, exported to JSON, JSONL, CSV or XML.

The three capture styles share one parsing interface. The same selector code moves from a static page to a dynamic one by swapping the fetcher, with no second dialect for rendering JavaScript. The XHR requests behind a page can be recorded too, and the data an endpoint returns is usually tidier than the HTML that renders it.

One small detail earns its place. Debugging an endpoint in a browser, copy as cURL yields a complete request; paste that command into Scrapling's interactive shell and it becomes a reusable statement, ready to run once and then fold into a script. The first pass of assembling headers by hand disappears.

Three openings in 0.5.1

The remote session matters because it separates the browser from the process doing the scraping. It connects over a CDP endpoint to a Chromium already running, reuses the existing context by default, inherits the login, and leaves open tabs in place when the session closes. Carrying login state through a crawl used to mean writing and reading state files over and over.

Motion and dragging are for agents. A single screenshot records one instant, and animation, position after scrolling and slider controls have no before or after in a still image. Version 0.5.1 lets the agent read a run of frames and understand what the page is doing, then hold the mouse, move, pause to check and let go. Tool descriptions were compressed at the same time, so the same ability costs less of the context budget.

More than one door for an agent

The MCP server does the work: opening pages, clicking, filling forms, scrolling, screenshots and reading endpoint responses all became tools, so a client can be told to turn the parameters on a page into a table and finish the job. The agent skill does something else. It is a manual kept in step with releases, which a coding assistant consults while writing Scrapling code, so it stops rebuilding an API from stale memory. The command line and the interactive shell are the third path, good for getting one request working before it lands in a script.

Three paths, three kinds of people. Data pipelines mostly use the Python interface, terminal tinkerers like the shell, and those handing collection to an agent touch MCP.

The boundary is still the boundary

The stronger the tool, the earlier the question of what to take. Public documentation, public pricing and published policy papers are ordinary use. A site that plainly forbids automated access, or keeps its content behind a login, is not. The Spider can be told to respect robots.txt and limited to a set of domains, so a crawl stays inside its target. Frequency deserves restraint as well: once or twice a day shows what changed, and a stranger's server is not a load-testing rig.

Scrapling 0.5.1 is worth having not because it can scrape, but because what it scrapes is usable: a page goes in and clean Markdown comes out, an element that moved is recognised again, and an agent can take over the controls. The whole thing is open source and runs locally, at the price of being a tool that expects code, so anyone hoping for plug-and-play will be disappointed. The people who gain are those already building a knowledge base, or wiring a data path for an agent.