Firecrawl vs Scrapy vs Playwright vs BeautifulSoup

Search for how to scrape a website with Python and you’ll keep running into the same names. They don’t do the same job, though. BeautifulSoup parses HTML you've already fetched. Playwright drives a real browser. Scrapy is a framework for crawling thousands of pages. And Firecrawl does the whole job as an API. The right one depends on what your pages look like, how many you need, and what you do with the text afterwards.

The short version: use BeautifulSoup for static pages, Playwright for JavaScript-heavy sites, Scrapy for large or recurring crawls, and Firecrawl when you want clean markdown or JSON without running browsers or crawlers yourself.

To compare them properly, each tool below gets the same task: fetch a documentation page and turn it into markdown. Markdown is what you'd want if the text is going to a language model, and it shows how much cleanup each tool leaves to you.

Firecrawl vs Scrapy vs Playwright vs BeautifulSoup at a Glance

BeautifulSoup: Simple and Fast for Static Pages

BeautifulSoup only parses. You fetch the page with requests, strip the parts you don't want, and convert the rest to markdown with a library such as markdownify:

pip install requests==2.34.2 beautifulsoup4==4.15.0 markdownify==1.2.3

import requests
from bs4 import BeautifulSoup
from markdownify import markdownify

URL = "https://docs.python.org/3/library/asyncio.html"


def to_markdown(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "header", "footer", "aside"]):
        tag.decompose()
    main = soup.find("main") or soup.find(attrs={"role": "main"}) or soup.body
    return markdownify(str(main), heading_style="ATX")


html = requests.get(URL, timeout=30).text
print(to_markdown(html))

On server-rendered pages like Python's docs, this works well, and it's fast because there's no browser involved. However, on pages built by JavaScript, requests gets an empty shell. The cleanup rules only fit the layout you wrote them for, so every new site means adjusting them. And a site with bot protection returns a challenge page instead of the content.

Playwright: Scrape JavaScript Websites with a Real Browser

Playwright fixes the JavaScript problem by loading the page in a real browser. It installs the browser binary separately:

pip install playwright==1.64.1
playwright install chromium

The fetch changes, but the cleanup reuses the same to_markdown function:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(URL, wait_until="networkidle")
    html = page.content()
    browser.close()

print(to_markdown(html))

You now get the page as a user sees it, and you can click, scroll or log in before reading it. What you don't get is help with anything else: the markdown conversion, following links, retrying failures and dealing with bot detection are still your code. A browser per page is also heavy. Each one needs a few hundred megabytes of memory, so running many in parallel takes real infrastructure.

If you're coming from Selenium, Playwright does the same job with a simpler API and automatic waiting for elements. If you’d rather not run browsers yourself, hosted browser services such as Steel and Browserbase give you remote browsers you can drive with the same Playwright code.

Scrapy: Web Crawling at Scale

Scrapy is a different kind of tool. Instead of fetching one page, you write a spider that starts at a URL, extracts what it needs, and decides which links to follow. Scrapy handles the scheduling, concurrency, retries, throttling and robots.txt rules for you:

pip install scrapy==2.19.0 markdownify==1.2.3

This spider converts the asyncio page to markdown, then follows links to the other asyncio pages, up to 20 in total:

import scrapy
from markdownify import markdownify


class DocsSpider(scrapy.Spider):
    name = "docs"
    start_urls = ["https://docs.python.org/3/library/asyncio.html"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "AUTOTHROTTLE_ENABLED": True,
        "CLOSESPIDER_PAGECOUNT": 20,
    }

    def parse(self, response):
        main = response.css("[role=main]").get() or response.css("body").get()
        yield {"url": response.url, "markdown": markdownify(main, heading_style="ATX")}

        for href in response.css("[role=main] a::attr(href)").getall():
            if href.startswith("asyncio"):
                yield response.follow(href, self.parse)

Run it with scrapy runspider docs_spider.py -o docs.jsonl, and you get one JSON line per page. Scrapy only follows each URL once, and it spreads requests out so you don't hammer the site.

Scrapy is the right tool for thousands or millions of pages, where you need control over every request. Like BeautifulSoup, it doesn't run JavaScript. The scrapy-playwright plugin adds a browser for the pages that need one. The markdown cleanup is still yours to write.

Firecrawl: Scrape Any Page to Markdown with One API Call

Firecrawl does the fetching, rendering, cleanup and markdown conversion on its own servers. The same task is a single call:

pip install firecrawl==4.55.0

from firecrawl import Firecrawl

firecrawl = Firecrawl()  # reads FIRECRAWL_API_KEY
doc = firecrawl.scrape(URL, formats=["markdown"], only_main_content=True)
print(doc.markdown)

Firecrawl also crawls and maps whole sites, extracts JSON against a schema, and handles bot protection on the hosted service. The free plan includes 1,000 credits a month, with one credit per page, and paid plans start at $16 a month, billed yearly. The core is open source under AGPL-3.0 if you'd rather self-host, but some of the anti-bot handling is only in the hosted version. My Firecrawl guide covers crawling, mapping and extraction in more depth.

Using Scrapy, Playwright and BeautifulSoup Together

These tools aren't mutually exclusive, and production scrapers often use two. Playwright renders a page and BeautifulSoup parses page.content(), which suits people who prefer BeautifulSoup's API to Playwright's locators. Scrapy with the scrapy-playwright plugin crawls most pages over plain HTTP and opens a browser only for the ones that need JavaScript, so you don't pay the browser cost on every request. And Firecrawl can replace just the hard part: use Scrapy or your own code to decide which URLs to fetch, and send the difficult ones to Firecrawl.

Before reaching for a browser at all, open the page with your browser's developer tools and check the Network tab. Many JavaScript-heavy sites load their data from a JSON endpoint, and calling that directly with requests is faster and more reliable than rendering the page.

Web Scraping for LLMs and RAG

If the scraped text is going to a language model, the output format matters as much as the fetch. HTML wastes most of a context window on markup, navigation and scripts. Markdown with only the main content keeps the structure the model uses, such as headings, lists and code blocks, at a fraction of the tokens.

That's the dividing line in this comparison. BeautifulSoup, Playwright and Scrapy all leave the markdown conversion to you, and the cleanup rules in the examples above need to be adjusted for every site you add. Firecrawl returns markdown that's already cleaned.

Firecrawl Alternatives and Other Scraping Tools

Selenium is the older standard for browser automation and still had over 28 million Python downloads last month, though Playwright now has more than three times that. It’s mainly used for testing now, and for new scraping work Playwright is usually the easier choice.

Crawl4AI is an open-source crawler built on Playwright, aimed at LLM pipelines. It returns cleaned markdown and supports deep crawls and LLM-based extraction. It's the closest self-hosted alternative to Firecrawl.

Jina Reader is the quickest option to try: put https://r.jina.ai/ in front of any URL and you get markdown back. It works without an API key at 20 requests a minute.

Frequently Asked Questions

What's the best Python library for web scraping? There isn't one best library, because they solve different problems. BeautifulSoup is the easiest to start with, Playwright handles JavaScript, Scrapy is the most capable for large crawls, and Firecrawl needs the least code.

Is Firecrawl better than Scrapy? For small to medium jobs where you want clean output fast, Firecrawl needs far less code and handles JavaScript and bot protection for you. For very large crawls where cost per page matters and you want full control, Scrapy is usually cheaper to run.

Is Scrapy better than BeautifulSoup? They do different jobs. BeautifulSoup parses a page you've already downloaded. Scrapy downloads, schedules and crawls pages, and includes its own selectors. Many projects use BeautifulSoup for small scripts and Scrapy once they're crawling at scale.

Can BeautifulSoup or Scrapy scrape JavaScript websites? Not on their own, because neither runs JavaScript. Fetch the page with Playwright first, or add the scrapy-playwright plugin to Scrapy.

Is Playwright better than Selenium for web scraping? For most new projects, yes. Playwright waits for elements automatically, supports Chromium, Firefox and WebKit through one API, and handles modern single-page apps well. Selenium is still widely used, especially in existing test suites.