Crawlee for Python
Dev tools
web scraping library
SHOW HN2024
How HN took it
9 positive, 15 neutral or asking, 1 mixed, 7 critical, out of 32 comments about the project. Thread score +0.08.
Design: 2 for, 3 againstDocs: 2 for, 2 againstPerformance: 3 for, 0 againstIdea: 0 for, 2 againstPricing: 2 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Clean, minimal API that is easy to start with
From 3 comments
hey intev, - Crawlee has out-of-the-box support for headless browser crawling (Playwright). You don't have to install any plugin or set up the middleware. - Crawlee has a…
sauain on HNCrawlee isn’t any less configurable than Scrapy. It just uses different, in my personal opinion more approachable, patterns. It makes it easier to start with, but you can tweak…
mnmkng on HNI found crawlee a few days ago while figuring out a stack for a project. I wanted a python library but found crawlee with typescript so much easier that I ended up coding the…
ranedk on HN
Works well for JavaScript-rendered sites via Playwright
From 2 comments
I found crawlee a few days ago while figuring out a stack for a project. I wanted a python library but found crawlee with typescript so much easier that I ended up coding the…
ranedk on HNIt uses Playwright under the hood, so yes, it can do all of that, and more.
mnmkng on HN
Objections
Python documentation is incomplete; some features like tiered proxies are undocumented
From 2 comments
You'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would…
mdaniel on HNI literally had to go through the entire codebase the documentation is that lacking. It’s boring to document but imo it’s the lowest hanging fruit to get people moving down that…
bn-l on HN
Less flexible than middleware-based frameworks like Scrapy for custom extensions
From 2 comments
> You don't have to install any plugin or set up the middleware. That cuts both ways, in true 80/20 fashion: it also means that anyone who isn't on the happy path of the way that…
mdaniel on HN> Crawlee isn’t any less configurable than Scrapy. Oh, then I have obviously overlooked how one would be able to determined if a proxy has been blocked and evict it from the pool…
mdaniel on HN
Requests
Robots.txt and opt-out protocol support
From 1 comment
Does it have support for web scraping opt-out protocols, such as Robots.txt, HTTP and content tags? These are getting more important now, especially in the EU after the DSM…
Findecanor on HN
More code examples and documentation
From 2 comments
I'd suggest bringing more code snippets from the test cases to documentation as examples. Nice work though.
fforflo on HNYou'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would…
mdaniel on HN
Questions
How does it differ from Scrapy
From 1 comment
How is this different from Scrapy?
intev on HN
Whether it supports SSO-protected intranet sites and auto-login
From 2 comments
Can this work on intranet sites like sharepoint or confluence , which require employee SSO ? I was trying to build a small Langchain based RAG based on internal documents but…
manishsharan on HNCan you use this to auto-logon to systems?
bmitc on HN
Alternatives named
Scrapy
From 1 comment
How is this different from Scrapy?
intev on HN
Selenium
From 1 comment
How does it compare to selenium for python?
3abiton on HN
PhantomJS
From 1 comment
Does it have event listeners to wait for specific elements based on certain pattern matches. One reason i am still using phantomjs is because it simulates the entire browser and…
holoduke on HN
Golang colly
From 1 comment
Cache management is also what I had in mind. Ive been using golang+colly and the default caching behavior is just different enough from what I need. I haven't written a custom…
8organicbits on HN