Crul
Dev tools
web scraping and api query tool
SHOW HN2023
How HN took it
7 positive, 19 neutral or asking, 2 mixed, 12 critical, out of 40 comments about the project. Thread score -0.03.
Execution: 1 for, 6 againstIdea: 3 for, 3 againstDesign: 0 for, 1 againstPricing: 0 for, 1 againstDocs: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Filter command is powerful for exploring data
From 1 comment
I just started playing with CRUL recently to try to map out different media download links on various websites, in an effort to avoid clicking around looking for hidden content.…
dishwishy on HN
Slick tool
From 1 comment
That's a pretty slick tool you guys built!
the_giver on HN
Objections
Closed source, though open sourcing is planned
From 2 comments
looks like it is closed source: https://github.com/crul-oss
2h on HNThis looks very cool. Congrats on the launch! Have you thought about open sourcing it? I realize you want to sustain a business, but an open core model with commercial features…
imiric on HN
Slow default throttle of one request per second per domain
From 1 comment
> status: complete (28.284 seconds / 14 results) Why was it so slow? Is there a default delay between requests or something?
KomoD on HN
Struggles with nested JSON and structural website changes
From 2 comments
I played around with this trying it on a few tricky cases. At least for the initial step of getting the data in a tidy format, it wasn't immediately obvious to me how crul could…
djhn on HNHow does crul handle the dynamic nature of the web? Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is…
jawns on HN
Name is too close to curl
From 1 comment
Man, this looks cool, the name's closeness to "curl" irritates me..
netsharc on HN
Requests
Open source the project
From 2 comments
This looks very cool. Congrats on the launch! Have you thought about open sourcing it? I realize you want to sustain a business, but an open core model with commercial features…
imiric on HNlooks like it is closed source: https://github.com/crul-oss
2h on HN
Add a Crul datasource or FDW integration
From 2 comments
Hey, just watched the video. This looks super useful! I'm the founder of WunderGraph (https://wundergraph.com) and we allow our users to easily integrate multiple data sources…
jensneuse on HNI'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could…
chatmasta on HN
Support for HTML table extraction
From 1 comment
Do you have an example of how to turn an html table element into a CSV? I saw the open and the scrape commands, but wasn't sure where to go from there.
throwaway_e9463 on HN
Questions
How to handle anti-bot blocking
From 1 comment
How do you plan on tackling anti-bot blocking?
asdadsdad on HN
How it handles changes in web page structure
From 1 comment
How does crul handle the dynamic nature of the web? Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is…
jawns on HN
Alternatives named
curl and jq
From 1 comment
Please share some data wrangling challenges that support this statement: "The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and…
1vuio0pswjnm7 on HN
Puppeteer, Playwright, Selenium
From 1 comment
Hi HN, we’re Carl and Nic, the creators of crul (https://www.crul.com), and we’ve been hard at work for the last year and a half building our dream of turning the web into a…
portInit on HN
Postman
From 1 comment
Hi HN, we’re Carl and Nic, the creators of crul (https://www.crul.com), and we’ve been hard at work for the last year and a half building our dream of turning the web into a…
portInit on HN
Airbyte and Splitgraph
From 1 comment
I'll make this same offer for Splitgraph :) If you feel like writing a Postgres FDW then we can add it to the engine on the backend, so that anyone with a Postgres client could…
chatmasta on HN