Copy from URL
AI
ai chatbot web content extraction
SHOW HN2024
2points
6comments
#3,379of 3,515 in 2024
Top 97%of 2024 by points
How HN took it
QuietFewer than 5 comments about the project
0 positive, 0 neutral or asking, 0 mixed, 1 critical, out of 1 comments about the project.
Idea: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
Objections
Robots.txt is easily ignored anyway, so the tool adds little
From 1 comment
Or you know you can just ignore robots.txt like everyone else. What do you think robots.txt does exactly?
JSDevOps on HN
Bypassing restrictions is seen as disrespecting the rule of law
From 1 comment
Aka "isn't it annoying to respect the rule of law?" Reminded of this post by https://twitter.com/GergelyOrosz/status/1808515221817885017
Juliate on HN
Questions
What does robots.txt actually do, given it can be ignored
From 1 comment
Or you know you can just ignore robots.txt like everyone else. What do you think robots.txt does exactly?
JSDevOps on HN