Crawlab
Dev tools
web crawler management
SHOW HN2019
116points
18comments
#406of 2,195 in 2019
Top 19%of 2019 by points
How HN took it
CuriousMostly questions and neutral comments
2 positive, 4 neutral or asking, 1 mixed, 0 critical, out of 7 comments about the project. Thread score +0.18.
Docs: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Flexibility and appealing Web UI
From 1 comment
Hi, Thanks for the upvotes. Crawlab is a golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and…
tikazyq on HN
Good project, cool stuff
From 2 comments
A good project, how does it compare to python based scrapy.org. For admin you can use scrapyhub open source or also another project https://github.com/Gerapy/Gerapy
dragonsh on HNCool stuff. Does it really run any language, or only languages that have had integrations written?
lidHanteyk on HN
Objections
Discussion and docs mostly in Chinese; hard for non-Chinese speakers to contribute
From 1 comment
Certainly interesting to see English's domination increasingly challenged on open source tech projects. However this makes contributing harder for non-Chinese speakers. I had a…
pattusk on HN
Demo login with admin/admin does not work
From 1 comment
Looks like a cool project, however I can't seem to get into the demo (it seems to indicate using admin/admin but that doesn't work). Would be great to have an english language…
atymic on HN
Requests
English login page and English documentation
From 2 comments
Looks like a cool project, however I can't seem to get into the demo (it seems to indicate using admin/admin but that doesn't work). Would be great to have an english language…
atymic on HNSounds like you're using redis as a message broker for tasks here. Are you using redis streams?
captainmarble on HN
Questions
How does it compare to scrapy-based admin tools?
From 1 comment
A good project, how does it compare to python based scrapy.org. For admin you can use scrapyhub open source or also another project https://github.com/Gerapy/Gerapy
dragonsh on HN
Does it really run any language?
From 1 comment
Cool stuff. Does it really run any language, or only languages that have had integrations written?
lidHanteyk on HN
Alternatives named
Gerapy
From 1 comment
A good project, how does it compare to python based scrapy.org. For admin you can use scrapyhub open source or also another project https://github.com/Gerapy/Gerapy
dragonsh on HN
scrapydweb, spiderkeeper
From 1 comment
There are a couple of crawler management projects: scrapydweb, spiderkeeper, gerapy, crawlab. The first three are based on scrapyd.
tikazyq on HN