[{"data":1,"prerenderedAt":245},["ShallowReactive",2],{"project-20737035":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":40,"post_text":43,"what":44,"said":95,"maker_in_thread":153,"shape":160,"grid":164,"attempts":207,"same_week":208,"maker_shows":33,"maker_hidden":237,"reader_version":238,"card":239},"20737035","Crawlab","Open-Source Web Crawler Admin Platform That Runs Any Language","title","Show HN: Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language","https:\u002F\u002Fgithub.com\u002Ftikazyq\u002Fcrawlab","github.com","tikazyq",1566216763,2019,116,18,406,2195,19,"notable","dev_tools","web crawler management","web_app",[24,25,26,27],"web crawler","scrapy","crawler management","distributed crawler","curious",0.176,{"positive":31,"neutral":32,"mixed":33,"critical":34},2,4,1,0,7,{"idea":37,"docs":38,"execution":39},{"positive":34,"critical":34},{"positive":33,"critical":34},{"positive":34,"critical":34},[34,34,41,41,34,42,34,42,34,34,34,34,41,34,41,34,41,34],3,5,null,{"problem":45,"audience":43,"pricing":43,"motivation":56,"time_to_build":43,"stack":63,"openness":92,"stage":93,"team_size":94},{"value":46,"source":47,"evidence":48},"Managing and scheduling web crawlers across nodes","inferred",[49,53],{"id":50,"by":51,"text":52},"20743226","chrischen","Scrapy is a web crawler. This project is a web crawler management UI\u002Fplatform, so it presumably manages your scrapy crawlers\u002Finstances and schedules them.",{"id":54,"by":11,"text":55},"20742942","Hi, Thanks for the upvotes. Crawlab is a golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and…",{"value":57,"source":58,"evidence":59},"Promoted in mainland China tech sites; wants more contributors and English support","maker",[60],{"id":61,"by":11,"text":62},"20747471","Thanks all for the upvoting and positive feedbacks for Crawlab. The reason why Crawlab is mainly focused on Chinese is because it was initially promoted in mainland China tech…",[64,68,72,76,80,84],{"value":65,"source":58,"evidence":66},"Go",[67],{"id":54,"by":11,"text":55},{"value":69,"source":58,"evidence":70},"Python",[71],{"id":54,"by":11,"text":55},{"value":73,"source":58,"evidence":74},"Scrapy",[75],{"id":54,"by":11,"text":55},{"value":77,"source":58,"evidence":78},"Puppeteer",[79],{"id":54,"by":11,"text":55},{"value":81,"source":58,"evidence":82},"Selenium",[83],{"id":54,"by":11,"text":55},{"value":85,"source":86,"evidence":87},"Redis","commenter",[88],{"id":89,"by":90,"text":91},"20745051","captainmarble","Sounds like you're using redis as a message broker for tasks here. Are you using redis streams?","unknown","launched","team",{"praise":96,"objections":112,"requests":127,"questions":133,"alternatives":142},[97,101],{"value":98,"source":58,"evidence":99},"Flexibility and appealing Web UI",[100],{"id":54,"by":11,"text":55},{"value":102,"source":86,"evidence":103},"Good project, cool stuff",[104,108],{"id":105,"by":106,"text":107},"20743191","dragonsh","A good project, how does it compare to python based scrapy.org. For admin you can use scrapyhub open source or also another project https:\u002F\u002Fgithub.com\u002FGerapy\u002FGerapy",{"id":109,"by":110,"text":111},"20743278","lidHanteyk","Cool stuff. Does it really run any language, or only languages that have had integrations written?",[113,120],{"value":114,"source":86,"evidence":115},"Discussion and docs mostly in Chinese; hard for non-Chinese speakers to contribute",[116],{"id":117,"by":118,"text":119},"20744009","pattusk","Certainly interesting to see English's domination increasingly challenged on open source tech projects. However this makes contributing harder for non-Chinese speakers. I had a…",{"value":121,"source":86,"evidence":122},"Demo login with admin\u002Fadmin does not work",[123],{"id":124,"by":125,"text":126},"20743564","atymic","Looks like a cool project, however I can't seem to get into the demo (it seems to indicate using admin\u002Fadmin but that doesn't work). Would be great to have an english language…",[128],{"value":129,"source":86,"evidence":130},"English login page and English documentation",[131,132],{"id":124,"by":125,"text":126},{"id":89,"by":90,"text":91},[134,138],{"value":135,"source":86,"evidence":136},"How does it compare to scrapy-based admin tools?",[137],{"id":105,"by":106,"text":107},{"value":139,"source":86,"evidence":140},"Does it really run any language?",[141],{"id":109,"by":110,"text":111},[143,147],{"value":144,"source":86,"evidence":145},"Gerapy",[146],{"id":105,"by":106,"text":107},{"value":148,"source":58,"evidence":149},"scrapydweb, spiderkeeper",[150],{"id":151,"by":11,"text":152},"20746993","There are a couple of crawler management projects: scrapydweb, spiderkeeper, gerapy, crawlab. The first three are based on scrapyd.",{"comments":35,"answered":31,"key":154},[155,156,157],{"id":54,"text":55},{"id":151,"text":152},{"id":158,"text":159},"20742947","Thanks for the feedback. Actually I saw a lot of great Chinese projects on Github trending and sadly they are Chinese only. I would definitely agree they can do better by…",{"commenters":161,"max_depth":32,"maker_replies":35,"first_comment_mins":162,"lifespan_hours":163},11,420,26,[165,167,169,171,173,174,178,180,182,185,189,192,194,196,199,201,203,205],[54,34,34,166,11,33],"project\u002Fpositive\u002Fidea",[105,33,34,168,106,34],"other\u002Fneutral\u002Fnone",[50,33,41,170,51,34],"project\u002Fneutral\u002Fidea",[172,33,41,170,106,34],"20743426",[151,33,34,170,11,33],[175,31,42,176,177,34],"20740818","project\u002Fpositive\u002Fnone","realty_geek",[158,31,34,179,11,33],"other\u002Fneutral\u002Fdocs",[117,31,42,181,118,34],"project\u002Fpositive\u002Fdocs",[183,31,34,168,184,34],"20744293","VvR-Ox",[186,31,34,187,188,34],"20745174","commenter\u002Fneutral\u002Fnone","pictur",[190,31,34,179,191,34],"20745234","dvdkon",[61,41,34,193,11,33],"maker\u002Fneutral\u002Fdocs",[124,32,41,195,125,34],"project\u002Fmixed\u002Fexecution",[197,32,34,198,11,33],"20746565","maker\u002Fneutral\u002Fexecution",[89,42,41,200,90,34],"project\u002Fneutral\u002Fexecution",[202,42,34,200,11,33],"20746947",[109,204,41,170,110,34],6,[206,204,34,170,11,33],"20746967",[],[209,213,218,223,228,233],{"id":210,"name":211,"points":212,"category":20},"20734956","Chart.xkcd",435,{"id":214,"name":215,"points":216,"category":217},"20725311","Saag as a Service",303,"web_saas",{"id":219,"name":220,"points":221,"category":222},"20723366","ClojureScript pixel game engine with Blender live-reloading",285,"games",{"id":224,"name":225,"points":226,"category":227},"20704168","Homer",110,"languages",{"id":229,"name":230,"points":231,"category":232},"20738584","color.clss.hk",106,"creative",{"id":234,"name":235,"points":236,"category":222},"20745142","Easylang games",102,false,"v2",{"id":240,"name":5,"line":6,"ai":237,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":241,"top":18,"kw":242,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":243,"url":9,"mix":244,"praise":98,"objection":114,"sub":21},20737035,0.18,[24,25,26,27],"003305050000303030",[31,32,33,34],1794036579969]