[{"data":1,"prerenderedAt":326},["ShallowReactive",2],{"project-8616738":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":42,"post_text":45,"what":46,"said":91,"maker_in_thread":186,"shape":195,"grid":200,"attempts":288,"same_week":289,"maker_shows":44,"maker_hidden":319,"reader_version":320,"card":321},"8616738","pyspider","A Python Spider System with Web UI","title","Show HN: A Python Spider System with Web UI","https:\u002F\u002Fgithub.com\u002Fbinux\u002Fpyspider","github.com","binux",1416195303,2014,159,40,153,2272,7,"notable","dev_tools","web crawler","web_app",[24,25,26,27],"python web crawler","web scraping","spider scheduler","phantomjs fetcher","curious",0,{"positive":31,"neutral":32,"mixed":29,"critical":33},3,14,4,21,{"design":36,"idea":38,"docs":39,"execution":40,"performance":41},{"positive":37,"critical":33},2,{"positive":29,"critical":29},{"positive":29,"critical":29},{"positive":29,"critical":29},{"positive":29,"critical":29},[43,31,31,29,31,31,43,31,29,31,29,29,31,29,31,29,31,29,29,31,29,43,31,29,29,29,31,29,44,29,31,44,31,29,29,44,29,29,44,29],5,1,null,{"problem":45,"audience":45,"pricing":45,"motivation":47,"time_to_build":45,"stack":54,"openness":90,"stage":90,"team_size":90},{"value":48,"source":49,"evidence":50},"Came from a vertical search engine project needing to monitor 100+ sites and update pages within 5 minutes","maker",[51],{"id":52,"by":11,"text":53},"8622482","I'm working on a benchmarking suite https:\u002F\u002Fgist.github.com\u002Fbinux\u002F67b276c51e988f8e2c31 and meet some problem... pyspider comes from a vertical search engine project. we have two…",[55,64,72,78,84],{"value":56,"source":49,"evidence":57},"Python",[58,60],{"id":59},"post",{"id":61,"by":62,"text":63},"8616807","meowface","This looks really nice. The API seems more user-friendly than scrapy's.",{"value":65,"source":66,"evidence":67},"MySQL","commenter",[68],{"id":69,"by":70,"text":71},"8617019","mrmondo","Nice project! I do wish it supported a PostgreSQL backend rather than (or as well as I guess) MySQL.",{"value":73,"source":49,"evidence":74},"PhantomJS",[75],{"id":76,"by":11,"text":77},"8618072","http:\u002F\u002Fdemo.pyspider.org\u002Fdebug\u002Fjs_test_sciencedirect is a sample for this. There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some…",{"value":79,"source":49,"evidence":80},"Tornado",[81],{"id":82,"by":11,"text":83},"8619024","I want to make it a http proxy in the beginning. But I found it hard to do so. Then I post every to it, but haven't change the name. But it works like a proxy, that any request…",{"value":85,"source":49,"evidence":86},"RabbitMQ",[87],{"id":88,"by":11,"text":89},"8618200","the architecture of pyspider: http:\u002F\u002Fblog.binux.me\u002Fassets\u002Fimage\u002Fpyspider-arch.png And yes for centralized queue which is in scheduler. It's designed to satisfy about 10-100…","unknown",{"praise":92,"objections":105,"requests":128,"questions":151,"alternatives":172},[93],{"value":94,"source":66,"evidence":95},"Nice UX and API, more user-friendly than scrapy",[96,97,101],{"id":61,"by":62,"text":63},{"id":98,"by":99,"text":100},"8618002","_bitliner","I really like the flow\u002FUX. Congratulations! Nice job! What is the roadmap? I am really inside scraping, it is one of my daily job. I could consider to integrate it in one of my…",{"id":102,"by":103,"text":104},"8616759","bowlofstew","That is a nice tool....nice work!",[106,121],{"value":107,"source":66,"evidence":108},"Package layout uses a generic top-level 'libs' name that may collide when installed",[109,113,117],{"id":110,"by":111,"text":112},"8616840","zbb","Take a look at source code. The package hirarchy is not pythonic (use \"libs\" as top package is not a good idea).",{"id":114,"by":115,"text":116},"8617511","vertex-four","The issue is that if you have two packages installed, and both use \"libs\" as their top-level package, they'll collide. Use \"projectname.common\" instead.",{"id":118,"by":119,"text":120},"8617157","ngoldbaum","Because the name \"libs\" is now installed into the global module namespace. It's better to use a less generic name.",{"value":122,"source":66,"evidence":123},"Setup instructions for the PhantomJS proxy are unclear",[124],{"id":125,"by":126,"text":127},"8618965","maratc","You are running ``` phantomjs phantomjs_fetcher.js ``` and using it as proxy? The setup instructions are a bit unclear on this.",[129,140,144],{"value":130,"source":66,"evidence":131},"Comparison with scrapy",[132,136],{"id":133,"by":134,"text":135},"8617691","adam-_-","How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?",{"id":137,"by":138,"text":139},"8622344","skillachie","+1 Scrapy comparison please Can you compare to scrapy as requested by other posters. Why could you not build on top of scrapy and leverage celery for scheduling etc…",{"value":141,"source":66,"evidence":142},"PostgreSQL backend support",[143],{"id":69,"by":70,"text":71},{"value":145,"source":66,"evidence":146},"GUI-based scraper interface like Portia",[147],{"id":148,"by":149,"text":150},"8616830","Immortalin","Any plans for a gui based web scraper interface similar to portia?",[152,160,166],{"value":153,"source":66,"evidence":154},"How does it compare to scrapy in performance and use",[155,156],{"id":133,"by":134,"text":135},{"id":157,"by":158,"text":159},"8617069","bjblazkowicz","How's the performance compared to scrapy?",{"value":161,"source":66,"evidence":162},"How the queue and architecture work",[163],{"id":164,"by":99,"text":165},"8618174","Because I already have a powerful distributed architecture. I was curious about the architecture of pyspider. For example, how the queue is handled? Is it centralized? Is there a…",{"value":167,"source":66,"evidence":168},"What JavaScript page support means",[169],{"id":170,"by":99,"text":171},"8618013","Furthermore, what you mean with `Javascript pages supported`? Could I just specify where it has to click or do I need to make a reverse engineering of the ajax calls?",[173,178,182],{"value":174,"source":66,"evidence":175},"Scrapy",[176,177],{"id":61,"by":62,"text":63},{"id":133,"by":134,"text":135},{"value":179,"source":66,"evidence":180},"Portia",[181],{"id":148,"by":149,"text":150},{"value":183,"source":66,"evidence":184},"Celery",[185],{"id":137,"by":138,"text":139},{"comments":187,"answered":188,"key":189},13,10,[190,191,192],{"id":52,"text":53},{"id":76,"text":77},{"id":193,"text":194},"8619339","Images not downloaded default. Both the fetcher and the phantomjs proxy is totally async.",{"commenters":196,"max_depth":197,"maker_replies":187,"first_comment_mins":198,"lifespan_hours":199},24,8,9,29,[201,203,205,206,208,212,214,215,217,218,222,223,225,226,227,229,232,233,234,238,242,245,247,248,250,253,256,257,259,261,264,268,269,272,274,277,278,281,283,286],[61,29,43,202,62,29],"project\u002Fpositive\u002Fdesign",[133,44,31,204,134,29],"project\u002Fneutral\u002Fidea",[137,44,31,204,138,29],[52,44,29,207,11,44],"maker\u002Fneutral\u002Fidea",[209,37,31,210,211,29],"8617054","project\u002Fneutral\u002Fdocs","OedipusRex",[69,31,31,213,70,29],"project\u002Fneutral\u002Fnone",[98,33,43,202,99,29],[170,33,31,216,99,29],"project\u002Fneutral\u002Fexecution",[76,33,29,216,11,44],[219,33,31,220,221,29],"8619213","project\u002Fneutral\u002Fperformance","pknerd",[193,33,29,220,11,44],[224,33,29,207,11,44],"8618100",[164,33,31,216,99,29],[88,33,29,216,11,44],[228,33,31,220,126,29],"8618763",[230,33,29,231,11,44],"8618847","project\u002Fpositive\u002Fexecution",[125,33,31,210,126,29],[82,33,29,210,11,44],[235,43,29,236,237,29],"8617879","other\u002Fneutral\u002Fnone","kidsil",[239,240,31,210,241,29],"8617880",6,"erikb",[243,240,29,244,11,44],"8618105","maker\u002Fneutral\u002Fnone",[102,18,43,246,103,29],"project\u002Fpositive\u002Fnone",[148,197,31,204,149,29],[249,197,29,204,11,44],"8616863",[251,197,29,236,252,29],"8618795","prht",[254,197,29,236,255,29],"8619859","smoe",[157,198,31,220,158,29],[258,198,29,220,11,44],"8622943",[110,188,44,260,111,29],"project\u002Fcritical\u002Fdesign",[262,188,29,263,11,44],"8616897","commenter\u002Fneutral\u002Fdesign",[265,188,31,266,267,29],"8617101","project\u002Fneutral\u002Fdesign","redacted",[118,188,44,260,119,29],[270,188,31,266,271,29],"8617460","iamtew",[273,188,29,266,11,44],"8618032",[275,188,29,263,276,29],"8617105","paulhauggis",[114,188,44,260,115,29],[279,188,29,263,280,29],"8617574","fmueller",[282,188,29,263,115,29],"8620826",[284,188,44,260,285,29],"8617284","huskyr",[287,188,29,266,11,44],"8617319",[],[290,295,300,305,310,315],{"id":291,"name":292,"points":293,"category":294},"8608358","Explained Visually",353,"learning_science",{"id":296,"name":297,"points":298,"category":299},"8631158","LanguageExt",164,"languages",{"id":301,"name":302,"points":303,"category":304},"8615416","SineRider",137,"games",{"id":306,"name":307,"points":308,"category":309},"8631223","Picky Pint",134,"consumer_social",{"id":311,"name":312,"points":313,"category":314},"8623864","WikiWash",131,"productivity",{"id":316,"name":317,"points":318,"category":304},"8627793","Axis",110,false,"v2",{"id":322,"name":5,"line":6,"ai":319,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":29,"top":18,"kw":323,"maker":11,"domain":10,"tier":19,"attempts":29,"strip":324,"url":9,"mix":325,"praise":94,"objection":107,"sub":21},8616738,[24,25,26,27],"5330335303003030300305300030103130010010",[31,32,29,33],1794036525153]