[{"data":1,"prerenderedAt":400},["ShallowReactive",2],{"project-8787783":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":46,"post_text":49,"what":50,"said":86,"maker_in_thread":222,"shape":233,"grid":237,"attempts":361,"same_week":362,"maker_shows":33,"maker_hidden":392,"reader_version":393,"card":394},"8787783","Espion","Scrape the web by injecting JavaScript into web pages","title","Show HN: Scrape the web by injecting JavaScript into web pages","http:\u002F\u002Fespion.io\u002F","espion.io","TheEspion",1419335550,2014,109,54,255,2272,12,"notable","web_saas","web scraping service","web_app",[24,25,26,27],"web scraping","headless browser","proxy rotation","captcha solving","curious",-0.281,{"positive":31,"neutral":32,"mixed":33,"critical":34},4,16,1,11,32,{"idea":37,"pricing":40,"execution":42,"design":43,"performance":44,"privacy":45},{"positive":38,"critical":39},2,6,{"positive":41,"critical":33},0,{"positive":41,"critical":33},{"positive":33,"critical":41},{"positive":41,"critical":33},{"positive":41,"critical":33},[33,33,47,33,33,47,41,33,47,33,47,41,41,48,41,47,41,33,41,33,33,47,41,33,47,47,41,41,47,48,41,47,41,47,47,47,48,47,41,47],3,5,null,{"problem":51,"audience":61,"pricing":65,"motivation":71,"time_to_build":49,"stack":75,"openness":84,"stage":85,"team_size":84},{"value":52,"source":53,"evidence":54},"Extracting data from websites without building and maintaining scraping infrastructure","maker",[55,58],{"id":56,"by":11,"text":57},"8788293","Espion is similar to PhantomJS in that they're both headless browsers that you can inject JavaScript into. Espion comes with a lot more. First there is the infrastructure:…",{"id":59,"by":11,"text":60},"8788027","We target businesses that want to fulfil their web data extraction needs in-house rather than hiring a third-party provider – which we originally are. Espion was built for our…",{"value":62,"source":53,"evidence":63},"Businesses that want web data extraction in-house rather than from a third-party provider",[64],{"id":59,"by":11,"text":60},{"value":66,"source":53,"evidence":67},"Packages for a set number of pages, no timeframe; prices not final",[68],{"id":69,"by":11,"text":70},"8788102","The packages are for a set number of pages, with no timeframe. I don't have prices yet for a load in the millions of pages per hour. What kind of system do you feed data into?",{"value":72,"source":53,"evidence":73},"Built for the maker's own scraping needs first",[74],{"id":59,"by":11,"text":60},[76],{"value":77,"source":78,"evidence":79},"PhantomJS (headless browser, per commenter)","commenter",[80],{"id":81,"by":82,"text":83},"8788700","asdf123456","it's just PhantomJS wrapped up behind a cloud","unknown","prototype",{"praise":87,"objections":106,"requests":140,"questions":162,"alternatives":188},[88,95],{"value":89,"source":78,"evidence":90},"Good site design",[91],{"id":92,"by":93,"text":94},"8788845","Jayd2014","Does the location of Espion in Mauritius have anything to do with this? Are the servers located there?. Good design of the site and good work.",{"value":96,"source":78,"evidence":97},"Useful for businesses lacking time to build scraping setups",[98,102],{"id":99,"by":100,"text":101},"8787888","corford","Or for scraping anything non-trivial, do it yourself with Casperjs, AWS\u002FLinode\u002FDO, RabbitMQ and a bit of monitoring\u002Falerting from someone like Datadog. It will be cheaper and a…",{"id":103,"by":104,"text":105},"8788701","softdev12","The general sentiment on this thread pretty much sums up the idea of \"scraping as a service\". To me, there is definitely a legitimate business need to be able scrape. Whether…",[107,111,118,133],{"value":108,"source":78,"evidence":109},"Likely PhantomJS wrapped in a cloud service",[110],{"id":81,"by":82,"text":83},{"value":112,"source":78,"evidence":113},"IP pools get blacklisted at scale",[114],{"id":115,"by":116,"text":117},"8790131","ivanhoe","We run in-house data extraction structure very similar to this (spiders, phantomJS, ocr, anonymous proxies, etc), and it indeed takes some time to set it up properly. Main…",{"value":119,"source":78,"evidence":120},"Legal and abuse risk from scraping and spam",[121,125,129],{"id":122,"by":123,"text":124},"8788288","aikah","> CAPTCHAs and IP blacklistings are things you encounter routinely in perfectly legitimate and legal web scraping projects It's only legal if the site TOS says it is.You dont get…",{"id":126,"by":127,"text":128},"8789204","logn","What if instead of scraping, customers use the JS injection features for spamming? How are you approaching the liability issues involved, given that potentially anyone can sign…",{"id":130,"by":131,"text":132},"8788114","gingerlime","It sounds a bit in a grey area, in-between legit use and something slightly dodgy (mentioning IP blacklisting and overriding CAPTCHAS...) I'm curious to know what kind of…",{"value":134,"source":78,"evidence":135},"Slower than non-JS scraping",[136],{"id":137,"by":138,"text":139},"8790114","ankimal","Can you talk about performance? Having written both vanilla non-js scrapers (which get you far enough) and js scrapers for those pesky web pages, I can confirm that our js…",[141,148,155],{"value":142,"source":78,"evidence":143},"GUI for selecting elements and generating scraper code",[144],{"id":145,"by":146,"text":147},"8789688","ivanca","This is an interested idea, but what I would pay for is a GUI for scrapping, for example: clicking an element, seeing what classes it got, and be able to click each class to see…",{"value":149,"source":78,"evidence":150},"Video recording of scraped pages",[151],{"id":152,"by":153,"text":154},"8789271","AlwaysBCoding","Just wondering, can the scraper record video of the site it's on? I feel like that's something I have a lot of use cases for. Even if it's not possible is there some hack you…",{"value":156,"source":78,"evidence":157},"Clearer non-technical front page copy",[158],{"id":159,"by":160,"text":161},"8788077","dalacv","Since this looks to be a product targeting businesses, I would say you might want to dumb down your techie speak on the front page. Talk to more of the benefits and value that…",[163,174,181],{"value":164,"source":78,"evidence":165},"How it differs from PhantomJS or Zombie.js",[166,170],{"id":167,"by":168,"text":169},"8788165","LunaSea","How is it different than Phantom.js or Zombie.js ?",{"id":171,"by":172,"text":173},"8788603","gildas","So, is the Espion headless browser based on an existing one? If this is the case, which one?",{"value":175,"source":78,"evidence":176},"Who the target customer is",[177],{"id":178,"by":179,"text":180},"8787973","xyby","Who is your target customer?",{"value":182,"source":78,"evidence":183},"Legality of scraping and terms of service",[184,185],{"id":122,"by":123,"text":124},{"id":186,"by":11,"text":187},"8788403","It's more complex than that. Whether the site's TOS are enforceable is a matter of jurisdiction and may depend on the intent of the web scraper. Ryanair for example has lost…",[189,197,208,215],{"value":190,"source":78,"evidence":191},"PhantomJS and CasperJS",[192,193],{"id":99,"by":100,"text":101},{"id":194,"by":195,"text":196},"8788064","dale-cooper","Sounds more like phantomjs\u002Fcasperjs.",{"value":198,"source":78,"evidence":199},"Selenium",[200,204],{"id":201,"by":202,"text":203},"8789672","dmritard96","you can do this with selenium also btws",{"id":205,"by":206,"text":207},"8789785","aaronem","Selenium could do that for you, and there's an IDE available as a Firefox plugin; branching might be tricky and you'd need to build some tooling around it in general, but it…",{"value":209,"source":78,"evidence":210},"Kimono Labs",[211],{"id":212,"by":213,"text":214},"8790338","juddernaught","If you are looking for a GUI for scraping, check out kimono labs. It's a GUI tool for scraping, and requires no code to set up and can find all similar elements from the one you…",{"value":216,"source":78,"evidence":217},"Cheerio",[218],{"id":219,"by":220,"text":221},"8787861","bitplanets","This is cheerio with node.js and extra bells. Example: http:\u002F\u002Fprojs.hackhat.com\u002Fthe-making-of-the-hackernews-crawler-interesting-tips-to-know\u002F",{"comments":223,"answered":224,"key":225},13,10,[226,227,230],{"id":56,"text":57},{"id":228,"text":229},"8788130","Thanks, we'll definitely do that.",{"id":231,"text":232},"8788143","CAPTCHAs and IP blacklistings are things you encounter routinely in perfectly legitimate and legal web scraping projects. A typical example are businesses that want to monitor…",{"commenters":234,"max_depth":39,"maker_replies":223,"first_comment_mins":235,"lifespan_hours":236},34,36,19,[238,240,241,243,245,246,248,251,254,258,262,264,265,266,268,269,270,271,273,276,279,282,284,286,288,290,292,295,299,302,303,304,309,310,312,316,318,321,324,327,331,333,336,339,341,342,343,345,347,349,351,355,358,360],[219,41,33,239,220,41],"project\u002Fcritical\u002Fidea",[194,41,33,239,195,41],[56,41,41,242,11,33],"maker\u002Fneutral\u002Fidea",[171,41,47,244,172,41],"project\u002Fneutral\u002Fnone",[81,41,33,239,82,41],[99,33,33,247,100,41],"project\u002Fcritical\u002Fpricing",[249,33,47,244,250,41],"8787916","gondo",[252,33,47,253,100,41],"8788080","project\u002Fneutral\u002Fexecution",[255,33,41,256,257,41],"8788575","commenter\u002Fneutral\u002Fnone","eli",[259,33,33,260,261,41],"8789106","project\u002Fcritical\u002Fexecution","mhluongo",[159,38,47,263,160,41],"maker\u002Fneutral\u002Fdesign",[228,38,41,263,11,33],[115,47,33,239,116,41],[130,31,47,267,131,41],"project\u002Fneutral\u002Fidea",[231,31,41,242,11,33],[122,31,41,256,123,41],[186,31,41,256,11,33],[92,31,48,272,93,41],"project\u002Fpositive\u002Fdesign",[274,31,41,275,11,33],"8788895","maker\u002Fneutral\u002Fnone",[277,31,41,256,278,41],"8789202","sxv",[280,31,47,244,281,41],"8788979","psykovsky",[283,31,41,242,11,33],"8789224",[285,31,47,244,281,41],"8789430",[126,48,33,287,127,41],"project\u002Fcritical\u002Fnone",[289,48,41,275,11,33],"8789247",[137,39,33,291,138,41],"project\u002Fcritical\u002Fperformance",[293,39,41,294,11,33],"8791486","maker\u002Fneutral\u002Fperformance",[296,297,33,239,298,41],"8787890",7,"volker48",[145,300,47,301,146,41],8,"project\u002Fneutral\u002Fdesign",[205,300,41,256,206,41],[212,300,41,256,213,41],[305,306,33,307,308,41],"8787862",9,"project\u002Fcritical\u002Fprivacy","baldfat",[152,224,47,244,153,41],[311,224,41,275,11,33],"8789665",[313,34,47,314,315,41],"8788054","project\u002Fneutral\u002Fpricing","jalfresi",[69,34,41,317,11,33],"maker\u002Fneutral\u002Fpricing",[319,34,41,320,315,41],"8788275","other\u002Fneutral\u002Fnone",[322,34,47,294,323,41],"8788168","MoOmer",[325,34,41,326,315,41],"8788268","other\u002Fneutral\u002Fperformance",[328,34,48,329,330,41],"8788224","project\u002Fpositive\u002Fidea","willu",[332,34,41,320,315,41],"8788258",[334,34,41,256,335,41],"8789211","bolasanibk",[337,18,47,267,338,41],"8789147","Axsuul",[340,18,41,263,11,33],"8789196",[178,223,47,242,179,41],[59,223,41,242,11,33],[103,223,47,344,104,41],"project\u002Fmixed\u002Fidea",[346,223,47,242,179,41],"8789049",[348,223,48,329,116,41],"8790212",[167,350,47,267,168,41],14,[352,353,47,244,354,41],"8790064",15,"codexon",[356,353,41,357,11,33],"8791736","project\u002Fpositive\u002Fnone",[359,353,48,357,354,41],"8791894",[201,32,33,239,202,41],[],[363,368,373,378,383,388],{"id":364,"name":365,"points":366,"category":367},"8766865","Wysihtml",215,"dev_tools",{"id":369,"name":370,"points":371,"category":372},"8787733","Startup Fighter",117,"games",{"id":374,"name":375,"points":376,"category":377},"8774257","ping.gg",95,"infrastructure",{"id":379,"name":380,"points":381,"category":382},"8782320","reread",81,"consumer_social",{"id":384,"name":385,"points":386,"category":387},"8790970","Chicory",76,"languages",{"id":389,"name":390,"points":391,"category":372},"8782069","Rudolph Runner",61,false,"v2",{"id":395,"name":5,"line":6,"ai":392,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":396,"top":18,"kw":397,"maker":11,"domain":10,"tier":19,"attempts":41,"strip":398,"url":9,"mix":399,"praise":89,"objection":108,"sub":21},8787783,-0.28,[24,25,26,27],"1131130131300503010113013300350303335303",[31,32,33,34],1793990917179]