[{"data":1,"prerenderedAt":932},["ShallowReactive",2],{"project-7066479":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":50,"post_text":52,"what":53,"said":85,"maker_in_thread":208,"shape":221,"grid":225,"attempts":899,"same_week":900,"maker_shows":18,"maker_hidden":924,"reader_version":925,"card":926},"7066479","Kimono","Never write a web scraper again","title","Show HN: Kimono – Never write a web scraper again","http:\u002F\u002Fkimonify.kimonolabs.com\u002Fkimload?url=http%3A%2F%2Fwww.kimonolabs.com%2Fwelcome.html","kimonify.kimonolabs.com","pranade",1389823190,2014,717,230,6,2272,1,"legend","dev_tools","web scraping","web_app",[24,25,26,27],"web scraper","scraping api","web data extraction","no code scraping","mixed",0.104,{"positive":31,"neutral":32,"mixed":33,"critical":34},32,51,12,25,120,{"execution":37,"design":40,"idea":42,"privacy":45,"pricing":47,"performance":48,"docs":49},{"positive":38,"critical":39},5,14,{"positive":33,"critical":41},2,{"positive":43,"critical":44},8,4,{"positive":46,"critical":41},0,{"positive":46,"critical":18},{"positive":18,"critical":18},{"positive":46,"critical":46},[51,41,51,46,46,41,51,41,51,51,44,18,38,51,38,18,51,41,18,18,44,38,44,38,51,51,51,51,51,44,44,44,41,51,51,51,51,44,44,44],3,null,{"problem":54,"audience":62,"pricing":71,"motivation":78,"time_to_build":52,"stack":82,"openness":83,"stage":84,"team_size":83},{"value":55,"source":56,"evidence":57},"Writing and maintaining web scrapers to get data that has no API","commenter",[58],{"id":59,"by":60,"text":61},"7068521","bambax","> Web scraping. It's something we all love to hate. You wish the data you needed to power your app, model or visualization was available via API. But, most of the time it's not.…",{"value":63,"source":64,"evidence":65},"Developers and data users who need data from sites without APIs","inferred",[66,67],{"id":59,"by":60,"text":61},{"id":68,"by":69,"text":70},"7066705","phillmv","The UX is great and a journalists everywhere will thank you. But outside of government websites I don't see how a lot of this is even legal, per se?",{"value":72,"source":73,"evidence":74},"Free during beta for 30 APIs","maker",[75],{"id":76,"by":11,"text":77},"7067060","Yes, we're in beta right now, while we're still working out the bugs. For beta, it's free for 30 APIs.",{"value":79,"source":64,"evidence":80},"Scraping is painful to build and maintain, so make it easy",[81],{"id":59,"by":60,"text":61},[],"unknown","beta",{"praise":86,"objections":98,"requests":122,"questions":163,"alternatives":177},[87],{"value":88,"source":56,"evidence":89},"Useful for simplifying common scraping tasks",[90,94],{"id":91,"by":92,"text":93},"7066658","hcarvalhoalves","This is excellent. Even it if doesn't work for scraping all sites, it simplifies the average use case so much that it's not even funny. Feature proposal: deal with pagination.",{"id":95,"by":96,"text":97},"7067510","jlees","I like how you've thought through the end to end use case: not just generating an API, but actually making it usable. I've done my fair share of web scraping and it's not an easy…",[99,110],{"value":100,"source":56,"evidence":101},"Breaks on many sites and buggy saves\u002Fselection",[102,106],{"id":103,"by":104,"text":105},"7066834","randomdrake","The presentation is beautiful and the website is great, but the tech broke so I have no idea how or if this even works. This is a wonderful concept and one I've talked about…",{"id":107,"by":108,"text":109},"7067027","fsckin","Constructive Tone: I figured that it might be nifty to scrape cedar pollen count information from a calendar and then shoot myself an email when it was higher than 100 gr\u002Fm3.…",{"value":111,"source":56,"evidence":112},"Legality and IP bans from scraping at scale",[113,117,121],{"id":114,"by":115,"text":116},"7066720","georgemcbay","I've written more web scraping code than I care to admit. A lot of the apps that ran on chumby devices used scraping to get their data (usually(!) with the consent of the website…",{"id":118,"by":119,"text":120},"7068090","GigabyteCoin","I'm curious how you plan to avoid\u002Fcircumvent the inevitable hard IP ban that the largest (and most sought after targets) will place on you and your services once you begin to…",{"id":68,"by":69,"text":70},[123,136,143,153],{"value":124,"source":73,"evidence":125},"Authentication \u002F login support",[126,129,132],{"id":127,"by":11,"text":128},"7067106","Yes, it's one of the most popular feature requests. We don't support auth yet, but it's on our shortlist and we hope to have it ready soon.",{"id":130,"by":11,"text":131},"7070828","We're working on auth... it's the most requested feature at the moment. And we're still beta at the moent, so all usage is free",{"id":133,"by":134,"text":135},"7067092","DanBlake","Show me it working with authentication and you will have a customer. Scraping is always something you need to write because the shit you want to get is only shown when you are…",{"value":137,"source":73,"evidence":138},"Pagination and crawling",[139,142],{"id":140,"by":11,"text":141},"7066740","thanks. glad you like it. pagination, dynamic tabs (and crawling in general) is a big feature we really want to add soon. a lot of people are asking for it. the challenge will be…",{"id":91,"by":92,"text":93},{"value":144,"source":56,"evidence":145},"POST support and multiple targets from a list",[146,149],{"id":147,"by":11,"text":148},"7067173","No, we don't have POST support quite yet. We're working on a solution.",{"id":150,"by":151,"text":152},"7068069","thinkzig","Great work so far. The tool was very intuitive and easy to use. My suggestion: once I've defined an API, let me apply it to multiple targets that I supply to you programatically.…",{"value":154,"source":56,"evidence":155},"Image extraction",[156,160],{"id":157,"by":158,"text":159},"7066966","ThomPete","Thank you for building a tool I been wanting so I don't have to! Can't wait to play around with this tonight. Suggestion. Allow one to select images.",{"id":161,"by":11,"text":162},"7066711","It's a great suggestion, thanks! ... image extraction would be cool, and it's on our shortlist of features to build next",[164,172],{"value":165,"source":56,"evidence":166},"How it handles robots.txt and legality",[167,171],{"id":168,"by":169,"text":170},"7066623","ForHackernews","This looks really slick. What happens if a website you're scraping changes its design? Do you respect robots.txt?",{"id":68,"by":69,"text":70},{"value":173,"source":56,"evidence":174},"How it adapts when source sites change",[175,176],{"id":59,"by":60,"text":61},{"id":168,"by":169,"text":170},[178,185,192,199],{"value":179,"source":56,"evidence":180},"Simile\u002FMIT web augmentation research",[181],{"id":182,"by":183,"text":184},"7066977","dunham","The Simile group at MIT did something similar back around 2006. Automatic identification of collections in web pages (repeated structures), detection of fields by doing tree…",{"value":186,"source":56,"evidence":187},"ScraperWiki",[188],{"id":189,"by":190,"text":191},"7066934","rlpb","Are you familiar with ScraperWiki? I'm wondering how your work fits in with it. Edit: looks like they've moved away from that space, but have an old version available at:…",{"value":193,"source":56,"evidence":194},"SelectorGadget",[195],{"id":196,"by":197,"text":198},"7067616","tectonic","Just use http:\u002F\u002Fselectorgadget.com",{"value":200,"source":56,"evidence":201},"iMacros, Beautiful Soup, Nokogiri",[202,203,204],{"id":118,"by":119,"text":120},{"id":59,"by":60,"text":61},{"id":205,"by":206,"text":207},"7066580","jmcgough","Really sleek interface, and looks like it could be extremely useful (I just spent a few hours cranking out Nokogiri this morning). Oh, typo: \"Notice that toolbar at the toop of…",{"comments":209,"answered":210,"key":211},59,23,[212,215,218],{"id":213,"text":214},"7066859","Sorry you had a bad first experience. We've tested this on a lot of sites, and it works stably across a lot of different cases, but haven't solved it everywhere. Thanks for…",{"id":216,"text":217},"7067301","Thanks for the suggestions on tool tips... We can also expand the tutorials section to make this more clear.",{"id":219,"text":220},"7067850","Kimono can handle several pages with malformed and old\u002Fbad HTML. We're still in beta though, so we're handling more edge cases as we encounter them. Should work on HN main page.",{"commenters":222,"max_depth":223,"maker_replies":209,"first_comment_mins":44,"lifespan_hours":224},141,7,326,[226,228,230,233,237,239,243,247,250,253,255,258,262,265,269,272,276,278,281,284,287,290,293,297,300,302,305,308,310,312,315,318,321,325,327,330,334,337,340,341,342,346,350,353,356,358,362,363,365,368,370,373,376,378,382,384,386,387,389,391,392,393,396,399,401,404,406,408,410,411,413,415,416,418,420,422,425,429,434,436,439,442,445,449,452,456,459,461,465,467,469,472,475,477,479,481,482,485,487,490,494,496,500,504,506,511,514,517,520,523,526,528,531,535,539,541,543,545,548,550,552,554,556,559,563,565,568,570,572,575,579,584,588,590,594,596,600,602,604,606,609,610,612,614,616,619,621,623,627,630,632,637,639,642,647,651,655,659,660,664,666,671,674,677,681,683,687,692,693,695,697,699,703,707,709,713,715,719,722,726,728,732,736,740,742,746,748,751,753,756,758,762,765,769,772,774,779,783,786,788,792,796,800,803,807,809,813,815,818,820,824,828,832,836,838,842,845,849,851,855,859,863,866,871,875,879,883,887,891,895],[103,46,18,227,104,46],"project\u002Fcritical\u002Fexecution",[213,46,46,229,11,18],"maker\u002Fneutral\u002Fexecution",[231,46,38,232,104,46],"7066921","maker\u002Fpositive\u002Fexecution",[234,46,51,235,236,46],"7067285","project\u002Fmixed\u002Fdesign","faceyspacey",[216,46,46,238,11,18],"maker\u002Fneutral\u002Fdocs",[240,46,38,241,242,46],"7067558","project\u002Fpositive\u002Fexecution","sswaner",[244,46,51,245,246,46],"7067620","project\u002Fmixed\u002Fexecution","BinaryBird",[248,46,18,227,249,46],"7067651","ricardobeat",[251,46,18,227,252,46],"7067760","voltagex_",[219,46,46,254,11,18],"project\u002Fneutral\u002Fexecution",[256,46,51,257,252,46],"7068189","maker\u002Fneutral\u002Fnone",[259,46,46,260,261,46],"7069162","other\u002Fneutral\u002Fnone","tlarkworthy",[263,46,51,254,264,46],"7074680","Nilzor",[266,46,46,267,268,46],"7068193","other\u002Fneutral\u002Fdesign","wilg",[270,46,46,260,271,46],"7069032","tinco",[273,46,46,274,275,46],"7069618","commenter\u002Fneutral\u002Fnone","theg2",[277,46,46,274,271,46],"7069675",[279,46,46,274,280,46],"7070168","Flimm",[282,46,46,260,283,46],"7072523","SkyMarshal",[285,46,46,260,286,46],"7078008","bmelton",[288,46,46,260,289,46],"7070050","elwell",[291,46,46,267,292,46],"7069217","precisioncoder",[294,46,46,295,296,46],"7069714","commenter\u002Fneutral\u002Fdesign","LordIllidan",[298,46,46,267,299,46],"7069827","romaniv",[301,46,46,274,296,46],"7072159",[303,46,46,267,304,46],"7071167","Houshalter",[306,46,46,260,307,46],"7068904","roel_v",[309,46,46,260,249,46],"7069368",[311,46,46,229,11,18],"7070966",[313,46,46,260,314,46],"7069017","easy_rider",[316,46,51,254,317,46],"7069444","riffraff",[319,46,51,254,320,46],"7070361","ronaldx",[322,46,18,323,324,46],"7068338","project\u002Fcritical\u002Fnone","motyar",[182,18,51,326,183,46],"project\u002Fneutral\u002Fidea",[328,18,46,329,11,18],"7066988","maker\u002Fpositive\u002Fidea",[331,18,51,332,333,46],"7067124","project\u002Fneutral\u002Fnone","tlianza",[335,18,46,260,336,46],"7067140","losvedir",[338,18,46,260,339,46],"7068160","danso",[133,41,51,254,134,46],[127,41,46,229,11,18],[343,41,18,344,345,46],"7067808","project\u002Fcritical\u002Fprivacy","chii",[347,41,51,348,349,46],"7068071","project\u002Fneutral\u002Fprivacy","the_french",[351,41,46,260,352,46],"7068263","garyjob",[354,41,46,260,355,46],"7068559","davedx",[357,41,51,254,314,46],"7069025",[359,41,51,360,361,46],"7070496","project\u002Fneutral\u002Fpricing","billmalarky",[130,41,46,229,11,18],[364,41,51,360,361,46],"7143294",[366,41,46,260,367,46],"7072033","automately",[114,51,51,369,115,46],"project\u002Fneutral\u002Fperformance",[371,51,46,372,11,18],"7066780","maker\u002Fneutral\u002Fidea",[374,51,51,348,375,46],"7066878","Kudos",[377,51,46,257,11,18],"7066938",[379,51,51,380,381,46],"7067626","maker\u002Fneutral\u002Fprivacy","thatthatis",[383,51,46,260,352,46],"7068277",[385,51,46,274,375,46],"7069156",[118,44,51,254,119,46],[388,44,46,372,11,18],"7068122",[390,44,51,326,264,46],"7074694",[91,38,38,241,92,46],[140,38,46,372,11,18],[394,38,51,326,395,46],"7068880","kyriakos",[397,38,51,326,398,46],"7066990","scotty79",[400,38,46,372,11,18],"7067184",[402,38,51,326,403,46],"7070900","walden42",[405,38,46,257,11,18],"7071038",[407,38,18,227,398,46],"7072524",[409,38,51,326,352,46],"7068290",[107,16,18,227,108,46],[412,16,46,229,11,18],"7067046",[414,16,51,254,108,46],"7067096",[147,16,46,254,11,18],[150,223,38,417,151,46],"project\u002Fpositive\u002Fidea",[419,223,46,372,11,18],"7068075",[196,43,18,421,197,46],"project\u002Fcritical\u002Fidea",[423,43,46,274,424,46],"7067643","fizx",[426,43,46,427,428,46],"7068261","other\u002Fpositive\u002Fnone","deskglass",[430,431,38,432,433,46],"7066572",9,"project\u002Fpositive\u002Fdesign","sync",[189,435,51,326,190,46],10,[437,435,46,260,438,46],"7066969","Maxious",[440,435,46,274,441,46],"7067162","handelaar",[443,435,46,444,438,46],"7067196","other\u002Fneutral\u002Fpricing",[446,435,46,447,448,46],"7067379","commenter\u002Fneutral\u002Fpricing","yahelc",[450,435,46,274,451,46],"7067182","AznHisoka",[453,454,38,417,455,46],"7066750",11,"trey_swann",[457,33,38,417,458,46],"7067100","pknight",[460,33,46,372,11,18],"7067129",[462,463,18,227,464,46],"7066769",13,"fnordfnordfnord",[466,463,46,229,11,18],"7066791",[468,463,51,254,464,46],"7066829",[470,463,46,274,471,46],"7066943","nobodysfool",[473,463,46,274,474,46],"7067041","Dylan16807",[476,463,46,274,464,46],"7067057",[478,463,46,274,352,46],"7068282",[480,463,46,274,464,46],"7069168",[59,39,51,326,60,46],[483,39,51,254,484,46],"7068602","littledot5566",[486,39,46,229,11,18],"7070862",[488,39,51,254,489,46],"7070244","rpedela",[491,492,18,227,493,46],"7068463",15,"IbJacked",[495,492,46,229,11,18],"7068486",[497,498,38,432,499,46],"7067908",16,"eth",[501,502,18,227,503,46],"7067451",17,"alternize",[505,502,46,229,11,18],"7067628",[507,508,18,509,510,46],"7066838",18,"project\u002Fcritical\u002Fdesign","aqme28",[512,508,18,509,513,46],"7067108","misuba",[515,508,46,295,516,46],"7067275","roryokane",[518,508,46,295,519,46],"7067632","hobs",[521,508,46,295,522,46],"7067633","prolways",[524,508,46,295,525,46],"7067991","GBond",[527,508,46,295,375,46],"7069443",[529,508,46,295,530,46],"7067276","rfnslyr",[532,533,18,227,534,46],"7068991",19,"lips",[536,537,51,254,538,46],"7066827",20,"guptaneil",[540,537,46,229,11,18],"7066841",[542,537,18,227,538,46],"7066851",[544,537,46,229,11,18],"7067029",[546,547,38,432,352,46],"7068296",21,[95,549,38,417,96,46],22,[551,549,46,372,11,18],"7067836",[553,549,38,417,96,46],"7068363",[555,549,46,274,11,18],"7072237",[557,210,38,432,558,46],"7068919","chevreuil",[560,561,51,332,562,46],"7066698",24,"ph4",[564,561,46,332,11,18],"7066787",[566,34,51,332,567,46],"7067089","shekyboy",[569,34,46,332,11,18],"7067101",[571,34,51,332,567,46],"7067136",[573,34,46,574,11,18],"7067190","maker\u002Fpositive\u002Fnone",[576,577,38,432,578,46],"7066541",26,"ameister14",[580,581,38,582,583,46],"7066546",27,"project\u002Fpositive\u002Fnone","rafeed",[585,586,18,227,587,46],"7066885",28,"jfoster",[589,586,46,332,11,18],"7066905",[591,592,38,432,593,46],"7066608",29,"dmunoz",[595,592,46,574,11,18],"7066646",[597,598,38,417,599,46],"7070178",30,"critium",[601,598,46,257,11,18],"7072201",[157,603,38,582,158,46],31,[605,603,46,332,11,18],"7067019",[607,603,51,608,158,46],"7067065","project\u002Fneutral\u002Fdesign",[168,31,51,332,169,46],[611,31,46,332,11,18],"7066664",[613,31,51,257,381,46],"7067634",[615,31,46,332,11,18],"7067988",[617,31,51,257,618,46],"7068341","loceng",[620,31,46,274,381,46],"7069972",[622,31,46,257,11,18],"7072223",[624,625,38,417,626,46],"7067227",33,"blazespin",[628,629,18,227,489,46],"7067949",34,[631,629,46,229,11,18],"7067962",[633,634,46,635,636,46],"7066916",35,"other\u002Fneutral\u002Fidea","tlrobinson",[638,634,46,257,11,18],"7066961",[640,641,51,245,246,46],"7067622",36,[643,644,51,645,646,46],"7067903",37,"project\u002Fmixed\u002Fperformance","jval",[648,649,51,245,650,46],"7069347",38,"lucasnemeth",[652,653,38,432,654,46],"7066654",39,"twog",[656,657,51,245,658,46],"7066629",40,"jjcm",[161,657,46,257,11,18],[661,662,51,332,663,46],"7066576",41,"cbaleanu",[665,662,46,257,11,18],"7066694",[667,668,18,669,670,46],"7067721",42,"project\u002Fcritical\u002Fperformance","dikei",[672,668,46,673,11,18],"7067826","maker\u002Fneutral\u002Fperformance",[675,676,46,574,11,18],"7066547",43,[678,679,51,332,680,46],"7067163",44,"ewebbuddy",[682,679,46,257,11,18],"7067179",[684,685,51,332,686,46],"7069962",45,"shamsulbuddy",[688,689,51,690,691,46],"7067047",46,"project\u002Fmixed\u002Fpricing","catshirt",[76,689,46,360,11,18],[694,689,51,360,691,46],"7067119",[205,696,38,432,206,46],47,[698,696,46,257,11,18],"7066594",[700,696,51,701,702,46],"7066647","project\u002Fneutral\u002Fdocs","lstamour",[704,705,38,582,706,46],"7066737",48,"tchadwick",[708,705,46,257,11,18],"7066803",[710,711,51,245,712,46],"7066621",49,"kenrikm",[714,711,46,229,11,18],"7066677",[716,717,51,326,718,46],"7069169",50,"paul1664",[720,32,51,332,721,46],"7068415","keyurfaldu",[723,724,18,227,725,46],"7066830",52,"cullenmacdonald",[727,724,46,257,11,18],"7066946",[729,730,38,432,731,46],"7066577",53,"rmason",[733,734,38,582,735,46],"7066874",54,"xux",[737,738,51,254,739,46],"7066668",55,"bluejellybean",[741,738,46,229,11,18],"7066690",[743,744,51,254,745,46],"7070950",56,"mhluongo",[747,744,46,257,11,18],"7072192",[68,749,51,750,69,46],57,"project\u002Fmixed\u002Fnone",[752,749,46,380,11,18],"7066727",[754,749,18,344,755,46],"7066807","hartard",[757,749,46,380,11,18],"7066903",[759,749,46,760,761,46],"7066960","commenter\u002Fneutral\u002Fprivacy","_delirium",[763,749,46,760,764,46],"7066831","eli",[766,767,46,260,768,46],"7070987",58,"cycnusx",[770,209,51,254,771,46],"7068566","diegolo",[773,209,46,229,11,18],"7068576",[775,776,18,777,778,46],"7068779",60,"project\u002Fcritical\u002Fpricing","PhilipA",[780,781,51,254,782,46],"7067123",61,"dmritard96",[784,785,51,348,381,46],"7067986",62,[787,785,46,380,11,18],"7068010",[789,790,51,360,791,46],"7069366",63,"timov",[793,794,38,241,795,46],"7081106",64,"toddwahnish",[797,798,51,245,799,46],"7066600",65,"iurisilvio",[801,798,46,274,802,46],"7066610","trip41",[804,805,51,245,806,46],"7067698",66,"bigd",[808,805,46,229,11,18],"7068002",[810,811,51,245,812,46],"7068469",67,"yummybear",[814,811,46,229,11,18],"7068531",[816,817,18,227,395,46],"7068859",68,[819,817,46,229,11,18],"7070928",[821,822,38,417,823,46],"7066588",69,"szidev",[825,826,18,421,827,46],"7067049",70,"wprl",[829,830,38,241,831,46],"7069657",71,"narzero",[833,834,51,254,835,46],"7068072",72,"joshmlewis",[837,834,46,229,11,18],"7068083",[839,840,38,432,841,46],"7067157",73,"NicoJuicy",[843,840,46,844,11,18],"7067244","maker\u002Fneutral\u002Fdesign",[846,847,51,326,848,46],"7066986",74,"dome82",[850,847,46,372,11,18],"7067008",[852,853,46,260,854,46],"7067690",75,"abvdasker",[856,857,51,326,858,46],"7067985",76,"mswen",[860,861,38,432,862,46],"7067621",77,"aaronsnoswell",[864,861,46,274,865,46],"7067964","dclara",[867,868,38,869,870,46],"7066503",78,"project\u002Fpositive\u002Fperformance","taternuts",[872,873,18,421,874,46],"7068824",79,"byteface",[876,877,51,326,878,46],"7067319",80,"harryovers",[880,881,38,432,882,46],"7067217",81,"iamkoby",[884,885,38,582,886,46],"7067909",82,"rismay",[888,889,38,582,890,46],"7066523",83,"tonystark",[892,893,18,421,894,46],"7066890",84,"nnnn",[896,897,46,260,898,46],"7067668",85,"pyed",[],[901,906,910,914,917,921],{"id":902,"name":903,"points":904,"category":905},"7059919","Funded.io",167,"finance_crypto",{"id":907,"name":908,"points":885,"category":909},"7049960","Cabot","infrastructure",{"id":911,"name":912,"points":794,"category":913},"7030983","Sense","data",{"id":915,"name":916,"points":790,"category":20},"7066142","Houdini",{"id":918,"name":919,"points":785,"category":920},"7043295","My Internet life","productivity",{"id":922,"name":923,"points":781,"category":20},"7039509","Taplytics",false,"v2",{"id":927,"name":5,"line":6,"ai":924,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":928,"top":18,"kw":929,"maker":11,"domain":10,"tier":19,"attempts":46,"strip":930,"url":9,"mix":931,"praise":88,"objection":100,"sub":21},7066479,0.1,[24,25,26,27],"3230023233415351321145453333344423333444",[31,32,33,34],1793990902007]