[{"data":1,"prerenderedAt":495},["ShallowReactive",2],{"project-11587468":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":46,"post_text":48,"what":49,"said":81,"maker_in_thread":225,"shape":237,"grid":241,"attempts":456,"same_week":457,"maker_shows":33,"maker_hidden":487,"reader_version":488,"card":489},"11587468","Jam API","Jam API, turn any site into a JSON api using CSS selectors","title","Show HN: Jam API, turn any site into a JSON api using CSS selectors","http:\u002F\u002Fwww.jamapi.xyz\u002F","jamapi.xyz","gavino",1461835828,2016,300,82,65,2264,3,"legend","dev_tools","web scraping api","web_app",[24,25,26,27],"web scraping","css selectors","json api","html to json","debated",0.043,{"positive":31,"neutral":32,"mixed":33,"critical":31},12,17,2,43,{"pricing":36,"idea":39,"execution":41,"design":42,"docs":43,"performance":44,"privacy":45},{"positive":37,"critical":38},0,1,{"positive":40,"critical":18},6,{"positive":37,"critical":40},{"positive":33,"critical":37},{"positive":37,"critical":37},{"positive":37,"critical":38},{"positive":37,"critical":38},[47,37,37,18,47,37,18,38,38,18,18,37,37,37,47,18,47,37,18,18,18,18,37,38,18,38,37,47,37,37,37,18,38,47,47,38,38,18,37,38],5,null,{"problem":50,"audience":48,"pricing":62,"motivation":48,"time_to_build":48,"stack":70,"openness":78,"stage":79,"team_size":80},{"value":51,"source":52,"evidence":53},"Extracting structured data from websites without writing custom scraping code","inferred",[54,58],{"id":55,"by":56,"text":57},"11588609","yeukhon","> \"How will it make money\" I was trying to implement a Firefox add-on which navigates the web based on speech for my senior project (the theme was assistive technology). The…",{"id":59,"by":60,"text":61},"11588841","g00gler",">However, as cool as this is, it's just another way to \"scrape\" Isn't that the point? The demo seems like it'd be a lot easier, less verbose, and probably less brittle, than…",{"value":63,"source":64,"evidence":65},"Open source; no monetization plan stated","commenter",[66],{"id":67,"by":68,"text":69},"11591275","awinder","It's an open source project so, maybe it's not meant to make money?",[71],{"value":72,"source":64,"evidence":73},"Node.js",[74],{"id":75,"by":76,"text":77},"11587734","WA","Well no, because my browser isn't doing the request. The underlying Node app (the Jam API) does it. I found it: The API responds with an HTTP 500 error if you use CSS selectors…","open_source","launched","unknown",{"praise":82,"objections":109,"requests":148,"questions":177,"alternatives":190},[83,98],{"value":84,"source":64,"evidence":85},"Good idea and useful for scraping without custom code",[86,90,94],{"id":87,"by":88,"text":89},"11587649","adriancooney","This is a fantastic idea and I'm really surprised nothing like this has existed before, it seems like such a no-brainer. Great work.",{"id":91,"by":92,"text":93},"11592679","paulmd","I've been thinking about writing some website-to-JSON scrapers myself and this basically solves that problem (since I would have been going after CSS selectors or xpath anyway…",{"id":95,"by":96,"text":97},"11588169","chriswarbo","Very nice idea. Although scraping should always be a last resort, I could imagine using this for semi-serious purposes, i.e. when I care enough about the output, will be doing…",{"value":99,"source":64,"evidence":100},"Developer-friendly interface and low barrier to start",[101,105],{"id":102,"by":103,"text":104},"11590280","phsource","We launched WrapAPI (https:\u002F\u002Fwrapapi.com\u002F) a few weeks ago with the same functionality, but a bit more complex and powerful process to get set up. You can not only specify CSS…",{"id":106,"by":107,"text":108},"11592579","OJFord","Yes, yes, yes! I'm using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I've come up with is to…",[110,121,129,140],{"value":111,"source":64,"evidence":112},"Scrapers are fragile and break when page markup changes",[113,117],{"id":114,"by":115,"text":116},"11587994","geuis","I built something almost identical in 2011. It really doesn't have as much utility in practice as you think initially. CSS selectors are an interesting idea for extracting data…",{"id":118,"by":119,"text":120},"11587882","mappy","> I'm really surprised nothing like this has existed before But how would you monetize it? Unlike an RSS feed, you really don't know how the JSON response would be used, so you…",{"value":122,"source":64,"evidence":123},"Hard to monetize",[124,125],{"id":118,"by":119,"text":120},{"id":126,"by":127,"text":128},"11588360","bryanrasmussen","I think the wanting to know how it will be monetized \u002Fsupport itself is a reasonable question. If there isnt an answer you know not to build something using it, expecting it to…",{"value":130,"source":64,"evidence":131},"Sites block scraping via robots.txt and rate limits",[132,136],{"id":133,"by":134,"text":135},"11587815","fizx","https:\u002F\u002Fgithub.com\u002Ffizx\u002Fparsley\u002Fwiki looks pretty similar. Running this sort of thing as a service\u002Fapi never panned out for us because you are almost universally robots.txt…",{"id":137,"by":138,"text":139},"11588057","LoSboccacc","done it as well. at time specialized to organize web comics (it was way before google reader times). real issue is that popularity will get you blocked fairly quickly. see also:…",{"value":141,"source":64,"evidence":142},"Does not handle JavaScript-rendered pages",[143,147],{"id":144,"by":145,"text":146},"11587837","daw___","Wonderful idea. What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?",{"id":114,"by":115,"text":116},[149,159,163,170],{"value":150,"source":64,"evidence":151},"Support for HTTPS requests",[152,156],{"id":153,"by":154,"text":155},"11588184","nsgi","If it's going to be used for serious purposes it really needs HTTPS support, as most APIs do these days.",{"id":157,"by":76,"text":158},"11587634","HTTPS results in 500 Internal Server Error. Edit: Well no, it's only some sites. E. g. https:\u002F\u002Fmedium.com",{"value":160,"source":64,"evidence":161},"Support for JavaScript-rendered DOM",[162],{"id":144,"by":145,"text":146},{"value":164,"source":64,"evidence":165},"Form authentication with cookies",[166],{"id":167,"by":168,"text":169},"11587676","NicoJuicy","I'm actually wondering if it would be possible to add forms authentication to this? Eg. Post with some sort of css selecters and then a \"cookie memory\".",{"value":171,"source":64,"evidence":172},"Publish to npm",[173],{"id":174,"by":175,"text":176},"11588228","splatcollision","Nice work, thanks for adding the Github link. I can think of lots of immediate use for this. Consider publishing on NPM?",[178,186],{"value":179,"source":64,"evidence":180},"How will it make money?",[181,182],{"id":118,"by":119,"text":120},{"id":183,"by":184,"text":185},"11588088","lost_my_pwd","A little word of warning\u002Fencouragement. I did something similar a long time ago (JSONDuit), which got posted to HN by someone else. You will probably run into a healthy mix of…",{"value":187,"source":64,"evidence":188},"Will it render JavaScript before scraping?",[189],{"id":144,"by":145,"text":146},[191,202,210,214,218],{"value":192,"source":64,"evidence":193},"Kimono Labs",[194,198],{"id":195,"by":196,"text":197},"11588555","moeamaya","There was a YC company a few years back that got acquired by Palantir in February that did something very similar. https:\u002F\u002Fwww.kimonolabs.com\u002F",{"id":199,"by":200,"text":201},"11588357","karlcoelho1","If anyone remembers, they was a YC company that did exactly this. It was called Kimono Labs. I think it failed and just got acquired a year ago. \"Jam API\" will probably do way…",{"value":203,"source":64,"evidence":204},"Apifier",[205,209],{"id":206,"by":207,"text":208},"11588123","jancurn","Well, https:\u002F\u002Fwww.apifier.com does essentially the same thing, plus it supports JavaScript, can crawl through the whole website etc. Disclaimer: I'm a cofounder there",{"id":106,"by":107,"text":108},{"value":211,"source":64,"evidence":212},"WrapAPI",[213],{"id":102,"by":103,"text":104},{"value":215,"source":64,"evidence":216},"Parsley",[217],{"id":133,"by":134,"text":135},{"value":219,"source":64,"evidence":220},"xidel",[221],{"id":222,"by":223,"text":224},"11587670","xaduha","Using a 3rd party site to query HTML (which you should be able to do yourself, plenty of tools for that) isn't a fantastic idea. This one, for example…",{"comments":226,"answered":18,"key":227},4,[228,231,234],{"id":229,"text":230},"11587568","Ah, yep, you're right, forgot to change the URL. Updated now. Thanks for letting me know.",{"id":232,"text":233},"11587624","Regarding the \"index\" key, there are some JSON parsers for languages like Swift that will rearrange your JSON. By adding the index key, you'll still be able to sort after…",{"id":235,"text":236},"11587777","I had been trying to figure out what would be causing this issue, thanks for pointing it out, I've pushed a fix real quick that will respond whether JSON is invalid or a CSS…",{"commenters":238,"max_depth":40,"maker_replies":226,"first_comment_mins":239,"lifespan_hours":240},63,20,31,[242,244,247,250,252,255,258,260,261,263,266,269,272,273,275,278,279,281,283,284,286,288,290,293,295,298,301,304,305,307,310,313,317,319,322,323,326,328,331,333,334,337,340,344,346,350,354,357,359,363,367,370,371,373,376,379,384,386,390,393,395,398,400,403,405,407,409,412,414,415,417,421,425,427,430,434,436,439,441,445,449,452],[183,37,47,243,184,37],"maker\u002Fpositive\u002Fnone",[245,37,37,246,223,37],"11588324","commenter\u002Fneutral\u002Fnone",[248,37,37,246,249,37],"11588350","iamben",[251,37,37,246,223,37],"11588410",[253,37,37,246,254,37],"11588503","barranger",[256,37,37,246,257,37],"11589303","ryanlol",[126,37,18,259,127,37],"project\u002Fneutral\u002Fpricing",[67,37,18,259,68,37],[55,37,47,262,56,37],"project\u002Fpositive\u002Fidea",[264,37,47,262,265,37],"11589645","dpweb",[267,37,37,246,268,37],"11589851","enraged_camel",[270,37,37,246,271,37],"11596761","metasean",[87,38,47,262,88,37],[222,38,38,274,223,37],"project\u002Fcritical\u002Fidea",[276,38,37,246,277,37],"11587674","martinvol",[133,38,38,274,134,37],[118,38,38,280,119,37],"project\u002Fcritical\u002Fpricing",[282,38,18,259,154,37],"11588178",[59,38,47,262,60,37],[114,38,38,285,115,37],"project\u002Fcritical\u002Fexecution",[137,38,18,287,138,37],"project\u002Fneutral\u002Fnone",[289,38,18,287,168,37],"11589069",[291,38,37,246,292,37],"11595021","mickael-kerjean",[206,38,37,294,207,37],"other\u002Fneutral\u002Fnone",[296,38,37,294,297,37],"11588469","fiatjaf",[299,38,37,294,300,37],"11589503","novaleaf",[302,38,37,294,303,37],"11588154","lorenzhs",[195,38,37,294,196,37],[102,38,47,306,103,37],"project\u002Fpositive\u002Fdesign",[308,33,37,294,309,37],"11588293","ptwt",[311,33,37,294,312,37],"11592500","finnn",[314,18,18,315,316,37],"11587549","project\u002Fneutral\u002Fdocs","jstanley",[229,18,37,318,11,38],"maker\u002Fneutral\u002Fnone",[320,18,47,321,316,37],"11587590","project\u002Fpositive\u002Fnone",[232,18,37,318,11,38],[324,18,37,246,325,37],"11587698","JelteF",[327,18,37,246,96,37],"11588209",[329,18,18,315,330,37],"11587629","pmontra",[95,226,18,332,96,37],"project\u002Fmixed\u002Fidea",[167,47,18,287,168,37],[335,47,18,336,107,37],"11592613","project\u002Fneutral\u002Fidea",[338,40,47,321,339,37],"11587628","fryiee",[341,342,38,285,343,37],"11590971",7,"denishaskin",[106,345,47,306,107,37],8,[347,348,38,285,349,37],"11587873",9,"staticelf",[351,352,37,294,353,37],"11590985",10,"soheil",[355,352,37,246,356,37],"11591667","smadge",[157,358,38,285,76,37],11,[360,358,18,361,362,37],"11587659","project\u002Fneutral\u002Fexecution","danielsamuels",[364,358,18,365,366,37],"11587667","project\u002Fmixed\u002Fexecution","learned",[368,358,18,361,369,37],"11587681","diggan",[75,358,38,285,76,37],[235,358,37,372,11,38],"maker\u002Fneutral\u002Fexecution",[374,31,37,294,375,37],"11588603","MetaMetaApplyHN",[377,31,37,294,378,37],"11588877","Mahn",[380,381,18,382,383,37],"11589622",13,"project\u002Fneutral\u002Fdesign","loisaidasam",[174,385,47,321,175,37],14,[387,388,37,294,389,37],"11587900",15,"bartkappenburg",[391,388,37,294,392,37],"11587935","dkopi",[394,388,37,294,389,37],"11587942",[396,388,37,246,397,37],"11588229","toupeira",[399,388,37,294,134,37],"11587938",[401,388,37,294,402,37],"11588263","hegivor",[144,404,18,361,145,37],16,[406,404,37,372,11,38],"11587849",[408,404,18,361,145,37],"11587863",[410,404,38,285,411,37],"11588395","etatoby",[413,404,37,246,107,37],"11592641",[199,32,47,262,200,37],[91,416,47,262,92,37],18,[418,419,18,287,420,37],"11591976",19,"dimino",[422,419,38,423,424,37],"11592322","project\u002Fcritical\u002Fperformance","brbsix",[426,239,37,294,356,37],"11591545",[153,428,38,429,154,37],21,"project\u002Fcritical\u002Fprivacy",[431,432,18,382,433,37],"11587686",22,"thomasahle",[435,432,18,382,134,37],"11587881",[437,432,37,246,438,37],"11589014","jerf",[440,432,37,246,433,37],"11589041",[442,443,37,294,444,37],"11588398",23,"uberneo",[446,447,38,274,448,37],"11592104",24,"amelius",[450,447,37,451,356,37],"11593202","other\u002Fneutral\u002Fidea",[453,454,37,294,455,37],"11593276",25,"joelbondurant",[],[458,463,468,473,478,483],{"id":459,"name":460,"points":461,"category":462},"11603604","Alloy",148,"productivity",{"id":464,"name":465,"points":466,"category":467},"11599714","TeachCraft",129,"learning_science",{"id":469,"name":470,"points":471,"category":472},"11618550","GAM",115,"languages",{"id":474,"name":475,"points":476,"category":477},"11610832","Panther",108,"consumer_social",{"id":479,"name":480,"points":481,"category":482},"11589340","TinyRave",68,"creative",{"id":484,"name":485,"points":486,"category":462},"11617791","Gratefulness.io",56,false,"v2",{"id":490,"name":5,"line":6,"ai":487,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":491,"top":18,"kw":492,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":493,"url":9,"mix":494,"praise":84,"objection":111,"sub":21},11587468,0.04,[24,25,26,27],"5003503113300053503333013105000315511301",[31,32,33,31],1793990942158]