[{"data":1,"prerenderedAt":323},["ShallowReactive",2],{"project-15189099":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":43,"post_text":45,"what":46,"said":104,"maker_in_thread":208,"shape":216,"grid":220,"attempts":286,"same_week":287,"maker_shows":252,"maker_hidden":315,"reader_version":316,"card":317},"15189099","Page.REST","An API to fetch details from a web page as JSON","title","Show HN: Page.REST – An API to fetch details from a web page as JSON","http:\u002F\u002Fwww.page.rest","page.rest","laktek",1504757709,2017,114,32,383,2242,18,"notable","web_saas","web scraping api","api",[21,24,25,26,27],"open graph","oembed","json api","html parsing","curious",-0.241,{"positive":31,"neutral":32,"mixed":33,"critical":34},2,10,0,5,17,{"privacy":37,"execution":38,"performance":40,"idea":41,"pricing":42},{"positive":33,"critical":33},{"positive":33,"critical":39},1,{"positive":39,"critical":33},{"positive":33,"critical":31},{"positive":33,"critical":31},[33,44,44,44,34,33,33,44,44,39,39,33,44,39,33,44,44,39,33,34,33,33,33,44,33,39,33,33,33,44,33,33],3,null,{"problem":47,"audience":61,"pricing":71,"motivation":83,"time_to_build":45,"stack":90,"openness":101,"stage":102,"team_size":103},{"value":48,"source":49,"evidence":50},"Fetching structured details (selectors, OpenGraph, oembed) from a web page as JSON without writing a parser","inferred",[51,54,58],{"id":52,"by":11,"text":53},"15189804","It depends on the use case. Say you want to get all images on a page you'd just use `img`. If you are going to find JSON-LD blocks defined in a set of pages, you can use -…",{"id":55,"by":56,"text":57},"15189525","ganessh","If I have to know the elements' selectors, why should I prefer this service over using a HTML parser?",{"id":59,"by":11,"text":60},"15196260","Added support for OpenGraph extraction https:\u002F\u002Fwww.page.rest\u002F#open-graph",{"value":62,"source":49,"evidence":63},"Developers who need page metadata or content as JSON",[64,67],{"id":65,"by":11,"text":66},"15189836","It's actually a utility I built for my own use (I bootstrap Pragma full-time). I thought there might be an interest among other developers for a similar utility. That's why I…",{"id":68,"by":69,"text":70},"15189863","icebraining","Consider offering responses other formats, possibly protobuf. Then you can sell it as a solution for limited IoT devices where a full HTML parser is too much. If your service…",{"value":72,"source":73,"evidence":74},"Paid access token at $5, charged to reduce abuse; maker says the code is open source","maker",[75,78,79],{"id":76,"by":11,"text":77},"15189814","The idea of charging $5 was a stop gap measure to prevent abuse. There are limits set for access token too. I actually rushed to ship before putting out a TOS which mentions…",{"id":65,"by":11,"text":66},{"id":80,"by":81,"text":82},"15190291","xytop","It's my own server and I'm the developer of the service ;) I pay $3\u002Fmo for hosting and $10\u002Fy for domain so it doesnt cost me much. Regarding reliability: all the code is…",{"value":84,"source":73,"evidence":85},"Maker built it as a utility for own use and hoped other developers would want it; intended as a profitable service",[86,87],{"id":65,"by":11,"text":66},{"id":88,"by":81,"text":89},"15192328","Idea was to make a profitable service out of this. But I'm bad interpreneur :) I wanted to make an analog of http:\u002F\u002Fembed.ly\u002Fcode but lost interest in the middle. I can code…",[91],{"value":92,"source":93,"evidence":94},"Go","commenter",[95,98],{"id":96,"by":81,"text":97},"15191563","Usual traffic is 0-1 req\u002Fsec. Even now it is about 2-3 req\u002Fsec. It is running for more than 2 years now, never failed :) Golang is pretty stable. Also the server is behind…",{"id":99,"by":81,"text":100},"15194454","It's not DO. It's scaleway C1 arm server, even cheaper. Golang produces native code with very good runtime with green threads which makes it very easy to do async programming and…","open_source","launched","solo",{"praise":105,"objections":116,"requests":143,"questions":162,"alternatives":178},[106],{"value":107,"source":93,"evidence":108},"Cheap free alternative and useful idea",[109,112],{"id":110,"by":81,"text":111},"15190164","For absolutely 0-cost you can use http:\u002F\u002Fproc.link which will return oembed info for ANY url. General example: http:\u002F\u002Fapi.proc.link\u002Foembed?url=http%3A%2F%2Fpage.rest Youtube:…",{"id":113,"by":114,"text":115},"15190332","keyle","I did something similar a loooong time ago. Granted not as sexy. https:\u002F\u002Fgithub.com\u002Fkeyle\u002Fjson-anything",[117,125,132],{"value":118,"source":93,"evidence":119},"Users must know element selectors, reducing value over an HTML parser",[120,121],{"id":55,"by":56,"text":57},{"id":122,"by":123,"text":124},"15189642","AznHisoka","Yeah, It kind of defeats the point\u002Fvalue prop of an API.",{"value":126,"source":93,"evidence":127},"Abuse risk without clear API limits",[128],{"id":129,"by":130,"text":131},"15189789","cinooo","Someone could really abuse this service, I don't see any mention of API limits.",{"value":133,"source":93,"evidence":134},"$5 price and payment flow seem off-putting",[135,139],{"id":136,"by":137,"text":138},"15190050","kowdermeister","I would make the 5$ price and token validity much larger, like \"4rem\" or something, I was looking at the CC input field and thinking \"seriously? how much will you charge?\"",{"id":140,"by":141,"text":142},"15189746","geetfun","Looks interesting. I wonder what kind of market this app might serve. For larger apps, I would worry about support. 5 dollars per year tells me that the developer is doing this…",[144,151,155],{"value":145,"source":93,"evidence":146},"Support for scraping sites with anti-bot protection and high request rates",[147],{"id":148,"by":149,"text":150},"15194494","throwaway12s34","How do you handle sites that have scraper prevention? Such as captcha, IP throttling, etc. I'd pay $100-$500 per-month for a service that could reliably scrape some particularly…",{"value":152,"source":93,"evidence":153},"Support for other response formats such as protobuf",[154],{"id":68,"by":69,"text":70},{"value":156,"source":93,"evidence":157},"Await for JS frameworks to render before scraping",[158],{"id":159,"by":160,"text":161},"15191716","andrevv_6","Is there a await for JS Frontends to initialise before the scrape via a selector (i.e. Angular\u002FEmber etc)?",[163,174],{"value":164,"source":93,"evidence":165},"How the server cost is covered and whether it is reliable",[166,170],{"id":167,"by":168,"text":169},"15190259","afshinmeh","Very neat. But the question is, how they pay the cost of the server? I mean, do you think a web service without any API keys, subscriptions, etc. is reliable?",{"id":171,"by":172,"text":173},"15191512","have_faith","What levels of traffic does your server see on a normal day? has it ever gone down from a spike?",{"value":175,"source":93,"evidence":176},"Whether it handles scraper prevention like captchas",[177],{"id":148,"by":149,"text":150},[179,183,190,197,201],{"value":180,"source":93,"evidence":181},"proc.link oembed API",[182],{"id":110,"by":81,"text":111},{"value":184,"source":93,"evidence":185},"Yahoo YQL",[186],{"id":187,"by":188,"text":189},"15189741","squiggy22","Not unlike YQL - https:\u002F\u002Fdeveloper.yahoo.com\u002Fyql\u002F",{"value":191,"source":93,"evidence":192},"wrapapi.com",[193],{"id":194,"by":195,"text":196},"15190728","Luyanda","Uhm this is interesting. Reminds me of: https:\u002F\u002Fwrapapi.com",{"value":198,"source":93,"evidence":199},"embed.ly",[200],{"id":88,"by":81,"text":89},{"value":202,"source":93,"evidence":203},"link.fish",[204],{"id":205,"by":206,"text":207},"15194519","janober","If you are interested the same is possible with https:\u002F\u002Flink.fish\u002Fhn-early-access\u002F which has a Zapier integration. It allows to send an URL and it then returns its data. However,…",{"comments":209,"answered":44,"key":210},7,[211,212,215],{"id":52,"text":53},{"id":213,"text":214},"15190064","I bolded it for now. Need to rethink the presentation.",{"id":65,"text":66},{"commenters":217,"max_depth":209,"maker_replies":209,"first_comment_mins":218,"lifespan_hours":219},19,129,164,[221,223,225,227,229,231,234,235,238,240,242,243,245,247,249,250,251,253,254,255,258,261,264,266,268,271,273,274,278,279,281,283],[110,33,33,222,81,33],"other\u002Fneutral\u002Fnone",[167,33,44,224,168,33],"project\u002Fneutral\u002Fprivacy",[80,33,44,226,81,33],"maker\u002Fneutral\u002Fexecution",[171,33,44,228,172,33],"project\u002Fneutral\u002Fperformance",[96,33,34,230,81,33],"project\u002Fpositive\u002Fperformance",[232,33,33,222,233,33],"15193177","isgolanggood",[99,33,33,222,81,33],[236,33,44,237,168,33],"15192283","maker\u002Fneutral\u002Fidea",[88,33,44,239,81,33],"maker\u002Fneutral\u002Fpricing",[55,39,39,241,56,33],"project\u002Fcritical\u002Fidea",[122,39,39,241,123,33],[52,39,33,244,11,39],"maker\u002Fneutral\u002Fdesign",[68,39,44,246,69,33],"project\u002Fneutral\u002Fidea",[136,31,39,248,137,33],"project\u002Fcritical\u002Fpricing",[213,31,33,244,11,39],[187,44,44,246,188,33],[194,252,44,246,195,33],4,[140,34,39,248,141,33],[65,34,33,239,11,39],[256,34,34,257,141,33],"15194580","maker\u002Fpositive\u002Fnone",[113,259,33,260,114,33],6,"other\u002Fneutral\u002Fidea",[262,209,33,263,11,39],"15243825","project\u002Fneutral\u002Fnone",[59,265,33,263,11,39],8,[159,267,44,228,160,33],9,[269,267,33,270,11,39],"15196341","maker\u002Fneutral\u002Fperformance",[129,32,39,272,130,33],"project\u002Fcritical\u002Fexecution",[76,32,33,239,11,39],[275,276,33,222,277,33],"15191459",11,"chenster",[205,276,33,222,206,33],[148,280,44,228,149,33],12,[282,280,33,222,206,33],"15195832",[284,280,33,222,285,33],"15196296","hyder_m29",[],[288,293,297,302,307,311],{"id":289,"name":290,"points":291,"category":292},"15201152","Coder's Coloring Book",523,"learning_science",{"id":294,"name":295,"points":296,"category":20},"15207151","unarbitrate",431,{"id":298,"name":299,"points":300,"category":301},"15216650","Gridless Kits",373,"hardware",{"id":303,"name":304,"points":305,"category":306},"15207117","Hyperapp",242,"languages",{"id":308,"name":309,"points":310,"category":20},"15200100","Borderline.biz",187,{"id":312,"name":313,"points":314,"category":292},"15206982","Jupyter4kids",147,false,"v2",{"id":318,"name":5,"line":6,"ai":315,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":319,"top":18,"kw":320,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":321,"url":9,"mix":322,"praise":107,"objection":118,"sub":21},15189099,-0.24,[21,24,25,26,27],"03335003311031033105000301000300",[31,32,33,34],1793990963647]