[{"data":1,"prerenderedAt":456},["ShallowReactive",2],{"project-15611605":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":46,"post_text":49,"what":50,"said":86,"maker_in_thread":199,"shape":211,"grid":215,"attempts":418,"same_week":419,"maker_shows":47,"maker_hidden":448,"reader_version":449,"card":450},"15611605","Link.fish","API to extract data from websites as JSON","title","Show HN: Link.fish – API to extract data from websites as JSON","https:\u002F\u002Flink.fish\u002Fapi","link.fish","linkfish",1509638743,2017,201,77,160,2242,8,"notable","web_saas","web data extraction API","api",[24,25,26,27],"web scraping","data extraction","schema.org","website to json","curious",-0.141,{"positive":31,"neutral":32,"mixed":33,"critical":34},6,34,2,11,53,{"execution":37,"idea":40,"pricing":41,"performance":43,"docs":44,"design":45},{"positive":38,"critical":39},1,7,{"positive":33,"critical":38},{"positive":38,"critical":42},0,{"positive":42,"critical":33},{"positive":42,"critical":38},{"positive":38,"critical":42},[47,42,47,47,48,47,38,48,38,47,48,47,42,42,48,47,38,47,47,47,38,38,47,38,38,47,47,47,38,47,38,47,47,47,47,48,38,47,48,38],3,5,null,{"problem":51,"audience":59,"pricing":69,"motivation":49,"time_to_build":49,"stack":83,"openness":84,"stage":85,"team_size":84},{"value":52,"source":53,"evidence":54},"Extracting structured data like prices, geo coordinates and social profiles from websites","commenter",[55],{"id":56,"by":57,"text":58},"15611943","janober","No is not just bad luck. Currently did not concentrate so much on \"text pages\" like blogs or articles yet. Mainly on pages which contain more data like prices, geo coordinates,…",{"value":60,"source":61,"evidence":62},"Developers and businesses needing structured web data","inferred",[63,66],{"id":64,"by":11,"text":65},"15612413","Hard to say and compare in what regard exactly? Price wise, much cheaper. Data wise, it depends. Mainly on in what kind of data you are interested in. Will probably return better…",{"id":67,"by":11,"text":68},"15612511","Yes is similar to it in the regard that with both tools data from websites can be extracted.",{"value":70,"source":53,"evidence":71},"Credit-based tiers; full-browser rendering costs more credits; cheaper than Diffbot",[72,76,80],{"id":73,"by":74,"text":75},"15613143","RussianCow","I think the credit concept was created solely for this reason (from the page): \"There is only one exception if the page should be rendered with a full browser (not headless). In…",{"id":77,"by":78,"text":79},"15612424","raresp","At a first look it's much cheaper than Diffbot :)",{"id":81,"by":57,"text":82},"15612003","The reason is that we also offer that pages can be rendered with a full browser (to execute all JavaScript and make screenshots) which takes much more resources.",[],"unknown","launched",{"praise":87,"objections":107,"requests":139,"questions":160,"alternatives":175},[88,96],{"value":89,"source":53,"evidence":90},"Cheaper than alternatives like Diffbot",[91,92],{"id":77,"by":78,"text":79},{"id":93,"by":94,"text":95},"15615452","dwynings","Disclaimer: I work at Diffbot Major differences I can see (OP feel free to correct if I'm wrong): Link.fish * doesn't provide a web crawler * relies heavily on microdata,…",{"value":97,"source":53,"evidence":98},"Works well on many real estate and product sites",[99,103],{"id":100,"by":101,"text":102},"15652982","etewiah","I've been checking out link.fish for a while now - awesome product! My interest is in scraping real estate websites and it seems to do quite a good job with many of the sites…",{"id":104,"by":105,"text":106},"15612304","wiradikusuma","I tried a link to some product in some ecommerce…",[108,123,131],{"value":109,"source":53,"evidence":110},"Fails on many sites and returns 500 errors",[111,115,119],{"id":112,"by":113,"text":114},"15612783","callmeed","Having done a ton of scraping in the past (especially around ecommerce and products), this looks pretty cool. A couple comments in general: 1. Personally I think its better to be…",{"id":116,"by":117,"text":118},"15615352","ojanik","https:\u002F\u002Fen.wikipedia.org\u002Fwiki\u002FThe_Expanse_(TV_series) Error: 500 Message: { \"status\": 500, \"message\": \"Internal Server Error\" }",{"id":120,"by":121,"text":122},"15611798","lawl","I tried a random reddit thread. Did not fetch comments, only information about the submission. Then I tried it with HN. Same. Then I tried it with a github issue. Same. Then I…",{"value":124,"source":53,"evidence":125},"Relies on microdata and custom parsers, so it works only on specific sites",[126,130],{"id":127,"by":128,"text":129},"15619452","nailer","It works, but only on that site - a similar site (http:\u002F\u002Fstudio68london.net\u002Fwork\u002Ftimetable\u002F fails). I want to be able to point something at an arbitrary URL and have it extract…",{"id":93,"by":94,"text":95},{"value":132,"source":53,"evidence":133},"Credits abstraction is confusing",[134,135],{"id":112,"by":113,"text":114},{"id":136,"by":137,"text":138},"15611917","mmahemoff","Looks good, but why complicate the pricing with \"credits\" when 1 credit==1 request. The tiers already reference parallel \"requests\", so you could just say N requests instead of N…",[140,145,153],{"value":141,"source":53,"evidence":142},"Focus on one data type such as products or real estate",[143,144],{"id":112,"by":113,"text":114},{"id":100,"by":101,"text":102},{"value":146,"source":53,"evidence":147},"Browser extension to avoid blocking",[148,152],{"id":149,"by":150,"text":151},"15620053","sharpshadow","One solid way would be to offer the service as browser addon. With that you can avoid any blockage, because the user itself is doing it.",{"id":100,"by":101,"text":102},{"value":154,"source":53,"evidence":155},"Recursive contact information scraping",[156],{"id":157,"by":158,"text":159},"15612378","danielvinson","The feature I'd be looking for in this is to be able to recursively scrape for contact information (email, phone number, etc.)... that doesn't seem possible with this?",[161,168],{"value":162,"source":53,"evidence":163},"How does it compare to Diffbot",[164],{"id":165,"by":166,"text":167},"15612340","mericsson","Well done! How does this compare to diffbot's website extraction? https:\u002F\u002Fwww.diffbot.com\u002F",{"value":169,"source":53,"evidence":170},"What do priority tiers mean in practice",[171],{"id":172,"by":173,"text":174},"15611877","sjs382","In the pricing, different tiers have different \"priorities\". It would be helpful to know what these \"priorities\" mean in the real world. If I submit a request as a low-priority…",[176,181,188],{"value":177,"source":53,"evidence":178},"Diffbot",[179,180],{"id":165,"by":166,"text":167},{"id":93,"by":94,"text":95},{"value":182,"source":53,"evidence":183},"Apify",[184],{"id":185,"by":186,"text":187},"15612327","kashprime","Is this similar to Apify? https:\u002F\u002Fwww.apify.com\u002F",{"value":189,"source":53,"evidence":190},"Page.REST",[191,195],{"id":192,"by":193,"text":194},"15612952","laktek","Nice work! I also built something very similar called Page.REST (Show HN thread: https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=15189099) Page.REST supports extracting contents using CSS…",{"id":196,"by":197,"text":198},"15614130","holtalanm","This looks almost exactly like the functionality provided by https:\u002F\u002Fpage.rest\u002F",{"comments":200,"answered":31,"key":201},10,[202,205,208],{"id":203,"text":204},"15613300","Thanks a lot for all the feedback! 1. Yes will think about it. For just the API it would make definitely sense. However because the same technology currently also powers the…",{"id":206,"text":207},"15613191","Thanks, dito! Is actually already supported when a special parameter is set(on the API-Test-Box on the landing page it is not set). About the pricing. Was a longer process.…",{"id":209,"text":210},"15612930","Thanks a lot! Will investigate and fix. If you run into any other issues please keep them coming!",{"commenters":212,"max_depth":48,"maker_replies":200,"first_comment_mins":213,"lifespan_hours":214},47,4,143,[216,218,220,222,226,228,232,233,235,237,240,243,247,249,252,256,258,261,263,264,265,267,268,272,275,278,281,283,285,286,287,291,294,298,301,304,305,306,309,311,312,314,317,318,320,323,326,328,331,333,335,338,340,343,345,347,352,354,359,361,365,367,369,372,373,375,380,382,386,390,392,396,398,402,407,411,413],[112,42,47,217,113,42],"project\u002Fmixed\u002Fexecution",[73,42,42,219,74,42],"commenter\u002Fneutral\u002Fpricing",[203,42,42,221,11,38],"maker\u002Fneutral\u002Fnone",[223,42,47,224,225,42],"15614515","project\u002Fneutral\u002Fexecution","tomascot",[227,42,47,224,113,42],"15615899",[229,42,47,230,231,42],"15615417","project\u002Fneutral\u002Fidea","matt_wulfeck",[149,42,47,230,150,42],[192,38,48,234,193,42],"project\u002Fpositive\u002Fidea",[206,38,42,236,11,38],"maker\u002Fneutral\u002Fpricing",[238,38,47,221,239,42],"15613394","tomerbd",[241,38,42,242,193,42],"15613506","commenter\u002Fneutral\u002Fnone",[244,38,38,245,246,42],"15613540","project\u002Fcritical\u002Fexecution","TamDenholm",[248,38,42,242,193,42],"15613621",[250,38,48,251,193,42],"15613915","maker\u002Fpositive\u002Fexecution",[253,38,47,254,255,42],"15614214","project\u002Fneutral\u002Fnone","BrandoElFollito",[257,38,47,221,193,42],"15614587",[259,33,38,245,260,42],"15612828","jotaen",[209,33,42,262,11,38],"maker\u002Fneutral\u002Fexecution",[165,47,47,254,166,42],[64,47,42,254,11,38],[77,47,48,266,78,42],"project\u002Fpositive\u002Fpricing",[93,47,47,254,94,42],[269,213,42,270,271,42],"15612868","other\u002Fneutral\u002Fnone","pen2l",[273,213,42,270,274,42],"15613543","twblalock",[276,213,42,270,277,42],"15614500","always_good",[279,213,42,270,280,42],"15615396","mfrye0",[100,48,48,282,101,42],"project\u002Fpositive\u002Fnone",[136,31,47,284,137,42],"project\u002Fneutral\u002Fpricing",[81,31,47,284,57,42],[196,39,47,230,197,42],[288,18,38,289,290,42],"15613665","project\u002Fcritical\u002Fperformance","adventurer",[292,18,47,293,57,42],"15613951","maker\u002Fneutral\u002Fperformance",[295,296,42,270,297,42],"15612002",9,"TenJack",[299,296,47,300,57,42],"15612050","maker\u002Fneutral\u002Fidea",[302,200,42,270,303,42],"15615642","nl",[185,34,47,230,186,42],[67,34,42,221,11,38],[307,308,47,217,128,42],"15615246",12,[310,308,47,262,57,42],"15615467",[127,308,38,245,128,42],[120,313,38,245,121,42],13,[315,313,42,270,316,42],"15611890","Treegarden",[56,313,47,262,57,42],[319,313,42,242,121,42],"15611989",[321,313,38,245,322,42],"15612066","weego",[324,313,42,242,325,42],"15613229","nedwin",[327,313,47,221,57,42],"15614242",[329,313,38,245,330,42],"15612412","taternuts",[172,332,47,284,173,42],14,[334,332,47,293,57,42],"15611983",[336,332,47,337,173,42],"15613656","project\u002Fneutral\u002Fperformance",[339,332,47,293,57,42],"15614215",[341,332,47,342,173,42],"15614503","project\u002Fneutral\u002Fdocs",[116,344,38,245,117,42],15,[346,344,47,262,57,42],"15615476",[348,349,47,350,351,42],"15612945",16,"maker\u002Fneutral\u002Fdocs","tylerpachal",[353,349,42,221,11,38],"15613106",[355,356,38,357,358,42],"15611945",17,"project\u002Fcritical\u002Fdocs","noinput",[360,356,42,262,11,38],"15612125",[362,363,42,270,364,42],"15617805",18,"nreece",[157,366,47,254,158,42],19,[368,366,42,230,11,38],"15612428",[370,371,47,254,57,42],"15611752",20,[104,371,47,224,105,42],[374,371,42,224,11,38],"15612678",[376,377,47,378,379,42],"15615400",21,"project\u002Fneutral\u002Fdesign","gbrits",[381,377,47,378,57,42],"15615487",[383,384,48,234,385,42],"15611647",22,"JanKoenig",[387,388,47,224,389,42],"15612490",23,"swaraj",[391,388,42,224,11,38],"15612598",[393,394,38,289,395,42],"15613389",24,"attacomsian",[397,394,47,293,57,42],"15613954",[399,400,42,270,401,42],"15616014",25,"purplepotato",[403,404,48,405,406,42],"15616399",26,"project\u002Fpositive\u002Fdesign","taivare",[408,409,47,224,410,42],"15621320",27,"palani666",[412,409,47,224,57,42],"15621349",[414,415,38,416,417,42],"15612174",28,"project\u002Fcritical\u002Fidea","profalseidol",[],[420,425,429,434,439,444],{"id":421,"name":422,"points":423,"category":424},"15624206","Server.js",295,"languages",{"id":426,"name":427,"points":428,"category":424},"15644430","Javalin 1.0",271,{"id":430,"name":431,"points":432,"category":433},"15645276","Orange Forum",226,"infrastructure",{"id":435,"name":436,"points":437,"category":438},"15622140","CertStream",142,"security_privacy",{"id":440,"name":441,"points":442,"category":443},"15638201","OSX-ISO",135,"dev_tools",{"id":445,"name":446,"points":447,"category":433},"15611265","url-to-pdf",129,false,"v2",{"id":451,"name":5,"line":6,"ai":448,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":452,"top":18,"kw":453,"maker":11,"domain":10,"tier":19,"attempts":42,"strip":454,"url":9,"mix":455,"praise":89,"objection":109,"sub":21},15611605,-0.14,[24,25,26,27],"3033531513530053133311311333131333351351",[31,32,33,34],1794036560472]