[{"data":1,"prerenderedAt":325},["ShallowReactive",2],{"project-31693314":3},{"id":4,"name":5,"line":5,"line_source":6,"title":7,"url":8,"domain":8,"maker":9,"posted_at":10,"year":11,"points":12,"comments":13,"rank_in_year":14,"shown_in_year":15,"top":16,"tier":17,"category":18,"subcategory":19,"format":20,"keywords":21,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":34,"aspects":35,"strip":40,"post_text":42,"what":43,"said":80,"maker_in_thread":174,"shape":182,"grid":186,"attempts":285,"same_week":286,"maker_shows":31,"maker_hidden":316,"reader_version":317,"card":318},"31693314","Analyzing top HN posts with language models","title","Show HN: Analyzing top HN posts with language models",null,"jayalammar",1654865243,2022,117,43,533,2570,21,"notable","data","text clustering analysis","web_app",[22,23,24,25,26],"hacker news","text embeddings","clustering","umap","language models","debated",-0.103,{"positive":30,"neutral":31,"mixed":32,"critical":33},4,2,0,5,11,{"idea":36,"execution":37,"design":38},{"positive":32,"critical":31},{"positive":31,"critical":31},{"positive":32,"critical":39},1,[39,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,39,32,33,32,32,41,32,33,39,41,32,32,39,32,32,32,33,39,33],3,"Hi HN,\n\nI spent a few weeks looking at the top HN posts of all time. This included exploration, clustering, creating visualizations, and zooming in on what (to me personally) seems like some of the best discussions on here.\n\nThree things in this post:\n\n1- The interesting groups of HN posts\n\n2- The interactive visualizations that you can explore in your browser\n\n3- The data from this exploration -- this includes CSV of the titles as well as the text embeddings of 3,000 Ask HN articles.\n\nBlog post about this whole process here: [1]\n\n============\n\n1- The interesting groups of HN posts\n\nFrom the e",{"problem":44,"audience":50,"pricing":8,"motivation":54,"time_to_build":8,"stack":58,"openness":79,"stage":79,"team_size":79},{"value":45,"source":46,"evidence":47},"Finding patterns and insightful groups among the top Hacker News posts of all time","maker",[48],{"id":49},"post",{"value":51,"source":46,"evidence":52},"People interested in language models and Hacker News discussions",[53],{"id":49},{"value":55,"source":46,"evidence":56},"Exploring top HN posts and sharing interesting discussions",[57],{"id":49},[59,69,75],{"value":60,"source":46,"evidence":61},"Cohere Embed endpoint",[62,65],{"id":63,"by":9,"text":64},"31695096","Disclosure: These were made by Cohere's embeddings, a company where I work. The process should work on text embeddings from other sources.",{"id":66,"by":67,"text":68},"31694976","wpietri","The conflict of interest here concerns me. I don't object to content marketing, but I'd rather a) you were clear from the start that you work for this company and are promoting…",{"value":70,"source":46,"evidence":71},"UMAP",[72],{"id":73,"by":9,"text":74},"31694384","So it really depends on what you use for clustering. In this case, I'm clustering by the original embeddings so the UMAP results are different. I've also seen: 1- Clustering by…",{"value":76,"source":46,"evidence":77},"Google Colab notebook",[78],{"id":49},"unknown",{"praise":81,"objections":104,"requests":136,"questions":149,"alternatives":162},[82,98],{"value":83,"source":84,"evidence":85},"Interesting analysis and useful visualizations","commenter",[86,90,94],{"id":87,"by":88,"text":89},"31697424","victorianoi","Nice idea and analysis! I reproduced it as well with https:\u002F\u002Fgraphext.com and got similar clusters…",{"id":91,"by":92,"text":93},"31701528","renewiltord","I love this shit, dude. Not for any great insight. I just enjoy this sort of thing. Good stuff.",{"id":95,"by":96,"text":97},"31701484","arolihas","Love your blog posts and visualizations Jay, thanks for sharing!",{"value":99,"source":84,"evidence":100},"Embeddings and data shared freely for reuse",[101],{"id":102,"by":9,"text":103},"31695126","Thanks. I just added a disclosure to the comment (can't edit the parent anymore). The full embeddings are freely provided here without the need to use the service.",[105,116,123,129],{"value":106,"source":84,"evidence":107},"Undisclosed conflict of interest: author works at Cohere, which sells the embedding service",[108,109,112],{"id":66,"by":67,"text":68},{"id":110,"by":67,"text":111},"31699118","\"Normal\" doesn't mean good, so that line of argument doesn't work for me. But the specific \"stink\" I'm objecting to here is, as I said, a conflict of interest. In specific,…",{"id":113,"by":114,"text":115},"31713735","woojoo666","and the more fake reviews, the less I trust them. Likewise, the more people here advertising their products without proper disclosure, the less I'll trust the honesty of the…",{"value":117,"source":84,"evidence":118},"Clusters look dense and poorly separated in the UMAP projection",[119],{"id":120,"by":121,"text":122},"31694250","uniqueuid","Interesting, but it doesn't seem like the dimensionality reduction produces a good separation of topics. The UMAP projection looks pretty dense. Did you consider pruning or using…",{"value":124,"source":84,"evidence":125},"Recency bias in the dataset because top post counts and scores grew over time",[126],{"id":127,"by":88,"text":128},"31697509","BTW there is an implicit recency bias in the dataset, since 2017 the number of top 3K post became more frequently and the avg score is larger year after year as the community in…",{"value":130,"source":84,"evidence":131},"Ask HN posts are naturally biased toward engagement",[132],{"id":133,"by":134,"text":135},"31694222","toppy","I don't know how HN score metrics work but after some short review of the datafile [1] I've noticed a lot of the posts has the form of a simple questions and as such seems to be…",[137,145],{"value":138,"source":84,"evidence":139},"Try t-SNE or other clustering approaches for better separation",[140,144],{"id":141,"by":142,"text":143},"31694447","PaulHoule","Try t-SNE. I used to scoff at cluster plots until I saw those but with t-SNE… wow, those clusters are actually separated!",{"id":120,"by":121,"text":122},{"value":146,"source":84,"evidence":147},"Add additional engagement metrics to remove bias",[148],{"id":133,"by":134,"text":135},[150,155],{"value":151,"source":84,"evidence":152},"Why cluster on original embeddings rather than UMAP output",[153,154],{"id":120,"by":121,"text":122},{"id":73,"by":9,"text":74},{"value":156,"source":84,"evidence":157},"Is top score a sign of quality or just early posting",[158],{"id":159,"by":160,"text":161},"31698144","natch","There's too much fixation with \"top\" in our industry. Top voted tends to mostly be a function of early posting. Later posts don't get votes because they simply were not seen.…",[163,167],{"value":164,"source":84,"evidence":165},"graphext",[166],{"id":87,"by":88,"text":89},{"value":168,"source":84,"evidence":169},"Ploomber HN classifier",[170],{"id":171,"by":172,"text":173},"31694754","hoerzu","I grouped posts by topic: https:\u002F\u002Fn3ws.ploomber.io Explanation: https:\u002F\u002Fploomber.io\u002Fblog\u002Fhn_classifier\u002F",{"comments":175,"answered":41,"key":176},7,[177,178,181],{"id":102,"text":103},{"id":179,"text":180},"31693359","[1] https:\u002F\u002Ftxt.cohere.ai\u002Fcombing-for-insight-in-10-000-hacker-news-posts-with-text-clustering\u002F [2] https:\u002F\u002Fassets.cohere.ai\u002Fblog\u002Ftext-clustering\u002Faskhn_cluster_6a.html [3]…",{"id":63,"text":64},{"commenters":183,"max_depth":184,"maker_replies":175,"first_comment_mins":33,"lifespan_hours":185},22,9,47,[187,189,191,194,197,200,202,204,206,208,211,212,214,216,218,221,223,224,225,227,230,233,236,238,240,243,246,248,250,252,255,256,257,259,260,263,266,269,272,274,275,280,281],[66,32,39,188,67,32],"maker\u002Fcritical\u002Fidea",[102,32,32,190,9,39],"maker\u002Fneutral\u002Fnone",[192,32,32,193,67,32],"31699134","commenter\u002Fneutral\u002Fnone",[195,32,32,193,196,32],"31696161","rmbyrro",[198,32,32,193,199,32],"31696579","notatoad",[201,32,32,193,196,32],"31700804",[203,32,32,193,114,32],"31701048",[205,32,32,193,196,32],"31702674",[207,32,32,193,114,32],"31703592",[209,32,32,193,210,32],"31711142","tough",[113,32,32,193,114,32],[213,32,32,193,67,32],"31699130",[215,32,32,193,196,32],"31700782",[217,32,32,193,67,32],"31709676",[219,32,32,193,220,32],"31697318","mbesto",[110,32,32,222,67,32],"commenter\u002Fcritical\u002Fnone",[179,39,32,190,9,39],[63,39,32,190,9,39],[159,31,32,226,160,32],"other\u002Fneutral\u002Fnone",[228,31,32,226,229,32],"31698372","tra3",[231,31,32,226,232,32],"31700814","ultra_nick",[234,31,32,226,235,32],"31702491","mateo1",[120,41,39,237,121,32],"project\u002Fcritical\u002Fexecution",[73,41,32,239,9,39],"maker\u002Fneutral\u002Fexecution",[241,41,33,242,121,32],"31694439","project\u002Fpositive\u002Fexecution",[244,41,32,245,9,39],"31694592","maker\u002Fpositive\u002Fnone",[247,41,32,193,121,32],"31694648",[141,41,41,249,142,32],"project\u002Fneutral\u002Fexecution",[251,41,32,193,121,32],"31694482",[253,41,32,193,254,32],"31700641","nighthawk454",[87,30,33,242,88,32],[127,30,39,237,88,32],[258,30,41,249,88,32],"31697596",[171,33,32,226,172,32],[261,33,32,226,262,32],"31707800","chrisMyzel",[133,264,39,265,134,32],6,"project\u002Fcritical\u002Fidea",[267,264,32,268,9,39],"31694393","maker\u002Fneutral\u002Fidea",[270,264,32,271,134,32],"31694584","commenter\u002Fneutral\u002Fidea",[273,264,32,268,9,39],"31694791",[91,175,33,245,92,32],[276,277,39,278,279,32],"31694556",8,"project\u002Fcritical\u002Fdesign","bryanrasmussen",[95,184,33,245,96,32],[282,283,32,226,284,32],"31701170",10,"earthboundkid",[],[287,292,297,302,307,312],{"id":288,"name":289,"points":290,"category":291},"31710603","Vytal",313,"security_privacy",{"id":293,"name":294,"points":295,"category":296},"31755025","anonfriendly",306,"web_saas",{"id":298,"name":299,"points":300,"category":301},"31679811","Ory Kratos",290,"infrastructure",{"id":303,"name":304,"points":305,"category":306},"31726472","Tuc",241,"dev_tools",{"id":308,"name":309,"points":310,"category":311},"31724613","Animation Magic",238,"learning_science",{"id":313,"name":314,"points":315,"category":296},"31741018","My Side Project Rocks",220,false,"v2",{"id":319,"name":5,"line":5,"ai":316,"cat":18,"year":11,"points":12,"comments":13,"rec":27,"s":320,"top":16,"kw":321,"maker":9,"domain":322,"tier":17,"attempts":32,"strip":323,"url":-1,"mix":324,"praise":83,"objection":106,"sub":19},31693314,-0.1,[22,23,24,25,26],"","1000000000000000000001050030513001000515",[30,31,32,33],1794036621324]