[{"data":1,"prerenderedAt":296},["ShallowReactive",2],{"project-41544969":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":41,"post_text":42,"what":43,"said":76,"maker_in_thread":154,"shape":165,"grid":170,"attempts":259,"same_week":260,"maker_shows":38,"maker_hidden":288,"reader_version":289,"card":290},"41544969","Wordllama","Things you can do with the token embeddings of an LLM","title","Show HN: Wordllama – Things you can do with the token embeddings of an LLM","https:\u002F\u002Fgithub.com\u002Fdleemiller\u002FWordLlama","github.com","deepsquirrelnet",1726370714,2024,370,36,131,3515,4,"legend","ai","embedding models","library",[24,25,26,27,28],"word embeddings","token embeddings","semantic similarity","fuzzy deduplication","cpu embeddings","curious",0.174,{"positive":32,"neutral":33,"mixed":34,"critical":34},3,5,1,10,{"idea":37,"performance":40},{"positive":38,"critical":39},2,0,{"positive":39,"critical":34},[32,39,34,39,39,39,39,39,39,39,39,39,39,39,32,39,33,32,39,39,39,39,39,39,32,32,39,39,39,39,39,39,39,33,33,32],"After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes.\n\nHow can this be accomplished? The model is simply token embeddings that are avera",{"problem":44,"audience":50,"pricing":50,"motivation":51,"time_to_build":50,"stack":55,"openness":73,"stage":74,"team_size":75},{"value":45,"source":46,"evidence":47},"Lightweight semantic string tasks like dedup, ranking and clustering without deep learning runtimes","maker",[48],{"id":49},"post",null,{"value":52,"source":46,"evidence":53},"Wanted a lightweight utility for small LLM-related tasks like preparing inputs and building evaluators",[54],{"id":49},[56,61,65,69],{"value":57,"source":58,"evidence":59},"Python","inferred",[60],{"id":49},{"value":62,"source":46,"evidence":63},"Cython",[64],{"id":49},{"value":66,"source":46,"evidence":67},"numpy",[68],{"id":49},{"value":70,"source":46,"evidence":71},"sentence transformers",[72],{"id":49},"open_source","launched","unknown",{"praise":77,"objections":93,"requests":109,"questions":125,"alternatives":137},[78,86],{"value":79,"source":80,"evidence":81},"Tiny model size is an advantage over SBERT smallest models","commenter",[82],{"id":83,"by":84,"text":85},"41545765","authorfly","Nice. I like the tiny size a lot, that's already an advantage over SBERTs smallest models. But it seems quite dated technically - which I understand is a tradeoff for performance…",{"value":87,"source":80,"evidence":88},"Useful for game making",[89],{"id":90,"by":91,"text":92},"41547225","ttpphd","This is great for game making! Thank you!",[94,105],{"value":95,"source":80,"evidence":96},"Benchmarks clearly worse than MiniLM models",[97,101],{"id":98,"by":99,"text":100},"41549429","refulgentis","Its a 17 MB model that benchmarks obviously worse than MiniLM v2 (which is SBERT). I run V3 on ONNX on every platform you can think of with a 23 MB model. I don't intend for that…",{"id":102,"by":103,"text":104},"41545666","dspoka","Looks cool! Any advantages to the mini-lm model - it seems better on most mteb tasks but wondering if maybe inference or something is better.",{"value":106,"source":80,"evidence":107},"Dated technically compared to LLM-based embeddings",[108],{"id":83,"by":84,"text":85},[110,114],{"value":111,"source":80,"evidence":112},"Support for multiple similarity types such as semantic, NLI and noun-abstract",[113],{"id":83,"by":84,"text":85},{"value":115,"source":80,"evidence":116},"Multilingual or French language support",[117,121],{"id":118,"by":119,"text":120},"41546377","jcmeyrignac","Any plan for languages other than english? This would be a perfect tool for french language.",{"id":122,"by":123,"text":124},"41554888","Ey7NFZ3P0nzAe","Very interested in a multilingual version too! FYI huggingface hosts datasets too. And wikipedia has a nice portal for datasets :…",[126],{"value":127,"source":80,"evidence":128},"Whether embeddings can encode opposite meanings like burning and freezing",[129,133],{"id":130,"by":131,"text":132},"41549486","curl-up","I assume that by \"ChatGPT embeddings\" you mean OpenAI embedding models. In that case, \"burning\" and \"freezing\" are not opposite at all, with a cosine similarity of 0.46 (running…",{"id":134,"by":135,"text":136},"41550901","magicalhippo","This might be a dumb question but... if I get the embeddings of words with a common theme like \"burning\", \"warm\", \"cool\", \"freezing\", would I be able to relatively well fit an…",[138,143,147],{"value":139,"source":80,"evidence":140},"MiniLM \u002F SBERT models",[141,142],{"id":102,"by":103,"text":104},{"id":98,"by":99,"text":100},{"value":144,"source":80,"evidence":145},"OpenAI embedding models",[146],{"id":130,"by":131,"text":132},{"value":148,"source":80,"evidence":149},"TF-IDF with kmeans",[150],{"id":151,"by":152,"text":153},"41553869","xk3","With a large corpus (10,000+ sentences--each sentence is a \"document\" in my usecase) I can get similar results by kmeans clustering TF-IDF spmatrix vectors but it looks like this…",{"comments":33,"answered":32,"key":155},[156,159,162],{"id":157,"text":158},"41547150","Great ideas - I’ll run some experiments and see how feasible it is. I’d want to see how performance is if I train on a single type of similarity. Without any contextual…",{"id":160,"text":161},"41548893","That’s a really cool idea. I’ll think about it some more, because it sounds like a feasible implementation for this. I think if you take the magnitude of any token embedding in…",{"id":163,"text":164},"41547597","It’s certainly feasible. I’d need to put together a corpus for training and I’m not terribly familiar with what’s available for French language. I have done some training with…",{"commenters":166,"max_depth":167,"maker_replies":33,"first_comment_mins":168,"lifespan_hours":169},21,7,191,56,[171,173,175,177,179,181,183,185,187,189,192,194,197,201,203,205,207,209,211,214,216,219,222,225,227,229,232,234,237,239,242,244,246,248,250,254],[83,39,32,172,84,39],"project\u002Fmixed\u002Fidea",[157,39,39,174,11,34],"maker\u002Fneutral\u002Fidea",[98,39,34,176,99,39],"project\u002Fcritical\u002Fperformance",[130,39,39,178,131,39],"commenter\u002Fneutral\u002Fidea",[134,39,39,180,135,39],"other\u002Fneutral\u002Fidea",[182,39,39,180,84,39],"41553624",[184,39,39,178,84,39],"41553711",[186,39,39,178,131,39],"41553917",[188,39,39,178,84,39],"41559543",[190,39,39,191,131,39],"41559773","commenter\u002Fneutral\u002Fnone",[193,39,39,178,84,39],"41566415",[195,34,39,180,196,39],"41547680","warangal",[198,34,39,199,200,39],"41548649","other\u002Fneutral\u002Fnone","nostrebored",[160,34,39,202,11,34],"maker\u002Fpositive\u002Fidea",[118,38,32,204,119,39],"project\u002Fneutral\u002Fidea",[163,38,39,206,11,34],"maker\u002Fneutral\u002Fnone",[122,38,33,208,123,39],"project\u002Fpositive\u002Fidea",[151,32,32,210,152,39],"project\u002Fneutral\u002Fperformance",[212,18,39,199,213,39],"41548740","Der_Einzige",[215,18,39,199,152,39],"41553841",[217,33,39,199,218,39],"41545767","anonymousfilter",[220,33,39,199,221,39],"41547071","batch12",[223,33,39,199,224,39],"41547326","wakaru44",[226,33,39,191,221,39],"41551192",[102,228,32,210,103,39],6,[230,228,32,210,231,39],"41545720","lennxa",[233,228,39,199,84,39],"41545797",[235,228,39,199,236,39],"41545831","byefruit",[238,228,39,210,11,34],"41547290",[240,228,39,180,241,39],"41547721","ryeguy_24",[243,228,39,199,11,34],"41548318",[245,228,39,199,200,39],"41548693",[247,228,39,180,241,39],"41550389",[90,167,33,249,91,39],"project\u002Fpositive\u002Fnone",[251,252,33,208,253,39],"41548377",8,"visarga",[255,256,32,257,258,39],"41552424",9,"project\u002Fneutral\u002Fnone","johnthescott",[],[261,266,271,276,280,284],{"id":262,"name":263,"points":264,"category":265},"41521919","FixHub",928,"hardware",{"id":267,"name":268,"points":269,"category":270},"41539125","Meet.hn",768,"consumer_social",{"id":272,"name":273,"points":274,"category":275},"41517312","Konty",392,"dev_tools",{"id":277,"name":278,"points":279,"category":275},"41563958","Void",347,{"id":281,"name":282,"points":283,"category":275},"41530190","FlowTracker",280,{"id":285,"name":286,"points":287,"category":20},"41545335","Bullshit Remover",270,false,"v2",{"id":291,"name":5,"line":6,"ai":288,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":292,"top":18,"kw":293,"maker":11,"domain":10,"tier":19,"attempts":39,"strip":294,"url":9,"mix":295,"praise":79,"objection":95,"sub":21},41544969,0.17,[24,25,26,27,28],"301000000000003053000000330000000553",[32,33,34,34],1793991100300]