[{"data":1,"prerenderedAt":336},["ShallowReactive",2],{"project-37234887":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":44,"post_text":46,"what":47,"said":85,"maker_in_thread":180,"shape":192,"grid":196,"attempts":299,"same_week":300,"maker_shows":18,"maker_hidden":328,"reader_version":329,"card":330},"37234887","Fast vector similarity","Fast vector similarity using Rust and Python","title","Show HN: Fast vector similarity using Rust and Python","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Ffast_vector_similarity","github.com","eigenvalue",1692794077,2023,141,40,451,3109,15,"notable","data","similarity measures library","library",[24,25,26,27,28],"vector similarity","hoeffding's d","rust python","bootstrap similarity","embedding comparison","curious",-0.051,{"positive":32,"neutral":33,"mixed":34,"critical":35},2,7,0,3,12,{"design":38,"performance":40,"idea":41,"execution":42,"docs":43},{"positive":34,"critical":39},1,{"positive":34,"critical":32},{"positive":34,"critical":34},{"positive":39,"critical":34},{"positive":34,"critical":34},[39,34,35,35,34,35,34,34,45,34,35,34,34,45,34,34,35,34,34,34,35,34,34,34,34,39,34,35,34,34,39,34,34,34,34,34,34,34,34,34],5,"I recently found myself computing the similarity between lots of very high dimensional vectors (i.e., sentence embedding vectors from LLMs), and I wanted to try some more powerful measures of similarity\u002Fdependency than just Cosine similarity, which seems to be the default for everything nowadays because of its computational efficiency.\n\nThere are many other more involved measures that can detect more subtle relationships, but the problem is that some of them are quite slow to compute, especially if you're trying to do it in Python. For my favorite measure of statistical dependency, Hoeffding's",{"problem":48,"audience":54,"pricing":54,"motivation":55,"time_to_build":54,"stack":59,"openness":82,"stage":83,"team_size":84},{"value":49,"source":50,"evidence":51},"Cosine similarity is limited and more powerful dependency measures are slow to compute in Python","maker",[52],{"id":53},"post",null,{"value":56,"source":50,"evidence":57},"Maker needed to compute similarity on high-dimensional embeddings and wanted to learn Rust and Python packaging",[58],{"id":53},[60,64,68,72,78],{"value":61,"source":50,"evidence":62},"Rust",[63],{"id":53},{"value":65,"source":50,"evidence":66},"Python",[67],{"id":53},{"value":69,"source":50,"evidence":70},"Rayon",[71],{"id":53},{"value":73,"source":50,"evidence":74},"Maturin",[75],{"id":76,"by":11,"text":77},"37239788","Yeah, like the other commenter said, everything is in this file here: https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Ffast_vector_similarity\u002Fblob\u002Fmain\u002F.github\u002Fworkflows\u002Fbuild-wheels.yml If…",{"value":79,"source":50,"evidence":80},"GitHub Actions",[81],{"id":76,"by":11,"text":77},"open_source","launched","solo",{"praise":86,"objections":103,"requests":121,"questions":147,"alternatives":162},[87,99],{"value":88,"source":89,"evidence":90},"Fast performance versus NumPy","commenter",[91,95],{"id":92,"by":93,"text":94},"37238644","ashvardanian","Updated, thanks :)",{"id":96,"by":97,"text":98},"37239927","simonw","Always great to see projects like this that distribute wheels for a whole ton of different platforms - makes installation so much less painful if you don't need to ensure you…",{"value":100,"source":89,"evidence":101},"Wheels for many platforms make installation easy",[102],{"id":96,"by":97,"text":98},[104,111],{"value":105,"source":89,"evidence":106},"Library can consume all CPU cores and compete with other workloads",[107],{"id":108,"by":109,"text":110},"37241218","binary132","I realize you have a different usecase in mind, so the following feedback might not be that useful to you. I also haven’t reviewed your API design, so perhaps you have already…",{"value":112,"source":89,"evidence":113},"Brute force approach is slow compared with approximate search",[114,118],{"id":115,"by":116,"text":117},"37238064","spullara","It sounds like it is brute force comparison so I imagine any approximate solution will be a lot faster. I have a similar brute force search library in Rust. Though I didn't find…",{"id":119,"by":93,"text":120},"37238453","It’s a good start, but you can’t generally get even remotely close to hardware potential in Rust, let alone Python. I had to implement a separate C99 library to always trigger…",[122,130,140],{"value":123,"source":89,"evidence":124},"Add an option to limit parallelism or thread count",[125,126],{"id":108,"by":109,"text":110},{"id":127,"by":128,"text":129},"37243742","zacmps","I would say default to min(affinity, system threads, 16) and let the user input a specific number to override it. I had a problem recently where a library started a process per…",{"value":131,"source":89,"evidence":132},"Support passing a matrix or dataframe for batch queries",[133,137],{"id":134,"by":135,"text":136},"37240699","alpaca128","In the example it looks like this needs one JSON encoding\u002Fdecoding step per call for parameters & results which seems a bit inefficient, but maybe my intuition is wrong there? Or…",{"id":138,"by":11,"text":139},"37242642","Yes, you're right, and it's a good point. I'll add an additional python function that allows you to supply a pandas dataframe or numpy array for one of the arguments so you can…",{"value":141,"source":89,"evidence":142},"Write-up on GitHub Actions cross-compilation",[143],{"id":144,"by":145,"text":146},"37239441","tmaly","Could you do a future write up on the whole process of how you got Github actions to cross compile to different platforms? That sounds very fascinating.",[148,155],{"value":149,"source":89,"evidence":150},"How does it compare to FAISS?",[151],{"id":152,"by":153,"text":154},"37237835","DigitalNoumena","Nice work! How does it compare against FAISS?",{"value":156,"source":89,"evidence":157},"How is normalized mutual information computed?",[158],{"id":159,"by":160,"text":161},"37241142","foota","If you don't mind, how is this calculated? I found some references, but I'm not sure I understand.",[163,169,173],{"value":164,"source":50,"evidence":165},"FAISS",[166],{"id":167,"by":11,"text":168},"37238048","It's really quite different in goals to FAISS. FAISS is for if you have millions of stored vectors and want to do quick semantic similarity for retrieval. It manages to do this…",{"value":170,"source":89,"evidence":171},"USearch and SimSIMD",[172],{"id":119,"by":93,"text":120},{"value":174,"source":89,"evidence":175},"Qdrant",[176],{"id":177,"by":178,"text":179},"37245682","timvisee","Awesome work! At Qdrant we do this at scale. Store billions of vectors in a cluster of any size. Also in Rust which turned out to be an amazing choice, and fully open source. It…",{"comments":181,"answered":33,"key":182},17,[183,186,189],{"id":184,"text":185},"37242981","It's a good point, and the way my library is designed now it would be a big resource hog. Not sure the best way to handle that given that it's supposed to be a super easy to use…",{"id":187,"text":188},"37240501","Author here. I just wanted to add that I struggled mightily to include normalized mutual information in this library but simply couldn’t get it to run quickly enough with 15,000…",{"id":190,"text":191},"37242611","There are lots of different ways of doing it (for a reference, see https:\u002F\u002Fwww.mdpi.com\u002F1099-4300\u002F19\u002F11\u002F631 ), but this was the basic approach I was trying that I gave up on…",{"commenters":193,"max_depth":45,"maker_replies":181,"first_comment_mins":194,"lifespan_hours":195},22,191,32,[197,199,201,203,205,207,209,211,213,215,218,220,224,225,228,232,235,238,239,241,243,246,248,253,255,258,261,263,267,269,273,274,276,278,281,284,287,290,292,296],[108,34,39,198,109,34],"project\u002Fcritical\u002Fdesign",[184,34,34,200,11,39],"maker\u002Fneutral\u002Fdesign",[127,34,35,202,128,34],"maker\u002Fneutral\u002Fperformance",[204,34,35,200,109,34],"37254033",[187,39,34,206,11,39],"project\u002Fneutral\u002Fperformance",[159,39,35,208,160,34],"maker\u002Fneutral\u002Fidea",[190,39,34,210,11,39],"project\u002Fneutral\u002Fexecution",[212,39,34,210,11,39],"37242627",[96,32,45,214,97,34],"project\u002Fpositive\u002Fexecution",[216,32,34,217,11,39],"37240004","maker\u002Fneutral\u002Fexecution",[144,35,35,219,145,34],"maker\u002Fneutral\u002Fdocs",[221,35,34,222,223,34],"37239531","commenter\u002Fneutral\u002Fdocs","appplication",[76,35,34,219,11,39],[177,226,45,227,178,34],4,"project\u002Fpositive\u002Fnone",[229,45,34,230,231,34],"37241448","other\u002Fneutral\u002Fnone","kordlessagain",[233,45,34,234,11,39],"37243461","maker\u002Fneutral\u002Fnone",[152,236,35,237,153,34],6,"project\u002Fneutral\u002Fidea",[167,236,34,237,11,39],[115,236,34,240,116,34],"commenter\u002Fneutral\u002Fperformance",[242,236,34,202,11,39],"37238152",[244,33,35,237,245,34],"37246260","sps44",[247,33,34,208,11,39],"37247084",[249,250,34,251,252,34],"37239389",8,"other\u002Fneutral\u002Fidea","Xenoamorphous",[254,250,34,237,11,39],"37239839",[256,250,34,251,257,34],"37241184","esafak",[134,259,39,260,135,34],9,"project\u002Fcritical\u002Fperformance",[138,259,34,262,11,39],"project\u002Fneutral\u002Fdesign",[264,265,35,237,266,34],"37239297",10,"pajep",[268,265,34,237,11,39],"37239407",[270,271,34,230,272,34],"37243675",11,"matmulbro",[119,36,39,260,93,34],[275,36,34,206,11,39],"37238590",[92,36,34,277,93,34],"other\u002Fneutral\u002Fdocs",[279,36,34,240,280,34],"37238750","bluejekyll",[282,36,34,240,283,34],"37239188","jackmott42",[285,36,34,286,11,39],"37239437","other\u002Fneutral\u002Fperformance",[288,36,34,240,289,34],"37239696","steveklabnik",[291,36,34,240,11,39],"37239889",[293,294,34,230,295,34],"37237943",13,"softwaredoug",[297,294,34,298,11,39],"37238233","commenter\u002Fneutral\u002Fexecution",[],[301,306,310,315,319,323],{"id":302,"name":303,"points":304,"category":305},"37190743","flipreps",373,"creative",{"id":307,"name":308,"points":309,"category":305},"37194128","Just intonation keyboard",331,{"id":311,"name":312,"points":313,"category":314},"37188659","Rivet",327,"infrastructure",{"id":316,"name":317,"points":318,"category":305},"37188102","Avatartion",162,{"id":320,"name":321,"points":322,"category":314},"37224148","FlakeHub",154,{"id":324,"name":325,"points":326,"category":327},"37180017","Poozle",132,"ai",false,"v2",{"id":331,"name":5,"line":6,"ai":328,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":332,"top":18,"kw":333,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":334,"url":9,"mix":335,"praise":88,"objection":105,"sub":21},37234887,-0.05,[24,25,26,27,28],"1033030050300500300030000103001000000000",[32,33,34,35],1794036640579]