[{"data":1,"prerenderedAt":309},["ShallowReactive",2],{"project-37133163":3},{"id":4,"name":5,"line":5,"line_source":6,"title":7,"url":8,"domain":9,"maker":10,"posted_at":11,"year":12,"points":13,"comments":14,"rank_in_year":15,"shown_in_year":16,"top":17,"tier":18,"category":19,"subcategory":20,"format":21,"keywords":22,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":41,"post_text":44,"what":45,"said":94,"maker_in_thread":180,"shape":189,"grid":192,"attempts":266,"same_week":271,"maker_shows":300,"maker_hidden":301,"reader_version":302,"card":303},"37133163","Llama2 Embeddings FastAPI Server","title","Show HN: Llama2 Embeddings FastAPI Server","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fllama_embeddings_fastapi_service","github.com","eigenvalue",1692102687,2023,178,31,327,3109,11,"notable","ai","embeddings server","api",[23,24,25,26,27],"llama2 embeddings","embeddings api","semantic search","self-hosted embeddings","fastapi","roasted",-0.345,{"positive":31,"neutral":32,"mixed":33,"critical":34},1,7,0,8,16,{"idea":37,"performance":39,"execution":40},{"positive":33,"critical":38},6,{"positive":33,"critical":31},{"positive":31,"critical":31},[31,33,31,33,42,33,33,31,42,31,42,33,33,33,33,33,31,42,31,33,43,33,42,33,42,33,31,31,33,33,42,33],3,5,null,{"problem":46,"audience":44,"pricing":44,"motivation":53,"time_to_build":62,"stack":66,"openness":91,"stage":92,"team_size":93},{"value":47,"source":48,"evidence":49},"Getting embedding vectors from Llama2 and similar GGML models via a simple REST API","maker",[50],{"id":51,"by":10,"text":52},"37133203","Author here. I just wanted a quick and easy way to easily submit strings to a REST API and get back the embedding vectors in JSON using Llama2 and other similar LLMs, so I put…",{"value":54,"source":48,"evidence":55},"Wanted a quick, small, hackable self-hosted way to get Llama2 embeddings and explore LLM embeddings",[56,59],{"id":57,"by":10,"text":58},"37133819","I wanted something that works natively with llama-cpp and langchain that is also small and easy to hack on. And I also wanted everything to be seamlessly cached in SQlite. And…",{"id":60,"by":10,"text":61},"37138700","If all you want to do is make some basic semantic search, that’s probably true. But I strongly suspect we are only just now starting to scratch the surface of what’s possible…",{"value":63,"source":48,"evidence":64},"A couple of days",[65],{"id":51,"by":10,"text":52},[67,72,79,83,87],{"value":68,"source":48,"evidence":69},"FastAPI",[70],{"id":71},"post",{"value":73,"source":48,"evidence":74},"llama-cpp",[75,76],{"id":51,"by":10,"text":52},{"id":77,"by":10,"text":78},"37133745","The embeddings are computed using llama-cpp, but langchain makes a nice convenience wrapper to directly get them, so I use that. The embeddings are 4096 dimensional vectors. And…",{"value":80,"source":48,"evidence":81},"langchain",[82],{"id":77,"by":10,"text":78},{"value":84,"source":48,"evidence":85},"SQLite",[86],{"id":51,"by":10,"text":52},{"value":88,"source":48,"evidence":89},"FAISS",[90],{"id":51,"by":10,"text":52},"open_source","unknown","solo",{"praise":95,"objections":110,"requests":129,"questions":145,"alternatives":157},[96,104],{"value":97,"source":98,"evidence":99},"Clean and simple codebase","commenter",[100],{"id":101,"by":102,"text":103},"37133940","lgvld","Hi, Looks quite clean, congrats. Two questions: 1. Starting from this, what would be the proper way to create embeddings for a complete document (i.e. a long paragraph)? My goal…",{"value":105,"source":98,"evidence":106},"Useful features like token-level embeddings and similarity measures",[107],{"id":108,"by":10,"text":109},"37135518","Cool. I was wondering what Tanimoto meant, since I've tried to make myself familiar with all the useful similarity measures, and apparently it's just another name for Jaccard…",[111,122],{"value":112,"source":98,"evidence":113},"Raw Llama2 likely underperforms dedicated embedding models like bge, gte and e5",[114,118],{"id":115,"by":116,"text":117},"37133597","Palmik","It seems unlikely that raw llama2 will perform better than purpose made encoder models like bge [1], gte [2], e5 [3] or instructor, despite it's much much larger size (for the…",{"id":119,"by":120,"text":121},"37137050","readyplayeremma","The SGPT model is a very high performing text embeddings model adapted from a decoder. Using the same techniques with Llama-2 might perform better than you expect. I think…",{"value":123,"source":98,"evidence":124},"7B model is overkill for semantic embeddings",[125],{"id":126,"by":127,"text":128},"37138556","jerrygenser","I think it's still overkill though for semantic embedding, SBERT is on order of ~250mm parameters while smallest llama at 7b parameters.",[130,138],{"value":131,"source":98,"evidence":132},"Benchmarks against dedicated embedding models or OpenAI",[133,137],{"id":134,"by":135,"text":136},"37133401","lhr0909","Nice work and starred! I am curious how you get the embeddings computed, the # of dimensions for the embeddings and if you have run any benchmarks against OpenAI's offering?…",{"id":115,"by":116,"text":117},{"value":139,"source":98,"evidence":140},"Word-level highlighting examples for token embeddings",[141],{"id":142,"by":143,"text":144},"37136409","clbrmbr","I’d love to see some examples of how the token-in-context embeddings stack up against sentence-level. What new use-cases are unlocked? Perhaps semantic search with…",[146,150],{"value":147,"source":98,"evidence":148},"Does it need a GPU and does inference run on the VPS",[149],{"id":101,"by":102,"text":103},{"value":151,"source":98,"evidence":152},"How to embed whole documents versus sentences",[153,154],{"id":101,"by":102,"text":103},{"id":155,"by":102,"text":156},"37140968","Thank you very much! And when is it useful to compute an embedding for each sentence of a document\u002Fparagraph then?",[158,162,173],{"value":159,"source":98,"evidence":160},"bge, gte and e5 embedding models",[161],{"id":115,"by":116,"text":117},{"value":163,"source":98,"evidence":164},"InstructorEmbedding",[165,169],{"id":166,"by":167,"text":168},"37134411","villgax","I'd rather stick with InstructEmbedding instead of pandering to flavour of the month LLM. That way I keep my key components insulated from drastic changes",{"id":170,"by":171,"text":172},"37135228","kordlessagain","Here's an example of Instructor Embeddings w\u002FFeatureBase: https:\u002F\u002Fgist.github.com\u002Fkordless\u002Faae99946e7e2a5afccc83f3c4eeee65a Instructor Embedding rank high on various leaderboards…",{"value":174,"source":98,"evidence":175},"Torchserve",[176],{"id":177,"by":178,"text":179},"37133794","rvrs","What's wrong with just using Torchserve[1]? We've been using it to serve embedding models in production. [1] https:\u002F\u002Fpytorch.org\u002Fserve\u002F",{"comments":17,"answered":38,"key":181},[182,185,186],{"id":183,"text":184},"37133786","Thanks for pointing out those models. I see from a quick Huggingface search that the bge model is available in GGML format. You can trivially add new GGML format models to the…",{"id":60,"text":61},{"id":187,"text":188},"37135632","I totally agree when you're talking about a bunch of embeddings at once-- that's why the document level endpoint (and the token-level embedding endpoint) can optionally return a…",{"commenters":190,"max_depth":43,"maker_replies":17,"first_comment_mins":43,"lifespan_hours":191},19,249,[193,195,197,198,200,204,208,211,214,215,218,221,222,226,228,230,232,236,240,242,243,245,248,250,252,253,255,256,258,260,263,265],[115,33,31,194,116,33],"project\u002Fcritical\u002Fidea",[183,33,33,196,10,31],"maker\u002Fpositive\u002Fexecution",[126,33,31,194,127,33],[60,33,33,199,10,31],"project\u002Fneutral\u002Fidea",[201,33,42,202,203,33],"37142988","maker\u002Fneutral\u002Fnone","gsuuon",[205,33,33,206,207,33],"37134082","other\u002Fneutral\u002Fnone","yberreby",[209,33,33,206,210,33],"37134606","thejosh",[212,33,31,194,213,33],"37135479","jayalammar",[119,33,42,199,120,33],[216,33,31,194,217,33],"37136498","jauntbox",[219,31,42,199,220,33],"37135599","daturkel",[187,31,33,196,10,31],[223,31,33,224,225,33],"37135823","other\u002Fneutral\u002Fexecution","zh217",[170,227,33,206,171,33],2,[108,227,33,229,10,31],"maker\u002Fpositive\u002Fidea",[231,227,33,206,171,33],"37136421",[233,42,31,234,235,33],"37267315","project\u002Fcritical\u002Fperformance","skeptrune",[237,238,42,199,239,33],"37235292",4,"doxav",[177,43,31,241,178,33],"project\u002Fcritical\u002Fexecution",[57,43,33,229,10,31],[101,38,43,244,102,33],"project\u002Fpositive\u002Fexecution",[246,38,33,247,10,31],"37134009","project\u002Fpositive\u002Fperformance",[155,38,42,249,102,33],"maker\u002Fneutral\u002Fidea",[251,38,33,249,10,31],"37142614",[142,32,42,249,143,33],[254,32,33,249,10,31],"37136512",[166,34,31,194,167,33],[257,34,31,194,167,33],"37135131",[259,34,33,229,10,31],"37135175",[51,261,33,262,10,31],9,"project\u002Fpositive\u002Fidea",[134,261,42,264,135,33],"maker\u002Fneutral\u002Fperformance",[77,261,33,264,10,31],[267,270],{"id":268,"t":269,"points":238,"comments":33},"37059465",1691563957,{"id":4,"t":11,"points":13,"comments":14},[272,276,281,286,290,295],{"id":273,"name":274,"points":275,"category":19},"37125118","Outlines",854,{"id":277,"name":278,"points":279,"category":280},"37119942","Little Rat",555,"security_privacy",{"id":282,"name":283,"points":284,"category":285},"37083309","OFRAK Tetris",472,"games",{"id":287,"name":288,"points":289,"category":19},"37128293","AI-town",429,{"id":291,"name":292,"points":293,"category":294},"37114141","learndb-py",371,"learning_science",{"id":296,"name":297,"points":298,"category":299},"37075730","Applite",280,"dev_tools",15,false,"v2",{"id":304,"name":5,"line":5,"ai":301,"cat":19,"year":12,"points":13,"comments":14,"rec":28,"s":305,"top":17,"kw":306,"maker":10,"domain":9,"tier":18,"attempts":227,"strip":307,"url":8,"mix":308,"praise":97,"objection":112,"sub":20},37133163,-0.34,[23,24,25,26,27],"10103001313000001310503030110030",[31,32,33,34],1794036640154]