[{"data":1,"prerenderedAt":575},["ShallowReactive",2],{"project-40441945":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":34,"aspects":35,"strip":46,"post_text":48,"what":49,"said":87,"maker_in_thread":183,"shape":193,"grid":197,"attempts":536,"same_week":537,"maker_shows":32,"maker_hidden":567,"reader_version":568,"card":569},"40441945","Unify","Route your prompts to the best LLM","title","Show HN: Route your prompts to the best LLM","https:\u002F\u002Funify.ai\u002Fchat?default=true","unify.ai","danlenton",1716390440,2024,298,126,208,3515,6,"legend","ai","llm routing","web_app",[24,21,25,26],"llm router","model selection","llm benchmarks","curious",0.075,{"positive":30,"neutral":31,"mixed":32,"critical":33},9,26,2,12,49,{"pricing":36,"performance":38,"privacy":39,"idea":41,"docs":44,"execution":45},{"positive":37,"critical":32},0,{"positive":37,"critical":37},{"positive":40,"critical":37},1,{"positive":42,"critical":43},5,7,{"positive":40,"critical":37},{"positive":37,"critical":32},[47,37,37,37,40,37,47,47,42,42,47,42,47,47,47,47,47,47,40,42,42,40,47,37,42,37,40,47,42,40,47,42,40,40,40,40,37,40,47,42],3,"Hey HN, we've just finished building a dynamic router for LLMs, which takes each prompt and sends it to the most appropriate model and provider. We'd love to know what you think!\n\nHere is a quick(ish) screen-recroding explaining how it works: https:\u002F\u002Fyoutu.be\u002FZpY6SIkBosE\n\nBest results when training a custom router on your own prompt data: https:\u002F\u002Fyoutu.be\u002F9JYqNbIEac0\n\nThe router balances user preferences for quality, speed and cost. The end result is higher quality and faster LLM responses at lower cost.\n\nThe quality for each candidate LLM is predicted ahead of time using a neural scoring func",{"problem":50,"audience":56,"pricing":66,"motivation":74,"time_to_build":83,"stack":84,"openness":85,"stage":86,"team_size":85},{"value":51,"source":52,"evidence":53},"Choosing which LLM and provider to use for each prompt balancing quality, speed and cost","maker",[54],{"id":55},"post",{"value":57,"source":58,"evidence":59},"Developers and teams running LLM applications","inferred",[60,63],{"id":61,"by":11,"text":62},"40451627","Agreed, we've spoken to tons of users who reach out to us and start the conversation with \"we've tried to implement this ourselves\".",{"id":64,"by":11,"text":65},"40455363","however, as janekm says, we can't charge just based on cost savings. We would need the router points to be sufficiently compelling wrt quality, speed and cost (including our own…",{"value":67,"source":52,"evidence":68},"Charges same rates as backend providers with no margin; $50 free credits for new signups",[69,70],{"id":55},{"id":71,"by":72,"text":73},"40449633","jefftk","> Pricing-wise, we charge the same rates as the backend providers we route to, without taking any margins. We also give $50 in free credits to all new signups. What's your plan…",{"value":75,"source":52,"evidence":76},"Help apps reduce cost and latency while keeping quality as LLM use scales",[77,80],{"id":78,"by":11,"text":79},"40446698","Great question! Generally the neural network used for the router takes maybe ~20ms during inference. When deployed on prem, in your own cloud environment, then this is the only…",{"id":81,"by":11,"text":82},"40448602","Super helpful feedback, thanks for going so deep! I agree that for the really heavy agentic stuff, the router in it's current form might not be the most important innovation.…",null,[],"unknown","beta",{"praise":88,"objections":108,"requests":132,"questions":149,"alternatives":162},[89,101],{"value":90,"source":91,"evidence":92},"Useful idea for scaling LLM apps and cost control","commenter",[93,97],{"id":94,"by":95,"text":96},"40449956","aleksiy123","I wonder, have you also done any work on actually improving performance by enhancing the prompt or something similar? I always thought a product like this that could empirically…",{"id":98,"by":99,"text":100},"40442440","lovesdogsnsnow","This is interesting! Sort of a super mixture of experts model. What's the latency penalty paid with your router in the middle? The pattern I often see is companies prototyping on…",{"value":102,"source":91,"evidence":103},"Well executed product",[104],{"id":105,"by":106,"text":107},"40453785","billylo","Very well executed. Congrats. Signed up. The chatbot providers will enjoy seeing how they stack up against each other. Your data\u002Fanalytics\u002Falerts will be valuable to them.…",[109,116,125],{"value":110,"source":91,"evidence":111},"Prompt changes across models make treating LLMs as interchangeable bogus",[112],{"id":113,"by":114,"text":115},"40454590","weird-eye-issue","Sorry but I hate stuff like this We spend $30k+ per month with OpenAI and Anthropic Even minor prompt changes between minor model versions can have huge differences in output We…",{"value":117,"source":91,"evidence":118},"No clear business model; no margin taken yet",[119,120,121],{"id":71,"by":72,"text":73},{"id":98,"by":99,"text":100},{"id":122,"by":123,"text":124},"40447723","thomasfromcdnjs","Since you take no extra charges from the end user, the presumption is you would charge LLM's for giving them traffic?",{"value":126,"source":91,"evidence":127},"Router adds latency to time-to-first-token",[128,129],{"id":98,"by":99,"text":100},{"id":130,"by":11,"text":131},"40452908","So the neural scoring introduces ~20ms latency, but this only impacts time-to-first-token (not inter-token-latency). When using our public endpoints there is an additional ~150ms…",[133,143],{"value":134,"source":91,"evidence":135},"Local or on-prem deployment",[136,140],{"id":137,"by":138,"text":139},"40450415","Sontho","It's a good options to test local models I guess.",{"id":141,"by":11,"text":142},"40452249","actually we don't support local deployment quite yet, it's all run in our server. However, fully local deployment is on the roadmap, stay tuned!",{"value":144,"source":52,"evidence":145},"Support for bring-your-own API keys",[146],{"id":147,"by":11,"text":148},"40451610","Another point here is that some users prefer to use their own API keys for the backend providers (a feature we're releasing soon). Any \"discounts\" would then be harder to…",[150,155],{"value":151,"source":91,"evidence":152},"How the company will make money long term",[153,154],{"id":71,"by":72,"text":73},{"id":98,"by":99,"text":100},{"value":156,"source":91,"evidence":157},"Whether using GPT4 as judge for training labels is allowed",[158],{"id":159,"by":160,"text":161},"40447747","iknownthing","> It is trained in a supervised manner on several open LLM datasets, using GPT4 as a judge. Does this mean GPT4 predictions are used as labels? Is that allowed?",[163,169,175],{"value":164,"source":52,"evidence":165},"OpenRouter",[166],{"id":167,"by":11,"text":168},"40448329","definitely similar! I'm a fan of Alex and his work on OpenRouter :) Some of the main differences would be: - we focus on performance based routing, optimizing speed, cost and…",{"value":170,"source":52,"evidence":171},"Semantic router",[172],{"id":173,"by":11,"text":174},"40451664","Thanks for sharing! These are useful toos, but they are a bit different, more based on similarity search in prompt space (a bit like semantic router:…",{"value":176,"source":91,"evidence":177},"Mixture of experts models",[178,182],{"id":179,"by":180,"text":181},"40447161","3abiton","Isn't this what a MoE LLM does already?",{"id":94,"by":95,"text":96},{"comments":184,"answered":31,"key":185},53,[186,187,190],{"id":147,"text":148},{"id":188,"text":189},"40451590","that's a good point, impartiality would then be questioned",{"id":191,"text":192},"40451586","I certainly wouldn't complain about this lol",{"commenters":194,"max_depth":18,"maker_replies":184,"first_comment_mins":195,"lifespan_hours":196},62,44,154,[198,200,204,208,211,212,214,216,217,220,224,227,229,233,235,239,243,245,246,250,254,257,258,260,262,266,269,273,275,278,280,284,287,292,294,297,298,302,305,308,310,314,316,319,321,325,327,329,332,335,337,340,341,345,347,349,352,354,358,361,364,366,368,370,374,376,378,380,383,385,389,392,397,400,402,406,408,412,414,418,419,423,425,430,432,433,436,438,440,444,446,450,453,455,459,463,465,470,472,474,475,478,481,483,485,487,489,491,493,496,499,501,504,506,508,510,512,516,518,522,524,528,530,534],[71,37,47,199,72,37],"maker\u002Fneutral\u002Fpricing",[201,37,37,202,203,37],"40450010","other\u002Fneutral\u002Fpricing","kromem",[205,37,37,206,207,37],"40450132","other\u002Fneutral\u002Fnone","runako",[209,37,37,206,210,37],"40450263","loceng",[147,37,37,199,11,40],[213,37,37,206,210,37],"40467679",[188,37,37,215,11,40],"maker\u002Fneutral\u002Fnone",[191,37,37,199,11,40],[218,37,37,202,219,37],"40453000","namdnay",[221,37,37,222,223,37],"40450598","other\u002Fneutral\u002Fprivacy","cosmojg",[225,37,37,222,226,37],"40451451","ec109685",[228,37,37,199,11,40],"40451576",[230,37,40,231,232,37],"40453352","project\u002Fcritical\u002Fpricing","Aeolun",[234,37,37,199,11,40],"40453809",[236,37,47,237,238,37],"40453998","project\u002Fneutral\u002Fpricing","roynasser",[240,37,37,241,242,37],"40455279","commenter\u002Fneutral\u002Fpricing","janekm",[244,37,37,199,11,40],"40455329",[64,37,37,199,11,40],[247,37,47,248,249,37],"40456672","project\u002Fmixed\u002Fpricing","bitbuilder",[251,40,47,252,253,37],"40447697","project\u002Fneutral\u002Fnone","ianbicking",[255,40,37,206,256,37],"40448024","ankrgyl",[81,40,37,215,11,40],[98,32,47,259,99,37],"project\u002Fneutral\u002Fperformance",[78,32,37,261,11,40],"maker\u002Fneutral\u002Fperformance",[263,32,47,264,265,37],"40450947","project\u002Fneutral\u002Fprivacy","jonahx",[267,32,37,268,11,40],"40452169","maker\u002Fneutral\u002Fprivacy",[270,32,42,271,272,37],"40455788","project\u002Fpositive\u002Fprivacy","ColinHayhurst",[274,32,37,268,11,40],"40457142",[276,32,47,237,277,37],"40450997","nl",[279,32,37,199,11,40],"40451713",[281,47,42,282,283,37],"40447392","project\u002Fpositive\u002Fidea","swiftlyTyped",[285,47,37,286,11,40],"40447478","maker\u002Fpositive\u002Fidea",[288,289,47,290,291,37],"40452901",4,"project\u002Fneutral\u002Fidea","saberience",[293,289,37,261,11,40],"40452959",[295,42,47,252,296,37],"40449984","ceroxylon",[173,42,37,215,11,40],[299,18,42,300,301,37],"40454098","project\u002Fpositive\u002Fdocs","BrutalCoding",[303,18,37,304,11,40],"40457060","maker\u002Fpositive\u002Fdocs",[306,43,47,252,307,37],"40452950","Oras",[309,43,37,261,11,40],"40452996",[311,312,47,290,313,37],"40451852",8,"mzl",[315,312,37,261,11,40],"40451969",[317,30,47,290,318,37],"40452807","Animats",[320,30,37,215,11,40],"40452863",[322,323,47,290,324,37],"40447209",10,"joaquincabezas",[326,323,37,215,11,40],"40448274",[94,328,47,290,95,37],11,[330,328,37,206,331,37],"40450043","ramoz",[333,328,37,334,95,37],"40450055","commenter\u002Fneutral\u002Fnone",[336,328,37,215,11,40],"40451641",[338,33,47,259,339,37],"40452590","treprinum",[130,33,37,261,11,40],[342,343,37,206,344,37],"40465486",13,"aravindputrevu",[159,346,47,290,160,37],14,[348,346,37,215,11,40],"40447825",[350,346,40,351,160,37],"40448496","project\u002Fcritical\u002Fnone",[353,346,37,215,11,40],"40448635",[355,356,47,290,357,37],"40447261",15,"andrewstetsenko",[359,356,37,360,11,40],"40447884","maker\u002Fneutral\u002Fdocs",[105,362,42,363,106,37],16,"project\u002Fpositive\u002Fnone",[365,362,37,334,11,40],"40453842",[367,362,37,334,106,37],"40454156",[369,362,37,334,11,40],"40455400",[371,372,42,282,373,37],"40447448",17,"siquick",[375,372,37,363,11,40],"40447495",[122,377,47,237,123,37],18,[379,377,37,237,11,40],"40447864",[381,377,40,231,382,37],"40451095","t-writescode",[384,377,37,215,11,40],"40451793",[386,387,47,290,388,37],"40448757",19,"memothon",[390,387,37,391,11,40],"40452239","project\u002Fneutral\u002Fexecution",[393,394,37,395,396,37],"40447357",20,"other\u002Fneutral\u002Fidea","tyrw",[398,394,37,206,399,37],"40447722","darreninthenet",[401,394,37,206,11,40],"40447793",[403,404,47,391,405,37],"40462216",21,"metalwhale",[407,404,37,391,11,40],"40507050",[409,410,42,282,411,37],"40449863",22,"potatoman22",[413,410,37,206,11,40],"40451880",[415,416,37,206,417,37],"40451352",23,"_pdp_",[61,416,37,334,11,40],[420,421,47,290,422,37],"40452266",24,"t_mann",[424,421,37,290,11,40],"40452936",[426,427,40,428,429,37],"40447570",25,"project\u002Fcritical\u002Fexecution","xena",[431,427,37,215,11,40],"40447915",[179,31,47,290,180,37],[434,31,47,290,435,37],"40447319","sodality2",[437,31,37,334,11,40],"40448348",[439,31,37,290,11,40],"40447949",[441,442,42,363,443,37],"40452782",27,"santiagobasulto",[445,442,37,215,11,40],"40452847",[447,448,40,428,449,37],"40455392",28,"ModernMech",[451,448,37,334,452,37],"40455539","frontman1988",[454,448,37,334,449,37],"40456319",[456,457,37,206,458,37],"40456795",29,"Havoc",[460,461,47,290,462,37],"40450249",30,"dheera",[464,461,37,290,11,40],"40451828",[466,467,47,468,469,37],"40450539",31,"project\u002Fmixed\u002Fidea","swyx",[471,467,37,290,11,40],"40452198",[137,473,42,282,138,37],32,[141,473,37,252,11,40],[113,476,40,477,114,37],33,"project\u002Fcritical\u002Fidea",[479,476,40,477,480,37],"40454663","qeternity",[482,476,40,477,114,37],"40455207",[484,476,37,334,11,40],"40457231",[486,476,40,477,114,37],"40464133",[488,476,37,290,11,40],"40464726",[490,476,37,290,11,40],"40457110",[492,476,40,477,480,37],"40469591",[494,476,37,206,495,37],"40500143","tarasglek",[497,476,37,334,498,37],"40454945","thomashop",[500,476,40,477,114,37],"40455151",[502,476,37,334,503,37],"40455282","dilek",[505,476,37,215,11,40],"40457002",[507,476,37,290,11,40],"40455575",[509,476,40,477,114,37],"40456404",[511,476,37,290,11,40],"40456978",[513,514,47,252,515,37],"40450182",34,"cchance",[517,514,37,252,11,40],"40451847",[519,520,47,252,521,37],"40453143",35,"starwin1159",[523,520,37,252,11,40],"40453288",[525,526,42,282,527,37],"40451072",36,"lhousa",[529,526,37,215,11,40],"40451843",[531,532,47,290,533,37],"40448160",37,"Alifatisk",[167,532,37,535,11,40],"maker\u002Fneutral\u002Fidea",[],[538,543,548,553,558,562],{"id":539,"name":540,"points":541,"category":542},"40431126","Pls Fix",444,"web_saas",{"id":544,"name":545,"points":546,"category":547},"40429200","graphgame.sabrina.dev",321,"games",{"id":549,"name":550,"points":551,"category":552},"40392548","EquityVal",287,"finance_crypto",{"id":554,"name":555,"points":556,"category":557},"40443927","Neosync",246,"data",{"id":559,"name":560,"points":561,"category":557},"40432213","Openpanel",244,{"id":563,"name":564,"points":565,"category":566},"40441093","PBT",196,"dev_tools",false,"v2",{"id":570,"name":5,"line":6,"ai":567,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":571,"top":18,"kw":572,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":573,"url":9,"mix":574,"praise":90,"objection":110,"sub":21},40441945,0.08,[24,21,25,26],"3000103355353333331551305013513511110135",[30,31,32,33],1794036655166]