[{"data":1,"prerenderedAt":236},["ShallowReactive",2],{"project-42339302":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":27,"sentiment":28,"stance_mix":29,"scored_comments":33,"aspects":34,"strip":39,"post_text":41,"what":42,"said":74,"maker_in_thread":139,"shape":145,"grid":149,"attempts":198,"same_week":199,"maker_shows":140,"maker_hidden":228,"reader_version":229,"card":230},"42339302","Prompt Engine","Auto pick LLMs based on your prompts","title","Show HN: Prompt Engine – Auto pick LLMs based on your prompts","https:\u002F\u002Fjigsawstack.com\u002Fblog\u002Fjigsawstack-mixture-of-agents-moa-outperform-any-single-llm-and-reduce-cost-with-prompt-engine","jigsawstack.com","yoeven",1733489148,2024,93,21,886,3515,26,"notable","ai","llm routing","api",[21,24,25,26],"model selection","prompt optimization","llm api","curious",-0.222,{"positive":30,"neutral":31,"mixed":30,"critical":32},0,4,2,6,{"idea":35,"execution":37,"docs":38},{"positive":30,"critical":36},1,{"positive":30,"critical":36},{"positive":30,"critical":30},[36,40,40,30,30,40,30,40,30,36,30,30,30,30,30,30,30,30,30,30,30],3,"Nowadays, a common AI tech stack has hundreds of different prompts running across different LLMs.\n\nThree key problems:\n\n- Choices, picking from 100s of LLMs the best LLM for that 1 prompt is gonna be challenging, you're probably not picking the most optimized LLM for a prompt you wrote.\n\n- Scaling\u002FUpgrading, similar to choices but you want to keep consistency of your output even when models depreciate or configurations change.\n\n- Prompt management is scary, if something works, you'll never want to touch it but you should be able to without fear of everything breaking.\n\nSo we launched Prompt En",{"problem":43,"audience":49,"pricing":53,"motivation":54,"time_to_build":53,"stack":58,"openness":72,"stage":73,"team_size":72},{"value":44,"source":45,"evidence":46},"Picking the best LLM for each prompt and keeping outputs consistent as models change","maker",[47],{"id":48},"post",{"value":50,"source":45,"evidence":51},"Teams running hundreds of prompts across different LLMs",[52],{"id":48},null,{"value":55,"source":45,"evidence":56},"Reduce choice, upgrade and prompt management pain across LLM stacks",[57],{"id":48},[59,63],{"value":60,"source":45,"evidence":61},"Small trained model that picks an LLM per prompt",[62],{"id":48},{"value":64,"source":45,"evidence":65},"Runs prompts across 5-6 models and ranks outputs by similarity",[66,69],{"id":67,"by":11,"text":68},"42357426","Basically ^ - Prompt style is starting to standardize across models. Something we see happening more and more. - There are keyword triggers and prompt styles that perform better…",{"id":70,"by":11,"text":71},"42357497","We run tons of evals with public datasets from LLM Arena on common categories such as law, financing task, code, maths, etc., and pair them up with the public benchmarks such as…","unknown","launched",{"praise":75,"objections":83,"requests":106,"questions":121,"alternatives":133},[76],{"value":77,"source":78,"evidence":79},"Makers responded to criticism with detailed answers","commenter",[80],{"id":81,"by":11,"text":82},"42357391","Hey swyk! I have a similar perspective when it comes to model routing and even LLM frameworks that centralize the schema when almost every provider already supports OpenAI's…",[84,91,95,99],{"value":85,"source":78,"evidence":86},"Model routing is unnecessary because people tune prompts to specific models",[87],{"id":88,"by":89,"text":90},"42340287","swyx","congrats yoeven! i have been a skeptic of model routing like Martian, because while it sounds good in theory, 1) in practice, people tune prompts\u002Fbehavior\u002Foutput to models and…",{"value":92,"source":78,"evidence":93},"Router must be as smart as the strongest model, making it pointless",[94],{"id":88,"by":89,"text":90},{"value":96,"source":78,"evidence":97},"Labs will offer good-enough routing themselves",[98],{"id":88,"by":89,"text":90},{"value":100,"source":78,"evidence":101},"Homepage crashes with a client-side exception",[102],{"id":103,"by":104,"text":105},"42346654","bosky101","It's a pity your homepage crashes and replaces all content with the text ``` Application error: a client-side exception has occurred (see the browser console for more…",[107,114],{"value":108,"source":78,"evidence":109},"Explain how the training dataset was annotated",[110],{"id":111,"by":112,"text":113},"42340142","ankit219","The small model you trained, how did you annotate the dataset? Because most of the times output from big models will be subjective, and with not a drastic difference in quality.…",{"value":115,"source":78,"evidence":116},"Publish the list of models used for training",[117],{"id":118,"by":119,"text":120},"42340067","soco","I can't find that list of hundreds of models to save my life... is it there some place?",[122,129],{"value":123,"source":78,"evidence":124},"How are prompt differences across model families handled",[125],{"id":126,"by":127,"text":128},"42340337","htrp","I'd also ask how you're dealing with the idiosyncrasies across model families (its very different to prompt gemini vs claude vs gpt4o) when you are routing these lm inputs",{"value":130,"source":78,"evidence":131},"How the router was trained and annotated",[132],{"id":111,"by":112,"text":113},[134],{"value":135,"source":78,"evidence":136},"Martian and other LLM routers",[137,138],{"id":88,"by":89,"text":90},{"id":81,"by":11,"text":82},{"comments":140,"answered":32,"key":141},5,[142,143,144],{"id":67,"text":68},{"id":81,"text":82},{"id":70,"text":71},{"commenters":146,"max_depth":140,"maker_replies":140,"first_comment_mins":147,"lifespan_hours":148},11,98,50,[150,152,154,156,157,159,160,161,163,165,167,170,174,178,181,184,186,188,191,194,196],[88,30,36,151,89,30],"project\u002Fcritical\u002Fidea",[126,30,40,153,127,30],"project\u002Fneutral\u002Fexecution",[155,30,40,153,89,30],"42340349",[67,30,30,153,11,36],[81,30,30,158,11,36],"project\u002Fneutral\u002Fidea",[111,36,40,153,112,30],[70,36,30,153,11,36],[118,32,40,162,119,30],"project\u002Fneutral\u002Fdocs",[164,32,30,162,11,36],"42357526",[103,40,36,166,104,30],"project\u002Fcritical\u002Fexecution",[168,40,30,169,11,36],"42357506","maker\u002Fneutral\u002Fnone",[171,31,30,172,173,30],"42340386","other\u002Fneutral\u002Fnone","mg",[175,31,30,176,177,30],"42340725","commenter\u002Fneutral\u002Fnone","Kiro",[179,31,30,176,180,30],"42340887","throw2342412314",[182,31,30,172,183,30],"42340957","denuoweb",[185,31,30,176,173,30],"42341236",[187,31,30,176,183,30],"42341415",[189,31,30,172,190,30],"42340785","fragmede",[192,31,30,193,173,30],"42349205","other\u002Fcritical\u002Fnone",[195,31,30,176,190,30],"42351613",[197,31,30,172,173,30],"42355833",[],[200,204,208,213,218,223],{"id":201,"name":202,"points":203,"category":20},"42348513","Countless.dev",361,{"id":205,"name":206,"points":207,"category":20},"42356443","Cut the crap",356,{"id":209,"name":210,"points":211,"category":212},"42331270","Banan-OS",333,"infrastructure",{"id":214,"name":215,"points":216,"category":217},"42338327","Hacker Herald",251,"consumer_social",{"id":219,"name":220,"points":221,"category":222},"42329071","gitingest",185,"dev_tools",{"id":224,"name":225,"points":226,"category":227},"42359949","salmagundi",176,"languages",false,"v2",{"id":231,"name":5,"line":6,"ai":228,"cat":20,"year":13,"points":14,"comments":15,"rec":27,"s":232,"top":18,"kw":233,"maker":11,"domain":10,"tier":19,"attempts":30,"strip":234,"url":9,"mix":235,"praise":77,"objection":85,"sub":21},42339302,-0.22,[21,24,25,26],"133003030100000000000",[30,31,30,32],1793991106804]