[{"data":1,"prerenderedAt":254},["ShallowReactive",2],{"project-36958175":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":40,"post_text":42,"what":43,"said":76,"maker_in_thread":148,"shape":160,"grid":164,"attempts":216,"same_week":217,"maker_shows":177,"maker_hidden":246,"reader_version":247,"card":248},"36958175","PromptTools","open-source tools for evaluating LLMs and vector DBs","title","Show HN: PromptTools – open-source tools for evaluating LLMs and vector DBs","https:\u002F\u002Fgithub.com\u002Fhegelai\u002Fprompttools","github.com","krawfy",1690906995,2023,211,24,264,3109,9,"notable","dev_tools","llm evaluation testing","library",[24,25,26,27],"llm evaluation","prompt testing","vector database","llm benchmarking","loved",0.345,{"positive":31,"neutral":32,"mixed":33,"critical":34},5,6,2,0,13,{"idea":37,"execution":38,"design":39},{"positive":33,"critical":34},{"positive":34,"critical":34},{"positive":34,"critical":34},[41,41,34,34,34,34,41,31,34,41,34,41,34,31,34,41,34,31,34,31,41,31,41],3,"Hey HN! We’re Kevin and Steve. We’re building PromptTools (https:\u002F\u002Fgithub.com\u002Fhegelai\u002Fprompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts.\n\nEvaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time.\n\nBy “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples:\n- For generated JSON, SQL, or Python, ",{"problem":44,"audience":50,"pricing":55,"motivation":56,"time_to_build":55,"stack":60,"openness":73,"stage":74,"team_size":75},{"value":45,"source":46,"evidence":47},"Evaluating prompts, LLMs and vector databases is slow and painful","maker",[48],{"id":49},"post",{"value":51,"source":52,"evidence":53},"Engineers building products on LLMs","inferred",[54],{"id":49},null,{"value":57,"source":46,"evidence":58},"Makers found it hard to discover and evaluate open-source models after working with HuggingFace at Google",[59],{"id":49},[61,65,69],{"value":62,"source":46,"evidence":63},"Python",[64],{"id":49},{"value":66,"source":46,"evidence":67},"Jupyter notebook",[68],{"id":49},{"value":70,"source":46,"evidence":71},"Streamlit",[72],{"id":49},"open_source","beta","unknown",{"praise":77,"objections":94,"requests":105,"questions":127,"alternatives":139},[78],{"value":79,"source":80,"evidence":81},"Useful tooling for the LLM evaluation problem","commenter",[82,86,90],{"id":83,"by":84,"text":85},"36958802","catlover76","Super cool, the need for tooling like this is something one realizes pretty quickly when starting to build apps that leverage LLMs.",{"id":87,"by":88,"text":89},"36959289","neelm","Something like this is going to be needed to evaluate models effectively. Evaluation should be integrated into automated pipelines\u002Fworkflows that can scale across models and…",{"id":91,"by":92,"text":93},"36960853","tikkun","This looks great, thanks See also this related tool: https:\u002F\u002Fnews.ycombinator.com\u002Fitem?id=36907074",[95],{"value":96,"source":80,"evidence":97},"LLM auto-evaluators risk over-trust and circular scoring",[98,102],{"id":99,"by":100,"text":101},"36959046","fatso784","I like the support for Vector DBs and LLaMa-2. I'm curious as to whether and what influences compelled PromptTools, and how it differs from other tools in this space. For…",{"id":103,"by":100,"text":104},"36959384","What does 'top 50%' responses mean here, though? You'd need to have a ground truth of how 'good' each score was to calculate that --and if you had ground truth, no need to use an…",[106,113,120],{"value":107,"source":80,"evidence":108},"Add Qdrant support",[109],{"id":110,"by":111,"text":112},"36959491","esafak","I'd like to see support for qdrant.",{"value":114,"source":80,"evidence":115},"Add OpenAI function calling support",[116],{"id":117,"by":118,"text":119},"36964389","mmaia","I like that it's not limited to single prompts and allows to have chat messages. It would be great if `OpenAIChatExperiment` could also handle OpenAI's function calling.",{"value":121,"source":80,"evidence":122},"Provide a Dockerfile",[123],{"id":124,"by":125,"text":126},"36966117","robszumski","I'll put in a friendly request for a Dockerfile in the repo. I've been trying out AI tools as test cases for our supply chain security platform and had to cobble a Dockerfile…",[128,132],{"value":129,"source":80,"evidence":130},"How does it differ from other tools and what influenced it?",[131],{"id":99,"by":100,"text":101},{"value":133,"source":80,"evidence":134},"Where do thumbs up\u002Fdown feedback values go and are they persisted?",[135],{"id":136,"by":137,"text":138},"36958806","politelemon","Similar tool I was about to look at: https:\u002F\u002Fgithub.com\u002Fpromptfoo\u002Fpromptfoo I've seen this in both tools but I wasn't able to understand: In the screenshot with feedback, I see…",[140,144],{"value":141,"source":80,"evidence":142},"ChainForge",[143],{"id":99,"by":100,"text":101},{"value":145,"source":80,"evidence":146},"promptfoo",[147],{"id":136,"by":137,"text":138},{"comments":149,"answered":41,"key":150},7,[151,154,157],{"id":152,"text":153},"36959297","Great question, chainforge looks interesting! We offer auto-evals as one tool in the toolbox. We also consider structured output validations, semantic similarity to an expected…",{"id":155,"text":156},"36958880","Glad you think so, we agree! If you end up trying it out, we'd love to hear what you think, and what other features you'd like to see.",{"id":158,"text":159},"36959558","We've actually been in contact with the qdrant team about adding it to our roadmap! Andre (CEO) was asking for an integration. If you want to work on the PR, we'd be happy to…",{"commenters":35,"max_depth":161,"maker_replies":149,"first_comment_mins":162,"lifespan_hours":163},4,38,10,[165,167,171,173,176,178,180,182,184,186,188,190,192,194,195,197,198,200,202,204,208,211,214],[99,34,41,166,100,34],"project\u002Fmixed\u002Fidea",[168,34,41,169,170,34],"36959291","project\u002Fneutral\u002Fidea","hashemalsaket",[103,34,34,172,100,34],"commenter\u002Fcritical\u002Fidea",[174,34,34,175,170,34],"36959640","commenter\u002Fneutral\u002Fidea",[152,34,34,175,11,177],1,[179,34,34,175,100,34],"36959474",[124,177,41,181,125,34],"project\u002Fmixed\u002Fexecution",[83,33,31,183,84,34],"project\u002Fpositive\u002Fidea",[155,33,34,185,11,177],"maker\u002Fneutral\u002Fnone",[110,41,41,187,111,34],"project\u002Fneutral\u002Fexecution",[158,41,34,189,11,177],"maker\u002Fneutral\u002Fexecution",[136,161,41,191,137,34],"project\u002Fneutral\u002Fdesign",[193,161,34,191,11,177],"36958870",[87,31,31,183,88,34],[196,31,34,185,11,177],"36963921",[117,32,41,187,118,34],[199,32,34,189,11,177],"36964480",[91,149,31,201,92,34],"project\u002Fpositive\u002Fnone",[203,149,34,185,11,177],"36963911",[205,206,31,201,207,34],"36961888",8,"8awake",[209,206,41,185,210,34],"36963604","nivekt",[212,18,31,201,213,34],"36961077","pk19238",[215,18,41,185,210,34],"36964115",[],[218,223,228,233,238,242],{"id":219,"name":220,"points":221,"category":222},"36933603","San Francisco Compute",727,"infrastructure",{"id":224,"name":225,"points":226,"category":227},"36933452","Khoj",565,"ai",{"id":229,"name":230,"points":231,"category":232},"36944152","Markwhen",327,"productivity",{"id":234,"name":235,"points":236,"category":237},"36973400","Pretty Polly",325,"learning_science",{"id":239,"name":240,"points":241,"category":232},"36942308","Linkwarden",280,{"id":243,"name":244,"points":245,"category":20},"36958731","Magic Loops",248,false,"v2",{"id":249,"name":5,"line":6,"ai":246,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":250,"top":18,"kw":251,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":252,"url":9,"mix":253,"praise":79,"objection":96,"sub":21},36958175,0.35,[24,25,26,27],"33000035030305030505353",[31,32,33,34],1794036639311]