[{"data":1,"prerenderedAt":252},["ShallowReactive",2],{"project-41567192":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":36,"aspects":37,"strip":41,"post_text":43,"what":44,"said":90,"maker_in_thread":172,"shape":182,"grid":186,"attempts":215,"same_week":216,"maker_shows":36,"maker_hidden":244,"reader_version":245,"card":246},"41567192","Opik","An open source LLM evaluation framework","title","Show HN: Opik, an open source LLM evaluation framework","https:\u002F\u002Fgithub.com\u002Fcomet-ml\u002Fopik","github.com","calebkaiser",1726578104,2024,86,15,931,3515,27,"notable","ai","llm evaluation framework","library",[24,25,26,27,28],"llm evaluation","llm observability","llm tracing","model testing","pytest llm","curious",0.222,{"positive":32,"neutral":33,"mixed":34,"critical":35},2,3,1,0,6,{"execution":38,"idea":39,"design":40},{"positive":35,"critical":35},{"positive":32,"critical":35},{"positive":35,"critical":35},[33,42,35,42,35,35,33,35,33,35,35,35,35,33,35],5,"Hey HN! I'm Caleb, one of the contributors to Opik, a new open source framework for LLM evaluations.\n\nOver the last few months, my colleagues and I have been working on a project to solve what we see as the most painful parts of writing evals for an LLM application. For this initial release, we've focused on a few core features that we think are the most essential:\n\n- Simplifying the implementation of more complex LLM-based evaluation metrics, like Hallucination and Moderation.\n\n- Enabling step-by-step tracking, such that you can test and debug each individual component of your LLM application",{"problem":45,"audience":51,"pricing":56,"motivation":63,"time_to_build":67,"stack":68,"openness":87,"stage":88,"team_size":89},{"value":46,"source":47,"evidence":48},"Hard to write evals and trust LLM applications in production; opaque application behavior","maker",[49],{"id":50},"post",{"value":52,"source":53,"evidence":54},"Developers building LLM applications","inferred",[55],{"id":50},{"value":57,"source":47,"evidence":58},"Free open source version; free API key for cloud platform; Comet Artifacts need a free Comet account",[59,60],{"id":50},{"id":61,"by":11,"text":62},"41571993","Great question. First, we have a ton of respect for the work the DeepEval team is doing. That said, we took a fundamentally different approach in building Opik as an open source…",{"value":64,"source":47,"evidence":65},"Make evals for LLM applications less painful and improve trust in production LLM apps",[66],{"id":50},null,[69,75,79,83],{"value":70,"source":47,"evidence":71},"Python SDK",[72],{"id":73,"by":11,"text":74},"41580261","The decision to go with Java for the backend was because we feel Java is a bit more battle tested than Python for production (dependency management, concurrency, compilation…",{"value":76,"source":47,"evidence":77},"Java backend",[78],{"id":73,"by":11,"text":74},{"value":80,"source":47,"evidence":81},"TypeScript frontend",[82],{"id":73,"by":11,"text":74},{"value":84,"source":47,"evidence":85},"OpenAI Python library integration",[86],{"id":50},"open_source","launched","team",{"praise":91,"objections":107,"requests":115,"questions":126,"alternatives":152},[92,100],{"value":93,"source":94,"evidence":95},"Promising tool; useful for local experimentation with Ollama","commenter",[96],{"id":97,"by":98,"text":99},"41567374","tcsizmadia","It looks very promising! Congratulations, great tool! I can't wait to start experimenting with it. I plan to use it locally, with Ollama.",{"value":101,"source":94,"evidence":102},"Useful for university AI testing and QA course",[103],{"id":104,"by":105,"text":106},"41579948","trolan","I'm in a University course related to AI testing and quality assurance. This is something I'll definitely bring up and see how it can be used. With OpenAI comparability, hoping…",[108],{"value":109,"source":94,"evidence":110},"Colab ollama example threw a connection error on cell 2",[111],{"id":112,"by":113,"text":114},"41597605","itssteadyfreddy","Ran through some colabs. Signed up for a key and tested the ollama colab. Got a little error on cell 2, \"ConnectError: [Errno 99] Cannot assign requested address\" but the traces…",[116,122],{"value":117,"source":47,"evidence":118},"OpenTelemetry compatible ingestion endpoints",[119],{"id":120,"by":11,"text":121},"41568812","Good question! It mostly came down to implementation speed, as well as some uncertainty about performance\u002Foverhead. We will be releasing OpenTelemetry compatible ingestion…",{"value":123,"source":94,"evidence":124},"Out-of-the-box OpenRouter support",[125],{"id":104,"by":105,"text":106},[127,131,138,145],{"value":128,"source":94,"evidence":129},"How does it compare to Arize Phoenix?",[130],{"id":112,"by":113,"text":114},{"value":132,"source":94,"evidence":133},"Why Java for the backend instead of Python?",[134],{"id":135,"by":136,"text":137},"41575655","smcleod","Looks interesting, great to see it specifically calls out supporting LLM servers as first class citizens! I see some of the code is Java, that strikes me as an interesting choice…",{"value":139,"source":94,"evidence":140},"Why not implement OpenTelemetry directly?",[141],{"id":142,"by":143,"text":144},"41568516","hrpnk","Is there a reason you didn't just implement OpenTelemetry (OT) straight away? Curious about the trade offs to opt for a home-grown telemetry inspired by OT instead.",{"value":146,"source":94,"evidence":147},"How does it compare to DeepEval?",[148],{"id":149,"by":150,"text":151},"41571313","yu3zhou4","Hello! How does it compare to DeepEval (open source)?",[153,157,161,168],{"value":154,"source":94,"evidence":155},"DeepEval",[156],{"id":149,"by":150,"text":151},{"value":158,"source":94,"evidence":159},"Arize Phoenix",[160],{"id":112,"by":113,"text":114},{"value":162,"source":94,"evidence":163},"OpenLLMetry",[164],{"id":165,"by":166,"text":167},"41571151","baggiponte","Have you seen two “prototypes” of standard for LLM telemetry? One is openllmetry, maintained by the folks at TraceLoop. Seems the more popular. The other one is openinference…",{"value":169,"source":94,"evidence":170},"OpenInference",[171],{"id":165,"by":166,"text":167},{"comments":36,"answered":173,"key":174},4,[175,178,181],{"id":176,"text":177},"41567413","Awesome, thanks! If you run into any issues, you can open a ticket on the repo or ping me directly at caleb[at]comet.com",{"id":179,"text":180},"41580053","Fantastic to hear! Opik should work with OpenRouter out of the box, particularly if you are using the OpenAI Python client to interface with OpenRouter. Opik's integration with…",{"id":73,"text":74},{"commenters":183,"max_depth":173,"maker_replies":36,"first_comment_mins":184,"lifespan_hours":185},10,23,82,[187,189,191,193,194,195,199,201,202,204,205,207,209,212,214],[112,35,33,188,113,35],"project\u002Fmixed\u002Fexecution",[97,34,42,190,98,35],"project\u002Fpositive\u002Fidea",[176,34,35,192,11,34],"maker\u002Fneutral\u002Fnone",[104,32,42,190,105,35],[179,32,35,190,11,34],[196,32,35,197,198,35],"41606316","commenter\u002Fneutral\u002Fnone","pablomendes",[135,33,33,200,136,35],"project\u002Fneutral\u002Fnone",[73,33,35,200,11,34],[142,173,33,203,143,35],"project\u002Fneutral\u002Fdesign",[120,173,35,203,11,34],[165,173,35,206,166,35],"other\u002Fneutral\u002Fnone",[208,173,35,206,11,34],"41571248",[210,173,35,206,211,35],"41571281","kakaly0403",[149,42,33,213,150,35],"project\u002Fneutral\u002Fidea",[61,42,35,190,11,34],[],[217,222,227,232,236,240],{"id":218,"name":219,"points":220,"category":221},"41521919","FixHub",928,"hardware",{"id":223,"name":224,"points":225,"category":226},"41539125","Meet.hn",768,"consumer_social",{"id":228,"name":229,"points":230,"category":231},"41517312","Konty",392,"dev_tools",{"id":233,"name":234,"points":235,"category":20},"41544969","Wordllama",370,{"id":237,"name":238,"points":239,"category":231},"41563958","Void",347,{"id":241,"name":242,"points":243,"category":231},"41530190","FlowTracker",280,false,"v2",{"id":247,"name":5,"line":6,"ai":244,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":248,"top":18,"kw":249,"maker":11,"domain":10,"tier":19,"attempts":35,"strip":250,"url":9,"mix":251,"praise":93,"objection":109,"sub":21},41567192,0.22,[24,25,26,27,28],"350500303000030",[32,33,34,35],1794036661115]