[{"data":1,"prerenderedAt":272},["ShallowReactive",2],{"project-37038053":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":35,"aspects":36,"strip":41,"post_text":44,"what":45,"said":72,"maker_in_thread":166,"shape":174,"grid":178,"attempts":234,"same_week":235,"maker_shows":33,"maker_hidden":264,"reader_version":265,"card":266},"37038053","ChainForge","A visual tool for prompt engineering and LLM evaluation","title","Show HN: ChainForge, a visual tool for prompt engineering and LLM evaluation","https:\u002F\u002Fchainforge.ai\u002Fdocs\u002F","chainforge.ai","fatso784",1691430872,2023,177,29,331,3109,11,"notable","ai","llm evaluation tool","web_app",[24,25,26,27],"prompt engineering","llm evaluation","prompt testing","model comparison","loved",0.327,{"positive":31,"neutral":32,"mixed":33,"critical":34},7,10,1,0,18,{"idea":37,"docs":39,"design":40},{"positive":38,"critical":34},2,{"positive":34,"critical":34},{"positive":34,"critical":34},[42,43,43,34,43,42,42,34,42,43,43,43,34,34,34,34,34,43,34,42,43,34,42,34,43,43,43,42,34],5,3,"Hi HN! We’re been working hard on this low-code tool for rapid prompt discovery, robustness testing and LLM evaluation. We’ve just released documentation to help new users learn how to use it and what it can already do. Let us know what you think! :)",{"problem":46,"audience":55,"pricing":59,"motivation":60,"time_to_build":59,"stack":68,"openness":69,"stage":70,"team_size":71},{"value":47,"source":48,"evidence":49},"Finding the right prompt and evaluating and inspecting LLM outputs across prompts and models","maker",[50,52],{"id":51},"post",{"id":53,"by":11,"text":54},"37096555","We don’t view it as a competition here. These other tools are for LLM app building, which is great, but isn’t the focus of ChainForge. Instead, we’re focused on helping devs find…",{"value":56,"source":48,"evidence":57},"Developers and researchers who want to evaluate or audit LLMs",[58],{"id":53,"by":11,"text":54},null,{"value":61,"source":62,"evidence":63},"Academic project; a citation helps visibility and grant funding","commenter",[64],{"id":65,"by":66,"text":67},"37040744","priyanmuthu","Hi! I’m one of the great students working on this. This is merely a request to get more visibility. It will also help us get more grants. We don’t have any intention of…",[],"open_source","launched","team",{"praise":73,"objections":85,"requests":110,"questions":131,"alternatives":150},[74],{"value":75,"source":62,"evidence":76},"Useful for comparing outputs across models and parameterised queries",[77,81],{"id":78,"by":79,"text":80},"37038600","_puk","I think you should probably mention that its source is available! [0] I don't personally have a need for this right now, but I can really see the use for the parameterised…",{"id":82,"by":83,"text":84},"37039156","ericskiff","We just used this on a project and it was very helpful! Cool to see it here on HN",[86,100],{"value":87,"source":62,"evidence":88},"Licence and citation request considered unclear by some commenters",[89,93,96],{"id":90,"by":91,"text":92},"37039504","swyx","\"source available\" means one thing, but this is properly MIT open source and i believe the authors should receive credit for that in this age of frankenlicenses EDIT: ah: \"This…",{"id":94,"by":79,"text":95},"37040093","I didn't want to open that can of worms: \"If you use ChainForge for research purposes, or build upon the source code, we ask that you cite this project in any related…",{"id":97,"by":98,"text":99},"37040227","7moritz7","It says \"we ask\", not \"you have to\". I'd say it's open to interpretation that this is an informal request and not legally binding. Also I do want to open the can of worms and say…",{"value":101,"source":62,"evidence":102},"Less community activity than Flowise and langflow",[103,107],{"id":104,"by":105,"text":106},"37041379","sdesol","I can't comment on the features, but ChainForge has some catching up to do...mind share wise. Below are some community insights for langflow, Flowise and ChainForge…",{"id":108,"by":105,"text":109},"37042663","I sort of know where you are coming from, but I think it is hard to discount the fact that 80 new people took the time to create 99 issues in the last 6 weeks in Flowise. No…",[111,121],{"value":112,"source":62,"evidence":113},"Google Vertex AI support",[114,118],{"id":115,"by":116,"text":117},"37039999","koryk","I like it! Any plans to add Google Vertex AI support?",{"id":119,"by":116,"text":120},"37108870","I meant for completion, chat and embedding. https:\u002F\u002Fcloud.google.com\u002Fvertex-ai\u002Fdocs\u002Fgenerative-ai\u002Fchat\u002Ftest-chat-prompts#chat-query-drest . Some examples here. Vertex AI has the…",{"value":122,"source":62,"evidence":123},"Function call evaluation",[124,128],{"id":125,"by":126,"text":127},"37042065","mabcat","This looks excellent! It's a great interface for two things I'm struggling to make LlamaIndex do: explain and debug multi-step responses for agent flows, and cache queries…",{"id":129,"by":11,"text":130},"37096589","Thank you for the kind words! Looking at the photo, I think you wouldn’t need the last prompt node there. As far as evaluating functions go, that’s unfortunately a ways off. But,…",[132,143],{"value":133,"source":62,"evidence":134},"What prompt discovery means",[135,139],{"id":136,"by":137,"text":138},"37041645","maxlamb","What exactly is prompt discovery?",{"id":140,"by":141,"text":142},"37042078","toxicFork","AFAIK it's finding out what prompts to use for what LLM to get the answer your want E.g. this > Compare response quality across prompt permutations, across models, and across…",{"value":144,"source":62,"evidence":145},"How the tool was useful in practice",[146],{"id":147,"by":148,"text":149},"37041429","dekervin","May I ask, how was it useful ? I find it cool but I have a hard time justifying using it.",[151,158,162],{"value":152,"source":62,"evidence":153},"langflow",[154],{"id":155,"by":156,"text":157},"37039849","KRAKRISMOTT","It seems to be more powerful than langflow and flowise https:\u002F\u002Fgithub.com\u002Flogspace-ai\u002Flangflow https:\u002F\u002Fgithub.com\u002FFlowiseAI\u002FFlowise",{"value":159,"source":62,"evidence":160},"Flowise",[161],{"id":155,"by":156,"text":157},{"value":163,"source":62,"evidence":164},"LlamaIndex",[165],{"id":125,"by":126,"text":127},{"comments":167,"answered":33,"key":168},6,[169,170,173],{"id":53,"text":54},{"id":171,"text":172},"37096509","ChainForge looks similar visually, but is very different in practice. We target evaluating and inspecting LLM outputs, rather than building LLM applications. So, some things are…",{"id":129,"text":130},{"commenters":175,"max_depth":167,"maker_replies":167,"first_comment_mins":176,"lifespan_hours":177},16,37,137,[179,181,183,184,186,187,190,194,196,197,200,202,203,205,206,208,212,213,215,217,218,219,222,223,225,226,228,229,232],[78,34,42,180,79,34],"project\u002Fpositive\u002Fidea",[90,34,43,182,91,34],"project\u002Fneutral\u002Fnone",[94,34,43,182,79,34],[97,34,34,185,98,34],"commenter\u002Fneutral\u002Fnone",[65,34,43,182,66,34],[188,34,42,189,79,34],"37041042","project\u002Fpositive\u002Fnone",[191,34,42,192,193,34],"37041773","maker\u002Fpositive\u002Fnone","gsuuon",[195,34,34,185,126,34],"37040778",[155,34,42,180,156,34],[198,34,43,199,91,34],"37039882","project\u002Fneutral\u002Fdocs",[201,34,43,182,66,34],"37040767",[104,34,43,182,105,34],[204,34,34,185,91,34],"37042232",[108,34,34,185,105,34],[53,34,34,207,11,33],"project\u002Fneutral\u002Fidea",[209,34,34,210,211,34],"37101778","other\u002Fneutral\u002Fnone","brianjking",[171,34,34,207,11,33],[125,33,43,214,126,34],"project\u002Fmixed\u002Fdesign",[129,33,34,216,11,33],"project\u002Fneutral\u002Fdesign",[82,38,42,189,83,34],[147,38,43,207,148,34],[220,38,34,221,11,33],"37096627","maker\u002Fneutral\u002Fnone",[115,43,42,189,116,34],[224,43,34,182,11,33],"37096661",[119,43,43,182,116,34],[136,227,43,207,137,34],4,[140,227,43,207,141,34],[230,42,42,189,231,34],"37039429","trentearl",[233,42,34,192,11,33],"37096663",[],[236,241,246,250,255,260],{"id":237,"name":238,"points":239,"category":240},"36987920","Hydra 1.0",269,"data",{"id":242,"name":243,"points":244,"category":245},"37034951","Phind V2",221,"dev_tools",{"id":247,"name":248,"points":249,"category":20},"37050532","Airbyte",220,{"id":251,"name":252,"points":253,"category":254},"37064152","ToSNotify",190,"productivity",{"id":256,"name":257,"points":258,"category":259},"37062422","Putting the \"You\" in CPU",187,"learning_science",{"id":261,"name":262,"points":263,"category":20},"37034575","Axilla",161,false,"v2",{"id":267,"name":5,"line":6,"ai":264,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":268,"top":18,"kw":269,"maker":11,"domain":10,"tier":19,"attempts":34,"strip":270,"url":9,"mix":271,"praise":75,"objection":87,"sub":21},37038053,0.33,[24,25,26,27],"53303550533300000305305033350",[31,32,33,34],1794036639626]