[{"data":1,"prerenderedAt":325},["ShallowReactive",2],{"project-49800574":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":29,"sentiment":30,"stance_mix":31,"scored_comments":35,"aspects":36,"strip":46,"post_text":48,"what":49,"said":65,"maker_in_thread":174,"shape":181,"grid":185,"attempts":287,"same_week":288,"maker_shows":40,"maker_hidden":317,"reader_version":318,"card":319},"49800574","JevBench","A reproducible benchmark for typed decision models","title","Show HN: JevBench, a reproducible benchmark for typed decision models","https:\u002F\u002Fbenchmarkheaven.com\u002Fjev-models","benchmarkheaven.com","florianstandhar",1790082063,2026,154,39,420,4366,10,"notable","dev_tools","model benchmark","cli",[24,25,26,27,28],"llm benchmark","decision models","model leaderboard","calibration scoring","latency cost benchmark","roasted",-0.256,{"positive":32,"neutral":18,"mixed":33,"critical":34},3,0,13,26,{"performance":37,"privacy":39,"idea":41,"execution":42,"design":44,"docs":45},{"positive":33,"critical":38},4,{"positive":33,"critical":40},1,{"positive":33,"critical":40},{"positive":43,"critical":40},2,{"positive":33,"critical":38},{"positive":33,"critical":33},[40,40,33,33,40,47,40,32,40,32,33,33,33,33,40,40,40,33,33,33,33,40,32,32,32,33,32,32,33,32,32,47,47,32,40,40,33,40,40],5,"Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.\n\nJev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.\n\nJevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.\n\nA full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed",{"problem":50,"audience":56,"pricing":56,"motivation":57,"time_to_build":56,"stack":61,"openness":62,"stage":63,"team_size":64},{"value":51,"source":52,"evidence":53},"Compares typed decision models on accuracy, latency and price in one weighted score.","maker",[54],{"id":55},"post",null,{"value":58,"source":52,"evidence":59},"Maker wants a public, reproducible comparison of Jev against open and lookalike models.",[60],{"id":55},[],"open_source","launched","unknown",{"praise":66,"objections":85,"requests":126,"questions":147,"alternatives":162},[67,78],{"value":68,"source":69,"evidence":70},"Reproducible, open harness and public per-task outcomes make results checkable.","commenter",[71,75],{"id":72,"by":73,"text":74},"49812398","DylanMerigaud","534 English decisions in one full run sounds substantial.",{"id":76,"by":11,"text":77},"49814443","Thanks! Yeah, the code is open: https:\u002F\u002Fgithub.com\u002Ffstandhartinger\u002Fwho-is-right. Email classifying is definitely a good Jev usecase, I agree",{"value":79,"source":69,"evidence":80},"Open-sourced leaderboard is a useful, fast-growing comparison.",[81],{"id":82,"by":83,"text":84},"49823304","tamimio","For something that was announced a week ago, and now there are ~80 model competing is crazy!",[86,97,107,122],{"value":87,"source":69,"evidence":88},"Jev is closed and its claimed performance is unverified against cheap open clones.",[89,93],{"id":90,"by":91,"text":92},"49809071","hbrn","$40m in funding, 2 years in stealth. Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking…",{"id":94,"by":95,"text":96},"49809863","Retro_Dev","Isn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so…",{"value":98,"source":69,"evidence":99},"Benchmark coverage is narrow, mostly text classification tasks.",[100,104],{"id":101,"by":102,"text":103},"49814276","vessenes","I think it’s much more likely that these benchmarks are not good, in that they do not explore much of the space that jev was (likely) trained to cover. The doom demo is a good…",{"id":105,"by":91,"text":106},"49816019","So if benchmark says Jev sucks, it's benchmark's fault. And yet Typesafe doesn't share anything about their internal benchmark that apparently proves how amazing Jev is. Funny…",{"value":108,"source":69,"evidence":109},"The linked slop detector gives confident wrong answers on simple inputs.",[110,114,118],{"id":111,"by":112,"text":113},"49810063","542458","Keysmashing my keyboard resulted in 86% confidence that the text was AI written. I don't think this is a particularly good classifier, I've never seen an LLM output \"kad…",{"id":115,"by":116,"text":117},"49810079","ajs1998","I asked free chatgpt to give me some essays that will fool a slop detector and they all fooled this slop detector. Its best guess was \"6% slop probability 87% confidence answered…",{"id":119,"by":120,"text":121},"49811050","pushpendraw","the slop detector giving 86% confidence on a keysmash is the real finding here, not the leaderboard score. confident and wrong is worse than an LLM that just hedges.",{"value":123,"source":69,"evidence":124},"Leaderboard comparison shows little gap versus cheap browser models like SemIf.",[125],{"id":90,"by":91,"text":92},[127,133],{"value":128,"source":52,"evidence":129},"Add more task families beyond text classification, such as computer use and images.",[130],{"id":131,"by":11,"text":132},"49814421","Agree the coverage question is real. Today it's text decisions (intent, routing, extraction, scoring, harder multi-constraint cases). We're working on screen\u002Fcomputer-use and…",{"value":134,"source":69,"evidence":135},"Add more models to the leaderboard, including BART zero-shot.",[136,140,144],{"id":137,"by":138,"text":139},"49813649","faangguyindia","Can you please add our model to benchmark? https:\u002F\u002Fgambler-relay-us-west1.leo-fish.ts.net\u002Fdemo",{"id":141,"by":142,"text":143},"49810808","janalsncm","It is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.",{"id":145,"by":11,"text":146},"49814408","Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.",[148,155],{"value":149,"source":69,"evidence":150},"How is the benchmark protected from models tuned to the 534 public items?",[151],{"id":152,"by":153,"text":154},"49812306","adityamishra241","How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?",{"value":156,"source":69,"evidence":157},"Are there Jev-like models that run on Ollama?",[158],{"id":159,"by":160,"text":161},"49841547","joknoll","Are there any \"System One\" models or jev like models that run on ollama?",[163,167],{"value":164,"source":69,"evidence":165},"SemIf",[166],{"id":90,"by":91,"text":92},{"value":168,"source":69,"evidence":169},"Hugging Face jev-decision-index",[170],{"id":171,"by":172,"text":173},"49808495","sean_pedersen","Good project but this one also exists https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmultimodalart\u002Fjev-decision-index and the results do not seem to add up and also model sets are different...…",{"comments":38,"answered":32,"key":175},[176,177,180],{"id":131,"text":132},{"id":178,"text":179},"49814465","Happy to — please open an issue at https:\u002F\u002Fgithub.com\u002Ffstandhartinger\u002Fjevbench\u002Fissues with an endpoint or runnable code, the model and licence, and whether it was trained on the…",{"id":145,"text":146},{"commenters":182,"max_depth":38,"maker_replies":38,"first_comment_mins":183,"lifespan_hours":184},30,462,118,[186,188,190,192,194,196,199,203,207,209,212,215,217,220,223,226,228,230,234,237,240,243,247,251,254,256,257,260,263,264,267,270,273,274,278,279,280,282,284],[90,33,40,187,91,33],"project\u002Fcritical\u002Fperformance",[94,33,40,189,95,33],"project\u002Fcritical\u002Fprivacy",[101,33,33,191,102,33],"commenter\u002Fpositive\u002Fnone",[131,33,33,193,11,40],"maker\u002Fneutral\u002Fidea",[105,33,40,195,91,33],"project\u002Fcritical\u002Fidea",[197,33,47,198,102,33],"49820356","project\u002Fpositive\u002Fexecution",[200,40,40,201,202,33],"49810168","project\u002Fcritical\u002Fdesign","dmix",[204,40,32,205,206,33],"49810317","project\u002Fneutral\u002Fdesign","dogscatstrees",[208,40,40,201,202,33],"49810370",[210,40,32,205,211,33],"49810610","CBLT",[213,40,33,214,142,33],"49810783","other\u002Fneutral\u002Fdesign",[216,40,33,214,202,33],"49811013",[218,40,33,214,219,33],"49811286","anukin",[221,40,33,214,222,33],"49812603","spiderfarmer",[224,40,40,201,225,33],"49813959","wannabe44",[227,40,40,201,222,33],"49812575",[171,43,40,229,172,33],"project\u002Fcritical\u002Fexecution",[231,32,33,232,233,33],"49807791","other\u002Fneutral\u002Fnone","swyx",[235,32,33,232,236,33],"49808504","jldugger",[238,32,33,232,239,33],"49813597","Maxious",[241,32,33,232,242,33],"49808861","arbot360",[244,32,40,245,246,33],"49809120","project\u002Fcritical\u002Fnone","tomrod",[248,32,32,249,250,33],"49812939","project\u002Fneutral\u002Fdocs","meander_water",[252,38,32,249,253,33],"49810509","ks2048",[137,47,32,255,138,33],"maker\u002Fneutral\u002Fnone",[178,47,33,255,11,40],[152,258,32,259,153,33],6,"maker\u002Fneutral\u002Fexecution",[141,261,32,262,142,33],7,"project\u002Fneutral\u002Fexecution",[145,261,33,259,11,40],[159,265,32,266,160,33],8,"project\u002Fneutral\u002Fnone",[268,265,32,266,269,33],"49865512","luhgor",[82,271,47,272,83,33],9,"project\u002Fpositive\u002Fnone",[72,18,47,198,73,33],[275,276,32,249,277,33],"49808592",11,"nzoschke",[111,276,40,187,112,33],[115,276,40,187,116,33],[76,276,33,281,11,40],"maker\u002Fneutral\u002Fdocs",[119,283,40,187,120,33],12,[285,34,40,245,286,33],"49816723","MisterMunchkin",[],[289,294,299,304,308,312],{"id":290,"name":291,"points":292,"category":293},"49823738","Bastardica",862,"creative",{"id":295,"name":296,"points":297,"category":298},"49783133","Mini-AGI",277,"ai",{"id":300,"name":301,"points":302,"category":303},"49740105","Setup",251,"productivity",{"id":305,"name":306,"points":307,"category":20},"49801329","Drop",193,{"id":309,"name":310,"points":311,"category":298},"49776523","Tiny Brains",116,{"id":313,"name":314,"points":315,"category":316},"49746346","Snapdrop",112,"infrastructure",false,"v2",{"id":320,"name":5,"line":6,"ai":317,"cat":20,"year":13,"points":14,"comments":15,"rec":29,"s":321,"top":18,"kw":322,"maker":11,"domain":10,"tier":19,"attempts":33,"strip":323,"url":9,"mix":324,"praise":68,"objection":87,"sub":21},49800574,-0.26,[24,25,26,27,28],"110015131300001110000133303303355311011",[32,18,33,34],1794036710619]