JevBench
Dev tools
model benchmark
SHOW HN2026
How HN took it
3 positive, 10 neutral or asking, 0 mixed, 13 critical, out of 26 comments about the project. Thread score -0.26.
Performance: 0 for, 4 againstDesign: 0 for, 4 againstExecution: 2 for, 1 againstPrivacy: 0 for, 1 againstIdea: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Reproducible, open harness and public per-task outcomes make results checkable.
From 2 comments
534 English decisions in one full run sounds substantial.
DylanMerigaud on HNThanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree
florianstandhar on HN
Open-sourced leaderboard is a useful, fast-growing comparison.
From 1 comment
For something that was announced a week ago, and now there are ~80 model competing is crazy!
tamimio on HN
Objections
Jev is closed and its claimed performance is unverified against cheap open clones.
From 2 comments
$40m in funding, 2 years in stealth. Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking…
hbrn on HNIsn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so…
Retro_Dev on HN
Benchmark coverage is narrow, mostly text classification tasks.
From 2 comments
I think it’s much more likely that these benchmarks are not good, in that they do not explore much of the space that jev was (likely) trained to cover. The doom demo is a good…
vessenes on HNSo if benchmark says Jev sucks, it's benchmark's fault. And yet Typesafe doesn't share anything about their internal benchmark that apparently proves how amazing Jev is. Funny…
hbrn on HN
The linked slop detector gives confident wrong answers on simple inputs.
From 3 comments
Keysmashing my keyboard resulted in 86% confidence that the text was AI written. I don't think this is a particularly good classifier, I've never seen an LLM output "kad…
542458 on HNI asked free chatgpt to give me some essays that will fool a slop detector and they all fooled this slop detector. Its best guess was "6% slop probability 87% confidence answered…
ajs1998 on HNthe slop detector giving 86% confidence on a keysmash is the real finding here, not the leaderboard score. confident and wrong is worse than an LLM that just hedges.
pushpendraw on HN
Leaderboard comparison shows little gap versus cheap browser models like SemIf.
From 1 comment
$40m in funding, 2 years in stealth. Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking…
hbrn on HN
Requests
Add more task families beyond text classification, such as computer use and images.
From 1 comment
Agree the coverage question is real. Today it's text decisions (intent, routing, extraction, scoring, harder multi-constraint cases). We're working on screen/computer-use and…
florianstandhar on HN
Add more models to the leaderboard, including BART zero-shot.
From 3 comments
Can you please add our model to benchmark? https://gambler-relay-us-west1.leo-fish.ts.net/demo
faangguyindia on HNIt is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.
janalsncm on HNGood call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.
florianstandhar on HN
Questions
How is the benchmark protected from models tuned to the 534 public items?
From 1 comment
How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?
adityamishra241 on HN
Are there Jev-like models that run on Ollama?
From 1 comment
Are there any "System One" models or jev like models that run on ollama?
joknoll on HN
Alternatives named
SemIf
From 1 comment
$40m in funding, 2 years in stealth. Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking…
hbrn on HN
Hugging Face jev-decision-index
From 1 comment
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-index and the results do not seem to add up and also model sets are different...…
sean_pedersen on HN