OmniAI OCR Benchmark
Data
ocr benchmark
SHOW HN2025
146points
40comments
#510of 3,837 in 2025
Top 14%of 2025 by points
How HN took it
CuriousMostly questions and neutral comments
1 positive, 12 neutral or asking, 1 mixed, 9 critical, out of 23 comments about the project. Thread score -0.20.
Execution: 0 for, 4 againstPerformance: 1 for, 2 againstPrivacy: 0 for, 2 againstIdea: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Honest, open data release from OmniAI
From 1 comment
As far as I'm concerned, all of these specialty services are dead compared to a generalized LLM like OpenAI or Gemini. I wrote in a previous post about how NLP services were dead…
blindriver on HN
Gemini Flash 2.0 is a strong, cheap OCR option
From 2 comments
A big takeaway for me is that Gemini Flash 2.0 is a great solution to OCR, considering accessibility, cost, accuracy, and speed. It also has a 1M token context window, though…
jasonjmcghee on HNAs far as I'm concerned, all of these specialty services are dead compared to a generalized LLM like OpenAI or Gemini. I wrote in a previous post about how NLP services were dead…
blindriver on HN
Objections
GPT-4o as judge and structured extraction task is not a true OCR benchmark
From 2 comments
I’m definitely not getting that takeaway. This wasn’t even an OCR benchmark: the task was structured data extraction, and deterministic metrics were set aside in favor of…
anon373839 on HNGPT-4o as a judge to evaluate the quality of something which gpt4o is not inherently that good at. Red flag.
danielcampos93 on HN
Accuracy of 87-92% is still too error-prone for numbers and money
From 2 comments
YES >>5% accuracy can be worth a lot. Most surprising to me about these results is the BEST error rate was over 8% errors (91.7% accuracy) and the worse was 40%. Their method of…
toss1 on HNI think 87% to 92% accuracy really isn't much difference. You're still going to get errors to the point where the level and amount of checking you need to do isn't affected. Even…
blindriver on HN
VLMs can hallucinate and correct source documents, unsuitable for important OCR
From 2 comments
I’m definitely not getting that takeaway. This wasn’t even an OCR benchmark: the task was structured data extraction, and deterministic metrics were set aside in favor of…
anon373839 on HNWe've tested basic prompt injections within images, but not been able to reliably trigger any adverse effects. However there are two big bugs we've found with VLMs: 1. Correcting…
themanmaran on HN
Requests
Add open models like Qwen 2.5 VL and small VLMs
From 4 comments
Thank you for sharing this. Some of the other public models that we can host ourselves may perform in practice better than the models listed - e.g. Qwen 2.5 VL…
betula_ai on HNAnyone have tried comparing with Qwen VL based model? I heard good things about its performance on ocr compared to other self hostable model, but haven't really tried…
banditelol on HNYes I'd like to see this repeated with any of the small VLM's like IBM Granite or the HF Smols. Pretty much anything in the sub 7B range.
jimmySixDOF on HNWondering why Florence-2 is not on the list of models?
default_ on HN
Add free engines like Tesseract for comparison
From 1 comment
Also, what's strange is there's no free of paid OCR engine is added to the mix for the evaluation. Tessaract is built specially for OCR'in scanned documents, and it has a both…
bayindirh on HN
Benchmark prompt injection risks in OCR
From 1 comment
The benchmark I most want to see around OCR is one that covers risks from accidental (or deliberate) prompt injection - I want to know how likely it is that a model might OCR a…
simonw on HN
Questions
What is the privacy of documents sent to the cloud API?
From 1 comment
What is the privacy of the documents for the cloud service? There’s nothing in the privacy policy about data sent over the api.
bn-l on HN
How does this compare to Marker?
From 1 comment
How does this compare to Marker https://github.com/VikParuchuri/marker?
gbertb on HN
Which VLMs are used in OmniAI?
From 2 comments
What kind of VLMs are being used in OmniAI? I fine-tuned a Llama 3.2 Vision on a small dataset I created for extracting text without heavy cropping. Results are simply amazing in…
hasibzunair on HNWhat VLMs do you use when you're listing OmniAI - is this mostly wrapping the model providers like your zerox repo?
fzysingularity on HN
Alternatives named
Google Gemini
From 2 comments
As far as I'm concerned, all of these specialty services are dead compared to a generalized LLM like OpenAI or Gemini. I wrote in a previous post about how NLP services were dead…
blindriver on HNA big takeaway for me is that Gemini Flash 2.0 is a great solution to OCR, considering accessibility, cost, accuracy, and speed. It also has a 1M token context window, though…
jasonjmcghee on HN
Tesseract
From 1 comment
Also, what's strange is there's no free of paid OCR engine is added to the mix for the evaluation. Tessaract is built specially for OCR'in scanned documents, and it has a both…
bayindirh on HN
ABBYY FineReader
From 1 comment
> What would be interesting is finding which two OCR/AI systems make the most different mistakes and running documents against both. Flagging only the disagreements for human…
sgc on HN
Marker
From 1 comment
How does this compare to Marker https://github.com/VikParuchuri/marker?
gbertb on HN