[{"data":1,"prerenderedAt":356},["ShallowReactive",2],{"project-43549072":3},{"id":4,"name":5,"line":6,"line_source":7,"title":8,"url":9,"domain":10,"maker":11,"posted_at":12,"year":13,"points":14,"comments":15,"rank_in_year":16,"shown_in_year":17,"top":18,"tier":19,"category":20,"subcategory":21,"format":22,"keywords":23,"reception":28,"sentiment":29,"stance_mix":30,"scored_comments":34,"aspects":35,"strip":45,"post_text":47,"what":48,"said":72,"maker_in_thread":187,"shape":195,"grid":199,"attempts":317,"same_week":318,"maker_shows":31,"maker_hidden":348,"reader_version":349,"card":350},"43549072","OmniAI OCR benchmark","Qwen-2.5-32B is now the best open source OCR model","title","Show HN: Qwen-2.5-32B is now the best open source OCR model","https:\u002F\u002Fgithub.com\u002Fgetomni-ai\u002Fbenchmark\u002Fblob\u002Fmain\u002FREADME.md","github.com","themanmaran",1743526849,2025,211,47,295,3837,8,"notable","ai","ocr benchmark","other",[24,25,26,21,27],"ocr","open source ocr","qwen 2.5 vl","json extraction","curious",0.047,{"positive":31,"neutral":32,"mixed":33,"critical":31},6,13,1,26,{"idea":36,"design":38,"execution":39,"docs":40,"performance":41,"pricing":43},{"positive":33,"critical":37},0,{"positive":37,"critical":37},{"positive":33,"critical":37},{"positive":37,"critical":37},{"positive":42,"critical":33},3,{"positive":37,"critical":44},2,[46,42,42,37,37,46,42,42,42,37,42,42,42,37,37,37,42,46,33,37,46,46,42,37,37,37,46,42,33,37,33,42,33,37,33,37,42,37,33,37],5,"Last week was big for open source LLMs. We got:\n\n- Qwen 2.5 VL (72b and 32b)\n\n- Gemma-3 (27b)\n\n- DeepSeek-v3-0324\n\nAnd a couple weeks ago we got the new mistral-ocr model. We updated our OCR benchmark to include the new models.\n\nWe evaluated 1,000 documents for JSON extraction accuracy. Major takeaways:\n\n- Qwen 2.5 VL (72b and 32b) are by far the most impressive. Both landed right around 75% accuracy (equivalent to GPT-4o’s performance). Qwen 72b was only 0.4% above 32b. Within the margin of error.\n\n- Both Qwen models passed mistral-ocr (72.2%), which is specifically trained for OCR.\n\n- Gemma-",{"problem":49,"audience":55,"pricing":56,"motivation":64,"time_to_build":55,"stack":68,"openness":69,"stage":70,"team_size":71},{"value":50,"source":51,"evidence":52},"Compares open-source OCR and vision LLMs on structured JSON extraction accuracy","maker",[53],{"id":54},"post",null,{"value":57,"source":58,"evidence":59},"Benchmark code and dataset are open source; OmniAI product is paid","commenter",[60],{"id":61,"by":62,"text":63},"43551175","rustc","The model with the best accuracy in the linked benchmark is \"OmniAI\" (OP's company) which looks like a paid model, not open source [1]. [1]: https:\u002F\u002Fgetomni.ai\u002Fpricing",{"value":65,"source":51,"evidence":66},"Update the OCR benchmark with newly released open models",[67],{"id":54},[],"open_source","launched","unknown",{"praise":73,"objections":93,"requests":118,"questions":147,"alternatives":162},[74,89],{"value":75,"source":58,"evidence":76},"Qwen 2.5 VL performs very well on OCR and vision tasks",[77,81,85],{"id":78,"by":79,"text":80},"43551340","CSMastermind","I've been very impressed with Qwen in my testing, I think people are underestimating it",{"id":82,"by":83,"text":84},"43551927","fpgaminer","I've been consistently surprised by Gemini's OCR capabilities. And yeah, Qwen is climbing the vision ladder _fast_. In my workflows I often have multiple models competing…",{"id":86,"by":87,"text":88},"43552172","stavros","I wrote a small, client-side-JS-only app that does OCR and TTS on board game cards, so my friends and I can listen to someone read the cards' flavor text. On a few pages of text…",{"value":90,"source":51,"evidence":91},"Benchmark is fully open source with code and dataset",[92],{"id":54},[94,102,109],{"value":95,"source":58,"evidence":96},"Benchmark is run by an OCR vendor that ranks its own product first",[97,101],{"id":98,"by":99,"text":100},"43550807","azinman2","News update: OCR company touts new benchmark that shows its own products are the most performant.",{"id":61,"by":62,"text":63},{"value":103,"source":58,"evidence":104},"Open models produce errors in precision for real use",[105],{"id":106,"by":107,"text":108},"43559088","pmarreck","Its errors are interesting (averaging around one per paragraph). Semantically-correct, but wrong on precision (simple example, the English word \"ardour\" is transcripted as…",{"value":110,"source":58,"evidence":111},"Full benchmark costs are high and provider pricing varies widely",[112,115],{"id":113,"by":62,"text":114},"43551213","Reproducing the whole benchmark would be expensive, OmniAi starts at $250\u002Fmonth.",{"id":116,"by":11,"text":117},"43551259","Yes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models.…",[119,129,136],{"value":120,"source":58,"evidence":121},"Add bounding box support to the benchmark",[122,126],{"id":123,"by":124,"text":125},"43551523","ks2048","I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm…",{"id":127,"by":11,"text":128},"43552704","We're also looking to test qwen and other for the bounding box support. Simon Willison had a great demo page where he used Gemini 2.5 to draw bounding boxes, and the results were…",{"value":130,"source":58,"evidence":131},"Add cost and latency results",[132],{"id":133,"by":134,"text":135},"43550948","daemonologist","You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would…",{"value":137,"source":58,"evidence":138},"Include Surya and MiniCPM models",[139,143],{"id":140,"by":141,"text":142},"43552167","ianhawes","Is there a reason Surya isn’t included?",{"id":144,"by":145,"text":146},"43549671","sandreas","What about mini cpm v2.6?",[148,155],{"value":149,"source":58,"evidence":150},"How does Tesseract compare to LLM-based OCR",[151],{"id":152,"by":153,"text":154},"43555605","AndrewDucker","Tesseract can manage 99% accuracy on anything other than handwriting. Without being an LLM. Is there an advantage of using an LLM here?",{"value":156,"source":58,"evidence":157},"Why is Llama 90b priced so high",[158],{"id":159,"by":160,"text":161},"43551589","dylan604","One of these things is not like the others. $8.50\u002F1000?? Any chance that's a typo? Otherwise, for someone that has no experience with LLM pricing models, why is Llama 90b so…",[163,170,174,181],{"value":164,"source":58,"evidence":165},"Mistral OCR",[166],{"id":167,"by":168,"text":169},"43555254","hdjjhhvvhga","I'm very curious about the results - I've been using mistral-ocr for the last 2 weeks and it worked really well.",{"value":171,"source":58,"evidence":172},"Tesseract",[173],{"id":152,"by":153,"text":154},{"value":175,"source":58,"evidence":176},"Gemini 2.5 Pro",[177],{"id":178,"by":179,"text":180},"43551762","kapitalx","In addition, gemini Pro 2.5 does really well with bounding boxes, but yeah not open source :(",{"value":182,"source":58,"evidence":183},"Doctly",[184],{"id":185,"by":179,"text":186},"43551756","If you're limited to open source models, that's very true. But for larger models and depending on your document needs, we're definitely seeing very high accuracy (95%-99%) for…",{"comments":188,"answered":44,"key":189},4,[190,191,192],{"id":127,"text":128},{"id":116,"text":117},{"id":193,"text":194},"43551806","That was the cost when we ran Llama 90b using TogetherAI. But it's quite hard to standardize, since it depends a lot on who is hosting the model (i.e. together, openrouter, grok,…",{"commenters":196,"max_depth":197,"maker_replies":188,"first_comment_mins":198,"lifespan_hours":34},31,7,54,[200,204,206,210,212,214,218,220,223,227,230,232,234,236,238,239,240,244,247,251,253,255,257,260,261,262,265,268,270,273,275,280,282,284,287,289,291,294,295,297,300,303,305,308,310,312,314],[201,37,46,202,203,37],"43551006","project\u002Fpositive\u002Fidea","jauntywundrkind",[123,33,42,205,124,37],"project\u002Fneutral\u002Fdesign",[207,33,42,208,209,37],"43551752","project\u002Fneutral\u002Fidea","jsight",[185,33,37,211,179,37],"other\u002Fpositive\u002Fperformance",[178,33,37,213,179,37],"other\u002Fneutral\u002Fnone",[215,33,46,216,217,37],"43552076","project\u002Fpositive\u002Fexecution","michaelt",[127,33,37,219,11,33],"maker\u002Fneutral\u002Fnone",[221,33,42,208,222,37],"43552641","chpatrick",[224,33,42,225,226,37],"43553085","project\u002Fneutral\u002Fdocs","deepsquirrelnet",[228,44,42,229,107,37],"43552325","project\u002Fneutral\u002Fperformance",[167,44,37,231,168,37],"commenter\u002Fneutral\u002Fnone",[106,44,42,233,107,37],"project\u002Fmixed\u002Fperformance",[133,42,42,235,134,37],"project\u002Fneutral\u002Fpricing",[116,42,37,237,11,33],"maker\u002Fneutral\u002Fpricing",[159,42,42,235,160,37],[193,42,37,237,11,33],[241,42,37,242,243,37],"43552161","commenter\u002Fneutral\u002Fpricing","int_19h",[245,42,37,213,246,37],"43556385","nickpsecurity",[248,42,42,249,250,37],"43551686","project\u002Fneutral\u002Fnone","esafak",[82,188,46,252,83,37],"project\u002Fpositive\u002Fperformance",[254,46,37,213,124,37],"43551496",[152,31,33,256,153,37],"project\u002Fcritical\u002Fperformance",[258,31,37,259,203,37],"43559996","commenter\u002Fneutral\u002Fperformance",[78,197,46,252,79,37],[86,197,46,252,87,37],[263,18,42,249,264,37],"43552078","WillAdams",[266,18,37,231,267,37],"43552406","3abiton",[269,18,37,231,264,37],"43555896",[271,18,37,231,272,37],"43552622","jumploops",[274,18,37,231,264,37],"43553323",[276,277,46,278,279,37],"43552835",9,"maker\u002Fpositive\u002Fnone","codybontecou",[140,281,42,249,141,37],10,[144,283,42,249,145,37],11,[98,285,33,286,99,37],12,"maker\u002Fcritical\u002Fnone",[288,285,37,231,203,37],"43550853",[61,285,33,290,62,37],"project\u002Fcritical\u002Fpricing",[292,285,42,249,293,37],"43551091","johnisgood",[113,285,33,290,62,37],[296,285,37,237,11,33],"43551436",[298,285,33,299,293,37],"43551449","project\u002Fcritical\u002Fnone",[301,285,37,231,302,37],"43551661","qingcharles",[304,285,37,231,293,37],"43552290",[306,285,42,249,307,37],"43553147","ipsum2",[309,285,37,231,293,37],"43554713",[311,285,33,286,179,37],"43551815",[313,285,37,231,99,37],"43552117",[315,285,37,231,316,37],"43552236","axpy906",[],[319,324,329,334,338,343],{"id":320,"name":321,"points":322,"category":323},"43543241","Nue",756,"dev_tools",{"id":325,"name":326,"points":327,"category":328},"43543235","FluentSubs",472,"languages",{"id":330,"name":331,"points":332,"category":333},"43514383","Pomodoro",315,"hardware",{"id":335,"name":336,"points":337,"category":20},"43532967","WhatsApp MCP",229,{"id":339,"name":340,"points":341,"category":342},"43520953","pg-mcp",167,"data",{"id":344,"name":345,"points":346,"category":347},"43551767","Igatha",163,"infrastructure",false,"v2",{"id":351,"name":5,"line":6,"ai":348,"cat":20,"year":13,"points":14,"comments":15,"rec":28,"s":352,"top":18,"kw":353,"maker":11,"domain":10,"tier":19,"attempts":37,"strip":354,"url":9,"mix":355,"praise":75,"objection":95,"sub":21},43549072,0.05,[24,25,26,21,27],"5330053330333000351055300053101310103010",[31,32,33,31],1793991116404]