superglue
AI
llm benchmark
SHOW HN2025
How HN took it
3 positive, 0 neutral or asking, 0 mixed, 0 critical, out of 3 comments about the project.
Idea: 2 for, 0 againstPerformance: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Benchmark results are useful and surprisingly good for a first version
From 1 comment
v interesting benchmark, looking forward to see it evolve over time. actually surprisingly good results already. maybe add a couple harder APIs (or more complex queries) as well…
nimar on HN
Benchmarks like this are needed, since API performance varies a lot for agents
From 1 comment
Super cool that you all published these benchmarks. We've seen similar, where some APIs work REALLY well with agents, but convoluted ones just product churn in the agent tool…
kyleledbetter on HN
Objections
Benchmark results are not that surprising given experience with LLM API calls
From 1 comment
The reason we think this would be interesting to share here is that these llm benchmarks seem increasingly disconnected from reality. idc if the llm can solve a PhD math question…
sfaist on HN
Requests
Add harder APIs or more complex queries where current models fail
From 1 comment
v interesting benchmark, looking forward to see it evolve over time. actually surprisingly good results already. maybe add a couple harder APIs (or more complex queries) as well…
nimar on HN
Test OpenAI models such as o3 and the model superglue uses
From 2 comments
v interesting benchmark, looking forward to see it evolve over time. actually surprisingly good results already. maybe add a couple harder APIs (or more complex queries) as well…
nimar on HNLove hearing Sam Altman talking about feeling the AGI and seeing that million dollar reasoning models can't execute simple API calls despite having a lot of docs and the entire…
michael-fuest on HN
Benchmark Supabase APIs
From 1 comment
Super cool that you all published these benchmarks. We've seen similar, where some APIs work REALLY well with agents, but convoluted ones just product churn in the agent tool…
kyleledbetter on HN
Questions
How well does OpenAI's agent mode handle API tasks
From 1 comment
Love hearing Sam Altman talking about feeling the AGI and seeing that million dollar reasoning models can't execute simple API calls despite having a lot of docs and the entire…
michael-fuest on HN
Which model superglue used in the benchmark
From 1 comment
v interesting benchmark, looking forward to see it evolve over time. actually surprisingly good results already. maybe add a couple harder APIs (or more complex queries) as well…
nimar on HN