We’re living in an exciting time where AI models are surpassing human performance in various tasks. They’re becoming incredibly useful, and many individuals and companies are reaping real benefits from them.
However, to be truly valuable, these models shouldn’t just outperform humans on benchmarks; they also need to be faster and more cost-effective.
That’s why I think it’s time to shift our focus away from raw benchmark scores alone. Speed and cost matter just as much, if not more.
Let me put it this way: I don’t care if a model can ace a prestigious exam if it costs $1,000 per task and takes a month to run. A model that’s slightly less capable but significantly cheaper and faster is far more useful in the real world.
## The Problem with Benchmark Obsession
We’ve all seen it – posts comparing the performance of different models on a particular benchmark, like GPT-5 vs. Grok-4 on the ARC-AGI 2 benchmark. But what about the cost and speed of these models? Are they really practical for everyday use?
## A More Holistic Approach
I propose that we start looking at benchmarks differently. Instead of just focusing on raw scores, let’s consider the entire package: performance, speed, and cost. This will give us a more accurate picture of a model’s usefulness in real-world scenarios.
## The Future of AI
As AI continues to advance, it’s essential that we prioritize practicality alongside performance. By doing so, we’ll unlock the true potential of these models and make them more accessible to people and companies around the world.
What do you think? Should we be looking at benchmarks differently? Share your thoughts in the comments below!