I still notice when I read posts from different people on Linkedin and social media (like Twitter/X)
that they still post things like
THIS IS THE BEST MODEL
CURRENTLY MUCH BETTER THEN X,Y,Z
OH MY GOD X,Y,Z IS BLOWING AWAY THE COMPETITION!
and the list goes on and on, however they fail to actually understand the bigger difference between the different models and lastly to understand, what actually makes a model the best model?
To set the record straight, there is not one model that is best at everything.
One thing to understand is that when a vendor releases a new language model these days they are aiming to score high on different benchmarks.
While some of these benchmarks are academic benchmarks which try to determine the capability of a language model within different categories, some are more aimed at practical tests.
We also have benchmarks that are more coding based where they evaluate a LLM against a set of coding issues and check if the language model can solve them.
When I evaluate «how good» a LLM is, I always check on Artificial Analysis (https://artificialanalysis.ai/models) since they have a good list of benchmarks that they check LLMs against to determine the overall «intelligence». This intelligence score is just the sum of different benchmarks they run against an LLM.
When you are evaluating which LLM to use, you need to understand what is going to be the purpose of the usage. Is it as a virtual agent for end-users? Handling image input? OCR? Coding agents? Analysis of Norwegian poetry? Handling and summarization of large sets of text?
When vendors like OpenAI, Anthropic, Google release new LLMs they also like to boast about the capabilities with these benchmarks, but they aren’t that good at actually writing what they mean so therefore I wanted to write a short about these benchmarks, what they mean and their importance, such as seen with the release of Kimi K3 here →
Where they publish a lot of numbers and stats from different benchmarks. Which looks good but what does it mean? Therefore I wanted to highlight some benchmarks and what they actually mean.
| Category | Benchmark | What It Measures |
| Reasoning | ARC-AGI-2 | Abstract reasoning puzzles novel problem-solving/fluid intelligence via visual grid transformations |
| GPQA Diamond | Graduate-level, “Google-proof” expert Q&A in physics, chemistry, and biology (hardest subset) | |
| Coding | Terminal-Bench 2.1 | Agentic terminal coding. completing real coding tasks in a command-line environment |
| ProgramBench | Reconstructing complete software from only a binary + documentation; tests holistic architecture and modular design (extremely hard and top models pass few/no tasks) | |
| SWE-Marathon | Long sustained software-engineering task completion | |
| FrontierSWE | Frontier-difficulty real-world software engineering problems | |
| SWE-Bench | Resolving real GitHub issues by producing working patches that pass tests | |
| DeepSWE | Open agentic SWE model/benchmark for complex software engineering reasoning | |
| LiveCodeBench | Contamination-free competitive coding using recent, time-stamped problems | |
| Security / Exploitation | ExploitBench | Capability-ladder benchmark grading exploitation across 16 flags (crash → primitives → arbitrary read/write → control-flow hijack → code execution), instantiated on V8 bugs |
| ExploitGym | Whether AI agents can turn vulnerabilities into real attacks; pass/fail exploitation across ~898 instances (userspace, V8, Linux kernel) | |
| Instruction Following | IFBench | Adherence to precise, verifiable instructions and constraints |
| Agentic | T2-Bench (τ²-Bench) | Conversational agents in a dual-control environment — multi-step tool use in realistic customer-service-style tasks |
| Long Context | MRCR v2 (8-needle) | Multi-round retrieval of multiple targets (“needles”) across very long contexts (128k–1M tokens) |
| AA-LCR | Reasoning (not just retrieval) across ~100k tokens of multiple real-world documents to replicate knowledge work | |
| Knowledge | MMMLU | Massive multitask, multilingual knowledge across academic/professional domains |
| Multimodal | MMMU Pro | Multimodal understanding and reasoning across images, charts, and text |
| UI / Computer Control | OS-World | Agentic computer use operating a real OS/desktop GUI to complete tasks |
| Expert / Economic Tasks | GDPval-AA | Economically valuable, real-world knowledge work across occupations (Elo-rated) |
| Honesty | Bullshit Benchmark | Whether a model pushes back on nonsensical prompts with hidden false premises vs. confidently answering measures resistance to plausible-sounding nonsense |
However, it is important to note that these benchmarks are constantly changing. This happens because scoring metrics become saturated, test answers inadvertently leak into the model’s training data, or entirely new evaluations become necessary to precisely measure the capabilities of next-generation models.
In the case of Kimi-K3 and the benchmarks they published, its core strengths lie in its capabilities for coding and agentic tasks. While it is good at coding and agentic tasks, if I were to use it for handling and understanding Norwegian context or data, it still sucks…. While other frontier models are quite good at understanding Norwegian context, since it represents a bigger part of the training data.
Therefore it is important to understand that while a model might be good at many things, there is no model that is best at everything and which use-case you want to use it for should determine which model you should use.