While many are talking about sovereignty these days and how to become less dependent on the large tech companies and their language models such as Anthropic ( such as where Fable was blocked from the US government, since the government was scared of its capabilities) and now yesterday when it was announced that OpenAI had a “rogue” AI agent that compromised Huggingfaces infrastructure to try and gain access to the answers of the benchmark Exploitgym. Which shows that they do not have control of their own infrastructure (both OpenAI and Huggingface..)
So time to run locally?
Many are stating (at least on social media especially on linkedin with the AI influencers) that we can now run local LLMs which are equally as powerful as the big three (OpenAI, Anthropic, and Google Gemini) Well the answer is NO and NOPE.
If we look at Kimi-K3 which was just released, which is the largest open-source model in the market. When you see benchmarks like this, you can easily understand that what they did is pretty impressive, since it competes with GPT 5.6 and Fable on many of the same capabilities when it comes to agent workloads and code development.
While you can get some of the same capabilities on certain levels they are nowhere near the bigger models in terms of overall functionality and secondly the infrastructure needed to run these are not that easy to get by, but allow me to explain.
Firstly let us start with its capabilities.
1: Kimi-K3 while good at coding, it is nowhere near the other frontier models on cybersecurity, as posted by AISI UK Lab → https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-preliminary-assessment-kimi-k3s-cyber-capabilities
2: In Scandinavian languages it is missing a lot of data within its training set. This impacts that the model misinterprets and blends different languages together like Norwegian/Swedish/Danish.
The pattern is most likely that Kimi has been trying to build a model that competes with Anthropic/OpenAI/Google on the main benchmarks, but once you go outside of those it becomes a lot weaker, since it would make the model even larger and even more difficult to host on your own infrastructure.
To run models locally you need an inferencing engine that loads the weights of an LLM into memory and services calls to the LLM. There are many local LLM inferencing engines you can use on your own device/computer and server infrastructure.
To name a few…
- Ollama
- Windows AI Foundry
- Unsloth
- LocalAI
- AnythingLLM
- LMStudio
- Llama.cpp
- Msty
Enterprise/Datacenter versions
- vLLM (Open-Source)
- Openshift Enterprise AI (uses vLLM)
- Nutanix Enterprise AI
- VMware Private Cloud AI
- NVIDIA Triton
These inference engines loads the model weights of the LLM into memory and serve the LLM through an API which either the user or other applications can talk with directly.
If we look at one of the most popular latest models, Kimi K3 which is the first open-source model to reach 2.8 trillion parameters. When it comes to hosting LLMs, GPU memory is the single biggest constraint.
As a rule of thumb, an LLM needs about 2 GB of VRAM per billion parameters (on average). Meaning that Kimi K3 would require 5,600 GB (5.6 TB) of VRAM just to be loaded into memory before it is actually used. In addition this is not counting peak VRAM from weights, activations, KV cache, and framework overhead. Which will require even more VRAM.
From the technical documentation posted on Hugginface → https://huggingface.co/blog/ResterChed/kimi-k3-model-overview-mxfp4-quantization-open-wei
Hardware requirements: Minimum 1.4 TB aggregate GPU memory for weight loading. Practical minimum: 8-node cluster with 8x 80GB GPUs each (5.12 TB total, leaving headroom for KV-cache and activations). Which would mean 8-16 nodes of 8x H100/B200, and just for context each B200 cost anywhere from $30,000 to $50,000 per GPU, which is at least 1 million USD just for the GPUs.
Secondly, even if you have the hardware, you cannot run K3 locally on day one. K3 introduces two architectural components that existing open-source inference engines do not yet support:
- Kimi Delta Attention (KDA): Not in llama.cpp, Ollama, or LM Studio as of July 2026.
- Stable LatentMoE routing: Requires custom kernel support.
The minimum of 1.4 TB is a lot lower than what I wrote, but! This is because of a quantization technique used called MXFP4 (which I will get back to…)
When running models locally we have the ability to use a technique that is called LLM Quantization which reduces the amount of bits that is used to store a model’s weights and calculations. In short it is a technique to reduce the computational and memory costs of running inference by representing the weights and activations with low-precision data types like 8-bit integer (int8) instead of the usual 32-bit floating point (float32).
So the lower the quantization of a model the less vGPU memory it consumes and also increases the speed of the model on the current hardware. If we look at GLM 5.2 which is another great open-source model.
By default at 744B parameters it would consume close to 1.5 TB of memory at FP16. Using quantization we can reduce the memory requirements. Just to show what amount of memory we would still need to load the models.
| Quantization | Model Size |
| 1-bit | 223 GB |
| 2-bit | 245 GB |
| 3-bit | 290–360 GB |
| 4-bit | 372–475 GB |
| 5-bit | 570 GB |
| 8-bit | 810 GB |
Great! At 1-bit we reduced the requirements from 1.5 TB to 223GB, so what is the catch of this process? For many benchmarked LLMs , at 1-bit weights the LLM gets around 76.2% accuracy yet being 86% smaller! (As Unsloth publishes on their benchmarks with GLM 5.2)
This shows dynamically quantizing some layers to higher precision does not make the model 76% worse yet being 86% smaller, but only 24% less capable than the full 1.5TB model.
But what does “76% accuracy” actually describe?
76% top-1% does NOT mean “The largest planet is” becomes choosing 76% Jupiter and 24% Mars. The model is NOT “dumber” by 24%. For this, Jupiter is always 100%, and Mars is 0%. The 76% number includes filler words and stop words across the entire corpus for example asking:
“Write a summary” will get due to LLM sampling:
- Sure, here is a summary…
- Below you will find the summary:
- Which topic should the summary cover?
Each example is correct, but the [Sure, Below, Which] distribution is what changes. The baseline might use [Sure] 100% of the time, but now [Sure] is 76% and [Below] is 24%.
It does NOT mean that you get incorrect outputs like gibberish or incorrect outputs 24% of the time. Instead it means 24% of the top, it might choose the 2nd highest probable answer more.
For example asking “Create a poem” will produce vastly different results – 74% of the time it’ll be similar poems to the BF16 baseline, but 26% it’ll be another poem, but still correct. For factual answers where creativity and sampling is not involved, the model will always recall the same answer (for example “What is 2+2”, it’ll always say 4, not 5)
It should also be noted that different quants can behave in unexpected ways, there isn’t a linear degradation as you might expect. Sometimes a model just doesn’t like one quant so maybe for a specific model Q5 performs poorly and all Q4 quants are better.
Larger models are also affected less than smaller models. So a 32B is still pretty good at Q4 but a 1B model at Q4 is braindead and useless in many cases. Unless you use LLMs that are optimized or created for Q1 such as with Microsoft framework Bitnet, where they have custom created LLMs for 1-bit quantization (https://bitnet.live/) (which you can test here → https://demo-bitnet-h0h8hcfqeqhrf5gf.canadacentral-01.azurewebsites.net/)
So what are some of the quantization levels we have?.
FP32 = Full precision (32-bit)
TF32 = TensorFloat-32, NVIDIA’s 19-bit tensor format
FP16 = Half precision (16-bit)
BF16 = Brain Float 16, 16-bit with the same exponent range as FP32
FP8 = 8-bit floating-point (E4M3/E5M2)
MXFP4 = Microscaling FP4, a next-generation 4-bit floating-point format
NF4 = 4-bit NormalFloat, the format used in QLoRA
Q8 = 8-bit quantization
INT8 = 8-bit integer quantization
Q6 = 6-bit quantization
Q5 = 5-bit quantization
Q4 = 4-bit quantization
Q3 = 3-bit quantization
Q2 = 2-bit quantization
Q1 = 1-bit quantization
So this means that we can use some of these techniques to reduce the overall hardware requirements. Still in theory with Kimi K3 requiring “just” 1.4 TB of memory without any usage. In addition every token of context (the text you send to the model plus the text it generates) requires additional memory on top of the model weights themselves. Short conversations are fine, but if you’re feeding in documents or maintaining extended chat histories, context memory can add up quickly.
As a rule of thumb, 8K tokens of context adds about 1 GB per 10B parameters. Push that to 32K tokens and you’re looking at an extra 4 GB. Some newer models support 1million tokens or more, which means that it will require a lot more memory.
So if you plan to provide Kimi K3 to your own organization and support a high number of users, you are looking at alteast 4 TB of memory to support the high context + usage + KV cache. If we look at the GB200 NVL72 rack which costs in the range of 3 – 4 million USD.
Source: https://newsletter.semianalysis.com/p/h100-vs-gb200-nvl72-training-benchmarks (while GB200 Rack provides close to 13.4 TB of vGPU Memory, way more than what is needed for Kimi-K3. Still it would be close to 1 – 2 million USD just to buy the necessary hardware to run the platform (that is not including licenses and operational cost).
Now the plus side of having your own platform is that you do not need to pay the token tax…..or have to wait for the next session, because you used all your tokens.
If we look at Anthropic subscription pricing per month, it is close to 20$ per user per month. While it also has other features outside of just getting access to the LLM, but as a theoretical exercise.
If we have a company of 6,000 users that all require access to LLMs would it be cheaper with Kimi K3 on local infrastructure or an Anthropic subscription across a span of 5 years?
Anthropic = 6000 users x 20$ x 12 months x 5 years = 7.2 million USD
Kimi K3 = (Coreweave 42$ per node/hour x 18 nodes x 730 hours monthly x 12 (months) x 5 years = 33 million USD.
On paper, self-hosting looks 4–5x more expensive. But this comparison is apples-to-oranges, and it’s worth being honest about why:
The $20 Anthropic seat is a metered cost; it scales linearly with headcount, comes with usage caps, and bundles a lot more than raw inference (the app, features, updates, support). The 18-node cluster is the opposite of a fixed-capacity asset. That $33M buys the same cluster whether 600 people use it or 6,000, and it runs 24/7 whether or not anyone is typing. So the real number that matters isn’t cost-per-seat, it’s utilization cost per token actually served.
That flips the logic depending on scale. At 6,000 light users, the subscription wins easily. But push usage, heavy agentic workloads, long context, thousands of concurrent power users hammering the API all day and the metered model’s bill climbs while the cluster’s cost stays flat. There’s a crossover point where owning the hardware becomes the cheaper option per token. For most organizations at this size and usage profile, you’re nowhere near it.
And the $33M isn’t the whole story either, it’s just compute rental. Add licensing, the platform/MLOps team to keep KDA and LatentMoE routing running, model updates, and the opportunity cost of being locked to a model that’s weaker outside the headline benchmarks.
So, time to run locally?
The honest answer is: it depends on why you’re asking.
If the goal is to escape dependency on the big three for sovereignty, data control, or simply not wanting a US company to hold the keys then yes, models like Kimi K3 and GLM 5.2 finally make self-hosting a real option in a way it wasn’t 18 months ago. That’s a genuine shift, and it’s worth taking seriously.
But if the pitch you’re hearing on LinkedIn is “you can now run something as good as Fable or GPT-5.6 on your own hardware” that’s still a NOPE. Three things get glossed over every time:
Capability: Kimi K3 competes on the headline benchmarks — coding, agentic workloads — but the moment you step outside them it gets thinner fast. Weaker on cybersecurity, weaker on Scandinavian languages, weaker anywhere the training data ran shallow. The frontier labs aren’t just winning on the benchmarks you can see; they’re winning on the long tail you only notice in production.
Infrastructure: “Open weights” doesn’t mean “runs on your laptop.” Kimi K3 needs ~1.4 TB of GPU memory just to load at MXFP4, before context, KV-cache, and concurrency and its new architecture (KDA, Stable LatentMoE) doesn’t even run on Ollama or llama.cpp yet. Quantization buys you a lot, but there’s no free lunch… less memory, some accuracy, and a model that’s harder to serve at scale. Smaller models degrade hard under aggressive quants; large ones tolerate it but large ones are exactly the ones you can’t fit.
Cost. Self-hosting isn’t cheaper it’s a fixed-capacity bet. You stop paying the per-token tax, but you take on the hardware, the MLOps team, and the risk of being locked to a model that’s weaker off the main path. Whether that trade pays off comes down to utilization, not sticker price.
So the real question isn’t “can I run a local LLM?” you can. It’s “what am I actually buying?” And the answer is control, not parity. For some organizations regulated, sovereignty-driven, or running at massive sustained scale that control is absolutely worth it. For most, the frontier APIs are still cheaper, smarter, and someone else’s problem to operate.
The open-source models are catching up on the benchmarks. They haven’t caught up on everything else and everything else is where most of the work actually happens.