New project I am working on: a language model that runs entirely on one machine. No cloud, no API. The machine is not connected to the network, by definition, so everything has to run there.
The original version was a tweet in Hebrew.
The task sounds simple: choose which open-weight model is actually worth deploying. But the benchmarks published every other day by whoever just released a model do not always tell the whole truth, and I do not think this gets said enough.
The benchmark was not measured on the model you run
The numbers you see in benchmarks were measured on the full weights, at full precision, on server GPUs. What runs on your machine is almost always a 4-bit quantization, and in practice there is a whole ladder between 2 and 8 bits, and your RAM decides where on it you sit. It is not the same model. Sometimes the difference is negligible, sometimes there is none, and sometimes what you are running is the lobotomized cousin of the model in the benchmark. Nobody tells you in advance which case you are in.
There are exceptions: models trained or released directly in the compressed format, where what was measured is what runs. But in most cases you are running a version no official benchmark ever measured.
So the benchmark has to be built, not downloaded
Which leaves one honest option: a real test of real questions against real documentation, on the real hardware, in exactly the quantization that will run in production. A domain expert judges the answers blind, without knowing which model wrote what.
The constraint that shaped everything: 128GB of memory. The model has to fit alongside a 128k context and leave room for the system, so the practical ceiling is about 70GB of weights.
Hardware is a filter, not a footnote
The final hardware is not settled yet, by the way. It may end up a DGX Spark, but right now the direction is Apple Silicon. If the direction changes, some of the findings change with it, because on Apple Silicon MLX builds are significantly faster than GGUF, and does-a-proper-MLX-build-exist turned from a technical detail into a filter criterion. Strong models fell on that alone. On a Spark it is a different story, a CUDA stack, and a model disqualified here could pass there.
And the criterion that surprised me most: tool calling. More models were eliminated because they cannot call tools reliably than because of answer quality.
The five finalists
All Apache 2.0:
- Mistral Small 4 119B, MoE, runs MLX 4-bit (68GB)
- gpt-oss-120b, MoE, runs native MXFP4 (63GB)
- Gemma 4 31B, dense, runs MLX 4-bit (19GB)
- Granite 4.1 30B, dense, runs MLX 4-bit (18GB)
- Gemma 4 26B-A4B, MoE, runs INT4 QAT (16GB)
Two spoilers for the rest of the series. The model that looked most suitable on paper, one built exactly for tasks like this, is not on the list at all. Not because it is bad, but because the tooling cannot run it yet.
And what is already clear, and will be the heart of this series: most of the work was not benchmarking. It was getting models to run at all. Also, these guys are fairly accomplished liars. The best-looking answers of the entire round came from the model that invented its sources.
This is all for a client deployment where the data cannot leave the building, the kind of enterprise AI setup where just-call-an-API is not on the menu.
This is the first post in a series on deploying a fully local LLM. Next: making sense of local model formats, then the failure modes the benchmark caught. An earlier series covers running local models on a MacBook.
FAQ
Why do local LLMs underperform their benchmark scores?
Published scores are measured on the full weights, at full precision, on server GPUs. What runs locally is almost always a quantized version, usually 4-bit, and no official benchmark measured that version. Sometimes the difference is negligible and sometimes it is severe, and nothing tells you in advance which case you are in.
What matters most when choosing a local LLM for agent work?
In my testing, reliable tool calling eliminated more models than answer quality did. After that: whether the weights fit your memory ceiling with enough room left for context, and whether a well-maintained build exists for your runtime, such as MLX on Apple Silicon.