In the first post of this series I wrote that the final hardware was not settled. Since then a new card has entered the game: an NVIDIA RTX PRO 4500 Blackwell that is already sitting at the client.
And that quietly changes the question.
The original version was a tweet in Hebrew.
A 32GB card against two 120B finalists
On one hand, it has only 32GB of memory. The two 120B models that reached my finals weigh 63 and 68GB after compression, so they simply do not fit. On the other hand, it has GDDR7 with 896GB/s of bandwidth, more than three times the 273GB/s of a DGX Spark, and 30B models at 4-bit weigh around 18GB.
If a 30B model is good enough, there is no reason to buy a Spark with 128GB just to hold a 120B. The card is already there, it should be faster at generating tokens, and the whole solution gets cheaper and simpler. Should is the important word: I have not measured the full run on it yet, so there is no performance promise here.
The vocabulary: model, precision, packaging, engine
Before any results, the pile of acronyms that gets thrown around local models needs sorting: MLX, GGUF, safetensors, MXFP4, NVFP4. Four different things are hiding in there.
- The model is the recipe: the architecture and the weights it learned in training.
- Precision, or quantization, is how each number is represented: BF16, INT4, Q4_K_M, MXFP4.
- The packaging is the file format the weights ship in.
- The engine is the runtime that actually executes it.
That is why MLX-versus-GGUF is a useful shorthand but technically a crooked comparison.
GGUF is the file format and ecosystem of llama.cpp. Very portable; it runs on almost anything: CPU, Mac, NVIDIA and more.
MLX is the framework built around Apple Silicon and its unified memory. When I say MLX build, I mean a package of weights and configuration that the MLX engine knows how to load well on a Mac.
safetensors does not mean full precision either. It is packaging, and it can hold different kinds of tensors.
MXFP4 and NVFP4 are not packaging at all, but ways of representing numbers in 4 bits. Blackwell accelerates FP4 in hardware, but putting a 4-bit file on a Blackwell card does not automatically make it NVFP4. The weights, the runtime engine and the kernels all have to support the same path.
Why the same model behaves differently on two machines
On the Mac, a proper MLX build was significantly faster than GGUF. On the 4500, a build that exploits CUDA and FP4 may flip the picture. And when you compare MLX 4-bit on a Mac against GGUF or NVFP4 on NVIDIA, you are not testing just hardware. In practice you also swapped the numeric representation, the weight packaging and the runtime engine.
The new round
So I am sort of back at the beginning, just with prior knowledge, and opened another round focused on models around 30B whose weights fit in 32GB, in the quant that will run in production. What I care about is not only tokens per second. I also measure:
- How much context is left after the weights are loaded
- Whether tool calling works
- Which documents the model actually opened
- Whether it cites a source it never read
Those last two turned out to decide everything. They get the next post.
Picking hardware for an on-prem model is exactly the kind of decision I get pulled into in consulting work: the spec-sheet answer and the right answer are rarely the same one.
FAQ
Is MLX faster than GGUF on Apple Silicon?
In my testing on a Mac, a well-made MLX build of the same model was significantly faster than its GGUF build. That is not a law of nature: switching between them changes the numeric representation, the weight packaging and the runtime engine at once, so the gap varies by model and build quality.
Does running a 4-bit model on a Blackwell GPU automatically use NVFP4?
No. Blackwell accelerates FP4 in hardware, but the weights, the runtime engine and the kernels all have to support the same path. Loading any 4-bit file onto the card does not switch that path on by itself.