I promised results from my local model benchmark. Let's start with the models that failed, because they explain why most of the work in this project was not running a benchmark. It was building a benchmark you can believe at all.
The original version was a tweet in Hebrew.
The bullshitter: authoritative, detailed, and making it up
The most dangerous and spectacular failure was Granite 4.1 30B.
It wrote the longest, most authoritative answers of the round, eight to nine steps on average. If I had read only the answers, it would have looked like an excellent candidate.
But in 12 of 20 answers it cited documents it had never opened. Only 6 of 20 passed the bar I defined as defensible: an answer in the required format, with no citation to a source the model did not actually read.
This is exactly why the harness does not only store the final answer. It stores every search, every read, which document was opened, which passage came back, and what the model claimed to have read.
How the harness got there
To understand how we arrived at that design, go back to its first version.
At the start I loaded no document text into the context at all. The model got a catalog with file names and sizes, and a tool that let it pick a document and read a 10,000-character window at an offset. The idea was simple: there are about 685,000 tokens of documentation, no context holds that, let the model navigate.
What actually happened: models read window after window from the beginning of a document until the whole read budget ran out, and the windows started and ended mid-sentence, mid-table, mid-procedure. On one of the hard questions this took 226 seconds and ended with no answer at all.
Next step: I added a search tool and an explicit instruction, search first, then read around the hit. Same model, same question, same settings. The time dropped to 51 seconds and the model returned an orderly seven-step answer.
One problem remained: the model was reading fragments, not whole units of content.
Look at your data before you blame the model
So I went and profiled the data properly. The first thing that jumped out: it is extremely uneven. Three giant guides make up 76% of all the material. The 13 one-pagers together are only about 24,500 tokens, and eight more documents are small enough to return whole.
From there the current version was built:
- All the one-pagers are preloaded and serve as a topic map.
- A medium document is returned in full.
- In the three giant guides the model searches, then receives a whole section cut along the Markdown headings, not an arbitrary window in the middle of the text.
This is not classic RAG, where an external mechanism decides which chunks to push at the model. The model itself sees the topic map, decides where to search, and chooses which source to open. The retrieval decisions, good and bad, belong to the model, and that is exactly what I need to measure.
Every convenience changes behavior
A side effect worth knowing about: the models started reading less. Before the one-pagers were preloaded, gpt-oss-120b in reasoning mode opened a full guide on 20 of 20 questions. Afterwards, on 14. In fast mode the drop was from 18 to 11.
The summaries were meant to help the model find the source. Sometimes they simply supplied it, and the model decided there was no need to check the full guide.
The slacker: fast, clean, and not doing the job
One model, Nemotron 3.5 Lightning 30B, took that behavior to the opposite extreme.
On paper its result looks decent: 18 of 20 defensible answers, and the whole round finished in 27 minutes. But on 11 of 20 questions it did not invoke a single tool. Only 5 answers rested on a full guide, and it read less than a quarter of the material the leading models read.
It did not invent many sources because it barely used sources and made few claims. That is a clean result, but not the behavior I want from an assistant whose job is to research the documentation before answering.
The real question a benchmark should ask
So we already have two very different kinds of failure:
- The bullshitter: Granite read little, wrote a lot, and invented authority.
- The slacker: Nemotron was fast and clean, but often simply did not investigate.
And exactly between them sits the real question of the benchmark. Not who writes the prettiest answers, and not who produces the most tokens per second, but who can find the right material, read it, and answer without inventing what it did not find.
Worth noticing: everything described here happened before the domain expert reviewed a single answer for content. The setup failed models long before the content ever mattered.
One 30B model managed to do all three. It gets the next post.
The harness itself is the same shape as the agent systems I build for clients: the tools you hand a model and the evidence you log decide whether you can trust anything it says.
Earlier in this series: why public benchmarks do not match local models and making sense of local model formats.
FAQ
How do you detect that an LLM fabricated a citation?
Log every tool call, not just the final answer. My harness records every search, every read, which document was opened and which passage came back. A citation is fabricated when the answer references a document that never appears in that log. Reading only the final answers, the worst offender looked like the best model.
Is agentic document retrieval the same as RAG?
Not quite. In classic RAG an external pipeline decides which chunks get pushed into the context. Here the model itself sees a topic map, decides where to search, and chooses which source to open and read. The retrieval decisions, good and bad, belong to the model, which is exactly what the benchmark needs to measure.