August 2026 · Paul Jennings
Curation beats generalisation
A smaller, well-kept knowledge set answers domain questions better than a general-purpose model. That is easy to assert and harder to demonstrate, which is why I built something to test it.
August 2026 · Paul Jennings
A smaller, well-kept knowledge set answers domain questions better than a general-purpose model. That is easy to assert and harder to demonstrate, which is why I built something to test it.
A general-purpose model is fluent about everything and dependable about nothing in particular. Ask it something inside a specialism and you get an answer in the right register, assembled from whatever the training data happened to contain about that subject, weighted by how often people wrote it down rather than by whether it was correct. For a great many questions this is fine. For the questions organisations actually pay to have answered, it is not.
The failure is not that the model knows nothing. It is that it cannot tell you which of the things it knows are still true, which were superseded, which apply to your jurisdiction, your equipment, your standard, your version. Nobody curated that. It was scraped.
It means a person who knows the field decides what goes in and what stays out. Which document is current. Which of two contradictory sources wins, and why. What is missing and needs writing down before the person holding it retires. This is unglamorous work and it does not compress into a model card, which is why it tends to get skipped in favour of a larger context window.
It is also the entire advantage. A curated set is smaller, so retrieval has less to be wrong about. It carries provenance, so an answer can be traced back to the document it came from and checked by someone who would know. It has an owner, so when it goes stale there is a person responsible rather than a retraining schedule.
The hard part of this argument is not making it. It is proving it, because a wrong answer from a language model reads exactly like a right one. Confidence is not a signal. Fluency is not a signal. And a demo is not evidence: whoever built it chose the questions, and people choose questions their system answers well. I have done it myself.
If you cannot say how you would find out that you were wrong, you do not have a result. You have a preference.
So the useful thing to build first is not the product. It is the apparatus that tells you whether the product is any good.
A retrieval system is a stack of choices: which documents, how they are split, which embedding model turns them into vectors, which store holds those vectors, how many are retrieved, which language model writes the answer. Each of those is usually chosen once, early, by whoever was there, and then never revisited because changing one means the whole thing has to be re-judged by hand.
The harness makes that cheap. One fixed set of questions written by people who know the correct answers. Every component swappable. Change the embedding model and nothing else, run the set, see what moved. It turns an argument between engineers into a table.
Two things fall out of doing this that I did not expect. The first is how often the expensive choice is not the better one. The second is how much of the variation comes from the documents rather than the models — the same stack, pointed at a set someone has actually tended, behaves like a different product.
Numbers from a specific comparison go here — question count, models compared, and the size of the difference. Worth naming one result precisely rather than gesturing at all of them.
If the documents matter more than the model, the procurement question changes. It stops being which model are you using, which is answerable by anyone and predicts very little, and becomes whose knowledge is in here, who decided, and how would we know if it went out of date. Those are harder questions and they are the ones worth asking.
None of this makes the model unimportant, and none of it removes the failure modes that come with generating text. It moves the effort to where it pays: getting the knowledge right, and being able to show that you did.