In a demo, a system that always answers looks brilliant. In a real project, it is a hazard. The difference between an answer and a reliable answer is knowing where it comes from, and being willing to say “I don’t know” when it comes from nowhere.
The problem with the confident answer
A language model is trained to sound convincing, not to be right. Faced with a gap in information, its default behaviour is to fill it with something plausible. In casual conversation, that does not matter. In a law firm’s document repository, in the figures behind a committee’s decision, or in the pricing of a consumer goods group, a confident but invented answer is not a charming mistake: it is a decision made on sand.
The most treacherous part is that the error is invisible. An invented figure and a correct one are presented exactly as well. If the system does not distinguish between what it knows and what it is guessing, it hands that entire job to the reader, who is almost never in a position to do it.
Traceability: where every fact comes from
That is why we build systems so that every answer is traceable to its source. When the system states something, you can see which document, which data point, or which source it rests on. This is not a technical detail: it is what lets a professional validate an answer instead of trusting it blind.
The AI does the analysis —it reads, connects, summarises— so that the person decides with the full picture in front of them. The business decision is not made by the machine. That division of labour is not a cautious concession: it is the only way the machine’s speed and the person’s judgment add up instead of getting in each other’s way.
Better “pending confirmation” than invention
The rule that governs our systems is uncomfortable and deliberate: when in doubt, flag “pending confirmation” rather than fill the gap. A system that admits what it does not know is more useful than one that hides it, because it turns an invisible gap into a concrete task.
We prefer a system that says “this would need verifying” to one that delivers a round number with nothing behind it. The first is honest and saves work; the second is well presented and buries the risk exactly where it is hardest to find. Acknowledging a limit is not a weakness of the system: it is the proof that you can trust what it does state.
Never a black box
Traceability and “pending confirmation” are two sides of the same discipline: that the system is never a black box. You do not have to believe the answer; you can check it. And when there is nothing to check, the system says so. That is the only way AI applied to work that has consequences becomes reliable: not by being infallible, but by being verifiable.
What this means for your project
In practice, it means the technology we leave running inside your company behaves like a good analyst, not an oracle: it works thoroughly, shows its reasoning, and warns when it steps onto uncertain ground. The judgment remains yours. Our job is to give you the best possible information to exercise it, and to be honest about its limits. A tool you can trust is, above all, a tool that knows how to tell you when it should not be believed.