[ Finance ]
Why LLMs Hallucinate Financial Numbers: A Two-System Control Pattern
LLMs can generate plausible numbers without a controlled calculation. This two-system pattern is designed to keep material figures computed, traceable, and reviewable.
On this page
An LLM can hallucinate financial numbers because it predicts tokens and may be rewarded for guessing. On the arithmetic subset of TAT-QA, a tool-augmented Llama 2 system reached 56.7% exact-match accuracy; the base model was near 2%. Supervised fine-tuning contributed, so architecture alone did not cause the gain. The control lesson: let language and arithmetic run in different systems.
Why do LLMs produce numbers that sound right but are wrong?
An LLM produces a number the same way it produces a sentence: by predicting the next token from learned patterns and the supplied context. OpenAI's 2025 research on hallucination explains that pretraining contains fluent examples, not a true-or-false label beside every statement. Low-frequency facts invite plausible guesses, and accuracy-only evaluations can reward guessing rather than abstention.
The National Institute of Standards and Technology calls the failure confabulation: a generative system confidently presents false or internally inconsistent content. NIST ties the risk to the statistical design of generative models and warns that fabricated logic or citations can make an incorrect answer look more defensible than it is.
Finance adds a trap. The next token may be a dollar amount, a margin, an account number, or a period label. Fluency hides the category change. The model has not turned into a calculator because the output contains digits.
Is bad arithmetic the same as a hallucinated number?
Bad arithmetic is only one numeric failure. A system can calculate perfectly and still return the wrong financial answer because it selected the wrong source, period, entity, currency, account mapping, or metric definition. A correct formula applied to booked revenue when the CFO asked for recognized revenue is a precise mistake.
This article groups numeric failures into four classes: an invented input, the wrong records, a broken calculation, or the wrong business meaning. A calculator addresses arithmetic execution within the third class; it cannot validate the formula, source, period, or business meaning. Finance teams need controls around the full path from question to source records to reported figure.
| Failure | What happened | Example | Control |
|---|---|---|---|
| Invented input | The model supplied a fact that was absent | A vendor balance appears without a ledger record | Reject unsupported values; require source IDs |
| Wrong retrieval | The system fetched the wrong records or context | June actuals are mixed with May budget | Bind entity, period, version, and source |
| Wrong computation | The formula, join, sign, or FX logic is wrong | A credit is added instead of subtracted | Run deterministic code plus reconciliation tests |
| Wrong interpretation | The math is right but the business meaning is wrong | Booked revenue answers a recognized-revenue question | Use approved definitions and a named finance reviewer |
Why can't better prompting make the number safe?
Better prompts can improve instructions, elicit uncertainty, and reduce some errors. Better prompts cannot create a control that the surrounding system does not possess. Peer-reviewed research on intrinsic self-correction found that asking models to inspect their own reasoning without external feedback often failed to improve the answer and sometimes made it worse.
Retrieval-augmented generation helps with a different problem. The original RAG research found more factual and specific language than a parametric-only baseline. Retrieval can put the right policy, contract, or filing in view. Retrieval does not prove that the system selected the right rows, applied the right definition, or reconciled the result. Grounding supplies evidence. Computation and controls turn evidence into a figure.
Some newer models report lower hallucination rates, and tool-enabled systems can invoke calculators or execute code. The finance question is still architectural: which component produced the number, which inputs it used, which rules ran, and whether another operator can reproduce the result.
What architecture keeps generated prose away from calculated figures?
The control boundary has a deliberately blunt shorthand: LLMs predict text; they don't compute. A bare LLM is not a controlled calculation engine, although tool-enabled systems can execute code or call calculators. The two-system control pattern routes every material figure through approved data, versioned business logic, controlled execution, validation, and lineage before language is allowed to describe it.
| Stage | Primary job | System of record | Evidence retained |
|---|---|---|---|
| 1. Interpret | Resolve the question, metric, period, entity, and requested output | LLM with constrained schema | Parsed request and uncertainty |
| 2. Retrieve | Fetch approved records and governed definitions | ERP, warehouse, contracts, ontology | Record IDs, versions, access path |
| 3. Compute | Apply formulas, joins, allocations, signs, and FX rules | SQL, code, calculation service | Logic version and execution log |
| 4. Validate | Tie out totals, test tolerances, and surface exceptions | Reconciliation and test layer | Test results and unresolved items |
| 5. Explain | Draft the narrative from computed outputs only | LLM | Citations to figures and drivers |
| 6. Approve | Review meaning, materiality, and distribution | Named finance owner | Decision, timestamp, and changes |
Deterministic does not mean infallible. With the inputs, logic version, runtime, and configuration held fixed, deterministic execution produces the same output. Reproducibility makes errors easier to isolate, correct, and rerun. The design standard here is deliberately unforgiving: A number that's 95% right is 100% useless. That is a quality bar, not a measured failure threshold.
The Committee of Sponsoring Organizations of the Treadway Commission makes the operating point explicit in its current guidance on internal control over generative AI. Rapid adoption compresses workflows. Opaque reasoning, model drift, and frequent configuration changes can threaten reporting integrity unless internal controls travel with the system.
What can AI not own in a finance workflow?
AI cannot be the accountable owner of a revenue definition, materiality threshold, policy exception, control conclusion, or board recommendation. A model can surface evidence and propose language. A named finance professional must decide whether the number answers the business question and whether it is fit to distribute.
Human review is not automatically a control. A reviewer who sees only a polished paragraph cannot verify the source, logic, or exceptions. The review surface must expose the records, definitions, calculation path, test results, and changes since the prior run. Otherwise the human is approving typography.
A finance AI system also needs permission to stop. Missing records, conflicting definitions, failed reconciliations, and out-of-tolerance results should produce an exception or an abstention, not a more eloquent guess.
How should a finance system handle each kind of number?
Use the Pluvo Number-Handling Contract before a figure enters a close pack, forecast, control file, or board deck. The contract applies Pluvo's four-class editorial taxonomy, labels what may create each value, specifies the evidence that must travel, and defines when the system must stop. It is an editorial control aid, not an audit standard.
| Output class | Allowed creator | Evidence that must travel | Stop condition |
|---|---|---|---|
| Sourced fact | Approved retrieval layer | Record ID, source system, timestamp, period, entity, and version | No approved record or conflicting records |
| Computed figure | Deterministic calculation service | Input record IDs, metric definition, formula or query version, run log, and reconciliation | Failed tie-out, tolerance breach, missing input, or unapproved logic |
| Estimate or scenario | Controlled model or simulation using human-approved assumptions | Method, assumption set, model version, range, sensitivity, owner, approval date, and random seed when applicable | Unlabeled estimate, stale assumption, or no sensitivity view |
| Business assumption | Named finance owner | Rationale, effective date, scope, version, and approver | No owner, contradictory definitions, or unresolved policy question |
| Narrative statement | LLM or human writer | References to the approved facts, figures, drivers, and exceptions it describes | The prose introduces a new material number or outruns the evidence |
A pilot that drops record IDs, metric definitions, calculation versions, or named approval on the way to production is still a demo. The post-mortem on failed AI pilots in finance shows how that trail disappears.
How does Pluvo apply the two-system pattern?
A finance question enters Pluvo in plain language. The ontology resolves the entities and governed definitions for the question; the deterministic compute layer produces the figure from connected records. The language model can then explain a number it did not invent.
Each figure retains its source records, applied definitions, and transformations. Separate controls govern who may view, run, edit, or approve the work. The model remains useful for language. It does not become the ledger because it writes a convincing sentence.
That boundary keeps probabilistic text generation from becoming the source of record. It cannot repair bad source data or disputed definitions, and faulty deterministic logic remains faulty. The gain is inspectability: finance can rerun the calculation, find the break, and correct it.
What evidence should travel with each financial number?
Ask for six artifacts: source records, the metric definition, the logic version, test results, unresolved exceptions, and a named approver. Then ask the vendor to rerun the figure with the same inputs, logic version, runtime, and configuration. A polished answer without those artifacts leaves finance to rebuild the control trail.
For one practical finance-AI control pattern each week, subscribe to the AI Finance Playbook.
When the board asks where the number came from, "the AI said so" is not an answer. It is the missing control.
Frequently asked questions
Why do LLMs hallucinate financial numbers?
A bare LLM generates likely tokens from patterns and context rather than executing a controlled financial calculation. A plausible amount can therefore appear even when the source, period, metric definition, or arithmetic is absent or wrong.
Can prompt engineering stop numeric hallucinations?
Prompt engineering can improve instructions and elicit uncertainty, but it cannot create missing source controls, deterministic calculations, reconciliations, lineage, or approval. Those protections must exist in the surrounding system.
Does RAG prevent financial hallucinations?
Retrieval-augmented generation can supply relevant source material, but retrieval alone does not prove that the correct records were selected, the correct business definition was used, or the resulting figure was computed and reconciled correctly.
What is deterministic computation in finance AI?
With approved inputs, versioned logic, runtime, and configuration held fixed, deterministic execution produces the same figure. Controlled code, SQL, or a calculation service performs the calculation while the LLM interprets requests and explains verified outputs.
Can deterministic finance software still be wrong?
Yes. Source data, definitions, formulas, mappings, and human decisions can all be wrong. Reproducible execution makes the failure easier to inspect so the team can isolate the cause, correct it, and rerun the result.
What must remain human-owned in AI financial analysis?
A named finance professional should own metric definitions, materiality, policy exceptions, unresolved reconciliations, control conclusions, and material recommendations. Human review is meaningful only when the reviewer can inspect the evidence and logic.
