Explainable AI in Finance: Accuracy Needs a Proof Trail

At 9:17 a.m. in an illustrative Thursday close, the board pack shows 68.0% gross marginGlossaryGross marginGross margin is gross profit as a percentage of revenue, what it includes (revenue minus COGS), and how to read it. For gross margin, the useful boundary is whether the movement comes from customers, contracts, billing, cash timing, recognition rules, churn, expansion, pricing, or usage behavior.Open full definition →. Explainable AIGlossaryExplainable AIExplainable AI is methods that make a model's outputs interpretable to humans. For explainable AI, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition → in finance means a qualified reviewer can retrace how that output was produced, what it means, and where the system must stop. Until sources, definitions, calculations, exceptions, changes, and approval can be retraced, the figure remains unverified.

The arithmetic works. Nobody can tell whether rebates sit inside revenue, which entities were included, or whether a late journal reached cost of salesGlossaryCOGSCOGS is the direct costs of producing the goods or services a company sells, and what it includes versus operating expenses. For COGS, the useful boundary is whether the movement comes from customers, contracts, billing, cash timing, recognition rules, churn, expansion, pricing, or usage behavior.Open full definition →.

NIST's Four Principles of Explainable AI treats decision accuracy and explanation accuracy as separate properties. This article uses a stricter finance operating test: a material output is not decision-grade until a qualified reviewer can verify what produced it.

A black box may land on the right number. Finance still cannot rely on the box until the number carries its proof.

What does explainable AI mean in finance?

Explainable AI in finance gives the right reviewer faithful evidence about how an output was produced, what the output means, and where the system must stop. The explanation must fit the decision. A controllerGlossaryControllershipControllership is the function responsible for accounting accuracy, the close, internal controls, and compliance. For controllership, the important details are the accounting period, source evidence, reviewer, materiality threshold, and control purpose that make the treatment auditable during close, reporting, and later review.Open full definition → checking gross margin needs source records and calculation logic, not a paragraph about model confidenceGlossaryConfidence scoreConfidence score is a model's estimated likelihood that its output is correct. For confidence score, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition →.

NISTIR 8312 gives four useful principles: the system supplies evidence or reasons, the explanation is meaningful to its audience, the explanation faithfully reflects the process, and the system recognizes its knowledge limits. The third principle is the trap. A polished rationale can sound useful while describing a process the system never followed.

The NIST AI Risk Management Framework 1.0 keeps valid and reliable separate from explainable and interpretable. It also says opaque systems can be accurate, while noting that opacity makes accuracy harder to determine and monitor over time. NIST says version 1.0 is under revision, so treat the framework as a current reference point, not a permanent last word.

Can an opaque AI output still be accurate?

Yes. An opaque system can produce a numerically correct output. The problem is that one correct output does not establish a reliable process, and the reviewer cannot tell whether the next result used the same population, definition, logic, or assumptions.

A correct number with a false explanation is a lucky answer.

Current Federal Reserve, OCC, and FDIC model-risk guidance makes a related point for covered banking models: even a sound model producing accurate outputs can carry high risk when misapplied. The April 2026 guidance explicitly excludes generative and agentic AIGlossaryAgentic AIAgentic AI is artificial intelligence that can plan, choose tools, and take multi-step actions toward a goal with limited human supervision. For agentic AI, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition →, so it is not a GenAIGlossaryGenerative AIGenerative AI is artificial intelligence that creates text, images, code, or other content from learned patterns and prompts. For generative AI, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition → rule. The analogy is narrow: accuracy holds only for the documented use, input population and calculation that were tested.

What must an AI explanation prove?

A finance-grade explanation must prove five things: source and scope, definition, computation, exceptions and changes, and human disposition. The Pluvo Number Proof Card below is an original editorial framework, not an audit standard or certification. The five fields organize a plausible answer for review without certifying it.

The Number Proof Card does not prove that the company's metric policy is wise. It proves which policy ran, which records entered the calculation, what changed, and who accepted the result for a named use. That is enough to turn disagreement into work instead of archaeology.

The Number Proof Card's final disposition is binary for the intended use: verified or unverified. A 68.0% figure with an empty exception field is unverified, even if the same number appeared last month.

Why is a model-generated rationale not an audit trail?

A post-hoc, free-form model rationale is another output to test. By itself, it is not evidence of the process that produced the answer. Generative systems can fabricate both a conclusion and the logic or citation that appears to support it. Replayable evidence helps establish provenance and test process fidelity.

The NIST Generative AI Profile treats confabulation as a core risk and recommends provenance, source and version records, test history, approval thresholds, fallback procedures, and evaluation of transformations. A fluent chain of thought supplies none of those artifacts by itself.

In the peer-reviewed NeurIPS paper Sanity Checks for Saliency Maps, researchers randomized model parameters and training labels. Some visually persuasive explanation methods remained largely unchanged. The domain was image saliency, not finance. A convincing explanation can still be detached from the process it claims to explain.

The mechanics of numeric confabulation belong in the two-system control pattern for hallucinated financial numbers. The proof-trail test is simpler. If the rationale cannot point to the exact records, definition, calculation, changes, and review, it is commentary.

What can AI not do, and what must stay human?

AI cannot own the meaning of gross margin, decide which contradiction is material, accept an unresolved exception, or approve a figure for the board. A system can assemble evidence, execute approved logic, detect breaks, and draft commentary. A named finance professional owns the definition and the decision.

Human review is useful only when the reviewer can challenge the result. The reviewer needs the source population, the formula or transformation, contrary evidence, change history, and authority to stop distribution. A green check beside an opaque answer is not review. It is decoration.

PCAOB standards provide a narrow evidence analogy, not a universal rule for every finance output. AS 1105 distinguishes inquiry, recalculation, and reperformance. Asking the operator is weaker than checking the math or independently executing the procedure. The six-question AI audit trail covers the formal workpaper version of that discipline.

How should finance test one AI-produced number?

Test one material number through four verbs: recalculate, reperform, reconcile, and hand off. The exercise should fit inside a real close, forecast, or board-pack workflow, not a prepared demo. The aim is independent reproduction of evidence and logic, not another favorable output.

• Recalculate the approved formula from the recorded inputs.
• Reperform the source selection and transformation without relying on the original operator's clicks.
• Reconcile the source population and output to an authoritative total, then inspect contradictory evidence and exceptions.
• Hand the Number Proof Card to a qualified reviewer who did not build the run. If that person cannot reproduce and defend the output, mark it unverified.

The AI governance checklist for finance and the vendor accuracy question guide cover the broader workflow and buying tests. The Number Proof Card stays deliberately small: one output, five fields, one disposition.

How does Pluvo separate explanation from calculation?

Pluvo keeps material calculations outside the language model. Approved data and versioned business logic produce the figure; the model explains it while the workflow retains the records, transformations, exceptions and approvals. The separation keeps narrative flexible without letting generated prose determine or overwrite the number.

Separating deterministic calculationGlossaryDeterministic computeDeterministic compute is a calculation engine that returns the same output for the same input every time. For deterministic compute, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition → from language generation does not make bad data good or disputed logic correct. It makes the break inspectable. Financial data lineage lets a reviewer move from the board-pack figure back to the calculation and source records, then forward again after a correction.

Finance EngineeringGlossaryFinance engineeringFinance engineering is the discipline of designing finance workflows, data models, automation, controls, and AI interfaces as an integrated operating system. For finance engineering, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition → is the discipline of building AI-native financeGlossaryAI-native financeAI-native finance is finance functions and tools built around AI and automation from the ground up rather than added onto legacy processes. For AI-native finance, the useful boundary is the data, tools, approvals, human review, evaluation standard, and decision the system may influence.Open full definition → systems that are accurate, governed, auditable, model-agnosticGlossaryModel-agnosticModel-agnostic is a system not tied to a single model or provider. For model-agnostic, the useful boundary is the data it uses, the tools it can call, the approvals it needs, the review standard, and the finance decision it may influence before the output is trusted or automated.Open full definition →, and directly tied to how the business actually operates.

For one evidence-backed finance AI control pattern each week, subscribe to the AI Finance Playbook.

At 9:17 a.m., gross margin is still 68.0%. The difference is that the number is now finished.

About the author

Vanessa Galarneau

CFO & COO

LinkedIn →