[ Finance ]
AI Pilot Post-Mortem Template for Finance Teams
Turn a failed finance AI pilot into an evidence-backed decision with a copyable post-mortem record for causes, controls, owners, and restart gates.
On this page
On August 22, 2025, a media inquiry challenged citations in an Australian government assurance review. The investigation that followed confirmed errors the delivery review had missed. An AI pilot post-mortem exists for this moment: preserve the failure, compare the promise with the actual workflow, test competing causes, and make a defensible decision.
In a letter later released by the Australian Department of Finance, Deloitte said the department had consented to a ChatGPT-based tool chain for code-base analysis. At the end of the engagement, the team also used MyAssist and ChatGPT for summarization and citation formatting. Review caught some errors, but not all; the later investigation found additional inaccurate output.
The publicly identified errors were citation errors. The documented review corrected some errors but did not catch them all before delivery. The broader control question is whether the review could demonstrate that each citation matched its source. A useful post-mortem names the output failure and the detection failure.
What belongs in an AI pilot post-mortem?
A finance AI pilot post-mortem needs eight records. Start with the promise, actual workflow, first trust break, and evidence available at that moment. Then record the impact, root and contributing causes, corrective actions with owners, and the go-forward decision. Leave one out and the review becomes a meeting transcript instead of a control record.
Google's Site Reliability Engineering guidance provides the operating skeleton: record the incident, impact, mitigation, root and contributing causes, and preventive actions. A finance pilot needs three additions. It must compare the claimed use with the actual use, preserve the financial evidence behind material outputs, and state who has authority to accept the residual risk.
| Stage | Work | Output |
|---|---|---|
| Frame | Compare the approved promise with what actually ran | Scope and workflow record |
| Reconstruct | Freeze the first trust break, impact, and evidence | Replayable event timeline |
| Diagnose | Test root and contributing causes across six lanes | Cause map and counterfactual |
| Decide | Assign actions and choose stop, narrow, restart, replace, or production | Owned decision record |
Do not begin with “the model hallucinated.” That may be true. It may also be the sentence that lets missing source data, an unversioned formula, an ambiguous metric, or a rubber-stamp review walk out the side door.
Where did trust first break?
Freeze the exact output before interviewing the room about why the pilot failed. Capture the source snapshot, model and prompt version, calculation path, user, intended use, reviewer action, and what happened next. Memory tidies a story. Logs keep the splinters.
The first trust break is the earliest observable moment when a user could no longer rely on the workflow as intended. It might be a wrong number, a correct number with no source trail, an exception the system hid, a reviewer override, or a result that arrived after the decision window closed.
People punish algorithmic mistakes differently. Across five forecasting experiments, Dietvorst, Simmons, and Massey found that people could lose confidence in an algorithm faster than in a human forecaster after observing comparable mistakes, even when the algorithm performed better overall. The studies were not enterprise generative-AI deployments. The finding gives teams a reason to inspect the encounter with error, but it does not establish how enterprise adoption should be measured.
Which six root-cause lanes should an AI pilot post-mortem test?
Use six root-cause lanes and allow more than one to be true. In this Pluvo framework, a causal factor is a condition whose removal would have prevented the event. A control gap is a condition whose removal would have detected or contained it inside the accepted threshold. A contributing factor made the event more likely or more expensive to contain.
| Lane | Diagnostic question | Typical evidence |
|---|---|---|
| Promise and scope | Did the approved claim match the system's actual capability and intended use? | Charter, demo script, requirements, acceptance criteria |
| Data | Were the right records complete, timely, authorized, mapped, and traceable? | Source extracts, period filters, data tests, lineage |
| Encoded logic | Were definitions, assumptions, calculations, thresholds, and transformations explicit and versioned? | Metric contract, formula, rule history, unit tests |
| Business context | Did the system know which entity, period, policy, audience, and decision applied? | Context record, workflow state, user role, exception notes |
| Operating controls | Could review, monitoring, permissions, fallback, and change management catch or contain the error? | Access log, review record, alerts, overrides, rollback plan |
| Model or tool | Did the component fail under conditions inside its tested operating boundary? | Evaluation set, error taxonomy, version comparison, reproducible test |
Finance teams don't have a data problem. They have a context problem. Sometimes they have both. The post-mortem earns the difference.
The categories are an editorial synthesis, not a standard. They translate the NIST AI Risk Management Framework into a finance investigation. NIST separates governance, intended context, measurement, and risk treatment because model performance alone cannot establish whether a system is fit for its actual use. NIST calls for appeal and override mechanisms, incident response and recovery, and assigned responsibilities to supersede, disengage, or deactivate systems whose performance or outcomes conflict with intended use.
PCAOB Auditing Standard 1105 offers a finance-native analogy. Audit evidence must be relevant and reliable. Company-produced information used as evidence must be tested for accuracy and completeness or supported by tested controls. The standard does not require an AI post-mortem, and ordinary pilot output is not automatically audit evidence. More pages of weak evidence do not repair the evidence itself.
If the diagnosis is still unclear, use the published seven AI finance failure modes as a symptom-to-test map, then return here to document the evidence and owner.
Which evidence proves whether the pilot met its promise?
Freeze evidence before repairing the pilot. A corrected formula and cleaner prompt may improve tomorrow's run while erasing the path to yesterday's failure. Preserve the failed state, then branch.
Start the replay bundle with the source snapshot, model and prompt version, calculation path, and run log. Then attach the approved charter, system map, rule versions, evaluation set, exceptions, reviewer notes, overrides, affected outputs, user feedback, and every corrective change.
The U.S. Government Accountability Office's AI Accountability Framework organizes evidence around governance, data, performance, and monitoring. GAO's monitoring practices call for plans, schedules, acceptable ranges for drift, performance changes, corrective actions, system versions, and change logs. The framework was written for federal agencies and assessors. A finance team can use the same record to show what changed, when it changed, and who accepted the result.
NIST AI 800-4, published in March 2026, drew on three practitioner workshops, a literature review, and thematic coding. It identifies functionality, operational, human-factors, security, compliance, and large-scale-impact monitoring, while noting barriers such as drift, fragmented logging, and the difficulty of scaling human-driven monitoring alongside rapid rollouts. The report maps open problems. It does not certify any one monitoring method.
Which post-mortem decisions must stay human?
A human finance owner must decide whether the evidence is sufficient and the error is material for the intended use. A person must also judge causality, apply policy or accounting judgment, and accept or reject the residual risk. The same person need not make every decision, but every decision needs a name.
Human review is not a spell. A reviewer who cannot see the source, rule, version, and exception is being asked to approve a feeling. A reviewer who can see the evidence can challenge the work, reject it, or narrow the system's authority.
The human owner also communicates the failure to affected report and control owners, data stewards, vendors, and decision-makers. Blameless does not mean ownerless. Remove the hunt for a culprit; keep the deadline and the accountable name.
Who owns corrective action after a failed AI pilot?
A corrective action belongs to the person with the authority and resources to change the failed condition. One named pilot owner remains accountable for closing the record. Each action needs an owner, due date, acceptance test, dependency, and closure evidence. Residual-risk acceptance belongs to the authorized finance, control, or policy owner.
The Finance AI Trust Checklist can supply the control questions for a restart, and the AI audit-trail guide shows what a reviewer will expect to inspect.
How should the post-mortem record a stop, restart, or production decision?
Stop when the use is unnecessary, unlawful, or uncontrollable. Narrow it when only a bounded task is supportable. Restart when corrective actions pass tests against preserved ground truth. Replace the component when it cannot meet the requirement. Productionize only after deployment-like testing proves the threshold, monitoring, fallback, and ownership.
| Decision | Use when | Required record |
|---|---|---|
| Stop | The use is unnecessary, unlawful, uncontrolled, or cannot produce evidence at acceptable cost | Reason, owner, shutdown steps, affected users, retained evidence |
| Narrow | One bounded task is supportable but the original promise exceeded the evidence | Revised scope, prohibited uses, new threshold, communication |
| Restart | Corrective actions can be tested against preserved ground truth and the review path is owned | Action closure, regression test, rollback plan, approver |
| Replace | A component or architecture cannot meet the requirement inside reasonable constraints | Comparison basis, migration plan, control continuity |
| Productionize | The workflow meets its threshold in deployment-like conditions and monitoring, fallback, and ownership exist | Sign-off, monitoring plan, change control, next review |
Before another vendor demo or internal build, turn the post-mortem into test cases. The AI vendor accuracy questions help translate the failed evidence into a tougher evaluation script. For the broader architecture diagnosis, use Why AI Pilots Fail in Finance.
What should a copyable AI pilot post-mortem template contain?
Pluvo's record is an editorial framework. It is not a standard, audit program, or validated assessment instrument. The record forces the team to preserve the failure, test competing explanations, assign the repair, and make one decision.
| Field | Complete before the review closes |
|---|---|
| Pilot identity | Name, owner, sponsor, reviewer, start date, decision date |
| Promise | User, intended decision, minimum functionality, performance threshold, control objective |
| Actual workflow | Systems, data snapshot, model/tool and version, prompt, rules, handoffs, human roles |
| First trust break | Timestamp, exact output, expected output, detector, source evidence, override, downstream effect |
| Impact and blast radius | Affected reports, periods, entities, decisions, people, controls, and external parties |
| Evidence inventory | Run log, source snapshot, lineage, versions, tests, exceptions, reviewer notes, user feedback |
| Root causes | Promise/scope, data, encoded logic, context, controls, model/tool; label root versus contributing |
| Counterfactual | Which change or combination of changes would have prevented the event or caught it inside the accepted threshold? |
| Corrective actions | Action, owner, due date, acceptance test, dependency, closure evidence |
| What stays human | Materiality, policy, causality, evidentiary sufficiency, residual-risk acceptance, communication |
| Decision | Stop, narrow, restart, replace, or productionize; named approver and rationale |
| Next gate | Regression set, deployment-like test, monitoring signal, fallback, rollback, review date |
| Open risks | Residual risk, accepted by whom, until when, with what limit on use |
For more evidence-first finance AI operating guides, subscribe to the AI Finance Playbook.
A bad citation can be corrected in a new file. The harder repair is the review system that let it pass. Name both. Then close neither until the evidence does.
Frequently asked questions
What is an AI pilot post-mortem?
An AI pilot post-mortem is a written evidence record of the pilot's promise, actual workflow, first trust break, impact, root and contributing causes, corrective actions, owners, and go-forward decision.
Who should attend a finance AI pilot post-mortem?
Include the finance owner, end user, data owner, builder or vendor, control or risk owner, and the person authorized to stop or restart the workflow. Add legal, security, audit, or accounting specialists when the intended use requires them.
Should an AI post-mortem be blameless?
The investigation should avoid personal blame so people disclose the real system conditions. Corrective actions still need named owners, deadlines, acceptance tests, and closure evidence. Blameless does not mean ownerless.
How do you tell whether the model caused the failure?
Using Pluvo's six-lane framework, evidence for a model or tool cause strengthens when the event can be reproduced with source data, context, rules, and controls held constant. Test interactions as well as the component in isolation.
What must an AI pilot post-mortem record before a restart?
The post-mortem should record closed corrective actions, the failed case as a regression test, results from deployment-like conditions, the revised acceptance threshold, monitoring, human review, fallback, rollback, residual risk, and a named restart approver.
Is reviewed AI output sufficient audit evidence?
Not automatically. For a PCAOB audit, the auditor must obtain sufficient appropriate evidence. When using company-produced information, the auditor tests its accuracy and completeness or the related controls, and evaluates whether it is sufficiently precise and detailed.
