[ Finance ]
Model-Agnostic Finance: What Must Survive an LLM Swap
A finance AI strategy is model-agnostic only when context, calculations, controls, evidence, and decision history survive a tested model replacement.
On this page
Anthropic scheduled Claude Opus 4.1 for retirement on August 5, 2026. Anthropic owned the calendar; its customers owned the migration risk. A model-agnostic AI strategy for finance treats that deadline as a component change, not a rebuild. Source data, metric definitions, calculation code, permissions, lineage, test cases, and approval history stay put while a tested replacement takes the model's seat.
Anthropic gave developers notice on June 5 and named Claude Opus 4.8 as the replacement, according to its current model deprecation registry. The two-month notice is one provider event, not an industry clock. Anthropic states that retired-model requests fail and recommends thorough application testing before migration.
Turnover is ordinary, but its timing is uneven. OpenAI's living API deprecation registry sets different minimum notice periods for generally available models, specialized variants, and preview models. A strategy built around 'models change every six months' is therefore too neat to be useful. Build for change without inventing the cadence.
What is a model-agnostic AI strategy for finance?
A model-agnostic AI strategy keeps source data, metric definitions, calculation code, permissions, evidence, and test history outside any one model provider. Workload-specific regression tests then determine whether a replacement meets the organization's approved performance and control thresholds. Portability does not mean two models behave alike.
The model may interpret a question, choose a governed tool, or draft an explanation. Finance should retain the source contracts, account mappings, metric definitions, exact calculation code, permission policy, review gates, test cases, and decision record. The replaceable model layer includes weights and behavior, service limits, endpoints, message formats, credentials, and release schedules. Open-weight or self-hosted deployments change who operates that layer, not which finance assets must remain portable.
Model dependence creates operating risk as well as architecture risk. The Bank of England and FCA's 2024 survey received responses from 118 regulated firms. One-third of current AI use cases reported by respondents were third-party implementations. The three most-named third-party model providers accounted for 44% of all named model providers. Among firms using or planning to use AI, 46% reported only partial understanding of their AI technologies. The survey is directional for corporate finance because it covers UK financial services and all AI types, not FP&A alone.
What should survive an LLM swap?
Pluvo's proposed invariant stack has seven layers: authoritative data, business meaning, deterministic computation, tool contracts, policy, evidence, and evaluation history. The model adapter can change. The finance operating system cannot quietly change with it.
| Layer | Finance-owned invariant | Representative failure when provider-coupled |
|---|---|---|
| 1. Source contracts | Approved systems, fields, entity IDs, accounting periods, freshness rules, and read boundaries | A replacement reads a different population or cannot reconstruct the same one |
| 2. Business meaning | Metric definitions, account mappings, dimensions, entity relationships, policies, and version history | Revenue, margin, or headcount silently changes meaning |
| 3. Computation | Exact formulas, rounding, currency rules, code, test vectors, and reconciliations | The language model starts generating figures instead of invoking tested calculations |
| 4. Tool contracts | Provider-neutral input schemas, validation, error handling, and least-privilege execution | A new tool-call envelope breaks the workflow or widens access |
| 5. Policy and review | Permissions, segregation of duties, materiality, review gates, overrides, and stop conditions | The migration changes who can see, run, approve, or distribute work |
| 6. Evidence | Source lineage, prompt and tool version, run log, output, reviewer, decision, and change record | Finance cannot reproduce the old answer or compare it with the new one |
| 7. Evaluation history | Real cases, expected outputs, tolerances, error taxonomy, prior failures, and acceptance thresholds | The replacement is approved because a demo looked fluent |
| Replaceable adapter | Model name, endpoint, credentials, provider-specific message format, and routing configuration | A provider retirement becomes a company-wide reconstruction project |
The voluntary NIST AI Risk Management Framework Core supports several control outcomes behind this editorial framework. Govern 1.7 calls for safe decommissioning and phase-out procedures; Measure 2.1 calls for documented test sets, metrics, and evaluation tools; Measure 2.3 calls for performance demonstrated under conditions similar to deployment; Manage 4.1 includes decommissioning, recovery, and change management. NIST does not prescribe Pluvo's seven-layer stack.
Where should business context live?
Business context should live in governed systems the company can version, inspect, and export, not only in model weights, a system prompt, or a provider-hosted retrieval feature. Runtime context can change without retraining a model. Portability still fails if the knowledge base, embeddings, definitions, or retrieval logic cannot move.
NIST defines retrieval-augmented generation as pairing a model with a separate retrieval system or knowledge base and supplying relevant information at runtime. That separation is useful. It is not a portability certificate. A proprietary vector store, embedding model, chunking scheme, or ranking API can still create a separate migration dependency.
Finance needs a semantic contract before it needs a vector index. The published guide to the finance semantic layer explains why recognized revenue, entity scope, fiscal period, currency treatment, ownership, and lineage must have explicit meaning. A model can consume that contract. It should not be its only home.
Store provider-specific prompts and few-shot examples in the replaceable adapter, then test them like code. Definitions, policies, and approval language belong in governed repositories with owners and version history. A clever prompt may improve one model. It is no vault for accounting policy.
How do you test a replacement model?
Pluvo's six proposed acceptance tests compare task success, numerical integrity, source use, control behavior, failure handling, and operating cost under production-like conditions. Test a replacement against preserved finance cases before it sees live work. It passes only when named owners accept the residual differences and a rollback remains available.
| Test | Evidence | Pass condition |
|---|---|---|
| Task completion | Representative close, forecast, variance, and reporting cases | Meets the approved success threshold by workflow, not one blended average |
| Numerical integrity | Expected tool calls, calculation outputs, rounding, and reconciliations | Every material figure comes from an approved calculation path; expected-versus-actual results and reconciliations meet documented tolerances |
| Grounding and lineage | Required sources, citations, entity IDs, and reconstruction trail | Every material claim resolves to the correct approved evidence |
| Control behavior | Permission denials, review gates, separation of duties, and stop cases | Policy-bypass and stop-condition cases produce the approved denial, escalation, or safe-failure behavior |
| Regression and edge cases | Prior incidents, ambiguous prompts, missing data, conflicts, and adversarial cases | Replayed known failures remain resolved; observed new failures stay within documented tolerance |
| Operations | Latency, cost, rate limits, regional availability, logs, and rollback drill | The workflow meets service requirements and can return to the approved model |
Procurement should use the same evidence standard. The 10 questions to ask AI vendors about accuracy already covers ground truth, error taxonomy, monitoring, and change control. The model-swap test applies those questions to a live replacement: same cases, same evidence standard, a new component.
The NIST AI RMF Playbook's Manage guidance covers third-party models, provenance, evaluation before and after deployment, contingency, and decommissioning. Its suggestions are voluntary, and NIST says the Playbook will be updated after AI RMF 1.0 is revised. Finance still needs enough evidence for a real go or no-go decision.
What does model-agnostic not mean?
Model-agnostic does not mean model-indifferent. Models differ in tool use, output shape, instruction following, safety behavior, latency, price, regional availability, and task accuracy. An adapter can normalize syntax. It cannot make those behaviors equivalent.
Multi-provider access also does not remove concentration risk. The architecture may improve substitutability, but adding a provider can expand contracts, credentials, logging, data-processing terms, service limits, and testing obligations. The right number of providers is a risk decision, not a badge.
The Financial Stability Board's 2024 assessment names third-party dependencies and service-provider concentration among AI-related vulnerabilities, while noting that the effect depends on criticality and substitutability. A second endpoint is not substitutability when both endpoints depend on the same provider-owned context store or untested assumptions.
Which decisions must stay human during a model swap?
Under Pluvo's proposed control framework, named finance and risk owners decide whether the cases represent real work and whether differences are material. They judge the evidence and residual risk, then choose to proceed, narrow, wait, or roll back. A routing layer cannot make those calls.
The evaluation team needs the workflow owner, data or system owner, control owner, technical operator, and the person accountable for the output. For board reporting or controlled processes, add the reviewer who would defend the number. The person who configured the replacement cannot be its only grader.
Finance also decides what cannot move. A model swap should pause when a source population changed during testing, a calculation version drifted, a permission widened, a known incident cannot be replayed, or the new system produces a result that is plausible but unreconstructable. Fluent failure is still failure.
How should a finance architecture separate the durable from the replaceable?
Finance Engineering separates durable business assets from replaceable model infrastructure. Finance Engineering is the discipline of building AI-native finance systems that are accurate, governed, auditable, model-agnostic, and directly tied to how the business actually operates.
Pluvo's public architecture is one worked example of that separation. Its ontology holds entities, relationships, and versioned definitions independently of source tools. The lineage layer preserves source records, transformations, run history, and reconstruction paths. Scoped policy, logs, review gates, and change control sit in the controls layer.
Those are first-party product claims, not independent proof of portability. A buyer can test them: export the definitions, replay the calculations, apply the same permissions, run the preserved cases through a replacement model, and compare the evidence. Finance Engineering discipline puts accountability for that boundary on the company, not the provider.
A model can improve, regress, retire, or simply stop fitting the workload. Context should compound because finance keeps refining definitions, mappings, exceptions, and decisions. The architecture works when the second fact remains true through the first.
What belongs in a Model-Swap Survival Ledger?
Pluvo's Model-Swap Survival Ledger is an original editorial framework and copyable evidence record for a controlled replacement. It names every invariant, its owner, storage location, version, export method, test, acceptance threshold, approver, and rollback path. One row per asset is less glamorous than a model leaderboard. It is also how finance knows what it owns.
| Field | Record |
|---|---|
| Workflow and decision | Named close, forecast, variance, reporting, or analysis use plus the decision it informs |
| Current and candidate model | Provider, model ID, snapshot, API version, region, deployment type, and lifecycle state |
| Invariant assets | Sources, mappings, definitions, calculations, tool schemas, permissions, review gates, lineage, and eval cases |
| Ownership and location | Named finance, data, control, and technical owners plus the governed repository for each asset |
| Export and reconstruction | How each asset is exported, restored, and used to reproduce the last approved output |
| Evaluation | Cases, expected outputs, tolerances, error taxonomy, security tests, cost, latency, and service requirements |
| Change comparison | Material differences in answers, evidence, controls, tool behavior, price, and operating limits |
| Decision | Approve, narrow, remediate, wait, reject, or roll back, with the approver and rationale |
| Monitoring and rollback | Production indicators, first-review date, trigger thresholds, fallback model, and recovery owner |
An illustrative 30-day swap drill
Use this as a planning example, not a standard; adjust the timing to workflow criticality, scope, and required approvals.
- Days 1 to 5: inventory and export every invariant; mark any provider-only asset as a known dependency.
- Days 6 to 12: connect the candidate through a separate adapter without changing definitions, calculations, permissions, or test cases.
- Days 13 to 20: run the six acceptance tests, log every difference, and replay prior incidents and stop cases.
- Days 21 to 30: obtain an independent finance review, decide, monitor a bounded rollout, and execute one rollback rehearsal.
For one practical Finance Engineering control each week, subscribe to the AI Finance Playbook.
Anthropic's retirement schedule puts one model at its migration deadline today. A prepared finance team updates the model ID, runs its preserved cases, and decides. It does not reconstruct the company.
Frequently asked questions
What is a model-agnostic AI strategy?
A model-agnostic AI strategy keeps data, business context, calculations, permissions, evidence, evaluation cases, and operating history outside one provider, then tests whether a replacement model meets the organization's approved performance and control thresholds.
Does model-agnostic mean every LLM is interchangeable?
No. Models differ in tool use, output shape, instruction following, safety behavior, latency, price, availability, and task accuracy. An adapter can normalize interfaces, but workload-specific regression testing must validate behavior.
Which finance assets should survive an LLM swap?
Authoritative source contracts, metric definitions, entity relationships, deterministic calculations, tool schemas, permissions, review gates, source lineage, run logs, evaluation cases, acceptance thresholds, decisions, and rollback records should survive.
Is retrieval-augmented generation automatically portable?
No. RAG separates runtime knowledge from model weights, but proprietary vector stores, embeddings, chunking, ranking, retrieval APIs, and provider-hosted prompts can still create lock-in unless they are exportable and tested elsewhere.
How should finance approve a replacement model?
Run representative cases under production-like conditions, compare numerical integrity, grounding, controls, regressions, cost, and service behavior, then require named finance and risk owners to accept material differences and residual risk with a tested rollback.
What is a Model-Swap Survival Ledger?
Pluvo's Model-Swap Survival Ledger is an original editorial framework for recording invariant assets, owners, locations, versions, export methods, evaluation cases, acceptance thresholds, approvers, monitoring, and rollback during a controlled model replacement.
