Explainable AI in Accounting: Show Your Work

Explainable AI in accounting means deploying AI tools whose reasoning can be traced, reviewed, and defended, not just trusted on faith. An explainable AI system surfaces the data it used, the logic it applied, and the confidence level behind each output, so auditors, CFOs, and regulators can evaluate it the same way they would evaluate any other piece of evidence. Without explainability, AI outputs are assertions without support, which is the one thing an accountant should never accept.

What Is Explainable AI in Accounting?

Explainable AI, often abbreviated XAI, is a category of methods and design principles that make AI outputs interpretable to the people relying on them. Standard machine-learning models, including the large language models increasingly embedded in accounting software, can reach accurate conclusions through processes that are opaque even to their developers. XAI techniques impose structure on that opacity: they surface which inputs drove a prediction, how much weight each variable carried, and where the model’s confidence was low.

In accounting specifically, explainability is not optional. Every workpaper, every journal entry, and every material judgment must be supportable. When an AI flags a revenue transaction as anomalous or a vendor payment as a potential duplicate, the engagement team needs to understand why. The “why” is what separates audit evidence from a black-box suggestion.

The Core Techniques

Three methods dominate practical XAI deployments in finance and accounting:

• SHAP (Shapley Additive Explanations): Assigns each input variable a contribution score for a specific prediction. An auditor can see that a revenue recognition flag was driven 60% by a contract modification clause, 25% by timing of cash receipt, and 15% by an unusual discount.
• LIME (Local Interpretable Model-Agnostic Explanations): Builds a simplified, interpretable approximation of the model’s behavior around a single prediction. Useful for one-off explanations of high-risk transactions.
• Rule-based surrogate models: Train a transparent decision-tree model to mimic a complex model’s outputs. Useful when an audit committee or regulator needs a plain-English summary of how an AI system makes decisions.

None of these methods are perfect. SHAP values depend on assumptions about variable independence. Surrogate models are approximations. The professional obligation is to understand the limitations of the technique used, document those limitations, and corroborate AI findings with conventional evidence where the stakes are high.

Why Explainability Is Now a Professional and Regulatory Requirement

Accounting has always been a discipline of documented reasoning. The move to AI-assisted work does not change that obligation; it raises the stakes.

PCAOB Standards and Inspection Focus

The PCAOB has made AI explainability a live inspection issue. In its 2025 inspection priorities, the Board identified the use of generative AI at public companies and broker-dealers as a formal emphasis area. Under AS 1105 (Audit Evidence), information processed by AI tools must be evaluated for relevance and reliability before it can support an audit conclusion. That evaluation cannot happen if the auditor does not understand how the AI reached its output.

PCAOB inspection reports have documented a specific deficiency pattern: firms using AI-assisted anomaly detection for journal-entry testing without documenting how the algorithm was calibrated, what data it consumed, or how false-positive results were resolved. The PCAOB’s position is that AS 1105 applies regardless of whether the underlying tool is a spreadsheet formula or a neural network. An AI output that cannot be explained cannot be treated as sufficient appropriate audit evidence.

The PCAOB’s QC 1000, A Firm’s System of Quality Control, was originally expected to take effect December 15, 2025, but the Board postponed it by one year to December 15, 2026. QC 1000 requires firms to identify risks to their practice, including technology risks, and design control responses. Approving and monitoring AI tools falls squarely within that framework.

The PCAOB has also been explicit in its public guidance that significant audit judgments must be made by the engagement team, not delegated to AI. Where AI assists in forming a judgment, the file must show the human review process and the basis for the engagement team’s independent conclusion. The PCAOB speech “AI and the Pursuit of Audit Quality” describes this as the central accountability question for AI in public company auditing.

The CFA Institute’s Stakeholder Framework

In August 2025, the CFA Institute Research and Policy Center published *Explainable AI in Finance: Addressing the Needs of Diverse Stakeholders*, a comprehensive review of XAI practices across financial services. Its central finding is that different stakeholders need different kinds of explanations. An audit committee member needs a plain-English summary of what the AI found and why it matters. An engagement partner needs confidence intervals and documentation of false-positive rates. A regulator inspecting the file needs a full methodology description and evidence of human review.

The report recommends that firms develop global standards for measuring explanation quality and invest in interfaces that work for non-technical users, not just data scientists. For accounting firms and finance teams building AI governance programs, the stakeholder-differentiated approach is a practical guide for structuring explainability documentation.

The NIST AI Risk Management Framework

The National Institute of Standards and Technology released its AI Risk Management Framework (AI RMF 1.0) in 2023, and the framework has become a reference standard for organizations designing AI governance programs. It identifies explainability and interpretability as core trustworthiness characteristics, meaning AI systems should be able to show stakeholders how they reach outputs. For accounting and finance teams, the NIST AI RMF provides a structure for categorizing AI risks, designing measurement controls, and documenting governance decisions, all of which feed directly into audit trails.

Building an AI Audit Trail in Practice

An audit trail for AI-assisted work has four components. Each one has a direct parallel in conventional accounting practice.

1. Input Documentation

Record what data the AI tool consumed: the population size, the time period, the data source, and any filters or exclusions applied. If the AI reviewed 14,000 journal entries and flagged 37, the workpaper should show which 14,000, why those 37, and whether the population was complete.

2. Model Description

Describe the AI tool and its methodology at a level that would allow a knowledgeable reviewer to assess its suitability. This does not require publishing proprietary source code, but it does require documenting the type of model, its training basis, any known limitations or bias risks, and the vendor’s explainability representations. For third-party AI tools embedded in audit software, firms should request and retain vendor methodology documentation.

3. Output Evaluation

Document how the engagement team evaluated each AI output, including flagged items that were cleared after review. The resolution of a false positive is itself evidence that the human review was real and substantive, not perfunctory.

4. Conclusion and Sign-off

The final judgment must be attributable to a named engagement team member, not the AI. The workpaper should show the human reasoning that converted the AI output into an audit conclusion, referencing the evidence that corroborated or contradicted the AI’s finding.

Explainable AI for Finance Teams, Not Just Auditors

Explainability requirements do not apply only to audit firms. CFOs and controllers deploying AI inside their own finance operations, for cash flow forecasting, accounts payable automation, revenue recognition decisions, or fraud screening, face the same obligation when those outputs feed into financial statements.

If an AI-generated estimate becomes a line item on the balance sheet, the finance team needs documentation showing how that estimate was produced, what the model’s assumptions were, and how sensitivity was tested. Under GAAP, significant estimates require disclosure of the methodology. An AI model does not change that disclosure requirement; it changes what methodology documentation looks like.

AI-driven accounts payable systems that approve or block vendor payments create a parallel need: an audit trail that shows, for any payment decision, which policy rule or risk signal drove the outcome. Without that trail, internal controls over financial reporting are materially weakened, because management cannot assert effective controls over a process it cannot fully explain.

Firms with AI-native audit practices are beginning to standardize these documentation layers as part of engagement delivery, producing source-linked workpapers where every AI-assisted finding includes the input data, the model rationale, and the human conclusion. The result is an audit file that is both faster to produce and easier to defend.

Practical Steps for Finance Leaders

For CFOs and controllers evaluating AI tools or reviewing their firm’s AI audit approach, five questions sharpen the conversation:

1. Can the tool explain a specific output? Ask the vendor for a demonstration of a SHAP or LIME explanation on a sample transaction. If the tool cannot produce one, explainability has not been built in.
2. What is the false-positive rate, and how is it documented? A tool that flags everything is not useful. A tool that flags too little may be missing real issues. The calibration documentation matters.
3. Who reviews AI outputs before they become audit evidence or financial statement inputs? The human reviewer’s identity, credentials, and sign-off process should be defined in advance, not improvised after the fact.
4. How does the AI tool handle edge cases and low-confidence outputs? Look for a design that routes uncertain outputs to human review rather than defaulting to an answer.
5. How will AI use be described in the audit report or financial disclosures? Transparency to external stakeholders, including audit committees and financial statement users, is the final layer of the explainability obligation.

For companies working with an audit team that uses AI tools, these same questions apply to your auditor’s methodology. You have a right to understand how AI outputs became audit conclusions, and your auditor has a professional obligation to be able to explain it.

Related reading: Evaluate AI Accounting Software: A CFO’s Vetting Guide

Frequently Asked Questions

What is explainable AI in accounting?

Explainable AI in accounting refers to AI tools and methods that surface the reasoning behind their outputs so that auditors, finance teams, and regulators can evaluate, document, and defend those outputs under professional standards. The goal is to make AI-assisted conclusions as traceable as any other piece of audit evidence or financial estimate.

Does the PCAOB require auditors to document how they used AI?

Yes. Under AS 1105, auditors must evaluate the relevance and reliability of information obtained or processed using technology-based tools before using it as audit evidence. PCAOB inspection reports have cited firms for using AI anomaly detection without documenting calibration, data sources, or false-positive resolution. QC 1000, effective December 15, 2026, requires firms to govern AI tools as part of their quality control system.

What is an AI audit trail in accounting?

An AI audit trail is documentation that records: (1) the data an AI tool consumed, (2) the methodology it used, (3) how the engagement team evaluated its outputs, and (4) the human judgment that converted those outputs into an accounting or audit conclusion. It is the AI-era equivalent of a conventional workpaper.

What are SHAP and LIME, and why do they matter for auditors?

SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-Agnostic Explanations) are two widely used XAI techniques that explain individual AI predictions in terms of input variables. For auditors, they provide a mechanism for satisfying AS 1105 documentation requirements: the auditor can record which factors drove an AI flag and by how much, then document the human evaluation of those factors.

Do CFOs and controllers need to worry about explainable AI even if their company is not publicly traded?

Yes. Any AI output that feeds into a financial statement, estimate, disclosure, or internal control is subject to documentation requirements under GAAP and, where applicable, lending covenants, investor agreements, or regulatory filings. The explainability obligation applies to the finance team deploying the AI, not only to the outside auditor reviewing it.

How does explainable AI relate to the NIST AI Risk Management Framework?

The NIST AI RMF identifies explainability and interpretability as core characteristics of trustworthy AI systems. For accounting and finance teams, the framework provides a structured approach to categorizing AI risks, measuring model behavior, and documenting governance decisions, all of which support the audit trail obligations described above.