Shadow AI, Deepfakes & Payables: Controls That Lower Your Total Cost of Risk

By James Benham, Co-Founder and CEO, JBK

The riskiest moments in payables rarely look dramatic. They arrive as routine change requests, an updated remit to address, a new routing number, or an urgent wire that cannot wait. In 2025, those requests are increasingly polished by generative AI. Messages are drafted by models that mimic a supplier’s tone. Executives appear on video with cloned voices and faces. This is not hypothetical. The FBI recorded $16.6 billion in reported cyber losses for 2024 and highlighted business email compromise as a persistent driver of investigations. A high-profile example shows the new reality. Arup, a UK-based Engineering group, confirmed that a Hong Kong employee was deceived by a multi-person deepfake video call that led to transfers totaling about $25 million.

Finance teams cannot predict the next ruse. However, they can make fraud uneconomical. The path is a short list of disciplines that directly interrupt how attacks succeed. When these controls are implemented and evidenced, they help brokers and carriers price risk with fewer unknowns at renewal, which lowers the total cost of risk.

Identity

Multi-factor authentication remains table stakes, yet many payables workflows still rely on factors that are easily relayed. Hackers using “man-in-the-middle” tools can steal your login session tokens and get past basic one-time codes, so text-message and simple push-notification logins aren’t safe. The durable improvement is phishing-resistant authentication with FIDO2 security keys, and passkeys that bind the login ceremony to the legitimate domain and hardware. The Cybersecurity and Infrastructure Security Agency urges organizations to adopt phishing-resistant methods as part of modern zero-trust programs and provides practical implementation guidance. 

Priority surfaces for payables are straightforward: single sign-on, corporate email, ERP and AP, banking portals, and administrator consoles. Block legacy protocols that bypass MFA and require step-up authentication for high-risk actions such as adding payees or changing bank instructions. The Verizon Data Breach Investigations Report reveals that the human element dominates breach patterns, indicating that raising the cost of initial access yields dividends across the entire attack chain. 

Verification

Every large wire-fraud case shares the same pattern: attackers try to change the process inside the same channel where trust was established. The control is to separate reception from validation. If a request to change payment instructions arrives by email or via a portal message, confirm it through a different, pre-established channel taken from the vendor master or an authenticated portal with MFA. Maintain dual control so the person who verifies is not the same person who initiates or approves.

Before either party changes bank details or acts on payment instructions, require verification over a second, independent channel (for example, a phone call to a known number on file). Do not rely solely on email, SMS one-time codes, or simple push prompts. The timing also aligns with ACH changes. NACHA’s Credit Push Fraud Monitoring rule changes take effect in 2026 and will require risk-based monitoring by originators, Originating Depository Financial Institution (ODFIs), and Receiving Depository Financial Institution (RDFIs). Programs that use out-of-band verification and keep clean audit trails will be better prepared.

Governance for AI

Shadow AI spreads because it removes friction. That convenience can create untracked data flows—when invoices, contracts, or banking details are pasted into chatbots, or when assistants get broad access to mailboxes and shared drives. Start by consolidating access to approved tools behind Single Sign-On (SSO) and Multi-Factor Authentication (MFA). Add data-handling guardrails with Data Loss Prevention (DLP) for uploads and copy/paste. Limit what models and agents can read or do in Enterprise Resource Planning (ERP), Accounts Payable (AP), vendor portals, and storage, the principle of least privilege.

Use the NIST AI Risk Management Framework to map use cases, risks, mitigations, and reviews. Reference the OWASP Top 10 for LLM Applications for concrete threats such as prompt injection and data exfiltration that now apply to finance assistants and document bots.  Major vendors and community research continue to publish patterns for defending against indirect prompt injection in mail and document agents, including human-in-the-loop checkpoints for potentially risky actions and telemetry for audit and forensics.

Many organizations have allowed AI assistants to scan broadly across mailboxes and shared drives and, in some cases, to modify vendor records or initiate payments. That turns one poisoned input into a business logic breach. Containment is both feasible and necessary. Separate retrieval from action so that answering a vendor inquiry does not grant the ability to alter the vendor master. Keep connectors in read-only modes by default. Require explicit consent and human review for any operation that could move money or change payment instructions. Treat all external content and attachments as untrusted until sanitized. Log prompts, tool calls, and outputs so anomalous access to banking or vendor entities can be investigated, and so you have the evidence carriers request during underwriting or claims.

Ransomware crews now go after backups first. Improve resilience by keeping immutable, offline-capable backups of identity systems, ERP and AP databases, document repositories, and test restores regularly. Federal #StopRansomware guidance stresses immutability and recovery drills because they cut downtime and claim severity. 

Follow the 3-2-1-1-0 rule: 3 copies of your data, on 2 different media, 1 off-site, 1 immutable or air-gapped, and 0 restore failures in testing.

Why this matters to insurance

These disciplines are observable. Brokers and carriers increasingly price on evidence rather than slogans. Suppose you can demonstrate phishing-resistant MFA on the right surfaces, provide sample records of out-of-band verification for vendor changes, prove scoped access for AI tools with DLP and audit logs, and produce reports from successful restoration drills. In that case, underwriters have fewer unknowns to price. Market data supports the value of control maturity. Marsh reported stabilization and later decreases in cyber pricing through late 2024, further softening into 2025 as capacity returned and insureds strengthened controls. Coalition’s 2025 claims analysis found ransomware remained the most costly category, but noted stabilization relative to prior spikes. 

What Payables Leaders Should Do Now

Criminals will keep evolving, and AI will make their stories smoother. Don’t chase every new trick; rather, raise the bar where fraud has to pass. Use phishing-resistant sign-in methods to block account takeovers. Confirm vendor changes through a second, pre-agreed channel and keep an audit trail. Limit what AI tools, models, and assistants can see or do, and design access on a least-privilege basis to contain mistakes. Keep immutable, offline-capable backups and practice restores so an extortion attempt is a headache, not a crisis. Do these well and you shift your loss curve, improve your terms, and move from constant triage to steady control, even as headlines chase the next deepfake.

Share This Article: