The most important safeguard in an AI deployment in regulated finance is often the person reading the output. Workers at more than 90% of companies already use personal chatbot accounts for daily tasks, while only about 40% of those companies pay for an official tool. Some lenders are treating AI literacy as a control alongside model governance. They train the people closest to a credit decision to interrogate whatever the model hands them, and regulators are starting to expect exactly that.
Prabhu Krishnammagari is the Head of AI Programs & Solutions at Finance House, a UAE-based financial institution. He joined in January after serving as Assistant Vice President, AI/ML and Site Reliability Engineer at Bank of America. He wants risk teams to challenge AI models rather than simply adopt them.
"I'm framing risk teams not as adopters of AI, but as challengers of AI. To become a challenger, a person has to understand what the model does," says Krishnammagari. In his view, the underwriter should always retain the final call on a credit decision, and training should make that judgment an active check on the model rather than a formality.
Governance has to start before the first login
Krishnammagari's view is that restrictions on AI are not enough on their own. "AI is now widely accessible, and across the industry people are already using it in their daily work. Restrictions alone are insufficient. Organisations also need approved tools, clear policies and employee education," he says.
In his view, data residency and the redaction of confidential information are foundational, but they are one layer of protection rather than the whole answer. Governance, clear policies and human oversight have to sit alongside them. Regulatory guidance in the UAE points the same way, naming effective human oversight as a core principle for financial institutions.
Technical controls are only part of the answer. The rest is people. "Deployment and literacy have to go together. Only when AI literacy goes up can people challenge or question the output," he explains.
A probability score is only as safe as its reader
For predictive models, his training approach begins with confidence scores and the thresholds that turn a probability into a decision. A model that returns a bare yes or no is a black box, and regulated lenders already answer to explainability requirements and model risk management frameworks. "When a model proposes the probability of something happening, the person looking at it should be able to construe what that probability means and where the AI could have gone wrong," Krishnammagari says.
Literacy also means knowing enough about training data to spot trouble before it reaches a borrower. "If someone builds a predictive model for a newly launched product, a good risk team should ask how the data was gathered. The quantity of data needed to train that model often isn't there yet," he adds. That question marks the difference between an engaged reviewer and a passive human overseer who clicks approve on whatever the system recommends. Enterprise AI policies often name a human in the loop without defining what that person is actually expected to catch. Krishnammagari's approach aims to write that job description around the specific ways models fail.
Fluent answers deserve the most suspicion
With generative AI, the risk Krishnammagari worries about most is how convincing a wrong answer can sound. "The number one risk with generative AI is 100% confident wrong answers. Someone who doesn't know an AI can be confidently wrong will take the output as correct. Someone who knows the wording can sound certain while the answer is silently wrong will challenge it," he says. Deloitte Australia learned that lesson publicly in 2025, partially refunding the government for a report containing a fabricated court quote and citations to research that didn't exist. The episode illustrates what happens when AI-generated claims make it through a review process without someone testing whether the underlying evidence exists.
Krishnammagari's training leaves stochastic gradient descent to the data scientists and focuses on the conceptual picture of how models get built and what data feeds them. "Once a risk professional understands the true nature of AI, that's when they'll be able to challenge it and verify the output at every step instead of blindly following what the AI said," he explains.
Engineering discipline cuts cost and contamination together
That literacy extends to the people building the systems, where Krishnammagari applies the same emphasis on understanding failure modes. In site reliability engineering, nothing ships until someone can see how it breaks, and Krishnammagari holds AI to a stricter version of that rule. "You should be two or three times more disciplined from a software engineering perspective when you're building AI than when you're building a normal software product," says Krishnammagari.
Much of that discipline concerns what goes into the model. Teams often dump every available document into a prompt for fear of missing something, a habit he considers counterproductive. Deterministic parsing and minimal inputs protect the context window, the part of the system most worth defending, and the approach mirrors the wider push toward enterprise context engineering. "When we give a byte of information to a model, we have to understand whether it makes sense or whether it's context contamination," he says.
He expected cost efficiency and engineering rigor to pull against each other, but found that the two could converge once a roadmap moved past its first steps. Compact, structured inputs lower the capability bar, so a model with half the power of a frontier release can often handle the work. Leaner models also change how value gets counted, at a moment when only 5% of companies report AI value at scale and headcount reduction has become the easy yardstick. Krishnammagari measures decision quality instead. "Underwriters make credit decisions a certain way. I'd rather have AI look at the same case through a different lens and add value. That's not something you measure in headcount," he says.
UBS has already turned AI fluency into a hiring screen for its junior intake. A more substantive test would reward candidates for catching the model in a mistake, a skill that looks unremarkable on a résumé until an examiner asks who reviewed the output. Krishnammagari prepares risk teams for that moment by demystifying AI. "Generative AI isn't magic. At its core, a language model is predicting the next word," says Krishnammagari. "I'm putting it plainly and bluntly."
The views and opinions expressed are those of Prabhu Krishnammagari and do not represent the official policy or position of any organization.
