By Mela Seyoum, in this Financial Advisor IQ , featuring Shane Cummings, CFP®, AIF®, Wealth Advisor & Director of Technology/Cybersecurity
Key Takeaways
- AI chatbots can provide incorrect or incomplete answers to financial questions.
- Errors become more common as financial scenarios become more complex.
- AI can be useful for general financial education and advisor productivity, but its output requires review.
- Context, calculations, risk considerations, and current information can affect AI-generated results.
- Advisors should understand how AI-powered tools work and periodically verify the technology behind them.
- AI adoption doesn’t eliminate the advisor’s role in reviewing and contextualizing information.
Why AI Chatbots Get Financial Questions Wrong
The most popular AI chatbots on average got 57% of financial advice questions wrong, a recent study found.
As financial advisors and clients alike increasingly use artificial intelligence, advisors should be wary of the limitations that large language models in particular have when it comes to delivering financial advice.
A recent study from Saturn, a London-based AI and technology provider for financial advisors, found that on average, across a variety of large language models and difficulty of questions, the models made mistakes 57% of the time.
On questions categorized as hard, models made mistakes 88% of the time, according to the study.
To test the models, Saturn had a paraplanner run 121 money-related questions through 18 free and paid AI models, including ChatGPT, Grok, Claude and CoPilot. The questions had a range of difficulty from general financial literacy to more complex multi-part scenarios with each question being run five times and tested against the paraplanner’s criteria, according to the study.
How Clients Are Using AI for Financial Questions
The question of AI’s accuracy on financial advice can pose challenges to advisors, especially as some clients begin to use these popular chatbots on their own time, potentially asking questions about their advisor’s recommendations, which they might not state outright, said Jacob Tally, financial advisor at Prospero Wealth.
For clients, using AI can be a “double-edged sword,” Tally said.
He added that in those cases, he might be able to tell that a client has been using AI based on the types of questions they have and has asked clients to share what information they have so he can review it and correct any mistakes or misunderstandings.
While clients using AI may create more work for the advisor, it can also be an opportunity to build more trust with the client, Tally said.
These chatbots are most helpful to clients for getting a better understanding of some general finance-related topics, but getting specific recommendations from it “opens up a big can of worms,” he said.
However, most adults aren’t blindly accepting financial recommendations from AI, according to a recent New York Life study, which found that only 23% of the 2,278 adults surveyed accepted the advice and went forward without seeking additional validation.
Shane Cummings, wealth advisor and director of technology and cybersecurity at registered investment advisor Halbert Hargrove, added that the firm’s younger clients tend to be more interested in using AI.
The New York Life study found that Generation Z was the most likely to use AI for financial advice, but 40% of the Gen Z adults surveyed also stated that having access to AI tools made them feel a greater need for a financial professional.
While the capabilities of AI have progressed significantly in a short amount of time, Cummings is still optimistic that AI won’t replace advisors, though he suggested that it might change how they give advice.
How Advisors Are Using AI in Their Work
Clients are not the only ones using AI. Advisors are starting to use these large language models as well, and when they’re doing so it’s important to thoroughly review any output from them, Prospero’s Tally said.
For example, as Tally was analyzing a scenario for a Roth conversion for some of an early retiree’s retirement assets, he wanted to build a customized tool in Excel and used Claude to help.
While it saved him a significant amount of time, there were many errors. Since he knew what to look for, he could detect where the model went wrong and fix it, and overall he still saved some time. But for clients who might try to do something like that on their own, they likely wouldn’t catch all those mistakes, Tally added.
“Don’t throw the baby out with the bathwater. It’s a good productivity tool, but you can’t take the results or the output of some of these tools and LLMs at face value,” Tally said.
What Are the Limitations of AI Financial Tools?
The Saturn study found that the most common mistakes LLMs made were leaving out critical information, citing wrong numbers and missing risk warnings.
Those errors are consistent with Tally’s experience using LLMs and are something he doesn’t necessarily see going away, since the objective of many AI tools is to create an affirmative output to the user that may not include all the necessary information.
Furthermore, the Saturn study did not take into account any background information about a particular scenario, type of client, tax codes or other limitations, which colors the results, Jesse Rosato, chief technology officer at Nitrogen, provider of an AI-powered suite of tools for advisors, said in an email to Financial Advisor IQ.
“The gap is context and tools, not intelligence,” Rosato wrote.
Toby Wade, co-founder and chief executive officer of DeepVest, provider of an agentic wealth management platform, added that the LLMs are not built for mathematical use cases, so having them perform calculations is not the best application.
“You don’t want to tell a client one day their portfolio risk is 20% and the next day it’s 10%,” Wade said.
He added that AI can also have problems reproducing the same answer for the same question, which can bring its accuracy into question.
Why Advisors Need to Verify AI-Generated Information
It’s also important for advisors to understand how any AI-powered tools are using LLMs and which models are being used, Halbert Hargrove’s Cummings said. That information will also need to be verified periodically, as any given tech vendor’s LLM of choice may change.
Cummings said he has seen both client and advisor perspectives on AI vary widely from disinterest and skepticism to eager adoption.
While there’s no backtracking when it comes to whether or not to use AI, the role of verifying the information it produces will become increasingly important, Cummings said.
