AI governance in finance: What CFOs need to know

Bruce Mullan:
"How many AI systems do you think you're using?" Sometimes they'll say three, sometimes they'll say four. And I say, "Well, what about Teams? That's got a meeting notation process in it. Word has got help in that. So when I start saying, "What about, what about, what about?" three becomes 30 very, very quickly. And what that means is for organisations is that all of a sudden we've got this AI capability that no one has really reviewed, vetted, nor approved and people are using it without training or without supervision.

Elinor Kasapidis:
Welcome to CPA Australia's With Interest podcast. I'm Elinor Kasapidis, Chief of Policy Standards and External Affairs here at CPA Australia. AI is already embedded in finance software, business systems, and the tools employees use every day. It's helping people analyse information, prepare reports, automate routine tasks, and make decisions faster.

But organisations still need to answer some key questions. Where is AI being used? What information is being given to it? Who is checking the outputs? And how is it genuinely improving decisions. To explore this more, I'm joined today by Bruce Mullan from AI Governance Partners. Welcome, Bruce.

Bruce Mullan:
Oh, thank you for having me, Elinor. It's a pleasure to be talking with you today.

Elinor Kasapidis:
Thank you so much. So you're in AI, you're immersed in it, and we're hearing a lot about how AI is transforming finance. What are you seeing in organisations today, but also how are organisations thinking about the future?

Bruce Mullan:
Well, the best way to look at AI usage or adoption in organisations is to break it down into three levels. So at the lowest level, it's really around those administration tasks where people are generating PowerPoints, emails, letters, correspondence, spreadsheet formulas, those kinds of things that are administrative, and that's where the AI is actually supporting how people do their work. So that's the lowest level.

The next level up is business processes and how AI is augmenting those processes. So this might be things like prepayments, accruals, generating budgets, routing workflows to people for approval, and so on and so forth. So that's the second level. The third level is where AI takes over and makes decisions, and that's the highest level. So why I've represented them as three levels is because at each level it gets harder and more difficult to implement. And so most companies and most CFOs are really at that first level.

They haven't gone into the automated processes, nor have they gone into, heaven forbid, automated decision making. So a lot of CFOs tell me that they feel like they're behind, but they're just as behind as everyone else. So everyone's kind of in that first level where AI is individually supporting their work. And so what I would encourage CFOs to think about is that in the future, you might have a team now of 10 people or 20 people in a finance team, that same finance team of the future might have 12 people with a hundred agents helping them do their work, or a team of 10 might have four people with 50 agents helping them support their work.

And the way AI is helping enable organisations do more with less will be quite profound. So CFOs really need to start thinking about what that future looks like and what are the steps they're going to need to go through to transition to that change because it's going to be quite a dramatic change. It'll probably get here quicker than they think, but it's not far away.

Elinor Kasapidis:
Thanks, Bruce. And breaking it down into those three stages, I think that people are starting to think about that process piece, the deep integration and the automated decision making is something for the horizon. But you've also argued that many organisations are actually using AI a little bit more than a CFO or a leader might realise. Why is there a gap, I guess, between some of the perceptions of leadership and the risk areas and the reality of the employees across the organisation?

Bruce Mullan:
Yeah. So when I meet with people, particularly boards and executive teams, CFOs, et cetera, I ask them, "How many AI systems do you think you're using?" And sometimes they'll say three, sometimes they'll say four. And I say, "Well, what about Teams? That's got a meeting notation process in it. Word has got help in that. What about Adobe where it's generating backgrounds for images?"

So when I start saying, "What about, what about, what about?" three becomes 30 very, very quickly and a large, medium-sized business might have three or 400 systems. Now, all of the vendors around the world have been gradually incorporating AI into their products. A lot of people didn't ask for it. And what that means is for organisations is that all of a sudden we've got this AI capability that no one has really reviewed, vetted, nor approved and people are using it without training or without supervision.

So that's in the embedded side of the systems that people are using already. The other part of the AI that's really critical is the personal productivity tools. In 2022 when ChatGPT first was released, it's free and people started using it for personal things, but then they started using it for their work. And then Claude came along and Perplexity and Copilot and so on and so forth. So people have been using these tools for quite a few years now.

And so statistics show that somewhere between 60 to 80% of people use these personal AI tools for work purposes without their employer knowing about it. So AI is already influencing your organisation. It's just a question of how it's doing it. Now the biggest statistic in that, and this was the Melbourne Business School research, was that 48% of people are doing this knowing that they're deliberately breaching a company policy.

So when people are using AI for company purposes, the incentive for the employee is great, I save money, I can look good, I can save time, I can get my work done, but the risk sits solely with the employer and with the CFO particularly. So you've got the data breach, you've got the privacy breach, you've got cybersecurity risk, and you've got the risk of an embarrassing mistake, which there's been plenty of those.

I know a company, an ASX listed company in April this year said," Great, we'll give Copilot to everyone because it's a fabulous tool. We think people can really use it." They switched it on and what they didn't realise is Copilot had access to all the employee payroll data inadvertently, so anybody could find out what anybody else's salary was. And so that's why the guardrails are important for that you just can't switch these tools on.

And the implications of that obviously is a privacy breach, you've got legal implications and compensation for people, you've got remediation, you've got disharmony, all those sorts of things. So it's really important that CFOs in their role start thinking about how is your team using AI and how should they be using it?

Elinor Kasapidis:
You touched on a lot of elements there. So there's the employee putting confidential information, commercial and confidence out into the public domain. There's also not having your IT system set up internally when you switch on the tools to actually secure access according to permissions, as well as the embedded AI in software that people are using and maybe not being aware of some of how those agents are being switched on and used. It's a lot. It's a lot.

So how do you see, whether it's a CFO or even at the board or the executive level, because I see a tension between employees, they just want to do the best job as fast as possible and they know the tool is out there. And for the younger generation, they're growing up as AI natives. They expect the employer to give them access to these things. At the same time, there are some serious consequences about getting it wrong, so an organisation may move too slowly for the employee. So how do we navigate that tension? What are some tips or tricks there?

Bruce Mullan:
Well, just looking at from the employee perspective to begin with, when I mentioned earlier that those three levels, so you've got the administration task, you've got the business processing, and you've got automated decision making. Each of those three levels of systems have got a shadow AI equivalent. So they can do all of those three things themselves in their own tools under a personal login. So the incentive for the employee is that it helps them produce great work or what they believe to be great work.

And we all know that ChatGPT comes with a warning, Claude comes with a warning, Perplexity has a warning, "This information may not be correct. It needs to be verified." So in the interests of taking the risk of putting confidential and commercial and confidence information into these tools, the employees are taking that risk on for the organisation. Now, if you ask an employee, are you using AI, what do you think they would tell you?

Would they tell you they are or they aren't? Would they be honest about their use of it? And it's the concealed use that CFOs should be most worried about because the inputs could be confidential IP that's going out into the world or it could be private data that's exposed for a privacy breach. The outputs could be incorrect, misleading, and then they form part of the information chain.

So when that information goes from that particular person's work into the next stage, which it might be a management report, then that's reviewed by the CFO, that goes to the executive, goes to the board, someone makes a decision and all of a sudden the information that they've been relying on is incorrect. And because AI, to some extent, wants to please us when it's producing information, it gives us the confidence that the information is accurate and that they don't actually need to check it. It's the garbage in, gospel out fallacy. So that's why it's so important that CFOs get on top of what people are doing in these tools with company information.

Elinor Kasapidis:
Which is a great segue to the next question, which is around that cascading consequence and garbage in, we tend to say garbage out, but I love the gospel out phrase. What's an example of that and how do you end up with this cumulative impact from a flawed AI output that was not disclosed by the employee?

Bruce Mullan:
So let's say we've got Chris. Chris is a financial analyst. He works in finance and the CFOs come to Chris says, "Chris, we're looking at an acquisition. I want you to find out all this information about our business, the company that we're acquiring, put it together and build me a financial model." Now, if I was doing that, I would be grabbing data from all the department heads about our forecast revenue, our costs from payroll, from salary perspective and procurement about what we spend. And I would gather information in the marketplace from economist reports about what's happening broadly.

And also I would go out and gather public information about the company we are acquiring. I would put all those documents into a folder and they would ask AI over the top, "Tell me what this looks like. Give me a financial forecast." Now, if I'm using AI to generate this and I don't quite know how to phrase my prompts correctly or establish my assumptions, and it will then read through all those documents, create a spreadsheet and create a financial model for you, but it may not declare the assumptions that it's making.

I'll give you an example. So the way sales recognise revenue and the way finance recognise revenues, one could be a cash basis, one could be an accruals basis. Now, if AI looks at that data and sees a conflict, unless you tell it to come back to me if you've got any conflict, it's going to assume one or the other and it might pick sales. So you might be forecasting on a cash basis and all the way up, that then becomes a decision from the board to acquire.

So you might spend 15 minutes generating a financial model, just making it up for a $5 million acquisition and that data that you are relying on is incorrect. So a good practice would be to set up and establish an assumptions table and therefore then the AI would then be referring to those assumptions and you would then ask it to tell you where any of those assumptions are in conflict so that you can...

The three things that you want is that you want the transparency of where AI's being used and some sort of audit about how it got to its decision that it's made and the explainability of the decision. So you need to be able to explain that model to somebody if anybody asks. And they're the three things that are really, really important. And this is where organisations can save time at the expense of getting it right.

Elinor Kasapidis:
If I go back to 10 years ago where junior analysts might put all of this into a spreadsheet and due to their lack of experience and knowledge might make exactly the same mistake, is there a difference or is it really around transforming the current process that you might have for quality assurance over human produced work into a more process-based thinking of the agent and what its potential to make errors is? So is it easy to change or is it a brand new kind of set of rules given that these errors can occur by humans?

Bruce Mullan:
The principles are the same. So when we used to build financial models from first principles back in 30 years ago, we would start with building our assumptions first. What are we assuming in this before we started building the model out? And that step might get missed now. So it's about best practise of how do you build a financial model, not so much whether it's AI or a spreadsheet or what tool it is. It's what are the things that we can do to make sure that we build in the quality and the understanding of what we are doing and what we are trying to achieve.

Elinor Kasapidis:
Therefore, with that example, was it a governance failure or an AI failure at the end of the day once you start to have these ramifications?

Bruce Mullan:
Governance, absolutely. Because you can make the same mistakes in a spreadsheet. You can have a spreadsheet with weak assumptions or poor modelling. It's just the choice of tool or weapon of choice is somewhat irrelevant. It's the guardrails that sit around that process to make sure that if you are a CFO and somebody's given you the outcome of a modelling calculation, how can you defend that if you need to defend that later on to make sure that I can explain it, it's transparent and we can fully audit? How do we get to that point? And that's the difference now with AI.

Elinor Kasapidis:
And you're probably just getting there a bit faster. So I can imagine from a CFO, the team's turning things around in a day when it used to take a week. And so it's really speeding up, I think that decision making pressure. Can you see how the new world of good decision making? So it's not necessarily about a productivity improvement, even though that's there, but how can these tools and as we build them and the teams use them improve decision making for the CFO and the board and the executive?

Bruce Mullan:
Yeah. So I think the enemy of every CFO is time because that role has changed. So in the progress of a career from, you mentioned the junior accountant role and the financial analyst role right through to becoming the CFO, for most of your career, you are nose down in a spreadsheet or nose down in the data. And what CFOs are being asked to do now is far broader than the work that they used to do. And making that transition is really important.

So it's the time they need to do their jobs well because what they're doing now, they're building relationships, they're working with the business, they're looking at future opportunities to scale, to acquire, to divest. They're not working on spreadsheets anymore. They're working with external stakeholders. They're working with regulators. So these things take time out of a CFO's day. And as a CFO, you need that time to be able to put a board paper together and to make sure that you can be ready for all of the questions you are going to get.

So that preparation is really, really important and you need to have that time to reflect, think and ponder. Now, the beauty of what the AI does for both the CFO in that role and the people in the finance team is it gives them their time back to do that context checking, to do that sanity checking of what they're producing, to look at other possibilities, to apply the sniff test as we call it, to make sure whatever they're producing is of high quality and that there are no mistakes in it. And it's not a productivity shift in the sense of, "Hey, I can produce more per hour." It's using that extra time to do the things that they never had time to do, which was the checking.

Elinor Kasapidis:
And you mentioned the board and the CFO preparing these insights and presenting information. Now let's say AI generated analysis becomes embedded and it comes from lots of different places, lots of different agents and humans are involved. The decision goes up to the board, it gets approved and it's the wrong decision. Where does accountability lie in that chain of command?

Bruce Mullan:
Well, there's four parties involved. There's Chris who was the financial analyst, there's the CFO, there's the board and there's a software vendor. And one of the things that's important, and I've learned this in my career, it's not whose fault it was, it's who gets the blame. So in that case, it's going to sit squarely with the board because of the Corporations Act around their fiduciary duties.

And the software vendors, well, it's a tool. You've used it. Hey, you've chosen to use it. That's not up to us. It's like a screwdriver. I can use a screwdriver to do all sorts of things, but it's just a screwdriver. So the board will be responsible for that and directors. So they need to know when something lands on their desk, how much AI was incorporated in this, how was it checked and how confident I can be in relying on that.

Now, I'll give you a really simple example. If Chris, who's the financial analyst, out of every document that Chris produces, it had a disclaimer at the bottom and says, "This was 50% AI generated." How do you think that would make Chris feel? What would the result be if he had to do that? And that is essentially the transparency that we're asking employees to tell us, but I'm not sure Chris wants to share that because a lot of the Chrises that I speak to don't want people to know how they're coming up with this stuff.

Elinor Kasapidis:
It's a great open question. And also it's around where does the value sit? Because we talk a lot around human in the loop. The board will turn then to the CFO to say, "Well, you gave me this." And then the CFO will turn to Chris, the analyst, go, "Well, I asked you the right questions." Chris will say, "Well, yeah, sure. 50% of it came from AI, but I was the one prompting. I was doing the checking." There's a bit of moral hazard. He could probably turn and say, "Well, it was 50% AI. It's not my fault." Or he could take accountability and say, "The important parts of what I was doing."

So I guess I'm hearing though a little bit of over governance or uncertainty about, well, if the board's asking the same questions that the CFO's asking, then poor Chris is like, "Well, is it better that I just do it myself, that I don't unlock these time savings or the benefits of developing an agent which could actually make me better, actually ask me better questions about my analysis or suggest new things?" And you don't want to repeat the same things all the way down the chain because then you unlock no product, you don't unlock any productivity. So how do you see that tension?

Bruce Mullan:
Well, I think what the CFO needs to do is to create the conditions for Chris to be able to use AI appropriately in that team and to use it openly. And that is a leadership and a culture question about how do we use AI and use it appropriately in our team and how do we be open about it? And if Chris is using these shadow AI tools to get the results that Chris feels he needs, well, the need is there.

So these tools are really powerful and it's exciting because as a junior accountant, I would never have had this capability. So it's pretty exciting now for somebody to have these tools and to do such a great job and to have these tools help them do their job better, but they have to do it safely and responsibly. And so that's where the organisation should say, "Don't worry, Chris. If you help work with us, we can provide the tools that you need to do, but within a safe environment."

So you want to create the conditions upon which people are open and honest about their AI use. Now, how would you do that? If you suspect somebody is using AI as part of their job, how would you find out if they're doing it or not? What process would you use as the CFO? Well, I just refer to my children. If you go to a child and say, "Okay. Who ate the cookie out of the cookie jar?" They're all going to say, "No, it wasn't me. It wasn't me. It wasn't me."

So asking someone directly, "Do you use AI?" is confrontational, it's abrasive, it's judgmental. It's all of those things that the employee then thinks, "Oh, I'm in trouble now. Maybe I'll just say, 'No, I didn't and I did it myself.'" But a better way of doing that would be to say, "Hey Chris, look at that modelling you've done. Tell me how you went about it. What assumptions did you make? How did you get to that recommendation? What were the challenges you found as you were doing it to come up with that? Chris, can you show me your assumptions table, all of that and walk me through it?" So taking the employee on that kind of a journey, you'll know pretty quickly.

If they understand the content, if they understand what they did and why they did it, then you know you're okay. But if they have to defer to the AI, you've got a problem and you'll know straight away, but you have to use open questions in a way that makes Chris comfortable to share. And that's usually not by asking him directly, "Did you use AI for this?" That's really not helpful.

Elinor Kasapidis:
We've spoken to many members across Asia Pacific who are on different stages of the journey, and I think the culture piece and the safety for the employee to own up, to actually experiment within an environment and providing the sandbox. And what they've actually found is that by enabling their people to experiment, to build, to try different things, they've created new products, new improved processes.

There's now, it's started a discussion around re-architecting job roles, processes, because as people start to think around how do I integrate AI, problems start to be solved and there is this real benefit, but you're right around, it's a mindset as well, isn't it? The CFO can't say, "Well, do the same thing. Tell me how you use AI, make sure that it's no worse than you, and I'm going to interrogate everything." So governance and culture kind of go together, don't they?

Bruce Mullan:
Absolutely. Absolutely. I think that we're past the point of no return now. People are using it and if you try to drive it underground or try to expose it, you'll probably just drive it further underground. So this is where the CFO needs to put on their thinking cap and work out, how do I get the information I need to know that my team are doing things responsibly? And it's about asking the right questions and not being the fun police or the no police because the worst thing for Chris would be, "Don't take away my tools."

Elinor Kasapidis:
Exactly.

Bruce Mullan:
That's what Chris doesn't want, right? So he wants to use those tools and it's great that he does, but he's got to use them safely and responsibly.

Elinor Kasapidis:
And just commercially as well, you've got competitors who are giving their employees the tools that are creating the right environments that do have that secure space, both from a security point of view as well as a cultural point of view, and they're going to race ahead if you stay behind. So I think, like you're sort of saying, I think many people are at that administrative stage.

They're now getting into that second sort of process stage. Automated decision making, so we see it evolving quite quickly, particularly in Asia, and we've got this sort of thing around how do the agents from different applications as well as those that are created by a business interact? So where you have the 20 agents, each of them are making decisions and the team or the CFO is getting that aggregated combined output at the end. What does governance look like in that space?

Bruce Mullan:
Yeah. And when these agents are operating, they're operating at scale. So it's not just one transaction that they're doing. They're doing hundreds of transactions at a time. So you need a way of monitoring your agents and alerting you when something's outside a boundary or an exception has been generated. So the new way of working will be around managing by exception. And that's a concept that's been around, I think since the agile days of 15 years ago, I first heard this concept of managed by exception.

And it's never been more important than now because I did an invoice automation project a year or two ago where we took 50,000 invoices, we automated them, the data capture, the routing, the coding. We didn't get to the approval stage of automation, but we did get to the stage where we went from 35 days from receiving the invoice to the date the invoice was actually approved down to less than seven days.

And we went from 57% on time payment to 92%. Those numbers are frighteningly amazing. And that just turned that whole procurement function around completely by having really good data. So it freed the time up to be able to look at the spend, not process invoices to achieve the spend. So the agents that were doing that, they're processing transactions in nanoseconds. So once the invoice hits the inbox, it picks it up and it's within the system within about five seconds.

So it's impossible to have a human in the loop looking at every transaction when you're dealing with 50,000 invoices. And what I found was the team, when we were first reviewing the work that was coming through, they were fatigued. They got tired from ticking and tying things as they were coming in because when you're trying to keep an eye on these agents, you've got to work out what should you be looking for and not trying to look at every transaction because it's always a risk equation.

Now, if you don't want any risk, the only way to do that is to eliminate it and not use AI, but you won't be able to do that. So you've got to manage the risks. So you've got to work out what are those things that you need to look for and what's the most important things to check and provide the tools for the finance team to keep an eye on things, to make sure things are on track. We've all heard about the rogue agents that go off and delete databases or issue 50,000 incorrect statements.

That's what you don't want. So Robodebt was a good example of a system that generated 400,000 debt notices that was incorrect because nobody checked. So the way you build in governance is understanding from the beginning to the end, where in the steps do we need to build in the checks? So the best example I can give you for that would be something like Ford versus Toyota.

So in the 1980s, there was a book called The Machine That Changed the World. It was about the car industry and how Ford were competing with Toyota. So in Ford, you had 30% of its workforce sitting at the end of the process. So when the car came off the production line, there was a fellow in Tennessee with a singlet on with a big rubber mallet who was just continually banging the cars into shape to get them roadworthy off the production line.

So that was 30% of their workforce was fixing defects after the cars had been built. In Toyota, there is no one at the end of the production line. There are no defects that come off the production line because all the defects are fixed along the way. So in an invoice processing context, for example, you would have all the checks along that journey to make sure by the time it got approved for payment, there's no doubt that it's correct.

And so every business process, you should be able to make sure the data's the correct data, make sure the right approvals are happening, make sure the right checks are happening at various stages. A lot of it will be automated so that you can then make sure that what it's producing is safe, responsible, and the right outcome for the business. So that's where the new world will look like because as I mentioned earlier, finance might be a small team of people, but they might have hundreds of agents operating on a day-to-day basis and they've got to be able to have the skills and the knowledge and understanding of how AI works to manage those agents.

Elinor Kasapidis:
There are examples throughout history that we can learn from and there is an opportunity, I guess, to transform processes and then re-engineer them using technology. And that's something that the accounting and finance profession has done for centuries. As we finish off this podcast, we get a lot of questions here at CPA Australia around the young accountants. So they may be thinking about studying accounting or they've just graduated, they're looking at becoming an advisor or an auditor or perhaps operating in a finance team. This is quite overwhelming for them because they say, "Are the agents taking my job?" What advice or how can they position themselves to actually take advantage of this new frontier?

Bruce Mullan:
Yeah. Well, first thing I would say was when I was studying accounting and I was a young accountant once, I mean, if I could be that young accountant today, I would absolutely love this opportunity because the world hasn't been defined yet. This is your opportunity to define what good looks like. So from a perspective of what the role of the accountant will be is there was a report that came out by the Department of Employment and Workplace Relations, which classified occupations in accordance to those that were most and least exposed to AI displacement.

And in the top 20% of the most exposed are accountants and accounting offices. The reason for that is that it's the routine cognitive tasks that accountants do that are appropriate to be replicated by a machine, but that's not all they do. So if you go and visit your doctor, your doctor's got all this technical knowledge about what's wrong with you, but they've also got a really good bedside manner and they can already also help you about what support do you need with this injury that you've got or this disease that you've got.

So the role of the accountant will be moving from the back office to the middle office and the back office will be done by the agents because that's all the routine cognitive stuff. And the idea is that frees you up to actually sit and partner with the business and get into the exciting and fun stuff.

Because if an organisation is growing and you are providing that financial advice and you're becoming the trusted advisor to help them shape their future and become a much more successful company, that's what the future is for the accountants. It's not sitting in a back office banging away on a spreadsheet.

We used to call them spreadsheet jockeys, but the spreadsheet jockeys will, in some respects, go the same way as the fax machine because the AI will do that bit. And you've got to have that human connection, bring your judgement, understand the business, work with the people doing the work.

You'll be working with doctors, nurses, engineers, instruction people, surgeon, all sorts of people that you will have a really rich and rewarding career if you focus on that side of the role and not the spreadsheet part of the role, which is what traditionally accountants have been known for.

Elinor Kasapidis:
No, that's great insight. And I think it's the fun stuff and it's the value adding stuff. It's the intellectually challenging stuff. And that is what our next generation is also looking for. So Bruce, I'd like to say a massive thank you for talking through a wide range of issues. We could continue for hours, but I think really it was that journey around where are people at organizationally on their journey?

What is the environment that you need to create so that employees can disclose and be transparent about their AI use and how do you enable them through the provision of governed and secure sandboxes and systems as well as the, I guess the capability. There's a whole capability piece as well. But also to talk about the board, the CFO, how they need to actually understand things and the opportunities to enhance our productivity and our processes. So what a journey. Thank you so much, Bruce.

Bruce Mullan:
Well, if there's one thing I could just leave you with, and that's with the CFOs. So what I recommend you do is sit down with your team and develop what I call an AI charter. It's about how your team will adopt and use AI, when will you use it, when will you not use it, and what your attitude towards AI is. Sit down with your team, start developing that. And that's maybe a one to two hour workshop.

And then that will start making the changes that you need because then you'll know that, hey, we've got a gap here. People are using it in this context and they shouldn't be, but it will guide you and your people about responsible AI. And it's the first step and it's a great starting point. So I highly recommend anybody listening to this podcast is to go into ChatGPT. What does an AI charter look like? It'll give you all the steps.

Elinor Kasapidis:
Excellent. Thank you so much, Bruce. And he's available to help you with your charter as well. So for our listeners and viewers, if you enjoyed the episode, subscribe to With Interest through your favourite podcast app or on our YouTube channel, and share this with someone in your organisation or network who is working through these questions. Thank you for listening.