Questions about governance in a world of agentic AI? What about measuring the cost? And still unsure where you should deploy it first?
You’re not alone. Our Lunch & Learn at SIFMA Ops made that perfectly clear. Industry participants and leaders packed into the session to discuss everything from measuring the cost and ROI of AI to staying in control in an agentic world.
Building agentic Ops is an industry-wide effort. Your peers are struggling with these same questions and concerns. So dive into this summary of the key discussion points to see how they are thinking about them.
The first takeaway will be a relief to anyone who doesn’t know where to start on delivering the agentic AI-driven cost reduction their CEO is pushing for…
The journey to agentic Operations is only just beginning
The majority of firms in our session reported that they are still in the very early stages of their agentic journey. Some, but not all, had taken steps to establish a dedicated AI function. They had appointed a Head of AI, with a direct line of communication to senior leadership.
Most participants were in the early stages of using AI - what we refer to as ‘AI-assisted’. Use cases in production involved Microsoft Copilot for tasks such as email automation. Only a few were experimenting with other forms of AI, such as small proofs of concept with Claude.
Very few firms reported using AI in an Operations context, although one firm shared that they were using it for exceptions resolution. There was a worry that AI efforts were largely focussed on the front office. This is a classic pattern in many financial firms. There is a risk that agentic AI could widen the capability chasm between the innovative front office and the forgotten middle and back offices.
Participants also shared that a lot of their AI usage is in development; segregated from the wider tech stack. Firms have small AI pilots demonstrating the value of the use case, but these are sandboxed due to information security concerns.
Which leads us neatly onto the second key takeaway - and probably the biggest question currently on your mind…
Governance concerns dominate the agentic AI debate
Keeping control in an agentic world is important. Many firms’ Technology and Information Security functions are exercising extreme caution, given the lack of a well-defined framework.
This is making governance a big blocker to AI adoption, with many participants in our roundtables sharing that they were struggling to get approval.
Central to this are the fears around ‘black box’ AI, that makes decisions or takes action without transparency. Firms are worried about how they will explain this to auditors.
Tied to this point is how firms can prove to auditors that they have the necessary controls in place around AI. This is the primary reason why infosec teams have adopted stricter approaches to internal governance. But it is slowing down firms’ ability to move fast.
Even the participants who are looking to partner with AI vendors to accelerate their adoption are struggling due to the internal red tape. Some firms admit to wondering whether the agentic use case is even beneficial enough to warrant the time they would have to spend getting approval.
This last point is particularly striking, because it mirrors something that already happens in Operations when change management policies clash with the need to act fast. End-user computing solutions are already an issue for both Operations and IT teams, where users resort to risky and opaque manual processes rather than go through a lengthy change management process.
Indeed, participants raised the idea that we’re just heading for EUC 2.0, where people develop their own agentic applications across the organisation outside of the proper governance structures.
Which raises the question: how should agentic AI be deployed?
Where to deploy, how to adopt
The traditional model of technology deployment - where it sits firmly within IT - is being thrown into question by AI. It makes it simple for any user to build agents and complex applications. How should this be enabled and managed to best balance agility and control?
Indeed, even IT teams themselves are unsure, with some participants in our roundtables sharing that they are uncertain what their role is in an AI-enabled organisation.
Most participants said that IT should be an ‘enabler’ of agentic AI for Operations. They should own the technology and especially the governance framework. But, again, there were questions over how such a model should work. Some participants believed their IT teams lacked the SME knowledge to understand the business value of agentic AI use cases.
People were also split over whether agentic AI capabilities should be centralised or decentralised. While some envisaged an AI centre of excellence, others believed that ‘transformation agents’ should sit within individual teams and share best practices back across the organisation.
The question of deployment is also tightly linked to the issue of enablement. Some participants shared that they are already trialling a centralised AI deployment. However, they have in some cases found that their AI teams don’t see training and enablement as part of their remit.
This is particularly troubling given that a common theme among the discussions was that these were severely lacking. Participants reported being given directives to adopt AI or ‘go build agents’, without getting much guidance or training on how to ensure success.
In some cases, training was limited to 15-minute videos on how to use AI.
But others shared more promising examples, such as using internal competitions to encourage teams to innovate. And enabling and upskilling new talent to thrive in an agentic environment was a top priority for several participants.
All of this work to ensure the right implementation and widespread adoption of AI will be for nothing if the benefits are minimal. So how do you measure the payoff of your investment in agentic AI?
Measuring benefits and calculating costs
Despite the uncertainty, there is a maturing of attitudes towards AI, with participants very focussed on the practicalities of agentic.
In particular, there is a growing awareness that AI is not a panacea, as some are claiming. The key to success is picking the right use cases where AI can make a real impact.
However, lingering questions around how to measure the return on investment of agentic AI is making it hard to onboard the right tools for the job.
Most participants were clear that the primary driver of agentic AI is the ability to deliver much faster. By ‘getting rid of the work that people don’t want to do anymore’, as one participant put it, firms are freeing up time to help scale the organisation faster.
Speed is easier to measure than cost, which is where lots of firms are struggling. There isn’t yet a well-defined framework for measuring the cost of agentic AI in production.
Firms are questioning whether they should measure it based on token consumption, for instance. Which raised another difficult point: it’s not easy to know how many tokens an agentic solution will consume in production, so how can you create a compelling business case for it?
This was definitely one of the biggest areas of uncertainty for our participants. Even measuring the ROI on early agentic use cases already in production is proving a challenge.
Foundational pillars for agentic Operations
We began this article by acknowledging the uncertainties surrounding AI. But not all of the questions being asked lack an answer. In many cases, while the best practice may need updating for an agentic world, there are already time-tested strategies you can follow when implementing AI.
Here are four essential pillars of agentic AI adoption.
Start with the problem, not the technology
Focus on the issues you are trying to solve - for example, control, risk reduction, or scalability. Transformation projects often go wrong when implementing technology for technology’s sake, and agentic AI is no different.
Once you've identified the problem, think hard about the desired end state. This doesn't happen organically, and defining a clear destination will help you to measure your progress and keep agentic projects on track.
Use agents for tasks that need them
Agentic AI isn’t the answer to every problem, although some vendors may try and tell you it is. Indeed, many of the AI use cases already creating value are using other forms of AI, such as generative AI, not agents.
Agents add value around the edges, for example:
- Interpreting exceptions
- Navigating policies and procedures
- Prioritising issues for resolution
- Coordinating handoffs between teams
Give your agents a mission
Agentic AI works best when it is deployed with a mission, not on a platform-by-platform basis. This means giving it a defined outcome, clear boundaries, points where it should escalate, and a human owner.
Interrogate AI partners on their governance approach
Agents can only ever be as effective as the governance surrounding them. Without control, you can’t trust them and therefore they don’t get deployed. Make sure any AI vendors can answer tricky questions, such as:
- Is the agent permissioned exactly like a user?
- Can I see who did what and when, broken down by human and agent actions?
- Can I trace every decision back to data, logic and training?
- Can I export controls, rules and configurations for audit?
Vendors who understand and cater to your governance requirements should be able to answer these.
The first steps into an agentic world
We are entering a new era of Operations. Agentic AI is changing everything from the day-to-day work performed by an Operations analyst to the role of a CIO and their Technology team.
While there are some first movers, the majority of firms are in the same position. By learning from each other, you’ll be able to overcome those concerns around cost, uncover the governance frameworks you need in order to keep control while fostering innovation, and build the best practices around empowering your teams and raising adoption.
These are big, important questions that need answering. But that’s because this is a big opportunity facing the industry. The choices you make today will shape the Operations of tomorrow.
Want to stay on top of all the latest agentic best practices? Subscribe to our newsletter.
