Agentic AI: Exploring the Benefits and Challenges of GenAI 2.0

There has been much discussion in recent months about this concept of Agentic AI (sometimes referred to as GenAI 2.0). Many people see it as being the key AI trend for 2025 (see here). Visionaries like Microsoft's Satya Nadella suggest that the use of AI agents will disrupt whole industries and eventually render traditional SaaS applications obsolete (see here). Not only will there be impacts on the SaaS-based software industry, but futuristics are also predicting that the need for search engines like Google and Bing will eventually be eliminated, and how we design and develop software will also be impacted. Their view is that Agentic AI promises to be one of the more disruptive technologies to the IT industry.

But what is Agentic AI, what are the benefits of this technology, and what are its challenges? Why do people think it will be so disruptive?

In my blog article this month, I will talk about the concept of Agentic AI, explain "what it is", and talk about the benefits that Agentic AI can bring to Enterprises looking to increase their investments in Artificial Intelligence and automation with the goal of reducing costs, increasing efficiencies and improving customer service.

I will also provide some ways that organisations can easily experiment now with Agentic AI (Microsoft have released some interesting new tooling in this area) and discuss some of the more promising Agentic AI use cases that you may want to incorporate into your current AI program planning.

But like most new technologies, Agentic AI has its challenges, and I'll talk about those as well. Whilst I am optimistic about the significance of this new technology trend, I do think the adoption curve will be slower than what many in the industry are predicting. This is because of the challenges associated with implementing Agentic AI. So, I don't think technologists who work in the SaaS, search or application development industries should be worried about their jobs just yet. But this doesn't negate the need for business and technology leaders to understand and plan for this new tech trend and for technologists to think about how they will need to adapt their skills to what will continue to be a changing IT labour market.

As with all my blog articles, my target audience for this blog is senior business executives who are charged with building their organisation's Digital Strategy, and as such, I seek to explain a complex technical topic in a simple, non-technical, business-centric way. The article is designed for those who have minimal knowledge of GenAI but will benefit the experienced GenAI practitioner as well. Feedback, comments and suggestions for improvement on my blog are always welcome. Suggestions for future topics are also very welcome.

A Happy New Year to all!

Some Background - Generative AI Version 1.0

Foundation Models - Changing the Economics of AI

For years, Artificial Intelligence (AI) required organisations to build specific AI models for each individual use case, whether it was machine vision, natural language processing, conversational agents or others. These models, trained on specific use case data, needed to be tuned and had to be constantly monitored for performance, updating the models regularly when performance degraded. Organisations needed to pay for the people and tools necessary to build and manage these individual AI models. As a result, AI was only really affordable for larger enterprises and organisations that could afford the infrastructure investments required.

Then, in 2017, the University of Toronto, in partnership with Google, wrote a research paper called "Attention is All You Need". See here if you'd like to read this paper. Without going into detail, this research paper fundamentally changed AI by giving organisations a computationally efficient way of building extremely large AI models. These extremely large models, trained on a variety of data, could be used for a variety of use cases, not just one. As a result, the concept of the "Foundation Model" was born.

The term "Foundation Model" was first coined by the Stanford AI team (see here) to refer to AI models that could be applied across a variety of use cases. These Foundation Models allow organisations to adopt a build once, use many times approach to AI. This radically changes the economics of AI by making even lower volume use cases economical for even very small organisations. It also allows organisations to use models built by other organisations in the same industry, reducing the minimum investments necessary and improving the economics of AI even further. Foundation models also facilitated the more recent development of Generative AI late in 2022.

Generative AI - a type of Foundation Model

By definition, Generative AI (GenAI) is any time you use AI Foundation Models to generate content, whether text, images or voice. Large Language Models (LLMs) are a form of Foundational, Generative AI that is used to understand text inputs and generate either a text or image response. I recently published a video on what Generative AI is, which you can watch here if you'd like further information and explanation.

ChatGPT, which was released for public consumption in November 2022 by OpenAI (see here) and which most people have played with, is the best example of a Large Language Model, which is a form of Foundational, Generative AI. Able to respond fluently and humanlike to many different types of questions and directions, ChatGPT demonstrated the immense capability of these Foundational GenAI models, and ever since, organisations have been racing to adopt this new technology because of the benefits it can provide.

The adoption rate of these foundational, generative AI solutions has been so fast that the use of ChatGPT has surpassed the adoption of Facebook, mobile phones, and even the Internet (see here). Hence, why I have so many clients experimenting with this technology, and why many organisations are asking big tech companies like Microsoft, AWS, Google, and IBM to help them deploy Foundational, Generative AI solutions into their organisations.

Recently, the next generation of GenAI, called "Agentic AI", has been starting to gain interest amongst early technology adopters.

Generative AI Use Cases - The Path to Agentic AI

Broadly speaking, the use cases for GenAI can be broken into three categories: Content Generation, Content Retrieval and Decision Making. Content Generation is easy. Ask ChatGPT to write a poem. It will do so in less than 10 seconds. Asking it to retrieve useful content related to a specific use case within your business is more challenging. You have to collate that content, make sure it is accurate and then use something called Fine Tuning or Retrieval Augmented Generation to ground your LLM in that content so that it responds correctly when asked a question. Harder still is using the LLM to make decisions. This often requires complex integration, design and testing.

Asking an LLM to route an email and then generate a useful response to that email that a call centre agent can use are simple examples of GenAI-based decision-making that are already in use by many organisations. But even these use cases are still not technically trivial as integration with existing systems, design and testing are required. The real value-added, more complex decisions I've seen people use LLMs for are activities like making asset investment recommendations or helping identify cancer in chest X-rays. There is always a human in the loop, but the LLM makes recommendations and helps with decisions, making the human faster, more efficient, and more effective.

Speaking from experience, the more advanced decision-making scenarios can be even more technically complex and require more effort to get right. But once you get them right, they generate much more value than the more straightforward Content Generation and Retrieval scenarios. It turns out LLMs can actually be quite good at making decisions. This has led to the next evolution in the use of GenAI called "Agentic AI", which started gaining interest late in 2023.

So what is Agentic AI (or Generative AI Version 2.0)?

Agentic AI, often referred to as GenAI 2.0, goes beyond using Large Language Models for simple content generation and retrieval and extends the LLMs existing ability to undertake basic decision-making.

Agentic AI systems are designed to autonomously pursue complex goals with limited direct human supervision by designing their own workflows and using dynamic combinations of available tools to interact with their environment. This ability to act autonomously and interact with their environment is what differentiates Agentic AI (or GenAI 2.0) from the previous iteration of GenAI.

The term "Agentic" itself comes from the idea that these systems have "agency" to make decisions, take actions, solve complex problems and interact with external environments (for reference, see here).

As such, Agentic AI represents a paradigm shift in terms of how we approach the use of Foundational Generative AI models within organisations.

One common way to achieve this ability for LLMs to pursue complex goals autonomously is to use ReAct and other advanced methods of LLM prompting, such as Chain of Thought (see here), to provide a single LLM with basic reasoning skills.

Another way is to have several LLM-based GenAI agents work together during the problem-solving process, undertaking various "roles" in achieving one unified goal. For example, when using Agentic AI in software development, it is common to have one LLM-based agent to write the software code, a second to do the QA on the code that is written by the first agent and make recommendations for improvements and a third agent to represent the user in terms of organising the work and defining the overall code requirements. It's the several LLMs working together that enable the basic problem-solving behaviour needed for Agentic AI.

This is why the term "system" is often used in conjunction with the term "Agentic AI", as Agentic AI solutions are often "systems" of multiple LLM-based agents working together.

There are still other techniques that researchers are experimenting with in their desire to develop more advanced Agentic AI. Regardless of how an Agentic AI system is designed and developed, all these Agentic AI systems have the following common characteristics...

Autonomy: The ability to take goal-directed actions with minimal human oversight. This minimises the workload on the part of the human users.
Reasoning: Contextual decision-making, allowing it to make judgment calls and weigh trade-offs.
Adaptable Planning: Dynamic adjustment of goals and plans based on changing conditions.
Language Understanding: Comprehending and following natural language instructions.
Workflow Optimization: Fluidly moving between subtasks and applications to complete processes efficiently

For further explanation, see here for a great video on what are Agentic AI systems from IBM.

What are the benefits and impacts of Agentic AI?

So far, we've talked about traditional AI and how the recent development of the Foundation model concept has not only changed the economics of AI but enabled the concept of Generative AI. We also talked about the multiple categories of GenAI use cases and that the use case category that generates the most benefit from GenAI is when you use GenAI to support decision-making. Finally, we discussed that Agentic AI extends the basic decision-making capability use case for LLMs with a view to creating autonomous systems that have the agency to undertake decisions with minimal human intervention. But what would be the benefits/ impacts of such systems?

Well, aside from the potential to further automate away many of the mundane, tedious tasks that we all have to undertake as part of our daily work lives, the real advantage of Agentic AI is in the impact it will have on how we interact with technology.

When people claim that these Agentic AI systems will make obsolete traditional SaaS-based applications, it's because, in traditional software, the end user is at the centre of the business process, undertaking all the necessary tasks to achieve the desired goal in the form of Create, Read, Update or Delete (CRUD) transactions. In recent years, workflows and automation tools (often called Robotic Process Automation or RPA) have been used to automate the most repetitive and fixed of these tasks and extend the efficiency of the human user. Still, these tools were limited to static processes where all possible variations of the process flow could be clearly articulated. Voice recognition may have also been used to, in some cases, eliminate the need for manual typing and make the users more efficient still. But when the process had a degree of ambiguity, was complex, or involved some form of problem-solving, or the verbal instructions provided were not specific enough, then humans were often required to intervene. So, humans were still critical for the execution of the business process.

With Agentic AI, the AI agent ( or agents ) have the ability to undertake a task as instructed by their human master unassisted, adapt to potentially ambiguous situations or unclear verbal statements and problem-solve when the process required to achieve the specified goal is unclear. In this context, humans no longer need to interact with the software directly; instead, they can give the AI agent instructions, either verbally or in writing, and the agent interacts with the software, undertaking the task on behalf of the user carrying out the task relatively unsupervised. Clearly, the human labour savings that result from this change in how we interact with technology could be significant.

Much like in the science fiction TV shows and movies of the past, advocates of Agentic AI systems see a near future where humans need only "talk" to a computer to achieve more and more complex tasks. Not only does this impact most SaaS-based business applications that rely on complex human-machine interfaces that support CRUD transactions, but it also impacts the use of Search engines, essential software development and so on.

As most SaaS licensing/ revenue models rely on per-user access licensing, this also means that software business models will undergo significant changes as one AI agent can service many human masters. As most search engines rely on advertising associated with users' reviews of search engine output, this business model will also require change as well. So, the change in IT business models across the industry could be pretty significant.

But there are Challenges with Agentic AI

Whilst Agentic AI is clearly a significant technology trend given its ability to change the way we interact with technology, and one can see many near-term plausible use cases with significant benefits, there are still many challenges that I think will slow the Agentic AI

adoption curve and cause it to take some time before Agentic AI systems are widespread.

These challenges include but are not limited to the following...

Poor Process / Problem Definition

As previously mentioned, one of the techniques to achieve Agentic AI is ReAct prompting. This is where the LLM prompt is written in such a way that the LLM first reasons what information or tasks it needs to undertake to solve a complex problem and then takes action to solve the problem. To define a ReAct prompt, you need to describe the process of solving a problem and agree on the general characteristics of the preferred solution.

Similarly, if you are using a system of LLMs together to reason through a complex problem, you also need to be able to guide that system in terms of the problem-solving process and provide a clear definition of success. This must mean that the problem-solving process is relatively defined and that there is organisational agreement on it.

I've been involved in many automation projects that used both RPA and AI to make a process more efficient over the last 10 years. One of the many challenges I've always had has been organisational change management, particularly when a business process crosses many departments within an organisation. Getting people to agree on a process and what the definition of good is has always been a challenge.

My point here is that for an organisation to make effective use of Agentic AI, there needs to be clarity around its business processes and organisational goals. Many organisations struggle to achieve that clarity even for the more critical aspects of their business. If humans can't articulate what they want, then how will a machine, even an intelligent Agentic AI-based machine, give them what they want?

Integration

For an agent to interact with its environment, it needs to be integrated with that environment. Over the years, organisations have gotten better at integrating their myriad of business systems. Yet many information workers still have to work with various applications to execute their day-to-day jobs. For example, I did a quick count of the applications on my laptop/ in my web browser, and there are over 40, 20 of which I use weekly and probably 10 I use daily. Mobile phones are the worst. Most people probably have over a hundred apps on their phones, many of which they use daily. My point here is that for the Agentic AI system to be of value, it will need to be able to interact with those sample applications directly on behalf of its human user. That integration won't be trivial.

Poor Underlying Data Quality

I've been involved in probably 20-plus GenAI pilots and production systems, and one of the issues that comes up time and again is the quality of the input data used in the LLM system.

For example, I worked with one utility (which will remain nameless), where we built a really effective GenAI solution that would review all its operating procedures and make recommendations to network operators in the case of network outages. As part of the development and testing process, the utility realised that even though it was an ISO shop and, therefore, had well-established documentation processes, many of its operating procedures were still out of date and/or needed to be rewritten. So when the operator asked the LLM a question, the answer might be wrong, not because of the LLM technology but because the underlying organisational data the LLM was basing its answer on was wrong.

There is an old adage in IT that "garbage in = garbage out," which still applies to Agentic AI as it does to traditional IT.

In-Effective Search and Poor Database Structure

In those 20-plus GenAI pilots and production systems that I have been involved with, the other problem that comes up quite regularly is getting the right data into the LLM for it to make an effective decision. This is particularly important when you have large amounts of data that the LLM needs to make sense of. This involves two parts: first, the database the data is stored within and how to make it easily accessible and searchable, and second, the mechanics of searching the database itself. I have seen many organisations fail in their deployments of GenAI over the last two years, not because of the GenAI technology but because of how they store and search for information that the LLM uses when they are asked either to retrieve information or make a decision. As a result, I have already written two articles on the topic which you can find here and here. This issue is particularly germane for those futurists who think GenAI will eliminate the current concept of a Search Engine. You may have an LLM front the Search Engine, but the importance of search will always be there.

So even if you have a good understanding of the business processes and what good is, have a fully integrated technical environment, and your data quality is above average, the LLM may still fail in its decision-making because it has trouble finding the right information to make an effective decision. If you think about it, these are all problems that challenge us now as human business decision-makers. So, I'm not sure why people believe Agentic AI will automatically solve these underlying business problems? This being said, most enterprises will have parts of the organisation where these problems are less prevalent and, as a result, may provide good opportunities to deploy Agentic AI solutions.

Microsoft's AutoGen and AutoGen Studio - A way for organisations to experiment with Agentic AI

In my preamble, I promised to talk about some tools that organisations can use right now to experiment with Agentic AI. Some of these are Microsoft AutoGen and AutoGen Studio.

AutoGen is an open-source framework that allows developers to build LLM applications via multiple agents that can converse with each other to accomplish tasks. On Sep 25, 2023, Microsoft Research introduced AutoGen, a framework for simplifying the orchestration, optimization, and automation of workflows for large language models.

AutoGen offers customizable and conversable agents that leverage the strongest capabilities of the most advanced LLMs, like GPT-4, while addressing their limitations by integrating with humans and tools and having conversations between multiple agents via automated chat.

Using AutoGen, developers can also flexibly define agent interaction behaviours. Both natural language and computer code can be used to program flexible conversation patterns for different applications. AutoGen is a generic infrastructure to build diverse applications of various complexities and LLM capacities.

By automating chat among multiple capable agents, one can easily make them collectively perform tasks autonomously or with human feedback, including tasks that require using tools via code.

Autogen includes the following features:

Multi-agent conversations: AutoGen agents can communicate with each other to solve tasks. This allows for more complex and sophisticated applications than would be possible with a single LLM.
Customization: AutoGen agents can be customized to meet the specific needs of an application. This includes the ability to choose the LLMs to use, the types of human input to allow, and the tools to employ.
Human participation: AutoGen seamlessly allows human participation. This means that humans can provide input and feedback to the agents as needed.

The agent conversation-centric design has numerous benefits, including that it:

Naturally handles ambiguity, feedback, progress, and collaboration.
It enables practical coding-related tasks, like tool use with back-and-forth troubleshooting.
Allows users to seamlessly opt in or opt out via an agent in the chat.
Achieves a collective goal with the cooperation of multiple specialists

To make it easier for organisations to deploy Agentic Applications using the AutoGen framework, Microsoft also created something called AutoGen Studio. AutoGen Studio is an AutoGen-powered AI app (user interface) to help you rapidly prototype AI agents, enhance them with skills, compose them into workflows and interact with them to accomplish tasks. It is built on top of the AutoGen framework, which is a toolkit for building AI agents. The best part of AutoGen and AutoGen Studio is that they are open source and, therefore, free for organisations to use and experiment with.

(Note: AutoGen Studio is meant to help you rapidly prototype multi-agent workflows and demonstrate an example of end-user interfaces built with AutoGen. It is not meant to be a production-ready applications)

For more information on AutoGen Studio and how to deploy it, see here.

Potential Agentic AI Use Cases

I've seen organisations use Agentic AI for a number of different types of use cases, from Travel Trip Planning to simple Software Application Development. There are whole web pages that are devoted to the topic of Agentic AI use cases and already provide code libraries to make it easier for organisations to prototype these technologies. For example, see here.

Some more common Agentic AI use cases are....

Customer Service - AI Agents that have the ability to solve more complex customer problems using customer-specific information
Supply Chain Management - AI Agents that can dynamically optimise logistic routes
Healthcare - AI Agents that can adjust strategies and set goals dynamically using real-time data and making recommendations to doctors
Software Development - using AI Agents to help create new applications quickly for specific use cases

In my experience, Customer Service and Software Development seem to be the two areas where organisations are experimenting with Agentic AI the most, and that represents areas where the opportunity and benefit warrant the technical complexity. But then I think what organisations need to do is filter all the potential applications of Agentic AI against the difficulties I mentioned in the previous section around process clarity, data quality, integration and the ability to find the correct information. This will give them a view of the best place to start on their Agentic AI journey.

Conclusions

It's clear that Agentic AI is a disruptive technology that could lead to significant change across many parts areas of IT and represents a paradigm shift in how we use technology. This being said, it does have its challenges, which means some organisations will likely be more successful in the use of Agentic AI than others. Consequently, this technology trend warrants careful consideration for those executives who are considering folding Agentic AI into their overall Digital Strategy.

If you have questions about this topic or want further information or assistance, please contact me at david@ai-savvy.com.au . You can also read more about GenAI at www.ai-savvy.com.au