Agentic Incident Enrichment, the Lowest Hanging Fruit of Agentic IT Use Cases

Agentic IT operations gets a lot more interesting when you stop asking what agents can do and start asking what you’d actually trust them to do.

In a previous vlog, I talked about “Bind Is Not a Grant” and demonstrated how we can use platforms like Tanzu and AgentMinder to control access to different MCP tools through identities, intents, and grants. That was pretty cool, but the actual use case was rather rudimentary. I was essentially showing how agents could interact with infrastructure, not necessarily why an enterprise would want them to.

Then I moved on to thinking about how to structure MCP servers. The idea was that instead of simply wrapping an API, maintaining its original structure, and exposing all the ugliness underneath, we should design MCP tools around the actual capabilities we want agents to have. For example, rather than exposing dozens of API calls related to virtual machines, we might want to provide meaningful capabilities around managing or investigating VMs. The same goes for networks, storage, Kubernetes, or whatever systems our agents need to interact with.

An MCP server should make it easier for an agent to accomplish a meaningful task, not force it to understand every detail of the underlying API. So now I want to move on to something else. What are the actual real-world use cases where enterprises can use agents to enhance their existing processes? Not because agents are cool or because we can connect an LLM to an API and make things happen, but because they can genuinely improve the way people work.

AI as an Exoskeleton

I like to think about using agents in the enterprise very similarly to how I think people should use regular AI chat. Let’s say you’re writing a document. You could ask an AI to write the entire thing for you, take the result, and send it to your boss. I’m sure some people do exactly that, but I don’t think that’s where AI provides its greatest value.

I want to bring my own ideas, experience, and opinions to the document. Then I want AI to help me enhance my capabilities: clean up the text, challenge my assumptions, suggest relevant facts, and generally improve the result. I think of AI as an exoskeleton for the human brain. It doesn’t have to replace what I do. It can make me better at doing it.

And I think we should approach enterprise agents in much the same way. Take infrastructure management, for example. We don’t necessarily want to release an agent into our network and allow it to roam around doing things. I certainly wouldn’t be particularly comfortable with that in a production environment. Not yet, at least. Instead, we should look at existing processes and identify places where agents can make those processes better. Processes where we understand the desired outcome, can define the boundaries, and can evaluate whether the agent has actually improved anything.

This touches on another important point that I think gets lost in some of the excitement around agentic AI: not everything requires an agent. Sometimes all you need is automation. If you have a well-defined process with predictable inputs and outcomes, traditional automation is probably the better solution. It’s generally easier to test, easier to understand, and more predictable.

So when should we use an agent? I think agents become interesting where automation starts struggling. Where there are too many variables to anticipate every possible scenario, where information is incomplete, or where some inference is required to determine what should happen next. The goal is not to replace deterministic automation with probabilistic reasoning. It’s to use reasoning where it actually contributes something.

Incident Enrichment: A Modest Use Case with Potentially Significant Value

One area that keeps coming up in my research and conversations with customers is troubleshooting and healing. IT organizations are responsible for more and more systems, and those systems are becoming increasingly interconnected. Applications span Kubernetes clusters, virtual infrastructure, networks, storage, load balancers, and multiple management platforms. And somehow, the IT organization is expected to maintain all of this while continuing to do more with less.

When something goes wrong, identifying the problem can be a significant part of the effort required to resolve it. We have monitoring systems generating alerts, observability platforms collecting metrics, and management systems maintaining operational information. But having the data doesn’t necessarily mean we know what to do with it. Often, an engineer needs to move between several systems, collect information, compare timestamps, understand dependencies, and try to determine which component is actually responsible.

What if we could use agents to do some of that investigation before a human even touches the incident? I’m talking about incident enrichment. An agentic system that responds to an incident, collects relevant information across multiple technical domains, correlates what it finds, and provides an initial diagnostic assessment.

No autonomous remediation, no agent deciding to restart a cluster or reconfigure the network. Just a better incident. Ideally, one that tells the operator what happened, what the likely cause might be, and what evidence supports that conclusion.

This is a relatively modest use case compared with the idea of fully autonomous IT operations, but modest doesn’t mean insignificant. If we can reduce the time required to identify the affected system or narrow down a root cause, we can potentially reduce the time to resolution. And when we’re talking about business-critical applications, that can have a very real impact.

Focus on the App

For this discussion, I want to focus on one specific critical application. We could try to build a system that investigates incidents across an entire environment, but I think starting with a defined application makes the problem more manageable. Let’s call it Critical App X.

When I say application, I don’t necessarily mean one executable or one Kubernetes deployment. I’m talking about something that delivers a business outcome. Critical App X might consist of multiple microservices, databases, and supporting components. It might span Kubernetes, virtual machines, and several infrastructure services. But from the business perspective, it’s one application that needs to work.

And for our example, something isn’t working particularly well. The application is experiencing latency. I chose latency deliberately because it’s one of those problems that can be frustrating to troubleshoot. When something is down, you often have a reasonably clear starting point. When something is slow, everything might appear to be running normally. The Kubernetes pods are healthy, the VMs are powered on, and the load balancer is functioning. Yet the application is slow.

Is it compute? Storage? Networking? Kubernetes? The application itself? Maybe it’s a combination of several things. This is where I think agentic investigation starts to become interesting. Instead of simply collecting more monitoring data, we can have an agentic system decide which evidence is relevant, investigate different possibilities, and produce a useful assessment.

But that raises a few architectural questions. How much context should we give an agent? How many systems should it be able to investigate? Should one agent handle everything, or should we have multiple specialized agents? And how do we make sure we can trust the outcome?

Thinking Through the Architecture

I recorded a whiteboard vlog exploring exactly these questions. Rather than presenting a finished architecture, I start with an incident affecting Critical App X and work through how an agentic system might investigate it across Kubernetes, VMware Cloud Foundation, and a load balancer. Along the way, I explore the tradeoffs between one large agent and multiple specialized agents, how MCP capabilities fit into the design, and why controlling access doesn’t necessarily solve the problem of incorrect reasoning.

The proposed incident-enrichment architecture, exploring the boundaries between incident response, domain-specific agents, MCP capabilities, and intent-based access control.

Watch the whiteboard session: Agentic Incident Enrichment

Watch on YouTube →

The video is my thinking process. What follows is where I think some of the more interesting engineering challenges begin.

The Agent Needs Context, Not Just Access

One of the first things we need to think about is what information an agent actually needs to investigate an incident. It’s tempting to give an agent access to a collection of MCP servers and assume it will figure everything out, but access to systems isn’t the same as understanding the environment.

Let’s say Critical App X is experiencing latency. Before the agent starts querying Kubernetes or VCF, it needs to know which resources are associated with that application. Which Kubernetes cluster hosts its workloads? Which namespaces and services belong to it? Which VMs support those workloads? Which load-balancer virtual services and pools handle its traffic? And how are all these components related?

That’s an application topology problem. If we don’t provide that context, the agent might spend much of its time discovering relationships that our infrastructure management systems already know. Worse, it might investigate unrelated resources simply because they happen to show unusual metrics.

This is where I would combine deterministic systems with agentic reasoning. Let existing inventory, monitoring, and topology systems establish the known relationships. Let automation assemble the incident context. Then let the agent decide what to investigate based on the symptoms and available evidence. We shouldn’t make agents rediscover information that already exists in a structured form, and we certainly shouldn’t confuse giving an agent more information with giving it better information.

The Difference Between Collecting Data and Producing Evidence

Here’s something else I think is important. An agent that collects a hundred metrics hasn’t necessarily improved an incident. It might have simply made the ticket longer. What we actually want is evidence, and there’s a meaningful difference.

A metric tells us that datastore latency increased. Evidence connects that observation to the incident we’re investigating. For example, the affected Kubernetes workloads are running on a particular worker node. That worker is hosted on a VM associated with a datastore. Datastore latency increased shortly before the application began experiencing problems. Now we have a relationship worth investigating, but we still don’t necessarily have a confirmed root cause.

And that’s where I think the design of the agent’s output becomes just as important as the design of its tools. I would want every diagnostic finding to distinguish between:

• Observations: What the agent actually retrieved from the systems.
• Evidence: The metrics, events, timestamps, and resource relationships supporting those observations.
• Assessment: What the agent believes those observations might mean.
• Uncertainty: What remains unknown, contradictory, or unverified.

That separation matters. If one agent reports that storage latency increased, another agent should be able to use that as an observation. But if the first agent concludes that storage caused the application slowdown, the second agent shouldn’t automatically treat that conclusion as a fact. Otherwise, we’re simply allowing one agent’s potentially incorrect reasoning to spread through the rest of the system.

Trust Isn’t Just About What an Agent Can Do

In my earlier Bind ≠ Grant work, I focused on controlling what agents are authorized to do. That remains extremely important. If an agent can access Kubernetes, VCF, and the load balancer, I want to make sure it can only invoke the capabilities appropriate for its job.

For incident enrichment, those capabilities should primarily involve inspection, diagnostics, and evidence collection. I don’t want the agent changing production infrastructure, and I don’t want it bypassing an identity or intent-enforcement layer to reach the systems directly.

But there’s another dimension of trust that we need to consider. What if the agent is authorized to do exactly what we intended, but reaches the wrong conclusion?

An agent might have perfectly constrained read-only access and still produce a misleading diagnosis. It might misunderstand a metric, miss an important dependency, or interpret correlation as causation. And if that diagnosis is written into an incident, a human operator might act on it.

So while externalizing authority can help control what an agent does to infrastructure, it doesn’t automatically make the agent’s reasoning trustworthy. This is why I find the question of agent specialization interesting. Maybe separating investigations across technical domains helps us produce cleaner findings. Maybe it reduces the amount of irrelevant context each agent needs to process. But perhaps the opposite is true. Perhaps one agent with broad visibility is better at understanding relationships across systems. I don’t think we should assume either answer. The architecture needs to be evaluated against actual incidents.

The Incident Is the Product

One thing I want to emphasize is that the output of this system isn’t really an agent conversation. It isn’t even the diagnosis itself. The product is a better incident.

Something an operator can open and immediately understand:

• What is affected, and what has already been investigated?
• Which observations are relevant, and what is the probable cause?
• How well is that assessment supported, and what remains uncertain?
• What should the engineer investigate next?
• What evidence can the engineer inspect to validate the conclusion?

If we can’t answer those questions, we haven’t really enriched the incident. We’ve just attached some AI-generated text to a ticket.

I also wouldn’t rely on an agent-generated confidence percentage by itself. A model saying it’s 95% confident doesn’t necessarily mean there’s a 95% chance it’s correct. For that number to mean something operationally, we’d need to validate and calibrate it against actual outcomes. Until then, I’d much rather see a clear explanation of the supporting evidence, the missing information, and any contradictory findings.

An agent that says, “I haven’t identified the cause, but I’ve ruled out these possibilities and found these anomalies,” might be considerably more useful than one that confidently invents an answer.

How Would We Know If It Actually Works?

This is the part that I think will ultimately determine whether incident enrichment becomes useful or just another interesting AI demo. We need to measure the outcome, and I don’t think the number of agents, the number of tools, or even the amount of information collected should be our primary success metrics. Those are implementation details.

What matters is whether the process improved:

• Diagnostic accuracy: Did we identify the correct problem domain or probable cause?
• Time to useful diagnosis: Did we give the operator actionable information faster?
• Evidence quality: Can the findings be traced back to actual system observations?
• False conclusions: How often did the agents point the operator in the wrong direction?
• Operational cost: How much time, how many tool calls, and how many tokens did the investigation consume?
• Human usefulness: Did the enriched incident actually save the engineer time?

There’s also a less obvious metric I’d like to understand: how often did the agent know when it didn’t know?

An incident investigation doesn’t always have enough information to establish a root cause. Sometimes the right outcome is identifying the next useful diagnostic step rather than presenting a conclusion. I think a trustworthy agentic system needs to be comfortable with that.

What I Want to Build Next

I have a VMware Cloud Foundation and VKS lab where I can start experimenting with this. My initial plan is to build an incident-enrichment workflow around a critical application and use controlled incidents to evaluate different approaches.

I want to compare a broad agent that investigates multiple systems with an architecture that delegates investigations to technical specialists. The important thing is that both approaches should operate under comparable authority constraints and investigate the same incidents. Otherwise, we won’t know whether differences in the results came from the agent architecture or from the information and tools we made available.

I’ll also need to think about how agents exchange findings, how incident state is maintained, and how to make the entire investigation traceable. And I want to explore how much of the diagnostic intelligence we can obtain from existing infrastructure management capabilities rather than asking an LLM to reinvent everything.

There’s no reason for an agent to independently figure out something that a mature infrastructure platform can already diagnose. The agent’s value may be in knowing which capability to invoke, interpreting the findings in the context of the incident, and correlating them with information from other systems.

This is also where the boundaries between agents, MCP tools, and deterministic automation become particularly interesting. I’m not convinced there’s one universally correct way to divide those responsibilities, but I think we can learn quite a bit by actually building the system and measuring what happens.

And While We’re at It… Agentic SDLC

There’s another use case I want to start exploring alongside incident enrichment, and it’s one I’ve already been experimenting with in my own lab: Agentic Software Development Lifecycle (SDLC).

If incident enrichment is about using agents to help us operate software, Agentic SDLC is about using agents to help us create it.

And I don’t just mean asking an AI to generate some code.

I’m interested in what happens when we introduce agents into the entire software development process. Agents that can take requirements, help design an application, write code, run tests, review their own work, and move changes through a controlled software delivery process.

Essentially, an agent-driven software supply chain.

Of course, that raises some very familiar questions. How much authority do we give these agents? How do we control what they can change? How do we validate the quality of their work? And where do we still need deterministic processes and human approval?

Interestingly, many of the architectural questions are the same ones we’re encountering with incident enrichment. The difference is that instead of agents investigating existing systems, we’re asking them to produce and modify software.

I want to explore both use cases in parallel, because I suspect we’ll learn quite a bit about enterprise agent architecture by looking at two very different kinds of work. More on that soon.

Back to the Exoskeleton

I started this discussion with the idea of AI as an exoskeleton, and I think incident enrichment is a good example of what that could mean for IT operations. We’re not trying to replace the engineer. We’re trying to give the engineer a head start.

Instead of spending the first part of an incident collecting information across different systems, the engineer receives a structured investigation with evidence, possible causes, and useful next steps. The human still validates the diagnosis and decides what to do, while the agents help make that decision better informed.

That’s the kind of enterprise use case I want to spend more time exploring. Not simply what we can make agents do, but where they can meaningfully enhance existing processes.

Of course, there’s still that unresolved architectural question: how many agents do we actually need to do this well? One large agent might be perfectly capable of handling the entire investigation. Several smaller agents might produce better findings. Or we might discover that all we’ve done is replace one agent with a rather expensive committee. That’s what the experiments are for.

More on this as it develops.