👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps, LLMOps, and AI Infra to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!
✉️ In Today’s Edition
AI agents are becoming increasingly important for DevOps, SRE, and platform engineering. But to use them effectively, you first need to understand how they work.
This edition covers the core concepts, including agent frameworks, tools, agent loops, state machines, approval gates, and memory.
In Part 2, we will put all the concepts together and build a working Kubernetes SRE agent that can investigate issues, propose fixes, wait for approval, apply changes, and verify the results.
Important Note: For a better reading experience, read the online version here.
🍒 Scale Your Infrastructure, Not Your Cloud Bill
Cherry Servers provides dedicated bare-metal and virtual servers for Kubernetes, AI/ML, databases, and other demanding workloads.
It can be a cost-effective option for teams that need predictable pricing and dedicated resources without paying for unnecessary cloud services. You can also save up to 31% on dedicated servers and up to 50% on virtual servers with fixed-term billing.
What is an AI Agent?
AI agents are becoming an important part of DevOps and platform engineering. But before building one, you need to understand what makes an application an agent.
Unlike a traditional application that follows a fixed set of instructions, an AI agent can autonomously decide what to do next based on the information it receives.
At a high level, you can think of an AI Agent as a combination of these components:
AI Agent = Agent Runtime + LLM + Tools + Memory/Context + Instructions
The following image illustrates a typical agent architecture.
Let's understand each component.
LLM: Think of the LLM as the brain of the agent. It helps the agent understand the request, reason, plan, and decide what to do next.
Agent Runtime: This is where the agent logic runs and coordinates different actions.
Tools: Tools allow the agent to actually perform actions. For example, a tool could be an API, custom function, CLI, or a database.
Memory/Context: Agents can maintain short-term or long-term memory. This helps them keep useful context while working on a task.
Instructions and Goals: These define what the agent should achieve, how it should behave, and the rules it must follow. Instructions may also define a role or persona for the agent.
The best way to understand the underlying concepts of an agent is by building one.
What Are We Building?
We are building a Kubernetes agent that troubleshoots the cluster using plain-English requests. It investigates the live cluster using resource details, events, and logs, identifies the likely root cause, and proposes a fix.
It applies a change only after a person approves it, then checks whether the change worked.
The following image illustrates the high-level architecture of the agent we are building.
In the above image,
LangChain provides the agent layer that connects the model, tools, instructions, and agent loop.
LangGraph provides the execution and state-management layer. It runs the workflow, tracks its state, and pauses before important actions to request human approval.
The agent calls Python tools that use the Kubernetes Python client to read and modify resources through the Kubernetes API.
Agent instructions define its goals and expected behavior. Tool restrictions, Kubernetes RBAC, and approval gates enforce what actions it is allowed to perform.
PostgreSQL stores the workflow state and messages through a LangGraph checkpointer, allowing the agent to pause and resume after an approval pause.
FastAPI provides the interface through which users send requests and approvals to the agent.
Don’t worry about these terms. It will all make more sense in the upcoming sections.
Before we get into hands-on, you must understand the key fundamentals of agents.
Agent Frameworks
We don't have to code the agent from scratch. There are frameworks that help us build it with the required agent features.
For example, agent frameworks provide common features such as:
Connecting the LLM to tools
Maintaining context
Asking for human approval
Adding safety checks and more.
To build our agent, we are using the following two popular open-source agent frameworks (Python libraries)
1. LangChain
LangChain is a framework for building agents and LLM-powered applications. It provides standard interfaces for connecting models, tools, messages, and the agent loop.
In our agent, LangChain handles the following three things.
Defining and connecting tools
Creating and managing messages in the format expected by the model
Connecting to the model through the Amazon Bedrock integration
2. LangGraph
LangGraph is a low-level orchestration framework and runtime for building long-running, stateful agents.
It defines which step runs next, what information it carries, and where it pauses. It also supports saving and resuming progress.
One of LangGraph’s key features is StateGraph, which lets us build an agent as a state machine (a stateful workflow) with clearly defined stages and transitions. This gives us more control over the agent’s execution flow.
Choosing Agent Loop vs State Machine
In agent development, you need to understand two important concepts.
Agent loop
State machine
Let's understand what they mean.
Agent loop
An agent loop allows the model to decide what to do next. It chooses an action, calls a tool, observes the result, and repeats the process until the task is complete.
ReAct (reasoning and acting) is a well-known architectural pattern for implementing this loop.
State machine
In a state machine, you define the possible states and the rules for moving between them. Based on the current state and conditions, the workflow can proceed, take a different path, repeat a step, or pause for human approval.
Combining both approaches
In our agent setup, we use a combination of an agent loop and a state machine. Loops are required at each step because the model calls tools repeatedly until it has enough information to answer.
Also, loops are triggered when a fix fails. For example, during the verification step, if the issue is not fixed, a retry loop is triggered
Note: Our workflow allows a maximum of three remediation attempts. If verification still fails after the third attempt, the workflow stops and reports that the issue remains unresolved.
The following image illustrates how loops and the state machine flow are used in our agent setup.
The complete workflow is implemented as a state machine, while ReAct loops are used within steps that involve the model.
Next, you need to know about tools in an AI agent.
Tools in an AI Agent
There are API's and command-line tools to interact with an application.
If an agent uses those APIs or command-line tools to perform actions such as reading data or updating a resource, you can refer to them as tools from the agent's perspective.
For example, if you create a custom Python function, get_pod_logs, that calls the k8s API, it is a custom agent tool.
However, if you use the kubectl logs command in bash, bash is the tool exposed to the agent, while kubectl logs is the command executed through it.
How about MCP Servers?
MCP server has built-in implementations to communicate with other apps via APIs, command lines, etc. So MCP servers expose tools directly to the AI agent. The agent just uses them.
For example, a Kubernetes MCP server might expose tools such as,
get_pod_logs
list_pods
The AI application connects to the MCP server through an MCP client and discovers the available tools. The agent can then select and call the appropriate tool for the task, as shown below.
The following image of the code snippet gives you a glimpse of the dedicated tool implementation. I will explain it in detail later in the implementation section.
Agent Approval Gate
Should we allow the agent to apply fixes automatically?
Not by default. Here is why.
Let's say there is a pod stuck in CrashLoopBackOff, and the logs show it's because the application can't connect to the database. But the agent may think the database URL is incorrect and plan to change it.
But the actual cause could be that the database is restarting or that a NetworkPolicy is blocking the connection. If the agent automatically changes the Deployment, it may trigger an unnecessary rollout.
Key Insight:
One reported real-world incident involved a Cursor AI coding agent running Claude Opus 4.6. The agent reportedly deleted PocketOS’s production database and its volume-level backups through a Railway API call while working on a staging task.
Approval gates solve these kinds of problems.
Whenever a new issue arises, the agent provides a fix and asks you to approve it, reject it, or provide a custom solution. The agent waits until you give an answer.
In our agent implementation, when we approve a plan, the agent won't apply it right away. A custom function called check_write_allowed() runs before applying the fix to verify whether the agent guardrail permits this action, as shown below.
Agent Memory
When you ask the agent, "Are my cluster pods healthy?"
Here, the first request may be simple, but the follow-up requests depend on information from the previous conversations. For example, after the agent investigates it, you may ask again, "What is their status now?"
Here, “their” refers to the cluster pods. The agent needs the previous conversation and investigation details to determine which Pods to check.
Memory is also important when the workflow pauses for approval. For example, the agent may have already identified the unhealthy Pod, found the likely cause, and proposed a fix, and paused the workflow for your approval.
At this point, the agent saves a checkpoint containing the messages, the collected information, the proposed fix, and the current workflow stage.
A checkpoint is a saved snapshot of the agent’s workflow at a specific point in time.
When you approve the fix, the agent loads the saved checkpoint and continues the workflow. It does not need to restart the investigation.
The LLM does not automatically remember all of this across separate requests. So, the agent needs memory to save and load the correct state for every request.
During development, we can use InMemorySaver to keep checkpoints in RAM. However, this data is temporary and will be lost when the application restarts.
In our agent, we use LangGraph’s PostgresSaver as the persistent checkpointer, with PostgreSQL as its storage backend. It stores snapshots of the graph state, including the message history and other workflow data.
Key Insight:
Production AI agents often use a combination of storage systems. Each storage type serves a different purpose.
PostgreSQL can store conversation history, workflow checkpoints, agent state, and other persistent data.
Redis can store cached results and temporary data. It can also help with queues and communication between agent workers.
A vector database can help the agent find related documents, runbooks, and past incidents based on meaning rather than exact keywords.
However, not every agent needs all three. The storage design depends on what the agent needs to remember and retrieve.
For example, a simple agent may only need PostgreSQL.
Agent Identity and Authentication
Approval gates control when an action can proceed. Permissions control what actions the agent can perform. But before granting those permissions, the receiving system must verify the agent's identity.
In our project, the backend Pod uses its assigned Kubernetes ServiceAccount when connecting to the Kubernetes API.
In larger environments where agents connect to remote MCP servers, internal APIs, or other agents, workload identity systems such as SPIFFE and SPIRE can provide short-lived, verifiable identities without relying on long-lived credentials.
We will cover workload identity for AI agents in a separate edition.
Key Insight
SPIFFE/SPIRE works like a Kubernetes ServiceAccount identity, but it can be used between workloads across clusters, clouds, and other environments.
What’s Next!
This edition was all about understanding the fundamental concepts behind AI agents.
In Part 2, we will build a Kubernetes SRE agent from scratch using LangChain and LangGraph.
The agent will investigate Kubernetes issues, collect logs and events, identify the likely cause, propose a fix, wait for approval, apply the fix, and verify the result.
This hands-on project will bring all the concepts we discussed today together and show you how they work in a real agent.
