Skip to main content
LeoNex News

Agents are "sessions", not "requests": sizing GKE resources for AI agents

Illustration: Agents are "sessions", not "requests": sizing GKE resources for AI agents
In this article

When coding agents move from a developer's laptop to a shared GKE cluster, an agent's working session is not like a web request. The session stays open all day but is only working for a small fraction of the time, so if each session holds a machine, most of the node bill pays for waiting time. Agent Substrate and AX let you keep the session open but give the machine back while the session is waiting, and according to the cost model in the article, the same 100 agents can drop from 34 nodes to 16 nodes.

An agent session is a session, not a request

Two common ways to run agents and their limitations

Two common approaches both suit web applications but show their limitations with agents.

This leads to a third option: keep the session and release the machine while the session waits.

Agent Substrate and AX: moving idle sessions into object storage

Two concepts need to be distinguished. A session is a multi-turn interaction between the developer and the agent, along with the conversation history. The compute serving a session is an isolated sandbox, called an actor in Agent Substrate or a Task in AX. Parking, meaning putting it into a paused state, acts on the sandbox, while the session stays open.

SuspendActor takes a snapshot of the idle actor into object storage and releases its worker. The next request restores the actor on a warm worker that has room. By default, Agent Substrate takes a memory snapshot and falls back to a file snapshot when a memory snapshot cannot be taken. Because many parked sessions share one small warm pool, the same 100 agents in the cost model need only 16 nodes instead of 34.

AX declares what an agent is and what it knows. A Task says what runs, including the image, command and limits; a Workspace says what the agent knows, including the repository and files. AX runs each Task as an actor. The ax suspend command parks a task and preserves /workspace. Currently AX requires a volume snapshot, so files come back on resume while the agent's process starts over from the beginning. AX delegates the suspend, resume and delete operations to Agent Substrate. The AX roadmap is expected to add budgets, policies and a gateway to limit what agents are allowed to do, along with a record of what each task has done.

What to decide before sizing the pool

Two details determine the bill even before sizing.

  1. The timing of parking is decided by whoever deploys it. Agent Substrate does not suspend idle actors on its own, so the agent harness or the service in front of it must call the suspend API after an idle timeout. Until then an idle actor keeps its worker. When idle actors occupy all the workers, a request for another actor waits up to 5 seconds by default and then fails with HTTP 503.

  2. The CPU commitment for each sandbox is also decided by whoever deploys it. An actor's size is determined by its CPU and memory limits, and each worker counts those limits against the capacity declared for the pool. The cost model sets this capacity exactly equal to the node's real capacity, so each actor holds its full limits. You can declare more than the node's real capacity, in which case CPU is overcommitted at the pool level rather than within each Pod.

How much parking saves on GKE

The result is that parking saves nearly 40% of the node bill. The pool fills roughly once in 3,000 turns, so prompts rarely hit a 503 error. If there are 2,5 prompts per hour instead of 2, the same 15 nodes fill up at 3% of turns, so the pool should be sized for the busiest hour.

The effectiveness of parking depends on how much the agents wait and on the size of the agent fleet:

  • At 2 prompts per hour, the saving is nearly 40%. At 4 prompts per hour, agents work 44% of the time, the pool grows to 27 nodes and the saving drops to just 4%. Above roughly 4 prompts per hour, one Pod per session becomes cheaper, so measure prompt frequency first.

  • The pool size is set by whoever deploys it. The autoscaling option in the GKE installation guide tracks worker Pods rather than suspended actors, so a scheduled job is needed to scale the pool to 0 outside working hours. A pool running the full 730 hours of the month costs nearly double the bill of the one-Pod-per-session model.

  • Fleet size matters: the Agent Substrate control plane costs the same for 10 or 1,000 sessions, so parking pays off from about 37 coding sessions, or 13 sessions with a hypothetical chat assistant working about a minute per hour.

When one Pod per session is cheaper

Overcommit changes the comparison. If a Pod requests less CPU than its limit, Kubernetes schedules the Pod by its request. A Pod requesting 0.25 vCPU with a 1 vCPU limit fits 12 Pods on a node, because 1 GiB of memory caps it at that level. In that case 100 coding agents fit in 9 nodes, fewer than the parked pool. Agent Substrate can also overcommit through the declared capacity on the worker pool, but the cost model does not account for this case.

Overcommit has the same cost on both sides: agents share CPU and slow down when neighboring agents are busy, for example during builds and test runs. So compare like for like, meaning Pods and actors with the same CPU guarantee, before deciding to park.

Key points

More in AI Engineering

Illustration: Context engineering and guardrails to "tame" coding agents
AI-ENG AI Engineering

Context engineering and guardrails to "tame" coding agents

Coding agents such as Cursor, Windsurf, Cline or Claude Code often deliver impressive results early on, but as a project grows with many files and branching business logic, they can break working features, pull in conflicting libraries or violate the architecture. The cause lies in an unstructured way of using them…

4 min read

Email newsletter

Receive a selection of new LeoNex News articles by email.

We only use this address to send the newsletter.

Receive a selection of new LeoNex News articles by email. You will receive a confirmation email; the subscription only takes effect after you confirm. Every email includes an unsubscribe link. Read the privacy policy