Cluster Agents

A Cluster Agent is a read-only SRE scoped to exactly one GKE cluster. It is a Hermes profile (cluster-<project>-<cluster>-<location>, sanitized to 63 characters) that the Platform Agent scaffolds at runtime inside its own pod — one per managed cluster, persisting on the data volume until that cluster is deleted. SOUL.md §6 makes delegation the default: work scoped to a single cluster’s live runtime (crash loops, OOMs, scheduling failures, mount errors, connectivity, autoscaling, storage, observability gaps) belongs to that cluster’s Cluster Agent, while fleet-wide audits, provisioning, and the GitOps write path stay with the Platform Agent.

Scoped by construction

Each profile is stamped from the agents/cluster/ template (baked into the image at /opt/cluster-template) by cluster_agent_profile.py:

• One cluster only. A KUBECONFIG pinned to the target cluster is written into the profile’s .env, and the cluster’s project/name/location are recorded as a cluster_identity block in its config and stamped directly into its SOUL.md so the worker has its identity in the system prompt on turn 1.
• Read-only toolset. The template config exposes only the gke and developer_knowledge MCP servers — no platform_control (provisioning), no GitOps write path. A Cluster Agent diagnoses; it never mutates cluster state and never opens pull requests. For GKE facts and error codes, it consults developer_knowledge first (search_documents, never the 50-a-day answer_query), falling back to DuckDuckGo web search only for third-party or open-source signatures.
• Its own skills. The single-cluster runtime-debugging skills ship in agents/cluster/skills/ (load-bearing change, observability, reliability, stall detection, storage, workload scaling, workload security, workload troubleshooting) — listed under their own heading in the skill catalog.

Lifecycle

A managed cluster and its Cluster Agent profile are created together and deleted together (SOUL.md §6). Three paths maintain that invariant:

1. Onboarding. When a merged change has produced a new cluster (gke-cluster-creation proposes it as a pull request) or the Platform Agent first brings one under management, the cluster-agent-lifecycle skill creates the profile.
2. On request. “Manage my cluster X in Y” invokes the manage-cluster skill, which verifies the cluster exists and creates its profile (idempotent); “stop managing” goes through the same skill, which routes the exclusion to where the install declares its scope.
3. Reconciliation. The hourly cluster-agent-reconcile job (a no_agent script job on the Planning Agent profile’s cron file) sweeps every project in scope (the management project alone unless spec.scope says otherwise): it creates a profile for every cluster that lacks one — including the management cluster kube-agents itself runs on, whose own workloads fail like any other cluster’s — and prunes a profile only when its cluster is definitively gone (a NotFound from gcloud container clusters describe), when the cluster is named in spec.scope.exclude.clusters (or, for one more release, in RECONCILE_EXCLUDE), or when its project has left the scope, over two runs that could each trust every lookup they made (the first marks the project retiring in fleet_scope.json, the next such run prunes; a management project that changes identity is marked on the run the change is seen, once the new project has listed its own clusters; spec.scope states the conditions). Ambiguous errors (auth, network, quota) never trigger deletion.

Which control-plane endpoint the profile uses

Scaffolding pins a kubeconfig scoped to the cluster, and which endpoint that kubeconfig names is decided per cluster. The DNS endpoint wins when it accepts external traffic. A cluster on the agent cluster’s VPC whose public endpoint is absent or gated by authorized networks, and whose DNS endpoint is closed, is reached over its private endpoint (privateClusterConfig.privateEndpoint) when that endpoint is in the agent cluster’s region or has control-plane global access, and when it admits the agent: no authorized-network list is enabled, or the list explicitly does not gate the private endpoint (masterAuthorizedNetworksConfig.privateEndpointEnforcementEnabled: false), or the two clusters share a subnet, or the agent cluster’s Pod range (clusterIpv4Cidr) lies inside a listed range. Otherwise the kubeconfig names the endpoint gcloud writes by default. The scaffold records the endpoint it chose, when one was decided, in the profile’s USER.md, probes the cluster once, and when the connection fails the onboarding log and the Cluster Agent’s preflight repeat that record and name what would open the cluster: the Pod range to add when the private endpoint’s list is the block, the agent’s egress address when the public endpoint’s is, control-plane global access when the region is, and the DNS endpoint when the VPC is. private-endpoint-selection.md has the full rule.

How delegation works

Delegation runs on the shared kanban board — agents never pass context to each other directly:

1. The Platform Agent resolves the cluster’s profile name with its get_cluster_profile_name tool, which also says whether that profile exists, and files a card: kanban_create(assignee="<profile>", body="<namespace/workload, symptom, time window>"). If the request omits the cluster name, list_cluster_profiles() returns every profile with the project, cluster and location it is pinned to, so the Platform Agent can find the cluster first.
2. The gateway’s dispatcher auto-spawns the Cluster Agent as a worker on that card. The card inherits the chat subscription of the card that created it, so the user sees the cluster’s progress in the thread.
3. The worker completes the card with the grounded root-cause analysis in result — the field the gateway posts into the requesting chat thread verbatim — and the machine-readable form of it, including the proposed manifest patch, in metadata.
4. The Platform Agent reads the result and, if you asked for the fix, submits it through the declarative workflow (submit-suggestion); if you asked for a diagnosis, the root cause and the proposed patch come back in the reply. The write path never moves to the cluster side.

For multi-cluster work the Platform Agent fans out one card per cluster, keeps its own card open while they run, and completes it with the synthesized answer once every per-cluster card has finished — the image refuses a completion that would hand back a dispatch receipt while they are still running (deploy/docker/patches/kanban_children_settled.py). A per-cluster card asked whether its cluster has something reports “not here” and completes (a card that needed the target blocks as a failure instead), and one still running after 30 minutes is stopped and settled as not finished on that cluster (deploy/docker/patches/kanban_fanout_reaper.py). See the workload-rebalancing skill for the pattern.

Event triage

A Kubernetes event alert arrives as a card like any other. The event watcher posts the event to the Session KV server, which records the chat thread it alerted in and then opens a session named k8s-evt-… on the gateway’s default profile — the Planning Agent — whose whole instruction is to file one card, to the agent scoped to the cluster that raised the event, carrying the diagnostic brief verbatim. A stall the stall-watch cron job finds takes the same path, with a brief that has the Cluster Agent run gke-stall-detection on the namespace.

The Cluster Agent finishes it with kanban_complete and nothing else, passing the whole report as result. Every card carries a subscription to the session it was filed from, and the notifier posts a subscribed card’s result to chat when the card turns terminal — so completing the card is the delivery, threaded under the alert it answers.

The report ends by inviting a reply — To authorize: reply 'apply', and where it proposes more than one remediation option, apply Option B to name one directly. That reply is ordinary chat ingress and lands on the Planning Agent, which did not do the investigation and cannot see the card. What makes it resolve is a row the notifier writes in the same step that posts the report, keyed to the thread it posted into: a pre_gateway_dispatch hook finds it and prepends the report to the user’s words, so apply Option B reaches an agent that knows what Option B was. Acting on it is still the Platform Agent’s job through submit-suggestion — the fix ships as a Pull Request, and nothing is written to the live cluster.

Security posture

A Cluster Agent shares the pod’s identity — the same KSA and GSA, so the same enforced IAM and RBAC ceilings apply (Security & IAM). The profile split is a scoping-down inside those ceilings: fewer MCP servers, no write skills, one pinned cluster context.

Where to go next

• Platform Agent — the orchestrator that owns Cluster Agent lifecycle and the write path.
• Architecture — where the cluster-* profiles sit in the pod.
• Skill catalog — the per-cluster runtime skill group.