Skip to content
kube-agents

Architecture

Ingress paths converge on the agent pod: user chat, scheduled cron, and the operator itself. Inside the pod, one Hermes gateway hosts co-located profiles: the Planning Agent (the default profile — the conversational front door and the only chat ingress), the Platform Agent (the platform profile — the privileged specialist that does the infrastructure work), and read-only per-cluster Cluster Agents (cluster-* profiles the Platform Agent scaffolds, one per managed GKE cluster). They coordinate through a shared kanban board: the Planning Agent files a card, the gateway’s dispatcher spawns the assigned specialist as a worker, and the completion is relayed back into the originating chat thread.

Component map

Section titled “Component map”
flowchart TB
    subgraph Ingress
        GC[Google Chat]
        SL[Slack]
        CR[cron jobs.json<br/>gateway ticker]
    end


    subgraph Cluster["Kubernetes cluster"]
        OP[k8s-operator<br/>controller manager]
        CRD[PlatformAgent CR]
        subgraph Pod["Agent pod (k8s-deployment)"]
            CA["Planning Agent<br/>(default profile)"]
            PA["Platform Agent<br/>(platform profile)"]
            CL["Cluster Agents<br/>(cluster-* profiles)"]
        end
        SEC[Secret<br/>API keys, tokens]
        LM[LiteLLM / vLLM]
        MT[Minty<br/>token broker]
        KMS[GCP KMS keyring]
    end


    subgraph External
        LLM[Gemini / Anthropic / OpenAI / Gemma]
        MCPGKE[GKE MCP server<br/>container.googleapis.com/mcp]
        GH[GitHub GitOps repo]
        OTEL[GKE Managed<br/>OTel + Prometheus]
    end


    GC -->|Pub/Sub subscription| CA
    SL -->|Socket Mode / bot token| CA
    CR -->|scheduled tick| CA


    CA -->|kanban card| PA
    PA -->|kanban card| CL


    OP -->|reconciles| CRD
    CRD -->|owns| Pod
    SEC --> Pod


    CA -->|Completions API| LM
    PA -->|Completions API| LM
    LM -->|hosted| LLM
    LM -.->|local GPU| LLM


    PA -->|MCP proxy| MCPGKE
    CL -->|MCP proxy, read-only| MCPGKE
    MCPGKE -->|read/write scoped| K8S[Kubernetes / GKE API]


    PA -->|submit-suggestion| MT
    MT -->|sign JWT| KMS
    MT -->|installation token| GH


    Pod -.->|OTel spans| OTEL
    CA -.->|reply / relayed progress| GC
    CA -.->|reply / relayed progress| SL

The request flows

Section titled “The request flows”

1. Chat request → reply

Section titled “1. Chat request → reply”
  1. A user posts in a Google Chat space (or DMs the Slack app).
  2. Google Chat publishes the event to a Pub/Sub topic; Slack routes via Socket Mode. The install’s chat-pubsub module sets up the topic and subscription.
  3. The Planning Agent (the pod’s default Hermes profile, image nousresearch/hermes-agent) consumes the event through Hermes’ bundled Google Chat adapter, patched for credential-free operation by google_chat_relay_patch.py (Slack uses slack_relay_patch.py in Socket Mode).
  4. The Planning Agent holds no infrastructure tools. It discovers the available specialists via its router MCP tool (list_agents, descriptions from each agent’s CAPABILITIES.md) and files a kanban card with the full request context (kanban_create); the originating chat session is auto-subscribed to the card.
  5. The gateway’s kanban dispatcher spawns the Platform Agent (hermes -p platform chat -q "work kanban task <id>"), which runs the tool-calling loop: system prompt (SOUL.md) + card body + available tools. Cluster reads go through the gke MCP server; mutations that touch infra route through the submit-suggestion skill (see flow 3) rather than direct kubectl. Work scoped to one cluster’s live runtime is delegated onward as a nested kanban card to that cluster’s read-only Cluster Agent (SOUL.md §6).
  6. The worker’s kanban_complete — its one-line summary followed by its full result — posts back into the same Chat thread, plus one message for each staged sub-card completion along the way. The gateway delivers those directly rather than through the Planning Agent, so the answer arrives verbatim. Console links are built from the shared templates in agents/platform/docs/gcp-console-links.md (per SOUL.md §5).

2. Cron tick → autonomous run

Section titled “2. Cron tick → autonomous run”
  1. The Planning Agent profile’s gateway — the only ticking one, since cron ticking belongs to a running gateway — evaluates its own jobs.json on schedule. Its profile-cron-tick job runs hermes cron tick against every named profile with work due, once a minute, which is what makes the Platform Agent’s own roster live (see What fires the schedule).
  2. The governance watchdogs live on that roster, agents/platform/cron/jobs.json. A due one runs in its own process with the Platform Agent’s persona, toolsets, skills and max_turns — no kanban card involved. The prompt is usually “read governance/<sop>.md in your profile home and execute”; the path is relative to the profile home, where profile_scaffold.py overlays the baked /opt/platform-template/governance/ directory.
  3. The agent reads the SOP, executes the diagnostic queries, produces findings.
  4. The six fleet audits hand their findings to the fleet-audit skill, which renders and maintains one GitHub issue per audit stream — the ledger — and promotes findings with a mergeable manifest into narrow per-finding remediation PRs linked back to it. A run with no findings closes the ledger as completed and returns [SILENT] — unless it could not read the whole fleet, in which case the ledger stays open and the gaps are reported, or it resolved findings on the way, in which case it says what closed.

The Planning Agent profile’s own job file — agents/chat/defaults/cron/jobs.json — holds nothing but no_agent script jobs: plain subprocesses, no LLM turn, since that profile’s toolsets are stripped to mcp-router, kanban and memory. Four ship: profile-cron-tick above; the hourly cluster-agent-reconcile sweep that keeps Cluster Agent profiles aligned with the live GKE fleet; and the bootstrap-inventory-scan / bootstrap-inventory-delivery pair behind first-run onboarding.

3. Remediation → GitHub PR

Section titled “3. Remediation → GitHub PR”
  1. The agent invokes submit-suggestion with a proposed diff (usually a YAML patch generated from an SOP).
  2. submit-suggestion writes the file(s), commits to a topic branch, and calls Minty for a signed installation token.
  3. Minty signs a JWT with a KMS-held private key and exchanges it for a short-lived GitHub App token.
  4. submit-suggestion pushes the branch and opens a PR against the configured GitOps repo.
  5. The agent posts a Chat message with the PR link.

Deployment topology

Section titled “Deployment topology”

Everything except the LLM (and, for hosted providers, the LLM API endpoint) runs inside your Kubernetes cluster:

  • Namespace: kubeagents-system.
  • Node pools: Autopilot, the shape a fresh install creates, has none to size — Google schedules the operator, Platform Agent, LiteLLM, and Minty, and the sandbox costs no extra nodes. On Standard the default node pool runs everything except the Platform Agent, which runs on the gVisor pool below; --gvisor=false puts it back with the others. If you use vLLM, provision a GPU node pool separately.
  • gVisor node pool for sandboxed skill execution, Standard only (the gke-cluster module’s enable_gvisor_node_pool). The sandbox is on unless the installer’s --gvisor=false turns it off, so a Standard install gets this pool; Autopilot ships the gvisor RuntimeClass and needs no pool.
  • Workload Identity bindings are pre-provisioned by the kube-agents-iam module — the Platform Agent gets a configurable permission set (read-only or custom). See Security & IAM.

Capacity and isolation

Section titled “Capacity and isolation”

The hub footprint (operator, Platform Agent, LiteLLM, Minty) is small and roughly fixed — it doesn’t grow with the number of clusters the agent manages, since those are read remotely through the GKE MCP server. What actually drives node capacity:

  • vLLM. Local inference needs a GPU node pool; hosted LiteLLM does not.
  • gVisor. Sandboxed skill execution puts the agent pod under the gvisor RuntimeClass, which is the default (--gvisor=false turns it off). On Autopilot, the default cluster shape, it costs no extra capacity. On Standard it means a gvisor-pool to run the pod on: one e2-standard-4 per zone, so three nodes on a regional cluster, with no autoscaling.

For production, prefer a dedicated cluster for the harness. The agent GSA can still be given broad GKE permissions through the custom permission set, so a hard cluster boundary limits blast radius if the execution sandbox is ever compromised. A shared cluster (with the kubeagents-system namespace for logical isolation) is fine for development.

Failure modes worth knowing

Section titled “Failure modes worth knowing”
  • LLM outage. LiteLLM proxies retries and can fall back to a secondary provider if configured. SOUL.md §4 retries via the recovery ladder; the agent will report the failure in Chat rather than hang.
  • Silent kanban workers. A delegated card only reaches the chat thread on completion or block. The Platform Agent is instructed (SOUL.md §0) to stage long jobs into scoped sub-cards and propagate the chat subscription onto each so progress stays visible; a worker that exits without kanban_complete/kanban_block fails the task.
  • Cron overlap. A long-running fleet audit can still be going when the next job fires — the audit schedules are staggered by 30 minutes, and github-repo-watcher ticks every 10 minutes regardless. Hermes serializes messages per session, so overlap manifests as queueing, not concurrency. The watcher is a script rather than a model run, so it queues behind nothing.
  • Minty token exhaustion. GitHub App installation tokens are 1-hour scoped. Minty caches and re-mints as needed; if the token refresher fails, submit-suggestion will surface it and route through the recovery ladder. github-repo-watcher has no ladder to route through — it is a script, not a model run — so a broker that refuses surfaces in the room as GITHUB_TOKEN_REFRESH_FAILED, which says a mint was attempted and refused rather than that nobody configured credentials. Why it refused is not carried with it — that reason code is rendered into a chat room, so it stays detail-free. The refusal itself is logged by the envoy-credential-proxy sidecar, which runs the refresh helper and records its stderr with token-shaped substrings redacted; read that, or Minty’s own logs.
  • Pub/Sub subscription drift. If the GChat Pub/Sub subscription is deleted out of band, no chat messages reach the Planning Agent and no error surfaces automatically. Cron-driven runs still work, and the kube-agents-observability skill helps triage.

Where to go next

Section titled “Where to go next”