PlatformAgent CRD

The PlatformAgent resource declares everything the operator needs to run one Platform Agent instance: which Hermes image, which service account, which chat integrations, and which framework-level toggles.

• API group / version: kubeagents.x-k8s.io/v1alpha1
• Kind: PlatformAgent
• Source: k8s-operator/api/v1alpha1/platformagent_types.go
• Sample: k8s-operator/examples/platformagent.yaml

Top-level shape

apiVersion: kubeagents.x-k8s.io/v1alpha1
kind: PlatformAgent
metadata:
  name: platformagent
  namespace: kubeagents-system
spec:
  harness: { ... } # execution environment + framework
  deployment: { ... } # container image, pull policy, containers, volumes
  security: { ... } # service account + Workload Identity
  telemetry: { ... } # OTLP collector endpoint (optional)
  networkPolicy: { ... } # generated egress NetworkPolicy (optional)
  integration: { ... } # Google Chat, Slack, Teams, git forge
  scope: { ... } # projects beyond the management project, and exclusions (optional)
  mode: next # component stack: next or today (optional; absent means today)

spec.deployment, spec.security, spec.telemetry, and spec.networkPolicy are inlined from the shared AgentSpec, so they are common to every agent type. spec.harness is required; spec.integration, spec.telemetry, spec.networkPolicy, and spec.scope are optional. spec.mode is an optional enum (today/next, absent means today), the switch from docs/designs/spec-mode-switch.md between the A2A next stack and the previous today one, which stays supported as a rollback for one release. The CRD declares no default, because the API server applies a schema default to every stored object it reads, and that would move every existing install with no field onto next. The chart sets it from platformAgent.mode, the Terraform composition from platform_agent_mode and install.sh from --mode (recorded in install.env as PLATFORM_AGENT_MODE). A first install through install.sh records next, and an install.sh run over a cluster that already runs kube-agents keeps the mode it runs unless told otherwise. The chart’s null and the composition’s today render no field at all, so a mode set on the CR by hand is left alone. Changing it on a running install is a mode switch rather than a setting: the operator rolls the agent and renders or retires the A2A stack. The next stack’s images ship from release 0.8.0, and the A2A gateway’s Slack backend from 0.9.0: on an earlier operator, Slack stays on the legacy path even under next. Under next the operator additionally renders the A2A stack — the NATS/JetStream component and its provisioning Job, an ingress NetworkPolicy fencing the bus, an egress NetworkPolicy fencing the session pods, a session-pod ResourceQuota, the A2A gateway Deployment, once a chat backend is configured (see the A2AGateway condition), with session-pod spawning armed, the capability verifier Deployment (two replicas, with its own ServiceAccount and a NetworkPolicy fencing it to DNS and the bus — it is the only bus principal permitted to read the cap KV bucket, and it needs nothing else; it also carries a PodDisruptionBudget of maxUnavailable: 1 and a kubernetes.io/hostname topology spread, because a submission arriving while no verifier is ready is refused terminally, so a node drain must not take both replicas at once, and the spread is ScheduleAnyway so a single-node install still comes up), and the auth callout that authenticates bus clients (its Deployment, Service, ServiceAccount, Role and RoleBinding, a cluster-scoped ClusterRoleBinding to system:auth-delegator, the identity-map ConfigMap <agent>-a2a-authmap, and the keys Secret <agent>-a2a-callout-keys; its two replicas also carry a PodDisruptionBudget of maxUnavailable: 1 and a kubernetes.io/hostname topology spread, because while no callout is ready the bus refuses every new connection, so a node drain evicts one callout at a time and waits for its replacement to be ready before taking the other (on a cluster with no other schedulable node, such as a single-node install, that wait lasts until the node is uncordoned or the drain overrides the budget), and the spread is ScheduleAnyway so a single-node install still comes up), the console server that serves the A2A console page and proxies its websocket to the bus (its Deployment and Service, <agent>-a2a-console, and a deny-all ingress NetworkPolicy, <agent>-a2a-console-netpol, so it is reachable only through kubectl port-forward, and that port-forward is also what hands out the console credential), plus ServiceAccounts for the provisioning Job and for the spawned session pods, so the callout has an identity to resolve each by. When that chat backend is Google Chat, the A2A gateway consumes the install’s Chat subscription in place of the Planning Agent, whose Google Chat platform is then not rendered: the gateway also carries spec.integration.googleChat.homeChannel, and the agent’s proactive messages reach that space through the gateway, which posts them as new threads there or as replies on its threads and nowhere else (with homeChannel unset, or not a spaces/<id> name, the agent posts nothing to Google Chat unprompted; unset, the CR carries the HomeChannelUnset condition). Disabling Google Chat afterwards, with no other backend configured, scales the existing gateway to zero replicas until a backend returns (see the A2AGateway condition). Flipping back to today tears that stack down, keeping the generated credentials Secret and the JetStream PVC. The callout’s keys Secret is not kept: a stale issuer key would make every answer the callout gives be refused, so it is deleted with the rest.

Under next the A2A gateway serves Prometheus counters on a metrics-only port, 9096 (container port a2a-metrics, set by A2A_METRICS_PORT), which the chart’s <name>-a2a-gateway-monitoring PodMonitoring scrapes behind platformAgent.podMonitoring; the metrics are listed on Observability. The gateway’s NetworkPolicy, <name>-a2a-gateway-netpol, renders wherever the gateway does, eval door or no door, and stays while the gateway is scaled to zero replicas for want of a backend; it admits the gke-gmp-system namespace and the operator’s own pods to that port and nothing else, the operator because its usage poller reads the gateway’s inbound-message counter into status.usage.chatMessagesInboundTotal.

spec.harness

Framework-level settings passed to Hermes. clusterName, location, and projectId are all required — the API server rejects a PlatformAgent that omits any of them. The credential proxy only renders its kubeconfig bootstrap (the gcloud container clusters get-credentials that gives the agent a usable kubectl context) when it has the complete triple; with one missing, every kubectl the agent runs resolves to localhost:8080 instead of a cluster.

Field	Type	Purpose
clusterName	string	Logical cluster name (e.g. cluster-a). Surfaces in observability and chat replies.
location	string	Cloud region (e.g. us-central1-a).
projectId	string	GCP Project ID of the cluster. Required.
hermes.dashboardEnabled	bool	Toggle the Hermes dashboard endpoint. Default true.
hermes.pluginsDebug	bool	Enable plugin-level debug logging. Default false.
hermes.agentHome	string	Path to the AGENT_HOME directory. Default /opt/data.
hermes.apiServerSecretRef.name + key	string	Secret holding API_SERVER_EXTERNAL_KEY, the credential outside callers present to the gateway pod’s agent-api-auth sidecar. Not API_SERVER_KEY — see How config reaches each profile.
hermes.sessionKVApiKeySecretRef.name + key	string	Secret holding the bearer token for the pod-local Session KV server (SESSION_KV_API_KEY). Optional; absent, the server rejects every request with 503.
hermes.sessionKVSaltSecretRef.name + key	string	Secret holding the HMAC salt used to pseudonymise chat identities (SESSION_KV_SALT). Optional; absent, the agent generates a per-pod salt and warns.
memory.memoryEnabled	bool	Toggle framework memory persistence. Default false.
memory.provider	string	Memory provider implementation. Default multiuser_memory; none for none. See below.
memory.userProfileEnabled	bool	Toggle per-user memory profiling. Default false.
eventWatcher.enabled	bool	Start the k8s-event-watcher. Default true; false is the emergency stop for an event storm (see below).
driftDetector.enabled	bool	Start the drift-detector. Default false, because it needs a Pub/Sub subscription a hand-written CR or a Helm-only install does not create; install.sh creates one and sets this to true unless told otherwise. See below.
driftDetector.subscription	string	Pub/Sub subscription the detector pulls audit records from. Unset takes the detector’s own default, which is the name the Terraform module creates.
driftDetector.gitopsManagers	string	Comma-separated managedFields field managers belonging to your GitOps controller, matched exactly — argocd-controller, flux. Unset means no card is ever annotated as possibly already reconciled.
stallWatch.excludeNamespaces	[]string	Namespaces the stall watch never reads, on every cluster, as exact names. Unset = the system namespaces and annotated namespaces are skipped. See below.
stallWatch.minStallAgeMinutes	int	Minutes a stall must last before the stall watch raises a new alert for it (0–1440). Unset = 0, alert on the first sweep that sees it.
stallWatch.kinds	[]string	Resources the stall watch scans, replacing its default list; [all] scans every namespaced kind. Unset = the default list.
tuning.<persona>.apiMaxRetries	int	Model-call retries before a run gives up. Unset = Hermes default 3.
tuning.<persona>.maxTurns	int	Iterations allowed in a single turn. Unset = Hermes default 90, except platform (see below).
tuning.maxInProgress	int	Board-wide cap on concurrent kanban workers. Unset = operator default 6.
tuning.maxSessions	int	Install-wide cap on concurrent A2A session pods (1–10000). Unset = operator default 10. Inert under mode: today; a separate lane from maxInProgress. A stream too small for it makes the provision Job refuse.
experimental.platformFrontDoor	bool	Unsupported. Run the gateway as the Platform Agent, so chat reaches it directly. Default false. See below.

dashboardEnabled publishes port 9119 on the agent Service, but nothing answers there: hermes dashboard binds 127.0.0.1, so a request arriving over the pod network gets connection refused. Reaching it means getting inside the pod’s network namespace — kubectl port-forward svc/<agent> 9119:9119 on an ordinary node pool, and scripts/hermes-dashboard-tunnel.py on a GKE Sandbox (gVisor) one, where port-forward is set up in the host-side netns and cannot see the sandbox’s listener. That script is canonical on the dashboard’s access path and on why the loopback bind is deliberate; the exec relay it uses to get inside the sandbox lives in scripts/exec_tunnel.py, shared with the E2E suite, which reaches the agent API the same way and for the same reason. The container’s readiness probe runs curl against loopback for the same reason a tcpSocket probe cannot work here: kubelet dials the pod IP, and nothing is listening on it.

sessionKVApiKeySecretRef is optional in the API but not in practice, and the 503 above is the milder half of what its absence costs. The k8s-event-watcher in the agent-api-auth sidecar authenticates to that same server, treats an empty SESSION_KV_API_KEY as fatal, and exits on every start — so no cluster events are watched at all, while the container stays Ready and the CR .status says nothing. An installation upgraded from before the key existed is the case that lands here; add the key to the agent Secret and restart the pod.

spec.harness.memory

provider picks which long-term memory implementation the agents load. Two ship in this repository, and the difference between them is the whole choice:

Value	Fits	What it costs to run	What it gives
multiuser_memory (default)	small or personal installs	nothing — a per-user Markdown file inside the pod	verbatim recall of everything, no ranking or search
kube_agents_memory	enterprise deployments	a Hindsight API server and a Postgres database in-cluster	ranked recall, per-user and shared scopes, consolidation
none	—	nothing	no provider; Hermes’ built-in store only

The split is about how the store is read, not about how good it is. The file provider concatenates everything into the system prompt on every turn, so it is bounded by the context window; Hindsight retrieves only what a question needs, so its cost per turn barely moves as the store grows. A fleet of a few clusters and a handful of people will not reach the bound, and paying for a database there buys nothing.

Anything else is passed through to Hermes untouched, so its own external providers (hindsight, mem0, openviking, …) work if you bring their configuration. none is this API’s spelling of Hermes’ empty string, which cannot be expressed here: an absent field takes the CRD default.

Only a Hindsight-backed provider reaches the specialist profiles, because only that one can be made read-only and scoped by tag. Under any other value the specialists get no provider at all and the Planning Agent keeps the store to itself.

memoryEnabled and userProfileEnabled are a different mechanism — Hermes’ built-in MEMORY.md / USER.md files, which have no per-user scoping. Both providers above replace that store rather than supplement it, so both run with memoryEnabled: false.

The installer’s MEMORY_ENABLED variable is that same built-in store and nothing more; the install copies it into this field unchanged. It is not a master switch — whether the agent remembers anything is provider’s question, and none is how that answers no.

The install reads only the provider: the chart’s hindsight.* values deploy the Hindsight store when the provider is Hindsight-backed (--memory=hindsight), and nothing when it is multiuser_memory or none. See docs/designs/memory.md.

spec.harness.eventWatcher

The k8s-event-watcher runs in the gateway pod’s agent-api-auth sidecar, streams warning events from every managed cluster, and posts each surviving incident to the pod-local Session KV server, which opens an autonomous triage session for it. enabled: false stops it from starting at all. Its Prometheus metrics are served on that sidecar’s port 9095 and scraped by the chart’s <name>-gateway-monitoring PodMonitoring on every replica; the chat_metrics plugin’s port 9097 on the platform-agent container has its own, <name>-gateway-chat-monitoring, which selects the pod carrying the leader label; see Concepts → Observability.

kubectl patch platformagent platform-agent -n kubeagents-system --type merge \
  -p '{"spec":{"harness":{"eventWatcher":{"enabled":false}}}}'

The field has to exist in the installed CRD for that patch to mean anything, and Helm installs CRDs on first install but never upgrades them. A chart-upgraded install therefore needs the CRDs applied first — worth doing ahead of the incident rather than during it, since a client that does not send strict field validation gets the unknown field pruned and an emergency stop that reports success and does nothing.

kubectl apply --server-side -f k8s-operator/config/crd/bases/

--server-side is not optional here. Client-side apply stores the whole object in the kubectl.kubernetes.io/last-applied-configuration annotation, and this CRD is far past the 262144-byte annotation cap, so a plain kubectl apply fails with metadata.annotations: Too long and leaves the CRD unchanged. make install in k8s-operator/ applies them the same way.

This is an emergency stop, not a tuning knob. It exists for the case where events arrive faster than the agent can triage them — a fleet-wide rollout gone wrong, a node pool flapping — and the cheapest way to get the agent back is to cut the inflow rather than chase the cards it has already been handed. It is all-or-nothing across every watched cluster: the watcher’s reason and namespace filters live in the sidecar’s entrypoint and are not exposed on the CRD, so there is no way to silence one noisy namespace through this field. If the board is merely busy rather than swamped, tuning.maxInProgress is the knob for that — it throttles how many cards run at once without losing the events.

Three consequences before you press it:

• It rolls the pod. The value reaches the sidecar as an environment variable, so changing it rewrites the pod template. During a storm that restart is usually wanted anyway — it is also what ends the sessions already running.
• It stops the inflow only. Kanban cards and sessions created from events already delivered keep running and still have to be dealt with on the board. It reclaims nothing either: the watcher’s kubeconfig, token projection, and mounts stay in place, and the sidecar keeps the memory request sized for the caches it is no longer running.
• Nothing turns it back on. An install left with the watcher off has no incident detection at all, and the container stays Ready throughout — the readiness probe covers the credential proxy, not the watcher. Two things say otherwise: a line in the sidecar log naming the consequence, and the EventWatcher condition on the CR (status below). Set enabled: true, or remove the field, to start watching again.

Unset means enabled. The watcher is how a fleet notices its own incidents, so an install that never mentions the field — which is every install today — keeps watching, and only an explicit false turns it off.

spec.harness.driftDetector

The drift-detector runs beside the watcher in the gateway pod’s agent-api-auth sidecar. It pulls GKE admin-activity audit records from a Pub/Sub subscription, drops every call that a person did not make or that changed nothing, and posts what is left to the same pod-local Session KV server the watcher posts to — so a change someone made to a cluster by hand arrives on the board as a card. The control plane, CI, and every service account are dropped alike; on a busy cluster that is the overwhelming majority of the stream. A change on another cluster in the project that no readable Cluster Agent profile names is logged in the sidecar and not reported, while the sidecar can read its profiles directory: the install’s scope ends at its profiles, so profile the cluster, exclude it under spec.scope.exclude.clusters, or fix the profile the sidecar’s log names. The management cluster is always reported, profile or not, and a cluster the sidecar had already joined when it started stays reported after its exclusion until the gateway pod restarts.

Through install.sh it is on unless you turn it off; the field itself still defaults to false. The two layers differ on purpose — the field is what a hand-written CR or a Helm-only install sets, and neither of those creates a subscription for the detector to read, so the API default stays off and the installer is what turns it on. It is not free either, and what it costs is GCP resources rather than cluster resources: the subscription it reads comes with an audit log sink and a Pub/Sub topic, all three from the drift-pubsub Terraform module, and the sink exports the admin-activity audit records of every GKE cluster in the project; on an install with a declared spec.scope, each project the scope lists gets a sink of its own into the same topic, and a change on a cluster there is enriched through that cluster’s Cluster Agent profile and reported like one on the management cluster. The terraform/examples/full-install composition instantiates the module when you set enable_drift_pubsub = true, and writes the field below when you set enable_drift_detector = true alongside it. Asking for the second without the first is refused by a precondition rather than applied.

If you installed with install.sh, one key covers both: ENABLE_DRIFT_DETECTOR in install.env writes the two variables together, and it defaults to true, so an install that says nothing gets the sink, topic, subscription and detector. To go without them, set ENABLE_DRIFT_DETECTOR=false in install.env. Put it in the file rather than relying on install.sh --enable-drift-detector=false: the run that creates install.env records that flag into it, but install.sh never rewrites that key afterwards, so on every later run the flag applies to the run you pass it to and is recorded nowhere — the next run resolves the default and provisions the three resources again, and upgrade.sh takes no such flag at all. The front doors regenerate terraform.tfvars on every run, so a value written into that file by hand does not survive the next one either.

This reaches an install created before the key existed. Its install.env records no choice, so the next install.sh or upgrade.sh run resolves the default and provisions the three resources — upgrade.sh --plan shows them as additions before you apply.

An install that has neither provisioned the ingress nor applied the module by hand has nothing for the detector to pull. Setting this field there anyway — which now takes a hand-edited CR or chart value, not the composition — gives a process that retries a failing pull for the life of the pod without ever reporting a change, and the pod stays Ready throughout, so an install you switched on by mistake looks exactly like a fleet nobody has touched.

spec:
  harness:
    driftDetector:
      enabled: true

Enabling is necessary and not sufficient. Outside the gateway the detector verifies the cluster name it was given against the cluster its credentials actually reach and exits on a mismatch; inside it, where the metadata server is closed, that probe times out and the triple is taken on trust (see the detector’s README). So the operator starts it only when projectId, location, and clusterName are all set on spec.harness as well. The API server already rejects a PlatformAgent that omits any of the three, so the case to watch for is one present but empty — that stays off however enabled reads, and unlike the watcher, nothing substitutes a placeholder name for it.

projectId has to be the project ID, not the project number. The detector matches it against the project on each audit record, which is always the ID, so a value that is nothing but digits can match nothing; the operator reports the detector as off rather than starting a sidecar that exits on every launch. Everything else reading spec.harness accepts a number, so an install can be healthy in every other respect and still be refused here.

subscription and gitopsManagers are the two knobs worth moving from a CR. Leave subscription unset unless you renamed the one the module creates.

gitopsManagers is not what separates automation from people — the detector already does that by principal, and a controller’s writes never reach a card. It names the field managers your GitOps controller writes under, as they appear in an object’s managedFields: argocd-controller, flux, whatever your own controller sets. That is a field-manager string, not a service-account email, and the two are rarely the same.

Matching one changes what a card says rather than whether it exists. When the detector finds a configured manager wrote the drifted fields after the change it is reporting, it marks the card as possibly already reconciled and names the manager. Unset is supported and simply loses that hint: ownership is still read and reported, and you get the same cards without the annotation.

Nothing else about the detector is exposed here. Which principals count as human, and which calls are dropped as failed or non-declarative, are compiled into the binary; what reaches it at all is set by the Terraform module’s log sinks, not by the CR.

spec.harness.stallWatch

The controller stall watch is the stall-watch cron job on the Platform Agent’s roster. Every 30 minutes it runs stall_report.py over the namespaces of each cluster that has a Cluster Agent profile, and raises one alert for each namespace with a new stall (autonomous watchdogs). This block tunes what it reads and when it alerts. Absent, the watch skips the system namespaces and annotated namespaces, alerts on the first sweep that sees a stall, and scans its default list of kinds.

spec:
  harness:
    stallWatch:
      excludeNamespaces: [load-test, payments-canary]
      minStallAgeMinutes: 30
      kinds: [deployments, statefulsets, gateways.gateway.networking.k8s.io]

excludeNamespaces names namespaces the watch never reads, on every cluster. Each item is an exact namespace name; there is no wildcard. If a namespace has an open alert when you exclude it, the watch completes the alert’s card with a note that the namespace was excluded, and the chat line says the same. Up to 100 names.

minStallAgeMinutes holds a new alert until at least one stall in the namespace has lasted that long. A stall’s age is the longer of two values: the age stall_report.py measures on the object (from its condition, its last spec write, or its warning events), and the time since the watch first saw the object stall. A spec write does not restart that second value, as long as the watch saw the object stall on one of its last two sweeps. A younger stall is not dropped: the watch keeps it in its ledger and checks it again on each sweep, so the alert follows on the first sweep after the stall is old enough. An object that starts to stall in a namespace with an open alert is added to that alert’s card at once. The range is 0 to 1440; unset or 0 alerts on the first sweep that sees the stall.

kinds replaces the default list of resources the watch scans: Deployments, StatefulSets, DaemonSets, Jobs, Gateway API Gateways and HTTPRoutes, and cert-manager Certificates. Write each item as kubectl names the resource, in lower case: deployments, gateways.gateway.networking.k8s.io. The single item all scans every namespaced kind the cluster serves, with one read per kind in each namespace, which is slow on a large fleet; the API server refuses all next to another item, and refuses Secrets, ConfigMaps, Events, Endpoints, EndpointSlices, ControllerRevisions and Leases, which the scan reports nothing from. Up to 20 items. A kind a cluster does not serve is left out on that cluster.

Separate from this block, the annotation kubeagents.x-k8s.io/stall-watch: exempt opts out without a CR change. On a namespace, the watch does not read the namespace, as if excludeNamespaces named it. On an object that stall_report.py reports (a Deployment, a Gateway, or any other kind the watch scans), the watch leaves that object’s rows out of its alerts. If that leaves an open alert with no stall, the watch completes the alert’s card with a note that names the annotation. The annotation is checked on each object alone: an exempt Deployment does not exempt the Gateway or the HTTPRoute in front of it. A Cluster Agent that runs the gke-stall-detection skill by hand still reports an exempt object.

The operator delivers the three values through the agent’s managed .env (see How config reaches each profile), so a change rolls the agent pod, and the agent cannot override a value from its own .env.

The chart sets this block from platformAgent.harness.stallWatch. On an install made with install.sh, put the block in the TF_VAR_extra_helm_values line in install.env, which a full-mode install.sh or upgrade.sh run applies; a helm upgrade --set value is dropped by the next full-mode run, and the watch then goes back to its defaults. install.env holds one such line, so if it already has a platformAgent key, put harness={stallWatch={...}} inside that key (Operator settings says how the line works):

TF_VAR_extra_helm_values='{platformAgent={harness={stallWatch={excludeNamespaces=["load-test"],minStallAgeMinutes=30}}}}'

On a hand-driven Terraform apply, set extra_helm_values in terraform.tfvars instead; on a bare Helm install, set the block in your values file.

spec.harness.tuning

Execution limits per agent persona, where <persona> is one of default (the Planning Agent front door), platform (the Platform Agent), or cluster (every Cluster Agent), plus the board-wide maxInProgress and the mode-next session-pod cap maxSessions.

The per-run limits are opt-in. The operator pins nothing of its own there: what a fleet needs depends on its model quota and on what its agents actually do, so a deployment doing short interactive work should not inherit limits raised for long-running batch work. Unset therefore means whatever the profile’s own config.yaml carries, and the default and cluster configs set no execution limit of their own — Hermes’ defaults apply there, 3 retries and 90 iterations. The platform profile is the exception: the image ships agent.max_turns: 250 in agents/platform/config.yaml because the fleet audits outgrow 90, and Config reference is canonical for why. Setting tuning.platform.maxTurns here still wins — the overlay is merged after the image force-sync — and removing it restores the image’s value rather than Hermes’.

maxInProgress is not. Unset renders 6, because the untuned case is the one that cannot absorb the alternative — see Why dispatch is capped by default below. Set it on the CR to raise or lower that. Either way the value is pinned in the managed scope at /etc/hermes, so it applies over whatever the agent’s own config.yaml says, and removing the field puts the default back.

spec:
  harness:
    tuning:
      maxInProgress: 8 # board-wide; raises the operator's default of 6
      platform:
        apiMaxRetries: 8
        maxTurns: 200
      cluster:
        apiMaxRetries: 8
        maxTurns: 150

Raised limits belong with the workload that needs them, not with the platform. A long-running, quota-hungry agent plugin should ship its own tuning — as a patch its installer applies — so that a deployment without it stays on Hermes defaults, and installing the plugin brings the limits it requires along with it.

The GKE Stockout Investigator is the worked example: agentplugins/gke-stockout-investigator/tuning.yaml records the reasoning behind each number, and its install.sh applies it.

The keys are personas rather than profile names because the profiles are not all known when the CR is written: Cluster Agent profiles are scaffolded at runtime, one per managed cluster, with generated names like cluster-<project>-<cluster>-<region>. cluster therefore applies to all of them at once — including ones onboarded after the pod last started, which pick the limits up as they are scaffolded.

Both limits matter because they fail the same way, and it is not an obvious way. A run that exhausts either stops mid-task without ever calling a terminal kanban tool. The card is charged a timed_out failure whose error text names how the turn ended — Iteration budget exhausted (N/M) for maxTurns, turn_exit_reason=all_retries_exhausted_no_response for an apiMaxRetries exhaustion that ends with no response — and retrying re-runs into the same wall, so read that text and the upstream error rate before suspecting the worker. One apiMaxRetries exhaustion is handled differently: when the retries were spent on provider rate limits (429s), the worker blocks its own card instead, kind transient, with the provider’s error text as the reason; kanban_unblock it once the quota window has passed. A second rate-limit block after an unblock sends the card to triage. A turn that fails outright, because the provider refused the call (an HTTP 401, a billing wall, a safety block) or the last retry ended on an error, is recorded by the worker itself: the error Hermes reported goes on the card as a comment, the run is charged as crashed and counted, and the card goes back to ready until it reaches its retry limit. The chat thread shows the generic “worker crashed” line for it, so read the card’s comment for the cause. The charge’s own error text names no provider error, so the dispatcher retries the card rather than parking it as an auth or quota failure. Goal-mode cards, which the dispatcher runs as a loop of turns judged against the card’s goal, get none of this: their failures exit through the dispatcher’s reaper as before, with exit 1, or exit 75 and a rate_limited run for a rate limit or a billing wall. A run that stops without a terminal call and reaches the dispatcher unexplained surfaces instead as a protocol violation, which describes the symptom and hides the cause; the image narrows that window in deploy/docker/patches/kanban_guardrail_exit.py.

Sizing notes: maxTurns is consumed mostly by repository exploration, so scale it against how much the agent has to read rather than how complex the request is. apiMaxRetries exists because Hermes’ default of 3 assumes an interactive session where a human retries; a background worker has nobody to retry it, so a transient burst of upstream 503s simply ends the run. A 429 retry waits out the delay Google states in the error body (up to 600 s) rather than a few seconds, so a burst shorter than that window is survived; see deploy/docker/patches/rate_limit_retry_delay.py. Raising maxTurns interacts with maxInProgress: a worker doing the work holds its slot for the whole task and there are only maxInProgress of them, so raising one is a reason to reconsider the other. A coordinator waiting on work it fanned out is the exception — see why dispatch is capped.

Why dispatch is capped by default

A kanban worker is not a coroutine. It is a full hermes … kanban task process — a few hundred MiB resident once its MCP proxies are up, and alive for as long as the task takes, which for an incident triage is minutes rather than seconds. Uncapped, the dispatcher starts one per queued card, and a burst of cluster events queues them faster than they retire.

What follows is invisible in the places you would look. The cgroup OOM killer takes a worker, not PID 1, so there is no container restart and no Kubernetes event; the pod stays Running and the only trace is pid not alive in the kanban ledger. The dispatcher’s retry budget is 1, so the card is stranded rather than re-dispatched, and the work it stood for is never done — a triage report that simply never arrives, with nothing anywhere reporting a failure.

6 is a floor for a deployment that has not measured itself, not a recommendation. It assumes the operator’s default 8Gi memory limit on the agent container, and the cost of being wrong is asymmetric: too low delays a delegated task, too high loses it silently. An install that lowered the agent’s memory limit through spec.deployment.resources (which replaces the defaults as a block) should set maxInProgress to fit: about 430 MiB per worker over the 1.8 GiB the pod holds idle, so a 4Gi limit fits about five. Raise it once you know your worker footprint and your model quota — that quota is the other shared resource, and for most deployments it binds before memory does.

The cap comes from the operator, not from the image or the agent’s own config.yaml. The operator pins kanban.max_in_progress in the managed scope at /etc/hermes: the CR’s value, or 6 when it says nothing. Hermes applies that file over the agent’s config on every load, so a value an existing volume was first seeded with, a removed override, or an edit the agent made to its own file has no effect. An install upgraded onto an operator that changes the default takes the new number on its first reconcile and the pod roll that follows; the dispatcher logs kanban dispatcher: max_in_progress=<n> at startup (Hermes’ _positive_int_setting in gateway/kanban_watchers_common.py, which logs each configured kanban cap as it reads it). The max_in_progress line in agents/chat/config.yaml applies only to an image run without the operator.

One slot is guaranteed to each class of card. A card is classed when an agent files it with kanban_create: one filed from an event-triage or cron-relay session is background, and one filed in a chat turn or through the inject and A2A doors is a user card. A card filed by hand, with hermes kanban create or from the dashboard, never passes through that step and is background unless it is given priority 100 or more (--priority 100); the dashboard can change a card’s priority afterwards. So are the children Hermes’ auto-decompose files for a card created with triage: true: they are written at priority 0, so a user card filed that way runs its decomposed work as background.

At a cap of 2 or more, background cards may hold every slot but one, so a question asked in chat starts at once even while triage is running, and user cards may hold every slot but one, so a door’s fan-out cannot silence alerts. The slots between go to whoever is first, and user cards sort first. At the default of 6 that is one slot for each class and four shared. The price is that a burst of alerts drains more slowly than it would with every slot open to it, and a burst of user work likewise. At a cap of 2 each class gets exactly one. At a cap of 1 nothing is held: a user card still goes ahead of waiting triage, but it waits for the running card to finish. A top-level user card that waits, because every slot is busy or the only free one is triage’s, gets one line in its thread: ⏳ Queued: the system is busy. Your request will start when a worker frees up. A card filed as part of a request already running (a worker’s child, or a card with a parent link) waits without one.

A full board is logged as what it is. When every slot is busy for six ticks in a row the gateway logs, at most every five minutes:

kanban dispatcher saturated: 6/6 worker slots busy (5 background, 1 user: t_ab12 @cluster-prod 14m [k8s-evt-], t_cd34 @cluster-prod 9m [k8s-evt-]); 1 user card(s) and 2 background card(s) waiting

The card list is shortened here. The real line names up to five cards that hold slots, oldest first within each board, boards in turn; a coordinator only waiting on its own children is left out, because it is not counted as running. The line gets ; N background card(s) held back because one slot is reserved for user cards or ; N user card(s) held back because one slot is reserved for background triage added when a held slot is what stopped them. When the only free slot is the one held for the other class, the line starts kanban dispatcher holding background cards: or kanban dispatcher holding user cards: instead. That line is load, not a fault. The older kanban dispatcher stuck: … Check profile health warning now means what it says: slots were free and still nothing started.

The arithmetic behind 6:

• Gateway memory. One worker measured about 430 MiB on a live install: hermes chat 207 MiB, two Node MCP proxies about 100 MiB each, a supervisor and an ssh. Six of them are about 2.6 GiB over the gateway’s 1.8 GiB idle set, about 4.4 GiB under the container’s 8Gi limit. A coordinator waiting on its own children gives its slot back but stays resident, so the process count can sit above six; the 8Gi limit leaves room for about fourteen workers at that size.
• Credential proxy. Every worker’s kubectl and gcloud runs through the proxy, which admits a request only when its children fit the proxy’s memory limit: 176 MiB per request after 320 MiB of fixed reserves. At its default 2Gi limit that is 9 at once (credentialProxyAdmittedRequests in k8s-operator/internal/controller/credential_proxy_manifests.go), held to 8 by its slot cap. At the old 1Gi it was 4. The 8 slots are shared: the stall watch and the cluster-agent reconcile each list four projects at a time through them. With neither listing, all 8 are free for the six workers; while one lists, 4 are; while both list at once, none, and a worker command waits up to 60 s for a slot and is then refused busy.
• Model quota. Per-install model rate limits are not measured. A small quota may see 429s at 6; if worker logs show them, lower maxInProgress.

If you raise maxInProgress, check your model quota, and know where the credential proxy stops it. The proxy’s default 2Gi already admits more than its slot cap of 8, so up to about eight workers’ worth of commands fit, fewer while a listing runs. Past that the slot cap binds, and the operator owns it: no CR field moves it, so raising the proxy’s memory in spec.deployment.credentialProxy.resources does not help, and a command beyond eight waits up to 60 s for a slot and is then refused busy. Keep that limit at 2Gi or more, because below it the memory budget binds first.

The cap counts running cards, not resident processes, and one case makes those differ: a coordinator waiting on work it fanned out is discounted, or it would hold the slot its own children need (kanban_scheduling.py). Peak memory is therefore the cap plus however many coordinators are waiting, held down by the dispatcher’s memory-pressure guard rather than by this number.

spec.harness.experimental

Opt-in switches with no compatibility promise. A field here may change meaning, change its default, or be removed outright in any release; an install that depends on one is expected to re-check it at every upgrade. Fields live here while the question they answer is still open — once it is settled the switch either graduates into a supported block or goes away.

platformFrontDoor

Makes the Platform Agent the profile the Hermes gateway runs as, so a chat message is handled by the agent that has the tools instead of arriving at the Planning Agent, which delegates through the router MCP server and the kanban board.

kubectl patch platformagent platform-agent -n kubeagents-system --type merge \
  -p '{"spec":{"harness":{"experimental":{"platformFrontDoor":true}}}}'

Three things change while it is on:

• The gateway container runs hermes --profile platform gateway run.
• profile-platform.overlay.yaml gains the three profile-shaped things only the default profile carried before: the toolsets each chat platform key resolves, the ingress plugins, and the kanban block. The adapters themselves are not copied — the managed scope at /etc/hermes is machine-global, so platforms.*, display.platforms and the board’s worker cap (tuning.maxInProgress) already land on this profile. The rest of kanban does have to follow the gateway, because the dispatcher and the notifier run in the gateway process and read their settings from its own home. The board itself does not move — Hermes anchors kanban.db at the shared root rather than the active profile, deliberately, so the dispatcher/worker handoff survives — so cards in flight are unaffected by the flip.
• The entrypoint stops force-syncing profiles/platform/config.yaml from the image and back-fills it instead, on the same terms as the default profile’s own file (except the remote MCP server entries and the plugins.enabled list, which follow the image), so /sethome and monitoring.install_id survive a restart — see How config reaches each profile.

Setting the field back to false reverses all three. The overlay records what it applied, so the keys are unapplied rather than left behind, and the force-sync resumes.

What it costs. The Planning Agent’s lockdown is its whole reason for existing: a front door with three toolsets, so an inbound message cannot reach the full Platform Agent tool surface before a card and a worker turn have framed it. With this on, an inbound message reaches that surface directly. The lockdown is deliberately not copied onto the platform profile — copying it would leave the Platform Agent unable to do the work the flag exists to let it do.

Known limits.

• One gateway means one profile, so this is not additive. While it is on, the Planning Agent persona sees no chat at all and the router MCP path is simply unused. Kanban delegation is not: the front door keeps dispatch_in_gateway, so it can still hand a card to a spawned worker — it just does so as the agent that could also have done the work itself.
• gateway.multiplex_profiles is still off, so a /p/<profile>/ prefix on an API request is ignored. Those requests now land on the Platform Agent rather than the Planning Agent.
• The hermes dashboard sidecar is deliberately left on the default profile, so the dashboard shows that profile’s sessions while the front door is the platform one.
• An AgentPlugin without a spec.targetProfile does not follow the gateway. The two halves that decide whether it loads at all stay on the default profile: the image volume is mounted at $HERMES_HOME/plugins/<name>, and the name is added to that profile’s plugins.enabled. With the flag on, the gateway is homed at profiles/platform and reads neither, so the plugin never registers — silently, since nothing errors, and its platforms: block does arrive here through the machine-global managed scope, configuring an adapter that has nothing to configure. The pubsub-platform adapter and the gke-stockout-investigator alert route that depends on it are both affected. Setting spec.targetProfile: platform moves the plugin and its platform_toolsets across.
• The default profile’s cron roster stops ticking. Hermes binds its cron ticker to one HERMES_HOME — the job store, the execution ledger and .tick.lock all resolve from the gateway process’s own home — so the one roster that ticks becomes profiles/platform/cron/jobs.json. The Platform Agent’s own watchdogs therefore tick natively, which is the upside; the cost is that the three jobs on the default roster never come due. Those are cluster-agent-reconcile, which scaffolds a profile for a newly onboarded cluster and prunes one for a deleted cluster; oobe, which runs first-run onboarding (the bootstrap-inventory-* entries ship disabled); and profile-cron-tick, the only thing that ticks a named profile’s own store — so every cluster-* roster goes quiet with it. Nothing errors and nothing is logged: a job that is never ticked simply stays scheduled with a next_run_at in the past.
• Per-user memory does not follow the gateway either. The front door’s provider is multiuser_memory, committed in agents/chat/config.yaml and reaching the default profile alone. The platform profile is configured as a specialist instead: read_only: true from the image, and a memory.provider its overlay blanks unless spec.harness.memory names a Hindsight-backed store. With the flag on the front door is that profile, so chat has no recall, no per-user profile and no retention — while memory stays in the profile’s toolsets advertising all three. The keys cannot simply be copied across, because profiles/platform/config.yaml is shared: it is also the home of every kanban-spawned specialist and every job on the platform cron roster. Lifting read_only there would let a specialist write the shared corpus, and un-blanking the provider would collapse those writes into one anonymous bucket, since a specialist carries no gateway identity to scope a per-user store by. Carrying memory to the front door needs those settings to be per-process rather than per-profile.
• First-run onboarding does not follow the gateway. Its job, oobe, is on the default roster the bullet above stops ticking, and its greeting hook — the bootstrap_onboarding plugin — resolves its once-per-deployment markers from the gateway’s home, so on the platform profile it would greet an already-onboarded install and promise a report nothing can deliver. The operator therefore leaves that plugin off the front door deliberately. Bring an install up with the flag off, let onboarding finish, then turn it on.
• A home channel set with /sethome, or by the first chat message, stays on the profile it was set on. The operator renders no home_channel of its own, so on an install that did not populate the CR’s Google Chat or Slack homeChannel the value lives only in the config file the gateway last wrote. Flipping the flag changes which file that is, and nothing carries it across. The Platform Agent’s own deliver: all watchdogs then tick in-process, where the delivery target is read from the environment alone — so every scheduled report posts nowhere while chat replies stay healthy and the install looks fine. Either populate homeChannel on the CR before flipping, or re-run /sethome after.
• chat_message_audit stops recording. It is a hook rather than a plugin, and hooks are only ever copied into the root home — nothing puts them on a named profile, and Hermes reads $HERMES_HOME/hooks and returns silently when the directory is absent. tool_call_audit is a plugin, is enabled on this profile and still records the inbound message with its session and pseudonymised user id, so what is actually lost is the record carrying the agent’s response text. Response content remains in the session store.
• The platform profile’s config.yaml stops being restored from the image on every restart. Handing the file to the agent is the point — that is what lets /sethome and monitoring.install_id survive — but the same change means a key the running agent writes there is not reverted at boot. Keys the image adds still arrive through the back-fill; keys already in the file stay as they were last written, except the remote MCP server entries, which follow the image (below). Operator-owned settings are unaffected: they come from the overlay and the /etc/hermes pins, both re-applied every boot.
• Cluster profiles that already exist are otherwise unaffected — their config, skills and scaffolding on disk are unchanged and they keep working. What stops is the scheduled work above, which includes cluster-agent-reconcile, so a cluster onboarded while the flag is on gets no profile until the flag goes back off.

shellSandbox

• enabled — the agent’s shell runs in a StatefulSet of its own, reached over SSH with the keypair in the agent’s credential Secret (SANDBOX_SSH_PRIVATE_KEY and its public half in <name>-shell-authorized-keys). This is not a toggle: false is refused with Degraded/ShellSandboxCannotBeDisabled, which changes nothing about the running workload. Absent or true are the same thing. With no keypair the sandbox Pod cannot start at all — the Secret it mounts is not optional — so the operator reports Degraded/ShellSandboxKeysMissing rather than leaving the Pod in ContainerCreating with the reason only in a Pod event. install.sh, upgrade.sh and the Terraform composition generate the pair; a bare helm install that passes none, or a kustomize install (INSTALL.md Method 2) that skipped its Step 2, reaches that state.
• image — overrides the sandbox image. Empty takes the operator’s default. A pinned image must be at least as new as the operator: the operator runs the image’s entrypoint in an init container that stages the image-shipped trees, and an older image does not have that mode, so the sandbox Pod never starts.
• runtimeClassName — runs the sandbox Pod under a sandboxed container runtime, gvisor being the one GKE offers. Unset by default. Separate from spec.deployment.availability.runtimeClassName, which governs the agent Pod: that Pod holds SQLite databases whose WAL mode gVisor corrupts on the gofer-backed mount, and setting the agent Pod’s field pins Hermes’ own databases to the DELETE journal mode (the Session KV store is not covered; see availability.runtimeClassName under spec.deployment), while the sandbox Pod holds none — so an install can sandbox the untrusted Pod without sandboxing, or slowing, the trusted one. On GKE Standard the cluster needs a node pool created with --sandbox type=gvisor; Autopilot ships the RuntimeClass natively. A RuntimeClass the cluster does not have leaves the CR Degraded naming it, rather than a Pod sitting Pending.

The GitHub-writing skills hand the credential broker file content and a commit message rather than a directory both sides mount, so the agent never holds a .git. There is no field for it: the broker keeps the checkout on its own state volume, which closes git’s config-driven exec surface — a hook, a pager, a filter.*.clean, an ext:: transport — and an install that could turn that off would be choosing to keep it open.

The credential broker is a Deployment of its own, never a container of this Pod, so a proxied command reaches it over a ClusterIP Service and nothing crosses as a path: a document travels on stdin, a kubeconfig as a GKE context name, a commit as file content. spec.security.workloadIdentityFederation is optional hardening on top of that, not what places it.

spec.deployment

Abstracts the pod/deployment configuration. The controller synthesises a Deployment from these plus the workspace ConfigMaps. Available fields:

• image — container image repository.
• tag — image tag. Applies only when image is set without a tag or digest, falling back to latest there; when image is omitted, the operator’s default platform-agent version applies instead.
• imagePullPolicy — one of Always, Never, IfNotPresent. Default IfNotPresent.
• imagePullSecrets — Secrets in the agent’s namespace holding registry credentials, as - name: <secret> entries. Referenced, not created: each must exist before the pod is scheduled. Pod-scoped, so it covers the agent, both injected sidecars, anything in initContainers/sidecars, and the OCI image volumes AgentPlugins mount.
• browserArgs — extra command-line args for the agent’s browser (e.g. --no-sandbox).
• availability.replicas — the gateway’s replica count, default 1. The operator runs the gateway as a single replica: a value above 1 renders exactly what 1 renders (one replica, Recreate, no leader election, ReadWriteOnce default claims), and the ReplicasLimited condition (status below) says so. High availability is not supported yet. Above one replica the gateway container would run leader_elect.py, which needs an in-cluster ServiceAccount token that container deliberately does not have, so it exits at startup; and the data volumes an install creates at one replica are ReadWriteOnce, so a replica scheduled on another node cannot attach them. The field is not refused at admission, so a CR that already asks for more keeps reconciling. On such an install, claims created earlier keep their access mode, since claims are created once; and a gateway that ran as a StatefulSet (more than one replica over ReadWriteOnce storages) is replaced by the Deployment, which mounts new claims. Set it to 1, or remove it, to clear the condition. That can leave the ReplicaSet from the higher count behind, still running pods that crash-loop: if gateway pods keep failing after the condition clears, find it with kubectl get rs -n <namespace> -l app=platform-agent-gateway (a ReplicaSet that isn’t the current one but still has replicas) and run kubectl scale rs <name> -n <namespace> --replicas=0.
• availability.runtimeClassName — pod runtime class (e.g. gvisor) for the agent Pod. Nested under availability alongside replicas, nodeSelector, tolerations and affinity. When set, the managed config also carries database.journal_mode: delete, and the entrypoint converts Hermes’ existing databases out of WAL once at start-up — state.db, kanban.db and the cron, project, evidence, response, memory and Discord stores Hermes opens through the same journal-mode helper: a sandboxed runtime serves the data volume over a gofer mount that accepts SQLite’s WAL mode and then corrupts it (#610). The Session KV store (session_kv.db on the system-metadata volume) sets WAL itself and is not covered by either. Clearing the field drops the pin, and the databases return to WAL on their next open.
• env — additional container environment variables.
• resources — requests and limits for the agent container, replacing the operator’s defaults as a block.
• credentialProxy.resources — requests and limits for the credential-proxy container, the broker that runs every credentialed command in a pod of its own. Merged over the operator’s defaults per key, unlike resources: a CR that sets only limits.memory keeps the default 500m CPU request, 1 CPU limit and 2Gi ephemeral-storage limit, which bounds the content workspace. The proxy’s state and /tmp emptyDirs (size limits 5Gi and 2Gi) follow that limit when it is raised above their defaults, so the kubelet does not evict the pod at the smaller figure. The broker sizes how many commands it admits at once from the memory limit, so that is the key to raise when the proxy is OOM-killed under a large fleet. Only cpu, memory and ephemeral-storage are accepted, the quantities the container declares. The operator refuses a memory limit below 672Mi (two commands at once), a limits.memory set alone below the 1Gi default request (set requests.memory with it), a request above its limit, a negative quantity, a zero limit and claims, which the proxy pod has no use for. A refused override, including an edit of one that was valid, renders the proxy Deployment at the operator’s defaults rather than at the last accepted override until it is corrected, which restarts the proxy once (its Deployment is Recreate), and the agent stays Ready whether or not the webhook is on. The Degraded condition carries one reason, and reports InvalidCredentialProxyResources when no higher-ranked Degraded cause is present. Where the validating webhook is enabled, admission refuses the same override at apply and the running proxy is untouched. Admission also warns when memory per CPU on the requests pair leaves the 1 to 6.5 GiB per vCPU band GKE Autopilot admits unchanged, because Autopilot then raises the smaller request and the chart’s quota preflight is short by the difference; Autopilot applies the band to requests only. It warns too for each of cpu and memory set under limits without the same key under requests, unless the limit equals the request it would be replaced by: Autopilot without bursting sets the limits equal to the requests, so a CR that sets only limits.memory: 3Gi runs at the 1Gi request there. Set the request to the same value, or enable bursting, under which the declared limits stand.
• agentAPIAuth.resources — requests and limits for the agent-api-auth sidecar in the gateway pod, the native container that authenticates the agent’s API-server calls and also runs the event watcher and the drift detector. Merged over the operator’s defaults per key, like credentialProxy.resources: a CR that sets only limits.memory keeps the default 150m CPU request, 384Mi memory request, 1 CPU limit and 2Gi ephemeral-storage limit. The event watcher reads the memory limit through the Downward API and sets its Go soft memory limit to half of it, so raise limits.memory when the watcher is OOM-killed under a large fleet of watched clusters. Overriding limits.cpu also changes the watcher’s GOMAXPROCS, which the Go runtime derives from the container’s CPU limit, so leave limits.cpu at the default unless you mean to. Only cpu, memory and ephemeral-storage are accepted. The operator refuses a request above its limit, a negative quantity, a zero limit, claims, and any other resource name, whether or not the webhook is on: it ignores the override, renders the sidecar at its defaults, and reports Degraded with reason InvalidAgentAPIAuthResources when no higher-ranked Degraded cause is present, the agent staying Ready. There is no memory floor, unlike the credential proxy, because the watcher derives no admission count from the limit. Where the validating webhook is enabled, admission refuses the same override at apply, and warns on the GKE Autopilot memory-per-vCPU band and on a cpu or memory limit set without the same key under requests, for the reasons credentialProxy.resources gives.
• initContainers / sidecars — standard init and sidecar containers. A volumeMounts entry naming a reserved volume is refused at admission, the same as extraVolumeMounts. That is the route that needs no user-authored volume at all, and so the one the name reservation exists for; see Reconcile behavior. Under spec.mode: next the render also writes two environment variables onto every sidecars entry — POD_NAMESPACE (downward API) and A2A_CAPABILITY_REQUIRED — with different precedence: the operator’s A2A_CAPABILITY_REQUIRED wins, so a value set for that name in the CR is discarded on every reconcile, while POD_NAMESPACE is supplied only as a default and a CR value for it survives. Both are inputs to the A2A capability check the sidecar executor performs, not container configuration; nothing else in the container is rewritten.
• extraVolumes — custom volumes for the main container. Both halves of the reservation apply: a reserved name, and a source that would carry the A2A bus credential, are each refused at admission; see Reconcile behavior.
• extraVolumeMounts — custom mounts for the main container. A mount has no source, so only the name half applies: an entry naming a reserved volume is refused at admission.
• sidecarVolumes — custom volumes for the sidecar containers. Same reservations as extraVolumes.
• The watchers’ broker token is reserved on all five surfaces. The webhook refuses the volume name watcher-credential-proxy-token on sidecars, initContainers, sidecarVolumes, extraVolumes and extraVolumeMounts. Where the webhook did not run, an extraVolumeMounts entry naming it degrades the CR with ForbiddenVolumeMount, as one naming the broker’s own volumes does, and no Pod is rendered. On the other surfaces the controller leaves out of the Pod template any user volume under that name, or projecting a token for the kubeagents-credential-proxy-watcher audience under another name, together with every volumeMount naming it. The VolumesDropped condition names those volumes. The token belongs to the gateway’s agent-api-auth container alone.
• A hostPath entry on either volume list is refused at admission, and on an install where the webhook did not run (the Helm chart ships it off, and one enabled through the chart fails open at its default failurePolicy: Ignore) the controller leaves it out of the Pod template at render, together with every volumeMount naming it on the agent container, the dashboard container, and the CR’s own sidecars/initContainers. The rest of the template renders as written, and the VolumesDropped condition (status below) names what was left out — as many entries as fit in 4096 bytes, then a count of the rest, and an entry longer than that on its own cut with .... The budget is the controller’s, not the schema’s: a condition message may hold 32768 characters, and the list stops well short of it because kubectl describe is where it is read and the first few entries are what anyone acts on. Volume names and host paths are the author’s, and nothing bounds their length or their number, so an unbounded list would be one spec away from failing the whole status write — Ready, the phase and every other condition with it.
• podAnnotations — annotations applied to the generated pod template.
• scaleToZero — when true, scales the deployment to 0 replicas (idle cost saving).

Default image: derived dynamically from the operator’s container image at runtime via the OPERATOR_IMAGE env var on the controller manager (e.g. ghcr.io/gke-labs/kube-agents/platform-agent:<version>), overridable operator-wide via the PLATFORM_AGENT_IMAGE env var on the controller manager (see Docker images § Private / custom registry), and falling back to ghcr.io/gke-labs/kube-agents/platform-agent:latest if neither is set. Rebuild with make dev-rebuild-agent ARGS="platform" for local iteration.

imagePullSecrets has the same operator-wide form, IMAGE_PULL_SECRETS on the controller manager, taking comma-separated Secret names. It differs from the image overrides in one way: a CR that sets imagePullSecrets replaces the operator’s list rather than merging with it, so an agent that names its own registry identity is stating it completely. See Docker images § Registry authentication.

spec.security

• serviceAccountName — the KSA the credential broker Pod runs as, with its Workload Identity binding. kubeagents-platform-agent by convention. The gateway Pod runs as its own <name>-gateway-agent, which the operator creates; a value naming that or another operator-created ServiceAccount is refused.

• serviceAccountAnnotations — passed through to the KSA. Typically holds iam.gke.io/gcp-service-account for Workload Identity binding.

• workloadIdentityFederation — audience (the pool provider’s full resource name) and serviceAccountEmail (the GSA to impersonate). Set both, or neither: a half-filled block is read as absent. Optional hardening: with it set, the credential broker takes its GCP identity from a projected token file in its own Pod rather than from the metadata server. Without it the broker uses the Workload Identity annotation on its own ServiceAccount; the gateway runs as a separate ServiceAccount with no annotation either way. The chart sets both from platformAgent.security.workloadIdentityFederation; nothing creates the pool, and those commands are in designs/agent-shell-sandboxing.md.

• egressPolicy — None (default) or Allowlist. Allowlist renders a default-deny egress NetworkPolicy on the agent Pod, with the link-local metadata server’s credential API left off the allowlist (the address is permitted on port 53 alone, where it is the Cloud DNS for GKE resolver). It blocks nothing today, and cannot — unless networkPolicy.enabled: false has withheld the gateway policy, in which case this becomes the Pod’s only policy and default-denies for real on an enforcing CNI. Everywhere else: adding a policy is monotone — policies selecting one Pod are unioned and the API has no deny rule — and the same Pod is already selected by <name>-gateway-netpol, which names 169.254.169.254/32 on port 53 only (the resolver; it opens no metadata path) and permits TCP 443 to 0.0.0.0/0 minus the private and link-local ranges. So enabling this widens the Pod’s permitted egress (by the broker on 8765, and by the managed collector namespace on 4317/4318 when the agent is not exporting telemetry) and narrows nothing; what you get is an auditable object, not enforcement. The gateway no longer holds a cloud identity and its policy opens no metadata path; narrowing <name>-gateway-netpol’s internet rule is what turns this field into a control, and the capability cost lands then. It will also do nothing on a cluster whose CNI does not enforce NetworkPolicy, which the operator cannot detect. See Credential isolation § Denying the sandbox the metadata server.

• egressAllowlist — tunes egressPolicy: Allowlist. controlPlaneCIDRs permits the Kubernetes API server on 443 (absent by default, which costs the agent container its API-server connection; a bare address — what the documented gcloud command emits for a public endpoint — is widened to a single-host prefix); extraRules are NetworkPolicy egress rules appended verbatim — except that a rule the operator will not render (an ipBlock reaching a metadata address, including in IPv4-mapped form; a rule with no to peers; a peer naming no ipBlock, podSelector or namespaceSelector; an invalid CIDR, in cidr or except; an except outside its cidr, which the API server would reject in a way that wedges the whole reconcile; or controlPlaneCIDRs wider than /16 IPv4 or /32 IPv6) is refused loudly rather than silently dropped: the agent goes Degraded/EgressAllowlistRefused, the workload stops being reconciled, and the policy keeps being rendered minus the refused destinations.

• scopedServiceAccountPool — the scoped service account pool: the switch that arms it and the project→service-account mapping it serves. Absent (the default) the pool is disarmed and the broker runs on the agent’s own identity, with CREDENTIAL_PROXY_SCOPED_SA_POOL=0 on its container. Leave it off for now: pool members currently hold no IAM grant, so an armed pool turns every mapped-project read into a Forbidden — see the caution on the pool section.

  • enabled — arms the broker, and is the only thing that does: a populated serviceAccounts list with enabled: false renders nothing, and a project declared in spec.scope arms nothing either. With it true the operator renders the mapping into the credential-proxy ConfigMap (scoped-sa-pool.json, version 2, sorted by projectId), mounts it read-only, and sets CREDENTIAL_PROXY_SCOPED_SA_POOL=1; the broker then mints a short-lived token for the account a request’s project maps to and refuses a project with no entry (403, rule gcp.scoped-sa.unmapped-scope) rather than falling back to the ambient credential. enabled: true with an empty list is rejected at kubectl apply, since the broker refuses to start on an empty pool.
  • serviceAccounts — one {projectId, serviceAccountEmail} entry per GCP project the agent may read, each account provisioned by Terraform in the host project (the composition’s scoped_service_accounts output is the source). Keyed on the project rather than the cluster: the project is the IAM unit a scope declaration is written in, and two clusters in one project share an account by design. The list is keyed on projectId so a duplicate project is rejected at kubectl apply rather than crashlooping the broker at startup; both fields are pattern-validated, and there is no entry cap — the Terraform module’s scoped_pool_max_accounts is the bound declared on the pool from the host project’s free service-account quota, and the plan holds the derived set to it before any entry reaches this list.

The Workload Identity target GSA (kubeagents-platform-gsa@<project>.iam.gserviceaccount.com) is created and bound by the kube-agents-iam Terraform module with one of these permission sets:

• read-only (default)
• custom (roles supplied via the installer’s --custom-roles, the composition’s project_roles)

spec.scope

Optional. Which GCP projects, folders, organisations, Shared VPC hosts and Metrics Scopes, beyond the project the agent runs in, the hourly Cluster Agent reconcile enumerates for GKE clusters, and which projects and clusters it leaves unmanaged. The chart renders it from platformAgent.scope, and the Terraform composition sets that value from scope in its tfvars, which the installer generates from SCOPE_PROJECTS, SCOPE_FOLDERS, SCOPE_ORGANIZATIONS, SCOPE_SHARED_VPC_HOSTS, SCOPE_METRICS_SCOPES, SCOPE_MAX_PROJECTS, SCOPE_EXCLUDE_PROJECTS and SCOPE_EXCLUDE_CLUSTERS in install.env, so on those paths the field is declared there and never by editing the PlatformAgent: the installer refuses a full upgrade over a spec.scope edited by hand until install.env records it (it prints the SCOPE_* lines that reproduce it) or the CR is put back, and a retag, or a hand-driven composition apply whose rendered scope is unchanged, leaves the edit in place because Helm sends only the difference between its rendered manifests. An empty scope is a present block with empty lists, rendered whenever the chart is given one (folders, organizations, sharedVpcHosts, metricsScopes and maxProjects only when the value carries the key, so a release record from before the chart knew them leaves the CR’s lists and cap alone on a retag; the composition always passes all five); a chart that is given nothing (the value is null, its default) renders no block, which leaves the field as it finds it while no earlier release rendered the block and removes it once one has, a removal the reconcile reads as no declaration (the management project alone, nothing retired). The composition binds the read roles before it writes the CR, which orders creation and not IAM propagation: a first install’s one-shot inventory sweep may still name a scoped project as denied, and the hourly reconcile creates its profiles once the grant has propagated. Absent, the reconcile lists the management project alone, every cluster there getting a Cluster Agent profile, keeps the last declaration’s exclusions and marks nothing newly retiring (a project an earlier block already marked retiring is still pruned on a clean run); an empty projects list in a present block drops the projects an earlier block declared, over two clean runs. The management project is always in scope and cannot be excluded.

spec:
  scope:
    projects: # explicit project IDs, in addition to the management project
      - payments-prod
      - payments-staging
    folders: # numeric folder IDs, every project beneath them at any depth
      - "123456789012"
    sharedVpcHosts: # Shared VPC host project IDs, every service project attached to each
      - shared-net-host
    metricsScopes: # Metrics Scope scoping-project IDs, every project each scope monitors
      - observability-hub
    maxProjects: 100 # the most projects a run lists, the management project included
    exclude:
      projects: # project IDs or shell-style globs, dropped after resolution
        - "*-sandbox"
      clusters: # one cluster each, by the full triple
        - projectId: payments-staging
          location: us-central1
          clusterName: scratch-cluster

• projects — project IDs whose clusters get profiles. The agent’s service account needs the read roles in each one (roles/container.clusterViewer, roles/container.viewer, roles/compute.viewer, roles/monitoring.viewer, roles/logging.viewer, roles/iam.securityReviewer, the read subset of the read_only_roles the Terraform composition binds in the management project). The composition binds the ones the management project holds, through the kube-agents-iam module’s scope input, in every project the same value names; a CR edited by hand needs the same grants made by hand. A project it cannot list is reported as denied (no role left that grants container.clusters.list; roles/iam.securityReviewer alone keeps a project listable but not manageable, since it carries no container.clusters.get: its clusters list and every profile create fails, which is why the composition refuses a scope whose bindings carry neither roles/container.clusterViewer nor roles/container.viewer; a project reads denied only once every listing role is gone), api-disabled (GKE API off in that project) or unreachable (anything else, including a listing that did not finish within the run’s listing budget) and its existing profiles are kept. Each list is capped at 100 entries, and the run lists at most maxProjects projects in total (100 by default), the management project included; an explicit project past that reads over-cap and is likewise kept but not listed.
• folders and organizations — numeric IDs of GCP folders and organisations. The design recommends folders until the scoped service account pool grants authority: a folder binding reaches every project beneath it, an organisation binding every project in the organisation, including ones nobody meant to manage, and one service account holds the read roles across all of them (Security & IAM). Each is resolved on every run with one Cloud Asset Inventory call (gcloud asset search-all-resources --scope=folders/<id> --asset-types=container.googleapis.com/Cluster), which names every GKE cluster in every project beneath it, at any depth, including projects created since the last run; no per-project listing follows for them. The asset index is what answers, not Resource Manager: a project moved into or out of a folder appears under its new parent once the index has caught up, which after a move can lag by more than an hour, and until then it reads as absent from the container. Because the index and not the declaration dropped it, a project that was reached through a still-declared container (whether or not it was also named in projects, so dropping the explicit entry while the project moves into the folder costs nothing, and the same holds when the folder is declared in the edit that drops the entry: a container the previous run did not know is what marks that edit, and the same day’s grace applies) and is absent from the index is kept, listed under unmanaged with that reason and the time it was first found absent (absentSince on its row), and is back in scope the run the index places it again. It retires sooner when the declaration speaks, by the container leaving the CR or an exclude.projects entry naming it, and otherwise once it has been absent for a day, over the ordinary two clean runs: that is what retires a project that was deleted or moved under a parent the CR does not declare, since a deleted project answers 403 to describe, never NotFound. A project dropped from projects, or whose folder leaves the CR, in the same edit that declares the folder it moved into is held that day too, since the index may not place it yet. A rollback to an operator that predates these fields renders a declaration without them, which the reconcile likewise reads as saying nothing about containers, so their members are kept. The agent’s service account needs roles/cloudasset.viewer and the read roles above on the container, and cloudasset.googleapis.com enabled in the management project: the Terraform composition binds both on each container its scope value names, through the kube-agents-iam module, and enables the API when a container is declared, and the installer checks before the apply that the credentials Terraform applies with can set IAM policy on each container and that no organisation policy forbids the API; a CR edited by hand needs the same grants made by hand, and recording such a container in install.env later hands its bindings to Terraform, which then needs that permission too. A container the run cannot read (denied, api-disabled, unreachable) is frozen: the members the previous snapshot reached through it are carried forward under that outcome, nothing is created under them, and the scope prune stays off for the run. A container whose members would cross the maxProjects cap reads over-cap: the members the run just resolved are carried reading over-cap, with nothing created under them, and because the run holds the full member list the prune is not held back. A project under two declared containers takes the live listing when either resolves, whichever sorted first, unless the cap is already reached. A member whose per-cluster call answers 403 reads denied for the run, unless it is also explicit or the management project, whose own listing decides. The snapshot’s containers array lists each container with its outcome and a project count: the projects the lookup returned for ok and over-cap, the members carried forward from the previous snapshot for a failed lookup, and a member project’s via names it (folders/<id>), or several sources when more than one produced it. A container whose members would take the run past the maxProjects cap reads over-cap with nothing created under it; its members are carried but not listed, and do not count against the containers after it, so a small folder declared beside a large one still lists. What an operator above the cap does: raise maxProjects to the size of the estate; exclude.projects globs subtract members before the count, so a folder that carries sandboxes or archives is trimmed by pattern; or declare the sub-folders that hold the clusters, each counted on its own.
• sharedVpcHosts and metricsScopes — project IDs of Shared VPC host projects, and of the scoping projects of Cloud Monitoring Metrics Scopes. Neither is a Resource Manager container. Each is resolved on every run to the projects it reaches (gcloud compute shared-vpc list-associated-resources <host> for the service projects attached to a host, the host itself not among them, so name it in projects if its own clusters are wanted; gcloud beta monitoring metrics-scopes describe locations/global/metricsScopes/<id> for the projects a scope monitors, the scoping project included), and those projects then take the projects path above: each lists its own clusters, needs the read roles in it, and reads denied, api-disabled or unreachable on its own. Nothing is inherited through a Shared VPC or a Metrics Scope, so the Terraform composition resolves each selector when it plans (its kube-agents-scope-resolver module), with the same reads as the run and made as the identity it applies with, and binds the read roles in every project resolved and in each scoping project, and roles/compute.viewer alone in a host not otherwise in scope, through the kube-agents-iam module; a read that identity cannot make fails the plan before anything is applied, naming the selector, and a monitored project it cannot name is left out by naming its project number in exclude.projects. A project attached or linked after the last apply reads denied until the next one binds it; a CR edited by hand needs the same grants made by hand. The lookups themselves need compute.projects.get in the host project (roles/compute.viewer carries it) and, for a Metrics Scope, to read the scope in the scoping project: roles/monitoring.metricsScopesViewer (resourcemanager.projects.get and resourcemanager.projects.list) is the narrowest role that grants it, and the read roles above carry both between them, with the Monitoring API enabled in the scoping project (a scope whose API is off reads api-disabled). The Monitoring API names monitored projects by project number, so the run names each with gcloud projects describe <number>, which needs resourcemanager.projects.get in that project; a monitored project the account cannot read cannot be named either, and is reported by number under the naming call’s outcome (denied for a 403), or under its ID once an earlier run has named it (every row a scope named by number keeps the number, retiring rows included); one that a Shared VPC host, a folder, an explicit entry or the management project names by ID is listed on that route, and its own listing decides its outcome. A monitored project whose ID the scope cannot carry, a legacy example.com:name ID, is reported by number as denied with the reason in the log rather than as a failed lookup, so it never holds the prune; name the number in exclude.projects to drop it. A monitored project the account cannot name and no earlier run has named holds the scope prune for the run, as a folder it cannot read does, because the bare number could be any project, including one the same edit dropped from projects; granting resourcemanager.projects.get there, or excluding the number, releases it. A project that is not a Shared VPC host resolves to no members. A selector’s projects fill the set after the explicit projects and before the containers, each reading over-cap on its own past the cap. A selector the run cannot read is frozen exactly as a container is: the members the previous snapshot reached through it are carried under that outcome, nothing is created under them, and the scope prune stays off for the run; the containers array carries a row for it under its via name (sharedVpcHosts/<host>, metricsScopes/<id>). There is no index and no lag behind either lookup: a project detached from the host or unlinked from the scope leaves the set on that run and retires over the ordinary two clean runs. A rollback to an operator that predates these fields keeps their members, as for containers.
• maxProjects — the most projects the reconcile lists per run, the management project included, 1 to 5000, 100 by default: explicit projects, selector members and container members fill it in that fixed order, and a project past it reads over-cap (profiles kept, nothing created). The reconcile lists with four workers at every cap and sizes its listing budget from it, 300 seconds per default cap’s worth of projects, and the bootstrap gate’s ceiling on the reconcile follows the budget, so an estate of a few hundred projects declares the number it has rather than reading half of itself over-cap; the snapshot records the cap in force under maxProjects. The Terraform composition refuses a plan whose explicit projects and selector members exceed it, while a selector is declared or the cap is below its default, and renders the same value here from its scope.max_projects (SCOPE_MAX_PROJECTS in install.env); a CR edited by hand takes the value directly. The 100-entry cap on each declared list is unchanged: a hand-written list past a hundred entries is the shape folders exist for.
• exclude.projects — IDs or globs matched against every resolved project ID, and, by an entry that names it exactly (a glob never matches a number), against the number a Metrics Scope named a monitored project by, on whichever route the project is reached, so long as the run can tie the number to the project’s ID: by naming it, which the grant an entry in projects or a declared folder carries allows, or from the number an earlier run named it by, which the snapshot keeps on the project’s row and, once the entry has dropped the project and it has no row, in its numbers map, so a number entry keeps matching a project the run named before and can no longer name, and a naming call that times out does not admit the project again. A number the run cannot tie to an ID matches only the member the Metrics Scope reports under that bare number. An entry that matches the management project, by ID or by the number a Metrics Scope named it by, is ignored and recorded in the snapshot, never applied.
• exclude.clusters — single clusters by projectId, location and clusterName, because a cluster name is unique only within a project and location. This replaces the RECONCILE_EXCLUDE environment variable, which matched bare names across every project and keeps working for one release alongside it. An excluded cluster gets no Cluster Agent, and fleet answers, the onboarding sweep and the fleet-audit collectors that list clusters leave it out. Still covering it: the daily upgrade-readiness watch, whose report enumerates by project; the networking audit’s subnet usage, since its pods still hold their IPs; the GCE audit’s instance listing; and the third-party gke MCP server’s list_clusters.

Rolling the release back past these fields: remove the whole spec.scope block from the CR first and re-add it after, because a cluster_agent_reconcile.py from before folders and organizations, or from before sharedVpcHosts and metricsScopes, reads the same scope.json, ignores those keys, and finds every container or selector member in its previous snapshot but not in its resolved set, which is the ordinary two-run retire; a CR without the block marks nothing newly retiring. On a default install the agent image follows the operator’s, so an operator rollback is an agent rollback too. The operator renders the block as scope.json in the agent’s config ConfigMap, mounted read-only at /etc/kube-agents/scope.json, so editing it moves the config hash and rolls the pod. The file records whether the CR carries a scope block at all: a CR without one declares nothing, so the reconcile lists the management project alone and marks nothing newly retiring (a project an earlier block already marked retiring is still pruned on a clean run), keeping the last declaration’s exclusions and carrying its projects, because a block goes missing on its own when a CR write passes an older operator’s webhook. To drop projects, empty projects and keep the block. Each reconcile run (other than --dry-run) writes what it resolved to fleet_scope.json beside the profiles on the data volume: the declaration the run applied, or on a run that could not read it the last one read (declared), every project with its outcome (ok, denied, api-disabled, unreachable, over-cap), the profiles whose project the scope never produced or whose prune waits for a clean run (unmanaged; a profile whose identity could not be read is kept but appears under the report’s skipped_no_identity, and under profiles once an earlier run has read its identity), any exclusion it declined to honour (ignoredExcludes), every folder, organisation, Shared VPC host and Metrics Scope with its outcome and member count (containers, with resolver reading asset-inventory once a folder or organisation is declared and explicit otherwise; it says nothing about the two selectors), and each profile’s project as read this run or, for a profile whose identity could not be read, as last read (profiles), which is what keeps such a profile attributed while it stays on the volume. A cluster named in exclude.clusters loses its profile on the next run, unconditionally, as RECONCILE_EXCLUDE always did. A project removed from the scope, or newly matched by an exclude.projects entry, has its profiles pruned over two clean runs: a run is clean when no project came back unreachable, every folder, organisation, Shared VPC host and Metrics Scope resolved ok or over-cap, the management project resolved, listed its own clusters and is the one the previous snapshot named, and the scope file was readable. The first clean run that finds a previously in-scope project absent records it as retiring in fleet_scope.json and keeps its profiles (listed under unmanaged); the next clean run deletes them, so a scope edit reverted before that run costs nothing. A management project that changes identity (RECONCILE_PROJECT re-pointed, or the metadata server naming another project) marks the old project retiring on the run the change is seen, once the new project has listed its own clusters, and the next clean run prunes, unless projects names the old project and no exclude.projects entry matches it; an answer from the gcloud config fallback that disagrees with the previous run is treated as unresolved instead: nothing is created or retired under the management project until an authoritative source, or a fallback answer that matches the previous run, names it. A run that cannot read the declaration is not a clean run and prunes nothing; one that reads a CR without a scope block keeps the exclusions of the last declaration it read and marks nothing newly retiring, so a rollback to an operator without the field neither re-onboards an excluded cluster nor, on the roll forward, prunes a project the rollback’s own CR write dropped. A profile the scope never produced is kept and listed, which is also what a project dropped while fleet_scope.json was lost between the two runs becomes: declare and drop it again to retire it. The onboarding sweep names any folder, organisation, Shared VPC host or Metrics Scope the reconcile could not resolve and, on an install whose scope resolved more than one project, any project it could not list, so a partial roster reads as partial; an install with one project and no container renders the sweep prompt it rendered before scopes existed.

spec.telemetry

• otlpEndpoint — the OTLP/HTTP collector base URL (no /v1/traces suffix; the exporters append their own per-signal path). Up to 2048 characters, http:// or https://.

Optional, and omitting it is the point: with the field absent the operator discovers an in-cluster collector, falls back to GKE Managed OpenTelemetry when it cannot establish what the cluster has, and disables export altogether when discovery finds no collector (otlpEndpointSource: None). Setting it pins the endpoint and suppresses discovery. The full precedence ladder, the discovery order, and the Helm value that drives LiteLLM and the NetworkPolicy alongside this field are on Deploy → Telemetry.

spec.networkPolicy

Configures the operator-generated egress NetworkPolicy.

• enabled (bool, optional) — toggle operator-managed NetworkPolicy generation. Default true (unset means on). Setting false stops generation and deletes the policies the operator manages for this agent: <name>-gateway-netpol, the <name>-fqdn-netpol FQDNNetworkPolicy, and the shared litellm-policy (if LiteLLM is present). Deletions check ownership / managed labels first, so a policy that the operator did not create survives. To opt only LiteLLM out of operator NetworkPolicy management while keeping the agent pod’s policies managed, set the annotation kubeagents.x-k8s.io/enable-litellm-network-policy: "false" on the PlatformAgent. When opted out, the operator deletes any managed copy of litellm-policy, leaving LiteLLM unselected (fail-open) unless a replacement policy is provided and managed out-of-band.

  Upgrade note: when upgrading from a chart version that shipped the static litellm-policy, Helm prunes the static policy on the first upgrade (unless the live object already carries helm.sh/resource-policy: keep, in which case Helm retains it and the operator adopts it). The operator recreates it once the new operator pod rolls out, acquires leader election, and reconciles. LiteLLM is unselected (fail-open) during this operator rollout window. To eliminate this window on an existing cluster, annotate the live policy before upgrading (kubectl annotate netpol litellm-policy helm.sh/resource-policy=keep -n <namespace>); Helm will retain the policy across the upgrade and the operator will adopt it via Server-Side Apply. Alternatively, pre-roll the new operator image before running helm upgrade to narrow the window to controller watch latency (~1s), or manage the policy out-of-band during the transition via litellm.networkPolicy=false. The same window opens on a fresh default install, with no live object to annotate; the chart README’s upgrade notes say how to cover it. Going the other way, from operator-managed back to the static copy, needs a handoff first — see Handing litellm-policy back to Helm in the chart README.

• dnsClusterIPs ([]string, optional, max 8 items) — pins the cluster DNS Service ClusterIPs in rule 1 of both the agent gateway policy (<name>-gateway-netpol) and the LiteLLM gateway policy (litellm-policy), suppressing dynamic discovery from kube-system/kube-dns. Each entry is a bare IPv4 or IPv6 address with no prefix. Admission bounds the IPv4 octets and rejects the leading-zero form (010.96.0.10) that Go’s net.ParseIP refuses, so the usual typos are apply-time errors; a malformed IPv6 literal can still get past it, in which case the operator drops the entry and falls back to discovery, and says so in its log.

• metadataDaemon (object, optional) — pins the node-local cloud metadata daemon IP in the metadata rule of litellm-policy (rule 4); the gateway policy has no metadata rule. Its one field, endpoint, is required within it, so metadataDaemon: {} is rejected; an explicit endpoint: "" suppresses the post-NAT metadata rule entirely, for datapaths without a post-NAT daemon. Leave it unspecified to let the operator discover the container port from the kube-system/gke-metadata-server DaemonSet on port metadata-server (promoting metadataDaemonIPSource to Discovered). If undiscoverable, it falls back to 169.254.169.252 on port 988. Overriding the endpoint explicitly opts out of port discovery and uses port 988.

• additionalEgress ([]EgressRule, optional, max 32 items) — appends custom CIDR and port egress rules to the generated agent gateway policy (<name>-gateway-netpol). (Does not apply to litellm-policy.) A peer CIDR broader than /12 (IPv4) or /48 (IPv6) is rejected at admission, so that a caller-supplied range cannot be widened into an unrestricted egress bypass. One shape gets past that check and is dropped by the operator instead: an IPv4-mapped IPv6 block such as ::ffff:0:0/96 is a 128-bit prefix by every textual measure, so it clears the IPv6 floor, and the operator re-measures it against the IPv4 floor once it has collapsed it to the IPv4 block it means. An except block that is not a strict subset of its peer’s CIDR is dropped too — the API server rejects the whole policy for one that is not — and a rule left with no usable peer is dropped whole — a rule carrying ports and no peer would otherwise permit egress to every destination. All three are logged, so the operator’s log is where a rule that did not take effect explains itself.

  A rule’s ports list is optional, and omitting it is not one of those drops: a rule with peers and no ports permits every port to those peers, which is what a NetworkPolicy egress rule with an empty port list means. Nothing is logged, because nothing was dropped. List the ports unless that is what you want.

  A peer’s except entries may be written bare (10.0.1.5, meaning a /32) as well as with a prefix, the same as cidr. Unlike cidr there is no prefix floor on them, because an except has to be a strict subset of its peer to be kept at all.

Annotations (kubeagents.x-k8s.io/dns-cluster-ip and kubeagents.x-k8s.io/metadata-daemon-ip) remain available as escape hatches and take precedence over spec.networkPolicy.

spec.integration

Enables external integrations. Only the enabled ones need to be present.

• googleChat — enabled (default false), projectId, topicName, subscriptionName, allowedUsers, homeChannel, and mode (default or debug, default default). When enabled, projectId, topicName, and subscriptionName are required (enforced by a CEL validation rule). Populated by the installer when Google Chat is enabled.
• slack — enabled (default false), botTokenSecretRef and appTokenSecretRef (Secret refs, required when enabled), allowedUsers, homeChannel, and homeChannelName. Populated by the installer when Slack is enabled. Under mode: next this field arms the A2A gateway’s Slack backend instead of the legacy consumer (the credential broker’s Socket Mode relay and the Hermes slack platform are not rendered), unless googleChat is enabled too, which holds the gateway and leaves Slack on the legacy path. Under next the gateway also carries homeChannel and posts the agent’s proactive messages into that channel and nowhere else (alerts, scheduled reports, and the fleet audit’s Block Kit report card, without its buttons), as new threads there or as replies on them; it must be a channel ID (C..., or G... for a private channel), and with it unset or malformed the agent posts nothing to Slack unprompted (unset, the CR carries the HomeChannelUnset condition). A kanban card’s report back to the Slack conversation that asked for it is not a proactive post: the gateway serves it with homeChannel unset too, though not with a malformed one, which leaves the gateway’s notify route off altogether. allowedUsers applies on the gateway as it does on the legacy path, as Google Chat’s does: an absent list (or a single empty string) admits every member, any other list admits only the member IDs it names, matched exactly. The gateway’s user_id principal map, the a2a-slack-principal-map Secret, is optional, mounted when present and read when the gateway starts. It does not admit anyone: a member the list admits and the map names is attributed by the mapped identity, one the map does not name by their Slack member id, and a member the list does not admit is refused whatever the map says. A member of another workspace posting in a Slack Connect channel shared with the install’s is not served, under any list. That Secret is the whole map: the principal-map ConfigMap is not mounted on a Slack gateway, so adding a Slack user to it admits no one. A multi-workspace install, whose bot-token Secret holds the legacy broker’s comma-separated list of tokens, must stay on mode: today: the gateway takes one workspace’s token. The operator does not check, because it arms on the CR alone, so an install already on next with a list token loses Slack when it upgrades to an operator that arms the gateway, with nothing in the CR’s .status to say why (the gateway pod’s log carries the cause), and gets it back only by switching the Secret to a single workspace’s bot token or the install back to today. The installer checks before it applies: install.sh and a full upgrade.sh refuse a next apply that hands Slack to the gateway with a token list, an email in allowedUsers, or a homeChannel that is not a channel ID (when an upgrade is refused). upgrade.sh’s operator and harness re-tags, a full upgrade onto a release that predates the check, and changes made outside the installer skip it. On version skew (a spec.mode this operator does not recognise) a next install’s Slack is consumed twice until the operator is upgraded: the frozen gateway keeps its own connection and the legacy consumer renders again. Slack delivers each event to one of them, and both admit by the same list, so a sender of the install’s own workspace whom the list admits is answered whichever receives the event; a member of another workspace is answered only if the legacy consumer receives it. Google Chat goes to the legacy consumer alone in that case.
• teams — enabled (default false), appIdSecretRef and appPasswordSecretRef (Secret refs, required when enabled), tenantId, allowedUsers, allowAllUsers, homeChannel, homeChannelName, and adaptiveCards (default true). Set when the Microsoft Teams integration is enabled.
• forges — the forges the agent talks to, up to 16. Each entry has a name (required; a DNS label that repositories refer to it by, unique in the list), provider (github or gitlab; omitted, it is github, and it is required for a forge at a host other than github.com), host (optional; defaults to the provider’s own — github.com, gitlab.com — and may name a self-managed instance: a GitLab instance on a gitlab forge, or a GitHub Enterprise Server instance on a github forge; never a host another provider serves, and never a host at or under a domain where GitHub runs its own services — github.com, githubusercontent.com, githubassets.com, githubapp.com, github.io, github.dev and ghe.com), namespace (optional; the organisation, user, or group a bare repository name on this forge belongs to, within the provider’s own limit — 39 characters on GitHub, a nested group path on GitLab), and credentialsRef (the name of a Secret in the agent’s namespace, for a provider whose credential is a token an administrator holds). GitHub’s credential is the GitHub App, so on a github forge credentialsRef is ignored and admission returns a warning saying so. On a gitlab forge it is required: the Secret holds the access token under the key token (a group or project access token, or a dedicated account’s personal access token on a tier without them), the operator mounts that key into the credential broker’s pod only, and the broker reads it on every call, so rotating the token is updating the Secret. The same Secret may also hold a read-only token under the key read-token (Reporter role, read_repository scope only). The operator mounts it beside token when the key is present, and the broker presents it only to clone the forge’s context_repos entries, never on a write; without the key a context repository clones with no credential. One GitHub host per install: github.com, or one GitHub Enterprise Server. A second github forge at another host is refused, and so is one host declared for two providers; two github forges at one host still name two organisations. The first github forge declared decides the host. While its host can be read, it keeps the host even when another of its fields is refused: the github_host key, the broker’s configuration and the egress stay on that host, and a later github forge at another host stays refused, and a change of the declaration that moves the host writes a GitHubHostChanged Warning Event, because the repositories registered on the old host are then no longer served. On a GitHub Enterprise Server the operator writes the host into the <agent>-gitops-state ConfigMap as the key github_host, which the agent reads (a github.com install has no such key); the credential broker calls the instance’s REST API at https://<host>/api/v3, and keeps the GitHub App token it mints for it in memory. The chart’s token minter mints on the instance too: it reads the GitHub forge and sets the instance’s API URL and CA on the minter. install.sh --gitops-host writes a GitHub Enterprise Server forge, through the Terraform composition’s gitops_host and github_ca_secret_name; see GitHub Enterprise Server in INSTALL.md. The chart owns this resource on such an install, so change the forge there, not with a hand edit of this resource. A forge at a self-managed host — a gitlab forge, or a github forge at a GitHub Enterprise Server host — may also set caBundleRef (name, and key, which defaults to ca.crt): a Secret in the agent’s namespace that holds the PEM CA that signed the instance’s TLS certificate. A CA certificate is public, but it lives in a Secret for its integrity: changing it chooses which servers the broker sends the token to, so it needs the same rights as changing the token Secret. The operator mounts that key into the credential broker’s pod, as it mounts the token, and the broker trusts the CA for this forge’s host and for no other, beside the system CAs, in its API client and in git; git follows no redirect from that host. On a GitHub Enterprise Server forge, the chart also mounts the CA into the token minter, which trusts it for all of its outbound calls, beside its image’s bundle (token minter guide). For this host only, the broker turns off Python’s strict X.509 checks, which refuse a CA certificate without Key Usage; the chain and the hostname are still verified. The mount is optional: until the Secret exists, calls to the forge answer FORGE_TLS_UNTRUSTED, naming the host and “the Secret or its key ”, and an update needs no restart. The field is refused, in the CRD and in admission, on a forge whose host is empty, gitlab.com or github.com in any spelling, because a private CA must never vouch for a host the public CAs sign. Two github forges at one host may not name two different CA bundles. A gitlab forge’s namespace and the groups of the repositories declared on it are also the groups the broker serves with that token. One gitlab forge per host: a second at a host already declared is refused. A gitlab forge with neither a namespace nor a repository is not given to the broker; admission returns a warning, and the operator writes the same text as a Warning Event (reason IntegrationWarning) on the PlatformAgent, since the chart ships the webhook off. A GitLab group may contain a dot, but a namespace or repository path whose first segment is a forge host (gitlab.com/acme, github.com/acme/infra, or the forge’s own self-managed host) is refused: name the group alone. A repository whose resolved URL is longer than 256 characters is refused too, because the credential broker reads no longer a reference. A missing Secret or token key does not stop the broker; calls to that forge answer FORGE_CREDENTIAL_UNAVAILABLE naming the host until the Secret exists, and the status does not report it. On an install that declares a gitlab forge beside GitHub, a bare owner/name still names a GitHub repository that the install registered by URL. The sandbox client sends the bare name as the URL of the registration, so crons, skill invocations and --repo arguments do not change. Use https://<host>/<path> or <host>/<path> for a GitLab repository. Also use it for a GitHub repository that the install did not register, that an administrator registered without a URL, or that has the same path as a repository on another forge. The broker serves two forges, so it refuses a bare owner/name in these cases. The declared forges also decide the forge hosts in the gateway’s FQDN egress policy (the enable-fqdn-network-policy annotation); GitHub’s hosts are always included.
• repositories — the repositories the operator registers, up to 64. Each entry has a forge (required; the name of an entry in forges), repository (required; up to 2048 characters), an optional namespace of its own that overrides the forge’s, and a role (required): gitops for the repository the agent publishes its own changes to, at most one; managed for another repository it may change; context for one it only reads. repository accepts HTTPS/HTTP (https://, http://), SCP-style SSH (git@...), SSH/Git protocols (ssh://, git://), owner/repo shorthand (e.g. gke-labs/kube-agents), and a bare name qualified by the namespace. The CRD refuses a repository naming an undeclared forge and a second gitops repository. The webhook also rejects whitespace, control characters, invalid syntax, the same repository declared twice (in any letter case), a GitHub repository name over 100 characters, a name that still ends in .git after one is stripped, and a host the forge’s provider does not serve — where the webhook is enabled; the chart ships it off, and one enabled through the chart fails open at its default failurePolicy: Ignore — so a repository on another forge is refused rather than rewritten onto the declared one. Every refusal names the entry at fault, e.g. spec.integration.repositories[1].repository. Reconciliation re-runs the same validation, since a chart install runs with ENABLE_WEBHOOKS=false: an invalid entry is left out of the state ConfigMap, its neighbours are still registered, and the resource surfaces Ready: False and Degraded: True (Reason: InvalidGitRepoURL). Where the refused entry is one the GitHub organisation is read from — a GitHub forge, or a gitops or managed repository unless every GitHub forge declares the same namespace, or unless the entry names on its own a gitops or managed repository that was accepted (a second declaration of it) — the operator also leaves the token minter policies as they were, and the agent’s GITHUB_ORG as the running pod has it (unset on a first install), rather than scoping either to another forge’s or another repository’s organisation. So a CR upgraded from a release that accepted a spelling this one refuses keeps its organisation, and its pod is not rolled for it. The same holds for the ConfigMap: while any github entry in managed_repos or context_repos does not parse — a spelling an earlier release accepted, added by hand — the operator adds token minter policies but prunes none, since the entry may be the repository a policy it tracks is for, and reports the hold as Degraded/MinterPruningHeld naming each entry by its list and position, such as managed_repos[1], never by its value, which can carry a credential; correcting or removing the entry lets the next reconcile prune. An entry of another type is not the minter’s and holds nothing, and on a GitHub Enterprise Server install neither is a github entry on github.com or with no host: the minter mints for the install’s GitHub host, and the agent skips such an entry, so the minter skips it too and it holds nothing. The operator appends each valid gitops and managed repository to managed_repos, and each context repository to context_repos, in the <agent>-gitops-state ConfigMap if absent, as {"type": "<provider>", "url": "https://<host>/<path>"} — https://github.com/<owner>/<name> on GitHub, https://<host>/<owner>/<name> on a GitHub Enterprise Server, https://gitlab.com/<group>/.../<project> or the self-managed host on GitLab. The entries carry no role, so the gitops repository is the first managed_repos entry, which is the one the agent’s token refresh mints for when no repository is named (the skills’ resolvers refuse to guess among several entries, so with a managed repository declared beside it a caller names its target with --repo); a gitops repository the list does not yet hold is inserted at the front, and an entry already present is never moved. While a declared gitops repository is refused, no managed repository is appended either, since it would take the first place. An existing entry counts as present when it names the same repository on github.com in another spelling — a .git suffix, a remote, a run of / after the host — but not in another case, since the agent’s --repo checks compare the spelling (so the declared spelling is added beside it), nor on www.github.com or ssh.github.com, with or without a scheme, nor under a type other than exactly github, nor in a spelling with two @, nor one over 256 characters, since the agent reads none of those. Reconciliation only adds, and keeps every existing entry as written, so removing a repository means removing it from the CR and editing the ConfigMap. managed_repos is read-write. context_repos names repositories the agent reads for declared intent — a Terraform repository an audit consults before it reports a posture as a finding — and never writes to; an administrator may also add entries to it by hand. An entry may add "ref": "<branch>" to read a branch other than the remote’s default; a ref is kept when it is made of letters, digits, ., _, /, - and, after the first character, @, and passes git’s branch-name rules (no .., @{, // or .lock component, no leading -); any other, an empty string, a JSON null or a value that is not a string (a number, a boolean) included, is refused with a warning naming it, and the declared-intent search skips that repository rather than reading its default branch in the pin’s place, so the ledger names it as not searched until the entry is corrected (omit the key to read the default branch); a ref on an entry naming the GitOps repository itself is ignored, because the audit reads that repository at the branch it publishes against. The operator never adds a ref, and keeps one an administrator added to a seeded entry. The agent’s skills read every managed_repos entry whose type is a forge the agent image serves (github, or gitlab named host/path), and only github entries with a GitHub URL in context_repos; any other entry is skipped with a warning naming the key. On a GitHub Enterprise Server install, a github entry on the instance’s host is read too, in both lists, and named <host>/<owner>/<name>; a github entry on github.com, on no host, or on any other host is skipped with a warning, so a repository registered before a move from github.com is not served on the instance under its old name. The credential broker’s repository gate reads every typed entry, keyed by its type, host and path, so an entry counts only for the forge whose provider it names. Each same-organization context_repos entry gets a read-only (contents: read) token minter policy, so a private context repository is readable through the credential broker’s content-mode clone, provided the GitHub App is installed on it — see Read-only tokens for context repositories. On a gitlab forge, a private context repository is readable the same way when the forge’s Secret holds read-token — see Private context repositories. Nothing that writes consults the key: a context repository stays refused by the broker’s commit and push.
• repositories[].baseBranch — on a gitops or managed entry, the branch every pull request (a merge request on GitLab) onto that repository must target; absent means the repository’s own default branch. A context entry may not set it: the CRD refuses one, because a context repository is never written and its branch pin is the ref in the <agent>-gitops-state ConfigMap. The deprecated github alias has no place for a base, so pinning one takes forges/repositories. For each accepted gitops or managed repository that sets one, the operator adds an entry to CREDENTIAL_PROXY_PINNED_BASES in the credential broker’s environment: a JSON array of {"repository": "https://<host>/<path>", "branch": "<baseBranch>"}, with the forge’s canonical host (github.com for every GitHub spelling, gitlab.com for www.gitlab.com) and the path that forge reads, nested groups included on GitLab. With no base set the variable is not rendered. The broker then refuses a pull request onto that repository that targets any other branch, a clone of it that names no branch checks the base out, and every pinned branch is refused a direct push on every repository. A repository is matched by host and path, so the same path on another host, or on another forge, is not pinned. The agent container and the shell sandbox get no copy, and spec.deployment.env can never set the variable. A CREDENTIAL_PROXY_BASE_BRANCH or GITOPS_BASE_BRANCH there still passes through: the broker refuses direct pushes to the branch it names but does not hold pull requests to it. The CRD schema holds the value to the branch names the broker accepts: up to 200 characters of letters, digits, ., _, / and -, starting with a letter or digit, not HEAD, with no .., /., //, @{ or .lock/, and no trailing /, . or .lock; a refs/heads/ prefix is allowed, and what follows it must start with a letter or digit, not be HEAD, and not start with refs/heads/ or heads/; a value may not start with heads/. So each base has one spelling, the one the broker targets and protects. Changing a base restarts the broker’s pod and no other. The Helm chart key is platformAgent.integration.repositories[].baseBranch; because helm upgrade does not update CRDs, the chart refuses to render it against an installed CRD that lacks the field.
• github — deprecated alias for one forge with provider: github: org is its namespace and gitRepo, when set, is one gitops repository on it; the two resolve identically. Set forges/repositories or github, not both; a resource setting both is refused by a CEL rule on the CRD and by the webhook. This is still the spelling install.sh and the full-install Terraform composition write for a github.com install (a GitLab or GitHub Enterprise Server install writes forges/repositories; see GitLab as the GitOps forge), and the Helm chart renders one GitHub forge with no credentialsRef, no caBundleRef, no host other than a spelling of github.com, and at most one repository, the gitops one, with no namespace of its own, this way whichever values key set it, because helm upgrade does not update CRDs and an older CRD would prune forges. It does so only where the forge declares a namespace or the repository, and the repository is one the operator would accept for GitHub: name, owner/name, or an http(s)://, ssh:// or git:// URL, a schemeless host or an scp remote on github.com, www.github.com or ssh.github.com, with no port, naming owner/name. A repository the operator would refuse renders as the lists, so the refusal names the entry that was written. A repository that sets baseBranch renders as the lists too, since the alias has no place for it.

Upgrade note: non-GitHub repository rejection

Upgrading to an operator that accepts provider: gitlab needs the CRD from the same release applied first: helm upgrade does not update CRDs, an older CRD’s enum refuses gitlab, and the chart fails the render with that message when it can read the installed CRD. Earlier operator versions passed arbitrary https:// and http:// URLs through to CleanRepoURLWithOrg without host validation. A repository now has to sit on a host its forge’s provider serves — for provider: github, github.com, www.github.com or ssh.github.com, or the forge’s declared GitHub Enterprise Server host. A forge at a host other than github.com must name its provider: the CRD refuses one without it, and the field has no default any more. A CRD of an earlier release stored provider: github on a forge written without one, so before you upgrade, check that every forge at a host other than github.com names the provider you mean: a forge written for a GitLab instance that way now reads as a GitHub Enterprise Server forge. Existing PlatformAgent resources pointing at another forge reconcile into phase: Degraded with condition Reason: InvalidGitRepoURL, and subsequent updates to those resources are rejected at admission by the validating webhook, where it is enabled, until the spec.integration.repositories entry — or the deprecated spec.integration.github.gitRepo — is corrected or removed. A repository on GitLab is admitted by declaring a gitlab forge with a credentialsRef and naming it in the entry’s forge; see the forges field above, and GitLab as the GitOps forge for install.sh’s way of doing it.

Upgrade note: repository spellings no longer rescued

Earlier operator versions also rewrote four malformed GitHub spellings into a working repository. Each is now refused, with the same Degraded condition and admission behaviour as above:

• A single path segment after a host, with the owner taken from org: github.gitRepo: github.com/infra (or https://github.com/infra, git@github.com:infra) and github.org: gke-labs used to mean gke-labs/infra. A URL with one segment names an account, not a repository, so only a bare name is qualified by the namespace. Write infra or gke-labs/infra.
• A colon in place of the first slash of a URL with a scheme: https://github.com:owner/repo, ssh://git@github.com:owner/repo. After ://, what follows the colon can only be a port, and git itself reads github.com:owner as the host. Write https://github.com/owner/repo, ssh://git@github.com/owner/repo, or git@github.com:owner/repo.
• An owner containing _ or ., which no GitHub account can have.
• A URL with any bracket in its userinfo, paired or not, or an unpaired one elsewhere in its authority: https://[TOKEN]@github.com/owner/repo, https://[x@github.com/owner/repo. Earlier versions read the host after the last @; the agent’s URL parser refuses these (with a bracket in the userinfo it accepts only an address literal as the host), so it reads no repository from the value either. A bracketed IPv6 host after the @ is unaffected.
• A schemeless remote with a : in its userinfo, or an empty one: user:token@github.com:owner/repo, a:b@github.com/owner/repo, :x@github.com/owner/repo, @github.com:owner/repo. Earlier versions cut everything up to the first @; git reads what comes before the first : as the host, so the first names the host user, and the agent reads none of them as a GitHub repository. Put a credential in the credential helper, not the value, and write git@github.com:owner/repo or https://github.com/owner/repo.

See k8s-operator/api/v1alpha1/platformagent_types.go for the exact struct definitions.

status

The operator writes observed state to the status subresource:

Field	Type	Purpose
phase	string	Overall state (Pending, Provisioning, Ready, Degraded, Failed).
observedGeneration	int64	The metadata.generation the status was last computed from. Behind metadata.generation from a spec edit until the reconcile that follows it.
address	string	Fully qualified domain name (FQDN) of the agent service.
lastReconcileTime	time	Timestamp of the last status write. A reconcile that changes nothing in the status leaves it where it was.
conditions	list	Standard metav1.Condition observations, keyed by type.
deploymentStatus.name	string	Name of the underlying Deployment.
deploymentStatus.readyReplicas	int32	Number of fully ready replicas.
serviceStatus.endpoint	string	Primary URL/IP (with protocol and port) to reach the agent.
storageStatus.bound	bool	Whether the primary PVC has been provisioned.
telemetry.otlpEndpoint	string	The OTLP collector the agent was wired to.
telemetry.otlpEndpointSource	string	Which rung answered: DeploymentEnv, Spec, OperatorEnv, Discovered, None, or Default.
networkPolicy.generated	bool	Whether the operator-managed NetworkPolicy is active. false when disabled, or not yet reconciled.
networkPolicy.dnsClusterIPs	[]string	The DNS ClusterIPs written into rule 1.
networkPolicy.dnsClusterIPsSource	string	Which rung answered: Annotation, Spec, OperatorEnv, Discovered, or Default.
networkPolicy.metadataDaemonIP	string	The post-NAT daemon IP in litellm-policy’s metadata rule, empty when suppressed.
networkPolicy.metadataDaemonPort	int32	The post-NAT daemon port in litellm-policy’s metadata rule, resolved from live DaemonSet or default (988).
networkPolicy.metadataDaemonIPSource	string	Which rung answered: Annotation, Spec, OperatorEnv, Discovered, Default, or Suppressed.
usage.activeInterfaces	[]string	The interfaces the spec enables, sorted; see below.
usage.sessionsTotal	int64	Sessions the agent opened, cumulative: the triage sessions the event watcher opened for events the daemon delivered (its k8s_event_watcher_triage_sessions_total; a session whose inject was filtered, suppressed or failed is not counted) plus the chat conversations a gateway opened a session for (the chat_metrics listener’s kubeagents_chat_sessions_total; on a next install the A2A gateway’s kubeagents_a2a_gateway_sessions_total too). A chat conversation that resumes after the session store’s retention window (seven days by default) counts again. Read every five minutes, monotonic like toolExecutionsTotal.
usage.eventsIngestedTotal	int64	Events the event watcher accepted for triage, cumulative; read from the watcher’s k8s_event_watcher_events_injected_total every five minutes. Observed-only events are not counted.
usage.toolExecutionsTotal	int64	Commands the credential broker ran, successful or not, plus requests it rejected or failed on before running (its success and error outcomes), cumulative; policy refusals, busy and abandoned are not counted. Read from kubeagents_tool_invocations_total every five minutes.
usage.remediationsProposedTotal	int64	Proposals the credential broker’s version-control route opened, cumulative: proposal-create requests the forge accepted, read from the broker’s kubeagents_vcs_requests_total under that verb and status="success" every five minutes, monotonic like toolExecutionsTotal. A refused or failed proposal is not counted.
usage.remediationsAppliedTotal	int64	Declared; nothing writes it yet (approvals are log records, not a metric).
usage.remediationsMergedTotal	int64	Proposals this install opened that the forge reported merged, cumulative. The broker counts each once per broker process when one of its own reads shows it merged, only for proposals whose author is its own login and whose merge is not earlier than the process started; read from kubeagents_vcs_proposals_merged_total every five minutes, monotonic like toolExecutionsTotal. A merge no read shows is not counted, and a credential whose login the broker cannot name counts nothing (the broker logs that once), so this under-counts and stays absent on such an install.
usage.chatMessagesInboundTotal	int64	Chat messages that reached the gateway, from every platform it serves, cumulative, counted at the dispatch hook, before the sender allowlist, so an unlisted user’s message counts: read from the chat_metrics plugin’s kubeagents_chat_messages_inbound_total on the platform-agent container’s port 9097 every five minutes, from the pod that runs Hermes (above one replica, the leader alone), summed across the pods that have led, and on a next install from the A2A gateway’s kubeagents_a2a_gateway_messages_inbound_total on its port 9096 as well, since Chat ingress is that gateway’s there; monotonic like toolExecutionsTotal.
usage.clustersRegistered	int64	Clusters the event watcher built a client for: the number of k8s_event_watcher_cluster_up series it exports, read every five minutes, the largest reading across gateway replicas. The watcher discovers its fleet once per process, so this is the fleet as of its last start; a cluster that joins or leaves is counted after the gateway pod restarts. A gauge: it falls after such a restart. Absent when the operator has no current reading: before the first poll, while the watcher is off, and after two polls in a row in which no gateway replica could be read, whatever the cause; a 0 is a reading.
usage.clustersMonitored	int64	How many of those are delivering events: the k8s_event_watcher_cluster_up series at 1. The difference from clustersRegistered is the clusters silently unwatched. A 0 is a reading, and the normal one while the watcher starts. Absent on the same terms as clustersRegistered: one poll in which no gateway replica can be read leaves both as they were, and a second in a row clears both. The CR’s events and the gateway pod’s state say whether the watcher is down, not running, or unreachable.
usage.lastActiveTime	time	The last poll in which toolExecutionsTotal, eventsIngestedTotal, sessionsTotal, remediationsProposedTotal, remediationsMergedTotal or chatMessagesInboundTotal moved; a scheduled job that runs one does, so it is not a record of human use alone.

usage.toolExecutionsTotal, usage.eventsIngestedTotal, usage.sessionsTotal, usage.remediationsProposedTotal, usage.remediationsMergedTotal, usage.chatMessagesInboundTotal, usage.lastActiveTime and the two cluster gauges are produced by the operator, on the leader and off the reconcile path: every five minutes it reads the credential broker’s, the event watcher’s, the chat plugin’s and, on a next install, the A2A gateway’s metrics listeners over the pod network, folds the counter series into totals it keeps with their per-pod baseline in the <name>-usage-counters ConfigMap, so the counters stay monotonic across pod, process and operator restarts, and patches the status only when it is behind. The gauges are the latest poll’s reading of the watcher’s body, written whenever they change in either direction, and they do not move lastActiveTime. The counters under-count rather than over-count: a listener that cannot be read keeps its baseline and its backlog is added when it is read again, but a backlog past the per-poll ceiling, an honest burst past it, or what a process counted after its last read and before it restarted, is lost once. A lastActiveTime that stops advancing while commands plainly run is the symptom of a listener the operator cannot reach; the CR’s events name the pod. It advances on any brokered command, including those a scheduled job runs – the Controller Stall Watch cron issues some every 30 minutes by default – so a recent lastActiveTime is not evidence that a person used the agent. The counters the table above marks as unwritten stay absent until a series exists for each (what lands them).

usage.activeInterfaces is dashboard unless harness.hermes.dashboardEnabled is false, plus googlechat, slack and teams for each integration entry with enabled: true. It is resolved from the spec on every reconcile and written by the Ready status update when it changes; a pass that parks the CR Degraded before that update leaves the previous value, so read it alongside the Ready condition. On an install whose served CRD predates the field, the API server drops it from every write; the operator notices from the write’s echo, stops treating the missing field as a change for five minutes at a time, and lands it once this release’s CRD is applied: within five minutes on a quiet install, at once when the Ready status update next writes for any other reason. The other status writers carry the field through as they read it, so a pass that ends Degraded lands nothing new. usage.remediationsAppliedTotal is declared in the schema and absent from every status until a series exists for it. The counters the operator does write, and usage.lastActiveTime, land through the poller above, which consults the same five-minute record: under a served CRD that predates status.usage they accumulate in the <name>-usage-counters ConfigMap and land with the first patch after this release’s CRD is applied. Under a served CRD at a previous schema, which has status.usage without the fields added later (the two cluster gauges, remediationsMergedTotal and chatMessagesInboundTotal), the first-schema counters keep landing and the newer fields are probed again every five minutes until this release’s CRD is applied.

These condition types appear in conditions; only Ready is always present:

Type	Written	Meaning
Ready	Always	Tracks phase; its reason and message carry whatever the reconcile is waiting on — including the reasons that park phase: Degraded (RuntimeClassNotFound, ShellSandboxCannotBeDisabled, ForbiddenVolumeMount, EgressAllowlistRefused, ModeNotRecognized, ShellSandboxKeysMissing). The last two render today’s stack in full and report at the end rather than returning early — ModeNotRecognized because it is a version skew rather than a bad spec, ShellSandboxKeysMissing because the objects are wanted in place so the Pod starts by itself once the Secret appears. Under mode: next a failed A2A provisioning Job parks the phase the same way with A2AProvisionFailed, naming the Job to inspect. That signal is prompt where the refusal is deterministic: the script exits 2 for a refusal a re-run cannot clear, the Job’s podFailurePolicy matches exit 2 with FailJob, and the Job fails from its first pod, so the condition is written in seconds. Any other failure — NATS unreachable, an unexpected command failure — matches no rule and still spends the backoffLimit: 20 retry budget, which is the slow path it used to take. On a fresh install the earlier symptom is an install that looks healthy — NATS 1/1, clients authenticating — while every stream read returns stream not found (err_code=10059); a fresh next install with no streams and no Degraded condition is this, not a render fault. An install that already has a bus reaches the same reason from the other end, with every stream and bucket present: the script’s closing check refuses when TASKS holds fewer consumers than tuning.maxSessions needs plus a reserve it sizes from the BRIDGE_CONCURRENCY a declared bridge sidecar sets, or, where the operator renders the bridge, from the operator’s A2A_BRIDGE_CONCURRENCY (10 when unset), up to 1024 workers, the bridge’s own queue capacity (the script’s refusal quotes both numbers, the rule it read the count by, and whether it capped it; this condition’s message on that refusal, where a sidecar’s entry took the bridge’s default because the render could not read it as a count — a valueFrom, or a reference to one — says the count is what the render read rather than what the CR declares, and states that rule; and the script prints a NOTE: on every run, refused or not, when an entry took the default or a sidecar carries envFrom with no BRIDGE_CONCURRENCY in env, a key the render does not read, since the budget may then be short for the real count), so declaring a bridge sidecar with more workers, or raising A2A_BRIDGE_CONCURRENCY, on an install whose bus already exists reaches it too, and there the symptom is a working bus. The remedy is not a stream edit: max_consumers is the one limit nats-server will not change on a stream that exists, answering an update that carries a different one with stream configuration update can not change MaxConsumers, and the rendered bus is pinned to nats:2.10. The message names the ones that do work — lower tuning.maxSessions until the budget fits the stream; where the CR declares a bridge sidecar with more workers than the bridge’s default of 2, declare it with fewer (a lower BRIDGE_CONCURRENCY, or none), or, where the operator renders the bridge, set the operator’s A2A_BRIDGE_CONCURRENCY to fewer workers (unset, the rendered bridge runs 10), which the message attributes the need to alongside maxSessions wherever the declared count moved the reserve, one worker included, and offers only above the default; or delete the TASKS stream and let provisioning recreate it at the width the CR asks for, which discards the 72h of task history that stream was holding. Those do not finish the same way, and the message splits them. Lowering tuning.maxSessions, or the bridge’s worker count, finishes by itself: either edit re-renders the provision Job, whose name digests that render, so a new Job appears and runs with nothing else to do. Deleting the stream does not — nothing re-reads the bus until the Job runs again, so delete the Job to re-run it, or leave it and the 24h TTL will. Deleting the Job before one of those ways out is taken — the two, or the three where the fewer-workers lever is offered — helps nothing: the same script refuses again against the same stream. Recreating the stream then costs one more step, and the order matters: once TASKS is back, restart the A2A gateway (kubectl rollout restart deployment/<agent>-a2a-gateway), and the agent workload <agent>-gateway with it where a Hermes bridge sidecar runs. Deleting a stream deletes every consumer on it, and neither of those two durable readers re-creates one, so a gateway left running keeps accepting delegations and spawning session pods while relaying no events, the bridge dispatches nothing, and the CR reads Ready over both — session pods recover by themselves, which hides it. Restarting before the stream is back only fails the subscribe.
Degraded	Only while degraded	Something in the spec, cluster state, or operator install cannot be honoured. Reason: InvalidGitRepoURL (an invalid entry in spec.integration.forges or spec.integration.repositories, such as a namespace outside the provider’s grammar or a repository on a host its forge’s provider does not serve; subsequent updates are rejected at admission until corrected), Reason: CorruptManagedRepos (unparseable managed_repos JSON in the <agent>-gitops-state ConfigMap; GitOps disabled), Reason: MinterPruningHeld (a github entry in that ConfigMap’s managed_repos or context_repos that this release cannot read; the operator prunes no minter policy while it is listed, so a repository removed from the lists keeps its write policy until the entry is corrected or removed. Ready and the phase are left as the workload reports them), or Reason: RBACIncomplete (not a spec fault but install skew — the ClusterRole bound to the operator denies a permission its RBAC markers declare, which is what an image deployed ahead of its manifests looks like; the message names the denied verbs and the fix. While InvalidGitRepoURL, CorruptManagedRepos or MinterPruningHeld holds the condition the RBAC reason is not written, and the operator’s startup log is the only trace). The refusals above ride the Ready condition instead of this one.
BusCredentialsReady	Under mode: next, and while a callout Deployment exists	Whether every replica the callout Deployment asks for is ready, and therefore both serving an identity map and attached to the bus — its readiness probe answers 503 unless both hold. It is the claim about the callout as a whole, not the rule a dispatcher waits on: a workload dispatched before the callout can authenticate it is refused with an error indistinguishable from a bad credential, and one callout replica serving on the current spec is enough to prevent that. The operator’s own gates do not read it. The bus provisioning Job is not created until the callout Deployment’s status shows one ready replica on any template (its principal is authenticated by every replica, all of which serve the current map), and the A2A gateway Deployment is not created until the callout Deployment’s status shows at least one replica that is both ready and on the current pod template (ReadyReplicas + UpdatedReplicas - Replicas >= 1, once the Deployment controller has observed the current generation, read from a copy of the Deployment whose generation has reached the one the operator’s own apply of the callout returned in that pass), and is reconciled normally once it exists. That gate is narrower than this condition on purpose. The callout’s replicas form a NATS queue group, so one serving replica answers every authorization request, and a callout whose second pod cannot schedule would otherwise withhold a first gateway indefinitely; the lower bound cannot read true while no current-template replica is serving, so a gateway created while this condition is False still has a serving replica to authenticate its sessions. Expect the two to differ: this condition reads False with 1 of 2 replicas ready while the gateway exists and dispatches. The hold is not silent: under mode: next the A2A gateway is one of the workloads Ready is a claim about, so while it is withheld by this gate the phase reads Provisioning and the Ready message names the Deployment. What waits on the callout is not the gateway’s own connection — the gateway is a static principal, NATS_USER=gateway with a rendered password, listed in auth_users and so exempt from the callout exactly as the Hermes bridge sidecar is. It is what the gateway spawns: a session pod authenticates with a projected token that only the callout can turn into a bus identity, so a gateway created while the callout is unready accepts delegations and spawns sessions that cannot connect. That is the ordering this condition exists to enforce, and the gateway’s creation is the last moment the operator can enforce it. Creation only, so a callout that goes unready later does not freeze a running gateway’s spec, and does not orphan the session pods that carry an ownerReference to it. The gate’s error directions are both false negatives, and each holds the gateway one more pass before the held reconcile’s requeue clears it: while a terminated callout pod is still counted, one ready replica on the current template reads as none; and on the pass that changed the callout’s pod template, the operator’s cached copy of the Deployment may still be the one from before that change, which the gate refuses rather than trusting counts that describe the previous template. Reason: CalloutServing (True), CalloutUnavailable, or CalloutAbsent. The messages differ in what they can name: CalloutServing gives the identity map version being served, the two CalloutUnavailable messages give the ready count against the count asked for, and CalloutAbsent gives neither, because there is no Deployment to read either from. Ready is counted against spec.replicas, not against the pods that exist: the callout rolls at maxSurge: 1 over maxUnavailable: 0, so a healthy roll spends its whole duration with more pods than it wants, and a callout coming up spends its window with fewer — a condition comparing the two numbers to each other would call the first an outage and the second serving. CalloutUnavailable also covers a spec change the Deployment controller has not observed yet, where the reported counts describe the spec before the roll; the message says so rather than reporting those counts as current. It does not assert that a named replica has observed a named map version, so a sub-second window after a re-render can report ready while a replica still serves the previous map. Removed, not set false, once the mode is not next and the callout Deployment is gone — both, so the condition never describes a callout that is still running. Re-derived from the callout Deployment on every exit from the reconcile, not at a point in its sequence, so a pass that parks Ready=False for an unrelated reason — a missing shell sandbox keypair, a refused spec.deployment, say — still reports it rather than leaving it absent or, worse, leaving the last value standing while the callout it named goes away. The one case it is neither written nor removed is version skew, a spec.mode this operator does not recognise: the bus a newer CRD rendered is deliberately left standing there, so the condition describing it is too. A CR flipped to today by an edit that is itself refused also keeps the condition, because the refusal withholds the teardown too and the callout is still running.
A2AGateway	Under mode: next, while the install configures no chat backend for the A2A gateway, and, for a gateway that was scaled to zero, after a backend returns until it has a ready replica; written and cleared by every pass that renders the stack, whichever phase it ends on (Ready, Provisioning, or Degraded for A2AProvisionFailed and ShellSandboxKeysMissing), so only a pass that stops before rendering leaves it as it was	status: False, Reason: NoChatBackend. The gateway is not run without a backend rather than run to crash-loop: neither spec.integration.googleChat nor spec.integration.slack is enabled, there is no discord-bot Secret with a token key in the namespace, and no door is armed on the operator. With no gateway Deployment yet, its creation is withheld. A Deployment that already exists is kept and applied at zero replicas, so the session pods that reference it stay. The message names the remedy. Ready does not wait on a gateway in either state; it says the gateway is not running. When a backend returns the gateway is created, or the existing one goes back to one replica, on the next pass: enabling Google Chat or Slack on the CR is seen at once, a door flag with the operator restart that sets it, and a Secret, which is not watched, within the reconcile’s 30 s requeue. An existing gateway scaled back up keeps the condition, now Reason: WaitingForReplica, and Ready waits on it, until a replica is ready; Ready’s message, not this one, says what the replica is waiting on. Rotate the discord-bot token by updating the Secret in place, since deleting and recreating it can scale a running gateway to zero for a pass if a reconcile lands in between. The running gateway picks the new token up by itself: DISCORD_TOKEN is in its secret-env digest, so a pass that reads the changed Secret rolls the pod (see Rotating a Secret rolls the pod), and kubectl rollout restart deployment/<agent>-a2a-gateway makes it immediate. A gateway at zero replicas is not digested, so its Secret may be gone.
BusProvisioned	Under mode: next, from the first pass that sees the bus provisioning Job complete, whichever phase that pass ends on; kept through the Job’s later re-runs and removed with the stack when the mode flips to today	status: True, Reason: ProvisionJobComplete. The record that lets Ready stop counting the Job: the finished Job is removed by its 24 h TTL and rendered again, and a spec edit renders a new one under a new digest, and neither re-run should flip a Ready install to Provisioning for the minute it takes. Until this condition exists the Job counts, so an install upgraded from an operator that never counted the Job (one that could read Ready over an unprovisioned bus) does not inherit a completion nobody saw. The message names the Job that first provisioned the bus. A re-run that fails is still reported as A2AProvisionFailed.
EventWatcher	Only while eventWatcher.enabled is false	status: False, Reason: DisabledBySpec. The emergency stop is still pressed and no cluster events are reaching the agent.
HomeChannelUnset	Under mode: next, while the A2A gateway holds a chat backend (Google Chat, or Slack with Google Chat off) whose homeChannel is empty	status: True, Reason: NoHomeChannel. The gateway arms proactive posts only on a home channel, so every alert, drift report and scheduled report is dropped, while replies in a conversation still arrive. Not Degraded, and Ready is unaffected: chat works, and an install may want no proactive posts. The message names the field and its install.env key. Written and cleared on a pass that ends Ready or Provisioning, and on one that renders the next stack and then parks Degraded (ShellSandboxKeysMissing, A2AProvisionFailed); a refusal that returns before the render leaves it as it stands. Set homeChannel to clear it.
VolumesDropped	Only after a pass that rendered the Pod template around a hostPath on spec.deployment.extraVolumes or .sidecarVolumes	status: True, Reason: HostPathVolumeDropped. The render leaves the named hostPath entries and every mount naming them out of the Pod template the operator applies; the message lists them by field and index, and on a spec carrying more of them than the controller’s 4096-byte list budget holds it lists as many as fit and counts the rest. Not Degraded: the Pod runs without them. It is a claim about that template and not about the Pod that is running, because nothing here waits on the rollout — see the note under the table. Written on whichever status the pass ends on, Ready or Degraded, because the drop happens at render and three of the reasons that park phase: Degraded render first — ShellSandboxKeysMissing among them, which is where a chart-default install sits, and a chart-default install is one with no webhook in front of it. The four refusals that return before the render — ForbiddenVolumeMount, ShellSandboxCannotBeDisabled, RuntimeClassNotFound, EgressAllowlistRefused — neither write it nor clear it. No Pod is rendered on those passes, so the workload still running is whatever the previous pass left, and a condition written there would report the entries out of a Pod this operator never wrote — on an install rolled forward over a CR a pre-fix operator gave real hostPath mounts, that reads as the security property being satisfied while the live Pod still mounts them. One already present is left exactly as it stands, because the pass that wrote it did render and that Pod is the one running; so while such a refusal holds, the message and the condition’s observedGeneration can both lag the spec until a pass reaches the render again. Remove the entries to clear it.
ReplicasLimited	Only while spec.deployment.availability.replicas is above 1, after a pass that rendered the gateway	status: True, Reason: HighAvailabilityNotSupported. The operator runs the gateway as a single replica, whatever the field asks for; the message names the requested count and why a second replica does not start. Not Degraded: the single replica runs. Written on whichever status the pass ends on, on the same terms as VolumesDropped: a refusal that returns before the render neither writes nor clears it. Set the field to 1, or remove it, to clear it.

Not everything the operator learns is a condition. Under mode: next, one thing the bus provisioning Job finds about a stream it could not change is recorded as an Event on the PlatformAgent instead: a TASKS stream created before the render carried a per-subject cap keeps running unbounded, and the pass that first sees the Job complete records a Warning Event, reason TasksSubjectCapMissing, naming the Job, the live and rendered caps, and the nats stream edit that closes the gap with what that edit costs. A stream carrying some other finite cap is drift, not that gap, and stays in the Job’s pod log. The Event is recorded once per Job run, so roughly daily while the gap stands, because the finished Job is removed by its TTL and run again; Events themselves expire on the cluster’s event TTL (an hour by default), so between runs it can be gone, and the next run records it again. Read it with kubectl describe platformagent or kubectl get events -n kubeagents-system --field-selector involvedObject.kind=PlatformAgent.

EventWatcher is absent on a healthy install rather than True, deliberately: the operator can say it asked for a watcher, but nothing here checks that one is alive, and a permanently-True condition would read as a health signal it is not. Disabling the watcher is also not a Degraded state — it is a decision somebody made, and phase stays Ready. VolumesDropped follows the same shape: it exists while the spec carries a hostPath and a pass has rendered around it, and it is the render-side half of the hostPath refusal, which the admission webhook makes at admission on installs where it runs.

It is a claim about the Pod template the operator rendered, not about the Pod that is running. The controller server-side-applies the template and returns as soon as the API server accepts it; nothing waits on the rollout. So on the first pass of an operator carrying this behaviour over a CR that a previous one rendered with real hostPath mounts, the condition reads True while Pods from the previous revision still mount them — until the roll replaces them, or indefinitely if it stalls at progressDeadlineSeconds. Ready does not close that window either: readyReplicas counts ready Pods across every ReplicaSet the Deployment owns, so one still-ready old Pod is enough for phase: Ready. The operator runs the gateway as a single replica (see availability.replicas above), so the strategy is always Recreate and the window is only as long as the replacement itself. While the operator can see the roll is unfinished — the workload has not observed the applied template, or it counts Pods that are not on it — the message says so in a sentence of its own. A CR parked Degraded by ModeNotRecognized, A2AProvisionFailed or ShellSandboxKeysMissing carries it on the same terms: those three refusals render before they park, so that status write reads the gateway workload back to see whether the roll has finished.

$ kubectl describe platformagent platform-agent -n kubeagents-system
...
  Conditions:
    Type:                 EventWatcher
    Status:               False
    Observed Generation:  3
    Reason:               DisabledBySpec
    Message:              Cluster event ingestion is disabled by spec.harness.eventWatcher.enabled=false. …

Telling a current status from a stale one

status.observedGeneration is the metadata.generation the status was last computed from, and the Ready condition, with any Degraded, EventWatcher or VolumesDropped condition written in the same pass, carries it in its own observedGeneration. Degraded/RBACIncomplete is the exception: it is written when the set of denied permissions changes and left in place otherwise, so its observedGeneration is the generation at which that set last changed. VolumesDropped is the other: a refusal taken before the render leaves it untouched, so while one holds its observedGeneration is that of the last pass which did render. A spec edit bumps metadata.generation at admission and leaves both behind until the next reconcile, so a Ready condition whose observedGeneration is below metadata.generation describes the previous spec, and kubectl wait --for=condition=Ready on its own can return on it. The field says the operator has processed that generation; it does not say the rollout it triggered has finished. Ready is still derived from replica counts, and during a rollout the replica satisfying it can be the previous one. Ready is a claim about three workloads (the gateway, the shell sandbox and the credential broker), and under mode: next about the whole stack the mode renders as well: the NATS StatefulSet, the auth callout Deployment, the provisioning Job’s first completion (recorded as BusProvisioned, after which the Job’s re-runs do not hold Ready), and the A2A gateway — a next install without any of them serves no A2A request, and the message names whichever is missing. That is also how the callout ordering gate becomes visible: while the operator holds a first gateway creation back for a callout replica to serve, the phase reads Provisioning and names the Deployment it is waiting on. A gateway withheld or scaled to zero because the install configures no chat backend is the one exception: Ready does not wait on it, and the A2AGateway condition says why it is not running. So to gate on a change, wait for the generation to be observed, then for the rollout of each workload, then for the condition:

gen=$(kubectl get platformagent platform-agent -n kubeagents-system -o jsonpath='{.metadata.generation}')
kubectl wait platformagent/platform-agent -n kubeagents-system --for=jsonpath='{.status.observedGeneration}'="$gen"
kubectl rollout status deployment/platform-agent-gateway -n kubeagents-system
kubectl rollout status statefulset/platform-agent-shell -n kubeagents-system
kubectl rollout status deployment/platform-agent-credential-proxy -n kubeagents-system
# under `mode: next` only, and the gateway only once a chat backend is
# configured. The verifier is not part of `Ready` — a rollout of it should not
# flip a serving install to Provisioning — but a submission arriving while no
# verifier is ready is refused terminally, so an install that stays down
# reports it as the `A2AVerifier` condition, which a Ready CR can carry.
kubectl rollout status deployment/platform-agent-a2a-gateway -n kubeagents-system
kubectl rollout status deployment/platform-agent-a2a-verifier -n kubeagents-system
kubectl wait platformagent/platform-agent -n kubeagents-system --for=condition=Ready

How config reaches each profile

A deployment runs several Hermes profiles from one pod: default (the Planning Agent front door), platform, and one cluster-* profile per managed cluster. The named profiles are each configured by an overlay merged into an image-built base at startup. The default profile is the exception: it takes the operator’s settings by two routes at once — an overlay merged into its config, and a read-only managed scope pinned over it.

Profile	Delivery	Who owns the file
default	Image-built base, writable on the PVC + profile-default.overlay.yaml merged at startup + a narrow set of keys pinned read-only at /etc/hermes	Agent owns the file, operator the pins
platform	Image-built base + profile-platform.overlay.yaml merged at startup	Image owns the base, operator the overlay
platform, with platformFrontDoor on	The same two inputs, but the base is back-filled rather than force-synced — and the /etc/hermes pins land here too, because that mount is machine-global	Agent owns the file, operator the overlay and the pins
cluster-*	Image-built base, copied when the profile is scaffolded and back-filled at startup + profileclass-cluster.overlay.yaml, plus profile-<name>.overlay.yaml if one exists	Image owns the base, operator the overlay

A cluster profile is the only one that can take two overlays: the class overlay carries tuning.cluster, which applies to all of them, and a plugin targeting one specific cluster produces a profile-<name> overlay for it as well. The class overlay merges first, so the per-profile file wins any conflict.

Why default is also pinned. The pins are the one change-control boundary the front door has: the agent’s own config file is writable, so without them a bad runtime edit survives a restart. (It is not a security sandbox — see the AgentPlugin trust boundary.)

What is pinned is narrow, on purpose. /etc/hermes is machine-global — one file for every profile in the pod, not just default — so it carries only what is identical for every profile and beyond the agent’s own repair: model.*, platforms.*, approvals.cron_mode, display.platforms, kanban.max_in_progress (the board is shared by every profile, and a burst of workers past it is lost to the OOM killer without a restart or an event), terminal.* (where the shell runs: one sandbox per Pod, reached the same way by every profile), when the agent Pod has a runtime class, database.journal_mode (one data volume per Pod, and a corrupted database is found only after the sessions in it are unreadable) and, when the A2A bridge is in the Pod, rendered by the operator or declared, and runs the api executor, one hooks.outbound entry that posts tool calls to the bridge’s loopback trace endpoint (one endpoint per Pod, signed with a key from the A2A credentials Secret). A list is one leaf, so while that entry is rendered it replaces any hooks.outbound list a profile sets in its own config.yaml: those hooks do not fire. The reasoning is that as long as a human can reach the agent (platforms) and the agent can reason (model), anything else it breaks it can be talked into fixing.

Everything else the operator owns for the front door goes in profile-default.overlay.yaml instead: plugins.enabled for AgentPlugins with no targetProfile, those plugins’ non-gateway config subtrees, and spec.harness.tuning’s default limits. Those are profile-shaped — pinning them machine-globally would hand the front door’s settings to every specialist — and they are all recoverable by an agent that can still talk and still reason. Nothing the operator renders appears on both routes. What appears on neither, and so stays the image’s alone, is each profile’s toolsets, mcp_servers and memory.

It is also the only profile whose config the running agent writes to: /sethome records the home channel there, the monitoring policy mints monitoring.install_id, and slash commands save preferences. Those two facts pulled in opposite directions, and the managed scope is what resolves them.

The rendering is published as the managed-config.yaml key of the <agent>-config ConfigMap and mounted read-only at /etc/hermes/config.yaml. Hermes treats that directory as an administrator layer and overlays it, per leaf key, on top of $HERMES_HOME/config.yaml at every load. Three things enforce it (hermes_cli/managed_scope.py):

• load_config deep-merges the managed dict on top of the agent’s own;
• save_config strips every managed leaf before writing, so a save cannot persist one;
• hermes config set rejects a managed key by name.

So $HERMES_HOME/config.yaml stays an ordinary writable file — /sethome and the install id work — while every leaf the operator renders is authoritative and immutable at runtime. Whatever ends up in the PVC file, the operator’s value is what loads, so a restart always heals. Earlier shapes did not manage both: mounting the render over $HERMES_HOME/config.yaml made the path read-only and failed every runtime write (/sethome with a permission error, the rest silently), and merging it into the file at startup left every merged key mutable, so an agent that repointed model.base_url at nothing kept that across restarts.

Hermes builds the terminal for some turns, among them a scheduled run and the turn a kanban wake starts, from the profile’s own .env and config.yaml, so the managed terminal.* leaves do not reach them on their own. The entrypoint copies them into every profile’s .env at start-up and refuses to start while a profile still resolves something else — see Container entrypoint.

platforms.<platform>.home_channel is deliberately not pinned, so /sethome can still set it from chat. The platform credentials and endpoints that have no config.yaml equivalent are pinned through a companion /etc/hermes/.env, which Hermes applies last with override=True and refuses to let the agent overwrite — without that, a container env var would beat the pinned platforms.* leaf.

That file also pins values that are not credentials at all. The first is API_SERVER_KEY=cluster-internal-trusted, the non-secret loopback sentinel the Hermes API server on 127.0.0.1:8642 validates. It is pinned here because Hermes’ stage2 hook generates a random API_SERVER_KEY into $HERMES_HOME/.env whenever that file carries none, and the PVC .env is applied with override=True too — ahead of the container env, behind /etc/hermes. Unpinned, the gateway ends up validating against a value no caller has, and the credential proxy, the startup probe and every in-pod loopback call get 401 Invalid gateway API key. The container entrypoint warns at boot when this pin and the container env disagree. The credential that guards the API from outside is API_SERVER_EXTERNAL_KEY, set from hermes.apiServerSecretRef; the sidecar authenticates the caller against it and swaps in the sentinel. The second is HERMES_HOME_MODE=2770, the mode Hermes re-applies to $HERMES_HOME at every process start. The container env carries it too, but that is the lowest-precedence of the three layers, so a HERMES_HOME_MODE=0777 line the agent writes into the PVC .env outranks it and widens every directory Hermes secures on the shared volume.

The third is KUBEAGENTS_MODE (today or next, from spec.mode) — the mode switch’s delivery contract (docs/designs/spec-mode-switch.md). It is pinned always, with the real value, because an absent key is a key the agent may write, and it is read back by exactly one module, agents/platform/scripts/runtime_mode.py.

The fourth is KUBEAGENTS_SCOPE_FILE=/etc/kube-agents/scope.json, where the reconcile finds the spec.scope declaration. The container env carries it too, for the same reason as HERMES_HOME_MODE: a line the agent writes into the PVC .env would otherwise outrank it and hand the reconcile a declaration the agent authored. Its readers are agents/platform/scripts/cluster_agent_reconcile.py, which resolves it every hour, and the platform MCP server’s fleet_scope tool (through fleet_scope_targets.py), which reads it when no reconcile snapshot answers, or when the last snapshot recorded no boundary.

The fifth is RECONCILE_PROJECT=, pinned empty. The reconcile reads that variable as the management project ahead of the metadata server, and a management project that changes identity retires the old one’s profiles, so a line the agent wrote into the PVC .env could otherwise re-point it and have two clean runs delete every profile of the real management project. Nothing in the operator or the chart sets the variable; the script treats the empty value as unset.

On GKE, GKE_PROJECT_ID is also pinned, to spec.harness.projectID. The reconcile reads it as the management project when RECONCILE_PROJECT is empty, so it is pinned for the same reason. A kind install does not pin it, because the container env there carries no project.

The stall watch’s three knobs from spec.harness.stallWatch are pinned too: STALL_WATCH_EXCLUDE_NAMESPACES (a comma list), STALL_WATCH_MIN_AGE_MINUTES and STALL_WATCH_KINDS (a comma list, or all). They are pinned on every install, empty when the CR does not set them, and the watch reads an empty value as its default. Each one changes what the watch reads or when it alerts, so a line the agent wrote into the PVC .env could otherwise silence alerts for a namespace nobody excluded. They are not in the container env.

One consequence of the render is worth knowing: the managed overlay is a leaf-level merge, and a list is a leaf, so a list rendered here replaces the image’s rather than unioning with it — for every profile at once. That is why the render emits no lists at all today, and why adding one is the change to think hardest about.

Why the others get overlays. Their config.yaml is assembled at image build time by merging the shared defaults with that profile’s own overlay, content the operator does not have; a cluster-* config additionally carries a runtime cluster_identity stamp that the reconciler matches profiles to clusters by. Rendering either file in full would fork the source of truth and, for cluster profiles, strip that identity record.

Every overlay is a key in the one <agent>-config ConfigMap, so a change to any of them moves the config hash and rolls the pod. That restart is required, not incidental: the merge happens once at startup, so a live ConfigMap update without a restart would be a no-op. The managed key shares the ConfigMap and so rolls the pod too, though for it the restart is belt-and-braces rather than required — it is mounted as a directory, not a subPath, so the kubelet propagates updates and Hermes re-reads the file when its mtime or size changes.

Startup is not the only moment a merge happens. Onboarding a cluster scaffolds a new profile without changing the ConfigMap, so nothing rolls the pod; that profile applies the overlays itself as it is created. Without it a Cluster Agent created between two pod starts would run on Hermes’ own defaults however the CR is tuned.

Ordering. The entrypoint force-syncs each profile’s image-owned files first, then merges the overlays. The reverse order would silently erase every overlay on each restart. The default profile’s config.yaml is the exception to the force-sync: it is the agent’s own file, and a force-sync is exactly what would throw the runtime’s edits away. It is instead seeded from the image on a fresh volume, and thereafter only back-filled — keys the image declares and the live file has lost are restored, keys it already holds are left alone, and the plugins.enabled list gains the names the image’s list has and it lacks (below). Its overlay is merged after that back-fill, so the operator’s settings are not undone by it.

The platform profile’s config.yaml becomes a second exception under platformFrontDoor, and for the same reason: the gateway is homed there, so that file is now the one /sethome and the monitoring policy write to. It leaves the force-sync list and is back-filled from the image template instead, on exactly the terms default gets — keys the template declares and the live file has lost are restored, keys it already holds are left alone. Its overlay merges after that back-fill as it always did. Everything else the image owns in that profile — the persona files, cron/, skills/, governance/, hindsight/ — still force-syncs either way.

A cluster-* profile’s config.yaml is never force-synced: it carries the cluster_identity stamp the cluster reconciler matches the profile to its cluster by, and an overwrite would strip it. It is back-filled from the cluster template instead, on the same fill-only terms — keys the template declares and the live file has lost are restored at the next start, and cluster_identity and every value the file already holds are left alone, except the retired memory.provider key, which the entrypoint drops. The persona files and skills/ beside it still force-sync.

The remote MCP server entries inside both of these files do follow the image (see the config reference). The back-fill recurses only through mappings and a server’s args is a list, so it would otherwise stay as the image that scaffolded the profile spelled it for the life of the volume. At every start the entrypoint repairs it in each cluster profile’s config.yaml, and in the platform profile’s when it is the front door: for a server the template routes through the credential proxy’s MCP relay (gke and developer_knowledge), the whole args list and the MCP_REMOTE_BEARER_FILE entry in its env become the template’s, so a hand edit to either is reverted at the next start; for any other remote server the image ships, only the User-Agent header value is set. Nothing else in the file is changed.

The plugins.enabled list follows the image for the same reason: it is a list, so the back-fill never enters it. In the default, platform and every cluster profile it gains the names the image template’s list has and the live list lacks, appended after the agent’s own; nothing is removed, so a name the agent dropped on purpose comes back, which is the price of a plugin that ships enabled. Without it a volume scaffolded before a plugin existed would carry the plugin into every image and never load it.

Merge semantics. These differ between the two mechanisms, which is the easiest thing to get wrong here. In a startup overlay — every profile including default — maps merge recursively, lists union, and scalars are replaced by the overlay; precedence, lowest to highest, is Hermes built-in default → the value committed in agents/<persona>/config.yaml → the operator overlay from the CR. In the managed scope the merge is per leaf key, so a list replaces rather than unions, and it wins over everything else because it is applied at every load rather than once at startup.

Two writers, two authorities. Both spec.harness.tuning (operator policy) and an AgentPlugin’s spec.config (plugin-supplied) land in the same overlay file, but not with equal rights. A plugin’s config is restricted to approvals, platforms, and platform_toolsets, and for an untargeted plugin only platforms reaches the machine-global managed scope — the rest goes to the front door’s overlay. The agent subtree holding the execution limits is dropped from plugin config and writable only by the operator. That is a coordination boundary rather than a security one — plugin code executes in-process and could change these at runtime — but it keeps limits with board-wide consequences in one reviewable place.

Rotating a Secret rolls the pod

Credentials reach the agent pod as environment, through SecretKeyRef, and a container’s environment is fixed for the life of the pod: editing the Secret changes nothing a running container can see. So the operator does for Secrets what the config hash does for ConfigMaps. It reads the Secret keys the rendered pod spec consumes as environment, digests them with an HMAC-SHA256 keyed by the UID of each Secret they come from, and stamps the result on the pod template as kubeagents.x-k8s.io/secret-env-hash. Rotating one of those keys moves the digest, which changes the template, which rolls the pod onto the new value. Three pods are stamped: the gateway, the credential proxy, and under mode: next the A2A gateway, which on a Slack-armed install is the pod that reads the Slack tokens. The mode: next A2A auth callout also reads Secrets as environment (its bus password and its issuer and encryption seeds) but is not stamped, so after rotating one of those, run kubectl rollout restart deployment/<agent>-a2a-callout. The digest is keyed because the annotation is readable by anyone who can read pods: an unkeyed hash would let that reader verify guesses at a low-entropy value offline, whereas the UID is on the Secret object, and reading it takes the same get on the Secret that reads the values. (A UID also travels on Events and owner references that point at the Secret; the operator creates neither.)

Five details decide whether you will see it happen.

• Within fifteen minutes, not immediately. The operator does not watch Secrets — it holds no list or watch on them, deliberately — so nothing wakes a reconcile when one changes. A healthy pass instead asks to be requeued after secretEnvReprobeInterval, and the re-read happens then. kubectl rollout restart deployment/<agent>-gateway still works and is immediate.
• Only what the pod reads as environment. A key no container references is not in the digest, and editing it rolls nothing. Neither does a key the pod mounts: the kubelet refreshes a mounted Secret file in place, so hashing one would roll a pod over a change it was going to see anyway. The gateway mounts exactly one item of platform-agent-secrets that way, SANDBOX_SSH_PRIVATE_KEY, and because an init container copies it into an emptyDir at pod start, rotating that one key still needs a restart — as it did before this change. The shell sandbox mounts its own <agent>-shell-authorized-keys, and deliberately never names platform-agent-secrets at all. The A2A gateway has a second such mount: on a Slack-armed install it mounts the a2a-slack-principal-map Secret, and the gateway reads that map once, when it starts, so an edit to it needs kubectl rollout restart deployment/<agent>-a2a-gateway.
• Whichever Secret the pod actually names. The refs are read off the rendered pod spec, so a CR that supplies its own SecretKeyRef pointing at a different Secret is covered without naming it anywhere.
• A missing Secret is not an error. It digests to a marker, so creating the Secret later moves the digest and rolls the pod, and an install whose credentials arrive after the agent behaves the way you would expect. A Secret the operator cannot read for any other reason — an API error rather than a NotFound — keeps the digest the last good pass computed, so a blip neither rolls the pod nor stops the rest of the reconcile.
• Recreating a Secret rolls the pod once, even with the same values. The digest’s key is built from the UID of every Secret it reads, which an in-place edit, kubectl apply, or a patch keeps and a delete-and-create (including kubectl replace --force) replaces. A metadata-only write — a new label or annotation — changes neither the UID nor the values and rolls nothing.

The roll is a stop-start. At the default single replica the gateway’s update strategy is Recreate, so the old pod is terminated before the new one starts and the agent is unreachable across the gap — up to the startup budget of roughly fifteen minutes on a cold image pull. Expect one such restart per agent the first time an operator carrying this change reconciles: the annotation is new, so the first pass adds it and the template changes once, whether or not anything was rotated. The A2A gateway joined the stamped pods later, so the first operator that stamps it restarts each mode: next A2A gateway once in the same way. An operator upgrade that changes how the digest is computed restarts each stamped pod once in the same way.

Reconcile behavior

• On create/update, the controller ensures the Deployment, Service, ServiceAccount, and ConfigMaps match the spec.
• On delete, it garbage-collects owned resources (note that litellm-policy is owned by Deployment/litellm rather than PlatformAgent, so deleting PlatformAgent preserves litellm-policy while LiteLLM is still running to prevent fail-open egress).
• The admission webhook (behind cert-manager) validates the spec before it’s persisted; it enforces at most one PlatformAgent per cluster, forbids sensitive environment variable overrides (API_SERVER_KEY, HERMES_HOME) and privileged containers/volumes (hostPath), requires each imagePullSecrets entry to name a Secret, and acts as a name-based tripwire against obvious privileged service account names (cluster-admin, system:admin). The hostPath refusal has a second layer in the controller: a hostPath volume that reaches the reconcile anyway — the Helm chart ships the webhook off, and one enabled through the chart fails open at its default failurePolicy: Ignore — is left out of the Pod template at render with every mount naming it, and the VolumesDropped condition reports it. Note that full RBAC least-privilege enforcement is handled by controller- and pipeline-level policies rather than the admission webhook.
• The webhook also refuses a user volume or mount on spec.deployment that would carry the A2A bus credential by a route other than the operator’s own projection, in every mode: the reserved volume name a2a-bus-token, a serviceAccountToken projection for the bus audience (a2a-bus) under any name, or a secret volume or projected secret source naming one of the Secrets the operator renders with bus credentials in them (<name>-a2a-nats-creds, <name>-a2a-nats-config, <name>-a2a-callout-keys). On an install running the A2A surface — spec.mode: next, and also a CR whose spec.mode this build does not recognise — the render leaves the same entries, and every mount naming them, out of the Pod. That happens whether or not admission ran; it is not a fallback for a skipped webhook, though a skipped webhook is when it is the only thing left. env references to those Secrets are not checked. This is a guard against misconfiguration rather than a boundary between the containers of one Pod, since a ServiceAccount token is pod-scoped.
• The kubeagents.x-k8s.io/prevent-deletion: "true" annotation on a PlatformAgent blocks deletion of the resource via the validating webhook (ValidateDelete). This serves as an accidental-deletion guardrail rather than an authorization control — ValidateUpdate does not block removing the annotation, so any principal with update permissions can patch the annotation off before deleting.
• The kubeagents.x-k8s.io/enable-litellm-network-policy: "false" annotation on a PlatformAgent opts the shared litellm-policy out of operator reconciliation and deletes any managed copy without affecting the agent pod’s own NetworkPolicy. Note that deleting the managed policy leaves LiteLLM unselected (fail-open) unless a replacement NetworkPolicy is managed out-of-band.
• The kubeagents.x-k8s.io/otlp-collector-namespace annotation sets the collector namespace for litellm-policy when LiteLLM exports to an in-cluster collector whose namespace cannot be derived from spec.telemetry.otlpEndpoint.
• The kubeagents.x-k8s.io/litellm-upstream-egress annotation adds one egress rule to litellm-policy, to the model server an openai_compatible gateway calls. Its value is compact JSON in one of two forms: {"namespace":"models","port":8000} admits that namespace on that TCP port (for an in-cluster Service, the server pod’s port), and {"cidrs":["10.20.0.0/24"],"port":443} admits those ranges on that port. port is 1 to 65535, namespace a DNS-1123 label, and cidrs holds 1 to 50 entries, each no wider than /12 (IPv4) or /48 (IPv6) and none reaching 169.254.0.0/16, fe80::/10 or a metadata address. A value that breaks any of these, sets both forms or neither, or carries an unknown field is ignored: the operator logs Ignoring invalid annotation value with the reason, adds no rule, and the reconcile carries on. Inference gateway covers how the chart derives the value from litellm.openaiCompatible.
• The kubeagents.x-k8s.io/network-policy-enforcement: absent-accepted annotation is a record, not a control: the Terraform composition stamps it when an install onto an existing cluster chose to proceed without NetworkPolicy enforcement, and the operator does not read it. Installing without NetworkPolicy enforcement says what it means.
• Under the Helm chart, platformAgent.annotations is the route to these annotations. The chart stamps three of them itself: enable-litellm-network-policy from litellm.networkPolicy=false, otlp-collector-namespace from a non-empty telemetry.collectorNamespace, and litellm-upstream-egress from litellm.openaiCompatible when the provider is openai_compatible and the server needs a rule. When it does, an entry that disagrees with the value fails the render — see PlatformAgent annotations in the chart README.
• The Helm chart renders and applies the CR (the install engine drives it through terraform apply); you can also edit it directly with kubectl edit.

Where to go next

• k8s-operator/README.md — build and test the controller locally.
• Quick start (GKE) — how the CR gets applied in a fresh install.