Operator overview

The k8s-operator is a Kubernetes controller that turns a PlatformAgent custom resource into a running Platform Agent Deployment plus everything it needs — Service, ServiceAccount, RBAC, PersistentVolumeClaims, and ConfigMaps for the agent config and logging. It also runs mutating (defaulting) and validating admission webhooks for the PlatformAgent type (see Admission webhooks).

Source: k8s-operator/. Full README: k8s-operator/README.md.

Layout

k8s-operator/
├── api/v1alpha1/           # PlatformAgent type definitions (Kubebuilder)
├── cmd/                    # manager entrypoint
├── config/                 # Kustomize base for the operator + integrations
├── internal/               # controller reconciler + admission webhook logic
├── examples/               # sample PlatformAgent CR
├── Dockerfile              # controller manager image
└── Makefile                # generate, build, test, deploy

What the operator manages

Custom resources in the kubeagents.x-k8s.io/v1alpha1 API group:

• PlatformAgent — declares a Platform Agent instance, container image, service account, chat integrations, and harness toggles.
• AgentPlugin — declares OCI plugin extensions, secret environment variables, and allowed configuration overrides targeted to a PlatformAgent.
• AgentProfile — under spec.mode: next only, declares one kind of A2A agent pod; the operator renders its ServiceAccount, bus identity and agent card. See AgentProfile CRD.

The controller reconciles a PlatformAgent into:

• A Deployment (named <name>-gateway) for the Platform Agent, running the Hermes runtime with a Fluent Bit log-forwarding sidecar and an agent-api-auth sidecar that terminates the PlatformAgent API bearer key, runs the k8s-event-watcher, and runs the drift-detector where an install has enabled it. The gateway executes nothing the model wrote; which credentials it does hold, and why, are on Credential isolation.
• A StatefulSet (named <name>-shell) and its Service, the shell sandbox: sshd on 2222, the durable /opt/data, and the wrappers that stand in for gcloud and kubectl. This is the pod that runs model-authored commands, and its ServiceAccount carries no Workload Identity annotation.
• A Deployment (named <name>-credential-proxy), a ClusterIP Service on port 8765, and a NetworkPolicy narrowing who may reach it (the sandbox and the gateway on 8765; pods in gke-gmp-system, where the managed-Prometheus collector runs, and the operator’s own pods on the metrics-only port 8766) — the credential broker, which holds every credential in the install and executes the real CLIs on the sandbox’s behalf. See Credential isolation.
• A Service fronting the gateway Deployment (API port 8642, plus dashboard port 9119 when the dashboard is enabled).
• A PodDisruptionBudget selecting the Deployment’s pods, maxUnavailable: 1 at every replica count. That declares the agent evictable rather than blocking node drains, and it stays correct when the agent is scaled — a budget keyed to the replica count would deadlock drains the first time someone scaled back to one.
• A ServiceAccount (annotated for Workload Identity) plus RBAC — a viewer ClusterRoleBinding and an “explorer” ClusterRole with its own ClusterRoleBinding.
• PersistentVolumeClaims for the agent’s data and system metadata.
• ConfigMaps for the pod: config overlays merged into each Hermes profile’s config.yaml at startup (including the whole rendered config for the default, Planning Agent, profile — see how config reaches each profile), a SETTINGS.md (GKE scope) mounted into /opt/data/, the spec.scope declaration as scope.json mounted at /etc/kube-agents/, and a Fluent Bit config for the logging sidecar. Each profile’s base config is baked into the image and scaffolded at startup.
• A ConfigMap (named <name>-usage-counters) holding the totals and per-pod baseline behind status.usage’s counters, owned by the CR, so it is collected with it, and written outside the reconcile loop; the operator writes it on the leader, at most once per five-minute poll, when a total or the per-pod baseline changed; a quiet poll writes nothing.
• Optional integrations wired through the CR spec.integration block: Google Chat (Pub/Sub topic/subscription), Slack (bot/app token secret refs), and version control (forges and the repositories on them — GitOps, managed and read-only context — with github as a deprecated alias).
• Under spec.mode: next, additionally the A2A stack (NATS, bus provisioning, the auth callout, the capability verifier and, once a chat backend is configured, the A2A gateway), and a hermes-bridge container in the agent pod, the bus’s task executor, added once the bus is provisioned unless the CR declares its own — see the PlatformAgent CRD page for what it renders.

Custom resource shape

apiVersion: kubeagents.x-k8s.io/v1alpha1
kind: PlatformAgent
metadata:
  name: platformagent
  namespace: kubeagents-system
spec:
  harness:
    clusterName: cluster-a
    location: us-central1-a
    projectId: example-project
    hermes:
      dashboardEnabled: true
      pluginsDebug: false
      apiServerSecretRef:
        name: platformagent-secrets
        key: api-key
  deployment:
    # Image is optional and omitted here on purpose. Omit it to use the
    # operator's default image (its PLATFORM_AGENT_IMAGE env var for
    # private-registry installs, else the public ghcr.io image; see the Docker
    # images page). Set it only to pin an image/registry for this agent:
    #   image: registry.example.com/kube-agents/platform-agent
    imagePullPolicy: IfNotPresent
  security:
    serviceAccountName: kubeagents-platform-agent
    serviceAccountAnnotations:
      iam.gke.io/gcp-service-account: kubeagents-platform-gsa@<project>.iam.gserviceaccount.com
  integration:
    googleChat:
      # subscription config...

harness.clusterName, harness.location, and harness.projectId are all required. The credential proxy only bootstraps a kubectl context when it has the complete triple; leave any one out and every kubectl call the agent makes resolves to localhost:8080 instead of a cluster.

Full walkthroughs: PlatformAgent CRD, AgentPlugin CRD and AgentProfile CRD.

Admission webhooks

The manager serves a mutating (defaulting) and a validating webhook for PlatformAgent. The Kustomize install registers them with failurePolicy: Fail. The Helm chart leaves them off by default (operator.webhooks.enabled=false, because the chart cannot install the cert-manager they need) and registers them at operator.webhooks.failurePolicy, which defaults to Ignore. The Terraform full-install composition turns them on, since enable_webhooks defaults to true there. So on a supported install the webhooks are registered — but under Ignore an unreachable one admits the object with validation skipped rather than failing the apply, and on a fresh full-install that is the first PlatformAgent rather than an edge case: Helm applies the webhook configurations ahead of the cert-manager Certificate and the CR in the same release, which is why the default is Ignore at all. Setting it to Fail is supported and documented (see the chart README). Controls that must hold regardless are enforced in the render as well as at admission.

The webhook server listens on port 10250, not Kubebuilder’s usual 9443. GKE creates one firewall rule from the control plane to the nodes, and it permits only tcp:443 and tcp:10250. The API server dials the endpoint pod IP on the Service’s targetPort, so on a private cluster a webhook on any other port is unreachable until someone adds a VPC firewall rule for it — per cluster, by hand. Serving on 10250 lands inside the rule GKE already made. It does not collide with the kubelet, which binds 10250 on the node IP in a different network namespace.

The port is set in three places that must agree, and a test under make test fails if they drift: the --webhook-port flag default in the webhook package, the manager containerPort, and the Service targetPort. The Service port stays 443 regardless — that is what the *WebhookConfiguration clientConfig resolves to, not what crosses the network.

Serving on a different port

On a cluster where 10250 is not the reachable port — one that scopes GKE’s rule to node IPs, or a non-GKE cluster with its own constraints — moving --webhook-port on its own wedges the cluster. The flag moves only the listener; the Service keeps sending the API server to 10250, nothing answers, and failurePolicy: Fail blocks every PlatformAgent write. That is the outage this port change exists to prevent, reached from the other side.

All three have to move together, so the override is a Kustomize patch rather than a flag:

# config/webhook-port-patch.yaml, referenced from your overlay's `patches:`
- target:
    kind: Deployment
    name: controller-manager
  patch: |
    - op: add
      path: /spec/template/spec/containers/0/args/-
      value: --webhook-port=8443
    - op: replace
      path: /spec/template/spec/containers/0/ports/1/containerPort
      value: 8443
- target:
    kind: Service
    name: webhook-service
  patch: |
    - op: replace
      path: /spec/ports/0/targetPort
      value: 8443

Changing the compiled-in default instead of patching means changing the flag default in the webhook package as well — the test reads both manifests and fails if either still names the old port. --webhook-port rejects anything outside 1–65535 at startup rather than letting controller-runtime fall back to its own 9443 default.

Upgrading from an operator that served 9443

Re-apply the manifests; do not bump the image alone. targetPort lives in the Service, so a kubectl set image — or any pipeline that rolls the tag without re-applying config/webhook/ — leaves the Service pointing at 9443 while the new pod listens on 10250, which is the wedge described below. make deploy IMG=$IMG applies both.

Applying both together still leaves a short window: the Service starts sending traffic to 10250 the moment it is applied, and the old pod does not answer there. Any PlatformAgent write in the gap between the Service change and the new pod becoming Ready fails closed. It is seconds on a healthy rollout, but schedule the upgrade accordingly rather than alongside a PlatformAgent change.

If the API server cannot reach the webhook, failurePolicy: Fail means every PlatformAgent create, update, and delete fails with a timeout — including the edits you would use to fix it. Errors read context deadline exceeded or failed calling webhook. To recover, and to roll back a bad webhook deployment:

kubectl delete validatingwebhookconfiguration kubeagents-validating-webhook-configuration
kubectl delete mutatingwebhookconfiguration kubeagents-mutating-webhook-configuration
kubectl -n kubeagents-system set env deploy/kubeagents-controller-manager ENABLE_WEBHOOKS=false

That leaves the cluster with the validation coverage a chart install with operator.webhooks.enabled=false has. The CRD’s own schema and CEL rules still run – those belong to the API server, not to the webhook – as does whatever the render enforces on its own. What goes away is everything the operator’s admission checks add on top, the bus-credential refusals among them. Re-apply with make deploy IMG=$IMG once the cause is fixed.

An image ahead of its ClusterRole

The webhook port above is one instance of a general skew: make deploy ships the ClusterRole with the image, and only when it is re-run. A controller deployed from a floating tag such as :latest is upgraded on its next pod reschedule while the applied ClusterRole stays put, and the first verb the newer controller needs that the older role lacks fails every reconcile with forbidden. Two things say so.

make deploy refuses an IMG whose tag is latest, main, master, HEAD, dev, or missing, unless ALLOW_MUTABLE_IMG=1 is set for a cluster that will be thrown away before its next reschedule.

The controller checks its own permissions with SelfSubjectAccessReview when it starts and every five minutes after, on its own ticker rather than on the reconcile worker. A denial is one RBAC self-check failed line in the startup log naming the denied permissions (patch poddisruptionbudgets.policy) and the fix, and a Degraded condition with reason RBACIncomplete carrying the same text on each PlatformAgent it reconciles that is not already Degraded for another reason. Reconciles continue; the steps that need the missing verb still fail until the manifests are re-applied. Re-applying them with the same image tag restarts nothing and triggers no reconcile, so the condition follows the next reconcile after the probe notices: every five minutes while the condition stands and the reconcile completes, or, while the reconcile itself is failing on the missing verb, on controller-runtime’s retry backoff, which caps at about seventeen minutes.

Related resources

• PlatformAgent CRD — reference for PlatformAgent custom resource.
• AgentPlugin CRD — reference for AgentPlugin custom resource.
• k8s-operator/README.md — build, test, and run the operator locally.
• scripts/installer/README.md — the installer helper scripts (install.env loader, tfvars generator), at the repository root rather than under k8s-operator/.