Observability

The Platform Agent (Hermes) Deployment exports OpenTelemetry traces and two sets of Prometheus metrics, the event watcher’s from its agent-api-auth sidecar and the inbound chat message count from the Hermes container, and the credential-proxy Pod exports the broker’s; LiteLLM and vLLM export both OpenTelemetry traces and Prometheus metrics to GKE Managed telemetry, and the Hindsight memory API exports Prometheus metrics. Container logs go to Cloud Logging. The Platform Agent’s persona also generates Cloud Console links inline in Chat replies whenever it’s discussing telemetry.

What gets exported

Prometheus metrics

LiteLLM — request latency, per-model token counts, error rates on its /metrics endpoint (port 8080). Scraped by GKE Managed Prometheus via the litellm-monitoring PodMonitoring shipped in the LiteLLM integration base (k8s-operator/config/integrations/litellm/base/podmonitoring.yaml).
vLLM — per-request latency histograms, queue depth, and GPU/KV-cache stats when running local models on GPU node pools. Exposed on its own /metrics endpoint and scraped by GKE Managed Prometheus.
Hindsight — the Planning Agent’s memory store. Retrieval and reranking latency (hindsight_operation_duration_seconds), HTTP request counts and durations, database pool wait times, and the token spend of its own extraction and consolidation calls (hindsight_llm_*, which bill through LiteLLM). Served on the API’s ordinary HTTP port (8888), not a separate metrics listener, and scraped via the hindsight-monitoring PodMonitoring in k8s-operator/config/integrations/hindsight/podmonitoring.yaml. Recall latency is dominated by the reranker, so hindsight_operation_duration_seconds is the signal to watch — see docs/designs/memory.md for why. The Postgres StatefulSet exports nothing; the pgvector/pgvector image ships no exporter.
Event watcher — per watched cluster: events seen, filtered, injected and deduplicated, inject errors, session creates, triage sessions opened (k8s_event_watcher_triage_sessions_total, the creates whose event was delivered), active incidents, and k8s_event_watcher_cluster_up, the one series that says events are flowing from that cluster. Served by the k8s-event-watcher in the gateway pod’s agent-api-auth sidecar on port 9095 (container port event-metrics) and scraped by the <name>-gateway-monitoring PodMonitoring the chart renders where the cluster serves the PodMonitoring API; platformAgent.podMonitoring, null by default, forces it on or off. The listener also serves process_start_time_seconds, a gauge captured once at process start rather than re-derived per scrape, which the operator reads to tell a restart from a counter that fell. The operator’s gateway NetworkPolicy admits the managed collector and the operator’s own pods on that port. In Cloud Monitoring the collector’s own cluster and location labels take precedence, so the watcher’s arrive as exported_cluster and exported_location. The watcher’s README says which series to alert on and how.
Credential broker — every command the sandbox’s wrappers send to the broker’s exec route, as kubeagents_tool_invocations_total{tool,subcommand,status}, where status is success (exit 0), error (a non-zero exit, a rejected request, or a broker fault), blocked (a policy refusal), busy (the broker’s command slots stayed full, or its child memory budget held the command, for the whole wait, so it never started) or abandoned (the caller hung up while its command was queued or running; a running one was killed); kubeagents_tool_execution_duration_seconds{tool}, a histogram from 50ms to 60s over the commands that ran; kubeagents_credential_proxy_requests_total{endpoint,status_code} for the credentialed listener’s traffic by route family; the admission queue the child memory budget creates, as kubeagents_credential_proxy_admission_wait_seconds{bound}, a histogram from 100ms to the broker’s 60s refusal bound over every admission (the 15s edge is the cap the Cluster Agent preflight puts on one brokered call), where bound is slot or budget for what held a request that waited (with the budget on, a wait the slot cap never touched counts as budget) and none for a request admitted at once (an idle broker’s observations are all none: a measured zero that blames neither bound, and a label that does not change with the memory limit), and kubeagents_credential_proxy_admission_refusals_total{bound}, the requests answered busy, by the bound that refused them: those two, refresh-lock for a forge refresh that waited out the refresh lock, session for the session role’s own limit, and other for a bound the broker did not name (a forge refresh that stepped aside for a vcs verb and was told busy counts under budget, which is what held it); kubeagents_credential_proxy_slots_in_use and kubeagents_credential_proxy_slot_cap, kubeagents_credential_proxy_child_memory_reserved_bytes and kubeagents_credential_proxy_child_memory_budget_bytes, the last absent while the budget is off (a limit under its floor, or none), which is the sign to look for on an install whose proxy never queues and never refuses; kubeagents_vcs_requests_total{verb,status} for the version-control route by verb (the route table’s vocabulary, or other) and the same outcome words as the tool counter; kubeagents_vcs_proposals_merged_total, the broker’s own proposals its reads showed merged, once each per process; and process_start_time_seconds, captured once at process start, which the operator reads for the same reason as the watcher’s. Read the advice by bound. A median wait in the tens of seconds, or refusals, under budget is the broker telling you to raise the proxy’s memory limit (spec.deployment.credentialProxy.resources). The same under slot says the slot cap of eight binds, and raising the limit changes nothing: narrow what the fleet reconcile and the stall watch list at once. At the operator’s 2Gi default the budget admits nine, so slot is the bound a default install sees; the design behind the figures is docs/designs/credential-proxy-child-memory-budget.md. The git the broker runs itself for its version-control and workspace routes is not a tool invocation and is not counted here. Served on a metrics-only listener, port 8766 (container port cred-metrics), separate from the credentialed port so the collector and the operator are admitted to counters and nothing else, and scraped by the <name>-credential-proxy-monitoring PodMonitoring behind the same platformAgent.podMonitoring switch. Every label value is a static enum or a word from a closed vocabulary — for kubectl the policy’s read verbs plus the write verbs a refusal is counted under, for gcloud the command groups the policy reads plus the few a refusal is counted under (a gcloud command is labelled by its group, container or iam, never by a verb), for git and gh the broker’s own lists — so nothing a caller sends reaches a series: an unlisted verb or an unserved executable counts under other.
A2A gateway (the spec.mode: next stack only) — kubeagents_a2a_gateway_messages_inbound_total{platform}, each chat message the gateway accepted as a turn, counted before the sender is verified, by googlechat, slack, discord, console or other (a door’s turns are not chat and are not counted), which the operator reads into status.usage.chatMessagesInboundTotal beside the plugin’s series; kubeagents_a2a_gateway_sessions_total{platform}, each conversation the gateway opened a session for, counted once under the platform of its first message, which the operator reads into status.usage.sessionsTotal; kubeagents_a2a_gateway_task_terminals_total{state,source}, each task terminal the gateway delivers to a conversation, where state is completed, failed, canceled or rejected and source is executor (the executor’s own account), supervisor (the gateway’s word about an executor that died or never ran), gateway (the task never reached the bus, so a NATS outage shows here) or gateway-never-started (no executor took the task inside A2A_FIRST_EVENT_GRACE); kubeagents_a2a_gateway_gchat_events_received_total, the Google Chat events pulled from the broker’s relay; kubeagents_a2a_gateway_gchat_pulls_total{outcome}, the relay pulls by events, empty or failed, so a Chat install that receives nothing shows empty rising rather than looking healthy; process_start_time_seconds, captured once at start, which the operator reads for the same reason as the watcher’s; and kubeagents_a2a_gateway_chat_messages_total{outcome}, the structured chat messages (a report, a question, a plan) the agent sent beside its text, counted each time one is posted or edited in: rendered (laid out, on Slack), text_only (a backend that shows text only posted the text), refused (Slack could not take it and the text posted instead) or invalid (it broke a rule and was dropped, so the text posted instead, or only its reaction state was bad and was dropped), so a rising refused or invalid on a Slack install usually means structured messages are reaching Slack as plain text; and kubeagents_a2a_gateway_notify_marks_dropped_total, the reaction updates from the board’s cards that the gateway dropped because it could not tell which of your messages they belonged to while the agent was working on another, each one a reaction that may be left stale. A value outside those lists counts under other, and no label carries a task, a conversation or a reason. Served on a metrics-only listener, port 9096 (container port a2a-metrics), separate from the doors’ ports, and scraped by the <name>-a2a-gateway-monitoring PodMonitoring behind the same switch.

The Platform Agent (Hermes) container serves one Prometheus endpoint of its own, the chat_metrics plugin’s /metrics on port 9097 (container port chat-metrics): kubeagents_chat_messages_inbound_total{platform}, the chat messages that reached the gateway’s dispatch hook, counted before the sender allowlist, by googlechat, slack, teams, api (the gateway’s own API port) or other; kubeagents_chat_sessions_total{platform}, the sessions the gateway opened, counted by the session_store plugin on the first message of each under the same labels; and process_start_time_seconds. Scraped by its own PodMonitoring, <name>-gateway-chat-monitoring, which selects the pod carrying kubeagents.io/is-leader: "true": above one replica only the leader runs Hermes and has the port, and at one replica the operator puts the label on the one pod. Its other runtime signals surface as OpenTelemetry traces (below) and tool_call_audit audit records (below); pod-level CPU/memory is available through the Kubernetes metrics API (kubectl top).

The operator reads some of these series itself, every five minutes: the broker’s kubeagents_tool_invocations_total and the watcher’s k8s_event_watcher_events_injected_total become status.usage.toolExecutionsTotal and status.usage.eventsIngestedTotal on the PlatformAgent, the broker’s kubeagents_vcs_requests_total under verb="proposal-create" and status="success" becomes status.usage.remediationsProposedTotal and its kubeagents_vcs_proposals_merged_total becomes status.usage.remediationsMergedTotal (a merge no broker read shows is not counted, so that field under-counts rather than over-counts), and the chat plugin’s kubeagents_chat_messages_inbound_total becomes status.usage.chatMessagesInboundTotal (on a next install the A2A gateway’s kubeagents_a2a_gateway_messages_inbound_total joins it there), and status.usage.sessionsTotal is the watcher’s k8s_event_watcher_triage_sessions_total (the sessions it opened whose event the daemon delivered) plus the chat plugin’s kubeagents_chat_sessions_total (plus the A2A gateway’s kubeagents_a2a_gateway_sessions_total on next), all kept monotonic across restarts, and the watcher’s k8s_event_watcher_cluster_up becomes status.usage.clustersRegistered (the series present) and status.usage.clustersMonitored (those at 1), the latest reading rather than a total. That is where an install without a collector sees the counts and how much of its fleet is being watched (CRD reference).

OpenTelemetry traces

LiteLLM and vLLM export spans directly to the GKE OTel collector (gke-managed-otel namespace). That collector is the default, not a requirement — see Deploy → Telemetry for pointing the deploy at your own.
Hermes exports session, tool-call, and MCP spans via the hermes_otel plugin, enabled in every profile config (agents/chat/config.yaml for the Planning Agent, agents/platform/config.yaml for the Platform Agent, and the agents/cluster/config.yaml template for the per-cluster Cluster Agents).
Traces route to Google Cloud Trace.
All of this assumes a collector. On a cluster that has none, the operator disables the agent’s exporter rather than pointing it at one that is not there — status.telemetry.otlpEndpointSource reads None, and export resumes on its own within 15 minutes of a collector appearing. See Deploy → Telemetry.

Cloud Logging

All container stdout/stderr is ingested by Cloud Logging by the GKE log agent. Cluster and pod labels flow through automatically. The Platform Agent writes its own logs to files under /opt/data/logs/*.log; a fluent-bit sidecar tails that shared volume and streams the lines to stdout so they reach Cloud Logging alongside every other container. Audit records arrive as structured jsonPayload fields rather than as text to parse. The tool_call_audit plugin and the chat_message_audit hook write theirs to a file of their own, logs/audit.jsonl in the profile’s home, which the same sidecar tails as JSON (below). The credential broker’s own log is one JSON object per line, the record that says why it refused to start included — severity, timestamp, logger, message, and exception when a traceback is attached — and each command on the broker’s exec route adds tool_execution_audit records whose request_id, principal, tool, subcommand and status are fields of their own (plus pod, the caller’s pod when its token was pod-bound, which is how a session pod’s command is tied to its conversation), with exit_code and duration_ms once a command has run (a command killed after its caller left has the duration only) and rule on a refusal. The record’s status is started, completed, blocked, rejected, failed, abandoned or busy, a different vocabulary from the metric’s above: completed says the command ran, whatever its exit code, and exit_code says how it went. Those fields carry identifiers and outcomes only. The record’s message is the broker’s log text, which names identifiers rather than values: the executable; on a policy refusal, the rule id and, for a read-only refusal, the verb path or the name of the flag it tripped on; on a workspace-lease refusal or a containment rejection, the requested directory or kubeconfig path; on a scoped-credential refusal, the project, location and cluster asked for; on a rejection because the broker could not fetch a cluster’s credentials, what its own gcloud said on stderr, truncated; and never an argument value, an input, or the output of the caller’s command.

Nothing in the envoy-credential-proxy container may log credential material. That container, in the credential-proxy Pod, is the one place holding cluster credentials, GCP tokens, and chat secrets, and everything it writes to stdout leaves the cluster through the path above. The same rule covers the event watcher and the drift detector, which both run in the gateway Pod’s agent-api-auth sidecar: they log identifiers — cluster, namespace, pod, event reason, profile directory, and for the detector the principal and resource name a change names — and never a token, a kubeconfig body, or a request header.

The broker records two bodies. One is the stdout and stderr of its start-up shell bootstrap when that command fails, truncated but not redacted. The other is the GitHub refresh helper’s stderr, because a broker that refuses a mint is otherwise recorded nowhere — the caller gets a reason code with no detail, and the reason code is all a chat room ever sees; that text is passed through a redactor that blanks GitHub token and JWT shapes before it is logged. The rules the code follows to keep it that way are in docs/credential-isolation-design.md.

Session metadata plumbing

Every Chat message carries session context (space ID, user, thread) that flows through Hermes as OpenTelemetry span attributes and out to Cloud Trace. The session_store and session_otel_bridge plugins that do this run on the Planning Agent profile, which owns chat ingress. The trace is documented in docs/designs/gchat-session-metadata-data-flow.md.

Inline Console links

SOUL.md §5 requires the agent, whenever it’s discussing telemetry, tracing, logs, or debugging, to generate clickable Cloud Console links using the active project ID. The URL templates live in agents/platform/docs/gcp-console-links.md — a shared runtime reference baked into the agent image at /opt/defaults/docs/ and inlined directly into SOUL.md §5 — covering Logs Explorer, Trace Explorer, Metrics Explorer, and the GKE Workloads console. The agent substitutes the runtime project ID and formats the links as Markdown so they render clickable in Chat.

Auditing the agent itself

The kube-agents-observability skill audits the harness’s own telemetry — logs, traces, metrics, API/dashboard observability of the Platform Agent. Use it when triaging “why did the agent do X?” or “why isn’t the agent responding?”.

Tool-call audit

The tool_call_audit plugin (enabled on the Planning Agent and Platform Agent profiles; Cluster Agent profiles enable only hermes_otel) writes per-tool-call records for every skill invocation and MCP tool call, and the chat_message_audit hook writes a record at each stage of a chat turn: chat_message_start, one chat_message_step per agent iteration, chat_message_end. Each record is one JSON object: event_type (and audit_event, the same value under the name the Admin Console queries), severity, timestamp, and the event’s own fields: tool, status and task_id on the tool-call records, with duration_ms on the one that ends a call; status on the approval records; session_id and a pseudonymised principal on gateway_dispatch and the chat-turn records. The tool-call records also carry the session_id of the session the call ran in, empty for a call outside any session. Arguments, results and message text are redacted (AuditRedactor) and truncated by the emitters, and inside Hermes the whole line then passes through Hermes’ own redact_sensitive_text, the engine its RedactingFormatter applies to agent.log and whose token list is the wider of the two, forced so that a profile’s security.redact_secrets: false does not reopen a trail that leaves the pod. Each emitter appends its record to logs/audit.jsonl under the profile’s home, one JSON object per line: /opt/data/logs/audit.jsonl for the Planning Agent, /opt/data/profiles/<name>/logs/audit.jsonl for a named profile. The two emitters rotate that file themselves at 5 MiB with three backups, the bound Hermes keeps by default on its agent.log. They are not its only possible writer, though: any process under the agent’s uid with a path to the profile’s logs/ can append a line the sidecar ships as a record, which is the volume’s boundary rather than the sidecar’s. The fluent-bit sidecar tails every profile’s copy with its JSON parser, so the record’s keys arrive as fields of their own, jsonPayload.event_type, jsonPayload.tool and so on, with jsonPayload.file_path naming the file and therefore the profile; no regex over a Hermes log line is involved, so a Hermes release that changes its log format does not touch the trail, and a line in agent.log shaped like a record does not enter the structured trail (the Admin Console’s text-form query still reads such lines, as it reads the records written that way before the file existed). The writes from the gateway and its worker processes are serialised by a lock file beside the audit file, audit.jsonl.lock, which nothing tails; the lock is taken without blocking and retried for 1.5 s, after which the record takes the fallback below rather than hold up the tool call the plugin’s hook is part of. A record the emitter cannot write to the file, because the volume is full or read-only, the lock is held past that wait, or Hermes refuses to materialise the profile’s directory, is printed instead to the emitting process’s stdout as the same JSON line, and an ERROR naming the file and the error, without the record, goes to that process’s agent.log. What the two reach depends on the process. From the gateway the stdout is the container log, so the GKE log agent ships the record to Cloud Logging as jsonPayload under the agent container rather than the sidecar and jsonPayload.audit_event:* still finds it, and the ERROR lands in the Planning Agent’s agent.log, which the sidecar tails. From a kanban worker neither reaches Cloud Logging: the stdout is the worker’s captured transcript, the board’s logs/<task>.log on the same volume, and the ERROR lands in the named profile’s agent.log, which the sidecar’s *.log input under /opt/data/logs/ does not reach; on a full volume the transcript write fails too, and the ERROR is the only trace. jsonPayload.audit_event:* in Logs Explorer is the query for the trail (event_type is on the broker’s records too; audit_event is only on these). Roll the operator with or before an agent image that carries these emitters: under an operator whose sidecar configuration predates the agent.audit input, the file is written and nothing tails it, and the records reach Cloud Logging only once the operator’s ConfigMap carries that input.

Where to go next

Deploy → Telemetry — install-side details on the GKE Managed OTel and Prometheus config.
Reference → Attribution — how a tool call ties back to the authenticated human.