Telemetry

The shipping deploy wires the Platform Agent, LiteLLM, and vLLM into GKE Managed telemetry so you don’t run your own OTel collector or Prometheus. Container logs go to Cloud Logging automatically. If you already run a collector, point the deploy at it instead.

For what’s exported and how the agent surfaces it in Chat replies, see Concepts → Observability. This page covers deploy-side details.

What runs where

Signal	Producer	Collector	Destination
Metrics	LiteLLM, vLLM, Hindsight, event watcher, credential broker, A2A gateway (spec.mode: next)	GKE Managed Prometheus	Cloud Monitoring
Traces	LiteLLM, vLLM, Hermes	GKE OTel collector (gke-managed-otel)	Cloud Trace
Container logs	All containers	GKE built-in log agent	Cloud Logging
Tool-call audit	Hermes tool_call_audit plugin, chat_message_audit hook	fluent-bit sidecar (tails the audit file as JSON), then the GKE log agent	Cloud Logging
Broker audit	Credential broker (JSON lines on its stdout)	GKE built-in log agent	Cloud Logging

GKE Managed Prometheus

Enabled at the cluster level (default on new GKE Standard clusters, opt-in on older). LiteLLM, vLLM, and the Hindsight memory API expose Prometheus /metrics endpoints (LiteLLM on port 8080, vLLM on port 8000, Hindsight on port 8888); managed Prometheus scrapes them via PodMonitoring resources shipped with each integration (the LiteLLM operator base at k8s-operator/config/integrations/litellm/base/podmonitoring.yaml, the Hindsight one at k8s-operator/config/integrations/hindsight/podmonitoring.yaml, and the vLLM example manifests under examples/). The event watcher in the gateway pod’s agent-api-auth sidecar exposes one too, on port 9095, the chat_metrics plugin in the same pod’s platform-agent container another on port 9097, and the credential broker exposes its own on a metrics-only listener, port 8766; the chart renders the <name>-gateway-monitoring, <name>-gateway-chat-monitoring (which selects the pod carrying kubeagents.io/is-leader: "true", the one that runs Hermes) and <name>-credential-proxy-monitoring PodMonitorings that scrape them wherever the cluster serves the PodMonitoring API unless platformAgent.podMonitoring says otherwise, and the operator’s policies on both pods admit the collector and the operator’s own pods on those ports and no other peer. Under the spec.mode: next stack the A2A gateway serves its own on port 9096, scraped by a fourth, <name>-a2a-gateway-monitoring; its policies, rendered wherever the A2A gateway runs, admit the collector and the operator’s own pods on 9096 and no other peer.

Where token spend lives

LiteLLM is the only component that knows what a request cost, and its prometheus callback publishes that as litellm_spend_metric_total — a counter labelled with the real model and provider it routed to alongside the requested_model alias the agent asked for:

litellm_spend_metric_total{api_provider="gemini",model="gemini-3.5-flash",requested_model="model-default",…} 2.907

That metric is the source of truth for spend, and it is already scraped by the PodMonitoring above.

Hermes’ own per-session cost fields are not. A session records estimated_cost_usd: 0.0, cost_status: unknown, and cost_source: none no matter how many tokens it burned. This is a consequence of routing every agent through the gateway rather than a misconfiguration: Hermes prices a turn either from a built-in table keyed by a first-party provider (gemini, anthropic, openai) or from pricing published by the endpoint’s own /v1/models. Agents here are configured provider: custom against LiteLLM, which misses the table, and LiteLLM’s /v1/models carries no pricing by design — it is the OpenAI-compatibility shim, and points at its own /model/info for pricing, which nothing probes. Naming a real model instead of the model-default alias does not help; the custom route is what misses, not the alias.

Read the Prometheus counter, not the session fields.

OpenTelemetry

The Hermes runtime enables the hermes_otel plugin (enabled in every profile config — Planning Agent, Platform Agent, and the Cluster Agent template). Its trace backend is baked into the image pointing at http://opentelemetry-collector.gke-managed-otel.svc.cluster.local:4318/v1/traces (deploy/docker/Dockerfile), which forwards to Cloud Trace. That bake is a fallback: at container start the entrypoint rewrites the backend of every plugin config it can see — the runtime root and each existing profile — to whatever OTEL_EXPORTER_OTLP_ENDPOINT says, and profiles scaffolded later (the per-cluster Cluster Agents) get the same treatment when they are created. Leave the variable unset and the baked endpoint stands, unless discovery resolves None (or OTEL_SDK_DISABLED=true is set), which disables the plugin and clears backends so no spans are sent to the missing collector (see Discovery below).

LiteLLM (via the otel callback and OTEL_EXPORTER_OTLP_ENDPOINT) and vLLM (via --otlp-traces-endpoint) are configured in their deployment manifests to export directly to the same collector — no per-component collector deploy.

Pointing at your own collector

If you run your own collector — say a Service otel-collector in namespace otel-collector — you do not have to edit any manifest. The endpoint resolves per PlatformAgent at reconcile time, in this order:

#	Source	status.telemetry.otlpEndpointSource
1	spec.deployment.env[OTEL_EXPORTER_OTLP_ENDPOINT]	DeploymentEnv
2	spec.telemetry.otlpEndpoint	Spec
3	OTEL_COLLECTOR_ENDPOINT on the operator	OperatorEnv
4	In-cluster discovery found one	Discovered
5	Discovery ran and this cluster has none	None
6	The GKE managed collector	Default

Rungs 1–3 suppress discovery entirely — no probe, no API calls. The result is on the resource, which is the only way to tell “discovered the managed collector” from “fell back to it”:

kubectl get platformagent -A -o jsonpath='{.items[*].status.telemetry}'

Discovery

The operator probes a fixed list of well-known Services by name — gke-managed-otel/opentelemetry-collector first, then otel-collector/otel-collector, opentelemetry/opentelemetry-collector, opentelemetry-operator-system/otel-collector, observability/otel-collector, monitoring/otel-collector — and if none exist, lists Services matching app.kubernetes.io/name=opentelemetry-collector, then app.kubernetes.io/component=opentelemetry-collector, then app=opentelemetry-collector. Multiple matches are sorted by (namespace, name) and the first wins; the losers are logged.

A Service qualifies on a TCP port named otlp-http, else http-otlp, else numbered 4318. A collector exposing only 4317 is skipped, not selected — the agents and LiteLLM speak http/protobuf and hermes_otel POSTs to /v1/traces, so a gRPC-only pick would fail on every span while looking configured. Point at it explicitly if that is what you want. Discovery is structural: a Service can expose 4318 and still refuse OTLP (wrong receiver, mTLS). No TLS is inferred, so an https:// endpoint has to be set explicitly.

Results are cached cluster-wide for 5 minutes, including the “found nothing” answer. When the answer is None or Default the operator re-probes every 15 minutes, so a collector installed after the agent still gets picked up without a restart. API errors are never cached: the last known good answer is reused — including a cached None — rather than flapping and rolling the pod. An install that narrows the operator’s cluster-wide RBAC on services makes discovery return Default silently, and otlpEndpointSource is what makes that visible.

None and Default are different answers and the distinction is the point. None means the probe completed and there is no collector here, so the agent is given no OTEL_EXPORTER_OTLP_ENDPOINT at all and runs with OTEL_SDK_DISABLED=true; the collector egress rule is left out of its gateway NetworkPolicy for the same reason (the opt-in allowlist policy makes the opposite choice, for the reason given under credential isolation). Default means nobody established that — discovery is switched off via OTEL_COLLECTOR_DISCOVERY=false, or no probe has completed — and the GKE managed collector is assumed, which is what installs did before discovery existed.

Wiring the managed endpoint on a cluster that does not have it is what the None rung exists to stop. What it stops specifically is the OpenTelemetry SDK’s metric exporter, which reads OTEL_EXPORTER_OTLP_ENDPOINT directly: on a collector-less cluster it POSTs /v1/metrics to a name that never resolves, once per export interval, for the life of the pod, and the resulting traceback buries every real line in the gateway log. Omitting the variable on its own would not stop it — with nothing set the SDK falls back to its own default of http://localhost:4318 and trades the DNS failure for a refused connection at the same rate — so None sets OTEL_SDK_DISABLED=true rather than merely leaving the endpoint out.

None also silences the hermes_otel plugin, which carries agent spans. When OTEL_SDK_DISABLED=true is set (or HERMES_OTEL_ENABLED=false is configured), the container entrypoint disables hermes_otel (enabled: false) and clears its backends list, so no spans are created or exported to the fallback URL and the gateway logs [hermes-otel] ✗ Disabled via config (enabled=false) at start-up. Setting HERMES_OTEL_ENABLED=true in spec.deployment.env overrides this if spans should be exported regardless.

Nothing here overrides configuration: an operator who sets rung 1 or 2 gets that endpoint on a collector-less cluster. Setting OTEL_SDK_DISABLED=false in spec.deployment.env re-enables the SDK, but on its own it does not give the exporter anywhere to send — pair it with rung 1 or 2 if you want a working exporter rather than a localhost one.

Helm

Discovery only covers the agents. LiteLLM’s exporter is rendered by Helm, while its NetworkPolicy is managed by the operator on default installs (or rendered by Helm when either platformAgent.enabled or operator.enabled is false), so one chart value drives all three:

telemetry:
  otlpEndpoint: "http://otel-collector.otel-collector.svc.cluster.local:4318"
  collectorNamespace: "" # for an in-cluster collector whose host does not name its namespace

telemetry.otlpEndpoint	PlatformAgent CR	LiteLLM env (only when litellm.otel=true)	NetworkPolicy egress namespace
"" (default)	field omitted	managed collector	gke-managed-otel
set	spec.telemetry.otlpEndpoint	the value	derived from the host, or collectorNamespace; external endpoints: see below

Empty means “do not decide here”. When platformAgent.enabled: true and operator.enabled: true (the default), the operator dynamically manages the agent’s NetworkPolicy using runtime collector discovery, while LiteLLM’s exporter and NetworkPolicy default to the GKE Managed OpenTelemetry collector (gke-managed-otel). When either is false, Helm renders a static litellm-policy with the shipping default. Setting the value explicitly pins both the agent and LiteLLM together — a release can never have the agent discover collector A while LiteLLM exports to B and the policy opens egress only to B’s namespace.

litellm.otel stays a separate switch, and it defaults to off. Naming a collector does not turn the LiteLLM otel callback on, because that callback aborts every LLM request on DNS failure — too severe to flip as a side effect. So on a default install telemetry.otlpEndpoint moves the agents and the egress rule; the LiteLLM OTEL_EXPORTER_OTLP_ENDPOINT variable does not exist until you also set litellm.otel=true.

The namespace is read off the endpoint host when it names an in-cluster Service (<svc>.<ns> or <svc>.<ns>.svc…). A vendor endpoint or a bare hostname has no namespace to read, and with litellm.otel=true both renders — the operator’s on the default install, Helm’s static copy when operator.enabled or platformAgent.enabled is false — emit no OTLP egress rule for it (unless telemetry.collectorNamespace names one). The exporter then leaves over the policy’s port-443 rule, which excepts private ranges, so the endpoint has to be a public host on port 443: an external endpoint on any other port fails the render because the exporter would be blocked, and so do the shapes the render can recognise as in-cluster or private on 443 (an IP literal inside a range that render’s 443 rule excepts — the RFC 1918 blocks on both, plus CGNAT, link-local, and the unique-local, link-local and multicast IPv6 ranges on the operator’s — a single-label hostname, and on the static render any IPv6 literal, whose 443 rule is IPv4-only), and so does an endpoint the render cannot read a port off (a scheme other than lowercase http:// or https://, or userinfo, a query, or a fragment in the authority); a DNS name that happens to resolve to private address space is blocked without a render error. telemetry.collectorNamespace and the otlp-collector-namespace annotation have to be namespace names (lowercase RFC 1123 labels) while the check applies. The check stands aside wherever nothing selects LiteLLM: litellm.networkPolicy=false, or on the default install platformAgent.networkPolicy.enabled=false or the enable-litellm-network-policy: "false" annotation. With the callback off there is no LiteLLM export for a rule to serve: the operator emits none and the static copy keeps gke-managed-otel. telemetry.collectorNamespace is for an in-cluster collector whose host does not name its namespace (an IP literal, a bare Service name): set it and both renders open 4317/4318 to that namespace and skip the port-443 check.

Kustomize and examples

The kustomize LiteLLM base stays on the managed collector; to point it elsewhere, install through the chart and set telemetry.otlpEndpoint, which moves the exporter env and the egress namespaceSelector together. The vLLM manifests under examples/ still carry the managed endpoint literally — edit --otlp-traces-endpoint there if you redirect the rest.

Cloud Logging

Container stdout/stderr is ingested automatically by the GKE log agent. Pod, namespace, and cluster labels are attached; you can query per-pod in Logs Explorer.

Session metadata

Chat session context (space ID, user, thread) flows through Hermes as OTel span attributes. Trace lookup by session ID works out of the box. Full data flow: docs/designs/gchat-session-metadata-data-flow.md.

Console links

The persona (SOUL.md §5) surfaces direct Cloud Console URLs in Chat replies. Templates are documented on Concepts → Observability.

Non-GKE clusters

The current wiring assumes GKE Managed OTel and Prometheus. On other Kubernetes distributions:

• Deploy an OTel collector. You do not have to reconfigure the hermes_otel plugin by hand — see Pointing at your own collector; a collector at one of the well-known names is picked up automatically, and anything else is one chart value.
• Deploy Prometheus (kube-prometheus-stack works) and enable platformAgent.serviceMonitor: true or let the default null auto-detect the ServiceMonitor CRD. Alternatively, deploy VictoriaMetrics Operator and use platformAgent.vmPodScrape. The chart’s monitoring knobs cover the agent’s pods; for LiteLLM (port 8080), vLLM (port 8000), and Hindsight (port 8888), add scrape jobs or ServiceMonitors of your own. Set litellm.podMonitoring=false and hindsight.podMonitoring=false either way, since the PodMonitoring CRD is GKE’s and their templates do not auto-detect CRD presence. Important NetworkPolicy note: Operator-managed NetworkPolicies on the agent pods admit metrics ingress only from gke-gmp-system. On clusters enforcing NetworkPolicy, set platformAgent.monitoringScraperNamespace: <namespace> in chart values to render admitting NetworkPolicies for ports 9095, 9097, 8766, and 9096, or deploy an admitting NetworkPolicy manually. The gateway admitting policy includes rules for the agent API (8642/8643 from within the namespace), the GMP collector in gke-gmp-system (9095/9097), and operator poller metrics (9095/9097 assuming operator.enabled: true in the chart release namespace) alongside the scraper namespace, isolating inbound traffic on the gateway pod to this explicit peer set when operator-managed NetworkPolicies are disabled. Credential proxy (8766) and A2A gateway (9096) remain guarded. Additionally, set platformAgent.serviceMonitorLabels (e.g. release: <stack-release-name>) to match Prometheus serviceMonitorSelector (or configure serviceMonitorSelectorNilUsesHelmValues: false).
• Configure a log-forwarding agent (Fluent Bit, Vector) to your log backend.

The Hermes runtime and integrations are collector-agnostic; the shipping config is GKE-specific.