Governance SOPs

Governance SOPs are the fleet-wide playbooks the Platform Agent executes on schedule (via cron watchdogs) or on request. They codify how the agent audits, remediates, and standardises clusters — separating the strategy from the tactics (skills).

The SOPs live in agents/platform/governance/.

The nine audit SOPs

Nine SOPs back the enabled fleet audits. They share one shape: enumerate the fleet, run read-only checks, write a validated findings file, and hand it to the fleet-audit skill, which owns the stream’s ledger issue and any remediation pull requests it spawns. Each check in each SOP states how it is read — its exact command, or the collector script that runs it — its flag-when predicate, an explicit do NOT flag list, a severity, an impact sentence, a recommendation, and a remediation kind — so a finding is either reproducible or it is dropped.

compliance_audit_sop.md

Security & RBAC posture, daily. Sixteen checks: privileged and SYS_ADMIN containers, host namespace sharing, hostPath mounts, cluster-admin and wildcard grants on bound roles only, namespaces with no enforcing NetworkPolicy, default ServiceAccount token automount, Workload Identity disabled, node pools exposing the legacy GCE metadata endpoint, public control planes with no authorized networks, Pod Security restricted gaps, stalled Config Connector objects, image references that name no specific bytes, tokens mounted for ServiceAccounts nothing has granted, LoadBalancer Services publishing a management port to the internet, and bindings whose subject is everyone.

Invoked by the compliance-audit watchdog.

obtainability_audit_sop.md

Workload reliability, daily — the question “which workloads break when I upgrade a node pool, and which ones cannot scale?” Twenty-three checks, over workload templates (not live Pods) and the Services, CronJobs, autoscalers and volumes around them: missing requests and memory limits, multi-replica workloads with no PodDisruptionBudget, drain-blocking and overlapping PDBs, unscaled and unscalable Deployments, autoscalers allowed to reach one replica, hostname and single-zone pinning, missing and unachieved spreading, missing readiness and liveness probes and liveness probes that fire first, single-replica Service-backed Deployments, rollouts and update strategies that drop traffic, preStop hooks the grace period cuts short, Services that select no pod or name a port no container declares, CronJobs that never succeed or pile up, and single-node volumes claimed from two nodes.

Invoked by the obtainability-audit watchdog. The cron id predates the rename.

security_patch_orchestrator_sop.md

Upgrade & patch readiness, weekly. Control-plane and node-pool versions compared against every version gcloud container get-server-config still offers at the cluster’s location, and against its release channel’s default, node skew against GKE’s two-minor ceiling, fleet-wide minor spread, clusters on no release channel, autoUpgrade/autoRepair off, missing maintenance windows, upgrade-blocking maintenance exclusions, deprecated node image variants, and absent upgrade notifications. For a cluster found behind, it also runs the fleet-upgrade-verification readiness grade and reports the upgrade as blocked when a drain-blocking PodDisruptionBudget would hold it — GKE holds each node’s drain for up to an hour and then evicts the workload anyway — or node-pool skew would stop it from completing.

The SOP forbids the words “vulnerable”, “unpatched”, and “CVE” in its findings: there is no vulnerability feed in this environment, so every finding is version currency or upgrade-policy hygiene — or, for the one check that asks how the upgrade would go, the held drain or the refused move above. Invoked by the security-patch-orchestrator watchdog.

fleet_wide_cost_analysis_sop.md

Fleet waste, weekly. Fourteen checks, run by a collector script the agent starts: over-requested, under-requested, unsized and idle workloads, judged against a week of Cloud Monitoring usage; orphaned PersistentVolumes, unconsumed PVCs, unattached Compute Engine disks, idle reserved IPs, orphaned load-balancer resources, under-allocated node pools, the scale-down blockers pinning them, terminal-pod accumulation, idle namespaces still holding billable objects, and Artifact Registry repositories with no cleanup policy.

Findings are reported in resource units — GiB, vCPU, node and object counts — never dollars. There is no billing export to price against, and the SOP treats a fabricated figure as worse than no figure. No remediation it emits may delete a PV, PVC, namespace, disk, snapshot, or address. Invoked by the fleet-wide-cost-analysis watchdog.

fleet_consistency_drift_sop.md

Fleet consistency drift, weekly. For each configuration facet — release channel, Shielded Nodes, logging and monitoring config, network policy, node auto-provisioning, Binary Authorization, label keys — it computes what the majority of comparable clusters do and reports the outliers, and it reports a cluster whose missing environment label keeps it out of every comparison.

The baseline is derived from the live fleet and nowhere else. That is what makes this one runnable where the retired blueprint-sync and standardization-validator SOPs were not: it needs no master blueprint, no CMDB, and no standards document. Invoked by the fleet-consistency-drift watchdog.

stockout_prevention_sop.md

Capacity obtainability & ComputeClass resilience, daily. Twelve checks over ComputeClasses, GCP reservations, workload affinities, and regional capacity: missing fallback machine families and dimension diversity, missing On-Demand floors for Spot priority lists, large-shape (>32 vCPU) obtainability risks, excessive priority rules causing autoscaler starvation, mixed disk generations on PV-attached ComputeClasses, Hyperdisk generation compatibility, regional quota saturation, Spot preemption and obtainability risks, single-zone standard node pools, GCP reservation bypasses or unallocated capacity mismatches, autoscaler out-of-resources visibility indicators, and dangling or invalid ComputeClass configurations.

Invoked by the stockout-prevention watchdog.

ai_security_audit_sop.md

AI workload security, daily — “who can reach my models, what can rewrite them, and where did their weights come from?” Six checks over the workloads a three-pronged discriminator identifies as AI workloads (a container image naming a known inference runtime, a container requesting an nvidia.com/gpu / google.com/tpu, or a container declaring a named model-provider credential such as OPENAI_API_KEY or HF_TOKEN): inference endpoints on external LoadBalancers, model repositories trusted to execute their own code (--trust-remote-code), model weights mounted writable by the serving process, model artifacts pulled from an unpinned source, model-registry credentials in plaintext environment variables, and model-server images on floating tags.

It deliberately does not evaluate the model. Prompt-injection resistance, jailbreak susceptibility, output filtering, and training-data provenance are real AI risks that no kubectl read can decide, and the SOP treats an unfalsifiable finding in a public issue as worse than no finding. It also stays off the generic container-hardening surface — privileged containers, host namespaces, RBAC, NetworkPolicy, and Workload Identity on AI workloads all belong to compliance_audit_sop.md, which already audits them there, so one object never carries two verdicts in two ledgers. Invoked by the ai-security-audit watchdog.

gcp_networking_fabric_sop.md

GCP Networking Fabric & VPC IPAM, daily. Checks VPC subnet IP exhaustion risks, Cloud NAT port exhaustion, PSC routing deadlocks, MTU mismatches across VPC peerings, Cloud Armor policy anomalies, and management ports open to the whole internet.

Invoked by the gcp-networking-fabric-audit watchdog.

gce_compute_fleet_sop.md

GCE Compute Engine and MIG fleet, daily. It checks instances, Managed Instance Groups, sole-tenant node groups and snapshots for startup-script failures in serial port output, MIGs that cannot converge on their target size, sole-tenant headroom exhaustion, and snapshots orphaned by a deleted source disk. Ops Agent guest telemetry is deliberately not audited by this stream.

Invoked by the gce-compute-fleet-audit watchdog.

The retired SOPs

The SOPs behind the five watchdogs retired from the roster — blueprint sync, policy propagation, global capacity orchestration, standardization validation, and lifecycle deprecation — are no longer in the tree or the image; git history has them. As written, each depended on an input a stock install does not provide — a master blueprint, a /opt/defaults/templates/ directory, a corporate patterns document — or duplicated an audit above, so reviving one means writing a new SOP against what an install actually has, not restoring the old file.

SOPs that are not fleet audits

inventory.md is not a fleet audit. It is the first-boot environment discovery procedure behind first-run onboarding, which lists the fleet once and audits the clusters no Cluster Agent covers; the onboarding gate files the per-cluster audits itself. Its companions run as separate cards on the same one-shot path: cluster_inventory_audit_sop.md is what each Cluster Agent follows on the one cluster it is pinned to, returning structured metadata that the onboarding gate’s hand-off script merges into the raw file, and inventory_prioritize_sop.md scores the merged findings against a fixed rubric, registers all of them in the findings queue, and renders the short /opt/data/INVENTORY.md that reaches the user from the top of that order.

The k8s-event-watcher daily recap has no SOP here: it is a no_agent script, eod_report_generator.py, that renders the recap deterministically from the session ledger and prompts no model, so its description is a design note rather than agent material — docs/designs/eod-event-watcher-daily-report.md.

How SOPs work

Each SOP is a Markdown file that opens with a **Purpose:** line and a **Data sources:** line naming exactly what the run may read, followed by a single ## Execution Checklist broken into numbered steps (loose convention, not enforced). The audit SOPs additionally close with a ## Red Lines section — the things the run must never do, stated as prohibitions rather than guidance.

The cron watchdog invokes the SOP by prompting the agent to read governance/<sop>.md relative to its profile home and execute it. profile_scaffold.py overlays the baked /opt/platform-template/governance/ directory there at container start; there is no /opt/defaults/governance/, so an absolute path of that shape does not resolve.

SOPs vs. skills

• A skill is a reusable capability (how to onboard an app, how to submit a PR, how to open and close an audit run).
• An SOP composes skills into a fleet-wide procedure with a policy for when to act.

The division of labour in the audit streams is deliberate: the SOP decides what is true, the skill decides what happens to it. The model reasons, runs read-only commands, and emits evidence; fleet-audit’s helper owns every forge call, through the version-control broker, and renders every body itself — the stream’s ledger issue and the remediation PRs promoted from it. The SOPs forbid hand-writing any of those bodies or invoking git directly, which is what keeps the nine ledgers uniform and their run-to-run deltas computable.

The audit jobs preload the skill through their cron entry ("skills": ["fleet-audit"]). An SOP that needs no preloaded skill can omit the key or leave it empty — the run loads what it needs.

Where to go next

• Autonomous watchdogs — the schedules that invoke SOPs.
• Skill catalog — the capabilities SOPs compose.
• Declarative workflow — how SOP-generated remediations become PRs.