Live checks

The static rules judge a manifest on its own terms. Live checks judge it against reality: they fetch what the cluster has actually been doing and compare the proposed change to it.

This is the class of finding that no amount of YAML review catches, because the manifest is internally consistent — it is only wrong for this workload, in this cluster.

What they compare

Check	The question
Limit cut vs usage	You are lowering a limit. Is it below what this workload actually uses?
HPA capacity vs demand	Can the new max replicas meet the demand this workload has actually seen?
Network policy vs observed edges	Would this policy sever a call path that is genuinely in use?
Node selector vs matching nodes	Do any nodes actually match the selector you are narrowing to?
Replica drain vs load	Can the remaining replicas carry observed load during a drain?
Startup vs observed boot	Is the startup probe budget shorter than this workload’s real boot time?
Probe timeout vs latency	Is the probe timeout below observed endpoint latency?
Shutdown grace vs observed	Is the grace period shorter than this workload takes to shut down?
Grace period vs in-flight	Would termination cut off requests still in flight?
PDB blocks drain	Does the disruption budget leave a zero eviction budget?
Topology concentration	Would this land the workload on a single node?
Config reference vs inventory	Do the ConfigMaps and Secrets this manifest references actually exist?

Credentials

Live checks call the control plane, so they need:

EAC_API_URL       # control plane base URL
EAC_API_TOKEN     # API token
EAC_ORG_ID
EAC_CLUSTER_ID    # which cluster's observed behaviour to compare against
EAC_PROJECT_ID

Without them the gate still runs — it just runs the static rules only. It does not fail because it could not reach the control plane.

Roll each check out in three stages

Every live check follows the same environment-variable contract, and the middle setting is the one that makes adoption safe:

Variable	Effect
DEPLOYMENT_RISK_<CHECK>_ENABLED	Turn the check on at all.
DEPLOYMENT_RISK_<CHECK>_GATE_FAILING	Whether it can fail the gate, or only warn.
DEPLOYMENT_RISK_<CHECK>_MARGIN	Safety margin before it fires.
DEPLOYMENT_RISK_<CHECK>_WINDOW_HOURS	How far back the observation window reaches.
DEPLOYMENT_RISK_<CHECK>_TIMEOUT_SECONDS	How long to wait for the data.

So the sane rollout is:

1. Off — the default for most checks.
2. Enabled, not gate-failing — the finding appears in the PR comment, the build stays green. Run here until you trust the signal on your own traffic.
3. Enabled and gate-failing — now it blocks merges.

Tip

Skipping stage 2 is how a gate gets disabled entirely after its first false positive. Let each check prove itself on real PRs before you give it a veto.

Some checks take additional bounds — MIN_CALLS on the network-policy edge check, MIN_STARTS on the startup-boot check — so a conclusion is never drawn from one or two observations.

Timeouts fail open

Each live check has its own timeout. If the control plane is slow or unreachable, that check is skipped and the gate proceeds on what it has. A CI gate that blocks every merge because a telemetry query timed out would be removed from the pipeline within a week — so it does not do that.

The trade-off is explicit: a skipped live check is not a passed live check, and the report says which ones did not run.