Traefik runtime

The Ingress view shows the routes you declared. With Traefik runtime on, Runtimez also reads what Traefik actually built and served:

• Traefik’s own status for every router — enabled, warning or disabled, with the error Traefik reports for it.
• Request counts by status code for each route, from Traefik’s Prometheus metrics.

Root-cause analysis and Ask Runtimez use both. When a workload behind Traefik fails, the analysis can show that Traefik answered its requests with 503s because the backend had no ready pods — or rule the edge out because Traefik served the route normally.

It stays read-only. The agent reads Traefik’s router API with one GET per Traefik pod per sync and scrapes the metrics endpoint Traefik already serves.

Before you start

• Traefik runs from its official Helm chart, or at least in a container named traefik — the chart’s default.
• A cluster may run several Traefik controllers; each one needs its API exposed as in step 3. See Several Traefik controllers.

1. Turn on Traefik in Runtimez

Open Organization → Integrations and switch on the Traefik card under Add-ons. Changing it needs edit rights on the organization.

2. Turn it on in the agent

Add both settings to the values file you install the agent with:

discovery:
  traefik:
    enabled: true      # scrape Traefik's request metrics
    runtime:
      enabled: true    # read Traefik's router table through the API server

Then upgrade the agent:

helm repo update
helm upgrade runtimez-agent runtimez/agent --namespace runtimez -f values.yaml

Pass your values file, not –reuse-values

--reuse-values replays the values of your previous install and skips any default a newer chart adds.

runtime.enabled adds one read-only permission to the agent’s ClusterRole: get on pods/proxy. See RBAC & security model for what that grants.

The agent’s collector pods restart on every helm upgrade, so they pick up the new metrics settings. If a GitOps tool such as Argo CD renders the chart with helm template, restart them yourself:

kubectl -n runtimez rollout restart daemonset/runtimez-agent-otelcol-agent

3. Expose Traefik’s API inside the cluster

Traefik serves its router table at /api/http/routers on its internal traefik entrypoint — but only when something routes to it, and recent Traefik charts leave that off. Turn on the chart’s dashboard route:

# Traefik Helm chart values
ingressRoute:
  dashboard:
    enabled: true

This creates an IngressRoute on the traefik entrypoint that sends /api and /dashboard to Traefik’s internal API. The chart does not put that entrypoint on the Traefik Service, so the API stays unreachable from outside the cluster. Keep it that way: do not attach the route to web or websecure.

• Traefik installed without its chart: start it with --api.insecure=true. Despite the name, that serves the API on the same internal traefik entrypoint. Runtimez also needs to find the pod:
  • the agent looks for the label app.kubernetes.io/name: traefik, or an image whose repository name ends in traefik — traefik, library/traefik and k3s’s rancher/mirrored-library-traefik all count;
  • the collector scrapes a container named traefik whose pod carries the prometheus.io/scrape: "true", prometheus.io/path and prometheus.io/port annotations for the metrics entrypoint.
• k3s: put the chart values in the traefik HelmChartConfig in the kube-system namespace.

Check it with the same request the agent makes. The traefik entrypoint listens on 8080 by default and on 9000 in older charts; the agent reads the port from the pod, and so can you:

kubectl get pods -A -l app.kubernetes.io/name=traefik
kubectl -n <traefik-namespace> get pod <traefik-pod> -o jsonpath='{.spec.containers[0].args}' | tr ',' '\n' | grep -i 'entrypoints.traefik.address'
kubectl get --raw "/api/v1/namespaces/<traefik-namespace>/pods/<traefik-pod>:<traefik-port>/proxy/api/http/routers" | head -c 400

No entrypoints.traefik.address argument means the default, 8080.

A JSON list of routers means it works. A 404 means nothing routes to the API yet.

Optional: richer metrics

Setting	Where	What it adds
--metrics.prometheus.addRoutersLabels=true	Traefik, in additionalArguments	Request counts per router, not only per service.
discovery.traefik.telemetry.metrics.histograms: true	Agent values	Request-duration histograms. About 60 series per route, so they are off by default.

When something is missing

Runtimez does not fill a gap with a guess. If Traefik’s API is not exposed, the analysis says so and treats each router’s status as unknown. If no Traefik metrics arrived for the window, it says the telemetry was not read rather than reading silence as zero traffic.

Traefik’s allowEmptyServices

This Traefik option decides what happens when a route’s backend has no ready pods:

• true — Traefik keeps the route and answers 503.
• false, Traefik’s default — Traefik drops the route and answers 404.

Runtimez reads the setting from the Traefik pod’s arguments and explains the failure either way. You do not need to change it.

Troubleshooting

No Traefik metrics, and you use the Prometheus Operator. Traefik’s chart removes its prometheus.io/* pod annotations when metrics.prometheus.serviceMonitor.enabled is true, and the Runtimez collector finds Traefik through those annotations. Add them back in the Traefik chart values:

deployment:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9100"   # your metrics entrypoint's port

No Traefik metrics, and Traefik runs on tainted nodes. The collector scrapes Traefik from the same node, and its DaemonSet has no tolerations by default. Add yours to the agent values:

otelcol:
  tolerations:
    - key: dedicated
      operator: Equal
      value: ingress
      effect: NoSchedule

The router read times out. The request travels from the Kubernetes API server to the Traefik pod on the traefik entrypoint’s port.

• Private GKE clusters allow only ports 443 and 10250 from the control plane by default. Add a firewall rule for the Traefik API port.
• On EKS, if your nodes use their own security group, it must accept that port from the cluster security group.
• A default-deny NetworkPolicy in Traefik’s namespace must allow it too.

Still 404 after turning on the dashboard route. List the route by its full resource name:

kubectl get ingressroutes.traefik.io -n <traefik-namespace>

On clusters that still carry Traefik v2’s CRDs, the short name ingressroute resolves to the older traefik.containo.us group and can hide the new route.

Several Traefik controllers

A cluster can run more than one Traefik — an internal and a public one, say, or two versions during a migration. Runtimez reads each controller on its own: its router table, its entrypoints and providers, and its request counts.

A route is matched to the controllers whose configuration admits it: the provider its kind comes from (kubernetescrd for an IngressRoute, kubernetes for an Ingress), its ingress class, the namespaces the controller watches and the entrypoints it names. Each of those controllers’ view is shown on the route, and the route’s state is the worst of them: a route one controller serves and another rejects shows as unserved, with the rejecting controller named. Request and error counts are summed across the controllers that carry the route. Root-cause analysis rules the edge out only when no controller reports the route down, and blames Traefik only when every controller that admits the route does.

When Runtimez cannot tell whether a controller admits a route — its arguments cannot be read from the pod spec, providers set in a configuration file, say, or its providers select objects by label — it says so and leaves that route’s verdict open instead of trusting the other controller’s table.

The Risk tab’s Traefik rules run against one controller, the primary: the one on the lowest Traefik version, then the one created first.

Limitations

• Up to eight Traefik controllers per cluster. Further controllers are not read.