Trace sampling

Trace volume is the fastest-growing line on an observability bill, and most of it is healthy traffic that nobody will ever open. Sampling decides what you keep. This tab makes that decision explicit, testable, and reversible.

Tail, not head

The decision is made after the trace completes, not when the first span arrives. That is what makes it safe to drop aggressively: errors and slow traces can be kept while the healthy bulk of the same endpoint is dropped.

Head sampling cannot do this — it decides before it knows how the request went, so keeping 5% means losing 95% of your errors too.

The five settings

Setting	What it does
Monthly GB	Your trace budget. Deliberately unset by default — an invented number would silently drive allocation and show up as a figure you never chose.
Latency threshold	Traces slower than this are kept regardless of rate. Defaults to 1000 ms.
Deploy boost	Temporarily raises retention around a deploy, when you most need the detail.
Per-signature thresholds	Each operation gets its own latency bar, instead of one global number that is wrong for both your 5 ms cache lookup and your 2 s report.
Neighbour retention	Keeps the traces adjacent to a kept one, so a retained error has context rather than sitting alone.

Deploy boost, per-signature thresholds and neighbour retention are on by default.

What is always kept

Independent of rules and rates, a trace survives if it is an error or exceeds the latency bar — plus its neighbours when that setting is on. The budget shapes the healthy remainder, not your incidents.

The keep-reason breakdown shows exactly which mechanism retained each slice of traffic, so you can see whether your budget is going to errors, latency, or baseline sampling.

Shadow mode — decide before you enforce

A policy is in one of two modes:

Mode	Effect
Shadow	The policy is evaluated and reported, but nothing is dropped.
Enforced	The policy is applied.

Shadow mode is the whole point of the workflow. It reports what would have happened on your real traffic:

• Received spans, decided traces and spans, kept traces and spans
• Projected drop percent
• Wire bytes in vs projected and stored bytes in — the actual saving, in bytes
• Keep reasons — which rule retained what, and what share of traffic each accounts for

There is also a reconciliation block, because tail sampling has edge cases and hiding them would be dishonest: late spans forwarded, late spans shed after the decision, malformed spans, spans dropped by truncation, and traces shed under pressure.

Run in shadow, read the projection, then enforce.

Suggestions

Rather than asking you to guess, Runtimez derives suggestions from your observed traffic — the high-volume, low-value paths worth dropping first. Each is suggested, then approved or dismissed by a named person, with the decision recorded.

Creating a rule

Rules come in two kinds.

Rate rules

Keep a percentage of matching traces.

• Name — what this rule is for.
• Match — key/value pairs against trace attributes, with a match type.
• Sample percent — the share to keep.

Rules are ordered, and the order is editable by drag. The first match wins, so put narrow rules above broad ones.

Ignore rules

Match traffic that should never be considered at all — health checks, readiness probes, synthetic monitors. Same shape: name, match, match type.

Preview before you save

Every rule can be previewed against real traffic before it is saved. Use it — the preview is the difference between “this looks like a health check” and “this matches 12% of production”.

Rules match the ROOT span

A drop rule is evaluated against the root span of the trace. A rule written against an attribute that only appears on a child span will not match, and the trace stays. If a rule’s preview count is zero when you expected thousands, this is almost always why.

The budget breakdown

Shows the target throughput and where your allowance is actually going. Your monthlyGb is capped by your plan’s allowance — the cap is commercial, and the tab shows both numbers so you can tell “I set this low” from “my plan stops here”.

A worked rollout

1. Set Monthly GB to your intended budget and leave the latency threshold at 1000 ms.
2. Put the policy in Shadow.
3. Let it run across a representative window — include a deploy and a busy period.
4. Read projected drop percent and wire bytes projected. If the projection does not save enough, add rate rules; if it drops something you need, check the keep reasons.
5. Review suggestions and approve the obvious health-check and probe paths as ignore rules.
6. Preview each new rule before saving it.
7. Switch to Enforced.
8. Come back after a week and compare against Cardinality & cost rules, where the realised saving is attributed per rule.

What this does not do

Sampling reduces what is stored, not what is produced. Your services still emit every span and the agent still receives them — the saving is in the pipeline and the store. If you need to cut emission itself, that is an instrumentation change, not a sampling one.