Sampling

The decision is made once, at the root

A trace is kept or dropped whole. The decision is taken when the root span starts and inherited by every child, so a run is never half recorded.

This is the rule everything else follows from. A trace with a random 30% of its spans missing does not read as a sampled trace. It reads as a bug in the system you are observing: a tool call that apparently never happened, a model call with no parent, a gap where the retrieval should be. You will spend an afternoon on it before you work out that the gap is your observability tool and not your agent.

The default keeps everything

With no sampling configured the policy is always, and that is the right default until volume says otherwise. Sampling trades away the run you will want to read. Pay that only when the storage or the bandwidth makes you.

The five strategies

Strategy	Behaviour
always	Keep everything. The default.
never	Keep nothing.
ratio	Keep a fixed fraction, drawn at random once per trace at the root.
deterministic	Keep a fixed fraction, decided by hashing the trace id. Every collector reaches the same answer without talking to any other.
tail	Defer the decision until the trace finishes, then keep every error and every slow run whatever the rate says.

Setting a rate

--sample, or MCPD_SAMPLE, takes a fraction from 0.0 to 1.0 and configures the deterministic strategy. A value outside that range stops the daemon at startup with a message naming what you passed. It does not clamp, because a mistyped rate that quietly becomes 1.0 is discovered a month later by the storage bill.

Every SDK reads the same fraction, as sample in the client configuration or MCPD_SAMPLE in the environment. Sampling in the client is cheaper: a span that was not sampled is never queued and never exported, so it costs no bandwidth at all. Sampling in the daemon costs you the round trip but keeps the rate in one place, which is what you want when a dozen services report into one instance.

Of the five strategies, the daemon reaches always and deterministic: --sample and a project's sample in the config file both set the deterministic strategy. never, ratio and tail are implemented in the policy type and are not reachable from a flag or the config file in this build.

Why hashing lets a fleet agree

Run several mcpd instances behind a load balancer and the spans of one trace do not all arrive at the same one. A sampler that flips a coin per instance, or per span, would keep some of a trace and drop the rest, which is the failure the first rule exists to prevent.

The deterministic strategy maps the trace id onto the full 64-bit range and keeps the trace when that value falls below the threshold the rate implies. Every instance computes the same number from the same id, so they all agree, with no shared state, no coordination protocol and nothing to fall out of sync. The same id also gives the same answer on a retry, so a replayed batch cannot half arrive.

Two details make that mapping honest. It reads the trailing sixteen hex characters of the id: OpenTelemetry requires trace ids to be random, but some producers derive the leading bytes from a timestamp, and a sampler reading those would correlate with the time of day rather than sample. When the id is not hex at all, it falls back to a real hash with a finalising mix instead of degenerating into keeping everything, so the realised rate stays close to the rate you asked for.

An upstream decision wins in the SDKs

When an SDK continues a trace from an inbound W3C traceparent, it takes the sampled flag with the ids and does not apply its own rate. This holds both ways: the flag set keeps the trace whatever the local rate, and the flag clear drops it whatever the local rate.

The daemon does not do this in this build. Its OTLP door keeps the trace-flags byte on each span, in the attribute mcpd.otlp.flags, but the daemon's sampler does not read it. A daemon with a rate applies that rate to every trace, whatever the flag says.

The alternative is worse than it sounds. If each service applied its own rate, a trace crossing four of them would be kept by two and dropped by two, and what you would have at the end is not a distributed trace. It is a set of holes in the shape of one.

Tail sampling keeps the runs worth reading

A head sampler decides before it knows anything. It is cheap, and it is blind: at a tenth, it keeps a tenth of the failures too. The traces you actually open are the rare ones, and rarity is exactly what a ratio sampler is worst at.

900

650

400

150

That is what a latency recording looks like: a flat baseline and a handful of excursions. The baseline is where the volume is and the excursions are where the answers are. Tail sampling holds the trace until it ends, then keeps it if anything in it errored, keeps it if it ran longer than the threshold you set, and otherwise applies the same deterministic hash as before. Keeping errors is on by default, because the traces you most need are the rare ones and a ratio sampler is most likely to throw them away.

The cost is buffering. A tail sampler must hold every unfinished trace in memory until it can decide, which is why it is a deliberate choice and not the default. As above, the tail strategy is implemented in the policy type and is not reachable from a flag or the config file in this build; the daemon's ingest path applies the head decision.

Knowing what was dropped

Every span the sampler discards increments mcpd_traces_sampled_out_total, exported at /metrics in Prometheus text format. Write the rate down somewhere your team reads. A missing trace with a documented sample rate is an explanation; a missing trace with no explanation is a support ticket, and the person filing it is usually you.

Next

• The other way to store less: Redaction and retention.
• Every flag and environment variable: Configuration.
• Reading /metrics and the rest of the daemon's own signals: Running it.