Dapr 1.9.3

Fixes traces not being reported in certain circumstances

Problem

With tracing enabled, Dapr operators can choose the sampling rate using the spec.tracing.sampling option in the Dapr configuration. In Dapr 1.9.0-1.9.2, when a request coming to the Dapr runtime contains a traceparent header, the decision on whether to sample the request or not is solely based on the "sampling bit" set in the header's value.

This behavior was introduced with Dapr 1.9.0 during the transition to the new tracing framework based on the OpenTelemetry (OTEL) standard. Although this behavior is compliant with the W3C specs for distributed tracing, it caused Dapr to omit sending traces to the telemetry collector (e.g. Zipkin, Azure Monitor, etc) in many instances.

Impact

The issue impacts users on Dapr 1.9.0-1.9.2 who have enabled collection of traces.

Based on reports from Dapr users, developers building apps with ASP.NET Core seem to be particularly impacted, as the .NET Core framework can automatically include a traceparent header in every request made to the Dapr sidecar which has the "sampling bit" set to 0 (disabled).

Root cause

Older versions of Dapr (before 1.9.0) ignored the "sampling bit" in the traceparent header when making decisions on whether to sample a request. That behavior changed in Dapr 1.9.0, where the decision made by the caller with the traceparent header determines the Dapr runtime's sampling choice too.

Solution

We have patched the way Dapr makes decisions on whether to sample a request and submit the trace to the telemetry collector (e.g. Zipkin, Azure Monitor, etc). The new behavior consists of:

• If the traceparent header has the "sampling bit" set to 1 (enabled), the request is always sampled.
• If the "sampling bit" is 0 (disabled), Dapr decides on whether to sample the request based on its own internal policies. For example, with spec.tracing.sampling set to 1, all requests are traced; instead, a value of 0 generates no trace. Numbers in-between 0 and 1 cause Dapr to sample only a fraction of requests.

This behavior, while more lax, remains compliant with the W3C specs for distributed tracing.