gRPC trace context is not properly propagated to the sidecar #1857

Activity

1. mtaghavi2005

   added

   kind/bugSomething isn't workingSomething isn't working

   on Jun 23, 2026
2. WhitWaldo commented on Jun 24, 2026

   

   WhitWaldo

   on Jun 24, 2026

   Contributor

   More actions

   With the exception of Workflows, the Dapr .NET is a library that wraps the functionality provided by the Dapr runtime via HTTP and gRPC and not something that does anything that needs to be itself instrumented.

   If you really want to track the network hops, I suggest adding their autoinstrumenter for the HTTP client to your own project to automatically capture those traces.

   But absent Workflows where there's a distinct state machine managed in the SDK and the runtime is merely the orchestrator of it (and you already instrumented that last release) which warrant its own telemetry capture, or within the Dapr Agents or other agentic frameworks atop Dapr (which themselves includes relevant spans), this isn't necessary across the Dapr Client and I'm reluctant to add it here.
3. mtaghavi2005 commented on Jun 25, 2026

   

   mtaghavi2005

   on Jun 25, 2026

   ContributorAuthor

   More actions

   Thanks for the clarification. I completely agree that the Dapr .NET SDK should not create additional spans for its normal client operations. That isn't what I'm proposing.

   The issue I'm trying to address is context propagation, not SDK instrumentation.

   The expected trace hierarchy should look something like this:

   Application operation
       └── gRPC call to Dapr sidecar
               └── Dapr sidecar operation
                       └── Infrastructure / external service
   

   However, what I currently observe is:

   Application operation
       └── gRPC call to Dapr sidecar
   
   Dapr sidecar operation
       └── Infrastructure / external service
   

   Although all spans belong to the same trace, the sidecar-created span is not correlated as a child of the application operation that initiated the Dapr call.

   To me this suggests one of two possibilities:

   1. The trace context is not being propagated to the runtime in the format the runtime expects.
   2. The runtime receives the propagated context but cannot reconstruct the parent span correctly.

   According to the Dapr documentation, the primary propagation mechanism for gRPC is grpc-trace-bin:

   https://docs.dapr.io/operations/observability/tracing/w3c-tracing-overview/

   I also checked the Java SDK, which explicitly propagates the binary gRPC trace context, consistent with the documentation.

   also experimented with [OpenTelemetry.Instrumentation.GrpcNetClient](https://www.nuget.org/packages/opentelemetry.instrumentation.grpcnetclient) (currently still in beta). When it is enabled, the trace hierarchy becomes correct.

   The instrumentation documentation explains that when downstream HttpClient instrumentation is suppressed, the gRPC instrumentation becomes responsible for propagating the ActivityContext and Baggage:

   https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Instrumentation.GrpcNetClient/README.md#suppressdownstreaminstrumentation

   While I haven't yet verified exactly which gRPC metadata is being written by the instrumentation, the fact that enabling it results in the expected parent-child relationship suggests that the missing piece is related to context propagation rather than additional SDK spans.

   Originally, I assumed this should be handled by the .NET SDK, which is why I opened this PR. However, while investigating further, I found that the Dapr runtime already attempts to support both propagation mechanisms: it first looks for grpc-trace-bin, and if that is not present, it falls back to extracting the context from the W3C traceparent header.

   This fallback was originally introduced because grpc-dotnet propagated traceparent instead of grpc-trace-bin:

   • Add traceparent header in gRPC metadata for grpc-dotnet client dapr#1824
   • Allow to pass W3C trace header or grpc-trace-bin to dapr api call #300

   The comment for the related code is still here:
   https://github.com/dapr/dapr/blob/5b2ea177b3c7200dea233cbf884436ec40cf968c/pkg/diagnostics/grpc_tracing.go#L309

   Because of that, I'm no longer convinced the SDK is necessarily the right place for the fix. It is possible that the runtime is not reconstructing the parent span correctly from the traceparent fallback. I'm currently investigating that path before drawing any conclusions.

   My PR was intended to demonstrate one possible solution, but my actual goal is simply to ensure that the Dapr sidecar spans are correctly parented under the application operation, regardless of whether the solution ultimately belongs in the SDK or in the runtime.

   If I determine that the issue is in the runtime rather than in the SDK, I'll close this issue and open an issue against the runtime instead.
4. WhitWaldo commented on Jun 25, 2026

   

   WhitWaldo

   on Jun 25, 2026

   Contributor

   More actions

   We had some issues with trace propagation at the runtime a few releases ago and it's possible that the fix at the time changed this behavior, but I'm not immediately able to find the PR for this that I'm thinking of.

   Please do let me know what your investigation yields - I'll bring this thread up on our maintainer call next week and see if anyone on the runtime team has any thoughts on it.
5. mtaghavi2005 commented on Jun 27, 2026

   

   mtaghavi2005

   on Jun 27, 2026

   ContributorAuthor

   More actions

   I investigated this further based on your suggestion and wanted to share the findings.

   I added some temporary logging to the Dapr runtime gRPC tracing path and confirmed that the runtime can correctly extract the parent context from the incoming traceparent. So this does not appear to be a Dapr runtime parsing issue.

   What I observed is that, for the same Dapr gRPC call, the application sends a valid traceparent (correct trace ID), but its parent span ID corresponds to the Grpc.Net.Client transport activity rather than the logical application operation.

   Since traceparent is propagated by Grpc.Net.Client, I looked into the grpc-dotnet implementation to understand why the propagated parent corresponds to the transport activity rather than the application activity. During that investigation, I found what appears to be a related issue:

   grpc/grpc-dotnet#2516

   That discussion points out that Grpc.Net.Client creates a transport Activity when Activity.Current != null, even if there is no ActivitySource listener.

   To validate this, I tested a locally patched version of grpc-dotnet that disables creating the transport Activity solely because Activity.Current != null. With that change, without any modifications to the Dapr .NET SDK and without propagating grpc-trace-bin, the trace hierarchy became correct.

   At this point, it appears more likely that the underlying issue originates in grpc-dotnet.

   However, if the Dapr project prefers not to rely on the current grpc-dotnet behavior, the proposed PR still provides an alternative by propagating Dapr's documented gRPC trace context (grpc-trace-bin), preserving the logical application context for the runtime.
6. WhitWaldo commented on Jun 28, 2026

   

   WhitWaldo

   on Jun 28, 2026

   Contributor

   More actions

   Excellent research - I appreciate you taking the time to work through it. Let me ask you this - if Grpc.Net.Client is patched to fix the issue and I've accepted the PR you proposed to patch the Dapr .NET SDK (e.g. both patches are applied), will it result in the activity being improperly set or have no effect?

   If the latter, I think that makes the case for merging your PR as-is. But if the former, I think you or I should simply submit a PR in their repo to fix the issue upstream and update to the new version when available.

   👍React with 👍1Reacted by Mohammad Taghavi
7. mtaghavi2005 commented on Jun 28, 2026

   

   mtaghavi2005

   on Jun 28, 2026

   Last edited by mtaghavi2005

   ContributorAuthor

   More actions

   Thanks, that's a good question.

   My expectation is that if Grpc.Net.Client is fixed upstream and this SDK PR is also applied, the SDK change should have no negative effect. The SDK change doesn't create or replace any Activity; it only propagates additional context derived from the current logical activity. So with both fixes applied, both traceparent and grpc-trace-bin would represent the same logical parent, and the runtime would simply continue preferring grpc-trace-bin.

   That said, after tracing this through the runtime and Grpc.Net.Client, I actually agree with your suggestion that the proper fix should be in grpc-dotnet rather than relying on a Dapr-specific workaround.

   I'll put together a PR in the grpc-dotnet repository based on these findings and see what the maintainers think. If the issue can be resolved upstream, I think that would be the ideal outcome. In that case, we could simply remove the grpc-trace-bin propagation from the Dapr .NET SDK later, and everything should continue to work as it does today, but with the correct parent-child relationship.

   Update:

   PR in the grpc-dotnet repository created:
   grpc/grpc-dotnet#2744

   🎉React with 🎉1Reacted by Whit Waldo
8. WhitWaldo

   added this to the v1.18.x - SDK Patches milestone on Jul 23, 2026
9. WhitWaldo

   closed this as completedin #1858on Jul 24, 2026