Dapr 1.9.6

Fixes Azure Service Bus components failing to recover from certain failures on publishing

Problem

In certain situations, often related to transient service failures, the Azure Service Bus components (bindings.azure.servicebusqueues and pubsub.azure.servicebus) could enter into a "bad" state and users would not be able to publish messages anymore, with all operations failing with a "context deadline exceeded" error. Dapr was not able to recover from the failures automatically and required a restart of the daprd process.

Impact

This impacts Dapr users who publish messages to Azure Service Bus using one of the two supported components: bindings.azure.servicebusqueues and pubsub.azure.servicebus.

Impacted Dapr versions include Dapr 1.9.0-1.9.5.

Root cause

In some cases, which we believe are in consequence of transient service failures, the connection between Dapr and Azure Service Bus that is used to publish messages could enter into a "bad" state. Any attempt at publishing a message would then fail with a "context deadline exceeded" error, with Dapr unable to recover automatically.

Dapr 1.9 already included some checks to attempt to recover from those failures, by forcing the connection with Azure Service Bus to be re-established. However, our checks were not catching all failures reported by the service that required reconnecting.

Solution

We have updated the Azure Service Bus components to catch more instances of errors that require re-creating the connection used to publish messages to Azure Service Bus.

We have also added more logs that will help operators and Dapr developers to be able to diagnose issues more effectively in the future.