RabbitMQ - v3.12.0-rc.4

RabbitMQ 3.12.0-rc.4

RabbitMQ 3.12.0-rc.4 is a candidate of a new feature release.

Highlights

This release includes several new features, optimizations, and graduates (makes mandatory) a number of feature flags.

The user-facing areas that have seen the biggest improvements in this release are

• Optimizations for both quorum and classic queues: improved throughput, lower throughput variability, lower latency, lower memory footprint
• More mature and efficient implementation of (non-mirrored) classic queues v2 (CQv2)
• Classic queue lazy and non-lazy modes no longer apply: classic queues v2 always behave very similarly
  to the lazy mode in earlier release series: moving data to disk aggressively and only keeping a subset of data in memory
• Significantly reduced MQTT and Web MQTT memory footprint per connection
• OAuth 2, OIDC, IDP support
• Even more configurability of the OAuth 2 plugin

This release also features many internal API improvements in preparation to 4.0
with Khepri.

See Compatibility Notes below to learn about breaking or potentially breaking changes in this release.

Release Artifacts

RabbitMQ releases are distributed via GitHub.
Debian and RPM packages are available via Cloudsmith mirrors, as well as PackageCloud.

Community Docker image, Chocolatey package, and the Homebrew formula
are other installation options. They are updated with a delay (usually a few days).

Erlang/OTP Compatibility Notes

This release requires Erlang 25.0 or later.
This introduces feature parity for x86- and ARM64-based CPUs: Erlang 25 offers the JIT and
modern Flame Graph profiling tooling
for both of those major CPU architectures.

Provisioning Latest Erlang Releases explains
what package repositories and tools can be used to provision latest patch versions of Erlang 25.x.

Erlang 26

Erlang 26 introduces a number of breaking changes around networking and TLS. At the moment,
RabbitMQ should not be considered fully compatible with Erlang 26. Erlang 26 compatibility
will be announced separately.

Upgrading to 3.12

Documentation guides on upgrades

See the Upgrading guide for documentation on upgrades and RabbitMQ change log
for release notes of other releases.

Required Feature Flags

RabbitMQ 3.12.0 will require all feature flags from the 3.11.x release series to be enabled before upgrading,
similarly to how 3.11.0 required all feature flags introduced before 3.9.0.

If the feature flags are not enabled, RabbitMQ 3.12 and later nodes will refuse to start.

Mixed version cluster compatibility

RabbitMQ 3.12.0 nodes can run alongside 3.11.x nodes. 3.12.x-specific features can only be made available when all nodes in the cluster
upgrade to 3.12.0 or any other patch release in the new series.

While operating in mixed version mode, some aspects of the system may not behave as expected. The list of known behavior changes is covered below.
Once all nodes are upgraded to 3.12.0, these irregularities will go away.

Mixed version clusters are a mechanism that allows rolling upgrade and are not meant to be run for extended
periods of time (no more than a few hours).

Compatibility Notes

More Feature Flags Gratuate to Core Features ("Always Enabled")

RabbitMQ 3.12.0 will require all feature flags from the 3.11.x release series to be enabled before upgrading.

If the feature flags are not enabled, RabbitMQ 3.12 and later nodes will refuse to start.

Minimum Supported Erlang Version

Starting with this release, RabbitMQ requires Erlang 25.0 or later versions. Nodes will fail to start
on older Erlang releases.

Erlang 25 as our new baseline means much improved performance on ARM64 architectures, profiling with flame graphs
across all architectures, and the most recent TLS 1.3 implementation available to all RabbitMQ 3.11 users.

Client Library Compatibility

Client libraries that were compatible with RabbitMQ 3.11.x will be compatible with 3.12.0.

Getting Help

Any questions about this release, upgrades or RabbitMQ in general are welcome on the RabbitMQ mailing list.

Changes Worth Mentioning

Release notes are kept under rabbitmq-server/release-notes.

Core Server

Enhancements

• Quorum queues now sustain higher throughput with large backlogs. They also offer higher throughput
  on average, and in particular when Single Active Consumer is used.

GitHub issue: #7553

• Reduced memory footprint, improved memory use predictability and throughput of classic queues.
  This particularly benefits classic queues with longer backlogs.

GitHub issues: #4522, #7516

• Reduced peak memory footprint of quorum queues.

GitHub issue: #7175

• Improved stream leader distribution efficiency. Mostly relevant to environments with lots of
  streams.

GitHub issue: #6440

• All feature flags introduced before 3.11.0 are now required to be enabled. Make sure
  all feature flags are enabled before upgrading to 3.12.0.

GitHub issue: #7219

• Plugin and core API: all schema data store operations are now available via the rabbit_db_* modules.
  Plugin maintainers should switch to them for an easier transition to a future 4.0 release.

GitHub issues: #6430, #6821

• Plugin and core API: node data directory now should be accessed using rabbit:data_dir/0.

GitHub issue: #6462

• Initial (incomplete) Erlang 26 compatibility

GitHub issues: #7443, #7900,
#7921, #8057

• There is now a way to pre-configure users and their permissions for newly created virtual hosts:

ini default_users.qa_user.vhost_pattern = qa.* default_users.qa_user.tags = policymaker,monitoring default_users.qa_user.password = fd237824441a78cd922410af4b83f0888186a8d7 default_users.qa_user.read = .* default_users.qa_user.write = .* default_users.qa_user.configure = .*
This is primarily useful in environments where RabbitMQ is provided as a service but
customers (clients) have control over virtual hosts.

Contributed by @illotum (AWS).

GitHub issue: #7208.

• Policies can now be defined to only apply to specific queue types.

For example, it is now possible to have two policies matching all queue names ('.*')
but one applicable only to one queue type, and the other only to different queue type,
without conflicts.

For example, this policy will only apply to quorum queues and not streams or classic queues:

bash rabbitmqctl set_policy at-least-once-dead-lettering ".*" '{"dead-letter-strategy": "at-least-once"}' --apply-to quorum_queues

• Nodes with a lot of classic queues now shut down up to 10 times faster.

GitHub issue: #7745

• Reduced metric collection-induced memory use spikes every two minutes in environments with
  a lot of mostly idle queues.

GitHub issues: #7825, #7896

• Delivery acknowledgement timeout now can be configured per queue (using a policy)
  or per consumer (using optional consumer arguments).

GitHub issue: #5437

• It is now possible to limit the maximum number of virtual hosts that can be created
  in the cluster.

Contributed by @SimonUnge (AWS).

GitHub issue: #7777

• Nodes with a lot of (non-mirrored) classic queues with v2 storage enabled
  by default (using classic_queue.default_version = 2) now boot faster.

GitHub issue: #7676

• It is now possible to limit how many shovels or federation links can run on a node
  using rabbitmq.conf:

ini runtime_parameters.limits.shovel = 10 runtime_parameters.limits.federation = 10

Contributed by @illotum (AWS).

GitHub issue: #7917

• Quorum queues: The x-delivery-count header will no longer be added to messages the first time they are delivered (x-delivery-count=0)
  as it has a potential performance impact on consumer deliveries.

GitHub issue: #7732

• Quorum queues will now log if they could not apply policy changes, for example,
  because there was no quorum of replicas online, or the queue was going through
  a leader election.

GitHub issue: #7853

Bug Fixes

This release includes all bug fixes shipped in the 3.11.x series.

• It is now possible to use at-most-once dead lettering with streams (dead letter
  messages to a stream).

GitHub issue: #7846

• It is now possible to use streams in combination with the event exchange.

GitHub issue: #7846

• Superstream could fail
  to elect a single active consumer (SAC) in certain consumer churn conditions.

GitHub issue: #7743

• Quorum queues now have a mechanism of shrinking down to just one replica.
  This is meant to be used exclusively for disaster recovery when a majority
  of nodes hosting replicas of a queue were permanently lost.

GitHub issue: #8322

• Classic mirrored queues that had a length limit defined on them
  handled the overflow of messages differently from the current elected leader,
  eventually causing exceptions and replica restarts.

GitHub issue: #7579

• When a queue was deleted and instantly re-declared with a different type,
  polling consumers (those that use basic.get) ran into an exception.

GitHub issue: #4976

CLI Tools

Enhancements

• rabbitmq-queues rebalance now supports streams in addition to classic and quorum queues.

GitHub issue: #6440

• rabbitmq-streams restart_stream is a new command that can be used to kick off
  a new leader election for a stream. This is another option that can be used
  to rebalance stream leaders across cluster nodes.

GitHub issue: #6440

• Classic queue metrics are now available via rabbitmq-diagnostics observer.

GitHub issue: #3314

• rabbitmqctl update_vhost_metadata is a new command that can be used to update
  the description, default queue type, or tags of a virtual host:

``` shell
rabbitmqctl update_vhost_metadata vh1 --tags qa,quorum,team3,project2

rabbitmqctl update_vhost_metadata vh1 --description "QA env 1 for issue 37483"

rabbitmqctl update_vhost_metadata vh1 --description "QQs all the way" --default-queue-type "quorum"

rabbitmqctl update_vhost_metadata vh1 --description "streaming my brain out" --default-queue-type "stream"
```

GitHub issue: #7914

• rabbitmqctl set_permissions_globally is a new command that sets up user permissions in all existing virtual hosts.

GitHub issue: #1000

• rabbitmq-diagnostics cluster_status now lists how many CPU cores are available to individual nodes, plus a total.

GitHub issue: #7135

Bug Fixes

• rabbitmq-stream stream_status failed with an exception if one of the replicas was offline.

Contributed by @gomoripeti (CloudAMQP)

GitHub issue: #8270

MQTT Plugin

Enhancements

• Significant (up to more than 90% with some workloads) reduction in memory footprint of MQTT connections,
  and a double digit % reduction in memory footprint of MQTT-over-WebSockets (Web MQTT) connections.

GitHub issues: #5895, #7091, #7234

• The x-mqtt-dup header will no longer be present for consumer deliveries as it wasn't used correctly.

Management Plugin

Enhancements

• Support for Identity Provider-initiated Login (IDP).

GitHub issues: #6015, #6201,
#6247

• Potentially breaking change: deprecated UAA-specific configuration keys were removed:
  management.enable_uaa, management.uaa_*.

GitHub issue: #7887

• Classic queues now list their storage implementation version on the details page.

Contributed by @gomoripeti (CloudAMQP)

GitHub issue: #8244

Bug Fixes

• Default queue type key used when add a virtual host via the HTTP API has changed from
  defaultqueuetype to default_queue_type. default_queue_type will be supported
  for a period of time for backwards compatibility.

GitHub issue: #7738

• It was impossible to return to a tab that had a filter expression
  that was not a valid regular expressions.

Now such expressions will be used as regular text filters.

GitHub issue: #8008

OAuth 2 AuthN/AuthZ Backend Plugin

Enhancement

• It is now possible to configure (or strip) the scope prefix used by this plugin when translating
  token scopes to RabbitMQ permissions.

To do so, override auth_oauth2.scope_prefix in rabbitmq.conf:

ini # "rabbitmq." is the default prefix auth_oauth2.scope_prefix = rabbitmq.

GitHub issue: #8001

• Several variables ({username}, {vhost} and JWT claims that are single string values)
  now can be used (expanded) in topic operation authorization.

GitHub issue: #7178

HTTPS AuthN/AuthZ Backend Plugin

Bug Fixes

• The authorization backend could run into an exception when used in
  combination with other backends.

GitHub issue: #7864

Consul Peer Discovery Plugin

Bug Fixes

• Consul peer discovery now correctly determines if the node uses long or short names.

Contributed by @SimonUnge (AWS).

GitHub issue: #4230

Dependency Changes

• ra was upgraded to 2.6.1
• osiris was upgraded to 1.5.1
• cowlib was upgraded to 2.12.1

Source Code Archives

To obtain source code of the entire distribution, please download the archive named rabbitmq-server-3.12.0.tar.xz
instead of the source tarball produced by GitHub.