Celery - v5.4.0

Celery v5.4.0 and v5.3.x have consistently focused on enhancing the overall QA, both internally and externally.
This effort led to the new pytest-celery v1.0.0 release, developed concurrently with v5.3.0 & v5.4.0.

This release introduces two significant QA enhancements:

• Smoke Tests: A new layer of automatic tests has been added to Celery's standard CI. These tests are designed to handle production scenarios and complex conditions efficiently. While new contributions will not be halted due to the lack of smoke tests, we will request smoke tests for advanced changes where appropriate.
• Standalone Bug Report Script: The new pytest-celery plugin now allows for encapsulating a complete Celery dockerized setup within a single pytest script. Incorporating these into new bug reports will enable us to reproduce reported bugs deterministically, potentially speeding up the resolution process.

Contrary to the positive developments above, there have been numerous reports about issues with the Redis broker malfunctioning upon restarts and disconnections. Our initial attempts to resolve this were not successful (#8796).
With our enhanced QA capabilities, we are now prepared to address the core issue with Redis (as a broker) again.

The rest of the changes for this release are grouped below, with the changes from the latest release candidate listed at the end.

What's Changed

• Add a Task class specialised for Django (#8491)
• Add Google Cloud Storage (GCS) backend (#8868)
• Added documentation to the smoke tests infra (#8970)
• Added a checklist item for using pytest-celery in a bug report (#8971)
• Bugfix: Missing id on chain (#8798)
• Bugfix: Worker not consuming tasks after Redis broker restart (#8796)
• Catch UnicodeDecodeError when opening corrupt beat-schedule.db (#8806)
• chore(ci): Enhance CI with workflow_dispatch for targeted debugging and testing (#8826)
• Doc: Enhance "Testing with Celery" section (#8955)
• Docfix: pip install celery[sqs] -> pip install "celery[sqs]" (#8829)
• Enable efficient chord when using dynamicdb as backend store (#8783)
• feat(daemon): allows daemonization options to be fetched from app settings (#8553)
• Fix DeprecationWarning: datetime.datetime.utcnow() (#8726)
• Fix recursive result parents on group in middle of chain (#8903)
• Fix typos and grammar (#8915)
• Fixed version documentation tag from #8553 in configuration.rst (#8802)
• Hotfix: Smoke tests didn't allow customizing the worker's command arguments, now it does (#8937)
• Make custom remote control commands available in CLI (#8489)
• Print safe_say() to stdout for non-error flows (#8919)
• Support moto 5.0 (#8838)
• Update contributing guide to use ssh upstream url (#8881)
• Update optimizing.rst (#8945)
• Updated concurrency docs page. (#8753)

Dependencies Updates

• Bump actions/setup-python from 4 to 5 (#8701)
• Bump codecov/codecov-action from 3 to 4 (#8831)
• Bump isort from 5.12.0 to 5.13.2 (#8772)
• Bump msgpack from 1.0.7 to 1.0.8 (#8885)
• Bump mypy from 1.8.0 to 1.9.0 (#8898)
• Bump pre-commit to 3.6.1 (#8839)
• Bump pre-commit/action from 3.0.0 to 3.0.1 (#8835)
• Bump pytest from 8.0.2 to 8.1.1 (#8901)
• Bump pytest-celery to v1.0.0 (#8962)
• Bump pytest-cov to 5.0.0 (#8924)
• Bump pytest-order from 1.2.0 to 1.2.1 (#8941)
• Bump pytest-subtests from 0.11.0 to 0.12.1 (#8896)
• Bump pytest-timeout from 2.2.0 to 2.3.1 (#8894)
• Bump python-memcached from 1.59 to 1.61 (#8776)
• Bump sphinx-click from 4.4.0 to 5.1.0 (#8774)
• Update cryptography to 42.0.5 (#8869)
• Update elastic-transport requirement from <=8.12.0 to <=8.13.0 (#8933)
• Update elasticsearch requirement from <=8.12.1 to <=8.13.0 (#8934)
• Upgraded Sphinx from v5.3.0 to v7.x.x (#8803)

Changes since 5.4.0rc2

• Update elastic-transport requirement from <=8.12.0 to <=8.13.0 by @dependabot in https://github.com/celery/celery/pull/8933
• Update elasticsearch requirement from <=8.12.1 to <=8.13.0 by @dependabot in https://github.com/celery/celery/pull/8934
• Hotfix: Smoke tests didn't allow customizing the worker's command arguments, now it does by @Nusnus in https://github.com/celery/celery/pull/8937
• Bump pytest-celery to 1.0.0rc3 by @Nusnus in https://github.com/celery/celery/pull/8946
• Update optimizing.rst by @alexmclarty in https://github.com/celery/celery/pull/8945
• Doc: Enhance "Testing with Celery" section by @Nusnus in https://github.com/celery/celery/pull/8955
• Bump pytest-celery to v1.0.0 by @Nusnus in https://github.com/celery/celery/pull/8962
• Bump pytest-order from 1.2.0 to 1.2.1 by @dependabot in https://github.com/celery/celery/pull/8941
• Added documentation to the smoke tests infra by @Nusnus in https://github.com/celery/celery/pull/8970
• Added a checklist item for using pytest-celery in a bug report by @Nusnus in https://github.com/celery/celery/pull/8971
• Added changelog for v5.4.0 by @Nusnus in https://github.com/celery/celery/pull/8973
• Bump version: 5.4.0rc2 → 5.4.0 by @Nusnus in https://github.com/celery/celery/pull/8974

New Contributors

• @danyi1212 made their first contribution in https://github.com/celery/celery/pull/8690
• @Mulugruntz made their first contribution in https://github.com/celery/celery/pull/8696
• @Viicos made their first contribution in https://github.com/celery/celery/pull/8743
• @em1le made their first contribution in https://github.com/celery/celery/pull/8747
• @robotrapta made their first contribution in https://github.com/celery/celery/pull/8753
• @amweiss made their first contribution in https://github.com/celery/celery/pull/8791
• @andyzickler made their first contribution in https://github.com/celery/celery/pull/8806
• @dingxiong made their first contribution in https://github.com/celery/celery/pull/8783
• @Watkurem made their first contribution in https://github.com/celery/celery/pull/8825
• @50-Course made their first contribution in https://github.com/celery/celery/pull/8826
• @s-t-e-v-e-n-k made their first contribution in https://github.com/celery/celery/pull/8838
• @murrple-1 made their first contribution in https://github.com/celery/celery/pull/8841
• @hann-wang made their first contribution in https://github.com/celery/celery/pull/8663
• @tobinus made their first contribution in https://github.com/celery/celery/pull/8489
• @haimjether made their first contribution in https://github.com/celery/celery/pull/8868
• @hsujeremy made their first contribution in https://github.com/celery/celery/pull/8881
• @beneltayar made their first contribution in https://github.com/celery/celery/pull/8903
• @carlosp420 made their first contribution in https://github.com/celery/celery/pull/8915
• @lukasz-leszczuk-airspace-intelligence made their first contribution in https://github.com/celery/celery/pull/8919
• @alexmclarty made their first contribution in https://github.com/celery/celery/pull/8945

Full Changelog: https://github.com/celery/celery/compare/v5.3.6...v5.4.0