Airflow - 2.4.0


New Features

  • Add Data-aware Scheduling (AIP-48 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-48+milestone%3A%22Airflow+2.4.0%22>_)
  • Add @task.short_circuit TaskFlow decorator (#25752)
  • Make execution_date_or_run_id optional in tasks test command (#26114)
  • Automatically register DAGs that are used in a context manager (#23592, #26398)
  • Add option of sending DAG parser logs to stdout. (#25754)
  • Support multiple DagProcessors parsing files from different locations. (#25935)
  • Implement ExternalPythonOperator (#25780)
  • Make execution_date optional for command dags test (#26111)
  • Implement expand_kwargs() against a literal list (#25925)
  • Add trigger rule tooltip (#26043)
  • Add conf parameter to CLI for airflow dags test (#25900)
  • Include scheduled slots in pools view (#26006)
  • Add output property to MappedOperator (#25604)
  • Add roles delete command to cli (#25854)
  • Add Airflow specific warning classes (#25799)
  • Add support for TaskGroup in ExternalTaskSensor (#24902)
  • Add @task.kubernetes taskflow decorator (#25663)
  • Add a way to import Airflow without side-effects (#25832)
  • Let timetables control generated run_ids. (#25795)
  • Allow per-timetable ordering override in grid view (#25633)
  • Grid logs for mapped instances (#25610, #25621, #25611)
  • Consolidate to one schedule param (#25410)
  • DAG regex flag in backfill command (#23870)
  • Adding support for owner links in the Dags view UI (#25280)
  • Ability to clear a specific DAG Run's task instances via REST API (#23516)
  • Possibility to document DAG with a separate markdown file (#25509)
  • Add parsing context to DAG Parsing (#25161)
  • Implement CronTriggerTimetable (#23662)
  • Add option to mask sensitive data in UI configuration page (#25346)
  • Create new databases from the ORM (#24156)
  • Implement XComArg.zip(*xcom_args) (#25176)
  • Introduce sla_miss metric (#23402)
  • Implement map() semantic (#25085)
  • Add override method to TaskGroupDecorator (#25160)
  • Implement expand_kwargs() (#24989)
  • Add parameter to turn off SQL query logging (#24570)
  • Add DagWarning model, and a check for missing pools (#23317)
  • Add Task Logs to Grid details panel (#24249)
  • Added small health check server and endpoint in scheduler(#23905)
  • Add built-in External Link for ExternalTaskMarker operator (#23964)
  • Add default task retry delay config (#23861)
  • Add clear DagRun endpoint. (#23451)
  • Add support for timezone as string in cron interval timetable (#23279)
  • Add auto-refresh to dags home page (#22900, #24770)

Improvements

  • Add more weekday operator and sensor examples #26071 (#26098)
  • Add subdir parameter to dags reserialize command (#26170)
  • Update zombie message to be more descriptive (#26141)
  • Only send an SlaCallbackRequest if the DAG is scheduled (#26089)
  • Promote Operator.output more (#25617)
  • Upgrade API files to typescript (#25098)
  • Less hacky double-rendering prevention in mapped task (#25924)
  • Improve Audit log (#25856)
  • Remove mapped operator validation code (#25870)
  • More DAG(schedule=...) improvements (#25648)
  • Reduce operator_name dupe in serialized JSON (#25819)
  • Make grid view group/mapped summary UI more consistent (#25723)
  • Remove useless statement in task_group_to_grid (#25654)
  • Add optional data interval to CronTriggerTimetable (#25503)
  • Remove unused code in /grid endpoint (#25481)
  • Add and document description fields (#25370)
  • Improve Airflow logging for operator Jinja template processing (#25452)
  • Update core example DAGs to use @task.branch decorator (#25242)
  • Update DAG audit_log route (#25415)
  • Change stdout and stderr access mode to append in commands (#25253)
  • Remove getTasks from Grid view (#25359)
  • Improve taskflow type hints with ParamSpec (#25173)
  • Use tables in grid details panes (#25258)
  • Explicitly list @dag arguments (#25044)
  • More typing in SchedulerJob and TaskInstance (#24912)
  • Patch getfqdn with more resilient version (#24981)
  • Replace all NBSP characters by whitespaces (#24797)
  • Re-serialize all DAGs on airflow db upgrade (#24518)
  • Rework contract of try_adopt_task_instances method (#23188)
  • Make expand() error vague so it's not misleading (#24018)
  • Add enum validation for [webserver]analytics_tool (#24032)
  • Add dttm searchable field in audit log (#23794)
  • Allow more parameters to be piped through via execute_in_subprocess (#23286)
  • Use func.count to count rows (#23657)
  • Remove stale serialized dags (#22917)
  • AIP45 Remove dag parsing in airflow run local (#21877)
  • Add support for queued state in DagRun update endpoint. (#23481)
  • Add fields to dagrun endpoint (#23440)
  • Use sql_alchemy_conn for celery result backend when result_backend is not set (#24496)

Bug Fixes

  • Have consistent types between the ORM and the migration files (#24044, #25869)
  • Disallow any dag tags longer than 100 char (#25196)
  • Add the dag_id to AirflowDagCycleException message (#26204)
  • Properly build URL to retrieve logs independently from system (#26337)
  • For worker log servers only bind to IPV6 when dual stack is available (#26222)
  • Fix TaskInstance.task not defined before handle_failure (#26040)
  • Undo secrets backend config caching (#26223)
  • Fix faulty executor config serialization logic (#26191)
  • Show DAGs and Datasets menu links based on role permission (#26183)
  • Allow setting TaskGroup tooltip via function docstring (#26028)
  • Fix RecursionError on graph view of a DAG with many tasks (#26175)
  • Fix backfill occasional deadlocking (#26161)
  • Fix DagRun.start_date not set during backfill with --reset-dagruns True (#26135)
  • Use label instead of id for dynamic task labels in graph (#26108)
  • Don't fail DagRun when leaf mapped_task is SKIPPED (#25995)
  • Add group prefix to decorated mapped task (#26081)
  • Fix UI flash when triggering with dup logical date (#26094)
  • Fix Make items nullable for TaskInstance related endpoints to avoid API errors (#26076)
  • Fix BranchDateTimeOperator to be timezone-awreness-insensitive (#25944)
  • Fix legacy timetable schedule interval params (#25999)
  • Fix response schema for list-mapped-task-instance (#25965)
  • Properly check the existence of missing mapped TIs (#25788)
  • Fix broken auto-refresh on grid view (#25950)
  • Use per-timetable ordering in grid UI (#25880)
  • Rewrite recursion when parsing DAG into iteration (#25898)
  • Find cross-group tasks in iter_mapped_dependants (#25793)
  • Fail task if mapping upstream fails (#25757)
  • Support / in variable get endpoint (#25774)
  • Use cfg default_wrap value for grid logs (#25731)
  • Add origin request args when triggering a run (#25729)
  • Operator name separate from class (#22834)
  • Fix incorrect data interval alignment due to assumption on input time alignment (#22658)
  • Return None if an XComArg fails to resolve (#25661)
  • Correct json arg help in airflow variables set command (#25726)
  • Added MySQL index hint to use ti_state on find_zombies query (#25725)
  • Only excluded actually expanded fields from render (#25599)
  • Grid, fix toast for axios errors (#25703)
  • Fix UI redirect (#26409)
  • Require dag_id arg for dags list-runs (#26357)
  • Check for queued states for dags auto-refresh (#25695)
  • Fix upgrade code for the dag_owner_attributes table (#25579)
  • Add map index to task logs api (#25568)
  • Ensure that zombie tasks for dags with errors get cleaned up (#25550)
  • Make extra link work in UI (#25500)
  • Sync up plugin API schema and definition (#25524)
  • First/last names can be empty (#25476)
  • Refactor DAG pages to be consistent (#25402)
  • Check expand_kwargs() input type before unmapping (#25355)
  • Filter XCOM by key when calculating map lengths (#24530)
  • Fix ExternalTaskSensor not working with dynamic task (#25215)
  • Added exception catching to send default email if template file raises any exception (#24943)
  • Bring MappedOperator members in sync with BaseOperator (#24034)

Misc/Internal

  • Add automatically generated ERD schema for the MetaData DB (#26217)
  • Mark serialization functions as internal (#26193)
  • Remove remaining deprecated classes and replace them with PEP562 (#26167)
  • Move dag_edges and task_group_to_dict to corresponding util modules (#26212)
  • Lazily import many modules to improve import speed (#24486, #26239)
  • FIX Incorrect typing information (#26077)
  • Add missing contrib classes to deprecated dictionaries (#26179)
  • Re-configure/connect the ORM after forking to run a DAG processor (#26216)
  • Remove cattrs from lineage processing. (#26134)
  • Removed deprecated contrib files and replace them with PEP-562 getattr (#26153)
  • Make BaseSerialization.serialize "public" to other classes. (#26142)
  • Change the template to use human readable task_instance description (#25960)
  • Bump moment-timezone from 0.5.34 to 0.5.35 in /airflow/www (#26080)
  • Fix Flask deprecation warning (#25753)
  • Add CamelCase to generated operations types (#25887)
  • Fix migration issues and tighten the CI upgrade/downgrade test (#25869)
  • Fix type annotations in SkipMixin (#25864)
  • Workaround setuptools editable packages path issue (#25848)
  • Bump undici from 5.8.0 to 5.9.1 in /airflow/www (#25801)
  • Add custom_operator_name attr to _BranchPythonDecoratedOperator (#25783)
  • Clarify filename_template deprecation message (#25749)
  • Use ParamSpec to replace ... in Callable (#25658)
  • Remove deprecated modules (#25543)
  • Documentation on task mapping additions (#24489)
  • Remove Smart Sensors (#25507)
  • Fix elasticsearch test config to avoid warning on deprecated template (#25520)
  • Bump terser from 4.8.0 to 4.8.1 in /airflow/ui (#25178)
  • Generate typescript types from rest API docs (#25123)
  • Upgrade utils files to typescript (#25089)
  • Upgrade remaining context file to typescript. (#25096)
  • Migrate files to ts (#25267)
  • Upgrade grid Table component to ts. (#25074)
  • Skip mapping against mapped ti if it returns None (#25047)
  • Refactor js file structure (#25003)
  • Move mapped kwargs introspection to separate type (#24971)
  • Only assert stuff for mypy when type checking (#24937)
  • Bump moment from 2.29.3 to 2.29.4 in /airflow/www (#24885)
  • Remove "bad characters" from our codebase (#24841)
  • Remove xcom_push flag from BashOperator (#24824)
  • Move Flask hook registration to end of file (#24776)
  • Upgrade more javascript files to typescript (#24715)
  • Clean up task decorator type hints and docstrings (#24667)
  • Preserve original order of providers' connection extra fields in UI (#24425)
  • Rename charts.css to chart.css (#24531)
  • Rename grid.css to chart.css (#24529)
  • Misc: create new process group by set_new_process_group utility (#24371)
  • Airflow UI fix Prototype Pollution (#24201)
  • Bump moto version (#24222)
  • Remove unused [github_enterprise] from ref docs (#24033)
  • Clean up f-strings in logging calls (#23597)
  • Add limit for JPype1 (#23847)
  • Simply json responses (#25518)
  • Add min attrs version (#26408)

Doc only changes

  • Add url prefix setting for Celery Flower (#25986)
  • Updating deprecated configuration in examples (#26037)
  • Fix wrong link for taskflow tutorial (#26007)
  • Reorganize tutorials into a section (#25890)
  • Fix concept doc for dynamic task map (#26002)
  • Update code examples from "classic" operators to taskflow (#25845, #25657)
  • Add instructions on manually fixing MySQL Charset problems (#25938)
  • Prefer the local Quick Start in docs (#25888)
  • Fix broken link to Trigger Rules (#25840)
  • Improve docker documentation (#25735)
  • Correctly link to Dag parsing context in docs (#25722)
  • Add note on task_instance_mutation_hook usage (#25607)
  • Note that TaskFlow API automatically passes data between tasks (#25577)
  • Update DAG run to clarify when a DAG actually runs (#25290)
  • Update tutorial docs to include a definition of operators (#25012)
  • Rewrite the Airflow documentation home page (#24795)
  • Fix task-generated mapping example (#23424)
  • Add note on subtle logical date change in 2.2.0 (#24413)
  • Add missing import in best-practices code example (#25391)

Details

date
Sept. 19, 2022, 9:11 a.m.
name
Apache Airflow 2.4.0
type
Minor
👇
Register or login to:
  • 🔍View and search all Airflow releases.
  • 🛠️Create and share lists to track your tools.
  • 🚨Setup notifications for major, security, feature or patch updates.
  • 🚀Much more coming soon!
Continue with GitHub
Continue with Google
or