airflow db export-archived doesn't page results

[dominikkedziak]

Apache Airflow version

3.0.1

If "Other Airflow 2 version" selected, which one?

All 2.x

What happened?

While using airflow db export-archived command it fetches all the results at once instead of paging them.

In case of big instances and archiving a lot of data a lot of memory is needed to use this command, othervise it ends up with OOMErrors

Exact line:
https://github.com/apache/airflow/blob/3.0.1/airflow-core/src/airflow/utils/db_cleanup.py#L163

What you think should happen instead?

use server-side cursor for paging if available.

How to reproduce

Create a database that has tables with the size bigger than your memory and then try to archive data with use of
airflow db export-archived command.

Operating System

K8s - pods with Ubuntu

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.