create and manage scripted data pipelines as code (Python)Īirflow organizes your workflows into DAGs composed of tasks.run workflows that are not data-related.orchestrate data pipelines over object stores and data warehouses.Astronomer.io and Google also offer managed Airflow services. ![]() (And Airbnb, of course.) Amazon offers AWS Managed Workflows on Apache Airflow (MWAA) as a commercial managed service. According to marketing intelligence firm HG Insights, as of the end of 2021 Airflow was used by almost 10,000 organizations, including Applied Materials, the Walt Disney Company, and Zoom. Airflow’s proponents consider it to be distributed, scalable, flexible, and well-suited to handle the orchestration of complex business logic. Written in Python, Airflow is increasingly popular, especially among developers, due to its focus on configuration as code. Airbnb open-sourced Airflow early on, and it became a Top-Level Apache Software Foundation project in early 2019.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |