Data orchestration plays a crucial role in seamlessly managing the flow of data between different systems. Apache Airflow, a popular open-source technology initially developed by Airbnb, has emerged as a leading tool for data orchestration. Now, Astronomer, the primary commercial sponsor of the Apache Airflow project, introduces its latest Astro platform update, offering comprehensive enterprise support, enhanced security, and advanced management capabilities.
The Evolution of Airflow: From Data Analytics to AI and ML
While Airflow initially gained traction in the realm of data analytics and business intelligence by facilitating the orchestration of data pipelines, its utility has expanded to encompass artificial intelligence (AI) and machine learning (ML) workloads. According to Julian LaNeve, CTO at Astronomer, Airflow excels in several areas: “Airflow is very good at a couple of things, one is just basically writing and running data pipelines. Airflow lets you define pipelines as code, so you can do anything that the code will let you do which is essentially boundless.”
Over the years, Airflow has gained popularity for enabling organizations to define, build, and deploy data pipelines with ease. It seamlessly integrates with major data platforms and cloud provider systems such as Snowflake, Databricks, AWS, Microsoft, and Google Cloud. However, as LaNeve pointed out, managing Airflow at an enterprise scale can be challenging. This is where Astronomer steps in, offering a managed service for Apache Airflow and supplementing the core open-source technology with additional capabilities.
Introducing the Astro Platform and Enhanced Capabilities
With its Astro platform update, Astronomer takes data orchestration to new heights by introducing a host of enhanced capabilities. One of the fundamental challenges in data pipelines is establishing secure connections to data sources. The Astro update addresses this challenge by introducing a new connection management feature that provides governance, visibility, and security for data pipelines. LaNeve explained, “We’ve built a connection management feature into the Astro platform that lets an administrator come in and define connections to Snowflake, Databricks, and anywhere that Airflow can access.”
Furthermore, the Astro platform update streamlines upgrades and rollbacks of data pipeline configurations. Users can easily revert to previous configurations in case of pipeline failures, ensuring that production workloads remain uninterrupted. The platform also performs compatibility checks to ensure smooth execution of the updated code.
Astronomer: Empowering AI Workflows
Astronomer has emerged as a go-to solution for AI workflows. The company has integrated with several vendors in the AI landscape, including OpenAI, Cohere, Pinecone, OpenSearch, Weaviate, and pgvector. Moreover, Astronomer has developed a reference architecture that guides organizations in building and deploying large language model (LLM) applications. An exemplary demonstration of this architecture is the ask.astronomer.io application, which utilizes a retrieval augmented generation (RAG) approach to pull information from various sources.
According to LaNeve, Airflow and the Astro platform also find extensive usage in training AI models. He emphasized the importance of training models with the latest data reliably, and Astronomer and Airflow are specifically designed to fulfill this need.
