Stay ConnectedWelcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward.
Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems.
Podcast Webpage: https://www.astronomer.io/podcast/
Copyright: © All rights reserved
When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.
Key Takeaways:
(02:03) Airflow as a service streamlines Uber’s data workflows.
(06:16) Serialization boosts security and reduces errors.
(10:05) Java-based scheduler improves system reliability.
(13:40) Custom recovery model supports emergency pipeline switching.
(15:58) No-code UI allows easy pipeline creation for non-coders.
(18:12) Backfill feature enables historical data processing.
(22:06) Regular updates keep Uber aligned with Airflow advancements.
(26:07) Plans to leverage Airflow’s latest features.
Resources Mentioned:
https://www.linkedin.com/in/shahshobhit/
https://www.linkedin.com/in/maheshwarisumit/
Uber -
https://www.linkedin.com/company/uber-com/
https://airflow.apache.org/
https://airflowsummit.org/
Uber -
https://www.uber.com/tw/en/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Efficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows.
Key Takeaways:
(01:55) Astrafy helps companies manage data using Google Cloud.
(04:36) Airflow is central to Astrafy’s data engineering efforts.
(07:17) Datasets and Pub/Sub are used for real-time workflows.
(09:59) Pub/Sub links multiple Airflow environments.
(12:40) Datasets eliminate the need for constant monitoring.
(15:22) Airflow updates have improved large-scale data operations.
(18:03) New Airflow API features make dataset updates easier.
(20:45) Real-time orchestration speeds up data processing for clients.
(23:26) Pub/Sub enhances flexibility across cloud environments.
(26:08) Future Airflow features will offer more control over data workflows.
Resources Mentioned:
https://www.linkedin.com/in/andrea-bombino/
Astrafy -
https://www.linkedin.com/company/astrafy/
https://airflow.apache.org/
https://cloud.google.com/
dbt -
https://www.getdbt.com/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Data orchestration is evolving faster than ever and Apache Airflow 3 is set to revolutionize how enterprises handle complex workflows. In this episode, we dive into the exciting advancements with Vikram Koka, Chief Strategy Officer at Astronomer and PMC Member at The Apache Software Foundation. Vikram shares his insights on the evolution of Airflow and its pivotal role in shaping modern data-driven workflows, particularly with the upcoming release of Airflow 3.
Key Takeaways:
(02:36) Vikram leads Astronomer’s engineering and open-source teams for Airflow.
(05:26) Airflow enables reliable data ingestion and curation.
(08:17) Enterprises use Airflow for mission-critical data pipelines.
(11:08) Airflow 3 introduces major architectural updates.
(13:58) Multi-cloud and edge deployments are supported in Airflow 3.
(16:49) Event-driven scheduling makes Airflow more dynamic.
(19:40) Tasks in Airflow 3 can run in any language.
(22:30) Multilingual task support is crucial for enterprises.
(25:21) Data assets and event-based integration enhance orchestration.
(28:12) Community feedback plays a vital role in Airflow 3.
Resources Mentioned:
https://www.linkedin.com/in/vikramkoka/
https://www.linkedin.com/company/astronomer/
The Apache Software Foundation LinkedIn -
https://www.linkedin.com/company/the-apache-software-foundation/
https://www.linkedin.com/company/apache-airflow/
https://airflow.apache.org/
https://www.astronomer.io/
The Apache Software Foundation -
https://www.apache.org/
Join the Airflow slack and/or Dev list -
https://airflow.apache.org/community/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Data and AI are revolutionizing HR, empowering leaders to measure performance and drive strategic decisions like never before.
In this episode, we explore the transformation of HR technology with Guy Dassa, Chief Technology Officer at 15Five, as he shares insights into their evolving data platform. Guy discusses how 15Five equips HR leaders with tools to measure and take action on team performance, engagement and retention. He explains their data-driven approach, highlighting how Apache Airflow supports their data ingestion, transformation, and AI integration.
Key Takeaways:
(01:54) 15Five acts as a command center for HR leaders.
(03:40) Tools like performance reviews, engagement surveys, and an insights dashboard guide actionable HR steps.
(05:33) Data visualization, insights, and action recommendations enhance HR effectiveness to improve their people's outcomes.
(07:08) Strict data confidentiality and sanitized AI model training.
(09:21) Airflow is central to data transformation and enrichment.
(11:15) Airflow enrichment DAGs integrate AI models.
(13:33) Integration of Airflow and DBT enables efficient data transformation.
(15:28) Synchronization challenges arise with reverse ETL processes.
(17:10) Future plans include deeper Airflow integration with AI.
(19:31) Emphasizing the need for DAG versioning and improved dependency visibility.
Resources Mentioned:
https://www.linkedin.com/in/guydassa/
15Five -
https://www.linkedin.com/company/15five/
https://airflow.apache.org/
MLflow -
https://mlflow.org/
DBT -
https://www.getdbt.com/
https://kubernetes.io/
RedShift -
https://aws.amazon.com/redshift/
15Five -
https://www.15five.com/
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Unlocking engineering productivity goes beyond coding — it’s about managing knowledge efficiently. In this episode, we explore the innovative ways in which Dosu leverages Airflow for data orchestration and supports the Airflow project.
Devin Stein, Founder of Dosu, shares his insights on how engineering teams can focus on value-added work by automating knowledge management. Devin dives into Dosu’s purpose, the significance of AI in their product, and why they chose Airflow as the backbone for scheduling and data management.
Key Takeaways:
(01:33) Dosu's mission to democratize engineering knowledge.
(05:00) AI is central to Dosu's product for structuring engineering knowledge.
(06:23) The importance of maintaining up-to-date data for AI effectiveness.
(07:55) How Airflow supports Dosu’s data ingestion and automation processes.
(08:45) The reasoning behind choosing Airflow over other orchestrators.
(11:00) Airflow enables Dosu to manage both traditional ETL and dynamic workflows.
(13:04) Dosu assists the Airflow project by auto-labeling issues and discussions.
(14:56) Thoughtful collaboration with the Airflow community to introduce AI tools.
(16:37) The potential of Airflow to handle more dynamic, scheduled workflows in the future.
(18:00) Challenges and custom solutions for implementing dynamic workflows in Airflow.
Resources Mentioned:
Apache Airflow - https://airflow.apache.org/
Dosu Website - https://dosu.dev/
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
Harnessing data at scale is the key to driving innovation in autonomous vehicle technology. In this episode, we uncover how advanced orchestration tools are transforming machine learning operations in the automotive industry. Serjesh Sharma, Supervisor ADAS Machine Learning Operations (MLOps) at Ford Motor Company, joins us to discuss the challenges and innovations his team faces working to enhance vehicle safety and automation. Serjesh shares insights into the intricate data processes that support Ford’s Advanced Driver Assistance Systems (ADAS) and how his team leverages Apache Airflow to manage massive data loads efficiently. Key Takeaways: (01:44) ADAS involves advanced features like pre-collision assist and self-driving capabilities. (04:47) Ensuring sensor accuracy and vehicle safety requires extensive data processing. (05:08) The combination of on-prem and cloud infrastructure optimizes data handling. (09:27) Ford processes around one petabyte of data per week, using both CPUs and GPUs. (10:33) Implementing software engineering best practices to improve scalability and reliability. (15:18) GitHub Issues streamline onboarding and infrastructure provisioning. (17:00) Airflow's modular design allows Ford to manage complex data pipelines. (19:00) Kubernetes pod operators help optimize resource usage for CPU-intensive tasks. (20:35) Ford's scale challenges led to customized Airflow configurations for high concurrency. (21:02) Advanced orchestration tools are pivotal in managing vast data landscapes in automotive innovation. Resources Mentioned: Serjesh Sharma - www.linkedin.com/in/serjeshsharma/ Ford Motor Company - www.linkedin.com/company/ford-motor-company/ Apache Airflow - airflow.apache.org/ Kubernetes - kubernetes.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Data failures are inevitable but how you manage them can define the success of your operations. In this episode, we dive deep into the challenges of data engineering and AI with Brendan Frick, Senior Engineering Manager, Data at GumGum. Brendan shares his unique approach to managing task failures and DAG issues in a high-stakes ad-tech environment. Brendan discusses how GumGum leverages Apache Airflow to streamline data processes, ensuring efficient data movement and orchestration while minimizing disruptions in their operations. Key Takeaways: (02:02) Brendan’s role at GumGum and its approach to ad tech. (04:27) How GumGum uses Airflow for daily data orchestration, moving data from S3 to warehouses. (07:02) Handling task failures in Airflow using Jira for actionable, developer-friendly responses. (09:13) Transitioning from email alerts to a more structured system with Jira and PagerDuty. (11:40) Monitoring task retry rates as a key metric to identify potential issues early. (14:15) Utilizing Looker dashboards to track and analyze task performance and retry rates. (16:39) Transitioning from Kubernetes operator to a more reliable system for data processing. (19:25) The importance of automating stakeholder communication with data lineage tools like Atlan. (20:48) Implementing data contracts to ensure SLAs are met across all data processes. (22:01) The role of scalable SLAs in Airflow to ensure data reliability and meet business needs. Resources Mentioned: Brendan Frick - https://www.linkedin.com/in/brendan-frick-399345107/ GumGum - https://www.linkedin.com/company/gumgum/ Apache Airflow - https://airflow.apache.org/ Jira - https://www.atlassian.com/software/jira Atlan - https://atlan.com/ Kubernetes - https://kubernetes.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
A 13% reduction in failure rates — this is how two data scientists at Astronomer revolutionized their data pipelines using Apache Airflow. In this episode, we enter the world of data orchestration and AI with Maggie Stark and Marion Azoulai, both Senior Data Scientists at Astronomer. Maggie and Marion discuss how their team re-architected their use of Airflow to improve scalability, reliability and efficiency in data processing. They share insights on overcoming challenges with sensors and how moving to datasets transformed their workflows. Key Takeaways: (02:23) The data team’s role as a centralized hub within Astronomer. (05:11) Airflow is the backbone of all data processes, running 60,000 tasks daily. (07:13) Custom task groups enable efficient code reuse and adherence to best practices. (11:33) Sensor-heavy architectures can lead to cascading failures and resource issues. (12:09) Switching to datasets has improved reliability and scalability. (14:19) Building a control DAG provides end-to-end visibility of pipelines. (16:42) Breaking down DAGs into smaller units minimizes failures and improves management. (19:02) Failure rates improved from 16% to 3% with the new architecture. Resources Mentioned: Maggie Stark - https://www.linkedin.com/in/margaretstark/ Marion Azoulai - https://www.linkedin.com/in/marionazoulai/ Astronomer | LinkedIn - https://www.linkedin.com/company/astronomer/ Apache Airflow - https://airflow.apache.org/ Astronomer | Website - https://www.astronomer.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Mastering the flow of data is essential for driving innovation and efficiency in today’s competitive landscape. In this episode, we explore the evolution of data orchestration and the pivotal role of Apache Airflow in modern data workflows. Ben Tallman, Chief Technology Officer at M Science, joins us and shares his extensive experience with Airflow, detailing its early adoption, evolution and the profound impact it has had on data engineering practices. His insights reveal how leveraging Airflow can streamline complex data processes, enhance observability and ultimately drive business success. Key Takeaways: (02:31) Benjamin’s journey with Airflow and its early adoption. (05:36) The transition from legacy schedulers to Airflow at Apigee and later Google. (08:52) The challenges and benefits of running production-grade Airflow instances. (10:46) How Airflow facilitates the management of large-scale data at M Science. (11:56) The importance of reducing time to value for customers using data products. (13:32) Airflow’s role in ensuring observability and reliability in data workflows. (17:00) Managing petabytes of data and billions of records efficiently. (19:08) Integration of various data sources and ensuring data product quality. (20:04) Leveraging Airflow for data observability and reducing time to value. (22:04) Benjamin’s vision for the future development of Airflow, including audit trails for variables. Resources Mentioned: Ben Tallman - https://www.linkedin.com/in/btallman/ M Science - https://www.linkedin.com/company/m-science-llc/ Apache Airflow - https://airflow.apache.org/ Astronomer - https://www.astronomer.io/ Databricks - https://databricks.com/ Snowflake - https://www.snowflake.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. #AI #Automation #Airflow #MachineLearning
Data orchestration is revolutionizing the way companies manage and process data. In this episode, we explore the critical role of data orchestration in modern data workflows and how Apache Airflow is used to enhance data processing and AI model deployment. Hannan Kravitz, Data Engineering Team Leader at Artlist, joins us to share his insights on leveraging Airflow for data engineering and its impact on their business operations. Key Takeaways: (01:00) Hannan introduces Artlist and its mission to empower content creators. (04:27) The importance of collecting and modeling data to support business insights. (06:40) Using Airflow to connect multiple data sources and create dashboards. (09:40) Implementing a monitoring DAG for proactive alerts within Airflow. (12:31) Customizing Airflow for business metric KPI monitoring and setting thresholds. (15:00) Addressing decreases in purchases due to technical issues with proactive alerts. (17:45) Customizing data quality checks with dynamic task mapping in Airflow. (20:00) Desired improvements in Airflow UI and logging capabilities. (21:00) Enabling business stakeholders to change thresholds using Streamlit. (22:26) Future improvements desired in the Airflow project. Resources Mentioned: Hannan Kravitz - https://www.linkedin.com/in/hannan-kravitz-60563112/ Artlist - https://www.linkedin.com/company/art-list/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Streamlit - https://streamlit.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Data engineering is constantly evolving and staying ahead means mastering tools like Apache Airflow. In this episode, we explore the world of data engineering with Alexandre Magno Lima Martins, Senior Data Engineer at Teya. Alexandre talks about optimizing data workflows and the smart solutions they've created at Teya to make data processing easier and more efficient. Key Takeaways: (02:01) Alexandre explains his role at Teya and the responsibilities of a data platform engineer. (02:40) The primary use cases of Airflow at Teya, especially with dbt and machine learning projects. (04:14) How Teya creates self-service DAGs for dbt models. (05:58) Automating DAG creation with CI/CD pipelines. (09:04) Switching to a multi-file method for better Airflow performance. (12:48) Challenges faced with Kubernetes Executor vs. Celery Executor. (16:13) Using Celery Executor to handle fast tasks efficiently. (17:02) Implementing KEDA autoscaler for better scaling of Celery workers. (19:05) Reasons for not using Cosmos for DAG generation and cross-DAG dependencies. (21:16) Alexandre's wish list for future Airflow features, focusing on multi-tenancy. Resources Mentioned: Alexandre Magno Lima Martins - https://www.linkedin.com/in/alex-magno/ Teya - https://www.linkedin.com/company/teya-global/ Apache Airflow - https://airflow.apache.org/ dbt - https://www.getdbt.com/ Kubernetes - https://kubernetes.io/ KEDA - https://keda.sh/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Managing data workflows well can change the game for any company. In this episode, we talk about how Airflow makes this possible. Larry Komenda, Chief Technology Officer at Campbell, shares how Airflow supports their operations and improves efficiency. Larry discusses his role at Campbell, their switch to Airflow, and its impact. We look at their strategies for testing and maintaining reliable workflows and how these help their business. Key Takeaways: (02:26) Strong technology and data systems are crucial for Campbell’s investment process. (05:03) Airflow manages data pipelines efficiently in the market data team. (07:39) Airflow supports various departments, including trading and operations. (09:22) Machine learning models run on dedicated Airflow instances. (11:12) Reliable workflows are ensured through thorough testing and development. (13:45) Business tasks are organized separately from Airflow for easier testing. (15:30) Non-technical teams have access to Airflow for better efficiency. (17:20) Thorough testing before deploying to Airflow is essential. (19:10) Non-technical users can interact with Airflow DAGs to solve their issues. (21:55) Airflow improves efficiency and reliability in trading and operations. (24:40) Enhancing the Airflow UI for non-technical users is important for accessibility. Resources Mentioned: Larry Komenda - https://www.linkedin.com/in/larrykomenda/ Campbell - https://www.linkedin.com/company/campbell-and-company/ 30% off Airflow Summit Ticket - https://ti.to/airflowsummit/2024/discount/30DISC_ASTRONOMER Apache Airflow - https://airflow.apache.org/ NumPy - https://numpy.org/ Python - https://www.python.org/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
The world of timekeeping for knowledge workers is transforming through the use of AI and machine learning. Understanding how to leverage these technologies is crucial for improving efficiency and productivity. In this episode, we’re joined by Vincent La, Principal Data Scientist at Laurel, and Jim Howard, Principal Machine Learning Engineer at Laurel, to explore the implementation of AI in automating timekeeping and its impact on legal and accounting firms. Key Takeaways: (01:54) Laurel's mission in time automation. (03:39) Solving clustering, prediction and summarization with AI. (06:30) Daily batch jobs for user time generation. (08:22) Knowledge workers touch 300 items daily. (09:01) Mapping 300 activities to seven billable items. (11:38) Retraining models for better performance. (14:00) Using Airflow for retraining and backfills. (17:06) RAG-based summarization for user-specific tone. (18:58) Testing Airflow DAGs for cost-effective summarization. (22:00) Enhancing Airflow for long-running DAGs. Resources Mentioned: Vincent La - https://www.linkedin.com/in/vincentla/ Jim Howard - https://www.linkedin.com/in/jameswhowardml/ Laurel - https://www.linkedin.com/company/laurel-ai/ Apache Airflow - https://airflow.apache.org/ Ernst & Young - https://www.ey.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
Discover the cutting-edge methods Vibrant Planet uses to revolutionize geospatial data processing and resource management. In this episode, we delve into the intricacies of scaling geospatial data processing and resource allocation with experts from Vibrant Planet. Joining us are Cyrus Dukart, Engineering Lead, and David Sacerdote, Staff Software Engineer, who share their innovative approaches to handling large datasets and optimizing resource use in Airflow. Key Takeaways: (00:00) Inefficiencies in resource allocation. (03:00) Scientific validity of sharded results. (05:53) Tech-based solutions for resource management. (06:11) Retry callback process for resource allocation. (08:00) Running database queries for resource needs. (10:05) Importance of remembering resource usage. (13:51) Generating resource predictions. (14:44) Custom task decorator for resource management. (20:28) Massive resource usage gap in sharded data. (21:14) Fail-fast model for long-running tasks. Resources Mentioned: Cyrus Dukart - https://www.linkedin.com/in/cyrus-dukart-6561482/ David Sacerdote - https://www.linkedin.com/in/davidsacerdote/ Vibrant Planet - https://www.linkedin.com/company/vibrant-planet/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io/ Vibrant Planet - https://vibrantplanet.net/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning
The world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment. This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning Engineer at Astronomer. They delve into the intricacies of data orchestration, generative AI and the practical applications of these technologies in modern data workflows. Key Takeaways: (01:51) The pressure to engage in the generative AI space. (02:02) Generative AI can elevate data utilization to the next level. (02:43) The transparency issues with commercial AI models. (04:27) High-quality data in model performance is crucial. (06:40) Running new models on smaller devices, like phones. (12:19) Fine-tuning LLMs to handle millions of task failures. (16:54) Teaching AI to understand specific logs, not general passages, is a goal. (21:56) Using Airflow as a general-purpose orchestration tool. (22:00) Airflow is adaptable for various use cases, including ETL and ML systems. Resources Mentioned: Julian LaNeve - https://www.linkedin.com/in/julianlaneve/ Atronomer - https://www.linkedin.com/company/astronomer/ David Xue - https://www.linkedin.com/in/david-xue-uva/ Apache Airflow - https://airflow.apache.org/ Meta’s Open Source Llama 3 model: https://ai.meta.com/blog/meta-llama-3/https://ai.meta.com/blog/meta-llama-3/ Microsoft’s Phi-3 model: https://www.microsoft.com/en-us/research/publication/phi-3-technical-report-a-highly-capable-language-model-locally-on-your-phone/ GPT-4 - https://www.openai.com/research/gpt-4 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning
Understanding the critical role of data integration and management is essential for driving business success, particularly in a dynamic environment like a luxury casino resort. In this episode, we sit down with Siva Krishna Yetukuri, Cloud Data Architect at Wynn Las Vegas, to explore how Airflow and other tools are transforming data workflows and customer experiences at Wynn Las Vegas. Key Takeaways: (02:00) Siva designs and builds cutting-edge data pipelines and architectures. (02:54) Wynn is building a data platform to drive surveys and marketing strategies. (05:00) Airflow is the backbone of data ingestion, curation and integration. (07:00) Custom operators in Airflow enhance monitoring and reporting. (09:00) Excitement surrounds the use of Airflow 2.9 and its new features. (08:32) A metadata database drives Airflow workflows and captures metrics. (12:31) Understanding Airflow fundamentals in layman’s terms simplifies complexity. (16:33) Transitioning from Control-M to Airflow eases building complex workflows. (24:06) ML models for volume and freshness anomalies improve data quality. (20:15) DAGs are often auto-generated, simplifying the process for engineers. Resources Mentioned: Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Databricks - https://databricks.com/ Great Expectations - https://greatexpectations.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning
The integration of data and AI in sports is transforming how teams strategize and perform. Understanding how to harness this technology is key to staying competitive in the rapidly evolving landscape of baseball. In this episode, we sit down with Alexander Booth, Assistant Director of Research and Development at Texas Rangers Baseball Club, to explore the intersection of big data, AI and baseball strategy. Key Takeaways: (03:00) Alexander Booth's role and responsibilities at the Texas Rangers. (03:33) The implementation of multiple cameras and pose tracking in stadiums. (06:16) The importance of Airflow in organizing data orchestrations. (06:22) The demand for faster data among modern baseball players. (11:01) The necessity of scalable solutions for handling large data sets. (15:00) How weather data influences game strategy. (15:46) The impact of advanced technology on decision-making in baseball. (18:00) The role of AI and machine learning in player and game analysis. (22:26) The use of dynamic tasks in Airflow for better data management. Resources Mentioned: Apache Airflow - https://airflow.apache.org/ Statcast - https://www.mlb.com/statcast Google BigQuery - https://cloud.google.com/bigquery/ Databricks - https://databricks.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning
Welcome back to the Airflow Podcast. This week, we met up with Ben Wisegarver, a staff data scientist at Reddit who runs their data warehousing and data engineering functions. Reddit users generate petabytes of data every day that needs to be processed, stored, and analyzed by a wide breadth of backend services. Our conversation with Ben touches on everything from Airflow as a tool for career mobility across the data stack to scaling out a self-service data architecture across many teams. For folks interested, our team at Astronomer is growing rapidly and we're on the hunt for new folks to join in a variety of different roles. If you're passionate about Airflow and interested in building the future of data engineering, please get in touch. You can check our current job postings at careers.astronomer.io, but we're constantly updating our listings to accommodate new hiring needs. Please feel free to email me directly at pete@astronomer.io if you're passionate about what we're doing and think you'd be a good addition to the team. Mentioned Resources: Careers: https://careers.astronomer.io Guest Profile: Ben Wisegarver: https://www.linkedin.com/in/ben-wisegarver-54566576
Welcome back to the Airflow Podcast. This week, we met up with Albert Franzi and Carlos Escura from Typeform. Typeform is a tool that allows you to build beautiful interactive forms that you can use for a wide variety of use cases, including customer surveys, employee engagement, product feedback, and market research to name a few. In our conversation, we discussed Airflow as a tool for GDPR compliance, the concept of self-service data and how it allows your data operations team to function as a data platform team, and some of the more specialized infrastructure tooling that the Typeform team has built out to support their internal teams. For folks interested, our team at Astronomer is growing rapidly and we're on the hunt for new folks to join in a variety of different roles. If you're passionate about Airflow and interested in building the future of data engineering, please get in touch. You can check our current job postings at careers.astronomer.io, but we're constantly updating our listings to accommodate new hiring needs. Please feel free to email me directly at pete@astronomer.io if you're passionate about what we're doing and think you'd be a good addition to the team. Mentioned Resources: Dag Factory: https://github.com/ajbosco/dag-factory Astronomer Careers: https://careers.astronomer.io Guest Profiles: Albert Franzi: https://www.linkedin.com/in/albertfranzi/?originalSubdomain=es Carlos Escura: https://www.linkedin.com/in/carlosescura/en-us/
After a bit of a break, we're back with the third official episode bundle of The Airflow Podcast. In this batch, we'll get a little bit deeper with current Airflow users and maintainers on core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating Airflow, specifically as we go through the release process of Airflow 2.0. This week, we met up with Brian de la Motte and Florian Hines at Netlify. Netlify provides an extremely popular toolset for building and deploying JAMstack sites. They provide hosting services, CI, DNS, authentication, and managed backend tools that help users run and operate static sites at scale. The team over there recently adopted Airflow to help decouple orchestration logic from a complex collection Spark jobs and are currently in the process of expanding their Airflow footprint to accommodate a broader group of interesting use-cases. Disclaimer: we get a bit of a surprise about halfway through the episode when Brian tells us that they had recently signed up for Astronomer- we promise that it wasn't a planted ad :). Please contact pete@astronomer.io if you'd like to get in touch regarding future episodes. Hope you enjoy! Guest Profiles: Brian de la Motte: https://www.linkedin.com/in/brian-de-la-motte/ Florian Hines: https://www.linkedin.com/in/florianhines/
This week, we linked up with Airflow release manager, core committer, and Astronomer platform engineer Ash Berlin-Taylor to discuss the Airflow 2.0 roadmap [1]. There is some great stuff in the works around performance, autoscaling, and usability that we're excited about. In this episode, Ash lends his thoughts on the design, implementation, and value-add around all of the upcoming features, including: - The Knative Executor - A modern and real-time UI - A production-grade API - Improved scheduler and webserver performance - An official production Docker image for Airflow We hope you enjoy! Please email pete@astronomer.io if you have thoughts on topics you'd like to see covered in future episodes. Separately, some good folks from the Airflow community are running a user survey that will help collect some useful information around the Airflow UX. If you have five minutes to spare, filling out the following form will help the core Airflow committers to shape the project roadmap: https://forms.gle/XAzR1pQBZiftvPQM7 [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
This week, we had the pleasure of meeting up with Jarek Potiuk, Principal Software Engineer at Polidea and Apache Airflow committer, to discuss his most recent contribution to the community, Airflow Breeze. Jarek deeply values developer productivity and realized while building a team of Airflow committers that, in order to open a PR on the project, passing unit tests and waiting for the CI build was a cumbersome process that could take up to a few hours. Breeze seeks to improve that experience for Airflow committers and lower the barrier-to-entry of contribution for folks that are new to the open-source community. You can read more about Airflow Breeze here: https://www.polidea.com/blog/its-a-breeze-to-develop-apache-airflow/#the-apache-airflow-projects-setup
This episode kicks off season 2 of The Airflow Podcast. In this next season, we'll focus on the future of Airflow and chat with leading members of the community to paint a picture of what's to come. We're pumped to be diving back into this project and look forward to the great conversations we have lined up. This week, we chatted with James Malone, Product Manager of Google's Cloud Composer. James had some interesting things to say about open source at Google and where his team plans on contributing most to the project going forward. As always, thanks for listening and please email pete@astronomer.io if you have any feedback or would like to be considered as a guest.
This week, we met up with Ash Berlin-Taylor to discuss the recent 1.10 release, what it's like to be a release manager for an open source project, Airflow's bid to graduate from incubating status, and the next phase of Airflow project development. As mentioned in our podcast intro, we at Astronomer are hiring Data Engineers who are passionate about contributing to open source and making Airflow great. Please shoot us an email at humans@astronomer.io if you're interested in hearing more about the fully-remote opportunity. Check us out at www.astronomer.io if you're interested in seeing a demo of our platform.
This time, we met up with WePay's Joy Gao to talk through her work on the RBAC components in the recent Airflow 1.10 release. We dove deep into what inspired her work and took some time to discuss what it's like to be a woman contributing to a predominately male open-source community. Hope you enjoy! If you'd like to get started using Airflow in your org, check out our recently-launched Spacecamp program here: https://www.astronomer.io/spacecamp Feel free to email me at pete@astronomer.io with any feedback or if you'd like to be considered as a guest!
In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, so it was really interesting to hear both of their perspectives on the project. Enjoy!
This week, we’ll examine conversations with both old guests and new to paint a comprehensive picture of Airflow’s pain points. While we still undoubtedly believe that Airflow is the future of ETL, it’s important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community’s attention will help shape the future of the project. We’ll talk with Thomas La Piana (1:36), Data Engineer at OrderMyGear, Frank Hsu (14:20), Data Engineer at mines.io, and Alan Cruickshank (27:41), business insights and data manager at tails.com. Check out our open-source library of Airflow plugins at github.com/airflow-plugins, and feel free to contribute anything that you've been working on! If you're interested in being on the podcast or have any feedback on how you think we could make it better, shoot me an email at pete@astronomer.io
On this episode, we linked up with Erik Bernhardsson (@erikbern), creator of Luigi and CTO of Better Mortgage. We chatted about everything from the motivations behind Luigi's creation and his current thoughts on Airflow- we hope you enjoy! Check out: - Erik's blog at erikbern.com - Our open-source library of Airflow plugins at github.com/airflow-plugins All podcast feedback is hugely appreciated- feel free to email me at pete@astronomer.io if you have any thoughts.
In this episode, we dive into Airflow Best Practices and include longer portions of interviews with Alan Cruickshank (1:30), Business Insights and Data Manager at Tails.com, Chris Riccomini (7:27), Principal Software Engineer at WePay, and Bolke de Bruin(31:45), Head of Advanced Analytics Technology at ING. Hope you enjoy! We're still working to get better at podcasting, so please send over any feedback to pete@astronomer.io. We really appreciate hearing what the community has to say, and your feedback is hugely helpful in making us better. If you're interested in Astronomer Spacecamp, a guided Airflow development course, you can find more info on that here: https://www.astronomer.io/blog/announcing-astronomer-spacecamp/ We also launched our Managed Airflow on Product Hunt last week- you can check that out here: https://www.producthunt.com/posts/apache-airflow-on-astronomer Thanks so much for listening!
Episode 2 of The Airflow Podcast is here to discuss six specific use cases that we’ve seen for Apache Airflow. Here’s the lineup: Patrick Atwater (@patwater), Water Data Projects Manager at ARGO Labs: 2:03-5:35 Maksime Pecherskiy (@mrmaksimize), CDO of San Diego: 5:35-23:06 Scott Halgrim (@shalgrim), Data Engineer at Zapier: 23:06-27:27 Bolke de Bruin (@bolke2028), Head of Advanced Analytics at ING: 27:27-39:46 Chris Riccomini (@criccomini), Principal Software Engineer at WePay: 39:46-54:20 Ben Gregory (@benbeingbin), Data Engineer (and noted craft soda enthusiast) at Astronomer: 54:20-1:14:38 Contribute to our open-source library of Airflow plugins at github.com/airflow-plugins Contact us at www.astronomer.io if you’re interested in Spacecamp: A guided development program to get your team up and running on Airflow.
For the first episode of the Airflow Podcast, we met up with Maxime Beauchemin, creator of Airflow, to explore the motivations behind its creation and the problems it was designed to solve. We asked Maxime for his definition of Airflow, the design principles behind hook/operator use, and his vision for the project. Speaker list: Pete DeJoy - Product at Astronomer Viraj Parekh - Data Engineer at Astronomer Maxime Beauchemin - Software Engineer at Lyft, creator of Airflow Talk mentioned at the end of the podcast- Advanced Data Engineering Patterns with Apache Airflow: http://www.ustream.tv/recorded/109227704 Maxime's Blog: https://medium.com/@maximebeauchemin
A sneak peek at our upcoming podcast about Apache Airflow. Featured in this clip (in order of appearance): Pete DeJoy - Product Specialist at Astronomer Patrick Atwater - Water Data Projects Manager at ARGO Labs Maksime Pecherskiy - Chief Data Officer of the City of San Diego Bolke de Bruin - Head of Advanced Analytics at ING
Our TOPPODCAST Picks 
