The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Latest Episodes:

Airflow on Kubernetes Jun 29, 2018

Show notes

In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, so it was really interesting to hear both of their perspectives on the project. Enjoy!

Pain Points May 31, 2018

Show notes

This week, we’ll examine conversations with both old guests and new to paint a comprehensive picture of Airflow’s pain points. While we still undoubtedly believe that Airflow is the future of ETL, it’s important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community’s attention will help shape the future of the project. We’ll talk with Thomas La Piana (1:36), Data Engineer at OrderMyGear, Frank Hsu (14:20), Data Engineer at mines.io, and Alan Cruickshank (27:41), business insights and data manager at tails.com. Check out our open-source library of Airflow plugins at github.com/airflow-plugins, and feel free to contribute anything that you've been working on! If you're interested in being on the podcast or have any feedback on how you think we could make it better, shoot me an email at pete@astronomer.io

Competitors: Luigi Apr 13, 2018

Show notes

On this episode, we linked up with Erik Bernhardsson (@erikbern), creator of Luigi and CTO of Better Mortgage. We chatted about everything from the motivations behind Luigi's creation and his current thoughts on Airflow- we hope you enjoy! Check out: - Erik's blog at erikbern.com - Our open-source library of Airflow plugins at github.com/airflow-plugins All podcast feedback is hugely appreciated- feel free to email me at pete@astronomer.io if you have any thoughts.

Best Practices Mar 21, 2018

Show notes

In this episode, we dive into Airflow Best Practices and include longer portions of interviews with Alan Cruickshank (1:30), Business Insights and Data Manager at Tails.com, Chris Riccomini (7:27), Principal Software Engineer at WePay, and Bolke de Bruin(31:45), Head of Advanced Analytics Technology at ING. Hope you enjoy! We're still working to get better at podcasting, so please send over any feedback to pete@astronomer.io. We really appreciate hearing what the community has to say, and your feedback is hugely helpful in making us better. If you're interested in Astronomer Spacecamp, a guided Airflow development course, you can find more info on that here: https://www.astronomer.io/blog/announcing-astronomer-spacecamp/ We also launched our Managed Airflow on Product Hunt last week- you can check that out here: https://www.producthunt.com/posts/apache-airflow-on-astronomer Thanks so much for listening!

Use Cases Feb 22, 2018

Show notes

Episode 2 of The Airflow Podcast is here to discuss six specific use cases that we’ve seen for Apache Airflow. Here’s the lineup: Patrick Atwater (@patwater), Water Data Projects Manager at ARGO Labs: 2:03-5:35 Maksime Pecherskiy (@mrmaksimize), CDO of San Diego: 5:35-23:06 Scott Halgrim (@shalgrim), Data Engineer at Zapier: 23:06-27:27 Bolke de Bruin (@bolke2028), Head of Advanced Analytics at ING: 27:27-39:46 Chris Riccomini (@criccomini), Principal Software Engineer at WePay: 39:46-54:20 Ben Gregory (@benbeingbin), Data Engineer (and noted craft soda enthusiast) at Astronomer: 54:20-1:14:38 Contribute to our open-source library of Airflow plugins at github.com/airflow-plugins Contact us at www.astronomer.io if you’re interested in Spacecamp: A guided development program to get your team up and running on Airflow.

The Origins of Airflow Feb 06, 2018

Show notes

For the first episode of the Airflow Podcast, we met up with Maxime Beauchemin, creator of Airflow, to explore the motivations behind its creation and the problems it was designed to solve. We asked Maxime for his definition of Airflow, the design principles behind hook/operator use, and his vision for the project. Speaker list: Pete DeJoy - Product at Astronomer Viraj Parekh - Data Engineer at Astronomer Maxime Beauchemin - Software Engineer at Lyft, creator of Airflow Talk mentioned at the end of the podcast- Advanced Data Engineering Patterns with Apache Airflow: http://www.ustream.tv/recorded/109227704 Maxime's Blog: https://medium.com/@maximebeauchemin

Season One Teaser Jan 18, 2018

Show notes

A sneak peek at our upcoming podcast about Apache Airflow. Featured in this clip (in order of appearance): Pete DeJoy - Product Specialist at Astronomer Patrick Atwater - Water Data Projects Manager at ARGO Labs Maksime Pecherskiy - Chief Data Officer of the City of San Diego Bolke de Bruin - Head of Advanced Analytics at ING

Our TOPPODCAST Picks

Follow Us

Stay Connected

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Related Podcasts

Links

Stay Connected