The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Latest Episodes:

Orchestrating Retail Data Pipelines at Saks Global Jul 16, 2026

Show notes

Saks Global runs one of the largest retail data operations in the US, with around 8 million SKUs flowing across point-of-sale, e-commerce, catalog, and fraud detection systems into Snowflake. In this episode, [Shailesh Kadam](linkedin.com), Architect at [Saks Global](saks.com), joins Kenten to walk through how Airflow acts as the nervous system tying it all together, why they moved from self-managed Kubernetes to Astro, what is driving their Airflow 3 upgrade, and how they are approaching agentic AI, MCP, and credential security.

Key Takeaways:

00:00 Introduction.
01:31 Saks Global today. Shailesh describes the business after separating e-commerce from brick and mortar and acquiring Neiman Marcus, and the modern cloud-native stack on AWS, Snowflake, and Airflow.
02:50 8 million SKUs in motion. Why every name, image, inventory, and price change has to flow in near real time across operational systems.
04:50 What the pipelines look like. Point-of-sale ingestion, fraud signals to third parties like Fiserv, and hourly product catalog feeds out to Meta and Google.
07:30 Moving off self-managed Kubernetes to Astro. Shailesh contrasts past experience with Kubernetes, IBM Tivoli, and Control-M against running on Astro.
09:35 Upgrading to Airflow 3. Event and asset-based scheduling, DAG versioning, task isolation, and using Otto to convert DAGs in a phased rollout.
13:13 Agentic AI and MCP on the roadmap. How Saks plans to use Airflow's MCP for LLM-driven product classification and to feed Snowflake analyses like churn and spend.
18:01 Securing PII and credentials. Secrets backends, cloud secret manager integration, key rotation, and keeping credentials out of DAG code.
21:02 Wishlist for Airflow. Interactive data lineage across DAGs and a UI-based debugging interface for support teams.

Resources Mentioned:

[Apache Airflow](airflow.apache.org)
[Astro](astronomer.io/product)
[Otto, the Astronomer data engineering agent](astronomer.io)
[Snowflake](snowflake.com)
[Saks Global](saks.com)

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

What's New in Apache Airflow® 3.3 Jul 09, 2026

Show notes

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at [Astronomer](astronomer.io), joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.

Key Takeaways:

00:00 Introduction.
01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.

Resources Mentioned:

#AI #Automation #Airflow

Running Airflow 3 in a regulated environment at OTPP Jun 25, 2026

Show notes

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at [Ontario Teachers' Pension Plan](otpp.com), joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.

Key Takeaways:

00:00 Introduction.
01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.

Resources Mentioned:

[Ontario Teachers' Pension Plan](otpp.com)
[Apache Airflow](airflow.apache.org)
[Astronomer](astronomer.io)
[Cosmos](astronomer.io/cosmos)

#AI #Automation #Airflow

Managing a Customer Analytics Platform with Airflow at Skimlinks Jun 11, 2026

Show notes

Skimlinks runs a reporting platform that serves around 2,000 weekly publisher users, and the data infrastructure behind it runs on Airflow. In this episode, Julian Larralde, Director of Data Engineering at Skimlinks, walks through the stack, the migration from external task sensors to event-driven Assets, and a YAML-based DAG factory the team built to onboard new publishers without rewriting Python.

Key Takeaways

00:00 Introduction.
00:45 What Skimlinks does and how it operates as an affiliate marketing network aggregator for publishers.
02:12 Julian's team and the data platform they own: a reporting portal that serves ~2,000 weekly publisher users.
03:07 The stack: real-time ingestion into BigQuery, Airflow as the orchestrator, raw / silver / gold layers, and Apache Druid as the serving database for sub-second BI queries.
04:50 Reusing the same data marts for ~100 internal customers across marketing, finance, operations, and account management.
06:25 Airflow as the single orchestrator: BigQuery operators for SQL business logic, plus raw file exports for the largest publishers.
08:08 Moving from external task sensors to datasets (now Assets) and what the migration actually solved.
09:18 Why sensor polling created scheduler load and worker overload, and how event-driven Assets fixed both.
10:15 The lineage view in the Airflow UI that came as a bonus after the Assets migration.
10:49 The vision for multi-tenant Airflow inside Skimlinks: replacing cron, Rundeck, and team-local Airflow instances with a shared platform.
14:31 Building a custom DAG factory with YAML configuration for onboarding new publishers.
17:33 Breaking a single Python class into single-responsibility components for the DataPipe project.
19:07 Adding a Pydantic layer so misconfigured YAML fails at DAG parse time instead of run time.
20:31 Using AI assistance to guide refactoring decisions and generate tests across the new class structure.
22:34 What Julian wants from Airflow next: asset watchers paired with data contracts.

Resources Mentioned

Skimlinks - skimlinks.com
Apache Airflow - airflow.apache.org
Astronomer - astronomer.io
Google BigQuery - cloud.google.com/bigquery
Apache Druid - druid.apache.org
Pydantic - docs.pydantic.dev
Looker - cloud.google.com/looker

#AI #Automation #Airflow

Building a custom Tableau provider for Airflow at JLR Jun 04, 2026

Show notes

JLR is the UK's largest automotive manufacturer, behind brands like Range Rover, Jaguar, Defender, and Discovery. In this episode, Najeeb Sulaiman, Senior Data Engineer at JLR, walks through how Airflow orchestrates data across manufacturing, supply chain, and finance — including a custom Tableau provider his team built (after the community version dropped PAT authentication) and a CI/CD pipeline that validates DAGs before they reach production.

Key Takeaways:

00:00 Introduction.
00:48 What JLR makes: luxury vehicles under the Range Rover, Jaguar, Defender, and Discovery brands.
01:42 Najeeb's team in the Data and AI Office, supporting manufacturing, supply chain, finance, and commerce analytics.
03:25 Airflow as the central nervous system of the JLR data stack — the orchestrator that connects every source and downstream system.
05:01 How JLR uses Tableau, and the two modes for getting data in: live connection and scheduled extract refresh.
06:24 Why scheduled Tableau refreshes go stale: they aren't aware of when the data pipeline actually finished.
08:09 First attempt at solving it: Python scripts calling the Tableau REST API directly.
08:47 Why the script approach didn't scale across teams — code duplication and version drift.
10:00 Trying the community Airflow Tableau provider and hitting the PAT authentication roadblock.
12:21 Building a custom provider on top of the community one to keep PAT auth.
13:30 Treating CI/CD as a deployment gate for Airflow DAGs at JLR's scale.
15:23 What the CI/CD pipeline actually catches: top-level code making external calls, import errors, and Airflow 3 compatibility.
17:47 How the gate blocks broken DAGs from reaching production.
18:30 What Najeeb wants from Airflow next: native integration testing, better OpenTelemetry support, and built-in lineage.

Resources Mentioned:

JLR - jaguarlandrover.com
Apache Airflow - airflow.apache.org
Astronomer - astronomer.io
Tableau - tableau.com
Tableau REST API - help.tableau.com/current/api/rest_api/en-us/REST/rest_api.htm
Airflow Tableau provider (community) - airflow.apache.org/docs/apache-airflow-providers-tableau

#AI #Automation #Airflow

Orchestrating 2,000 Airflow pipelines at Luiza Labs with Mateus Ferreira May 28, 2026

Show notes

Running Airflow at the scale of a national retailer means more than just scheduling. It means giving non-engineers a path to ship DAGs, and classifying thousands of runs to know which ones need attention. In this episode, Mateus Ferreira, Senior Data Engineer at Luiza Labs (the technology arm of Magazine Luiza, one of Brazil's largest retailers), joins Marc to talk about the patterns his team uses to run 2,000+ Airflow pipelines across more than four petabytes of data.

Key Takeaways:

00:00 Introduction
01:11 Mateus introduces himself and Luiza Labs, the technology arm of Magazine Luiza (Magalu), one of Brazil's largest retailers (founded 1957). 1,000+ physical stores, multi-region operations, and a data team that has to handle the variability that comes with all of it.
04:33 Lu Brain, Magalu's AI initiative built around their character Lu, and how AI fits into the data work.
06:47 The data reliability engineering channel where AI summarizes Airflow errors with confidence scores and posts a suggested fix in chat.
08:30 How Airflow became the heart of orchestration. Coming from Control-M in banking, then GCP, then consolidating on Cloud Composer to centralize roughly 2,000 pipelines.
14:23 The YAML wrapper that lets non-engineers ship DAGs. Reads namespace, tables, and Spark options. Handles CDC, JDBC full, and JDBC incremental collection types with checkpoints. All changes go through data reliability engineering.
17:20 Why metadata is the most valuable asset in the AI era, and how the wrapper makes data lineage observable across 2,000 pipelines.
18:26 The Data Reliability Engineering team. A 10-person group that is the window to the company, handling maintenance, validation, corrections, and optimization for the business unit pipelines.
20:09 Operating at four petabytes of data.
21:24 Why they built custom Spark operators. Cost drove the move off the DataprocOperator. The custom operator exposes Spark driver and executor sizing as Airflow parameters and generates the Kubernetes manifest.
24:36 The monitoring dashboard built on the Airflow metadata DB. A timeline view that shows how many DAGs run each hour, used to spread scheduling across the day.
26:37 Classifying DAGs by their last five runs: success, partially correct, intermittent, total failure. A reusable observability pattern.
29:57 How to reach Mateus, and a closing thought in Portuguese on appreciating the good old times while you are living them.

Resources Mentioned:

Apache Airflow (airflow.apache.org)
Magalu Cloud / MGC
Luiza Labs (luizalabs.com) and Magazine Luiza / Magalu
Astro Observe (https://www.astronomer.io/product)
Mateus Ferreira on LinkedIn (linkedin.com/in/mateusmferreira)

#AI #Automation #Airflow

Enhancing DAGs for Data Processing with William Orgertrice III at Cargill May 21, 2026

Show notes

In the data engineering world, the difference between a pipeline that works and one that's truly production-ready often comes down to a handful of deliberate decisions. William Orgertrice III, Data Engineer at Cargill, joins us to share the DAG design and monitoring practices he presented at Airflow Summit 2025 and how his team is rolling out Airflow across 60+ internal teams as part of Cargill's new Minerva data platform.

Key Takeaways:

00:00 Introduction.

01:45 Cargill is one of the largest privately owned companies in the US, operating across 70 countries and serving 125+ markets.

03:45 William's team on the Cargill Data Platform supports 60+ internal teams, providing data products that drive decisions across finance, inventory and operations.

05:10 Cargill chose Airflow as a core component of its new Minerva data platform to replace older ETL tooling with a more supportable, observable stack.

06:26 Native SLA sensors and dependency management were specific features that made Airflow the right fit for Cargill's batch ingestion pipelines.

09:00 Cargill is running Airflow through Astronomer as their managed solution, with some teams already in production.

13:22 Every task in a DAG should have a single, documented purpose — one task doing everything makes troubleshooting significantly harder.

14:40 A DAG that never enters a failed state but keeps running indefinitely will spend compute budget without alerting anyone.

15:25 In shared Airflow environments, embedding contact information and owner tags in DAGs ensures the right team is reached when something breaks upstream.

21:00 William flags connection testing as a friction point in pipeline development — verifying a connection string before building the full job would reduce iteration time.

Resources Mentioned:

Cargill | Website

https://www.cargill.com/food-beverage

Airflow Community on Slack

https://airflow.apache.org/community/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow

Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer May 14, 2026

Show notes

In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.

Shrividya Hegde, a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong.

Key Takeaways:

00:00 Introduction.

04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first.

07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries.

08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem.

13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output.

15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable.

17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code.

23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions.

Resources Mentioned:

Shrividya Hegde

https://www.linkedin.com/in/shrividya-hegde-shri-91562365/

Astronomer | LinkedIn

https://www.linkedin.com/company/astronomer/

Astronomer | Website

https://www.astronomer.io

Women in Data | Website

https://womenindata.mn.co/landing

Apache Airflow Slack

https://airflow.apache.org/

Shrividya's Medium writing

https://medium.com/@shrihegde

Shrividya’ Substack writing

https://substack.com/@shrividyahegde

#AI #Automation #Airflow #MachineLearning

Orchestrating DBT With Cosmos and Airflow with Filip Kunčar at ShipMonk Product Development May 07, 2026

Show notes

We explore how a third-party logistics platform built its entire data orchestration layer on Airflow, and what that makes possible for developer teams and merchant-facing products alike.

Filip Kunčar, Platform Director at ShipMonk Product Development, discusses migrating from a closed source tool to Airflow, orchestrating dbt with both Cosmos and the BashOperator and using Airflow to power customer-facing data delivery.

Key Takeaways:

00:00 Introduction.

01:07 ShipMonk is a third-party logistics company guaranteeing two-day delivery across the US. The data platform team's mission is to lower cognitive load for developers working with data.

05:13 ShipMonk migrated to Airflow in 2022, moving away from a closed-source UI-based tool, driven by the need for a code-first approach, open source extensibility and broad cloud provider support.

10:02 The team uses Cosmos for developer-facing visibility and lineage and BashOperator for internal pipelines where runtime performance matters.

12:20 Switching from Cosmos to the BashOperator for a frequently running pipeline reduced runtime from over 15 minutes to three minutes.

13:14 Because the full dbt chain runs inside Airflow, a configurable downstream DAG can deliver processed data directly to each merchant's preferred destination, with secrets management and SLA tracking already handled.

15:03 Per-team alerting is hooked to each DAG by owner and severity, so teams can react to SLA breaches immediately.

18:09 ShipMonk uses Airflow in three ways for AI: authoring DAGs faster with skills, orchestrating AI workloads in Lambda and containers and using Astronomer's skills repo to simplify Airflow version upgrades.

Resources Mentioned:

Filip Kunčar

https://www.linkedin.com/in/filipkuncar/

ShipMonk Product Development

https://www.linkedin.com/company/shipmonk-product-development/

ShipMonk | Website

http://www.shipmonk.com

Astronomer Cosmos

http://www.astronomer.io/cosmos

Astronomer AI Skills Repo

http://www.github.com/astronomer/airflow-llm-providers-demo

Datadog

http://www.datadoghq.com

#AI #Automation #Airflow #MachineLearning

Building Airflow CTL with Buğra Öztürk at Mollie Apr 30, 2026

Show notes

Buğra Öztürk, Senior Data Engineer at Mollie and Committer and PMC member on the Apache Airflow project, joins us to walk through Airflow CTL — what it is, how it differs from the existing Airflow CLI and where it is headed under AIP-94.

Key Takeaways:

00:00 Introduction.

03:10 Buğra has contributed to Airflow since 2022, from docs changes up to Committer and PMC member — a path he hopes inspires others to start small and contribute.

04:05 Airflow CTL solves secure user interaction by abstracting database credentials behind the public core API.

05:13 Airflow CLI and Airflow CTL are complementary — CLI handles administration and database management while CTL handles secure user interactions via the API.

07:08 Airflow CTL authenticates via the API, acquires a JWT token and stores it securely in the OS keyring — running on the user's machine and never requiring direct database access.

08:21 Concrete use cases include local DAG development without the UI and CI/CD automation using headless mode with short-lived JWT tokens.

10:08 AIP-94 describes the long-term vision — decoupling all remote commands from the Airflow CLI and routing them through Airflow CTL.

13:12 Airflow CTL is currently at 0.X and already being used in CI and deployment automations. The move to 1.0 with full CLI parity is the next milestone under AIP-94.

16:09 Multi-team deployment becoming generally available in a future Airflow release is Buğra's most-anticipated upcoming feature beyond Airflow CTL.

Resources Mentioned:

Buğra Öztürk

https://www.linkedin.com/in/bugraozturk93/

Mollie

https://www.linkedin.com/company/mollie/

Mollie | Website

https://www.mollie.com/

Apache Airflow CTL

https://airflow.apache.org/

AIP-94 on Airflow Confluence

https://lists.apache.org/thread/d2o1pr78wxdp1wozq519stp0pkcv6k6c

Apache Airflow GitHub