TopPodcast.com
Menu
  • Home
  • Top Charts
  • Top Networks
  • Top Apps
  • Top Independents
  • Top Podfluencers
  • Top Picks
    • Top Business Podcasts
    • Top True Crime Podcasts
    • Top Finance Podcasts
    • Top Comedy Podcasts
    • Top Music Podcasts
    • Top Womens Podcasts
    • Top Kids Podcasts
    • Top Sports Podcasts
    • Top News Podcasts
    • Top Tech Podcasts
    • Top Crypto Podcasts
    • Top Entrepreneurial Podcasts
    • Top Fantasy Sports Podcasts
    • Top Political Podcasts
    • Top Science Podcasts
    • Top Self Help Podcasts
    • Top Sports Betting Podcasts
    • Top Stocks Podcasts
  • Podcast News
  • About Us
  • Podcast Advertising
  • Contact
Not in our directory?
Add Show Here
Podcast Equipment
Center

toppodcastlogoOur TOPPODCAST Picks

  • Comedy
  • Crypto
  • Sports
  • News
  • Politics
  • True Crime
  • Business
  • Finance

Follow Us

toppodcastlogoStay Connected

    View Top 200 Chart
    Back to Rankings Page
    Software How-To

    Data Engineering Podcast

    This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

    Advertise

    Copyright: © 2024 Boundless Notions, LLC.

    • Apple Podcasts
    • Google Play
    • Spotify
    Latest Episodes:
    Hire And Scale Your Data Team With Intention Jun 13, 2022
    Show notes

    Summary

    Building a well rounded and effective data team is an iterative process, and the first hire can set the stage for future success or failure. Trupti Natu has been the first data hire multiple times and gone through the process of building teams across the different stages of growth. In this episode she shares her thoughts and insights on how to be intentional about establishing your own data team.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.
    • Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
    • Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
    • Unstruk is the DataOps platform for your unstructured data. The options for ingesting, organizing, and curating unstructured files are complex, expensive, and bespoke. Unstruk Data is changing that equation with their platform approach to manage your unstructured assets. Built to handle all of your real-world data, from videos and images, to 3d point clouds and geospatial records, to industry specific file formats, Unstruk streamlines your workflow by converting human hours into machine minutes, and automatically alerting you to insights found in your dark data. Unstruk handles data versioning, lineage tracking, duplicate detection, consistency validation, as well as enrichment through sources including machine learning models, 3rd party data, and web APIs. Go to dataengineeringpodcast.com/unstruk today to transform your messy collection of unstructured data files into actionable assets that power your business.
    • Your host is Tobias Macey and today I’m interviewing Trupti Natu about strategies for building your team, from the first data hire to post-acquisition

    Interview

    • Introduction
    • How did you get involved in the area of FinTech & Data Science (management)?
    • How would you describe your overall career trajectory in data?
    • Can you describe what your experience has been as a data professional at different stages of company growth?
    • What are the traits that you look for in a first or second data hire at an organization?
      • What are useful metrics for success to help gauge the effectiveness of hires at this early stage of data capabilities?
    • What are the broad goals and projects that early data hires should be focused on?
      • What are the indicators that you look for to determine when to scale the team?
    • As you are building a team of data professionals, what are the organizational topologies that you have found most effective? (e.g. centralized vs. embedded data pros, etc.)
    • What are the recruiting and screening/interviewing techniques that you have found most helpful given the relative scarcity of experienced data practitioners?
    • What are the organizational and technical structures that are helpful to establish early in the organization’s data journey to reduce the onboarding time for new hires?
    • Your background has primarily been in FinTech. How does the business domain influence the types of background and domain expertise that you look for?
    • You recently went through an acquisition at the startup you were with. Can you describe the data-related projects that were required during the merger?
      • What are the impedance mismatches that you have had to resolve in your data systems, moving from a fast-moving startup into a larger, more established organization?
      • Being a FinTech company, what are some of the categories of regulatory considerations that you had to deal with during the integration process?
    • What are the most interesting, unexpected, or challenging lessons that you have learned along your career journey?
    • What are some of the pieces of advice that you wished you knew at the beginning of your career, and that you would like to share with others in that situation?

    Contact Info

    • LinkedIn
    • @truptinatu on Twitter
    • Trupti is hiring for multiple product data science roles. Feel free to DM her on Twitter or LinkedIn to find out more

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • SumoLogic
    • FinTech
    • PII == Personally Identifiable Information

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Simplify Data Security For Sensitive Information With The Skyflow Data Privacy Vault Jun 06, 2022
    Show notes

    Summary

    The best way to make sure that you don’t leak sensitive data is to never have it in the first place. The team at Skyflow decided that the second best way is to build a storage system dedicated to securely managing your sensitive information and making it easy to integrate with your applications and data systems. In this episode Sean Falconer explains the idea of a data privacy vault and how this new architectural element can drastically reduce the potential for making a mistake with how you manage regulated or personally identifiable information.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking all of that information into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how you can take advantage of active metadata and escape the chaos.
    • Modern data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days or even weeks. By the time errors have made their way into production, it’s often too late and damage is done. Datafold built automated regression testing to help data and analytics engineers deal with data quality in their pull requests. Datafold shows how a change in SQL code affects your data, both on a statistical level and down to individual rows and values before it gets merged to production. No more shipping and praying, you can now know exactly what will change in your database! Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Visit dataengineeringpodcast.com/datafold today to book a demo with Datafold.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Sean Falconer about the idea of a data privacy vault and how the Skyflow team are working to make it turn-key

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Skyflow is and the story behind it?
    • What is a "data privacy vault" and how does it differ from strategies such as privacy engineering or existing data governance patterns?
    • What are the primary use cases and capabilities that you are focused on solving for with Skyflow?
      • Who is the target customer for Skyflow (e.g. how does it enter an organization)?
    • How is the Skyflow platform architected?
      • How have the design and goals of the system changed or evolved over time?
    • Can you describe the process of integrating with Skyflow at the application level?
    • For organizations that are building analytical capabilities on top of the data managed in their applications, what are the interactions with Skyflow at each of the stages in the data lifecycle?
    • One of the perennial problems with distributed systems is the challenge of joining data across machine boundaries. How do you mitigate that problem?
    • On your website there are different "vaults" advertised in the form of healthcare, fintech, and PII. What are the different requirements across each of those problem domains?
      • What are the commonalities?
    • As a relatively new company in an emerging product category, what are some of the customer education challenges that you are facing?
    • What are the most interesting, innovative, or unexpected ways that you have seen Skyflow used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Skyflow?
    • When is Skyflow the wrong choice?
    • What do you have planned for the future of Skyflow?

    Contact Info

    • LinkedIn
    • @seanfalconer on Twitter
    • Website

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Skyflow
    • Privacy Engineering
    • Data Governance
    • Homomorphic Encryption
    • Polymorphic Encryption

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Bringing The Modern Data Stack To Everyone With Y42 Jun 06, 2022
    Show notes

    Summary

    Cloud services have made highly scalable and performant data platforms economical and manageable for data teams. However, they are still challenging to work with and manage for anyone who isn’t in a technical role. Hung Dang understood the need to make data more accessible to the entire organization and created Y42 as a better user experience on top of the "modern data stack". In this episode he shares how he designed the platform to support the full spectrum of technical expertise in an organization and the interesting engineering challenges involved.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • The most important piece of any data project is the data itself, which is why it is critical that your data source is high quality. PostHog is your all-in-one product analytics suite including product analysis, user funnels, feature flags, experimentation, and it’s open source so you can host it yourself or let them do it for you! You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. Give it a try today with their generous free tier at dataengineeringpodcast.com/posthog
    • Your host is Tobias Macey and today I’m interviewing Hung Dang about Y42, the full-stack data platform that anyone can run

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Y42 is and the story behind it?
    • How would you characterize your positioning in the data ecosystem?
    • What are the problems that you are trying to solve?
      • Who are the personas that you optimize for and how does that manifest in your product design and feature priorities?
    • How is the Y42 platform implemented?
      • What are the core engineering problems that you have had to address in order to tie together the various underlying services that you integrate?
      • How have the design and goals of the product changed or evolved since you started working on it?
    • What are the sharp edges and failure conditions that you have had to automate around in order to support non-technical users?
    • What is the process for integrating Y42 with an organization’s data systems?
      • What is the story for onboarding from existing systems and importing workflows (e.g. Airflow dags and dbt models)?
    • With your recent shift to using Git as the store of platform state, how do you approach the problem of reconciling branched changes with side effects from changes (e.g. creating tables or mutating table structures in the warehouse)?
    • Can you describe a typical workflow for building or modifying a business dashboard or activating data in the warehouse?
    • What are the interfaces and abstractions that you have built into the platform to support collaboration across roles and levels of experience? (technical or organizational)
    • With your focus on end-to-end support for data analysis, what are the extension points or escape hatches for use cases that you can’t support out of the box?
    • What are the most interesting, innovative, or unexpected ways that you have seen Y42 used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Y42?
    • When is Y42 the wrong choice?
    • What do you have planned for the future of Y42?

    Contact Info

    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Y42
    • CDTM (Center for Digital Technology and Management)
    • Meltano
      • Podcast Episode
    • Airflow
    • Singer
    • dbt
      • Podcast Episode
    • Great Expectations
      • Podcast Episode
    • Airbyte
      • Podcast Episode
    • Grouparoo
      • Podcast Episode
    • Terraform
    • OpenTelemetry
      • Podcast.__init__ Episode

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Sponsored By:

    • PostHog: ![Post Hog](https://files.fireside.fm/file/fireside-uploads/images/c/c6161a3f-a67b-48ef-b087-52f1f1573292/K-hligJW.png) PostHog is an open source, product analytics platform. PostHog enables software teams to understand user behavior – auto-capturing events, performing product analytics and dashboarding, enabling video replays, and rolling out new features behind feature flags, all based on their single open source platform. The product’s open source approach enables companies to self-host, removing the need to send data externally. Try it out today at [dataengineeringpodcast.com/posthog](https://www.dataengineeringpodcast.com/posthog)

    Support Data Engineering Podcast

    Data Cloud Cost Optimization With Bluesky Data May 30, 2022
    Show notes

    Summary

    The latest generation of data warehouse platforms have brought unprecedented operational simplicity and effectively infinite scale. Along with those benefits, they have also introduced a new consumption model that can lead to incredibly expensive bills at the end of the month. In order to ensure that you can explore and analyze your data without spending money on inefficient queries Mingsheng Hong and Zheng Shao created Bluesky Data. In this episode they explain how their platform optimizes your Snowflake warehouses to reduce cost, as well as identifying improvements that you can make in your queries to reduce their contribution to your bill.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • The most important piece of any data project is the data itself, which is why it is critical that your data source is high quality. PostHog is your all-in-one product analytics suite including product analysis, user funnels, feature flags, experimentation, and it’s open source so you can host it yourself or let them do it for you! You have full control over your data and their plugin system lets you integrate with all of your other data tools, including data warehouses and SaaS platforms. Give it a try today with their generous free tier at dataengineeringpodcast.com/posthog
    • Your host is Tobias Macey and today I’m interviewing Mingsheng Hong and Zheng Shao about Bluesky Data where they are combining domain expertise and machine learning to optimize your cloud warehouse usage and reduce operational costs

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Bluesky is and the story behind it?
      • What are the platforms/technologies that you are focused on in your current early stage?
      • What are some of the other targets that you are considering once you validate your initial hypothesis?
    • Cloud cost optimization is an active area for application infrastructures as well. What are the corollaries and differences between compute and storage optimization strategies and what you are doing at Bluesky?
    • How have your experiences at hyperscale companies using various combinations of cloud and on-premise data platforms informed your approach to the cost management problem faced by adopters of cloud data systems?
    • What are the most significant drivers of cost in cloud data systems?
      • What are the factors (e.g. pricing models, organizational usage, inefficiencies) that lead to such inflated costs?
    • What are the signals that you collect for identifying targets for optimization and tuning?
    • Can you describe how the Bluesky mission control platform is architected?
      • What are the current areas of uncertainty or active research that you are focused on?
    • What is the workflow for a team or organization that is adding Bluesky to their system?
      • How does the usage of Bluesky change as teams move from the initial optimization and dramatic cost reduction into a steady state?
    • What are the most interesting, innovative, or unexpected ways that you have seen teams approaching cost management in the absence of Bluesky?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Bluesky?
    • When is Bluesky the wrong choice?
    • What do you have planned for the future of Bluesky?

    Contact Info

    • Mingsheng
      • LinkedIn
      • @mingshenghong on Twitter
    • Zheng
      • LinkedIn
      • @zshao9 on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Bluesky Data
      • Get A Free Health Check For Your Snowflake From Bluesky
    • RocksDB
    • Snowflake
      • Podcast Episode
    • Trino
      • Podcast Episode
    • Firebolt
      • Podcast Episode
    • Bigquery
    • Hive
    • Vertica
    • Michael Stonebraker
    • Teradata
    • C-Store Paper
    • Ottertune
      • Podcast Episode
    • dbt
      • Podcast Episode
    • infracost
    • Subtract: The Untapped Science of Less by Leidy Klotz

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    A Multipurpose Database For Transactions And Analytics To Simplify Your Data Architecture With Singlestore May 30, 2022
    Show notes

    Summary

    A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required. By supporting fast, in-memory row-based queries and columnar on-disk representation, it lets your transactional and analytical workloads run in the same database. In this episode SVP of engineering Shireesh Thota describes the impact on your overall system architecture that Singlestore can have and the benefits of using a cloud-native database engine for your next application.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
    • So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Shireesh Thota about Singlestore (formerly MemSQL), the industry’s first modern relational database for multi-cloud, hybrid and on-premises workloads

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what SingleStore is and the story behind it?
    • The database market has gotten very crouded, with different areas of specialization and nuance being the differentiating factors. What are the core sets of workloads that SingleStore is aimed at addressing?
      • What are some of the capabilities that it offers to reduce the need to incorporate multiple data stores for application and analytical architectures?
    • What are some of the most valuable lessons that you learned in your time at MicroSoft that are applicable to SingleStore’s product focus and direction?
    • Nikita Shamgunov joined the show in October of 2018 to talk about what was then MemSQL. What are the notable changes in the engine and business that have occurred in the intervening time?
      • What are the macroscopic trends in data management and application development that are having the most impact on product direction?
    • For engineering teams that are already invested in, or considering adoption of, the "modern data stack" paradigm, where does SingleStore fit in that architecture?
      • What are the services or tools that might be replaced by an installation of SingleStore?
    • What are the efficiencies or new capabilities that an engineering team might expect by adopting SingleStore?
    • What are some of the features that are underappreciated/overlooked which you would like to call attention to?
    • What are the most interesting, innovative, or unexpected ways that you have seen SingleStore used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on SingleStore?
    • When is SingleStore the wrong choice?
    • What do you have planned for the future of SingleStore?

    Contact Info

    • LinkedIn
    • @ShireeshThota on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • MemSQL Interview With Nikita Shamgunov
    • Singlestore
    • MS SQL Server
    • Azure Cosmos DB
    • CitusDB
      • Podcast Episode
    • Debezium
      • Podcast Episode
    • PostgreSQL
      • Podcast Episode
    • MySQL
    • HTAP == Hybrid Transactional-Analytical Processing

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Unlocking The Value Of Data Across The Organization Through User Friendly Data Tools With Prophecy May 23, 2022
    Show notes

    Summary

    The interfaces and design cues that a tool offers can have a massive impact on who is able to use it and the tasks that they are able to perform. With an eye to making data workflows more accessible to everyone in an organization Raj Bains and his team at Prophecy designed a powerful and extensible low-code platform that lets technical and non-technical users scale data flows without forcing everyone into the same layers of abstraction. In this episode he explores the tension between code-first and no-code utilities and how he is working to balance the strengths without falling prey to their shortcomings.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
    • So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
    • Your host is Tobias Macey and today I’m interviewing Raj Bains about how improving the user experience for data tools can make your work as a data engineer better and easier

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • What are the broad categories of data tool designs that are available currently and how does that impact what is possible with them?
      • What are the points of friction that are introduced by the tools?
      • Can you share some of the types of workarounds or wasted effort that are made necessary by those design elements?
    • What are the core design principles that you have built into Prophecy to address these shortcomings?
      • How do those user experience changes improve the quality and speed of work for data engineers?
    • How has the Prophecy platform changed since we last spoke almost a year ago?
    • What are the tradeoffs of low code systems for productivity vs. flexibility and creativity?
    • What are the most interesting, innovative, or unexpected approaches to developer experience that you have seen for data tools?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on user experience optimization for data tooling at Prophecy?
    • When is it more important to optimize for computational efficiency over developer productivity?
    • What do you have planned for the future of Prophecy?

    Contact Info

    • LinkedIn
    • @_raj_bains on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Prophecy
      • Podcast Episode
    • CUDA
    • Clustrix
    • Hortonworks
    • Apache Hive
    • Compilerworks
      • Podcast Episode
    • Airflow
    • Databricks
    • Fivetran
      • Podcast Episode
    • Airbyte
      • Podcast Episode
    • Streamsets
    • Change Data Capture
    • Apache Pig
    • Spark
    • Scala
    • Ab Initio
    • Type 2 Slowly Changing Dimensions
    • AWS Deequ
    • Matillion
      • Podcast Episode
    • Prophecy SaaS

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Cloud Native Data Orchestration For Machine Learning And Data Engineering With Flyte May 23, 2022
    Show notes

    Summary

    Machine learning has become a meaningful target for data applications, bringing with it an increase in the complexity of orchestrating the entire data flow. Flyte is a project that was started at Lyft to address their internal needs for machine learning and integrated closely with Kubernetes as the execution manager. In this episode Ketan Umare and Haytham Abuelfutuh share the story of the Flyte project and how their work at Union is focused on supporting and scaling the code and community that has made Flyte successful.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Data lake architectures provide the best combination of massive scalability and cost reduction, but they aren’t always the most performant option. That’s why Kyligence has built on top of the leading open source OLAP engine for data lakes, Apache Kylin. With their AI augmented engine they detect patterns from your critical queries, automatically build data marts with optimized table structures, and provide a unified SQL interface across your lake, cubes, and indexes. Their cost-based query router will give you interactive speeds across petabyte scale data sets for BI dashboards and ad-hoc data exploration. Stop struggling to speed up your data lake. Get started with Kyligence today at dataengineeringpodcast.com/kyligence
    • Your host is Tobias Macey and today I’m interviewing Ketan Umare and Haytham Abuelfutuh about Flyte, the open source and kubernetes-native orchestration engine for your data systems

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Flyte is and the story behind it?
    • What was missing in the ecosystem of available tools that made it necessary/worthwhile to create Flyte?
    • Workflow orchestrators have been around for several years and have gone through a number of generational shifts. How would you characterize Flyte’s position in the ecosystem?
      • What do you see as the closest alternatives?
      • What are the core differentiators that might lead someone to choose Flyte over e.g. Airflow/Prefect/Dagster?
    • What are the core primitives that Flyte exposes for building up complex workflows?
      • Machine learning use cases have been a core focus since the project’s inception. What are some of the ways that that manifests in the design and feature set?
    • Can you describe the architecture of Flyte?
      • How have the design and goals of the platform changed/evolved since you first started working on it?
    • What are the changes in the data ecosystem that have had the most substantial impact on the Flyte project? (e.g. roadmap, integrations, pushing people toward adoption, etc.)
    • What is the process for setting up a Flyte deployment?
    • What are the user personas that you prioritize in the design and feature development for Flyte?
    • What is the workflow for someone building a new pipeline in Flyte?
      • What are the patterns that you and the community have established to encourage discovery and reuse of granular task definitions?
      • Beyond code reuse, how can teams scale usage of Flyte at the company/organization level?
    • What are the affordances that you have created to facilitate local development and testing of workflows while ensuring a smooth transition to production?
      • What are the patterns that are available for CI/CD of workflows using Flyte?
    • How have you approached the design of data contracts/type definitions to provide a consistent/portable API for defining inter-task dependencies across languages?
    • What are the available interfaces for extending Flyte and building integrations with other components across the data ecosystem?
    • Data orchestration engines are a natural point for generating and taking advantage of rich metadata. How do you manage creation and propagation of metadata within and across the framework boundaries?
    • Last year you founded Union to offer a managed version of Flyte. What are the features that you are offering beyond what is available in the open source?
      • What are the opportunities that you see for the Flyte ecosystem with a corporate entity to invest in expanding adoption?
    • What are the most interesting, innovative, or unexpected ways that you have seen Flyte used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Flyte?
    • When is Flyte the wrong choice?
    • What do you have planned for the future of Flyte?

    Contact Info

    • Ketan Umare
    • Haytham Abuelfutuh

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Flyte
      • Slack Channel
    • Union.ai
    • Kubeflow
    • Airflow
    • AWS Step Functions
    • Protocol Buffers
    • XGBoost
    • MLFlow
    • Dagster
      • Podcast Episode
    • Prefect
      • Podcast Episode
    • Arrow
    • Parquet
    • Metaflow
    • Pytorch
      • Podcast.__init__ Episode
    • dbt
    • FastAPI
      • Podcast.__init__ Interview
    • Python Type Annotations
    • Modin
      • Podcast.__init__ Interview
    • Monad
    • Datahub
      • Podcast Episode
    • OpenMetadata
      • Podcast Episode
    • Hudi
      • Podcast Episode
    • Iceberg
      • Podcast Episode
    • Great Expectations
      • Podcast Episode
    • Pandera
    • Union ML
    • Weights and Biases
    • Whylogs
      • Podcast Episode

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Sponsored By:

    Designing And Deploying IoT Analytics For Industrial Applications At Vopak May 16, 2022
    Show notes

    Summary

    Industrial applications are one of the primary adopters of Internet of Things (IoT) technologies, with business critical operations being informed by data collected across a fleet of sensors. Vopak is a business that manages storage and distribution of a variety of liquids that are critical to the modern world, and they have recently launched a new platform to gain more utility from their industrial sensors. In this episode Mário Pereira shares the system design that he and his team have developed for collecting and managing the collection and analysis of sensor data, and how they have split the data processing and business logic responsibilities between physical terminals and edge locations, and centralized storage and compute.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
    • So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
    • Your host is Tobias Macey and today I’m interviewing Mário Pereira about building a data management system for globally distributed IoT sensors at Vopak

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Vopak is and what kinds of information you rely on to power the business?
    • What kinds of sensors and edge devices are you using?
      • What kinds of consistency or variance do you have between sensors across your locations?
    • How much computing power and storage space do you place at the edge?
      • What level of pre-processing/filtering is being done at the edge and how do you decide what information needs to be centralized?
      • What are some examples of decision-making that happens at the edge?
    • Can you describe the platform architecture that you have built for collecting and processing sensor data?
      • What was your process for selecting and evaluating the various components?
    • How much tolerance do you have for missed messages/dropped data?
    • How long are your data retention periods and what are the factors that influence that policy?
    • What are some statistics related to the volume, variety, and velocity of your data?
      • What are the end-to-end latency requirements for different segments of your data?
    • What kinds of analysis are you performing on the collected data?
    • What are some of the potential ramifications of failures in your system? (e.g. spills, explosions, spoilage, contamination, revenue loss, etc.)
    • What are some of the scaling issues that you have experienced as you brought your system online?
    • How have you been managing the decision making prior to implementing these technology solutions?
    • What are the new capabilities and business processes that are enabled by this new platform?
    • What are the most interesting, innovative, or unexpected ways that you have seen your data capabilities applied?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on building an IoT collection and aggregation platform at global scale?
    • What do you have planned for the future of your IoT system?

    Contact Info

    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Vopak
    • Swinging Door Compression Algorithm
    • IoT Greengrass
    • OPCUA IoT protocol
    • MongoDB
    • AWS Kinesis
    • AWS Batch
    • AWS IoT Sitewise Edge
    • Boston Dynamics

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Insights And Advice On Building A Data Lake Platform From Someone Who Learned The Hard Way May 16, 2022
    Show notes

    Summary

    Designing a data platform is a complex and iterative undertaking which requires accounting for many conflicting needs. Designing a platform that relies on a data lake as its central architectural tenet adds additional layers of difficulty. Srivatsan Sridharan has had the opportunity to design, build, and run data lake platforms for both Yelp and Robinhood, with many valuable lessons learned from each experience. In this episode he shares his insights and advice on how to approach such an undertaking in your own organization.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, dbt models, Airflow jobs, and business intelligence tools, reducing time to detection and resolution from weeks to just minutes. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more.
    • Your host is Tobias Macey and today I’m interviewing Srivatsan Sridharan about the technological, staffing, and design considerations for building a data platform

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what your experience has been with designing and implementing data platforms?
    • What are the elements that you have found to be common requirements across organizations and data characteristics?
    • What are the architectural elements that require the most detailed consideration based on organizational needs and data requirements?
    • How has the ecosystem for building maintainable and usable data lakes matured over the past few years?
      • What are the elements that are still cumbersome or intractable?
    • The streaming ecosystem has also gone through substantial changes over the past few years. What is your synopsis of the meaningful differences between todays options and where we were ~6 years ago?
    • How did your experiences at Yelp inform your current architectural approach at Robinhood?
    • Can you describe your current platform architecture?
      • What are the primary capabilities that you are optimizing for?
    • What is your evaluation process for determining what components to use in your platform?
      • How do you approach the build vs. buy problem and quantify the tradeoffs?
    • What are the most interesting, innovative, or unexpected ways that you have seen your data systems used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on designing and implementing data platforms across your career?
    • When is a data lake architecture the wrong choice?
    • What do you have planned for the future of the data platform at Robinhood?

    Contact Info

    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • Robinhood
    • Yelp
    • Kafka
    • Spark
    • Flink
      • Podcast Episode
    • Pulsar
      • Podcast Episode
    • Parquet
    • Change Data Capture
    • Delta Lake
      • Podcast Episode
    • Hudi
      • Podcast Episode
    • Redshift
    • BigQuery
    • Informatica
    • Data Mesh
      • Podcast Episode
    • PrestoDB
    • Trino
    • Airbyte
      • Podcast Episode
    • Meltano
      • Podcast Episode
    • Fivetran
      • Podcast Episode
    • Stitch
    • Pinot
      • Podcast Episode
    • Clickhouse
      • Podcast Episode
    • Druid
    • Iceberg
      • Podcast Episode
    • Looker
      • Podcast Episode

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Scaling Analysis of Connected Data And Modeling Complex Relationships With The TigerGraph Graph Database May 09, 2022
    Show notes

    Summary

    Many of the events, ideas, and objects that we try to represent through data have a high degree of connectivity in the real world. These connections are best represented and analyzed as graphs to provide efficient and accurate analysis of their relationships. TigerGraph is a leading database that offers a highly scalable and performant native graph engine for powering graph analytics and machine learning. In this episode Jon Herke shares how TigerGraph customers are taking advantage of those capabilities to achieve meaningful discoveries in their fields, the utilities that it provides for modeling and managing your connected data, and some of his own experiences working with the platform before joining the company.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
    • This episode is brought to you by Acryl Data, the company behind DataHub, the leading developer-friendly data catalog for the modern data stack. Open Source DataHub is running in production at several companies like Peloton, Optum, Udemy, Zynga and others. Acryl Data provides DataHub as an easy to consume SaaS product which has been adopted by several companies. Signup for the SaaS product at dataengineeringpodcast.com/acryl
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the leading end-to-end Data Observability Platform! Trusted by the data teams at Fox, JetBlue, and PagerDuty, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, dbt models, Airflow jobs, and business intelligence tools, reducing time to detection and resolution from weeks to just minutes. Monte Carlo also gives you a holistic picture of data health with automatic, end-to-end lineage from ingestion to the BI layer directly out of the box. Start trusting your data with Monte Carlo today! Visit http://www.dataengineeringpodcast.com/montecarlo?utm_source=rss&utm_medium=rss to learn more.
    • Your host is Tobias Macey and today I’m interviewing Jon Herke about TigerGraph, a distributed native graph database

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what TigerGraph is and the story behind it?
    • What are some of the core use cases that you are focused on supporting?
    • How has TigerGraph changed over the past 4 years since I spoke with Todd Blaschka at the Open Data Science Conference?
    • How has the ecosystem of graph databases changed in usage and design in recent years?
    • What are some of the persistent areas of confusion or misinformation that you encounter when explaining graph databases and TigerGraph to potential users?
    • The tagline on your website says that TigerGraph is "The Only Scalable Graph Database for the Enterprise". Can you unpack that claim and explain what is necessary for a graph database to be suitable for enterprise use?
    • What are some of the typical application and system architectures that you typically see for end-users of TigerGraph? (e.g. polyglot persistence, etc.)
    • What are the cases where TigerGraph should be the system of record as opposed to an optimization option for addressing highly connected data?
    • What are the data modeling considerations that end-users should be thinking of when planning their storage structures in TigerGraph?
    • What are the most interesting, innovative, or unexpected ways that you have seen TigerGraph used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on TigerGraph?
    • When is TigerGraph the wrong choice?
    • What do you have planned for the future of TigerGraph?

    Contact Info

    • LinkedIn
    • @jonherke on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Links

    • TigerGraph
    • GraphQL
    • Kafka
    • GQL (Graph Query Language)
    • LDBC (Linked Data Benchmark Council)

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Previous 1 21 22 23 24 25 52 Next

    Related Podcasts

    Know How… (Video)

    1

    Know How… (Video) Education
    JavaScript Jabber

    2

    JavaScript Jabber Education
    Programming Throwdown

    3

    Programming Throwdown Software How-To
    Software Engineering Radio – the podcast for professional software developers

    4

    Software Engineering Radio – the podcast for professional software developers Software How-To
    The Changelog: Software Development, Open Source

    5

    The Changelog: Software Development, Open Source How To
    Coding Blocks

    6

    Coding Blocks Software How-To
    footer-logo

    Contact Us

    Toll Free: 844-670-7747

    Links

    • Home
    • Top Charts
    • Networks
    • Apps
    • Independents Podcasts
    • Podcast Advertising
    • Podcast News
    • Contact Us
    • About Us
    • Analytics & Insights

    Stay Connected

      Privacy, Terms of Use & Our Code of Ethics Protecting Content Creators Copyrights