TopPodcast.com
Menu
  • Home
  • Top Charts
  • Top Networks
  • Top Apps
  • Top Independents
  • Top Podfluencers
  • Top Picks
    • Top Business Podcasts
    • Top True Crime Podcasts
    • Top Finance Podcasts
    • Top Comedy Podcasts
    • Top Music Podcasts
    • Top Womens Podcasts
    • Top Kids Podcasts
    • Top Sports Podcasts
    • Top News Podcasts
    • Top Tech Podcasts
    • Top Crypto Podcasts
    • Top Entrepreneurial Podcasts
    • Top Fantasy Sports Podcasts
    • Top Political Podcasts
    • Top Science Podcasts
    • Top Self Help Podcasts
    • Top Sports Betting Podcasts
    • Top Stocks Podcasts
  • Podcast News
  • About Us
  • Podcast Advertising
  • Contact
Not in our directory?
Add Show Here
Podcast Equipment
Center

toppodcastlogoOur TOPPODCAST Picks

  • Comedy
  • Crypto
  • Sports
  • News
  • Politics
  • True Crime
  • Business
  • Finance

Follow Us

toppodcastlogoStay Connected

    View Top 200 Chart
    Back to Rankings Page
    Software How-To

    Data Engineering Podcast

    This show goes behind the scenes for the tools, techniques, and difficulties associated with the discipline of data engineering. Databases, workflows, automation, and data manipulation are just some of the topics that you will find here.

    Advertise

    Copyright: © 2024 Boundless Notions, LLC.

    • Apple Podcasts
    • Google Play
    • Spotify
    Latest Episodes:
    Understanding The Role Of The Chief Data Officer Aug 22, 2022
    Show notes

    Summary

    The position of Chief Data Officer (CDO) is relatively new in the business world and has not been universally adopted. As a result, not everyone understands what the responsibilities of the role are, when you need one, and how to hire for it. In this episode Tracy Daniels, CDO of Truist, shares her journey into the position, her responsibilities, and her relationship to the data professionals in her organization.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
    • Your host is Tobias Macey and today I’m interviewing Tracy Daniels about the role and responsibilities of the Chief Data Officer and how it is evolving along with the ecosystem

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what your path to CDO of Truist has been?
      • As a CDO, what are your responsibilities and scope of influence?
    • Not every organization has an explicit position for the CDO. What are the factors that determine when that should be a distinct role?
      • What is the relationship and potential overlap with a CTO?
    • As the CDO of Truist, what are some of the projects/activities that are vying for your time and attention?
    • Can you share the composition of your teams and how you think about organizational structure and integration for data professionals in your company?
    • What are the industry and business trends that are having the greatest impact on your work as a CDO?
      • How has your role evolved over the past few years?
    • What are some of the organizational politics/pressures that you have had to navigate to achieve your objectives?
      • What are some of the ways that priorities at the C-level can be at cross purposes to that of the CDO?
    • What are some of the skills and experiences that you have found most useful in your work as CDO?
    • What are the most interesting, innovative, or unexpected ways that you have seen the CDO position/responsibilities addressed in other organizations?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working as a CDO?
    • When is a distinct CDO position the wrong choice for an organization?
    • What advice do you have for anyone who is interested in charting a career path to the CDO seat?

    Contact Info

    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Links

    • Truist
    • Chief Data Officer
    • Chief Analytics Officer

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Bringing Automation To Data Labeling For Machine Learning With Watchful Aug 14, 2022
    Show notes

    Summary

    Data engineers have typically left the process of data labeling to data scientists or other roles because of its nature as a manual and process heavy undertaking, focusing instead on building automation and repeatable systems. Watchful is a platform to make labeling a repeatable and scalable process that relies on codifying domain expertise. In this episode founder Shayan Mohanty explains how he and his team are bringing software best practices and automation to the world of machine learning data preparation and how it allows data engineers to be involved in the process.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Sifflet solves this problem by acting as an overseeing layer to the data stack – observing data and ensuring it’s reliable from ingestion all the way to consumption. Whether the data is in transit or at rest, Sifflet can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams’ on their preferred channels. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses. Listeners of the podcast will get $2000 to use as platform credits when signing up to use Sifflet. Sifflet also offers a 2-week free trial. Find out more at dataengineeringpodcast.com/sifflet today!
    • The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest. Go to dataengineeringpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Shayan Mohanty about Watchful, a data-centric platform for labeling your machine learning inputs

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Watchful is and the story behind it?
    • What are your core goals at Watchful?
      • What problem are you solving and who are the people most impacted by that problem?
    • What is the role of the data engineer in the process of getting data labeled for machine learning projects?
    • Data labeling is a large and competitive market. How do you characterize the different approaches offered by the various platforms and services?
    • What are the main points of friction involved in getting data labeled?
      • How do the types of data and its applications factor into how those challenges manifest?
      • What does Watchful provide that allows it to address those obstacles?
    • Can you describe how Watchful is implemented?
      • What are some of the initial ideas/assumptions that you have had to re-evaluate?
      • What are some of the ways that you have had to adjust the design of your user experience flows since you first started?
    • What is the workflow for teams who are adopting Watchful?
      • What are the types of collaboration that need to happen in the data labeling process?
      • What are some of the elements of shared vocabulary that different stakeholders in the process need to establish to be successful?
    • What are the most interesting, innovative, or unexpected ways that you have seen Watchful used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Watchful?
    • When is Watchful the wrong choice?
    • What do you have planned for the future of Watchful?

    Contact Info

    • LinkedIn
    • @shayanjm on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • Watchful
    • Entity Resolution
    • Supervised Machine Learning
    • BERT
    • CLIP
    • LabelBox
    • Label Studio
    • Snorkel AI
      • Machine Learning Podcast Episode
    • RegEx == Regular Expression
    • REPL == Read Evaluate Print Loop
    • IDE == Integrated Development Environment
    • Turing Completeness
    • Clojure
    • Rust
    • Named Entity Recognition
    • The Halting Problem
    • NP Hard
    • Lidar
    • Shayan: Arguments Against Hand Labeling

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Collecting And Retaining Contextual Metadata For Powerful And Effective Data Discovery Aug 14, 2022
    Show notes

    Summary

    Data is useless if it isn’t being used, and you can’t use it if you don’t know where it is. Data catalogs were the first solution to this problem, but they are only helpful if you know what you are looking for. In this episode Shinji Kim discusses the challenges of data discovery and how to collect and preserve additional context about each piece of information so that you can find what you need when you don’t even know what you’re looking for yet.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Sifflet solves this problem by acting as an overseeing layer to the data stack – observing data and ensuring it’s reliable from ingestion all the way to consumption. Whether the data is in transit or at rest, Sifflet can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams’ on their preferred channels. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses. Listeners of the podcast will get $2000 to use as platform credits when signing up to use Sifflet. Sifflet also offers a 2-week free trial. Find out more at dataengineeringpodcast.com/sifflet today!
    • The biggest challenge with modern data systems is understanding what data you have, where it is located, and who is using it. Select Star’s data discovery platform solves that out of the box, with an automated catalog that includes lineage from where the data originated, all the way to which dashboards rely on it and who is viewing them every day. Just connect it to your database/data warehouse/data lakehouse/whatever you’re using and let them do the rest. Go to dataengineeringpodcast.com/selectstar today to double the length of your free trial and get a swag package when you convert to a paid plan.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Shinji Kim about data discovery and what is required to build and maintain useful context for your information assets

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you share your definition of "data discovery" and the technical/social/process components that are required to make it viable?
      • What are the differences between "data discovery" and the capabilities of a "data catalog" and how do they overlap?
    • discovery of assets outside the bounds of the warehouse
    • capturing and codifying tribal knowledge
    • creating a useful structure/framework for capturing data context and operationalizing it
    • What are the most interesting, innovative, or unexpected ways that you have seen data discovery implemented?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on data discovery at SelectStar?
    • When might a data discovery effort be more work than is required?
    • What do you have planned for the future of SelectStar?

    Contact Info

    • LinkedIn
    • @shinjikim on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • Select Star
      • Podcast Episode
    • Fivetran
      • Podcast Episode
    • Airbyte
      • Podcast Episode
    • Tableau
    • PowerBI
      • Podcast Episode
    • Looker
      • Podcast Episode

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Useful Lessons And Repeatable Patterns Learned From Data Mesh Implementations At AgileLab Aug 06, 2022
    Show notes

    Summary

    Data mesh is a frequent topic of conversation in the data community, with many debates about how and when to employ this architectural pattern. The team at AgileLab have first-hand experience helping large enterprise organizations evaluate and implement their own data mesh strategies. In this episode Paolo Platter shares the lessons they have learned in that process, the Data Mesh Boost platform that they have built to reduce some of the boilerplate required to make it successful, and some of the considerations to make when deciding if a data mesh is the right choice for you.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
    • Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows. Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month. For more information on Prefect, visit dataengineeringpodcast.com/prefect.
    • The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
    • Your host is Tobias Macey and today I’m interviewing Paolo Platter about Agile Lab’s lessons learned through helping large enterprises establish their own data mesh

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you share your experiences working with data mesh implementations?
    • What were the stated goals of project engagements that led to data mesh implementations?
    • What are some examples of projects where you explored data mesh as an option and decided that it was a poor fit?
    • What are some of the technical and process investments that are necessary to support a mesh strategy?
    • When implementing a data mesh what are some of the common concerns/requirements for building and supporting data products?
      • What are the general shape that a product will take in a mesh environment?
      • What are the features that are necessary for a product to be an effective component in the mesh?
    • What are some of the aspects of a data product that are unique to a given implementation?
    • You built a platform for implementing data meshes. Can you describe the technical elements of that system?
      • What were the primary goals that you were addressing when you decided to invest in building Data Mesh Boost?
    • How does Data Mesh Boost help in the implementation of a data mesh?
    • Code review is a common practice in construction and maintenance of software systems. How does that activity map to data systems/products?
    • What are some of the challenges that you have encountered around CI/CD for data products?
      • What are the persistent pain points involved in supporting pre-production validation of changes to data products?
    • Beyond the initial work of building and deploying a data product there is the ongoing lifecycle management. How do you approach refactoring old data products to match updated practices/templates?
    • What are some of the indicators that tell you when an organization is at a level of sophistication that can support a data mesh approach?
    • What are the most interesting, innovative, or unexpected ways that you have seen Data Mesh Boost used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Data Mesh Boost?
    • When is Data Mesh (Boost) the wrong choice?
    • What do you have planned for the future of Data Mesh Boost?

    Contact Info

    • LinkedIn
    • @axlpado on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • AgileLab
    • Spark
    • Cloudera
    • Zhamak Dehghani
    • Data Mesh
    • Data Fabric
    • Data Virtualization
    • q-lang
    • Data Mesh Boost
    • Data Mesh Marketplace
    • SourceGraph
    • OpenMetadata
    • Egeria

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Optimize Your Machine Learning Development And Serving With The Open Source Vector Database Milvus Aug 06, 2022
    Show notes

    Summary

    The optimal format for storage and retrieval of data is dependent on how it is going to be used. For analytical systems there are decades of investment in data warehouses and various modeling techniques. For machine learning applications relational models require additional processing to be directly useful, which is why there has been a growth in the use of vector databases. These platforms store direct representations of the vector embeddings that machine learning models rely on for computing relevant predictions so that there is no additional processing required to go from input data to inference output. In this episode Frank Liu explains how the open source Milvus vector database is implemented to speed up machine learning development cycles, how to think about proper storage and scaling of these vectors, and how data engineering and machine learning teams can collaborate on the creation and maintenance of these data sets.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Sifflet solves this problem by acting as an overseeing layer to the data stack – observing data and ensuring it’s reliable from ingestion all the way to consumption. Whether the data is in transit or at rest, Sifflet can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams’ on their preferred channels. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses. Listeners of the podcast will get $2000 to use as platform credits when signing up to use Sifflet. Sifflet also offers a 2-week free trial. Find out more at dataengineeringpodcast.com/sifflet today!
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Frank Liu about the open source vector database Milvus and how it simplifies the work of supporting ML teams

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Milvus is and the story behind it?
    • What are the goals of the project?
      • Who is the target audience for this database?
    • What are the use cases for a vector database and similarity search of vector embeddings?
      • What are some of the unique capabilities that this category of database engine introduces?
    • Can you describe how Milvus is architected?
      • What are the primary system requirements that have influenced the design choices?
      • How have the goals and implementation evolved since you started working on it?
    • What are some of the interesting details that you have had to address in the storage layer to allow for fast and efficient retrieval of vector embeddings?
    • What are the limitations that you have had to impose on size or dimensionality of vectors to allow for a consistent user experience in a running system?
      • The reference material states that similarity between two vectors implies similarity in the source data. What are some of the characteristics of vector embeddings that might make them immune or susceptible to confusion of similarity across different source data types that share some implicit relationship due to specifics of their vectorized representation? (e.g. an image vs. an audio file, etc.)
    • What are the available deployment models/targets and how does that influence potential use cases?
    • What is the workflow for someone who is building an application on top of Milvus?
    • What are some of the data management considerations that are introduced by vector databases? (e.g. manage versions of vectors, metadata management, etc.)
    • What are the most interesting, innovative, or unexpected ways that you have seen Milvus used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Milvus?
    • When is Milvus the wrong choice?
    • What do you have planned for the future of Milvus?

    Contact Info

    • LinkedIn
    • fzliu on GitHub
    • Website

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • Milvus
    • Zilliz
    • Linux Foundation/AI & Data
    • MySQL
    • PostgreSQL
    • CockroachDB
    • Pilosa
      • Podcast Episode
    • Pinecone Vector DB
      • Podcast Episode
    • Vector Embedding
    • Reverse Image Search
    • Vector Arithmetic
    • Vector Distance
    • SIGMOD
    • Tensor
    • Rotation Matrix
    • L2 Distance
    • Cosine Distance
    • OpenAI CLIP
    • Knowhere
    • Kafka
    • Pulsar
      • Podcast Episode
    • CAP Theorem
    • Milvus Helm Chart
    • Zilliz Cloud
    • MinIO
    • Towhee
    • Attu
    • Feder
    • FPGA == Field Programmable Gate Array
    • TPU == Tensor Processing Unit

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda Jul 31, 2022
    Show notes

    Summary

    Exploratory data analysis works best when the feedback loop is fast and iterative. This is easy to achieve when you are working on small datasets, but as they scale up beyond what can fit on a single machine those short iterations quickly become long and tedious. The Arkouda project is a Python interface built on top of the Chapel compiler to bring back those interactive speeds for exploratory analysis on horizontally scalable compute that parallelizes operations on large volumes of data. In this episode David Bader explains how the framework operates, the algorithms that are built into it to support complex analyses, and how you can start using it today.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Data stacks are becoming more and more complex. This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating the quality of the data and causing teams to lose trust. Sifflet solves this problem by acting as an overseeing layer to the data stack – observing data and ensuring it’s reliable from ingestion all the way to consumption. Whether the data is in transit or at rest, Sifflet can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams’ on their preferred channels. All thanks to 50+ quality checks, extensive column-level lineage, and 20+ connectors across the Data Stack. In addition, data discovery is made easy through Sifflet’s information-rich data catalog with a powerful search engine and real-time health statuses. Listeners of the podcast will get $2000 to use as platform credits when signing up to use Sifflet. Sifflet also offers a 2-week free trial. Find out more at dataengineeringpodcast.com/sifflet today!
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing David Bader about Arkouda, a horizontally scalable parallel compute library for exploratory data analysis in Python

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Arkouda is and the story behind it?
    • What are the main goals of the project?
      • How does it address those goals?
      • Who is the primary audience for Arkouda?
    • What are some of the main points of friction that engineers and scientists encounter while conducting exploratory data analysis (EDA)?
      • What kinds of behaviors are they engaging in during these exploration cycles?
    • When data scientists run up against the limitations of their tools and environments how does that impact the work of data engineers/data platform owners?
    • There have been a number of libraries/frameworks/utilities/etc. built to improve the experience and outcomes for EDA. What was missing that made Arkouda necessary/useful?
    • Can you describe how Arkouda is implemented?
      • What are some of the novel algorithms that you have had to design to support Arkouda’s objectives?
      • How have the design/goals/scope of the project changed since you started working on it?
    • How has the evolution of hardware capabilities impacted the set of processing algorithms that are viable for addressing considerations of scale?
      • What are the relative factors of scale along space/time axes that you are optimizing for?
      • What are some opportunities that are still unrealized for algorithmic optimizations to expand horizons for large-scale data manipulation?
    • For teams/individuals who are working with Arkouda can you describe the implementation process and what the end-user workflow looks like?
    • What are the most interesting, innovative, or unexpected ways that you have seen Arkouda used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Arkouda?
    • When is Arkouda the wrong choice?
    • What do you have planned for the future of Arkouda?

    Contact Info

    • Website
    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Links

    • Arkouda
    • NJIT == New Jersey Institute of Technology
    • NumPy
    • Pandas
      • Podcast.__init__ Episode
    • NetworkX
    • Chapel
    • Massive Graph Analytics Book
    • Ray
      • Podcast.__init__ Episode
    • Dask
      • Podcast Episode
    • Bodo
      • Podcast Episode
    • Stinger Graph Analytics
    • Bears-R-Us
    • 0MQ
    • Triangle Centrality
    • Degree Centrality

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    What "Data Lineage Done Right" Looks Like And How They're Doing It At Manta Jul 31, 2022
    Show notes

    Summary

    Data lineage is the roadmap for your data platform, providing visibility into all of the dependencies for any report, machine learning model, or data warehouse table that you are working with. Because of its centrality to your data systems it is valuable for debugging, governance, understanding context, and myriad other purposes. This means that it is important to have an accurate and complete lineage graph so that you don’t have to perform your own detective work when time is in short supply. In this episode Ernie Ostic shares the approach that he and his team at Manta are taking to build a complete view of data lineage across the various data systems in your organization and the useful applications of that information in the work of every data stakeholder.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
    • The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
    • Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows. Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month. For more information on Prefect, visit dataengineeringpodcast.com/prefect.
    • Your host is Tobias Macey and today I’m interviewing Ernie Ostic about Manta, an automated data lineage service for managing visibility and quality of your data workflows

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you describe what Manta is and the story behind it?
    • What are the core problems that Manta aims to solve?
    • Data lineage and metadata systems are a hot topic right now. What is your summary of the state of the market?
      • What are the capabilities that would lead a team or organization to choose Manta in place of the other options?
    • What are some examples of "data lineage done wrong"? (what does that look like?)
      • What are the risks associated with investing in an incomplete solution for data lineage?
      • What are the core attributes that need to be tracked consistently to enable a comprehensive view of lineage?
    • How do the practices for collecting lineage and metadata differ between structured, semi-structured, and unstructured data assets and their movement?
    • Can you describe how Manta is implemented?
      • How have the design and goals of the product changed or evolved?
    • What is involved in integrating Manta with an organization’s data systems?
      • What are the biggest sources of friction/errors in collecting and cleaning lineage information?
    • One of the interesting capabilities that you advertise is versioning and time travel for lineage information. Why is that a necessary and useful feature?
    • Once an organization’s lineage information is available in Manta, how does it factor into the daily workflow of different roles/stakeholders?
    • There are a variety of use cases for metadata in a data platform beyond lineage. What are the benefits that you see from focusing on that as a core competency?
    • Beyond validating quality, identifying errors, etc. it seems that automated discovery of lineage could produce insights into when the presence of data assets that shouldn’t exist. What are some examples of similar discoveries that you are aware of?
    • What are the most interesting, innovative, or unexpected ways that you have seen Manta used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Manta?
    • When is Manta the wrong choice?
    • What do you have planned for the future of Manta?

    Contact Info

    • LinkedIn
    • @dsrealtime01 on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • Manta
    • Egeria
    • OpenLineage
      • Podcast Episode
    • Apache Atlas
    • Neo4J
    • Easytrieve

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Re-Bundling The Data Stack With Data Orchestration And Software Defined Assets Using Dagster Jul 24, 2022
    Show notes

    Summary

    The current stage of evolution in the data management ecosystem has resulted in domain and use case specific orchestration capabilities being incorporated into various tools. This complicates the work involved in making end-to-end workflows visible and integrated. Dagster has invested in bringing insights about external tools’ dependency graphs into one place through its "software defined assets" functionality. In this episode Nick Schrock discusses the importance of orchestration and a central location for managing data systems, the road to Dagster’s 1.0 release, and the new features coming with Dagster Cloud’s general availability.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today I’m interviewing Nick Schrock about software defined assets and improving the developer experience for data orchestration with Dagster

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • What are the notable updates in Dagster since the last time we spoke? (November, 2021)
    • One of the core concepts that you introduced and then stabilized in recent releases is the "software defined asset" (SDA). How have your users reacted to this capability?
      • What are the notable outcomes in development and product practices that you have seen as a result?
    • What are the changes to the interfaces and internals of Dagster that were necessary to support SDA?
    • How did the API design shift from the initial implementation once the community started providing feedback?
    • You’re releasing the stable 1.0 version of Dagster as part of something called "Dagster Day" on August 9th. What do you have planned for that event and what does the release mean for users who have been refraining from using the framework until now?
    • Along with your 1.0 commitment to a stable interface in the framework you are also opening your cloud platform for general availability. What are the major lessons that you and your team learned in the beta period?
      • What new capabilities are coming with the GA release?
    • A core thesis in your work on Dagster is that developer tooling for data professionals has been lacking. What are your thoughts on the overall progress that has been made as an industry?
      • What are the sharp edges that still need to be addressed?
    • A core facet of product-focused software development over the past decade+ is CI/CD and the use of pre-production environments for testing changes, which is still a challenging aspect of data-focused engineering. How are you thinking about those capabilities for orchestration workflows in the Dagster context?
      • What are the missing pieces in the broader ecosystem that make this a challenge even with support from tools and frameworks?
      • How has the situation improved in the recent past and looking toward the near future?
      • What role does the SDA approach have in pushing on these capabilities?
    • What are the most interesting, innovative, or unexpected ways that you have seen Dagster used?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on bringing Dagster to 1.0 and cloud to GA?
    • When is Dagster/Dagster Cloud the wrong choice?
    • What do you have planned for the future of Dagster and Elementl?

    Contact Info

    • @schrockn on Twitter
    • schrockn on GitHub
    • LinkedIn

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
    • To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers

    Links

    • Dagster Day
    • Dagster
      • 1st Podcast Episode
      • 2nd Podcast Episode
    • Elementl
    • GraphQL
    • Unbundling Airflow
    • Feast
    • Spark SQL
    • Dagster Cloud Branch Deployments
    • Dagster custom I/O manager
    • LakeFS
    • Iceberg
    • Project Nessie
    • Prefect
      • Prefect Orion
    • Astronomer
    • Temporal

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Writing The Book That Offers A Single Reference For The Fundamentals Of Data Engineering Jul 24, 2022
    Show notes

    Summary

    Data engineering is a difficult job, requiring a large number of skills that often don’t overlap. Any effort to understand how to start a career in the role has required stitching together information from a multitude of resources that might not all agree with each other. In order to provide a single reference for anyone tasked with data engineering responsibilities Joe Reis and Matt Housley took it upon themselves to write the book "Fundamentals of Data Engineering". In this episode they share their experiences researching and distilling the lessons that will be useful to data engineers now and into the future, without being tied to any specific technologies that may fade from fashion.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
    • Prefect is the modern Dataflow Automation platform for the modern data stack, empowering data practitioners to build, run and monitor robust pipelines at scale. Guided by the principle that the orchestrator shouldn’t get in your way, Prefect is the only tool of its kind to offer the flexibility to write code as workflows. Prefect specializes in glueing together the disparate pieces of a pipeline, and integrating with modern distributed compute libraries to bring power where you need it, when you need it. Trusted by thousands of organizations and supported by over 20,000 community members, Prefect powers over 100MM business critical tasks a month. For more information on Prefect, visit dataengineeringpodcast.com/prefect today.
    • Your host is Tobias Macey and today I’m interviewing Joe Reis and Matt Housley about their new book on the Fundamentals of Data Engineering

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Can you explain what possessed you to write such an ambitious book?
    • What are your goals with this book?
    • What was your process for determining what subject areas to include in the book?
      • How did you determine what level of granularity/detail to use for each subject area?
    • Closely linked to what subjects are necessary to be effective as a data engineer is the concept of what that title encompasses. How have the definitions shifted over the past few decades?
      • In your experiences working in industry and researching for the book, what is the prevailing view on what data engineers do?
      • In the book you focus on what you term the "data lifecycle engineer". What are the skills and background that are needed to be successful in that role?
    • Any discussion of technological concepts and how to build systems tends to drift toward specific tools. How did you balance the need to be agnostic to specific technologies while providing relevant and relatable examples?
    • What are the aspects of the book that you anticipate needing to revisit over the next 2 – 5 years?
      • Which elements do you think will remain evergreen?
    • What are the most interesting, unexpected, or challenging lessons that you have learned while working on writing "Fundamentals of Data Engineering"?
    • What are your predictions for the future of data engineering?

    Contact Info

    • Joe
      • LinkedIn
      • Website
    • Matt
      • LinkedIn
      • @doctorhousley on Twitter

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Links

    • Fundamentals of Data Engineering (affiliate link)
    • Ternary Data
    • Designing Data Intensive Applications
    • James Webb Space Telescope
    • Google Colossus Storage System
    • DMBoK == Data Management Body of Knowledge
    • DAMA
    • Bill Inmon
    • Apache Druid
    • RTFM == Read The Fine Manual
    • DuckDB
      • Podcast Episode
    • VisiCalc
    • Ternary Data Newsletter
    • Meroxa
      • Podcast Episode
    • Ruby on Rails
    • Lambda Architecture

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Joe Reis Flips The Script And Interviews Tobias Macey About The Data Engineering Podcast Jul 17, 2022
    Show notes

    Summary

    Data engineering is a large and growing subject, with new technologies, specializations, and "best practices" emerging at an accelerating pace. This podcast does its best to explore this fractal ecosystem, and has been at it for the past 5+ years. In this episode Joe Reis, founder of Ternary Data and co-author of "Fundamentals of Data Engineering", turns the tables and interviews the host, Tobias Macey, about his journey into podcasting, how he runs the show behind the scenes, and the other things that occupy his time.

    Announcements

    • Hello and welcome to the Data Engineering Podcast, the show about modern data management
    • When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
    • RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
    • Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
    • Your host is Tobias Macey and today we’re flipping the script. Joe Reis of Ternary Data will be interviewing me about my time as the host of this show and my perspectives on the data ecosystem

    Interview

    • Introduction
    • How did you get involved in the area of data management?
    • Now I’ll hand it off to Joe…

    Joe’s Notes

    • You do a lot of podcasts. Why? Podcast.init started in 2015, and your first episode of Data Engineering was published January 14, 2017. Walk us through the start of these podcasts.
    • why not a data science podcast? why DE?
    • You’ve published 306 of shows of the Data Engineering Podcast, plus 370 for the init podcast, then you’ve got a new ML podcast. How have you kept the motivation over the years?
    • What’s the process for the show (finding guests, topics, etc….recording, publishing)? It’s a lot of work. Walk us through this process.
    • You’ve done a ton of shows and have a lot of context with what’s going on in the field of both data engineering and Python. What have been some of the major evolutions of topics you’ve covered?
    • What’s been the most counterintuitive show or interesting thing you’ve learned while producing the show?
    • How do you keep current with the data engineering landscape?
    • You’ve got a very unique perspective of data engineering, having interviewed countless top people in the field. What are the the big trends you see in data engineering over the next 3 years?
    • What do you do besides podcasting? Is this your only gig, or do you do other work?
    • whats next?

    Contact Info

    • LinkedIn
    • Website

    Closing Announcements

    • Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
    • Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
    • If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@themachinelearningpodcast.com) with your story.
    • To help other people find the show please leave a review on iTunes and tell your friends and co-workers

    Parting Question

    • From your perspective, what is the biggest gap in the tooling or technology for data management today?

    Links

    • Podcast.__init__
    • The Machine Learning Podcast
    • Ternary Data
    • Fundamentals of Data Engineering book (affiliate link)

    The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA

    Support Data Engineering Podcast

    Previous 1 19 20 21 22 23 52 Next

    Related Podcasts

    Know How… (Video)

    1

    Know How… (Video) Education
    JavaScript Jabber

    2

    JavaScript Jabber Education
    Programming Throwdown

    3

    Programming Throwdown Software How-To
    Software Engineering Radio – the podcast for professional software developers

    4

    Software Engineering Radio – the podcast for professional software developers Software How-To
    The Changelog: Software Development, Open Source

    5

    The Changelog: Software Development, Open Source How To
    Coding Blocks

    6

    Coding Blocks Software How-To
    footer-logo

    Contact Us

    Toll Free: 844-670-7747

    Links

    • Home
    • Top Charts
    • Networks
    • Apps
    • Independents Podcasts
    • Podcast Advertising
    • Podcast News
    • Contact Us
    • About Us
    • Analytics & Insights

    Stay Connected

      Privacy, Terms of Use & Our Code of Ethics Protecting Content Creators Copyrights