Data Mesh Radio

Latest Episodes:

Summer Hiatus Announcement - Back in August Jun 02, 2024

Taking a needed break to focus on getting healthy. Be back in August!

#306 Building with People for People - Swisscom's Data Mesh Approach and Learnings - Interview w/ Mirela Navodaru May 26, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Mirela's LinkedIn: https://www.linkedin.com/in/mirelanavodaru/

In this episode, Scott interviewed Mirela Navodaru, Enterprise and Solution Architect for Data, Analytics, and AI at Swisscom.

Some key takeaways/thoughts from Mirela's point of view:

Specifically at Swisscom, it's not about doing data mesh. They want to make data a key part of all their major decisions - operational and strategic - and data mesh means they can put the data production and consumption in far more people's hands. Data mesh is a way to achieve their data goals, not the goal.
When you are trying to get people bought in to something like data mesh, you always have to consider what is in it for them. Yes, the overall organization benefiting is great but it’s not the best selling point 😅 try to develop your approach to truly benefit everyone.
Data literacy is crucial to getting the most value from data mesh. Data mesh is not about throwing away the important knowledge your data people have but it's about unlocking the value of the knowledge your business people have to be shared with the rest of the organization effectively, reliably, and scalably.
?Controversial? You really have to talk to a lot of people early in your data mesh journey to discover the broader benefits to the organization. That way you can talk to people's specific challenges to get them bought in. When designing your journey, it is important to get input from a large number of people.
When talking data as a product versus data products, the first is the core concept and the second is the deliverables. Scott note: this is a really simple but powerful delineation
"No value, no party." If there isn't a value proposition, there shouldn't be any action. You need to stay focused on value because there are so many potential places to focus in a data mesh implementation.
You have to balance value at the use case level to the domain versus more global value to the organization. At the end of the day, everything you do should add value to the organization but sometimes use cases are much more focused at the domain and that's perfectly expected and acceptable.
Data mesh, to really change the organization in the right way, needs top level buy-in. You can't only be the data team trying to head down the data mesh path.
Everything in data mesh is about iterating to better. You need the space and room to learn as you go along. You can - and must - deliver value before you've got everything figured out perfectly.
Relatedly, you will learn how to better iterate towards value throughout your journey. It will be tough at the start as with any learning journey.
Obviously, data mesh is a large cultural change. You need to have empathy and give people the chance to grow instead of trying to move too fast. Upskilling, especially around data literacy, is crucial.
There are two very valuable aspects of data mesh: the value you deliver via use cases along the way and the value you get from learning to do data better across your organization. The first is from integrating data into far more of your decisions and the second means you can react more quickly to new opportunities and build scalable and reliable approaches to data management.
Something like data mesh is a big change. But it shouldn't be a shock to people. You can do it gradually and incrementally while you deliver value. One of the best ways to lose people is to thrust disruptive change on them instead of working with them through the change to prevent large-scale negative disruptions.
There are so many areas where data mesh helps organizations, whether it is getting away from silos, reducing redundancy, improving quality and reliability, etc. It's not just about doing data management itself better, which has been the focus of most data approaches historically.
Again, data work is not the point. The point is to make your colleagues better at their job through being more informed. That comes down to the data but it's never the actual point, it's the vehicle to delivering value.
Transparency and managing expectations - and communication in general - are crucial to doing data mesh well. You need to have that space to learn and iterate. Let people know what you are doing and especially why you are doing it.
Data modeling in data mesh is of course a challenge. But it's important to have some level of common language between the domains or you will have data silos. It's a balance but it's crucial to give domains flexibility but also create easy paths for people to combine data across domains.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#305 Combining the Technical and Business Perspectives for Data Mesh - Interview w/ Alyona Galyeva May 19, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Alyona's LinkedIn: https://www.linkedin.com/in/alyonagalyeva/

In this episode, Scott interviewed Alyona Galyeva, Principal Data Engineer at Thoughtworks. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Alyona's point of view:

?Controversial? People keep coming up with simple phrasing and a few sentences about where to focus in data mesh. But if you're headed in the right direction, data mesh will be hard, it's a big change. You might want things to be simple but simplistic answers aren't really going to lead to lasting, high-value change to the way your org does data. Be prepared to put in the effort to make mesh a success at your organization, not a few magic answers.
!Controversial! Stop focusing so much on the data work as the point. It's a way to derive and deliver value but the data work isn't the value itself.
Relatedly, ask what are the key decisions people need to make and what is currently preventing them from making those decisions. Those are likely to be your best use cases.
When it comes to Zhamak's data mesh book, it needs to be used as a source of inspiration instead of trying to use it as a manual. Large concepts like data mesh cannot be copy/paste, they must be adapted to your organization.
It's really important to understand your internal data flows. Many people inside organizations - especially the data people - think they know the way data flows across the organization, especially for key use cases. But when you dig in, they don't. Those are some key places to deeply investigate first to add value.
On centralization versus decentralization, it's better to think of each decision as a slider rather than one or the other. You need to find your balances and also it's okay to take your time as you shift more towards decentralization for many aspects. Change management is best done incrementally.
?Controversial? A major misunderstanding of data mesh that some long-time data people have is that it is just sticking a better self-serve consumption layer on top and we can continue to do monolithic data work under the hood. Be prepared for lots of friction in convincing some data architects that this isn't just a reskin or another layer on top of the enterprise warehouse or data lake.
For data mesh, it's crucial to understand necessary changes at the technical and the business level. You can't only work on one but you also don't have to 'solve' them at the start, make progress. It's like with the four principles, you need to thin slice change across the technical and business aspects rather than only focusing on one.
You can sell data engineers on data mesh by making their work more meaningful and impactful. Instead of mostly firefighting - which is the case in many organizations - they can focus on shipping new features and adding incremental value.
With data mesh, you want people focusing on more than just making data valuable - what is valuable will change so how do you make your data products evolvable and maintainable?
You always want to be focused on addressing people's pain points in data mesh, driving towards value. That's how you can get data people bought in as well, not just business people.
!Controversial! Doing aggregated data products across domains is usually the data mesh inflection point - basically answering can data mesh work in your organization. If you can't get that cross domain collaboration going well, you should consider another model like hub and spoke.
Relatedly, aggregating data across multiple domains is where there is usually the most value for an organization. But it's very hard to find good champions there because you need more vision and more hard work to collaborate across domain boundaries. Identify the people with the vision early in your journey, even if it's often better to actually only start working with them once you have more momentum and data products.
Too often, there is a rush to build _something_ instead of the right thing. Don't get fooled by the idea that data work always creates value. Even if the client or business partner asks you to build something of value, always circle back to the use cases. As much as we'd like to build universal data products, they just don't exist.
Relatedly though, don’t get so focused only on trying to build the consumer-aligned data products for hyper-specific use cases that you miss the forest for the trees. Sometimes the use case is something like 'we need to understand what data we even have to be able to use it to address our current problems in XYZ business line.'
To kind of sum it up: stop focusing on what you can build first. Focus on what you should build and then look at the realities. What matters to the business and why? Then focus on what's possible and what will deliver sustainable and maintainable value through data work.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#304 Getting Your Data Mesh Journey Moving Forward - Interview w/ Chris Ford and Arne Lapõnin May 12, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Arne's LinkedIn: https://www.linkedin.com/in/arnelaponin/

Chris' LinkedIn: https://www.linkedin.com/in/ctford/

Foundations of Data Mesh O'Reilly Course: https://www.oreilly.com/videos/foundations-of-data/0636920971191/

Data Mesh Accelerate workshop article: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html

In this episode, Scott interviewed Arne Lapõnin, Data Engineer and Chris Ford, Technology Director, both at Thoughtworks.

From here forward in this write-up, I am combining Chris and Arne's points of view rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from Arne and Chris' point of view:

Before you start a data mesh journey, you need an idea of what you want to achieve, a bet you are making on what will drive value. It doesn't have to be all-encompassing but doing data mesh can't be the point, it's an approach for delivering on the point 😅
Relatedly, there should be a business aspiration for doing data mesh rather than simply a change to the way of doing data aspiration. What does doing data better mean for your organization? What does a "data mesh nirvana" look like for the organization? Work backwards from that to figure where to head with your journey.
A common early data mesh anti-pattern is trying to skip both ownership and data as a product. There are existing data assets that leverage spaghetti code and some just rename them to data products and pretend that's moved the needle.
"A data product is a data set + love." The real difference between a data product and a data set is that true ownership and care.
?Controversial?: Another common mesh anti-pattern is trying to get too specific with definitions or prescriptive advice. There isn't a copy/paste approach that will work and getting a specific definition of a data product doesn't really change much. Mindset is far more important than definitions.
It can be very helpful to have some simple checklists around your data products. While there is no prescriptive way to build, checklists remove a lot of the uncertainty for teams asking 'am I doing this right?' It gives some simple reassurances that you aren't missing out on key pieces of what they're building.
?Controversial?: Most organizations probably don't need to do a ton of pre-work before starting on a data mesh implementation. They need some achievable goals, a roadmap for how they plan to achieve those goals, and a lot of willpower to push things forward and keep going when the going gets tough. You also need an enticing vision for people to buy into.
THIN SLICE! Don't try to take everything on at once but also don't try to skip over any of the four pillars. There's a reason they haven't changed from Zhamak's initial blog post. Scott note: don't try to argue the governance pillar wasn't in the first blog post, it just wasn't called out separately…
Three key questions to answer if you are considering data mesh: A) Do you have sufficient scale? B) Do you have a strategy that depends on deriving value from data? C) Are you prepared to take advantage of the autonomy Data Mesh will afford to your product teams? If you don't have satisfactory answers to those three questions, data mesh is probably not right/overkill for you.
If people don't see the strong need to transform your business through data, it's likely to lead to troubles 6-9 months into your data mesh journey. If you aren't addressing key organizational pain points or delivering value, you will likely lose support for your data mesh implementation initiative. Doing data better has to be valued to get more budget to keep going.
Another anti-pattern is focusing too much on use cases at the expense of the platform and the journey. Data mesh is designed to work at scale and that only works by finding repeatable processes. You can't treat each data product like a one-off.
In order to get buy-in from the data engineers - or whoever are your data product developers - you need to invest in changing hearts and minds through the platform. If creating and managing data products is significantly harder than the old way of dealing with data, you will lose people quickly.
Read about the data mesh accelerate workshop 😅
When you think about first steps with data mesh, A) build buy-in at the strategic level that you want to actually start leveraging your data for high-value purposes; B) find use cases to support those strategic initiatives; and C) make sure you are ready to actually thin slice and not try to only tackle on pillar - you have to be ready to take on a LOT of challenging work.
!Controversial!: None of the four data mesh principles are all that useful on their own. Scott note: there's a figure Zhamak has that explains why all four are necessary in conjunction that's very helpful here.
It's easy to want to skip bringing all your key stakeholders into alignment early in your data mesh journey. But you need matching expectations and shared understandings of what you are trying to accomplish and why. Scott note: this doesn't mean everyone has to be bought in that they are first, there's a balance to be found here.
You need room to make mistakes and adjust your data mesh implementation because you will not get it all right at the start. Data mesh is as much about learning how to do data well as doing data well.
It's crucial to not just ask if you are succeeding with your data mesh implementation but measure that. It can be hard to measure but consider what matters to your implementation's success and find things to measure if you're succeeding in each of those areas. Otherwise, how do you know where to focus and optimize?
Subsidiarity: "everything should be decided as locally as possibly but no more so." Basically, there are many decisions that should be made in the domains but there are some that need to be made centrally. The challenge is figuring out which decisions should be made where 😅
There will be capability challenges in every organization when doing data mesh. That will impact initial decisions around how much to centralize or decentralize but as you upskill the teams, you may want to decentralize more. Find your equilibriums but equilibriums change. It's all about trade-offs!
Many people are too focused on exactly if they are doing data mesh instead of are they delivering value in a scalable way through data. That happened in microservices too and it took them 10+ years to really get to best practices. Data mesh is only 5 years old and only ~3 with any number of organizations attempting it - focus on getting better instead of being worried if you're the perfect picture of data mesh yet.
When talking about your data mesh success internally, you need to talk about the value from use cases AND the value of improving your data capabilities in general. You prove out you are delivering specific value along the way but also that you are getting more and more capable at doing the data work to make the organization better. Both are of valuable and you should promote the value of both aspects: use case value and capability value.
When talking about data mesh, use the ADKAR method: create Awareness and Desire, give them the Knowledge about how you're doing it, upskill people so they have the Ability to do data mesh, and finally constantly Reinforce the value and that it's important. Without touting your mesh successes, you'll lose momentum.
When looking for your first data mesh use case, look for something that has a customer impact - what can you do for them that you couldn't before. Personalization is a good example. Legal is potentially another place re reducing risk.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#303 Delivering What Matters - Value - Through Strong Business Collaboration - Interview w/ Saba Ishaq May 05, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Saba's LinkedIn: https://www.linkedin.com/in/sabaishaq/

Decide Data website: ttps://www.decidedata.com/

In this episode, Scott interviewed Saba Ishaq, CEO and Founder of her own data as a service consultancy, Decide Data, which also provides 3rd party DAaaS (Data Analytics as a Service) solutions.

Some key takeaways/thoughts from Saba's point of view:

"If you don't know what you want, you're going to end up with a lot of what you don't want." This is especially true in collaborating with business stakeholders when it comes to data 😅
Focus on delivering value through data instead of delivering data and assuming it has value. – “Not all data is created equal.”
As a data leader, it's your role to help people figure out what they actually want by asking great questions and being a strong partner when it comes to the data/data work. Don't only focus on the data work itself but it's very easy to do data work for the sake of it instead of something that is valuable.
To deliver data work that actually moves the needle, we need to start from what are the key business processes and then understand the pain points and opportunities. Then, good data work is about how do we support and improve those business processes.
Relatedly, that's also the best way to drive exec alignment - talking about their business processes and how they can be improved first, data work second. They will feel seen and heard and are far more likely to lean in. At the end of the day addressing business and operational challenges is what data and analytics is all about.
Deliver something valuable early in any data collaboration with a business stakeholder. You don't have to deliver an entire completed project but time to first insight is time to value and you build momentum and credibility with that stakeholder.
At the beginning of a project - and delivering a data product is itself a project - you should work with stakeholders to not just define target outcomes but also define how are you going to collaborate and communicate. You can't just get requirements, go away and build. Working with data should be iterative and should have an element of continuous improvement to evolve what you deliver as you build value.
Start any data work by asking someone about their business objectives, challenges, and target outcomes. You need your business stakeholders to have a clear vision of what they want to achieve, otherwise you are likely to be delivering only data work instead of business value that leverages your data work.
By doing deep discovery work, you can find where are the key lynchpins and value drivers in a use case. There are points of criticality that are easy to lose in a sea of potential requirements that are really requests. Find those crucial value leverage points!
Relatedly, you can use those value leverage points to keep your business execs engaged. They will - hopefully - see the importance and help you narrow in on what matters in their use case. Then it's no longer about the data work but the value to them.
?Controversial?: For data people, you have to balance career management and interesting project/technology work versus value delivery. That doesn't mean delivering value isn't interesting but it doesn't always mean getting to play with the latest and greatest. But if data people never get to have fun and play with cool tech, many will leave. It's a tough balance. Try to make the valuable work also interesting 😅.
Relatedly, try understanding the data team’s learning areas of interest and see how you can build seeds to foster their skill growth while making data work valuable. Sometimes it turns out to be a win-win situation.
Relatedly, be very transparent and communicate a lot to your data teams about what you are prioritizing and why. It's very easy to get lost in telling data people to do certain work rather than why they are doing that work. Keeping your data people in the loop of the why will keep them focused on what matters.
For many organizations, the rate of change of their technology - application and data technologies - is growing at faster than the rate of their people change management/transformation processes. You need parallel streams to modernize both or your people will fall further behind, leading to chaos.
?Controversial?: Relatedly, your overall org and/or digital transformation strategy should be tied to your data strategy. Otherwise, they will likely be heading in different directions, creating more challenges. Scott note: Benny Benford talked a lot about this in episode #244, going far together.
Data management is a very crucial element of digital transformation but it’s not the same thing as change management. The data team shouldn't be the ones leading the overall digital transformation of the organization. That's too much on a team that specializes in data rather than change management. If you are in that situation, it's a very tough spot to do well.
It's very important to focus on communication to stakeholders when you think about data governance and digital transformation. For many execs, these are foreign topics so you have to work hard to engage them and keep them leaning forward on the necessary work. Data governance is beneficial for everyone, so if explained and defined well people will engage willingly after knowing what’s in it for them.
As someone in the data team, you have to be well informed about digital transformation initiatives inside your organization. Otherwise, you will miss opportunities to align to those initiatives AND have all your data sources break when there is a migration you weren't told about 😅
It's easy to screw up the data steward/ownership conversation letting someone know they are responsible for the governance of their data. It's often a scary conversation for both parties. But it's necessary and you can show people why it makes sense and adds value to their work too!
Relatedly, link people's pain points to current weaknesses in the data governance. Show them they are causing issues for themselves and give them an easier path to fix it without having to learn everything about data work.
Data governance doesn't have to be some wholly - or holy 😅 - separate practice. It should just be part of normal work related to data. Make it less scary and more approachable for your business stakeholders. It's a team effort and it drives real, measurable benefits and value.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

No Episode This Week Apr 28, 2024

Craziness of the overseas move (including a faulty office chair... long story) are to blame. Back to the normally scheduled one episode a week next week!

Episode list and links to all available episode transcripts here.

#302 Finding and Delivering on a Good Initial Data Mesh Use Case - Interview w/ Basten Carmio Apr 21, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Basten's LinkedIn: https://www.linkedin.com/in/basten-carmio-2585576/

In this episode, Scott interviewed Basten Carmio, Customer Delivery Architect of Data and Analytics at AWS Professional Services. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Basten's point of view:

Your first use case - at the core - should A) deliver value in and of itself and B) improve your capabilities to deliver on incremental use cases. That's balancing value delivery, improving capabilities, and building momentum which are all key to a successful long-term mesh implementation.
When thinking about data mesh - or really any tech initiative - it's crucial to understand your starting state, not just your target end state. You need to adjust any approach to your realities and make incremental progress.
?Controversial?: Relatedly, it's very important to define what success looks like. Doing data mesh cannot be the goal. You need to consider your maturity levels and where you want to focus and what will deliver value for your organization. That is different for each organization. Scott note: this shouldn't be controversial but many companies are not defining their mesh value bet…
Even aligning everyone on your organization's definition of mesh success will probably be hard. But it's important to do.
For a data mesh readiness assessment, consider where you can deliver incremental value and align it to your general business strategy. If you aren't ready to build incrementally, you aren't going to do well with data mesh.
A common value theme for data mesh implementations is easier collaboration across the organization through data; that leads to faster reactions to changes and opportunities in your markets. Mesh done well means it's far faster and easier for lines of business to collaborate with each other - especially in a reliable and scalable way - and there are far better standard rules/policies/ways of working around that collaboration. But organizations have to see value in that or there may be mesh resistance.
As many have said, you must approach data mesh in a thin slice. Trying to focus too much on any pillar at the expense of the others leads to challenges. Scott note: Zhamak literally has a figure on this she shares often. It's easy to get unbalanced if you ignore a principle and fixing that takes more effort than thin slicing.
?Controversial?: As you build out your mesh capabilities, especially your platform, think about what you need to actually deliver on the use case(s) at hand and deliver only that. Don't get ahead of yourself. It's fun to build capabilities but it's easy to build a monstrosity of a platform instead of proper abstractions to make building and managing data products simple.
Doing data mesh well is all about managing trade-offs. There are trade-offs at the use case and implementation level. And it's okay to get those wrong, just look to fail fast rather than hold on to bad decisions. Don't be precious with your decisions and build in ways to make evolving and improving easier.
MVP can be minimum viable product or minimum valuable product. You need to define what value actually means for each use case. It's easy to point to pain and start solutioning but not end up addressing the pain or creating value. This will help you prioritize and deliver the most value compared to effort early.
Relatedly, having a clear perspective on value in your MVP means you are more easily able to change and adapt as you learn and deliver. If you thought value was going to come from X but early indications are Y is way more important or is much different than expectations, you can more easily pivot towards Y.
Data products don't magically create value. They should be bets on value delivery. So you need to have ways to gracefully retire data products when the cost exceeds the incremental value. Often times a bet was a good bet even when it didn't pay off or it is no longer paying off.
Continuous communication and driving buy-in, especially with C-level execs, is unfortunately crucial to your data mesh implementation success. If they don't value better data capabilities, few people will invest their time and effort to really reinvent the way your organization does data.
Relatedly, you need to tie your data mesh implementation to the overall business strategy. Execs need to know why you are doing something like data mesh and how it will deliver value. It's not an overnight success but it also can't be a 3 year project before value. Speaking to that gets more people to lean in so you can build momentum.
Finding and leveraging your data success champions is always hard but it's necessary to build that momentum. Think about building champions quite early.
When you're trying to get an exec to buy in to something like data mesh, always put it in terms of what's in it for them. Don't try to get them bought in to some grand vision first, why would they invest their valuable time and resources here? Find their pain points and what something like data mesh will do to address those specifically.
Relatedly, trying to get someone to just take on data ownership without the understanding of what the rest of the organization is going to do to make ownership much easier - mainly through the platform and federated governance - is probably going to be a … not fun conversation.
When thinking about data mesh, it should all come back to value. If you can't specifically point to value that will exceed your effort for doing something like data mesh, it's probably not worth it for your organization. What is valuable to your organization and how will data mesh help you capture that value?
Relatedly, as you are on your data mesh journey, you have to be honest in assessing if data mesh is really working for your organization. It's okay to stop if it's not.
The data mesh principles sound really great and helpful in the abstract. But they just might not be aligned to value with the way your company does business.
?Controversial?: It's perfectly acceptable to have hybrid approaches in a data mesh journey. There is a perception that everything should be decentralized and that just isn't always the best approach, don't get caught up in dogma. Scott note: the "decentralize everything" is a misunderstanding of Zhamak. Also, as you are on a journey, your current state won't match an ideal end state. Focus on evolving while delivering value, not trying to be perfect today instead of _getting_ to good.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#301 Learnings From 25+ Years in Data Quality - Interview w/ Olga Maydanchik Apr 14, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Olga's LinkedIn: https://www.linkedin.com/in/olga-maydanchik-23b3508/

Walter Shewhart - Father of Statistical Quality Control: https://en.wikipedia.org/wiki/Walter_A._Shewhart

William Edwards Deming - Father of Quality Improvement/Control: https://en.wikipedia.org/wiki/W._Edwards_Deming

Larry English - Information Quality Pioneer: https://www.cdomagazine.tech/opinion-analysis/article_da6de4b6-7127-11eb-970e-6bb1aee7a52f.html

Tom Redman - 'The Data Doc': https://www.linkedin.com/in/tomredman/

In this episode, Scott interviewed Olga Maydanchik, an Information Management Practitioner, Educator, and Evangelist.

Some key takeaways/thoughts from Olga's point of view:

Learn your data quality history. There are people who have been fighting this good fight for 25+ years. Even for over a century if you look at statistical quality control. Don't needlessly reinvent some of it :)
Data literacy is a very important aspect of data quality. If people don't understand the costs of bad quality, they are far less likely to care about quality.
Data quality can be a tricky topic - if you let consumers know that the data quality isn't perfect, they can lose trust. But A) in general, that conversation is getting better/easier to have and B) we _have_ to be able to identify quality as a problem in order to fix it.
Data quality is NOT a project - it's a continuous process.
Even now, people are finding it hard to use the well-established data quality dimensions. It's a framework for considering/measuring/understanding data quality so it’s not very helpful to data stewards / data engineers in creating data quality rules.
The majority of quality errors are not random, they come from faulty data mapping / bugs in pipelines. Having good quality rules will catch a large percentage of errors that can be fixed in bulk.
When thinking about getting started around data quality, it doesn't have to be complex and with lots of tools. It can be people looking at the data for potential issues and talking to producers. Then you can build a business case for fixing the data to get funding. You have to roll up your sleeves and talk to people but you can get forward momentum.
Data quality issues aren't inherently material to the business processes - they are only bad when they cause issues for the business. You have to find those actual business issues to get people to care and get funding for fixing it. Quality for the sake of quality is just extra cost. Do not create too many data quality rules that do not matter.
Relatedly, being able to show someone a relatively basic quality indicator early is far better than asking for a lot of budget to figure out the quality levels. You can do that with something as simple as random sampling 100-200 records and an hour of 1-2 people's time.
To understand which data quality challenges and use cases are the most important, data people simply have to learn more about the business. Good data quality is about fit for purpose and that means understanding the purposes :)
To find your initial good data quality use cases, look to mission criticality. What dashboards or reports are actually important to the company and why? Then work backwards to see if quality is an issue for those dashboards and reports. That's how you find your early buy-in to work on a quality initiative that can scale.
!Controversial!: Data contracts are not at all new, we just now have a good enough set of tools and technologies to be able to do them better at scale.
?Controversial?: Most are doing data contracts … not that well. For them, it's about the technology and not the process. There isn't a continuous approach. Scott note: Andrew Jones has said the same. It's about ensuring a process that results in quality data, not the tools.
For data contracts, there MUST be a feedback loop or we aren't actually delivering to needs, especially as needs evolve. Look to the widely used customer supply model for insights into what we need to achieve and how when it comes to data contracts.
Many companies are creating actual financial incentives tied to data quality in order to ensure people care about data quality. That's not right for every organization but it does send a clear message as to the importance of data quality.
You have to consider your data supply chain - if your interface for data input is bad, your data is very likely to be bad. People will simply enter garbage to move forward.
Doing data quality manually is not sustainable/scalable. But you don't need to start with expensive tools, you can get your arms around things initially pretty easily. It will help you identify your actual problems instead of spending time specifically on tools.
?Controversial?: Many vendors are selling their tools as the fix to data quality. But detecting data errors with the tools is only the start of the data quality improvements. Once errors are detected, root cause analysis for the errors needs to be performed and the processes / code need to be fixed. None of the data quality tools can do this. It is human’s job. Beware the snake oil.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#300 Panel: How to Treat Your Data Platform as a Product - Led by Michael Toland w/ Sadie Martin, Marta Diaz, and Sean Gustafson Apr 07, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Michael's LinkedIn: https://www.linkedin.com/in/mjtoland/

Marta's LinkedIn: https://www.linkedin.com/in/diazmarta/

Sadie's LinkedIn: https://www.linkedin.com/in/sadie-martin-06404125/

Sean's LinkedIn: https://www.linkedin.com/in/seangustafson/

The Magic of Platforms by Gregor Hohpe: https://platformengineering.org/talks-library/the-magic-of-platforms

Start with why -- how great leaders inspire action | Simon Sinek: https://www.youtube.com/watch?v=u4ZoJKF_VuA

In this episode, guest host Michael Toland Senior Product Manager at Pathfinder Product Labs/Testdouble and host of the upcoming Data Product Management in Action Podcast facilitated a discussion with Sadie Martin, Product Manager at Fivetran (guest of episode #64), Sean Gustafson, Director of Engineering - Data Platform at Delivery Hero (guest of episode #274), and Marta Diaz, Product Manager Data Platform at Adevinta Spain. As per usual, all guests were only reflecting their own views.

The topic for this panel was how to treat your data platform as a product. While many people in the data space are talking about data products, not nearly as many are treating the platform used for creating and managing those data products as a product itself. This is about moving beyond the IT services model for your data work. Platforms have life-cycles and need product management principles too! Also, in data mesh, it is crucial to understand that 'platform' can be plural, it doesn't have to be one monolithic platform, users don't care.

Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

You will hear "product mindset" a lot in this panel. It's important to embrace product management as a mindset and not an exact set of things to do and approaches to take. The whole point of a product mindset is to find what works and deliver reliably while focusing on value.
Be ruthless. Ruthlessly prioritize and ruthlessly focus on user centricity. In data, we have a tendency to fall in love with the tools instead of the jobs to be done. But a good platform is about making it easy to get high-value work done and that's rarely about exposing tooling to users.
Your platform isn't going to magically drive usage. There is change management required to get people to leverage your data platform more. Be prepared for that change management work and closely aligning with users, whether they are 'data people' or not.
Platforms and products in general are about scalability. Through a platform, instead of reacting to tickets, you build services and capabilities to address the problems that caused the tickets - far more scalable than reacting to the individual tickets, addressing the disease not the symptom. By building your platform as a product, you focus on what actual capabilities are within scope so you can continue to manage and expand your data platform.
As much as it would be amazing to build a data platform from scratch - think how amazing you could build it! - in many cases, you will have to build off of existing services and platforms. Don't be too dogmatic - what matters is continuing to get better not being perfect. Give yourself the space to improve the platform but products live in the real world and the real world of your organization has current/existing business needs and constraints.
Products - especially in software - are as much or more about evolution as they are about their form/function at initial launch. The same should happen with your platform. And the same should happen with your vision. Don't get locked in, don't get tunnel vision.
Generally speaking, most data platforms are still not serving the role a software platform - data or otherwise - should serve. They are still about the tech instead of the capabilities. You aren't alone if your platform doesn't meet the ideal vision. Your role is to make it better but it's still probably not going to be a sparkling beacon of perfection anytime soon 😅
Your data platform needs to be aligned to your company culture. So you have to meet people where they are and properly set down the easy path to where you want them to go. It's a long journey.

Other Important Takeaways (many touch on similar points from different aspects):

The product mindset is for the entire team, not just the platform leader and/or product manager. Your entire team will have to change their approach and mindset. Not overnight but it's hard to break old ticket-taker habits.
Make sure to put your work in the context of the business. It's easy to get bogged down in the platform engineering aspects or the data tools but your data platform is there to serve business purposes. Focus on what delivers value for the business.
Your platform doesn't have to address all your organization's data challenges at once, right at the start. Find where you are seeing the biggest challenges with delivering value - maybe listen back to episode #297 on the Data Value Chain - and look to focus there first. Build up to better.
"Build with [your users], not for them." Don't treat your platform as a project. Yes, that's the subject of the panel but it's very important to always keep close at heart.
User experience is such a crucial aspect of good software platforms. It's probably even more important when it comes to data platforms if you want to bring new users to producing and consuming data. But it's HARD and rarely discussed.
What is the goal of your platform? That might sound like an obvious question but it's not really when you go to answer it. Maybe it's "make it easier to work with data" but easier for who? Really map out good outcomes of a well-made platform.
Good products - at least good software products - aren't designed to be perfect at launch. Give yourself the space to not be perfect but set yourself up to understand where things can be improved and then iterate. Test, learn, iterate.
What are your data platform KPIs? What will measure if you are being successful? Really consider what bets you are making and why. Usage isn't always good but it's a good first indicator as you are getting moving.
To make your platform a product, you have to come back to the vision (or goal mentioned above). Otherwise, you have a collection of services. How is that vision tied to business value? If you need budget, do those holding the purse strings care about your tech decisions or business impact?
A good platform helps people get what they need done with the agency to do it. It limits - and where possible eliminates - bottlenecks.
A crucial aspect of product management is taking user needs and then translating that to build something to serve those needs. But when it comes to your data platform, the users and the builders (the data engineering team) usually don't speak the same language so you will probably have to spend more time translating between the users and the team building the platform than even in software. Be prepared for you data team to grumble about more meetings too 😅
Combine Simon Sinek's 'Start from the why' and user centricity. The why should be about what are you enabling your users to actually accomplish and what value that drives. Really focus on building something that drives value for the business instead of leveraging the coolest tech.
Relatedly, it's likely going to be very difficult to measure the return on investment on your data platform. Be prepared for those conversations but it can be pretty squishy.
At their heart, good products are about creating good user experiences and delivering/capturing value. Products need to deliver value to users and producers need to capture value as well. That is a complex topic when the value captured is internal, but it is still an appropriate mindset. If you aren't able to prove value, will you get further funding?
Consider your company needs relative to data maturity. An incredibly cutting edge platform for an organization that has low data maturity is not the right fit. Even if you have a cutting edge vision for their work, your 4 year old won't be able to out sculpt Michealangelo. Be realistic and use the platform to drive towards better data capabilities but you have to meet your organization at least close to where they are.
You might have difficulty getting people bought in on the vision of your data platform. If people view data as ticket-takers/a service instead of an integral part of the company's way of doing business, be prepared for the fun of lots of stakeholder alignment work.
Where possible, don't only look to sell people on your vision. Try to change - or possibly nudge at most - people's behavior through your platform. It's a subtle art and hard to pull off but it will mean people do what you and they both need but without having to build a million PowerPoint decks 😅
It's important to separate your concept of a data platform and the approach of treating it as a product. Otherwise, your platform can easily fall into the trap of designing to user wants instead of what is a platform's function - to make it easier to do work in a scalable, reliable, and repeatable fashion. You need to consider the purpose of the platform instead of throw product management at a collection of tools.
Leaders/execs want change. Leaders/exec do not want _to change_. Be prepared for that.
Users aren't necessarily ready for the data platform to be a product. They are used to the ticket-taker model. Be prepared to shift their mindset too, not just that of the (data) engineers building the platform. It won't be a simple switch over either - very deep topic…
Relatedly, you will probably need a longer run-way to transition your team to a product model for the platform. It is a mindset shift but it's also a big change in the ways of working. You will need to have some patience, you can't switch from a ticket model to only building your platform as a product overnight.
A key potential area of value for the platform is making it fast to prototype and get data in consumers' hands. It can reduce a ton of back and forth and also help you quickly discover if further work on a potential data product is useful or if it should be scrapped. Research spikes are very important in data and enabling them can be very valuable - if not always valued 😅
Make sure to think about the actual scope of your data platform. Products have scope, don't try to be all things to all people and don't bite off more than you can chew.
There's a reason SaaS offerings in the data space have post-sales and customer success engineers. Many of your users won't just simply be able to read the documentation and use your platform no matter how well you build and document it. Be prepared to ensure success through a bit of hand-holding. Ensuring usage that drives value is (usually) what makes software and data offerings successful.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#299 Empowering Development with Actionable Data - Interview w/ Carol Assis and Eduardo Santos Mar 31, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Carol's LinkedIn: https://www.linkedin.com/in/carol-assis/

Eduardo's LinkedIn: https://www.linkedin.com/in/eduardosan/

Continuous Integration book: https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380

Measure What Matters book: https://www.amazon.com/Measure-What-Matters-Google-Foundation/dp/0525536221

Inspired by Marty Cagan: https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507

Empowered by Marty Cagan: https://www.amazon.com/EMPOWERED-Ordinary-Extraordinary-Products-Silicon/dp/111969129X

In this episode, Scott interviewed Carol Assis, Data Analyst/Data Product Manager and Eduardo Santos, Professor and Consultant, both at Thoughtworks. To be clear, they were only representing their own views on the episode.

From here forward in this write-up, I will be generally combining both Carol and Eduardo's views into one rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from Eduardo and Carol's point of view:

At the end of the day, the team that produces the data will get the most use out of it 9/10 times. Getting teams used to developing with data in mind isn't just useful for the organization, it is for maximizing their own team's success.
Continuous integration is a crucial concept in general for learning how to automate and focus on delivering more, which leads to focusing on value. Read the book :)
?Controversial?: Data mesh is an extension of the continuous integration book/concept because of the focus on delivering value quickly and building to scale reliably.
There are many methodologies for understanding value delivery in software. We just have to adapt them better to data. Don't reinvent the wheel.
Far more organizations need to think about the goals of their products and then how to measure success against those goals _at product inception_. Design data into your products from the start.
Data people often make the data overly complicated for non-data people to grasp. What does the data tell us and what are some simple numbers? Then people can feel like they understand without going too deep into stochastic modeling or something.
Relatedly, engage data consumers' curiosity - including the producers that will consume their own data. Try to meet them where they are to get them to engage with data more. Lower the perceived bar to leveraging data.
Application development teams need convincing that working with data is 1) essential to understand their own success to further improve their products and 2) much easier than it has been historically. There is a LOT of scar tissue out there…
A potential good hook to get people to build their applications with data in mind is being able to show metrics and measure success from day one. The business side can get a better idea and are more likely to engage; it gives a communication bridge between developers and the business people.
Having data early in the application development cycle means you have more proof points for making your decisions - assuming you side with the data 😅 that makes it easier to justify decisions instead of people making guesses.
Measuring what matters is a crucial concept for the entire team to adopt. It will help people understand what data they need and why.
Pressing people on what to measure and then how to measure it crystalizes what bets they are making. How are they expecting users to interact with their applications?
For many business people, you may need someone playing the data translator role, translating data to business and business to data. Most organizations' data literacy is still quite low and again, you need to lower the bar to business people leveraging data.
?Controversial?: You don't need to start with data products. Yes, they are great, but teaching your teams that data matters is groundwork to head in the direction of something more scalable. A spreadsheet is a fine place to start. Focus on delivering insights that deliver value, then work towards the productized aspects :)
Ask people what action they will take once they have data. Get them in the mindset that data drives action and data that won’t drive action isn't where they should focus.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#298 Effective Partnering With Business Execs - Learnings from Another Data Mesh Journey - Interview w/ Jessika Milhomem Mar 24, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Jessika's LinkedIn: https://www.linkedin.com/in/jmilhomem/

In this episode, Scott interviewed Jessika Milhomem, Analytics Engineering Manager and Global Fraud Data Squad Leader at Nubank. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Jessika's point of view:

There are no silver bullets in data. Be prepared to make trade-offs. And make non data folks understand that too!
Far too often, people are looking only at a target end-result of leveraging data. Many execs aren't leaning in to how to actually work with the data, set themselves up to succeed through data. Data isn't a magic wand, it takes effort to drive results.
Relatedly, there is a disconnect between the impact of bad quality data and what business partners need to do to ensure data is high enough quality for them.
Poor data quality results in 4 potential issues that cost the company: regulatory violations/fines, higher operational costs, loss of revenue, and negative reputational impact.
There's a real lack of understanding by the business execs of how the data work ties directly into their strategy and day-to-day. It's not integrated. Good data work isn't simply an output, it needs to be integrated into your general business initiatives.
More business execs really need to embrace data as a product and data product thinking. Instead of a focus on only the short-term impact of data - typically answering a single question - how can we integrate data into our work to drive short, mid, and long-term value?
?Controversial?: In data mesh, within larger domains like Marketing or Credit Cards in a bank, it is absolutely okay to have a centralized data team rather than trying to have smaller data product teams in each subdomain. Scott note: this is actually a common pattern and seems to work well.
Relatedly, the pattern of centralized data teams in the domains leads to easier compliance with regulators because there is one team focused on reporting one view instead of trying to have multiple teams contribute to that view.
When you really start to federate data ownership, business execs can now partner far easier with other business execs in other domains leveraging data. Instead of having the central data team trying to translate, there is a focus on what needs to get done and the data work flows from that instead of the data work being the focus. It's the engine that powers their collaboration but it's no longer 'the point'.
Partnering with those who "are closer to the reality" of the business, it's easier and more likely to drive good outcomes. Meaning: not the senior execs. But the senior execs often have to be on board with the work and the target results. So work on communicating up but closely collaborating at lower levels.
Data for regulators often has a LOT of potential reuse for your own organization. Lean into finding those areas where you can do the data work once and get value twice :)
?Controversial?: Really consider role titles in data mesh. Data product owner might be too nebulous and quickly accumulate too many responsibilities. Data product manager is easier to understand the scope of responsibilities and the specific areas of focus. Scott note: this comes up A LOT and is generally starting with data product owner and moving to data product manager.
?Controversial?: Data leaders need to understand product management. To really scale data work, we have to start treating all aspects as a product practice. CTOs down to software engineers need to understand product management, it's time for the data org to as well.
Data leaders need to have significant communication skills while maintaining their understandings of data best practices. It's all a delicate balance but the data work doesn't speak for itself.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#297 Panel: Understanding and Leveraging the Data Value Chain - Led by Marisa Fish w/ Tina Albrecht, Karolina Stosio, and Kinda El Maarry, PhD Mar 17, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Marisa's LinkedIn: https://www.linkedin.com/in/marisafish/

Karolina's LinkedIn: https://www.linkedin.com/in/karolinastosio/

Tina's LinkedIn: https://www.linkedin.com/in/christina-albrecht-69a6833a/

Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/

In this episode, guest host Marisa Fish (guest of episode #115), Senior Technical Architect at Salesforce facilitated a discussion with Kinda El Maarry, PhD, Director of Data Governance and Business Intelligence at Prima (guest of episode #246), Tina Albrecht, Senior Director Transformation at Exxeta (guest of episode #228), and Karolina Stosio, Senior Project Manager of AI at Munich Re. As per usual, all guests were only reflecting their own views.

The topic for this panel was understanding and leveraging the data value chain. This is a complicated but crucial topic as so many companies struggle to understand the collection + storage, processing, and then specifically usage of data to drive value. There is way too much focus on the processing as if upstream of processing isn't a crucial aspect and as if value just happens by creating high-quality data.

A note from Marisa: Our panel is comprised of a group of data professionals who study business, architecture, artificial intelligence, and data because we want to know how (direct) data adds value to the development of goods and services within a business; and how (indirect) data enables that development. Most importantly, we want to help stakeholders better understand why data is critical to their organization's business administration strategy and is a keystone in their value chain.

Also, we lost Karolina for a bit there towards the end due to a spotty internet connection.

Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

If you want to dig deeper into the data value chain, consider looking into the value streams concept. What flows through your business in terms of process to generate value? Where are there points of value leakage? The same concepts are crucial in your value chain.
Organizations need to really educate their entire organization on the data value chain. Part of why there are so many issues in data from upstream changes by developers breaking downstream data is they simply don't know what parts of their data are used and why. Communication is a much bigger aspect of doing data than people think.
Even talking about the specific data value chain can cause people to focus too much on the data work instead of the business value delivered via data. The data value chain is crucial to understand but it's also crucial to understand data work doesn't inherently create value, it's about how it's used in the business. Dig into the value created and focus on working backwards from that to what data work needs to be done.
The data value chain is crucial for companies of all sizes across all industries. At its heart, the concept is about focusing on ensuring you aren't leaking value in your business value streams/pipelines. You need to focus on what drives value and how to improve the processes there.
Data value chains often cross line of business/domain boundaries. After all, a lot of the value of data is about combining information across those boundaries. That can mean cross-team handoffs, which make understanding and ensuring the success of those data value chains even harder. Who owns what isn't inherently understood/agreed to, you need to get specific.
It's important to not get overly focused on a single end-point of value when it comes to data work, especially when it comes to a data product. If we want re-use, we have to focus on the processes of creating reusable value. Maintaining that larger picture focus while still ensuring each data consumer can still get value from a data product is a very hard balance.
Focusing heavily on your data value chain is going to be hard. It means hard work and a lot of internal collaboration - and thus negotiation - across domain boundaries. You all have to be in it together to really get the best results - and some organizations aren't ready for that. But the hard work pays off because you are ensuring value actually gets created.
As with anything in data, you have to make bets. That doesn't mean every bit of data work will create significant value or even exceed the investment. But an approach like data value chain is crucial to understand 1) what bets are you making and why and 2) who owns what aspects of the data work. That can help you really focus on the what and why rather than focusing on outputs.

Other Important Takeaways (many touch on similar points from different aspects):

As with many things in data, ownership is crucial to understanding your value chain. The weakest points in a value chain are the handoffs between teams. Strong ownership, including of those handoffs, prevents value leakage (from the value streams concept).
To understand your data value chain, you will have to go deeper than many are willing to in the (dreaded?) operational plane. You have to understand what data you have, how it's collected, what data you can collect, etc. Some of it is working backward from what data you need/want but a lot of it is working from what data you have or can get.
Relatedly, the value you can create from data is heavily reliant on what matters to the business. To think about value, you have to understand your business processes and what generates actual value.
You really need to consider your approach to data collection and storage. How do you want to consider data that may have value but hasn't yet proven to have value? You don't want to have costs go out of control and most data is never back-cleaned/filled if it wasn't collected and stored for use. But you can't know all your data use cases at the launch of a new application or product. It's a balancing act.
There is a question of how mature do you need to be as an organization to actually really consider using data value chain as a framework instead of merely some principles to guide your work. It can be hard to get people to understand the value and what drives that value in data when they don't understand data work in general.
Relatedly, really digging into the data value chain can shine a light on underperforming activities inside and outside the data function. So you need to be prepared for some hard realizations and questions. Are you ready for transparency?
What aspect of data value chains fall on the business? It's a hard question. At the end of the day, data value chains are supportive of the business value chains/streams but it depends on who has ownership over data work: the lines of business or a centralized team. Your data value chains should have explicit ownership, at least of the different 'links' of the chain.
In data mesh especially but true in any data work, it's important to not see the data product as the end of the data value chain. The data product is there to make it easy for producers to reliably and scalably deliver value through data. But there is only value if that data is consumed, the value happens when someone takes action.
When launching new applications/products, you have to consider what data you might want to collect even if you don't need it right at the start. Especially if that is something like hardware where you can't augment many aspects of the devices once they've been deployed.
Focusing on data value chains is a mindset shift for most organizations, much like data as a product thinking. You need to get people to stop handwaving about aspects of data work and focus specifically on value and understanding that all parts of the data creation and transformation process are crucial to driving rich and sustainable value from data.
Even if you do a good job at understanding your data value chains, there will still need to be rework. But it can help you prioritize data rework - you aren't going to get your data preparation perfect, especially for multiple consumers, on the first try.
You have to be realistic about your data value. Your company probably won't value data and analytics that are internal facing as much as they do external-facing interactions until you prove out the value of treating those internal users with as much care. Part of that is getting specific about how much value you are generating and how :)
At some point in your value chain, you aren't dealing with raw data anymore. Think about who wants what and why. Most execs want aggregated information - again, that point of driving business value instead of data work. Make sure there is clear communication to drive outcomes instead of outputs.
A data value chain isn't about getting everything perfect upfront. Everything is about incremental delivery and getting better. What is the cost/benefit of that improvement? Get something out that works and is supportable/stable and then improve. Iteration is your friend.
When thinking about your data value chain, it's usually best to focus again on target business outcomes/objectives. After all, that is where the value is. You can get more business people interested in data work if you are constantly talking in their language about their key objectives.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#296 Patience in Product Thinking in Data - Building to Large-Scale Behavior Change - Interview w/ Darren Wood Mar 10, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Darren's LinkedIn: https://www.linkedin.com/in/darrenjwoodagileheadofproduct/

Darren's Big Data LDN Presentation: https://youtu.be/vUjoJrl_MEs?si=WzB0sBStVIAyqDJs

In this episode, Scott interviewed Darren Wood, Head of Data Product Strategy at UK media and broadcast company ITV. To be clear, he was only representing his own views on the episode.

Scott note: I use "coalition of the willing" to refer to those willing to participate early in your data mesh implementation. I wasn't aware of the historical context here, especially when it came to being used in war, e.g. the Iraq war of the early 2000s. I apologize for using a phrase like this.

Some key takeaways/thoughts from Darren's point of view:

Overall, when thinking about moving to product thinking in data, it's as much about behavior change as action. You have to understand how humans react to change and support that. You can't expect change to happen overnight - patience, persistence, and empathy are all crucial aspects. Transformation takes time and teamwork.
?Controversial?: In data mesh, it's crucial to think about flexibility and adaptability of your approach. Things will change, your understanding of how you deliver value will change. Your key targets will change. Be prepared or you will miss the main point of product thinking in data.
When choosing your initial domains and use cases in data mesh, think about big picture benefits. You aren't looking for exact value measurements for return on investment but you also want to target a tangible impact, e.g. if we do X, we think we can increase Y part of the business revenue Z%.
Zhamak defines a data product quite well in her book on data mesh. But data as a product is a much broader definition of bringing product management best practices to data. That's harder to define but quite important to get right.
When thinking about product discovery - what do data consumers actually need producers to create - there is often a big difference between consumers' initial suggested requirements and what they actually want. It's the role of data product management to bridge that gap and deliver what they need instead of what they requested.
?Controversial?: There is a big difference between data product management and regular product management: in regular product management, the ability to take requirements and go away and come back with something months later works. But data is about what's happening with the business now and needs to evolve as the understanding of requirements evolves.
Relatedly, data products need to be even less rigid than regular products because they are a reaction to the real world as it changes; you must focus on building something that flexible. Otherwise, you aren't going to be reflecting what matters.
When building data products, get to an MVP fast. Get something in people's hands and evolve it to what they need/want. Don't try to get it perfect. When it comes to doing data products well, there is far more collaboration than most people are used to around data work.
When considering what to build, data producers need to ask consumers what question they are actually trying to answer rather than what dashboard do they want. Outcomes over outputs. It's about what you are trying to do. Scott note: And then come to an agreement on the specifics - are you delivering data, the insights, or the 'so what'?
?Controversial?: Data products can - and probably should? - absolutely look to address multiple business questions. But when you are thinking about your MVP, focus on what is the most critical question and the minimum requirements to address that question. You can improve the product but getting to first valuable insights is a great initial milestone to build off.
?Controversial?: Similarly, in regular product management, you can go away for six months and come back with something new and shiny but in data, it might take that long just to change two metrics on a dashboard because it took that long to clean up the data. And at the end no one cares, no one understands what the value was, and worst still people are annoyed because they don't trust the new metrics.
Relatedly, maybe consider MLP instead of MVP. No, not 'My Little Pony' but Minimum Lovable Product. What is the minimum you can deliver that someone will love - what are the need to haves instead of the nice to haves? What is at least one feature that users will love and can you deliver that early to maintain engagement as you deliver more aspects of value?
Actually sit with users and see how they leverage your products. That's a crucial aspect of product management, it shouldn't be any different in data. It can be a bit harder sometimes to get to specifics but that doesn't let you off the hook. It's necessary work to do.
?Controversial?: Bring as many of the folks on the data product producer team as you can into the discussions with the initial data product consumer. That will give a more complete picture to the team. Don't treat the rest of the team as ticket takers internally. And you can probably find and address challenges earlier - e.g. flagging a low quality data source - if everyone is more informed.
As with anything in product management, prioritization is crucial. Focus on delivering value, not simply data products. Again, outcomes over outputs.
Initial delivery of the data product to a consumer isn't a mark of 'done', it's a great place to focus on how the consumer actually uses and interacts with the data product. Get a sense of the friction points to find places to further improve it. No more throwing things - data projects - over the wall and treating them as done.
Teams that understand product thinking in general are easier to teach product thinking around data. But for those teams that don't understand product thinking at all yet, you will need to spend more time with them. It can seem obvious but each team has different educational needs to bring them to product thinking for data.
You can't try to win over everyone to something like treating data as a product yourself. You need to find your champions and advocates and then provide platforms internally for them to spread the messages of value and success. It's not only about delivering value but showing that off a bit too to get others excited. Use momentum levers.
When choosing your first domains to work with and finding your first data products for data mesh, look for people that are enthusiasts. You want partners early on, not someone you have to constantly convince. Also look for data products that will be reusable to find additional users to provide additional value. It's as much about building momentum as it is about delivering value early.
?Controversial?: Behavior change is a crucial aspect of implementing data mesh. You have to understand behavior change takes time. And you have to know where you will be rigid and where you will be more flexible. If you say my way or the highway, most people are just going to head for the highway. You have to work out what is non-negotiable early.
Relatedly, behavior change happens at different paces for different people. That will happen inside your organization too. Look to build up the critical mass, that momentum, over time so people feel like they are joining a successful movement. But you need to do your internal PR to make sure they know about it - data success isn't self-evident.
During your transition to data mesh - and is it ever really done? -, you will have data sources that aren't part of the mesh. It will potentially be hard to integrate those with data from the mesh because your data in your data mesh has legal and governance at the core. Sources outside the mesh tend to have one-off policy approaches applied. Be prepared for some tension and consternation.
In most cases, people will inherently trust existing data sources over new data sources. That will be a challenge to your data mesh adoption. Even if they objectively know the quality is better with the new source, it's human nature to trust what you already know more all else being equal. Involve consumers heavily in the conversations about what is changing and why. Pushing change on people is likely to cause pushback. No one wants change to happen to them instead of with them.
When doing Domain Driven Design (DDD) in data mesh, do NOT try to start with all your domains at once. Look to find ones that are capable enough and that you can learn from to make it easier to partner with other domains in the future.
Relatedly, your domain map will change. That's a part of DDD. Don't try to hold onto things rigidly.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#295 Data Shouldn't be a Four-Letter Word - Making Data a Forethought - Interview w/ Wendy Turner-Williams Mar 03, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Wendy's LinkedIn: https://www.linkedin.com/in/wendy-turner-williams-8b66039/

Culstrata website: https://www.culstrata-ai.com/

TheAssociation.AI website: https://www.theassociation.ai/

In this episode, Scott interviewed Wendy Turner-Williams, Managing Partner at both TheAssociation.AI and Culstrata and the former CDO of Tableau.

TheAssociation.AI is "a global nonprofit business organization …focused on bridging the disciplines of AI, data, ethics, privacy, robotics, and security." It is focusing on things like networking and knowledge sharing to drive towards better outcomes including ethical AI.

Some key takeaways/thoughts from Wendy's point of view:

Right now, we try to break up the aspects of data into discrete disciplines - and then work on each completely separately - far too much. Privacy, security, compliance, performance, etc. Instead, we need to focus on the holistic picture of what we're trying to do and why.
Communication is key to effective data work and driving value from data. Hire product managers and focus on the why. Break through the historical perceptions of data as a service organization. Drive to what matters - outcomes over outputs - and focus on delivering value.
"What's the point of being focused on the data if you don't understand the business that the data is supposed to be used for?"
?Controversial?: "There is no transformation without automation." If you want data to play a part in transforming the business, you need to focus on automation. Data related work can't be toil work or most won't even do it.
"You will never be as successful as you can be as a data organization if you're not able to influence your IT partners, your product teams, your business teams."
For far too many companies, data is just an afterthought. It's not the core around how they build out initiatives. When you bolt-on the data to any aspect of the business instead of integrate it from the start and build with data in mind, it's far less impactful. You're always playing catch-up. Make data a forethought.
In many respects, data has become a 'four letter word' to lots of people - meaning it has a bad connotation.
There are a lot of internal politics around data. Data can mean power and it can also give people perspective on your team's performance. Try to work towards removing the politics if possible but also good luck… 😅
There is so much data in many large organizations that execs can't make sense of it. They often don't understand what data they will need to support their decisions or how to get in place the data they do know they need.
There's also often a disconnect between strategy and targets/feasibility when it comes to data. There may be a strategy of grow X product with a target of 'grow X product 15%' but there isn't a good reason why 15% is the target. It becomes a dartboard instead of data feeding into creating the goals.
Execution and tactical decisions are powered by data far less often than they should be. There is far too little thought or process around strategy and tactics enabled by and about creating data.
Many line of business or domain leaders are simply not great at data. They may be able to leverage insights but they don't get the information cycle, especially sourcing necessary data. Data teams need to partner with them effectively - that is definitely a two-way street.
?Controversial?: Relatedly, too many data people are focused on the data work itself instead of the impact of the work. There needs to be a better understanding of what teams are trying to accomplish with the data work. It's not about the pipeline, it's about the goal of the work and the impact.
You need internal processes and clear delineation of ownership or you will have multiple teams measuring the same things and getting different answers.
Far too often, people are myopic in focusing on their own job instead of how they fit into the bigger picture of the organization and delivering value to customers. That leads to teams not considering how they exchange information internally, only focusing on their own usage.
?Controversial?: Data teams need to spend more time creating their own data around the impact of data work and impact of issues like data downtime. Move past the service-only/cost-center perspective.
Lack of data fluency, especially among execs, causes so many issues. If people don't understand data, they don't understand how much they can trust it and thus won't rely on it.
Relatedly, there is a significant lack of understanding of upstream and downstream data and business processes and needs that could be fixed by better communication. What do you need from your upstream and how are your downstream users leveraging your data? Communicate!
Automating data work enables business partners to identify "business choke-points" and address them.
?Controversial?: You can't have AI without really understanding your business processes and how data supports those, how people combine data and their degree of trust and understanding of the data.

Wendy started out with her perspective that in some respects, data has become "a four letter word". There's so much data and everyone is trying to use it but everyone also feels inundated. Instead of being data-driven, we are data-flooded or data-dragged. And there is a major lack of tying the data work to the actual strategy and execution. Where do we need data to support our decisions? We need a strategy to get that data in place.

Relatedly, Wendy sees the major breakdown between strategy and goals when it comes to data. There may be a strategy to grow a product but how much growth is feasible, a good target? Why is that growth feasible? What does the data say about growing that product, e.g. the market dynamics and your positioning in the market? So when goals are set, it is a 'finger in the air' type guess as to how much it could grow or worse, simply how much leaders want a product to grow. And then what data do we need in place to enable the team managing that product to actually be able to grow it that much? How do we enable them to make smart tactical decisions?

Basically, it's a lot of things looping back on each other. We need data to set good strategic decisions. But we need a strategy to set up our ability to capture and analyze that data. We need data to make better tactical decisions. But most companies lack the ability to make good tactical decisions to get the necessary data in place. It's top-down driven but far too often the ones who understand what needs to be done for and with data are too far down in the organization and there's a communication gap. Thus, there is a significant lack of being data-driven. We have to admit the problem first. How to fix all of that is another fun process 😅

For Wendy, far too many organizations have data as an afterthought. And that leads to subpar understanding of what's actually happening with the business and lacking the information to fine-tune their strategic decisions. There isn't a strong strategic connection flowing from the business strategy to the data work. Execs aren't spending the time to really follow-through on exactly what the tactics should be to get the right data in place. Scott note: we literally have a panel on doing that, tying the data work to the business strategy and vice versa 😎 episode #251

In many parts of many organizations, e.g. Marketing, Wendy sees there being very competent people who just don't really understand how to do data well. They need a great partner. Should the marketing leader be focused on what data sources they need and why? Or should we be able to translate their needs into the work? But first, we need to actually be able to partner and they need to understand their needs. Data people can supercharge their efforts but the business partners need to lean in. Scott note: in data mesh, part of the role is enabling them to get better. We need people to up their fluency but doing data mesh or not, everyone starts somewhere and we need to help them level up.

On the flip side, Wendy also sees how often data people are stuck focusing on the data work instead of the business aspects of what that data work is tied to. Without the business context, all you are doing is pushing 1s and 0s. What do business partners need and why?! There needs to be the ability and the courage to just hammer out the understanding differences or the problems will persist.

Wendy also gave some specific examples of too many cooks in the kitchen relative to certain measurements. Instead of there being one official perspective or measurement for something like usage of a cloud product, in a previous role there were many measurements across engineering, finance, marketing, sales, etc. And every single one was different because they all used slightly different methodologies and even sources. So when they tried to look at success of the product, everything told a different story. And when they tried to have a simple bill the customers could understand, it was just not possible. While single source of truth is a complicated and overloaded term, one official source of truth for a question is something you should be able to rally around.

A big problem in many organizations is people are only focused on their own job and lose sight of the bigger picture and especially how they play into that bigger picture of the organization's success according to Wendy. Even if your role isn't directly improving the customer experience, your work can have a positive impact on that if you drive towards that goal. Sometimes, politics around data also gets in the way of collaboration across teams and lines of business.

Wendy talked about another persistent problem in data: the service model. If your data teams are only focused on supporting other teams, you can lose sight of your big picture impact as well as the impact of bad data. She believes data teams need to spend more time creating their own data around their impact and also quantifying the costs of data issues. What are the actual impacts to the organization? And do execs outside the data team understand data well enough to understand those impacts? If they don't understand data, can they even trust it enough to rely on it?

Circling back to the bigger picture, Wendy believes that teams can drive significant process improvements if they just understand the impact of their work - especially through data - upstream and downstream. What do they actually need from others? Who is consuming their data and why? What impact will changes have? How are communications set up to prevent issues and create strong understanding and trust? And then of course, try to automate as much as possible to lower the burden on everyone involved in the data flowing around :) As part of that, please just hire good product managers 😅

Wendy said, "You will never be as successful as you can be as a data organization if you're not able to influence your IT partners, your product teams, your business teams." Data is a team sport, data is about making the organization better. You need others to play with you or it won't work.

When thinking about actual business transformation around data, Wendy said, "There is no transformation without automation." Historically, doing data work has required a lot of effort. The business side just wants to leverage the data, help them automate as much as possible. Otherwise many - most? - business partners won't want to engage with the data and leverage data to improve their work - it's too much effort. Also, removing the friction from data work helps people identify the friction in general business processes. So automating the data work allows them to more easily identify and then address "business choke-points".

For Wendy, too many aspects of data work are treated as wholly separate disciplines instead of treating it as all part of one whole. Security, privacy, compliance/regulatory, performance, etc. We have to shift it left but also stop trying to treat them as discrete challenges to overcome instead of interoperating aspects of a working, scalable solution. Think data by design 😎 That's why she created TheAssociation.AI, "a global nonprofit business organization …focused on bridging the disciplines of AI, data, ethics, privacy, robotics, and security." She said, "there is no security, there is no privacy, there is no ethics, there is no AI without data," but we also need organizations to actually implement their policies into their data and data work. There isn't going to be ethical AI without someone leading that charge and TheAssociation.AI is looking to push that effort forward.

In wrapping up, Wendy circled back to the start. What is the point of doing data work. She said, "What's the point of being focused on the data if you don't understand the business that the data is supposed to be used for?" Being a data leader, especially the CDAO, is VERY tough because you often don't own much of the infrastructure if at all and have to do your work essentially via influence. But if you build the right relationships and understanding of the business, you can still have a major impact and drive significant value for your organization.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#294 Panel: Product Discovery and Data Discoverability in a Data Mesh World - Led by Ecem Biyik w/ Frannie Helforoush, Marta Debska-Barcinska, and Ole Olesen-Bagneux Feb 25, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#293 Adapting Product Management to Data - Finding the Customer Pain and the Value - Interview w/ Amritha Arun Babu Mysore Feb 18, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Amritha's LinkedIn: https://www.linkedin.com/in/amritha-arun-babu-a2273729/

In this episode, Scott interviewed Amritha Arun Babu Mysore, Manager of Technical Product Management in ML at Amazon. To be clear, she was only representing only own views on the episode.

In this episode, we use the phrase 'data product management' to mean 'product management around data' rather than specific to product management for data products. It can apply to data products but also something like an ML model or pipeline which will be called 'data elements' in this write-up.

Some key takeaways/thoughts from Amritha's point of view:

"As a product manager, it's just part of the job that you have to work backwards from a customer pain point." If you aren't building to a customer pain, if you don't have a customer, is it even a product?
Always focus on who you are building a product for, why, and what is the impact.
Data product management is different from software product management in a few key ways. In software, you are focused "on solving a particular user problem." In data, you have the same goal but there are often more complications like not owning the source of your data and potentially more related problems to solve across multiple users.
In data product management, start from the user journey and the user problem then work back to not only what a solution looks like but also what data you need. What are the sources and then do they exist yet?
Product management is about delivering business value. Data product management is no different. Always come back to the business value from addressing the user problem.
Even your data cleaning methodology can impact your data. Make sure consumers that care - usually data scientists - are aware of the decisions you've made. Bring them in as early as possible to help you make decisions that work for all.
?Controversial?: Try not to over customize your solutions but oftentimes you will still need to really consider the very specific needs of your consumers. Build for reuse but also build where your consumers are actually having their needs met. A mediocre solution for all is usually worse than a few specialized solutions.
Prioritization is crucial in product management. That applies to features within the products but also the products themselves. There are many potential use cases that won't be met because there isn't enough value. That's the name of the game, return on investment; it's not about capturing all value possible.
Communication and building relationships/trust are foundational in product management. It's an art as much as a science. If you can't have tough conversations and get alignment, it is FAR harder to build a product that meets customer's needs.
Relatedly, establish regular communication with your customers. You shouldn't only be talking to them when things go wrong. Stay on top of what is driving value for them and look to augment your product proactively, not only reactively.
Product management requires patience as much as diligence. Sometimes your data product/element violates its SLAs but it's an outlier, a one-off. Don't look to overreact and jump to changing things. But you obviously need to have serious conversations if elements aren't meeting expectations over a more extended time period.
If you aren't sure what products you should create in a new area, talk to people and find the points of friction. What are the pain points and is there enough value in addressing them to justify doing the work?
It's crucial to deeply converse with potential users of a data product/element to assess if it's really going to be worth the effort. There is always a chance you build something that isn't used/valuable but through deep investigation and ideation with potential customers, you can avoid that far more often.
When you are building something, even before it hits 'GA', get validation. You can save yourself a ton of effort in rework as you find a better solution sooner.
Product management is about collaborating to drive towards value. You are there to prioritize and coordinate. You don't have to know everything, but your job is to uncover as much understanding as possible to maximize your value creation and minimize wasted work.
Always ask what value building something for your customer will drive. But also ask what happens if we don't build it. What is the cost of not acting?
The only constant is change, especially in data. Leverage a "loosely dependent architecture" to be able to adapt to change. And be open and honest with customers that things will change. Emphasize you'll work with them to adapt to those changes.

Amritha started the conversation on some key differences between software product management and product management around data - whether specific to 'data products' or not. One similarity is the focus on solving a particular user problem but in data, you might have to build something to address multiple users' problems. A much bigger difference is that in data, you often don't own the entire process as you might be reliant on others to source your data. In software, you are generally building the data sourcing because you own the interaction creating the data. How the data is stored and collected throughout the upstream process impacts what you can do.

The user problem, the business value, and the user journey are some key guides to doing data product management well for Amritha. Keep coming back to those as you build out your solution. Focus on understanding what the user really needs and work backwards to the sources. And then of course focus on making sure you are actually addressing user needs when you deploy the solution. There are many reasons a data element may not be performing up to expectations so be prepared to deep dive; is there a problem with what you've built, what's feeding your data element - maybe sources have changed or there is a quality issue -, or is it just not performing to expectations because the hypothesis was wrong?

Amritha dug a bit more into some challenges specific to product management in machine learning and AI. While data scientists want clean data, when possible they want to even be part of the process of selecting the cleaning methodologies - even that can impact the data enough to change outcomes. So really start from the process of bringing them in as a stakeholder as soon as you can and don't throw data over the wall at them. And if you already have something developed, share your methodologies and help them figure out if it's the right fit for them or if something new needs to be developed. Again, we want reuse but we also want solutions that address their problems. Always a hard set of needles to thread.

"As a product manager, it's just part of the job that you have to work backwards from a customer pain point." Amritha questions if you are even building a product if you don't have a customer. What is the business value of the work? For a product manager of product without a customer, are you focused on your own thoughts and biases rather than the needs of consumers? "So the point here is that at any given point, you have to be cognizant of who are you building this for, why, and what that is the primary customer. And the secondary is: who else if I build this, what are the impacts it will have on my secondary customers, or other downstream or interacting applications?"

Amritha talked about one crucial rule in product management: prioritize. There are many use cases you _could_ solve but are they actually worth the effort? Think about what will impact your organization the most. Don't try to solve every use case and don't try to make products that can serve every potential customer - focus on delivering value. Scott note: this can be a slippery slope in data mesh. You want to take on use cases you actually can tackle when you are learning. Don't only go for the biggest value but also tackle problems where the juice is worth the squeeze, where the outcome is worth the effort.

In product management, Amritha believes it's absolutely crucial to understand the art and the science. The science is more about is this product specifically meeting the needs it was designed for. Basically, measuring the level of success and determining if that's good enough or especially is it _still_ good enough. But even that last bit can be a bit of art. The real art is all about communication and building relationships. If you build the world's objectively best product but no one trusts it or understands it enough to use it, it's not a valuable product. You must build strong relationships and have the tough conversations with stakeholders, earning their trust, to align on what needs to get built and why as well as when a product isn't meeting expectations. Establish regular lines of communication so it's not that the only time you talk to your customers, it's bad news or big changes. Continue to extract information from them to drive to business value.

When it comes back to the science, that's when Amritha believes you should dig into the why something isn't meeting expectations from the technical perspective :) And have some patience around that. Sometimes it's a blip on the radar, not anything more.

When figuring out what products/data elements you might want to build in a specific area, Amritha recommends digging into the potential workflows and user journeys. Start to really think about what you think could exist and why. But, instead of trying to ideate only yourself, go and talk to people and listen for their pain and points of friction. They may not even realize they have pain but you can find the challenges that people will want to address. Again, work backwards from the user journeys to discover what products you should build 😅

Amritha talked about how to make maximize the chance that what you're building will be used/valuable. A lot of it is simply digging in deep with potential customers in the ideation phase to make sure this will actually drive value. There are ways to do that but a lot of it is simply spending the time to really understand the likely impact of what you're building. As Alla Hale said in episode #122, "What would having this unlock for you?" Also, ask, "what if we don't do this, what is the impact of not doing this?" And make sure to get validation as you're building. It might be the value hypothesis was wrong or that you're building something that is the wrong or suboptimal way to address the challenge/opportunity. You can save yourself a lot of headaches and rework. It's all about that collaboration to drive to value.

In wrapping up, Amritha talked about how changes, especially in data, are inevitable. Make sure to communicate with consumers so they have realistic expectations. Sometimes those are proactive changes but often, you don't have that much control over changes, especially coming from upstream in data. Look to build in a way that can adapt and leverage a "loosely dependent architecture".

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#292 Aligning Your Data Transformation to the Business - Interview w/ Nailya Sabirzyanova Feb 11, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Nailya's LinkedIn: https://www.linkedin.com/in/nailya-sabirzyanova-5b724310b/

In this episode, Scott interviewed Nailya Sabirzyanova, Digitalization Manager at DHL and a PhD Candidate around data architecture and data driven transformation. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Nailya's point of view:

When it came to microservices and digital transformation, we aligned our application and business architectures. Now, we have to align our application, business, and data architectures if we want to really move towards being data-driven.
To do data transformation well, you must align it to your application architecture transformation. Otherwise, you have two things transforming simultaneously but not in conjunction.
It's crucial to involve business counterparts in your data architectural transformation. They know the business architecture best and the data architecture is there to best serve the business. That is a prerequisite to enable continuous business value-generation from the transformation.
Re a transformation, ask two simple questions to your stakeholders: What should this transformation enable? How should we enable it? It will give them a chance to share their pain points and their ideas on how to address them. The business stakeholders know their business problems better than the data people 😅
Your approach to data mesh, at the start and throughout your journey, MUST be adapted to your organization's organizational model and ways of working. Everyone starts from completely different places.
Data mesh won't work if you overly decentralize. You must find your balances between centralization and decentralization yourself.
?Controversial?: Historically, teams were charged for data work and resources but with something like data mesh, they can manage their data and data costs far more efficiently. Framework processes, tools, and skills help teams to identify which data is valuable for their own or other domains and requires investment, and which data or data processing operations are redundant, and thus, a source of savings.
?Controversial?: You should consider two phases of your early data mesh implementation: “foundation” - enabling teams to own their data by building corresponding teams, processes, and tools and “operationalization and scaling” - enabling/incentivizing them to share their data well with others. They have overlap but if you don't focus on enabling to own data for themselves, you may have trouble incentivizing them to even own their own data let alone share it.
To drive incentivization and prioritization well to do something like data mesh - or really most large-scale transformations - you need top-down support from the highest levels in the organization.
?Controversial?: In highly regulated industries, you will have domains that already have very strong governance practices. Focus on enabling them to safely manage their data within a new framework, rather than trying to change their ways of working.
Relatedly, focus on the frameworks and guidelines as well as the tooling to enable those domains that aren't nearly as advanced in their governance.
If you are looking to federate/decentralize to all domains at the same time, consider how you leverage central committees. If you don't have someone helping guide people towards some degree of consistency, you can't find repeatable scaling patterns/best practices and are likely to create data silos.
The level of importance your company places on your data transformation - mesh or otherwise - should determine how you combine it with your digital transformation. It might be under it as part of digital transformation, entirely separate but at a peer level, etc. But they should get aligned no matter what to have the best impact.
Everyone must understand that data mesh is journey. You will learn and adjust along the way.
Getting budget for your data transformation / data mesh journey is probably more political than many expect. The easiest way to get budget is high-level management attention. Scott note: but of course, it can be hard to get that buy-in first 😅

Nailya started the conversation on the need for application, business, and data architectures to all be aligned. She gave some history about how application and business architecture were brought into alignment when we started with microservices and digital transformation. But now we have to add data into the mix which makes things even more difficult. We need both the application and data architectures to be designed to specifically support the business goals.

When it comes to actually transforming your data architecture, Nailya believes the transformation should be led from the business side where possible. At the very least, the business side should be involved. They understand the business needs best so they can help direct the transformation to serve those needs. Great data work that doesn't support the business needs is often just a well-designed money pit 😅 You obviously need the strong data expertise but a transformation led exclusively by the data team is far less likely to align to business goals and priorities.

In a successful past digital and data transformation, Nailya used two simple questions to the key business stakeholders: What should this transformation enable? And how should we enable it? It gave the business stakeholders a chance to fully lay out their pain points as well as some ideas how to address them. That way, the team had a very broad perspective and could come back to each of the stakeholders with solutions that worked for them somewhat tailored to their needs and thoughts. You need repeatable patterns/approaches but you also need people to feel seen and heard in order to drive buy-in where you address their specific pains and ideas.

When asked about adapting data mesh to an organization's specific challenges, Nailya pointed to how every culture is so different and you need to take into account how people internally exchange information and work together as you design how you want to go forward. Overly decentralizing - so not doing any federation - or decentralizing too quickly won't work well. You have to find your balance between centralization and decentralization throughout your journey.

One interesting buy-in point Nailya mentioned was cost control over data work. Because teams have traditionally been charged for data resources and work by central data teams, they were not as involved in managing costs. Data mesh empowers business teams with tools to control cost-effectiveness of their data, and thus they can identify easier which data is valuable for their business and requires investment, and which data or data processing operations are redundant, and thus, a source of savings. They can see it as a chance to do things better and align better on what work is worth doing - the central data team might have done work that the domain doesn't see as valuable when really considering it more deeply. At first, it might be only for their internal-facing to the domain data work before we can get them bought in that they are now responsible for also sharing their data with other domains.

Nailya talked about her experience with a large-scale data mesh implementation. They focused on first enabling teams to own their own data. So again, giving them the chance to gain transparency to their most valuable data as well as define, align, and prioritize their data initiatives. Then, they started to work to incentivize and better enable them to share their data with the rest of the organization identifying new use cases and data customers. This may delay the biggest benefits of data mesh - high-quality, reusable data across domain boundaries - but it does mean that teams aren't struggling to own their data at the same time as learning to share it with others; this also helps with the incentivization challenge as they can take advantage of their data first for themselves before being asked to focus on sharing it with other domains.

Data governance in data mesh will - unsurprisingly - be hard for every organization in Nailya's view. If there are domains that already know how to handle their data well, work to enable them to better share their information but also don't try to push them towards central ways of working. If they can safely secure and share their data in a way the rest of the organization can consume it, don't get in their way. But you should also look to create frameworks and standards for those that aren't as mature to help guide them along. Scott note: this is an adjustment for those that are already somewhat decentralized with their data work. Again, adjust for your circumstances!

Nailya also recommends central committees to ensure teams are meeting some degree of conceptual consistency and also technical/architectural consistency. That way, you can really find your scalability patterns and best practices. Scott note: if you decentralize/federate all at once, this might be the best bet. But many - most? - are going domain by domain so this function is already embedded in an enabling team.

In her experience, Nailya believes that aligning your data and digital transformation is very important to be able to succeed at data mesh. Again, you need to align the application and data architectures with the business architecture. Really take stock - is your data transformation part of your overall digital transformation, are they at the same level and should be partnered, etc.?

Nailya again circled back to engaging with your business partners/stakeholders to have them help design your transformation efforts. They will lean in from feeling seen and heard but also, they know the most critical pain points best and can help direct where you should focus first.

The conversation finished up around getting and maintaining a budget around your data mesh implementation. For Nailya, this is typically far more political than many might expect. But if you have the proper top-down support and management attention/buy-in, you should at least be able to get going. You have to show value along the way but that should be part of the prerequisite to start: what value are you trying to capture and how will you measure it?

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#291 Panel: Data as a Product in Practice - Led by Jen Tedrow w/ Martina Ivaničová and Xavier Gumara Rigol Feb 04, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Jen's LinkedIn: https://www.linkedin.com/in/jentedrow/

Martina's LinkedIn: https://www.linkedin.com/in/martina-ivanicova/

Xavier's LinkedIn: https://www.linkedin.com/in/xgumara/

Xavier's blog post on data as a product versus data products: https://towardsdatascience.com/data-as-a-product-vs-data-products-what-are-the-differences-b43ddbb0f123

Results of Jen's survey 'The State of Data as a Product in the Real World' (NOT info-gated 😎👍): https://pathfinderproduct.com/wp-content/uploads/2023/12/2023-State-of-DaaP-Real-World-Study.pdf?mtm_campaign=daap-study&mtm_source=pp-blog&mtm_content=pdf-daap-study

In this episode, guest host Jen Tedrow, Jen Tedrow, Director, Product Management at Pathfinder Product, a Test Double Operation (guest of episode #98) facilitated a discussion with Martina Ivaničová, Data Engineering Manager and Tech Ambassador at Kiwi.com (guest of episode #112), and Xavier Gumara Rigol, Data Engineering Manager at Oda (guest of episode #40). As per usual, all guests were only reflecting their own views.

The topic for this panel was data as a product generally and especially how can we actually apply it to data in the real world. This is Scott's #1 most important aspect to get when it comes to doing data - especially data mesh - well. It's the holistic practice of applying product management approaches to data. It ends up shaping all the other data mesh principles and is a much broader topic than data mesh is in his view. But it can also be quite simple in concept when you really boil it down, it just takes patience and focus.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

At its core, data as a product is more an organizational mindset approach than anything else. It is something you work towards. It's not an overnight change but getting the mindset right first - at least with a small core group - will help the organization figure out how to best move towards treating your data as a product.
Data as a product and data product thinking must not fall into only thinking about data products - It's product thinking about data! In general product management, it isn't about creating a product, it's about creating an experience for the customer that generates value for them in some way. The best way to do that sustainably in data is a data product. But the value and experience are the point, that's your focus, the data product is merely a vehicle to deliver those.
Think about good and bad product experiences. The incomprehensible manuals and unintuitive design of a bad product. The awesome tutorials and documentation around a good one. We need to create easy paths for people to not only discover our data products but discover the best ways to generate value from them.
Data as a product includes considering your data sourcing strategy. If you think about a physical good, you don't just have the labor to put it together, you need to consider the materials that need to be combined to create the product. It's not just about what parts you have laying around, you manage the supply chain - hopefully reliably and scalably - to make your widgets. The same should be part of data product practices. It's not just what data you have but what data you will need.
If we actually want our organizations to be data-driven (whatever that means to you 😅), people need to be able to rely on the data. If we want to embed leveraging data into the DNA, the day-to-day operations, of the company, we need to make our data creation and management processes reliable. The best way to do that is via data products because it makes it easy and reliable for consumers to leverage data. The data isn't the point, it's the mechanism to drive business value.
A first step on the road to your organization treating data as a product is getting away from the data team as a service mindset. That they are ticket takers. Data shouldn't be a cost center or ticket-taking organization rather than a value generating one.
A key aspect of good products is usability. We need to focus more on usability in data. That is somewhat wrapped into user experience but there are other aspects that are even more often overlooked than UX. A lot of that usability falls on the platform as well so there isn't a different user experience for each data product.
Moving to a data as a product approach, instilling that data as a product mindset in your organization, will be hard. And it is quite a bit of cognitive load for people who haven't focused on data historically. Look to introduce the concept and changes needed to implement data as a product over time. This isn't a switch you flip.

Other Important Takeaways (many touch on similar points from different aspects):

Data as a product is a lot about the mindset and approach you bring to data. What have we learned from product management - in software and elsewhere - that we can bring to data?
Most good general product experiences - at least on the consumer side - don't need a ton of hand-holding. Can we actually get there on data? Do we want it to be that easy since it's still easy to misinterpret the data? This balance will be different for every organization and probably every data product.
Relatedly, fully self-service is a real question. You want to lower the amount of information requests to data owners but some of those questions can be value generating. So you want to build out the experience - especially documentation - to answer the basics but there's a question of how far you go relative to trying to document everything.
There's a major question about how much change management is involved in learning to treat your data as a product. Is it just a mindset shift? Probably not. But then how do you actually change the organization to start focusing on data as a product?
Jen said, "… the primary purpose of data as a product is to maximize data as utility."
There isn't a single solution or approach to 'solving' product management. There won't for data as a product either 😅 prepare yourself and your teams for that. It's going to take sustained learning and evolution to get better and better. There is no 'done' but there is a ton of value to accrue along that learning journey.
It's okay - if not ideal - to have multiple things in your organization called 'data products'. Not all have to meet the data mesh definition. But make it clear to people what you mean around data products - potentially call them mesh data products - or their thinking on data products will be, "okay, but just what the heck is a data product?" 😅
Software product management is to software products as data product management (the application of data as a product) is to data products. No one thinks software products and product management are the same. We shouldn't in data either.
Data work and learning how to do data well both generally have a high cognitive load. Be prepared for that. Don't expect everyone to get it right away, whether that is the why of treating data as a product or the how do we actually do this :)
Data as a product will be hard to instill even inside your data team. Again, this will take time and you have to let people know the why and the how. Similarly, you need to be prepared for sustained effort to communicate the benefits to those in the business. People aren't jumping up and down to own their own data and especially not to do it well 😅
Are people ready for the best product approaches in data? Doing product management well is about trying, fast failing, learning, then iterating. Are people ready for there to be more continual change in how data is served?
Part of product management is stakeholder management and especially communication. In data, we need to move past requirement gathering but we also have to find better ways to communicate value and get - and then retain - buy-in.
Product management, at the end of the day, is about capturing value through creating value for others. With data as a product, you need to understand what value is expected to be created from your work and where you might increase that value. But data doesn't have inherent value unless it is used so you need to stimulate data usage to create value.
An important benefit of managing your data as a product is the improvements in data user experience. That means more time spent on creating value instead of wrangling data.
Part of good product management is product marketing. That means discovering what data should exist but also finding champions, internally marketing successes, etc. You need people to see the value of the data work to get them to want to lean in. People aren't paying close enough attention to inherently know the impact of the data work, you have to tell them 😅
Products have owners. Treating your data as a product means your data has strong ownership. It's easy to say you are creating data products but really instilling that ownership is crucial to doing data at scale reliably.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#290 Applying Platform Engineering Best Practices to Your Mesh Data Platform - Interview w/ Tom De Wolf Jan 28, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Tom's LinkedIn: https://www.linkedin.com/in/tomdw/

Data Mesh Belgium: https://www.meetup.com/data-mesh-belgium/

Video by Tom: 'Platform Building for Data Mesh - Show me how it is done!': https://www.youtube.com/watch?v=wG2g67RHYyo

ACA Group Data Mesh Landing Page: https://acagroup.be/en/services/data-mesh/

In this episode, Scott interviewed Tom De Wolf, Senior Architect and Innovation Lead at ACA Group and Host of the Data Mesh Belgium Meetup.

Some key takeaways/thoughts from Tom's point of view:

Platform engineering, at its core, is about delivering a great and reliable self-service experience to developers. That's just as true in data as in software. Focus on automation, lowering cognitive load, hiding complexity, etc. If provisioning decision specifics don't matter, why make developers deal with them?
The key to a good platform is something your users _want_ to use not simply must use. That's your user experience measuring stick.
When building a platform, you want to hide a lot of the things that don't matter. But when you start, especially with a platform in data mesh, there will be many things you aren't sure if they matter. That's okay, automate those decisions that don't matter as you find them but exposing them early is normal/fine.
Relatedly, make that hiding easy to see through the curtain if the developer cares. Sometimes it matters to 5% of use cases but also often, engineers really want to understand the details just because they are engineers 😅 Make a platform where people can customize their experience where possible without going overboard.
?Controversial?: Few - if any - current tools in data are "aware" of the data product, they are still focused on their specific tasks instead of the target of creating an actual data product.
Relatedly, the developers should be able to focus on creating and maintaining data products instead of focusing on leveraging specific tools. We need platforms that allow them to deliver value through creating and managing data products, not a focus on working with tools.
?Controversial?: Data mesh without technology is just theory. It can't only be about the people - if you focus on evangelizing without anything practical to show, it is too theoretical or abstract for people. You need a platform early to be able to show people what you mean. Scott note: you need a thin slice that has at least some aspect of all the 4 mesh principles early or your implementation becomes lopsided.
Relatedly, get to something to show people in a demo as soon as possible with your mesh implementation so they can picture it and understand what you're trying to accomplish.
In data mesh, you will still have data developer power users that really want to dig deep. But a key focus of your platform should be to make it easier for non-power users to still build and maintain great and valuable data products. Expand the potential number of people creating data products by lowering the bar.
?Controversial?: The platform team shouldn't be a blocker to new data products being developed. However, you should probably have certain cost guidelines/guardrails so someone doesn't develop a very expensive data product - it should only go to a central team for oversight when cost becomes an issue. That way, you prevent unnecessary friction and costs simultaneously.
When there is an escalation because of a problem with a data product related to cost or governance, look to frame it as a collaborative deep-dive into an issue. Rather than a central 'you can't do this', it’s much more of a 'why is this made this way? Is this optimal or can we change it?' That collaborative discussion can keep people engaged and leaning in.
You can get domains more bought in on something like data mesh/data products by showing them how this new approach won't directly tie data schema to their application schema. That way, they can still easily make changes to their application schemas and not break their data products.
Good engineering is all about managing tradeoffs. Platform engineering is no different. You'll have to look at what should be specific versus generic. Orchestration is one area that should be very generic.
In data mesh, you want to think about the holistic flow of data and data product work. That's why we need a platform. But the tools aren't really that well built to work together. Be ready for that frustration of having to build on top of the tools to get them to play nicely.
?Controversial?: Even if you aren't doing data mesh, your platform should focus on abstractions. What matters and why is a fundamental question. You want people solving challenges that add value.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#289 Building the Right Foundations for Generative AI - Interview w/ May Xu Jan 21, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

May's LinkedIn: https://www.linkedin.com/in/may-xu-sydney/

In this episode, Scott interviewed May Xu, Head of Technology, APAC Digital Engineering at Thoughtworks. To be clear, she was only representing her own views on the episode.

We will use the terms GenAI and LLMs to mean Generative AI and Large-Language Models in this write-up rather than use the entire phrase each time :)

Some key takeaways/thoughts from May's point of view:

Garbage-in, garbage-out: if you don't have good quality data - across many dimensions - and "solid data architecture", you won't get good results from trying to leverage LLMs on your data. Or really on most of your data initiatives 😅
There are 3 approaches to LLMs: train your own, start from pre-trained and tune them, or use existing pre-trained models. Many organizations should focus on the second.
Relatedly, per a survey, most organizations understand they aren't capable of training their own LLMs from scratch at this point.
It will likely take any organization around three months at least to train their own LLM from scratch. Parallel training and throwing money at the problem can only take you so far. And you need a LOT of high-quality data to train an LLM from scratch.
There's a trend towards more people exploring and leveraging models that aren't so 'large', that have fewer parameters. They can often perform specific tasks better than general large parameter models.
Similarly, there is a trend towards organizations exploring more domain-specific models instead of general purpose models like ChatGPT.
?Controversial?: Machines have given humanity scalability through predictability and reliability. But GenAI inherently lacks predictability. You have to treat GenAI like working with a person and that means less inherent trust in their responses.
Generative AI is definitely not the right approach to all problems. As always, you have to understand your tradeoffs. If you don’t feed your GenAI the right information, it will give you bad answers. It only knows what it has been told.
Always start from the problem you are trying to solve rather than the approach you are trying to use. Then evaluate if GenAI is the right approach for that problem. Simple, fundamental stuff but it's crucial to remember: start with the problem before the proposed solution.
Many people are leaping to use GenAI because their past approaches to certain problems haven't worked. Dig into those pains. GenAI may or may not be the right approach but either way it can be great for surfacing persistent challenges.
Leverage people's enthusiasm for GenAI to have deeper conversations about general business challenges. It can really start to highlight friction points across organizational boundaries and who is responsible for what. Scott note: But as the data team, be careful not to try to fix the entire organization, that's not what you are responsible for 😅
Right now, despite all the hype, most organizations are still at most in small-scale PoCs around GenAI. There is less of an initial focus on return on investment versus what capabilities GenAI might unlock but there is also a focus on what risks GenAI may introduce. Despite the hype, many to most organizations are doing their diligence.

May started with three general approaches organizations are taking to generative AI (GenAI): 1) building their own LLMs from scratch, 2) fine tune specific, pre-trained existing LLMs, or 3) leverage pre-trained LLMs as is. Many organizations may want to do the first but it is prohibitively expensive to train your own LLMs from scratch just for the compute and you also need (very expensive) people with very specific expertise to do so. Tuning pre-trained models will likely become the standard approach for many organizations. However, being able to leverage LLMs on internal data in general requires "existing good quality data and solid data architecture."

When considering training a model from scratch, May also pointed to time as an issue. Typically, it takes at least three months to properly train an LLM from scratch. Parallel training is helpful but you need to fine-tune results and retrain so you can't just throw compute at it and make the process that much faster. So again, you need high quality data - and you need a LOT of it - plus a fair amount of time plus a ton of money. Once you are in production, it also takes a lot of money and effort to keep them running and tuned properly 😅 Luckily, according to some surveys Thoughtworks did, most organizations recognize training LLMs from scratch isn't the right call for them just yet.

May is seeing a trend of people moving away from the 'bigger is better' mentality. More people are starting to explore more targeted and specialized models that have fewer parameters. And often, for specific tasks, they perform better than the first L in LLMs. So we may see a trend towards more and more targeted LLMs/models. Scott note: Madhav Srinath really leaned into this in his episode, #264.

Humanity in general has benefited greatly from machines through predictability and reliability according to May. Essentially, if they are made well, you essentially know what you should/will get from machines. But GenAI is designed specifically to act like humans and humans are not predictable and often not that reliable. So people have to get used to interacting with machines that may give wrong answers and are designed - in a way - to do so 😅 We can't expect predictability and reliability from GenAI.

Relatedly, when thinking about where is GenAI the right choice versus like traditional machine learning/AI, May believes you really have to dig into the tradeoffs. If you really understand the problem set and what you are trying to accomplish, traditional ML/AI is probably the better approach for you. You need to really understand where the strengths of GenAI will play and feed it the data/information it needs to succeed, otherwise you'll be asking an uniformed and unpredictable entity to solve your most pressing business problems. That's probably not going to go well…

May talked about going back to the basics of problem solving when it comes to Generative AI: what problem are you trying to solve instead of what way are you trying to solve a problem and then finding your way back to the problem. It can sound obvious but really, many are in such a rush to leverage these tools, it's crucial to stop and consider. Start with the problem before the solution 😅

GenAI may also surface a number of internal business challenges that aren't spoken about or people have essentially given up on tackling previously according to May. We have a new tool in the toolbox so people want to see if it will be useful to tackle something they haven't been able to address well previously. Lean into GenAI as a conversational lubricant. GenAI may not be the right tool for every one of these challenges but it means there is more internal conversation and sharing :)

From what May is seeing, many to most organizations are still in the early experimenting and PoC phase with Generative AI. They are trying to figure out what opportunities GenAI brings and also what risks. Despite the hype, people are taking their time but they aren't as focused on initial return on investment, more to validate if they can actually leverage GenAI to create value. Also, there is strong trend towards domain-specific LLMs rather than general purpose ones, e.g. financial sector or media specific models.

May finished on the idea that data mesh and other data management paradigms are crucial to doing something like GenAI right. There is still a strong need for quality data that is accessible, interoperable, privacy-aware, secured, etc. to be able to leverage GenAI well.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Major Programming Announcement Jan 21, 2024

Announcing moving to one episode per week :)

#288 Panel: Master Data Management in a Data Mesh World - Led by Ole Olesen-Bagneux w/ Liz Henderson, Piethein Strengholt, and Samia Rahman Jan 18, 2024

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount!

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Ole's LinkedIn: https://www.linkedin.com/in/ole-olesen-bagneux-2b73449a/

Piethein's LinkedIn: https://www.linkedin.com/in/pietheinstrengholt/

Samia's LinkedIn: https://www.linkedin.com/in/samia-rahman-b7b65216/

Liz's LinkedIn: https://www.linkedin.com/in/lizhendersondata/

Ole's book The Enterprise Data Catalog: https://www.oreilly.com/library/view/the-enterprise-data/9781492098706/

Piethein's book Data Management at Scale (2nd Edition): https://www.oreilly.com/library/view/data-management-at/9781098138851/

Liz's blog: https://lizhendersondata.wordpress.com/

In this episode, guest host Ole Olesen-Bagneux, Chief Evangelist at Zeenea (guest of episode #82) facilitated a discussion with Piethein Strengholt, CDO at Microsoft Netherlands (guest of episode #20), Liz Henderson AKA The Data Queen, a board advisor, non-executive director, and mentor in digital and data at Capgemini (guest of episode #106), and Samia Rahman, Director of Enterprise Data Strategy, Architecture, and Governance at SeaGen/Pfizer (guest of episode #67). As per usual, all guests were only reflecting their own views.

The topic for this panel was modernizing master data management (MDM) and applying that to data mesh. It's a very challenging topic to cover because even people's general definition of MDM can be pretty different and there is a question between simply mastering data versus trying to globally compared to locally manage master data. It's a very tricky topic in data mesh. I sometimes use the term 'mastered data' because I think it is far more applicable in that situation than 'master data' - audio goes through a mastering process, so must the master data to actually reach a certain quality level. 'Master data' is more the core linking data. But even that is still just one person's definition.

Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Also, there was a bit of a misstep around intros if that gets a bit lost in there.

Scott's Top Takeaways:

The historical impression of MDM - striving for that single golden record - needs to change. Trying to head down that path in a federated/decentralized approach is even more difficult that in a centralized world. And the benefits just keep proving out to not be worth costs.
You need to consider your master data management strategy to do data mesh well. Do you want official sources of truth relative to specific questions? Who owns data quality for core linking data? Etc. You don't have to get it perfect at the start but if people don't trust the data on the mesh, your mesh implementation probably fails. Mastering data can improve the quality - especially the provability of that quality - and thus trust. And without strong linking data between data products, do you just have high quality data silos?
Relatedly, there is a tipping point in an organization related to size and complexity where you need to consider data mesh. There is a tipping point in a mesh implementation where you need to really start to push MDM. It doesn't have to be on day one but you should plan from the start around MDM.
MDM - at least done well - is not about getting to 'perfect' data, it's about understanding the needs of your organization and helping people get to the right quality level and providing core data when it's needed. Not all data needs mastering. Not all data is master data.
There's a difference between 'single source of truth' and 'most trusted source of truth'. It's absolutely okay to designate 'sources of truth' for specific questions. Other data sources may provide different perspectives on the same topic - e.g. customer is different in sales, marketing, and finance - but there needs to be one right, repeatable answer for things like regulatory reporting or financial statements 😅
One reason MDM is such a risk to data mesh is that, to be effective, some part of master data essentially has to be centrally managed. You need ways for domains to adhere to central standards, guidelines, policies, etc. Otherwise you risk silos. BUT centrally managed often leads to inflexibility. It can be tough to thread this needle. Look to provide the value add from MDM but limit the overhead and rigidity. How that will apply to your organization will be quite specific but go talk to others in your space to very specifically understand their approaches. No one-size-fits-all, no copy/paste. This is going to be very hard but don't skip it.
If you are going to do MDM in your data mesh implementation, much like with anything in data mesh: test, learn, and iterate. Don't do a huge upfront implementation or it will cost far too much and limit your agility and flexibility far too much. Plan some aspects of your MDM implementation out ahead of time but trying to do everything at the start is a massive anti-pattern.
There is a massive push and pull in data mesh - one where you have to find the right balance specific to your organization - between master data and maintaining the domain-level meaning and understanding of data in domain-specific data products. This is where the enterprise data warehouse often goes wrong: focusing on fitting the data together (master data) at the expense of its actual meaning and uniqueness. It's a balance. Scott note: I often call this local versus global maximization, this time related to business context :)

Other Important Takeaways (many touch on similar points from different aspects):

Liz asked an interesting question: if we are creating true high-quality data products, have we by definition mastered the data inside? Is that good enough? The answer to the first is maybe? The answer to the second seems to be no, we need to look at how data in a data product fits in the broader scope of the organization's data needs or we're just creating high-quality data silos as data products. That is that master data versus mastered data.
Do we only need to consider data that is broadly reusable across the organization for mastering? It can be quite complex - and political too 😅 - figuring out what data is deserving of being mastered and what data should be master data.
Historical MDM approaches have been extremely costly often without the return on investment to justify them. Look to consider a point Khanh Chau made way back in episode #44: if everyone is doing similar cleaning work instead of that being in the data product itself, your total cost of ownership skyrockets. In MDM, when there isn't a clear owner for data because it's so broadly used for linking, that may fall back to the data team. NOT IDEAL but such is life/the real world.
Be super clear internally as to your definitions around what MDM means and its application. It will look different in any organization. Much like the fun "what is a data product?" conversation.
Dive into the difference between reference data - e.g. standardized country codes - and master data. Data that can be combined but isn't of a high enough quality still needs to be mastered. It's not only about creating interoperable data but data you would actually want to use upon combining it.
As Piethein mentioned, MDM should not be managed at the domain levels. That creates a massive mess of contention.
MDM is especially key in industries with lots of externally purchased data - e.g. financial services or life sciences - because many domains all leverage the same data. It saves time and money to have that managed once instead of 10s of times 😅
Should data consumers push insights and data improvements back to their data producers? How do business logic and transformations of the data play into MDM? Does it matter what we call it as long as we allow value-add work to easily flow and become scalable?
Because of the data products approach with very clear ownership, data mesh may actually make MDM easier than the traditional centralized approach. There is - at least there is supposed to be - clear lineage, documentation, other metadata, ownership, etc. so we can all understand more easily what data we should use and for what purposes. Or at least ask the owners for more information to get a better understanding of that.
MDM without strong metadata ownership and management is just like data mesh without them: a disaster waiting to happen.
MDM and DDD: there is a complex interplay between how MDM and Domain Driven Design work together. Every domain has its own unique 'language', even a somewhat unique language for communicating externally, but you do need some more broad language of the organization. Getting that crisp so your data products fit a broader taxonomy or ontology is going to be a MASSIVE challenge.
If you don't have strong documentation and metadata management, you are far less likely to see a valuable MDM implementation, data mesh or otherwise. People will discover data and use it whether they really understand it or not 😅

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#287 Driving Data Value Through Creativity, Curiosity, Collaboration, and Communication - Interview w/ Tiankai Feng Jan 16, 2024

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount!

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#286 Mastering Master Data Management in a Modern World - Interview w/ Sue Geuens Jan 14, 2024

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount!

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Sue's LinkedIn: https://www.linkedin.com/in/suegeuens/

In this episode, Scott interviewed Sue Geuens, Director of Data Governance and Product Data at Elsevier. To be clear, she was only representing her own views on the episode.

We use the phrase MDM to mean master data management throughout the episode.

Some key takeaways/thoughts from Sue's point of view:

At the end of the day, if you want to do data governance well, it's about the people. Go talk to them, find out their specific needs and desires and work to tailor your language - and presumably your application of policies when possible - to their situations. People want good data, help them get there!
Relatedly, get good at telling stories about data work. Get people to lean in and get them involved. Personalize your communication!
While policies and standards are crucial, they are about creating better data for the organization. Try to leverage them as a carrot instead of a stick.
?Controversial?: Don't talk about someone owning data. That's scary for most. Find ways to get them excited about owning the data without making it scary by using different phrasing.
The key to doing data governance well is getting people to care. We need them to care about the data because others have to use it. And that means the people are the most important focus.
Data governance is too focused on 'governance' and that means oversight. The word governance has a bad connotation for a reason - it makes many potential allies uncomfortable. So governance folks have to really work to make it less scary.
Don't focus so much on the data aspects of data work when talking with stakeholders. It's about achieving outcomes through data, not data work itself. Focus on what gets your business partners excited and that's (unfortunately) usually not the data itself.
It's easy to fall into thinking about what you want from others in governance. But where you can add far more value is starting with what others want from you and working back towards solutions that accomplish both your and their goals.
?Controversial?: Prioritization in data governance work is crucial. A good method is looking for who shouts loudest for help. They are ready to lean in and are more likely to be leveraged as your governance advocates once you help them.
The two big reasons MDM initiatives have failed historically are 1) not having the governance, quality, and metadata embedded into the data and 2) striving for "perfect" data, that single golden record.
Relatedly, in data we don't need perfection, the juice isn't worth the squeeze. We need good enough and we need to reflect the realities that our world is ever changing and there are multiple perspectives on the same data that can all be right/correct.
It's easy to lose people if you start talking the 1s and 0s of data. Focus on finding stories that resonate with them. If they see the value of the work, you have a much better chance of getting them to actually do the work 😅
MDM in data mesh should be all about "ensuring that you get the right data for the right purpose at the right time for the right person."

Sue started the conversation as other data governance experts have - the word governance strikes at least discomfort if not fear into the hearts of many of our colleagues. We need to expect that discomfort and be active in dispelling the myths around data governance as it really is about achieving better outcomes for all. But that means more carrots than sticks, which can be a tall task when it comes to things like regulatory compliance. Basically, it's not easy 😅

Another aspect Sue pointed to is that many - most? - data people really like to talk data. So, instead of talking to outcomes, they talk about the data work, and data work for the sake of data work has kind of been one of the big historical challenges of data - instead we need to focus on the value that comes from the data work. If your business partners are already uncomfortable simply by the phrase data governance, not leaning into their value from the data work and target outcomes is likely to lose them even further. Start the conversation with what they might want from you, not what you might want from them.

Sue specifically said she starts partnering with people by focusing on those target outcomes and how might she be helpful to them. Especially, what are their expectations of her? By trying to walk in their shoes, she can come to better conclusions and find working solutions. It's about getting them to lean in. Scott note: and then she can trap them! In a virtuous bi-directional value trap of course…

Relatedly, prioritization in data governance is key in Sue's view. What are the problems that really matter? While the "who shouts loudest" test may not point to the most valuable problems, it often points to the problems people value most and thus you can find willing partners. Trying to enforce others to care about their data is a hard road but if people are ready for your help, you can make a huge difference and they are willing to lean in. Those are also likely to be your biggest advocates once you help them, gaining your governance efforts more momentum by leveraging champions.

There are many reasons why Sue believes people are skeptical of master data management (MDM). Historically, there were two big reasons MDM projects failed. The first is not really focusing on integrating MDM into the data so not having the governance, quality, and metadata embedded into the data and processes. The second is the drive towards perfection. Instead of focusing on what was good enough, there was this focus on the 'golden record'. That led to inflexibility, poor scaling, high costs, etc. Good data work isn't about being perfect, it's about being good enough.

Sue circled back to her focus on working with people. Good governance isn't about perfect data, it's about getting people to care about the quality of the data. That means working to get them to understand what is good enough and why should they care. It's not all just empathy - there needs to be some oversight and making it part of their job - but with humans in the loop, your data quality will be much better if you get people to care about who else uses their data and why.

When it comes to actually getting people to understand data governance work - whether MDM or anything else - Sue recommends personalizing your communication. While that may not scale perfectly, again, find your key stakeholders and partners. Stories about data work in a vacuum just don't resonate - Scott note: is there a physics/sound joke in there as there is no air for sound to resonate in a vacuum…? 😅 - Getting people to understand that the work has a purpose and it really is useful to specifically them is crucial. Don't talk to the 1s and 0s of data!

When it comes to specifically data ownership, Sue has seen just how scary that ownership word can be. It's not an easy task but we need to find ways to instill people with the excitement around ownership without the fear. Again, easier said than done but it's about getting things to the right place not about doing something right now. It will take time but it's better to do it right.

If you don’t take care with implementing data mesh well, Sue believes it will be a far bigger mess than if you didn't try data mesh at all. (Scott note: strong agree) You need to focus again back to what are you trying to accomplish and what data needs to be put in place to do that. MDM in data mesh should be about "ensuring that you get the right data for the right purpose at the right time for the right person."

In wrapping up, Sue emphasized the need for personalizing your communication around getting people to do data work with care and prioritize it. You need to be able to speak to them in their language and get them excited about the impact of the work.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#285 Getting Depth and Value From Generative AI - In Data Mesh and in General - Zhamak's Corner 33 Jan 11, 2024

Key points:

Thus far, most of the generative AI stuff Zhamak has seen is not that much of a differentiator. They are doing far better chat bots but that hasn't really changed the game.
When it comes to any ML work - and GenAI is just a subset of ML work - engineers need data products to make their data work easy. Reliable sources of data, ability to version, etc. Data mesh obviously plays well there.
Relatedly, we need to continue to make things easier for people to leverage data products for GenAI. Engineers shouldn't have to spend all their time moving data around and using many systems.
GenAI really could be game changing in data mesh but right now we don't have enough information to really do it well. We need far more metadata around things like data products.
GenAI often gives extremely shallow answers that just aren't that helpful. If we can get better answers, amazing. But right now, it's not there.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#284 Breaking Down the Monolith - Incentivizing Good Choices - Interview w/ Frederik Nielsen Jan 07, 2024

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Frederik's LinkedIn: https://www.linkedin.com/in/frederikgnielsen/

In this episode, Scott interviewed Frederik Nielsen, Engineering Manager at Pandora (the jewelry one, not the music one 😅).

Some key takeaways/thoughts from Frederik's point of view:

Your data technology and architecture choices incentivize certain behaviors. Consider what behaviors you want before you lock yourself in to anything.
Advice to past data mesh self: "construct a data architecture and platform that can adapt to the business requirements and wishes [which] will change over time." Build a composable platform as it's "easier to adapt to changing business requirements." Focus on decentralization features and make it decoupled and composable.
Trying to go too wide with your data mesh implementation at the start with all your domains makes it harder to really find your groove and build momentum.
Cost transparency can be a big driver for data mesh adoption. Teams want to understand their costs and many organizations are driving cost cutting initiatives. Decomposing the monolithic approach to data means better understanding the cost of individual pieces of data work.
Relatedly, when teams are responsible for their own costs, it's easier to spot when someone is making tradeoffs related to cost. It's a more tangible decision and can be a conscious decision to take on tech debt.
When taking a concept like data mesh to the highest levels in the organization, attach it to tangible use cases. Make it something that is worth their while, the 'juice must be worth the squeeze'. Focus on the strategic business goals and priorities.
It's okay to leverage management consultants. But your data ownership should very clearly be internal - external parties should not own any aspects if you want long-term success. Regarding consultants: "you would rather be driving them than them driving you."
It's absolutely normal for some teams to be more data mature than others. If teams raise their hands saying they need help with their data work, your culture is mature enough where teams ask for help and they should get it where possible.
?Controversial?: It's potentially better to focus on your more data mature teams first when going with data mesh so you can move faster early.
If possible, create golden paths or pre-configured approaches for less data mature teams to be able to still create data products.
It can be hard to show domains why they should move to data mesh. Focusing on use cases is probably the best approach but finding use cases enticing enough to each domain can be a challenge 😅
Tying your data initiatives to the company strategic priorities is crucial to get buy-in. E.g. personalization and omni-channel experience - how do you tie your use cases back to what is most important to the business?
At the heart of it, data mesh should be about driving business outcomes - especially the ones people really care about. Focus on that and you will have a far higher chance of success and getting/maintaining buy-in.
Make sure you build your data products in a scalable way. That means understanding when you need to put information into separate data products instead of trying to combine it all into one - that is just a mini enterprise data warehouse / microlith.
If your data team is remaining quite productive but the backlog is ever increasing as is the time between request and delivery, then your central data team might be a bottleneck. Consider addressing that with something like data mesh.
?Controversial?: Less mature domains can get a more "watered down" version of data mesh as they learn to actually manage and own their data. You don't need to start with the most complicated aspects and use cases first. Scott note: this can be a slippery slope
When mapping out potential use cases, ask the amount of effort - if it's even possible - to execute in your existing (non-data mesh) architecture. If it's not possible, data mesh can mean far more data capability for the organization, which can be a great selling point.
A decentralized architecture can mean cost savings by getting far more fine grained, e.g. shutting off test environments over the weekend or at night. You can find places to be more efficient far more easily.

Frederik started with a bit about how their initial data mesh journey started - and it wasn't great 😅 It was led by management consultants and was focused on real-time data with a very tangible use case. However, two things came from it: 1) a better understanding of what data mesh should actually be used for and 2) buy-in around a very specific use case at the highest levels. So while there was a misinterpretation of data mesh and the use case wasn't the best fit, there was still excitement about the term - and somewhat the actual meaning broadly internally. Making it tangible got people to see the potential benefits.

Cost transparency has been a major driver for data mesh internally according to Frederik. Because the costs in a large monolithic stack are very opaque, decomposing the architecture has led to a far better understanding of the cost of individual pieces of work. Because inflation concerns were a big factor for retail in 2023, there was a bigger focus on cost reductions. Being able to give teams the freedom to take different approaches but making them responsible for the costs has led to better cost efficiency - teams can choose more costly methods but those decisions are more exposed. Also, because you have much finer-grained control, there are far more levers to pull when it comes to cost savings, e.g. shutting off test and dev environments at night or scaling up and down dynamically.

Frederik talked about a common pattern when moving to data mesh: some teams are more data mature than others. There will be plenty of teams that need help when it comes to data mesh, especially building good data products. They are considering creating a sort of golden path or easy button approach for those teams that aren't as mature as well to make things relatively pre-configured instead of having to make many complex decisions.

When driving buy-in at the wider level for data mesh, Frederik talked about pitching data mesh as an entire organization transformation versus pitching use case by use case. He believes it's probably better to focus on the use cases but it can be hard to focus on the complete picture of everything you need when you are also focused on specific use cases. It's always a balance between what is needed only for the use case and what is good for the overall company approach to data.

For Frederik, there are two big company strategic priorities: personalization and omni-channel experience (experience across in-store and online). So much of what they have been focusing on is finding use cases that tie into at least one of the priorities because then there will be executive support. Constantly tying the data work back to what people care about shows an understanding of the business instead of doing data work for the sake of data work. However, these are very big challenges across many domains and teams. So making sure to do things in a scalable way and finding the right balance between data products with still high interoperability is crucial.

When discussing bottlenecks, Frederik talked about how the measure for the centralized data team becoming a bottleneck was when the time between a data request and the actually delivery was expanding. The backlog was ballooning even though the data team was quite productive. Many people will feel the pain of the increasing time to delivery, leverage that while still showing a productive team. If you are executing well but aren't succeeding, you need a new strategy.

Frederik talked about the fact your data technology and architecture decisions will incentivize certain behaviors. A monolithic platform incentivizes monolithic ownership and handing off work, responsibilities, etc. When they introduced Kafka, it enabled them to push ownership upstream to data producers because the new technology allowed data producers to more easily own their data. It's of course difficult to incentivize your desired behaviors but always think about what you want to happen and try to make that the easy/happy path.

When it comes to ownership of data, Frederik thinks maturity really matters. When you want to go down the path of data mesh, trying to get every domain to really be advanced with data is just not that realistic. Some teams just don't see data as their focus so if they won't leverage much data for analytical or ML/AI use cases, they are less likely to want to own their data. And less capable quite frankly.

Circling back to tangible use cases, Frederik talked about one really key use case that saw a really big uptake that they couldn't really accomplish before going the data mesh route. Being able to tie something to actual impact, that really helped people get more interested. Whether that is a business capability or directly impacting a business metric. Similarly, when trying to find new use cases, the team did a lot of user journey mapping. The data for that user journey lives in many systems so you need lots of teams participating to make the data available but it can have a big impact on business. Many companies probably can't do something that complex in their existing architecture. You can use the inability to do amazing things in your existing architecture as a potential selling point.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#283 Selling Data Mesh to Your C-Suite and Board - Mesh Musing 58 Jan 04, 2024

Quick Summary Points

Talk to the business strategy importance - data is there to make things better for the business. What could being better informed mean for your execs?
When people ask about the strategy, that is when you can mention data mesh. It isn't about doing data mesh but you also aren't inventing this whole-cloth. 100s to 1000s of organizations are already on the journey. But data mesh is not some magic phrase, it is merely a framing for doing data better at scale.
Think of the first hidden data demon from my upcoming mini-book: this is about getting to data driven, not being data dragged. This is about better equipping the people you thought were good enough to hire for their expertise and making them even better.
Think of the second hidden data demon: data isn't only about strategic decisions - this gets us into a place where we can make better day-to-day execution decisions too.
We don't get to skip leg day. I originally typed 'leg data' and maybe that's what we call the foundations 😅

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#282 Not Sweating the Small Stuff in Data Mesh - Interview w/ Mandeep Kaur Dec 31, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Mandeep's LinkedIn: https://www.linkedin.com/in/kaurmandeep80/

In this episode, Scott interviewed Mandeep Kaur, Enterprise Information Architect at Nordea Asset Management. To be clear, she was only representing her own views on the episode.

Nordea has been on their data mesh journey for a while and Mandeep has been trying to figure out best practices for the hundreds - thousands - of micro decisions in a journey. So how do we get comfortable with making so many calls?

Some key takeaways/thoughts from Mandeep's point of view:

"1) don't overthink it; 2) bring value out as soon as possible; [and] 3) evolution before completion."
The micro decisions in data mesh do matter, give them some thought. But it's important to simply get some perspective from the people who should know best and move forward. That can be from people inside or outside your organization but think about the blast radius of getting something wrong before you fix it. Most times it's smaller than you'd expect.
Your first question when considering data mesh: what value am I trying to get out of it? Think about what are the target value propositions and what does it do for the business if this is successful. If you don't have good answers, should you do data mesh?
The answers to the 'what value' question of your own mesh journey above should drive your strategy, where you should focus early and what will measure your success. And every organization will have different answers.
?Controversial?: There's a LOT of overthinking in most data mesh implementations 😅 come back to your anchoring points around ownership/accountability, product thinking, value proposition, etc. What's important? You can try something and see if it works and change it if it doesn't, don't get caught in analysis paralysis.
Relatedly, always focus on the value proposition. If you are delivering value, you can improve the other aspects as you move along and learn to do aspects of your journey better.
There's a major challenge in abstract communication, especially about something like data mesh: those doing the abstracting have so much more time and specific research that there will always be logic leaps and things that only map to their own mental model.
Get specific in your data mesh examples and anecdotes while providing abstractions. And be prepared to dive deeper in 1:1 conversations when it makes sense.
The internal communication of something like data mesh shouldn't only fall on the data team. Find/create your ambassadors/champions. They can communicate concepts and aspects well in the language of the business and have mental models closer to their peers.
?Controversial?: As the data team, our role is best served guiding people to the right ways to answer their own questions rather than answering things for them. That's part of data mesh but it's a good practice outside it too.
With data mesh, you will probably feel it's about 'getting it right'. It's as much about learning how to 'get it right' as it is about actually getting it right. That's product thinking for you.
When making your decisions, it's all about trade-offs. Think about what you aren't willing to trade-off. That will guide you more and more to your crucial decisions. Nothing is perfect and nothing is 100% sure.
When making your data mesh journey plan, you must take into account your competency gaps - either to fill them before your journey or how you will compensate for them and fill them as you go along. Scott note: ignore your competency gaps at your own - significant? - risk.
Set milestones for your data mesh journey so you can measure your success to some degree. It also makes you feel better about the progress you've made. Celebrate the success even if there is far to go!
We need to make business users understand data and technology are there to enable them to be better. Too many still see them as a threat.
?Controversial?: Something like data mesh might be seen as threatening to some data consumers by reducing their control and their value creation. Previously, many consumers were in charge of getting access to the data and doing the analysis but that is often pushed to the producer in data mesh. Scott note: this was a really interesting point that hasn't come up before
A product is only a product if it's providing value. Ingrain this into people doing data work.
Really drive to a common understanding of terms like a data product or data as a product. It's very easy for many people across the organization to interpret/understand them differently making communication extra challenging.
Product thinking isn't only focusing on the end product but the supply chain and component parts. That's even more important when that is relating to data; the inputs all matter and the reliability/sustainability of the inputs matter.
Transformation journeys are often quite disjointed at the start. You have to align people to start the transformation and people aren't moving together. There is a disjointed phase before the planning phase 😅 That's normal and to be expected.
If you focus on the target instead of the journey to get to the target, you're as likely to make things worse. You can't simply jump to the end target state, you have to learn how to get there and improve along the way. This is especially true with data mesh.

Mandeep started by discussing one of the key challenges in talking about data mesh: there are so many areas to cover that we often discuss things abstractly. Those abstractions are based on a significant amount of research, discussions, and related work that create a mental model. When we communicate those abstractions, it's hard to communicate the mental model as well. The listeners just aren't as deep into it so much of it goes over their (our) heads. So, we need to get far more specific with anecdotes and examples. We also can't forget the value of 1:1 conversations to drive to deeper understanding. That might not always be the most scalable but it is the best way to prevent misunderstandings. Basically, communication around data mesh is hard! Go talk to people. Scott note: Data Mesh Understanding exists for this reason…

When looking at how to get specific internally with data mesh communication, Mandeep is always on the lookout for her ambassadors or champions. Within a domain, they have strong domain knowledge to connect what you are trying to achieve with data mesh to what the domain is specifically focused on. And they can obviously communicate well in the language of the domain. Connecting the changes data mesh brings to real world problems helps people understand the what and the why.

There is a lot of risk of analysis paralysis in any data mesh implementation according to Mandeep. There are hundreds of 'micro decisions' but if you focus on the core aspects of what you're trying to do, that should guide you to the ones that matter the most. A bit of don't sweat the small stuff. Always come back to the value proposition because you can change things as you learn more. That's not to say be sloppy or careless, there are important aspects like using the right architecture, having strong ownership/accountability, product thinking, etc. But data mesh is as much about learning to get it right as getting it right. And always return to your trade-offs. What aren't you willing to trade-off and why? Once you answer that, more and more solutions become tenable and you can weigh the pros and cons.

Mandeep started to dig into the crucial first question to a data mesh implementation: what is the value you hope to get out of it? And there are different answers for each organization. Those answers will start to inform where you should focus and when in your mesh journey. That will help you set your plan because "a target without a plan is just a dream". And when you form your data mesh plan, think about what you have to adapt to your organization and why. This is not a copy/paste approach! You almost certainly will have competency gaps so how do you plan to fill those gaps and make progress while doing that? Or do you have to fill those gaps before starting the journey because they are journey blockers? Really consider the journey, not only the target outcome. Relatedly, set some milestones for your journey to help you measure your progress and celebrate the progress you've made. They might not be the best success measures once you're further along in your journey but that's okay, you can adjust. That's product thinking.

Mandeep wanted to stress three quick points: "1) don't overthink it. 2) bring value out as soon as possible. 3) evolution before completion."

Many business users still see technology as a threat rather than an enabler in Mandeep's view. They aren't even thinking about data yet, they are still on just tech 😅So there is a lot of work in communication to get them to see data as a major innovation enabler, something to drive their part of the business to new heights.

Another interesting aspect Mandeep talked about was that self-serve might actually be seen as threatening to data consumers. Previously, they controlled - to some degree - their own ability to get access to data but now, it's on the producing team and consumers only get what producers are willing to share. The consumers created the business value by doing the analysis and transformation and now that is pushed much more onto the data producers. Will consumers feel their power and importance is diminished? If the value of data work is attributed to the producers, will data fluent consumers still lean in to leveraging data as much as they did previously?

Mandeep returned to product thinking and her view that a product is only a product if it's providing value. You start from the value you are trying to deliver and work backwards. Build your KPIs around actually delivering value instead of simply creating data products with the hope they create value.

When thinking about data as a product, Mandeep encourages everyone to have conversations about it in their organization and what it will actually mean and look like in their specific organization. Because it's easy to assume everyone is on the same page when they really aren't. And that confusion will bite you in the end with more friction than clearing it up early.

Mandeep believes that in a transformation journey, it almost always starts as somewhat disjointed - a disruptive phase before the planning phase. Part of going on a journey is preparing for that journey. Once people are aligned, that is when you can really start all heading forward. You need pioneers or leaders, those front runners to show people it's safe. But it will still take some time before that alignment. Don't get concerned when that happens even if it feels like everyone should align after the first presentation 😅

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 31, 2023 Dec 31, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#281 Panel: Data Contracts and Data Mesh - Led by Jean-Georges Perrin w/ Amy Raygada and Andrew Jones Dec 28, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/

Amy's LinkedIn: https://www.linkedin.com/in/amy-raygada/

Andrew's LinkedIn: https://www.linkedin.com/in/andrewrhysjones/

Andrew's website: https://andrew-jones.com/daily/

Andrew's book: https://data-contracts.com/

Data contract standard project Bitol: https://lfaidata.foundation/projects/bitol/

JGP's blog: https://jgp.ai/

In this episode, guest host Jean-Georges Perrin, Data Innovation Consultant at ProfitOptics (guest of episode #130 and panelist in episode #227), facilitated a discussion with Amy Raygada, Senior Data Product Manager at Swiss Marketplace Group (guest of episode #165), and Andrew Jones, Principal Engineer and Author of the book on Data Contracts (guest of episode #29). As per usual, all guests were only reflecting their own views.

The topic for this panel was all about data contracts and how do we go about getting them in place. Much of it was about the general concept but some of it was specifically about how do we think about data contracts applying to data mesh. This was the first topic I really did a deep dive into in early 2022 and it has evolved but is definitely still evolving.

Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Data contracts are about trust and understanding. Trust that there is an owner and there are rules, there is a minder that knows this data matters. Trust that things aren't going to break - at least as often as many things in data have historically and they will be told if it breaks. And understanding that what you're getting isn't perfect and there are rules but also limitations. It's no longer buyer beware, consumers can understand what they should get.
To do data products well, you almost certainly need some concept of a data contract. Otherwise, you are essentially just putting out a data asset and calling it a product. Products come with guarantees of some sort.
Data contracts are about ensuring better outputs with less effort for all parties. They are a quality assurance mechanism but also a scaling mechanism. It's a printing press for data in a sense - reusability where you don't have to carve things in wood each time, you assemble the tiles to have it say what you want but it's more about arranging tiles than defining everything - carving from scratch in this analogy. Standardized aspects of contracts help both producers and consumers communicate about the aspects of a data product.
Like with anything related to data mesh - or really any good data practices - you can roll out data contracts over time. It's not a switch you flip and suddenly everything is covered. Start small and find value. Start with one or two teams / data products, figure how this can work in your organization, and then scale from there.
While many may see data contracts as additional overhead for data producers, it's quite often a safety mechanism for them. They (hopefully) don't want to break things for downstream consumers but they often don't know exactly how their data is used. Now we have a way for them to understand the impacts of their changes and easy mechanisms to get in touch with the users of their data. Far fewer emergency response tickets to data breakages.
Data contracts are very useful - potentially necessary? - when we think about interoperability between data products in a larger context. The contract isn't only about what is in the specific data product but how it relates to the rest of your data products, mesh or not. If you have interoperability standards or linking keys, those are important aspects to mention in a contract.
To realize the vision of data mesh, we have to be technology agnostic. There will be tons of vendors releasing their own versions and visions. But at the end of the day, to actually be able to let teams have the freedom to develop their data products to best serve users, we need approaches over tools. Scott note: If you can't tell, I am skeptical of tooling in this space…

Other Important Takeaways (many touch on similar points from different aspects):

Define your contracts where it's most likely to be updated. That's probably in the code for the data product, not having to go to some separate tool.
Circling back to understanding, data contracts set expectations. Literally, they contain the expectations of what you should get with the data product. Expectations setting and boundaries are crucial to good human communication :)
As always with data work, data contracts don't come for free. They take time for producers to engage with. Reduce the friction of dealing with contracts for producers but also incentivize them to actually leverage data contracts. Otherwise, it's just a request not a requirement.
There are two different main approaches to data contracts when it comes to breaking changes - to collaborate on changes before they happen or to alert people a breaking change has occurred. It's better to be the first but you might start with only the second capability and that's okay.
?Controversial?: Using data contracts only as a blame mechanism when data breaks is missing the point. They can be a GREAT collaboration tool for negotiating between producers and consumers. They are a great starting point for those negotiations and then an agreement tracking and enforcement mechanism.
?Controversial?: As I've noted many times before, contracts can be a double-edged sword. If you have consumers that never meet with producers and share information, that can lead to someone leveraging data they don't fully understand. Contracts can give people trust in the data products they discover without digging deep enough. It's a very nuanced and hidden issue.
Like any products practice, you will probably start out pretty raw and unsophisticated when doing data contracts. It's about getting to good, not starting there. Find value, find scale, find repeatability. Iterate to good, here's your permission to suck when you start.
Data contracts can behave as great automated communication tools. Instead of trying to find all your users to update them about an upcoming change, it's automatic. Without automation, Amy said data contracts are "just a bit more paperwork."
Data contract standards are important but must be extensible. Don't expect a standard to solve all your problems or fit all your needs, especially as they are just emerging.
There are many choices you have to make for your organization around your data contract setup. Who owns the data contract is especially important. It should probably lie with the owner but if you don't have clear ownership of a data product/asset, then it's more likely a fact sheet about your data product/asset, not a contract. That said, consumer-driven testing is great in software, will we have some aspect of it in data?
Circling back to communication, to get producers to lean into contracts, look to have real conversations with producers about the challenges the organization is having with data and things breaking. Work with them to find a better solution. They are typically software engineers, they like solving problems. Give them the KPIs that let them focus on data and solving things through contracts. It can't simply be more work, it needs prioritization.
Data consumers need to be accountable to watching for changes to the data products they use. You need a good mechanism to alert them but if they aren't paying attention and something breaks, that's on them. Everyone has accountabilities.
Should you have your security and privacy encoded in the data contract itself? I think it's early days there. It might live in the contract as the place of record for the platform or not. It's an interesting concept.
?Controversial?: Should we try to create the contract automatically and have a human change and validate or the other way around? Probably human with a template works best right now. Automation would be great but there's probably too much room for error.
We want super clean implementations of something like data contracts - one standard contract across the organization. But it's just not realistic at the end of the day, especially early in your data contract journey. Every organization is messy in its own way especially with multi-cloud and many platforms, this is no different.
How data quality plays into data contracts is a bit more complicated than people think. There are quality standards but checking if a data product actually complies with its SLAs and the standards is another interesting question that people are approaching differently, whether that quality enforcement is in the contract or not.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#280 Enabling Your Domains to Create Maintainable Data Products - Interview w/ Alexandra Diem, PhD Dec 24, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Alexandra's LinkedIn: https://www.linkedin.com/in/dralexdiem/

In this episode, Scott interviewed Alexandra Diem, PhD, Head of Cloud Analytics and MLOps at Norwegian insurance company Gjensidige.

Gjensidige's approach closely aligns with data mesh but they are starting with a focus on consumer-aligned data products as they have a well-functioning data warehouse and are not looking to replace what isn't broken.

Some key takeaways/thoughts from Alexandra's point of view:

Advice to past data mesh self: stop talking to people about data mesh, talk to the changes in the way of working. It can be very tiresome to try to explain data mesh instead of those changes. Data mesh isn't the point.
There aren't really any reasons we can't apply many software engineering best practices to data, it's simply we haven't done it broadly in the data world.
There is a push and pull between software best practices and data understanding. Consider which you see as more important and when. Do you bring data understanding to software engineers or software best practices to those with data understanding.
When you leverage pair programming between enablement software engineers and data analysts that understand the domain, the software engineers learn more about data and the domain and the analysts learn good software engineering/product practices. It's a win-win.
The people you enable to do work in a data mesh way should serve as ambassadors of your ways of working, especially within the domain. Both helping others learn and as champions. That provides organizational scale. You can't individually enable every person in a large company.
"Too many cooks spoil the broth." Think about having that 'two pizza team' kind of approach so you have concentrated understanding by those involved in creating data products who then can again help others learn. This is good for those in the domain and also for an enablement team bringing learnings back to a platform team.
Having a team with intimate knowledge of what data products/data product features have been built can speed time to market for other teams and improve reuse. Each time they sit with a team, that new team has far greater access to what's been built before, whether that is existing data sources, existing models or transformations, output ports, etc.
?Controversial?: With a central enablement team, your job is more to teach the domains how to do the work, get them to minimum viable data products. Otherwise, that central enablement just isn't scalable in a large organization.
?Controversial?: A perfectly filled data catalog still won't connect all the dots for consumers. Yes, good documentation is important but there still is a significant value in helping people connect the dots. Scott note: this shouldn't be controversial but is. It's also my 'data sherpa' pattern emerging yet again as highly valuable
If you can, make sure you have a shielding and prioritization mechanism for any central team or you can head back down the overloaded central data team as a bottleneck pattern/challenge.
As anyone in the organization, your ultimate role is value generation. Consider how the data teams do that. If it's an enabling team, it's helping teams to do data work quicker and better. Those teams don't care that you're doing that via data mesh.
Relatedly, terms like data product and self-service platform resonate far more than data mesh. Lean in to what generates value, not the implementation details behind the scenes. Potentially read The Lean Startup to dig deeper into this philosophy.
Data reuse is not actually that obvious of a concept to many. Probably because it's meant so much cleaning and manual work in the past from finding poorly owned data sources or processes. Train your domain teams to look for places to reuse what has come before them.
!Controversial!: Potentially look to build out your data products from a source of already clean data. That may be an existing data warehouse or something centrally managed. Scott note: this is a data mesh end-state anti-pattern but is it an anti-pattern when in transition? If something isn't broken, do you need to 'fix' it?
Relatedly, "[I] don't really see the point of having to destroy value before I should be able to generate new value. I can very happily just generate value on top of the value that I already have."
Hypothesis testing and fast fail are great software engineering practices but in data 1) it can be hard to hypothesis test value and 2) it can be quite hard culturally to get people to learn to iterate and embrace fast fail.

Alexandra started with a little about her background coming from academia into the commercial world and how that shaped her views of things. She was the first data scientist at a company so before she could really do data science, she essentially had to work as a software engineer; that meant learning many good software engineering practices. When she moved to data, she thought 'we should use these practices here too' and then she also came across Zhamak's posts on Martin Fowler's site and it all started to click.

Specifically at Gjensidige, Alexandra was brought in to lead of team of software engineers acting as an enablement team plus a platform team. Their role was to focus on bringing these good software practices - e.g. DevOps, automation, testing, etc. - to the data/business analysts to help them build data products. It has evolved to be more sophisticated but the team is still about enabling people to build data products.

Gjensidige already had embedded analyst teams in many of their domains so when Alexandra and team started to roll out the data mesh implementation, there were already a number of data-capable folks who understood the actual business aspects of the domains. That meant there wasn't the typical pushback on the domain actually owning their data products, it was more about enabling them to do so and build a maintainable and scalable product. This process of pair programming between her team of software engineers and the domain data experts means her team becomes more and more data fluent while the domain learns how to write good software code. They specifically leverage a model of two of her team and two of the analysts in a data product creation team to provide enough information exchange but not too much overhead. That intimate understanding of what has been created also helps her team to help find reuse in other domains - they more deeply understand what has been built and can direct teams towards it quickly. That speeds time to market as well for the new team. Lots of wins all around!

When looking at the central enablement team's strategy, Alexandra strongly believes in a minimum viable data product approach. Her team only has a handful of people and they have 25 analyst teams to work with. The team has to focus on getting each analyst team to capable via the first data product - again with only two analysts on the team - and then letting those two analysts propagate the knowledge to the rest of their own teams. Otherwise, the central team would be too overloaded. So again, the focus is on teaching the analyst teams how to build good data products and then moving on. Otherwise the central team just isn't scalable or you have so many people in the central team that it becomes far harder to find patterns and share information. The domains have to deliver value themselves so teaching them to do so and then moving on is a sustainable strategy.

When communicating with the rest of the organization, Alexandra rarely uses the term data mesh. She points to data product and self-service platform as things that resonate with people and help communicate what she's actually focused on doing: generating value. Most people don't care that the way you are generating value is data mesh. It's simply a mechanism. 'Lean' into that. Scott note: lean is a bad pun here because she mentioned how helpful The Lean Startup is to focusing on value generation.

One very interesting note Alexandra talked about was training the domains in reusing data. Historically, it's been very difficult to reuse data because you didn't have the information about how it was created and didn't really have a reliable source. Getting the spreadsheet from a colleague each month isn't that reliable 😅 so, you will likely need to train your domains on reuse, especially finding sources to reuse and how to see if something fits their purpose. That can be the producing team too, teaching them how to share what they've built to other parties that might want their data.

Alexandra noted that most of the data products they are building use an existing clean and well understood data source: the cloud data warehouse. They are leveraging a hub and spoke pattern from that warehouse for their products. People already know and trust the warehouse so it made sense to them to start there. Essentially, everything ends up as consumer-aligned data products in a sense. Relatedly, for Alexandra and team, they don't see a need to adhere to every aspect of data mesh, especially at the start of their journey. She said, "[I] don't really see the point of having to destroy value before I should be able to generate new value. I can very happily just generate value on top of the value that I already have." They had some things that were working well already and breaking it all down to fit the paradigm didn't make sense to them. However, she is aware of the additional challenges this can bring and made the conscious trade-off.

Scott note: this is an obvious data mesh anti-pattern because the upstream isn't directly from source systems - the teams building the data products don't control the source or their source-aligned data products. But if you don’t have an existing bottleneck from your cloud data warehouse, why fix something that isn't broken? This may become a bigger challenge later - Zhamak has written why not owning source data creates challenges - but if they are willing to take the tradeoff and understand those tradeoffs, is it a bad approach? I don't think so _in their case_ because the data warehouse is functioning well/isn't a bottleneck.

Alexandra talked about how to really embrace a culture around 'minimum viable x' in data. In data science, at least there is a good understanding of hypothesis testing but even then, it's often hard to embrace the necessary 'fast fail' model touted by things like The Lean Startup. Trying to understand how to hypothesis test value is also difficult and people have historically seen the challenges in iterating on anything data related. So there is a learning curve but also generally a necessary cultural change to embrace hypothesis testing and fast fail around data.

On advice to her past 'data mesh self', Alexandra gave a reasonably common response, circling back to an earlier point: stop talking about data mesh, at least early in the process. Data mesh is a set of guiding principles, not the answer. Talk to people about changes to their ways of working and target outcomes. Why are we taking on change? People hear data mesh and expect it to be some technology or technological approach. You can use the name when people ask for what you're calling the approach but selling it as doing data mesh doesn't help your business partners get it. It becomes a much more tiresome approach to specifically focus on data mesh instead of the ways things change and what matters to them.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 24, 2023 Dec 24, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#279 Driving Towards a Cohesive Developer Experience - At the Expense of Snowflake and Databricks? - Zhamak's Corner 32 Dec 21, 2023

Key Points:

We need API-first technologies in data. Not just offering APIs but being able to integrate seamlessly with each other via API. We have that in software but it's been a long-time coming in data. If we want an actual modern data stack, we need to have tooling providers make a real change.
Simple made easy: we need to make things simple for data product development and consumption. It's not simplistic but it removes unnecessary complexities.
Overall, there is such a trend in data where people aren't building things that remove toil - there is this assumption of increasing complexity of use cases but so much of the work is not that complicated. We need to make it so most people can do most of the work relatively easily without making it overly simplistic - easier said than done of course.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#278 Data Contracts for the Rest of Us - Approaching Contracts in Evolving Companies - Interview w/ Ryan Collingwood Dec 17, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Ryan's LinkedIn: https://www.linkedin.com/in/ryancollingwood/

In this episode, Scott interviewed Ryan Collingwood, Head of Data and Analytics at OrotonGroup. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Ryan's point of view:

Have empathy for yourselves and others in all things you do around data. You won't always get it right the first time. Build the relationships, build the trust to continually drive and iterate towards better.
In tech, far too often we hear what people need and provide a poor solution to actually solving their needs. It's focusing on the tech instead of the people.
Far too many technical solutions/approaches - e.g. data mesh, data contracts, etc. - are really presented for tech-heavy/forward companies e.g. startups. Most companies, large or small, are not capable to leverage the approaches as presented so they must be adapted for 'the rest of us' companies. Scott note: data mesh is like this
Far too often, these tech approaches focus purely on the tech instead of the people. That's partially because every org has a different culture so you can't cover them all; but if you only follow the approach as presented instead of focus on the people/ways of working in your org, it's far less likely to go well. You've implemented a great technical solution that no wants to or can use.
?Controversial?: "What are the trade-offs that I can make, while still being true to the value and the benefits that I want to get out of this?" Scott note: SO important to consider when looking at any technical pattern/approach. What is true to the value of the approach?
Data contracts really rely on 3 things: at least two parties, an agreement of some kind that is recorded, and access to data that conforms to that agreement. You can add value building beyond those 3 but you have to start somewhere and you can deliver value with something that only satisfies those 3.
?Controversial?: It's hard not to have a sense of imposter syndrome when you actually strip a concept down and implement something that doesn't look like the public examples. That's okay and to be expected. If you're delivering value reliably, you're probably doing something right 😅
The world changes all the time. Your systems will change. Your data sources will change. Your understanding of the world will change. Your processes will change. Create/use approaches that can handle change or you're just going to create more headaches down the road.
With a centralized data team, the data team is often considered the data producer, at least to the consumer in a data contract. So with strong testing, the data team can be far more sure about meeting their contracts with consumers.
Express data quality "in a way that engages people who are not you." Make it understandable. It doesn't have to be - and shouldn't be - rocket science.
Data contracts should be the culmination of multiple conversations. Because then the producer isn't just posting data, they understand the needs of the consumer(s) and can best serve them.
It can be incredibly helpful to just go talk to your business colleagues about data in their applications or that they use in the warehouse. They can explain the context of what is going on in the real world but you can show them how that is represented in data. Both people gain a lot of understanding.
Similar to data mesh, most people in your organization won't care you are specifically doing data contracts. Talk to them about what changes and why you are doing it and how it will deliver value. Speak in the language of the business, meet them where they are.
Also similar to data mesh, you don't need to convert the whole company upfront. Find an ally to test at a small scale and prove value. Use that to learn, get a champion, and show value to gain more converts.
Some data isn't worth cleaning if it fails your contracts testing. Really consider what is of value and negotiate that with your consumers. They may want 6 years of data but really only 6 months or weeks is of considerable value.
What is the risk, what is the fear of a piece of data being wrong? Really assess that. If it's relatively close, especially for now, will that be good enough? People need to consider that data is never 100% accurate so what is good enough and how comfortable are they with uncertainty.
?Controversial?: Relatedly, signal is what matters, not (usually) exact measurement. Get people used to finding the signal but also get them used to understanding how reliable that signal is and then acting on it to an appropriate degree.
If something you are measuring, some data point, isn't going to cause action no matter the result, why measure it?

Ryan started off with some framing of how he looks at tech approaches in general but especially how he started looking at data contracts. Most paradigms are presented as if every organization is very tech-y, like a tech startup. With data contracts, much of the content "…there was this assumption that you had multiple teams of people that had a fairly high degree of technical sophistication, … or maybe even data was their primary focus." So when a less tech-y company wants to leverage the paradigm, there is always some adjustments necessary 😅 and when it comes to those types of companies, it’s so much more about the people than the way most paradigms are presented. It makes some sense because every org's ways of working and culture are different but it still can feel very removed from reality for less tech-heavy companies.

When focusing specifically on data contracts, Ryan's company is far more batch than streaming. So trying to even leverage the best advice (Scott note: I highly recommend Andrew Jones for that), he had to adjust some aspects to a world where things were a bit more messy and with teams that aren't as data mature. When approaching how to tweak data contracts to still work, he asked the rhetorical but crucial question: "What are the trade-offs that I can make, while still being true to the value and the benefits that I want to get out of this?"

Ryan moved into what he sees as the minimum viable value aspects of data contracts. You need two parties, you need an agreement of some kind that is recorded, and you need access to data that conforms to the agreement*. As to the parts of the agreement, Ryan focused on two factors at the start: semantics and data quality. If people can't understand the data can they use it? If they don't understand the quality, can they really trust it enough to rely on it? So they worked to create a data dictionary and also provide people a better understanding of the different angles on data quality.

* Scott note: this could somewhat disagree with the idea many have around data contracts of merely publishing data with SLAs because while there is a consuming party, they aren't really part of the agreement, they only choose to use the data based on the existing SLAs/contract around it. There's lots of nuance but I HIGHLY believe in the communication-heavy aspect Ryan and Andrew Jones both present.

Often, when comparing with what was presented for a tech-heavy company to what is possible at a more regular organization can be disheartening according to Ryan. The idea that the end picture at your organization should look like the one presented is pervasive. So it's not only hard to adapt the approach but then you wonder if you even captured the value 😅 Can you even call it 'data contracts' or whatever you are working on?! Imposter syndrome is very common here. Scott note: you could definitely call what Ryan and team are doing data contracts :)

Ryan also talked about how in data contracts, you must build for change. Change is the only constant after all. So creating systems that don't handle change well is a great way to manufacture more headaches down the road. Much like in software testing, you can more easily tell when something no longer works and needs to be changed. And when the data team is the actual data producer - if the data team are the ones transforming the data, that's often the case or at least is the only group of people consumers talk to with a centralized data team - they are much more sure that what they are doing is correct.

Another key learning Ryan had along the journey was that when displaying data quality, make the metrics more easy to understand to the layperson. Historically, data quality has been measured with complex statistics. Most people can't easily read the charts from that to understand what's going on. Make the data quality metrics understandable so people can see progress but also get a sense of how well they can rely on data. It is a sad truth that you can deliver value but if you can't get others to see that value, it isn't valued. Showing that value gets people to lean in.

Ryan dug a bit deeper into creating systems that act with empathy. If you approach data contracts as consumers only get what the producer shares, that doesn't end up serving the end needs that well. But if you are treating the contracts as the culmination of multiple conversations, the producer can start to really understand the impact of bad data. How much work do data consumers have to do to actually use the data? This is where empathy and product thinking come in.

"…data, as we know, it is merely a side effect of activity, of stuff happening." Ryan believes we need to move past the 1s and 0s thinking in data and focus on what it reflects and how that impacts the people in the organization. Conversations can be hard but they give you the context necessary to maximize the impact of your deep systems work. Talking with people can help both parties bridge the gap between understanding what is happening in the real world versus the data 😅

Internally in Ryan's org, they wanted to review their general processes. Part of that was the uncomfortable truth that change, especially to processes, impacts the data. So that review created a great opportunity to start to implement data contracts. It wasn't about telling people they were doing data contracts, it was about getting people bought in to what value could be delivered if they did data quality and trust better. It just happened to be via data contracts.

When actually starting out, Ryan looked for one ally that was willing to take on some of the complexity of dealing with data contracts and saw the potential benefits. Instead of trying to convert the whole organization, it was contained and let Ryan learn how to implement data contracts well in his specific organization. That initial success gave him the confidence to move further and the success story to entice additional partners/allies.

Ryan discussed the push and pull of data quality and value. While it might be valuable to have a long history of data, is the cleanup worth it? Really have conversations and make hard choices that align to return on investment instead of merely do consumers want it. Similarly, people need to confront the idea of data being right or wrong. They need to consider what is the cost of some data being wrong, especially slightly off. If that's for a regulator, potentially high. But if it's your weekly marketing leads report and it's off by 0.2%, how big of a deal is that? And how much trust is lost if it's wrong? Can we get people to understand data is never 100% clean/right? Getting people to act on signals will likely be somewhat challenging but it's a better way to navigate than trying to wait for exact measurement in many - most? - cases.

Ryan wrapped up back on dealing with yourself and others with empathy. You might not get it right at first but if there's trust, you can iterate towards better together. That goes for your data, your processes, and your relationships.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 17, 2023 Dec 17, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#277 Mesh Momentum Versus Value - What to Choose When and Why - Mesh Musings 57 Dec 14, 2023

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#276 Making Self-Service Actually Work Well Safely - Interview w/ Kate Carruthers Dec 10, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Kate's LinkedIn: https://www.linkedin.com/in/katecarruthers/

Kate's 'Data Revolution' Podcast: https://datarevolution.tech/

In this episode, Scott interviewed Kate Carruthers, Head Of Business Intelligence at the UNSW AI Institute and Chief Data & Insights Officer at UNSW (University of New South Wales). To be clear, she was only representing her own views on the episode.

UNSW is not currently implementing data mesh but are preparing to be able to do so. This is a great lesson in building up the capabilities to move forward towards your goals but not rush.

Some key takeaways/thoughts from Kate's point of view:

Universities can teach us some really interesting perspectives on self-serve. Because universities are such complex organizations and so many departments are involved in deep investigations in very specific areas, they really are the only domain experts. So enabling them to even just own their own data can be very challenging, let alone helping them share with others safely.
Relatedly, each academic researcher is essentially a micro-domain themselves with their own ways of working. That just adds to the need to enable freedom in ways of working but still "keep them safe." Scott note: safety was a key theme of the conversation
"At the end of the day, data mesh is about controlling the bits that you need to control, and giving people the freedom to do what they need to do, safely."
"Technology is kind of the least of your problems." When it comes to data, be prepared to start with some people not even recognizing there is a problem with the current ways of working or a need to improve. Connect their pain to data immaturity to win them over.
The best way to win people over is show, don't tell. Show them the power of self-service instead of pitch them on it. Get a PoC going and get people to tangibly see - and hopefully soon touch - your self-service capabilities early.
Always look to anchor your data work - especially things like platform work - to a business need. How will doing the work impact the business? Why is it important to do and to do now?
When tying your data work to the overall business strategy for your organization, do NOT forget the people aspect. The relationships matter. Your work on the data team definitely isn't only about technical execution.
?Controversial?: Build a culture around data that is as focused on building human relationships as it is on building data pipelines and platforms.
?Controversial?: To share personal/sensitive information - e.g. PII - a producer should justify why it's appropriate and a data controller should review that. Keep humans in the loop.
Giving data owners (UNSW calls them data controllers) a say in how their data is actually used can get them more excited to share their data. It isn't a silver bullet to data sharing incentivization but it adds value to them.
Good conversations about access to sensitive data shouldn't be yes/no. They are about getting to what is acceptable and maximizing value within that framing. Get people to share what they are trying to accomplish and partner to best achieve it!
Invest in business analysts. They are your front-line to figuring out how to proceed around data and generate value. You need people who can speak business and data simultaneously to drive to great outcomes.
Find ways to prevent data puddles, especially places where people are copying data and not securing it well.
"People overestimate the power of making change really fast, and underestimate the power of … sustained incremental change."
Give people a mental map for change. It removes the fear of the change and lets them lean in. You are creating change with and through them instead of pushing change on them.
?Controversial?: ChatGPT and other GenAI can actually be a great benefit to education. We have to lean in to it as it's not as though students won't have access to these tools in their work life. So getting them to still learn but leveraging better tools is essential to their progress.

Kate started out with a bit about the catalyst for her current data journey towards data mesh. About 10 years ago, she saw that universities and especially UNSW were going to "undergo a very big digital transformation and that data would underpin it as an organization. [So] we would need to be on top of our data if we were going to be able to ride that wave." She also gave some color on what running a data office at a university entails. At UNSW, it's split into three general areas of administration, learning + teaching, and pure research.

There are some major challenges when it comes to providing data capabilities - especially self-service - to the academic research arm of a university according to Kate. They all have their own ways of working and want - demand? - freedom to work the way they want. Yet, the data team's job is also to "keep them safe." That safety has many facets as well. And the research capabilities of a university can mean some truly world-changing interdisciplinary collaboration. But that only happens if the teams can actually, you know, collaborate 😅

When it comes to the non-research area, Kate believes data mesh is an even better fit. "At the end of the day, data mesh is about controlling the bits that you need to control, and giving people the freedom to do what they need to do, safely."

As many guests have noted, Kate believes when it comes to your organization's data journey, "technology is kind of the least of your problems." It’s about people and often even getting them to recognize the problem with their ways of working and how better data maturity will help alleviate their problems. It's not just the data itself but their understanding and relationship to data.

Kate and team built a quick cloud warehouse PoC that showed people the ability to onboard new data sources in weeks instead of taking up to six months. Showing them instead of simply telling them really won people over. People could connect moving to a cloud data warehouse to business benefits. They also anchored it all to business needs. Yes, rebuilding their architecture to move to the cloud was going to be work but it meant speed to new data use cases and easier management.

When Kate was working to tie her team's work to the overall business strategy, she remained focused on the human relationships and people aspects of doing business. She really recommends building relationships with "customers" of your data work because then they feel comfortable to come to you with more types of problems and challenges. And sometimes that kind of culture/approach isn't for everyone and that's okay. If people aren't willing to treat customers as people, they aren't right for her team.

When asked about her frequent use of the word "safe", Kate talked about keeping people from misusing data or even misusing the trust people who provided that data - e.g. the students at UNSW - gave the organization. Anytime someone wants to share sensitive information like PII, there is a data controller that needs to review the justification. Keeping that human in the loop means there is a real understanding and consideration of 'is this okay?' On the flip side, the team has been proactive in sharing information that someone should have access to, e.g. a professor being able to know who is in their class and being able to contact them.

Kate mentioned that when they implemented the data controller review, the data producers were much happier. Previously, they had no real say in how their data was used but now, they are listened to. It also strengthened relationships because consumers had to actually collaborate with producers to get access to their data. It's creating interesting conversations and people can get more creative around data to achieve their goals with more data safety. And her investment in hiring a bunch of business analysts has created some great value leverage points.

Going back to keeping people and data safe, Kate talked about their struggles with data puddles - where people are copying data into lots of areas instead of accessing the data where it is. And they aren't securing that data well, which leads to more challenges and potential issues. But it's still a process to give people all the access they need and make that copying data less attractive. As like many areas, it's a work in progress 😅

Kate sees the attractiveness of moving fast but believes people need to focus more on sustained incremental change, that they overestimate the value of the former and underestimate the value of the latter. It's similar to transformation versus a change that will revert. Fast changes are far less likely to stick or even work. And people feel less of the suddenness and fight against it far less if at all when it is gradual incremental progress.

Another point Kate emphasized was that people need a mental map for change. If they don't understand what is changing and why, they will inherently fight back, even if the change is good for them. It's simply human nature to not want change. So take away the fear of change to make it easier for people. Basically create change with and through people instead of pushing change on them whether they want it or not 😅

The conversation wrapped up around GenAI, especially because Kate is involved in the UNSW AI Institute. She is seeing the open source large-language models (LLMs) improving at a rapid pace, sometimes multiple times a day. And there is a lot of promise even if things are early days. At UNSW, they are figuring out good ways to leverage GenAI in education instead of trying to fight against it like some math teachers did against calculators. It's here to stay so they have to adapt and adopt.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 10, 2023 Dec 10, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#275 Panel: Why Data Mesh Needs Digital and Org Transformation - Led by Benny Benford w/ Nailya Sabirzyanova, Iulia Varvara, and Stefan Zima Dec 07, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Benny's LinkedIn: https://www.linkedin.com/in/bennybenford/

Iulia's LinkedIn: https://www.linkedin.com/in/iuliavarvara/

Nailya's LinkedIn: https://www.linkedin.com/in/nailya-sabirzyanova-5b724310b/

Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/

In this episode, guest host Benny Benford, Founder and CEO at Datent - a data transformation focused consultancy/community - and guest of episode #244 facilitated a discussion with Iulia Varvara, Advisory Consultant in Digital and Organizational Transformation at Thoughtworks (guest of episode #268), Nailya Sabirzyanova, Digitalization Manager at DHL (guest of a soon-to-be-released episode), and Stefan Zima, Data Transformation Lead at Raiffeisen Bank International AG (guest of episode #270). As per usual, all guests were only reflecting their own views.

The topic for this panel was transformation when it comes to data and data mesh in general but especially understanding how organizational transformation must play a large part in a data mesh implementation to be successful. And that transformation is not simply making changes, it is making _lasting_ changes. Organizational transformation is a crucial aspect of doing data mesh even if it's not spoken about all that often.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Transformation means changing something. We aren't starting from scratch. You have to consider the starting points, not only the target end points - and in data mesh, there isn't really an end. Every organization's transformation starting point, whether a data mesh transformation or otherwise, will be unique so adjust your transformation journey plan accordingly.
There are so many reasons transformation initiatives, especially data transformations, can fail but a big one is not preparing for the long-term change necessary to make changes actually stick. It's easy to try to make changes but actually making them to last for the long run is something else entirely.
There needs to be a sense of urgency to drive forward a large-scale top down-driven organizational transformation. If there isn't a real business reason and one where there is a need - or at least a strong desire - to be addressed in the near-term, you are far more likely to lose momentum/sponsorship. And you need lots of momentum and sponsorship for large-scale sustainable transformation.
If you are trying to pitch something like data mesh, speak to real pain points. Just selling the potential benefits instead of solving real, painful existing challenges is not likely to win you as many converts. There's a reason painkillers are easier to sell than vitamins.
Transformation and product thinking have a lot in common. Org transformation is treating the organization as something like a product to improve over time. That means prioritization. You can't take everything on at once. Work with your stakeholders to make progress on what matters most.
Driving data transformation - data mesh or otherwise - will likely take a lot of education. There is a general sense people should be using data for additional use cases but really, many aren't thinking of the great ways they could use data. Help them find the link between their business priorities/pains and data.
Your business partners don't need to know the particulars of your transformation initiative. Sell them a story, give them an enticing vision. Why is this worth doing and what is the payoff? Stop taking them on the sausage factory tour against their will, give them - or promise them - a wonderful sausage tasting party instead.
Organizational transformation - data or otherwise - only happens when things change, when they transform. Sounds obvious but you really have to get your business partners to engage or your transformation won't be as successful and is likely to stall/fail. Trying to change the entire organization from just the data team is daunting at best.

Other Important Takeaways (many touch on similar points from different aspects):

Transformation might not be the best word since transformation implies an end state, an end to transforming. And while you should have some kind of target future state in mind, that's likely to be a target state along the journey rather than an end state. The only constant is change as Iulia said.
If you try to break your data mesh transformation down into very separate component parts and transform them separately, it's going to make it more difficult. You need to transform across technology, mindset, understanding, ownership, etc. simultaneously - pushing in the same cohesive direction. The organization has to be ready and capable for a change.
Relatedly, you can't start with some massive change or only look to deliver value starting in year 3. Find ways to make progress and deliver value - especially provable/marketable value - along the way. Incremental value delivery is the key to maintaining exec attention and sponsorship, which are crucial to maintaining momentum.
Also relatedly, this doesn't mean your progress on different aspects will all be at the same pace. Maturity, buy-in, capacity, etc. will determine how far you can transform different aspects and when. Don't try to wait for everything to move together.
A data mesh transformation driven from the bottom up and mostly by the data team is possible but will likely be harder - far harder? - than top-down. You need to constantly win more support but that can also have its advantages than something with fanfare but not a lot of specific direction.
Prioritization is key to doing org transformation well; so is measuring progress. If you aren't addressing the real pain points - or at least the pain points your exec sponsors care about 😅 - you will likely lose that sponsorship. Show them you are making progress against those pain points.
Get your business partners to tell you about their actual pain points. Not just about data but about areas where data may be able to improve their work. They will often literally tell you how to sell doing something like data to them by making them feel seen and heard, actually creating a plan to address their specific pain.
Relatedly, work with stakeholders to define success metrics around your progress. If you can continually show them incremental value and that you are addressing their needs, you are far more likely to be successful with your transformation initiative. But getting to clear metrics around the data work will be _hard_. Scott note: I'm writing a book on this for a reason 😅
?Controversial?: If you want to 'prove' value from data work, create a way for other teams to measure the value created. A data team claiming they created value versus Finance claiming the data team created value is a world of difference when it comes to credibility.
When it comes to prioritization of data transformation, should the data team really be setting the priorities? For certain aspects like the platform, probably. But really, the business should tell you what are the highest priorities where you should focus your work.
?Controversial?: The head of the data org should be there to enable other parts of the business to derive value from data. It's about making everyone else better.
Because of the central nature of many - most? - data teams in large organizations, too many people are used to them essentially being free - incremental data work doesn't typically cost the line of business or at least doesn't cost much. Transforming that mindset to get them to focus on extracting value from data work might be challenging.
As constantly comes up in almost every data mesh conversation, incentivizing data producers is hard. You should try to create structural incentivization at the organizational level to push some of the value created back to the data producers.
Data mesh isn't the point. It should never be the point. Zhamak has said this as well. We are looking for ways to achieve our goals and data mesh (hopefully) provides good framing to do that.
At the end of the day, data work should be about impact. Focus on impact with your business partners and they will be far more likely to continue to engage.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#274 Your Data Platform is a Product, Treat it Like One! - Interview w/ Sean Gustafson Dec 03, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Sean's LinkedIn: https://www.linkedin.com/in/seangustafson/

In this episode, Scott interviewed Sean Gustafson, Director of the Data Platform at Delivery Hero.

Delivery Hero has been on the data mesh journey for longer than most organizations, at least over 3 years.

Some key takeaways/thoughts from Sean's point of view:

It's extremely hard but still important to try to impact your culture through things like your data platform. Who are you trying to make information available to? How do you make it accessible? How do you make data ownership easier?
A key role of the data platform is that golden/easy path. Showing people easy ways to accomplish what they need with data products. Embed best practices into the platform when possible.
You need a product manager in your data platform team. It's easy-ish to build cool things in data but understanding and building to user needs is harder and a must. Treat your data platform as a product!
Relatedly, there isn't anything all that special about product management around the data platform. You can take what we've learned from other disciplines - especially software - and tweak it a bit for data. But it's not some arcane art.
Focus on KPIs around what you are building and why, especially for your data platform. It's very hard to measure developer productivity but that doesn't mean you just don't measure it.
?Controversial?: Be prepared to deal with a lot of qualitative data when measuring success around your data platform. Surveys work far better than most might think.
Good product managers balance the short and long-term. You don't want to make drastic and breaking changes to your data platform often but that doesn't mean you can't take bigger bets and shake things up. Just balance iterative improvements and the bigger picture. Scott note: Zhamak talks about Thomas Kuhn and cumulative progress versus paradigm shifts
In the same vein, make small bets where small bets will do but don't be afraid to make big bets when necessary.
?Controversial?: It will be hard to iteratively change a traditional centralized-focused data platform to do data mesh/decentralized ownership well. You want to at least consider a fresh start when looking at your mesh platform.
Tools like dbt have given a much broader group of people the ability to model their data. There are inherent problems if they don't do it well but we still need to encourage more people to do data work so we can get them better and producing great work.
Data products are a lot like APIs. There are many best practices we can take from APIs and apply them to data products.
Explaining data mesh to software engineers can be tough. They probably get the concepts given most are just software engineering concepts reconceptualized for data. But the biggest challenge is they will probably see data as a second class citizen to the underlying back-end systems. Scott note: Unfortunate but extremely common
In data incident management, e.g. data loss, you have to look at the prioritization but our general historical focus - how much money did we lose - just doesn't make sense in data. We have to take reliability engineering practices from software and tweak them to work with data but we can take incident management essentially as is. We just have to understand prioritization far better.

Sean started with a bit about how he sees his role as leading the data platform team. It is very challenging but still important in his view to try to shape culture even through the data platform. There are so many places in data mesh where there is friction, how do you make things easier as everyone transitions to product thinking and decentralized ownership? Just because you have mandates from the top, people need new ways to accomplish new goals. Make your platform reflect the type of data culture you want. Instill in people the understanding that they can and should participate in your data culture/work. Easier said than done of course.

Relatedly, Sean believes the data platform should show people the right way to do things, give them that easy path where possible. But still give them the freedom to do some aspects … not so right 😅

Treat your data platform as a product is something Sean strongly believes in. And to do that, you need someone acting as a product manager. It's not rocket science, we know how product management works and it's not very different when it comes to building a data platform. But you need someone specifically focusing on user needs. And part of that role is also to advocate new features and using the platform. Just because you built it, that doesn't mean people will use it.

When asked about iterating to good, Sean talked about how in product management, good practice is about making constant and small improvements but also balancing the bigger picture/big bets. It's not always about the big new platform but sometimes, it's okay to shake things up - make small bets when small bets are good enough but make big bets when necessary. But you have to do that by balancing the short-term and long-term picture. Fail fast and iterative improvements are crucial to good product thinking in software and we need to apply that to data. But again, big changes are okay if you properly build to them instead of trying to flip a switch. He specifically mentioned that it will be hard to iterate to a platform that does decentralized ownership well from one that was highly centralized. Not impossible but at least consider building that out more from scratch.

Sean talked about Generative AI and how it's starting to change lots of people's views internally about data. While previously, many software teams were at best reluctant/hesitant to model their data, there is a big interest from the software engineers to directly interact with the large language models (LLMs). Tools like dbt previously brought many new people to the data party, making it easy to model data - at least structurally - so hopefully GenAI will mean more people learning to model their data. There are inherent challenges but the more the merrier when it comes to people working to produce good data. We just have to make sure they learn how to do it well 😅many who are new to data modeling do it… not so well…

When it comes to product management, you need to measure how well you're doing. For Sean, that of course extends to the data platform. While KPIs can be somewhat hard around your data platform, that doesn't mean you get to slack off and not measure things. At Delivery Hero, right now they are using surveys to measure a number of things around their data platform rather than trying to measure things automatically without context. It also creates a lot of conversations in the data platform team about what are you trying to do and why, which prevents a lot of waste. It's not perfect but it's getting better. Scott Note: this is why I am writing a book on success factors then one on success metrics in data mesh 😅 this is HARD

Sean talked a bit about APIs and how much data products _should_ be treated like APIs. Not just versioning but tracking usage and having users register to use them. There's a lot to learn from how APIs evolved so we don't have to make the same mistakes in data. Scott note: Zhamak comments on this VERY frequently that API approaches are crucial to data mesh

When talking to software engineering people, Sean has found using data terminology, especially data mesh terminology, doesn't really resonate with them. We probably need to come up with new terms - or potentially use the terms Zhamak took from software and just make them about data too instead of inventing new terms. But be prepared for it all to fall back to that most software people will see the back-end systems as more important than the data. If you get them over that hump, it's far easier to get them bought in on data mesh. You may be able to win them over by showing them how the data is used internally.

Incident management in data is still pretty nascent in Sean's view. While on the software engineering side, there are very well established processes, often in data it has been more slapdash at best. No escalation, no prioritization, no formal process, no post mortem + shared learning, etc. The traditional measure around data issues - how much money did we lose - often isn't applicable to data. So we have to rethink what matters and why because our prioritization is often skewed.

Sean wrapped back to the start about how important culture is. Not just getting your organization to be data driven but setting up more and more people for success in your organization through their work with data.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 3, 2023 Dec 03, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#273 An API-First World in Data Integration - An Actual Modern Data Stack - Zhamak's Corner 31 Nov 30, 2023

Key Points:

The rush to categorize all of our tooling in data has caused many issues - we will see a big shake-up coming in the future much like happened in application development tooling.
So much of data people's time is spent on things that don't add value themselves, it's work that should be automated. We need to fix that so the data work is about delivering value.
We can learn a lot from virtualization but data virtualization is not where things should go in general.
Containerization is merely an implementation detail. Much like software developers don't really care much about process containers, the same will happen in data product containers - it's all about the experience and containers significantly improve the experience.
The pendulum swung towards decoupled data tech instead of monolithic offerings with 'The Modern Data Stack' but most of the technologies were not that easy to stitch together. Going forward, we want to keep the decoupled strategy but we need a better way to integrate - APIs is how it worked in software, why not in data?

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#272 Understanding and Valuing Your Organization's Data - Interview w/ Lauren Cascio and Chris Ensey Nov 26, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Contact email: Swimwith[at]gulpdata.com

Lauren's LinkedIn: https://www.linkedin.com/in/laurencascio/

Chris' LinkedIn: https://www.linkedin.com/in/censey/

In this episode, Scott interviewed Lauren Cascio, Chief Fish Wrangler, and Chris Ensey, CTO at Gulp Data.

From here forward in this write-up, L&C will refer to the combination of Lauren and Chris rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from L&C's point of view:

?Controversial?: Many organizations have an incorrect perspective that they mostly have a single type of data that's useful for each use case or need. Typically, their data is useful for many more internal use cases and also to organizations in far different industries.
Often, there is a lack of a data sharing culture in many organizations. There isn't anyone that really understands how data flows throughout the organization or especially how it _could_ flow to serve many untapped use cases.
There are many people emotionally attached to owning their own data but not in the product sense, they are focused on maintaining control rather than structuring it to be shared. So there are organizational challenges to data sharing in addition to technology.
Many organizations have a tough time justifying updating their data infrastructure, leading to more and more challenges with progressing their data journey. It's often hard to point to a tangible ROI on updating the data platform for instance.
Far too often, companies and LOBs know they want to analyze some information but they don't really know what they are analyzing it for. Instead of shaping data to make specific decisions, there is a focus on the visualization without a clear action in mind once the data tells them something. Drive towards what you care about and use data to answer those questions, the data doesn't speak for itself.
Your upper management has limited patience and a limited attention span. Focus on what matters to them and be crisp on delivering an outcome with data, not outputs. A dashboard is just a pretty picture unless it drives action/creates insights.
?Controversial?: It's often easier to get funding to prepare your data for external sale than investing in internal use cases. The simple reason is a tangible ROI. Look to frame your internal investments in data in the same way.
Find ways to open more communication about what data you have internally. You will be surprised by the number of new use cases emerging. There's so much untapped data internally that people would use if they only knew about it and could easily use it.
Many organizations aren't really thinking of the value of their data and how they protect it. If the data is so valuable to your organization, what kind of investment are you making in security and compliance to protect it?
!Controversial!: People's personal data getting shared, at least with some modicum of regulatory oversight, is for the greater good - e.g. more patient data to help fight disease or more financial information to help unbanked people get access to credit/capital.
If an organization wants to understand their overall data landscape, the best way to start is simply by starting and also having an end purpose in mind. Essentially, get conversations going and know why you are trying to understand your data. Is it to unlock new use cases, save costs, sell your data, etc.?

L&C started with discussing how many organizations view their internal data landscape/estate and how it's not a complete picture. There tends to be a perspective that an organization's data is only useful for their internal use cases and often that each set of data is only useful for one type of use case. And L&C just haven't seen that be true - internally, most orgs have data that could be useful to existing use cases . How that typically manifests is data silos where data that should be shared isn't because people aren't aware it exists. Or the other side is that data producers have no real idea of how their data is being used downstream by other parts of the company. Externally, most companies' data is often very useful to other organizations in entirely different sectors.

When asked about why lines of business have such a hard time understanding what data other LOBs have, L&C talked a bit about the technical challenges but much more about the organizational. In many - most? - organizations, lines of business have treated their internal data as overly precious, making sure it was structured specifically so they could use it. Trying to get them to structure it so others can use it is an emotional hurdle because it can feel like giving up that control. Thus, it's been hard for other LOBs to even know about the data across the organization, let alone use it. Add to that the challenge of businesses treating the data team and especially their infrastructure as a cost center and that further impedes their data journey. When it's hard to make the tangible business case for updating your data infrastructure, it's easy to fall further behind. If much of this sounds familiar, it's frequent hurdles towards implementing data mesh.

L&C talked about how typical it is where organizations understand they want to visualize their data but without a specific goal in mind. Just visualizing without an expectation of what it will be used for is not product thinking. What information do you need to make your external facing products better? What will cause you to act? Instead, it's about "what does the data tell us" which is not often aligned with taking actual action. That leads to wasted cycles and money; it also often leads to wasted buy-in from upper management - they really only have limited patience, spend it on what matters. Be crisp on what goals you are going after then develop the data and analysis to help you actually go after those goals.

It's far easier to get exec buy-in on selling your data externally than investing further for internal use in L&C's experience. That's because there is a tangible outcome at the end of the road. Look to try to shape your asks for additional funding based on that principle: a tangible ROI makes decisioning easier.

For L&C, there are many use cases that could be unlocked in most organizations if only people knew what data was available. Finding ways to discover and share more about what data you have internally is very helpful. Yes, a data catalog is great but finding better ways to make people aware of the available data will unlock new valuable use cases. An audit for what data to sell externally is one way to spark these conversations but there are many others :)

L&C pointed to two differing types of companies regarding selling their data. The first is low margin businesses. Because they are so reliant on volume, they end up with a considerable amount of data that they could potentially monetize. The other type of company is early stage companies that have yet to reach product market fit, especially B2B. They often think their data will be very valuable but selling data becomes a distraction far too easily. Focus on your core business, not small external monetization streams.

On the somewhat controversial topic of data monetization, how people's information is protected versus leveraged, L&C believe there is a greater good in general to your information being shared. While something like GDPR gives the perception of your data being protected, it's not really all that true - everyone's data is out there already 😅. Meanwhile, there is lots of potential good that can come out of more comprehensive data sharing, e.g. better information to fight diseases from more patient information or lower cost of items in retail stores from data generated in loyalty programs.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 26, 2023 Nov 26, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#271 The Importance of Repeatability of Language to Scalability - Mesh Musings 56 Nov 23, 2023

Important points:

There are places where nuance adds value. Many times, explicit definitions around data aspects like quality or even SRE metrics like uptime and query performance are not one.
Provide a simple way for producers to apply these scalable approaches - the platform should measure data quality metrics for example.
Data producers are having a hard enough time in general learning how to leverage data better. Find places to make it about learning about the information encapsulated in the data product, not learning a new set of SLAs for each data product.
Consumers will thank you too since it make their lives easier. With that, you should see more of an uptick in data usage.

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#270 Sustainable Data Transformation to Drive Towards Data Mesh - RBI's Journey So Far - Interview w/ Stefan Zima Nov 19, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/

In this episode, Scott interviewed Stefan Zima, Data Transformation Lead at RBI (Raiffeisen Bank International AG). To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Stefan's point of view:

No one has data mesh all figured out. Go talk to each other. But also don't be ashamed that you are running into challenges. So is everyone else. Data mesh implementers also need to share more of the anti-patterns they are finding.
Agile transformation really focuses a lot on communication and transparency. Both are very crucial to really any successful transformation initiative. Humans struggle with uncertainty and change so giving them a lot of information especially about the why prevents unnecessary pushback.
Relatedly, there are many things we can take from Agile transformation practices to apply to data/data mesh transformation. It's not a copy/paste but there's still much that is very relevant with some tweaks.
Many organizations are still focusing on technology-led transformation, whether data or digital in general. You must also change the mindset and organizational approaches if you want to be successful.
In banking, the rise of fintechs (financial technology companies) has made it clear that being nimble and quickly acting on data is crucial. Being data driven is required to remain competitive.
Data mesh can mean far less friction in getting to serving use cases. Instead of fighting against the data protection office, they are involved from the start. That time to market is especially crucial in banking now.
If you can, look to make your data sharing policies and approaches generic enough to only create friction when there truly is something different that should be examined further.
If you really want to be 'data-driven', if you really want to be a data company, you have to find and address the friction points in your data processes. Stop trying to simply get better at processes that stopped scaling and have high friction, find new ways.
Don't try to sell everyone on flying to the moon when you're struggling to get the garbage taken out every night. Get grounded in people's day-to-day challenges and sell them on getting better and better on delivering on them so they actually can start to get buy-in to the bigger picture.
Relatedly, talk to the vision and the goal but show the pathway to getting there. You can't just drop product thinking on the organization and expect overnight uptake.
Communication is incredibly important to transformation. You need top-down support, not just behind the scenes but in day-to-day internal communications. People need to understand transformation is a priority. Leverage your communication department.
To maintain mesh momentum, you really do have to prove out value and then get further buy-in from upper management - you don't just get to build without proving value. That proof of value will add more positive pressure on other domains to get on board.
To get people bought in, focus on incentives. Words and asking are all well and good but really, incentivization is the key to getting participation and buy-in.
In data mesh, it's easy to get too focused on either the small scale - only focusing on use cases - or the overall implementation. As a leader, you need to split your focus between the execution and the big picture. At some point, mesh needs to scale to make it worthwhile so that big picture matters even early.
?Controversial?: When doing an Agile transformation, the role of Agile coaches was quite helpful. Is there something similar that could work for data mesh?
Take the data product concept and find a clear definition for your own organization. And it's okay to have something that extends the concept. For example, at RBI, a dashboard is a business product, not a data product.
When it comes to risk mitigation, the Data Protection/Privacy Officer can just say no to every request and there is no risk taken, so job done, right? As data people, our job is to clearly explain the risks and how we are mitigating them to make it easy for the DPO to say yes. Talk to them early to design compliant solutions.
?Controversial?: Self-service is more than just providing access to the data and it's certainly more than just tooling/the platform. You have to teach people how to find, access, consume, and especially understand the data. And then teach them to also share what they've learned via sharing data in a safe and scalable way.
While incentivization is crucial, it's also often hard. E.g. what is the incentive for someone to put really complete metadata around a data product into the catalog. Can't any users just come to the owners with questions?
When you start your data mesh journey, make sure to start with FinOps. So much of data work is opaque, get your arms around costs and also cost savings. This will prevent challenges down the road 😅
Simply put, transformation takes time and energy/effort. If you try to do it as a big bang, all you do is blow things up.

Stefan started with a bit about his background and current focus. From having worked in data and transformation, the current transformation needs - data and digital in general - for many organizations are accelerating. There is a need to move towards more modern platforms and a self-service orientation of course but the organizational change especially around mindset cannot be overlooked.

As for why the banking industry is embracing data mesh and being data driven, Stefan mentioned that over the last five years or so, there have been many major industry changes. One aspect is the rise of fintechs that are built from day one with data in mind - to stay competitive, more traditional banks need to be able to move extremely quickly and the best - only? - way to do that intelligently is with data. Another aspect is simply the domains are getting more and more capable around leveraging information to gain competitive advantages so they need to feed that with quality data to win in the market.

When the bank started to see more and more friction - e.g. more pushback from the data protection officers to comply with ever advancing and increasing regulation - Stefan and team realized they needed a new approach. Instead of trying to make improvements to the existing processes, there needed to be new ways to get things done instead of relying on existing bottleneck creating interdependencies. That's part of what led them to data mesh - taking points of friction and shifting them left and reducing handovers meant faster time to market with new information and use cases.

Stefan talked about how many people in transformation and data push too far, too fast with their vision. When people are stuck in their day-to-day, getting them to imagine the company that could be in 5 years is at best inspiring but often not and it certainly doesn't help them today. Help them connect that vision of that 'data-driven future' to what you will do for them now. Don't expect people's mindsets to shift overnight.

Relative to transformation, Stefan discussed how important internal communications is. There is of course the messaging to drive understanding but also the communication by the leaders to show support. If you lack visibility of your top-down support, it's much harder to get people to take your efforts - data mesh or otherwise - seriously.

At RBI, Stefan noted that they are through their data mesh PoC phase. They were able to prove out significant value to upper management and get upper management to further buy-in. With that buy-in, there was more communication internally, getting more and more people aligned that data mesh was the way forward. But of course, data mesh is in some ways just a label and approach, it's not the point.

Stefan talked about what can we take from Agile transformation and successful business transformation in general to use for data/data mesh transformation. A big focus in Agile is on communication and transparency. When people feel informed and heard, they are far more likely to buy-in. Humans struggle with uncertainty. When it comes to data mesh, there will be all kinds of new roles and responsibilities so you need strong communication to keep people informed and not feeling lost. Even if that is about trying something and seeing if it works. That honesty and transparency will have far more people leaning in than trying to issue proclamations. And of course, be ready for some politically driven issues because we are changing the way people operate.

The ever present 'what is a data product?' conversation also happened at RBI according to Stefan. And it was important to define it in their own world - every organization has their own needs, understandings, and requirements/restrictions that will mean data products look slightly different. Specifically for RBI, a dashboard isn't a data product but is a business product, meaning there is still a clear ownership model. They also focused a lot on automation and risk assessment/mitigation. In a heavily regulated industry, risk is always a crucial factor. When it comes to working with the Data Protection Officers, it's important to make it safe and worthwhile for them to say yes when saying no is easy and prevents all risk.

Stefan then went into a bit about where RBI is headed around self-service and why that's so crucial to the company's data mesh ambitions. For him, self-service is far more than just giving people access to data. Yes, you need a platform but you need upskilling and data literacy, an understanding of compliance, an understanding of your overall data ecosystem, and an understanding of the tooling and processes. You also need to build a platform that can be leveraged by non-experts. You need to make it easy for the general populace of your organization to actually consume, understand, and produce data.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 19, 2023 Nov 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#269 Panel: Leading a Data Mesh Implementation (2nd Iteration) - Led by Vanessa Eriksson w/ Stefan Zima, Duncan Cooper, and Sid Shah Nov 16, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn.

Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here.

Vanessa's LinkedIn: https://www.linkedin.com/in/vanessaeriksson/

Sid's LinkedIn: https://www.linkedin.com/in/siddharthin/

Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/

Duncan's LinkedIn: https://www.linkedin.com/in/duncan-cooper-1113722/

In this episode, guest host Vanessa Eriksson, the first CDO in Sweden and the head of data advisory company Vanessa Eriksson AB facilitated a discussion with Duncan Cooper, Chief Data Officer for Northern Trust Asset Servicing, Sid Shah, Head of Data Monetization and Platform at Airtel (guest of episode #258), and Stefan Zima, Data Transformation Lead at Raiffeisen Bank International AG (guest of episode #270). As per usual, all guests were only reflecting their own views.

The topic for this panel was about the leader's role in a data mesh implementation and what these four panelists have learned in that role. This was the second iteration of a panel we will likely have about every six months or so - the first was episode #215 from April of 2023.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Regarding data mesh: get going but don't rush. Essentially, get started now but don't be in a hurry to try to get to some picture perfect end state. You need to take your time to make sure you are transforming instead of making changes that will unravel. Be brave and move forward into some uncertainty!
Relatedly, you will absolutely get many things "wrong" but wrong in a data mesh world can simply mean not right yet. We have ways to adapt/adjust and evolve as we learn and grow. Data mesh provides you the ability to iterate towards better constantly.
You _really_ should look to take inspiration and learnings from other data mesh implementations but also understand that your journey will look _considerably_ different, especially your starting point and initial focus areas. If you aren't prepared to adjust for your own situation, you aren't ready to do data mesh. Sid especially has focused on learning from other implementers and leveraging them to create credibility internally.
Stop treating data problems as technical problems. At least start to treat them as business problems first but really, they are sociotechnical problems - social/business challenges married to technical challenges wrapped in trying to share understanding. It's people/process/technology and people most importantly.
Relatedly, as Stefan said, data mesh is much more about mindset shifts than it is about technology shifts. You need to show people new ways of working and guide them to value. If you don't, you very likely won't change how people work with data and won't deliver the promised value of data mesh. And then you will lose your sponsorship.
While data mesh can be a powerful paradigm, one of the best things about it is giving us common framing and ways of speaking about what you are trying to do and accomplish. Scott note: this is partly why I created the data mesh community, this podcast, and my company - to make it easier to share information with each other about how to do data mesh, both with other implementers and within organizations.
There will be pushback from many parties to many aspects of data mesh. Be prepared for that and help people understand the reasoning behind the changes - what do these changes enable you to do? E.g. you aren't pushing data ownership onto the domains simply because as a central data team you're lazy - it's that they are the ones with the knowledge and skill to actually own their data well.
There are many big challenges in data mesh but the biggest is probably getting and then _retaining_ buy-in across many different groups. If you don't have leadership buy-in, what happens when you hit challenges? If teams won't take on ownership, can you move forward?

Other Important Takeaways (many touch on similar points from different aspects):

Duncan mentioned that in Agile, there is a tendency to focus on a specific problem. But the biggest value of data comes from reuse, not just that focus on one specific problem/use case. So be prepared to adapt methodologies like Agile to data where circumstances are far different.
If you aren't able to deliver incremental value - and prove/communicate that incremental value - you are highly likely to lose your credibility and/or sponsorship of your data mesh journey.
As is often mentioned, your business colleagues probably - almost certainly? - don't care about data mesh. They care about getting good value from data. Data mesh can help us accomplish that but it's not the point. Talk to them about what you are trying to achieve and why, not about the inner workings of data mesh. What changes for them and why?
Everyone already has ways of working with data. Instead of trying to change the way they work, look to intersect and gently nudge them in a better direction. The key to that is helping them understand what is changing and why.
If you treat Zhamak's writings - or any other content around data mesh - as gospel, you are headed for failure. Zhamak has said this herself. We are figuring out how to do data mesh and you need to adjust for your own organization to do data mesh well.
Be prepared for your organization to be even more messy/varied than you likely expect once you start to dig deeper into your ways of working in general and how those could be better improved by data. Your organization likely grew organically - different parts of a large forest behave quite differently even if it's one forest.
?Controversial Scott Note?: Someone made the point that to get buy-in, you can show people why it isn't sustainable for the organization to continue doing data the way we've been doing it. That generally isn't going to sway most people. They understand that but it's like saying everyone needs to do something like make their own clothes to fight climate change - you won't convince many…
Your organization's data culture needs to work like a moderately well-functioning society: you have rules and regulations that people need to abide by or face the consequences. Think about the phrase from "The Good Place": 'What do we owe each other?' Democratization/federation doesn't mean anarchy.
Your actual data mesh journey - maybe not the story you or others present at conferences - will be extremely messy. Organizations, people, life, etc. are all messy. Don't be worried about getting it perfect but worry about driving towards value.
?Controversial?: You need a certain level of maturity to be able to tackle data mesh but you also, as Stefan mentioned, need urgency. If there isn't a need to really change your approach to data, is data mesh worth the effort? Probably not…
Much like with any transformation project, you need time to see results. That doesn't mean you can't deliver incremental value along the way but this is not a quick change at the overall organization level. That's why you transform pockets - domains/LOBs - and work on the greater organization over time.
In Agile transformation we have Agile coaches. Do we need similar in data? Do we need evangelists? It seems you need someone connecting the dots for people. Just because people can self-serve doesn't mean they will really consider the 'art of the possible' with data. Have someone inspiring and coaching them.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#268 Adapting to and Adopting Product Thinking - Transforming Your Org for Sustainable Data Mesh - Interview w/ Iulia Varvara Nov 12, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.

Iulia's LinkedIn: https://www.linkedin.com/in/iuliavarvara/

In this episode, Scott interviewed Iulia Varvara, Advisory Consultant in Digital and Organizational Transformation at Thoughtworks. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Iulia's point of view:

If you are greatly changing your general approach to something - which data mesh does in many ways - you need to focus some amount on actual transformation. These approaches are not a switch you flip, it takes time and concerted effort to make lasting changes that work well.
If an organization hasn't really broadly embraced product thinking, starting with data as a product/product thinking in data can act as a catalyst for other aspects of the business to embrace product thinking.
You don't change the organizational mindset through words - you start using new ways of working that change people's mindset as they see the benefit of those ways of working. At the end of the day, talk is cheap.
To do data mesh well and have it work for an organization, it's best to tailor to their existing ways of working. Yes, change is necessary but a revolution is far less likely to work than an evolution. How are teams working and where can we make smaller tweaks?
Because you need to tailor your implementation to your own organization, any data mesh blueprint that will supposedly work for all organizations is likely to be snake oil at best.
?Controversial?: The first two principles of data mesh - domain data ownership and data as a product - have the most impact on the organizational operating model. Scott note: federated computational governance might be up there for some organizations too.
With data mesh - or really any change to your operating model - you need to make the goal(s) and measures of success visible to your people. That way, people can understand if progress is happening and what to prioritize.
In product thinking, everything really centers around user value. It will inform your strategy, vision, and business goals. Create feedback loops to constantly test against serving user value.
Create long-lived teams around your data products. If not, there is a significant risk of falling back to project thinking which will mean fewer innovations/experiments with the data and not enough subject matter expertise embedded into the data products to make them truly valuable.
Similarly: "stop funding the work, start funding the team." Stop trying to focus on exact tasks to be done instead of outcomes to achieve. That's key to product thinking.
Many people believe you have to completely change your company to start having domains own their data. Start with one or two domains, don't change the entire company upfront.
When funding the teams, there is an assumption that the team will experiment and find more and more things that are valuable to users. But that ability to _find_ value is crucial - you won't always know what is valuable ahead of time, give the team space to find value.
Also, by funding long-lived teams, they can learn more and more about what drives user value and keep an eye out for work that is no longer aligned to user value - you can shut down work that's no longer valuable. This is a rarer occurrence in data than people expect, shutting down work.

Iulia started with a few basics about general transformation and digital transformation, whether that includes data or not. To really be able to embrace a digital and data-driven future, organizations need to embrace product thinking across the entire organization. They need to align their strategy and operating model to be adaptable and flexible. If the organization has already embraced product thinking, few have really pushed that to data in her experience. But if the organization is new to product thinking entirely, then starting from the data side could create a strong catalyst because data is probably one of the hardest concepts to apply product thinking to - after taking on data, product thinking is far easier to grasp in other areas of the business.

In general, Iulia believes that mindset changes don't come from mandates. Instead, by implementing new ways of working, people's mindsets will start to shift once they see the impacts/benefits of those new ways of working. They see the benefit and change their mindsets. But that will of course take time and concerted effort - the mindset change by decree is faster but doesn't typically stick. Show don't tell.

As other guests have noted, Iulia pointed out to maximize the chance of a data mesh implementation succeeding, you have to take into account the existing ways of working, the organizational and team operating models. Yes, certain aspects will need to change but trying to completely change an organization's operating model is going to be too disruptive. Instead, align a transformation paradigm to how the organization already works so people can evolve and adapt. Don't throw them in the change deep end and don't throw the baby out with the bathwater. Of course, this also means there isn't some blueprint for data mesh that will work for all organizations.

In Iulia's book, the first two pillars of data mesh - domain-based data ownership and data as a product - are the two that have the biggest impact on the organizational operating model. She said, "When you start thinking about your data in terms of products, and put your user in the center of your attention, you try to organize all your efforts around the user needs. Right? You create this connection between the data team and the value." That is a big change to how most organizations work around data and it will take effort to make it happen.

In general, Iulia recommends that for any large operating model change, you really need to clearly communicate multiple things. What are the changes, why are you making them, what is the actual target outcome/goal, what are the measures of success, etc. That way, people can measure how well things are moving forward and more easily prioritize. "Because there would be so many things to be done at the beginning, that team really needs to have a clear understanding what to start with." Transformation will mean tens of changes, understanding where to start and why are crucial.

Specifically to product thinking, user value is your Northstar for Iulia. It will inform your strategy, vision, and business goals. Those business goals will be split into hypotheses of value for how you can reach the goals. This is where you start to allocate teams, to the actionable items from the hypotheses of value. But it all comes back to focusing on user value. Steer your work through feedback loops to focus on that user value and you have a great shot at implementing product thinking/focus well.

Iulia pointed to something many miss when it comes to treating your data as a product. If you don't have a long-lived data product team, it can cause many issues that significantly undercut the value of building data products. One is that you often lack the subject matter expertise in the data product team, so the information encapsulated is not nearly as deep or as relevant to the topic area for the data product. Another is that if the team isn't long-lived, will they really have the time and psychological safety to run experiments and innovate?

Similarly to data as a product, Iulia recommends going small, then sustaining, then scaling when it comes to domain ownership. Basically, start from one to two domains and go broader over time. Trying to reorganize your organization on day one so one or two domains can own their data in that time-frame, that's a TON of effort. Don't try to revolutionize your company to do data mesh, evolve and build the understanding as you go broader. You need to prove out value first too before you go broad.

In wrapping up, Iulia returned to the concept of funding the teams, not the work, and especially long-lived teams. When you fund the teams, they are able to focus on finding value. There isn't an expectation of the teams to be prescient, always knowing what will be valuable. And there isn't a need to simply react to tickets instead of finding what will be of value. The other aspect is that you can understand what should be decommissioned. Far too often, data work continues well past when it is valuable. But with a product mindset, teams can constantly be focused on user value and shut down things that no longer drive value.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 12, 2023 Nov 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#267 The Developer Experience - How Do We Delight Data Developers? - Zhamak's Corner 30 Nov 09, 2023

Key Points:

The current data product developer experience sucks. In software, we have one simple interface that manages all the pieces needed to get the job done, packages those components. In data, we have to jump through hoops and interfaces across many tools.
Right now, the developer has to manage everything themselves which is a ton of work. But it's also a big risk because of lifecycle management - if everything isn't packaged and deployed as one, you have dependencies drifting from each other.
There are many places from software we can learn from as to how to do this containerization. Ruby-on-Rails and CloudFoundry are good places to look.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#266 Leveraging Decades of Information Architecture Learnings to Do Data Well - Interview w/ Akins Lawal Nov 05, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Akins' LinkedIn: https://www.linkedin.com/in/akinslawal/

Schema for Success: https://www.schemaforsuccess.com/about

In this episode, Scott interviewed Akins Lawal, a Data Strategist. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Akins' point of view:

Far too often in data, people move to build _something_ instead of focusing on building good information architecture specific to the task at hand and the organizational goals and capabilities.
Good information architecture isn't about tech, it's about the principles and practices of how you're going to structure your data/information.
Keep going back to good product principles in information architecture: your focus should be on what are you trying to accomplish over what are you trying to build.
?Controversial?: Organizations need to focus far more on hiring for learning capacity instead of only for current skills. The world is changing too quickly to try to focus on specific skills for many data-intensive jobs.
Leadership buy-in ends up being the number one determining factor of success for projects and transformation according to many studies. Trying to proceed - even with the greatest plan ever - without that buy-in greatly reduces the chances of success.
Maturity models can be extremely helpful but they sometimes don't tell the full story. Look for pockets of maturity in your organization and see what can be copied/replicated and what can't when improving the maturity of other areas of the organization.
!Controversial!: There is something different about information exchange in person rather than entirely virtual. It is far more likely to create a deeper understanding if you can collaborate and whiteboard. Scott note: I'm 50/50 on this. A big question is cost benefit of mental energy, travel, etc. - better return doesn't always mean better return on investment.
Let people know "that data is more of a tool to empower … that also needs to be woven into the organizational fabric."
Reward people for (appropriately) sharing their data. At the start, accolades can be enough but if you really understand incentives, it should become something more deeply rewarded.
People, process, tools but it always comes back to people being the most important.
You can focus on and improve your data culture no matter the size of an organization from two to 200K.

In Akins' view, many more people need to slow down a bit to really consider what they are building, why, and for whom. Far too often, data people - and tech people in general - want to build something but we need to focus on good information architecture. We need to consider a lot about the specifics of why we are building something and also who will use it and how in order to maximize positive outcomes.

When starting with good information architecture, Akins recommends asking a lot about what the system will do and why. Map out the user flows even, how will people access information. That way, you can start to back into what kind of systems and approaches will support what you are trying to accomplish - it's not about what you are trying to build, it's about what are you trying to accomplish and building something that can help.

As you start to plan out your information architecture, Akins recommends you start finding your champions - you need people to rally around to get things moving forward. Then you plan out your process - there are many different tried and true processes for sharing context/information but it's important to find one that works well with the use case and your organization. You should be looking for happy mediums between all involved because no one will get everything they want if you're doing it well.

In Akins' view, if you want to become data driven as an organization, you need to focus on hiring learners, not just for current skill sets. Data - how to analyze it, leverage it, share it, etc. - is a lifelong problem/challenge. New tech and approaches always emerge. You want someone focused on staying up-to-date on best practices, not specifically proficient in one tool that could be close to obsolete in a few years.

Leadership buy-in is far and away the most important factor to new initiatives succeeding, according to multiple studies from groups like HBR and McKinsey. So Akins recommends to make sure you are aligning with those leaders and showing them the benefit of improving your data initiatives. A big reason so many data initiatives fail is that lack of buy-in and support. The best laid plans are still far less likely to succeed without support from above.

Akins talked about how maturity models can be a very helpful tool for finding what's already working in your organization relative to data work. You can find the patterns and then assess if you can make those into repeatable patterns for the rest of the organization or not. It's all about creating a situation where things can mature.

In person collaboration for Akins just kind of 'hits different' - you are better able to exchange context if you're in the same room and able to whiteboard. Potentially that's around driving meaning and trust? Virtual tools still have not caught up to the in-person collaboration capabilities. So if you aren't in person, he believes you will likely need to meet more often just to ensure you really understand each other.

Akins then talked about the importance of leveraging data to empower people and weaving that data understanding and empowerment into the fabric of the organization. How do you empower people to make more and better decisions with data? That's how you move towards being data driven.

Something Akins likes for driving data sharing and usage is incentivization: how are you rewarding people for sharing and leveraging data? Leaders should be giving teams and people that are sharing and using data well lots of accolades and potentially other rewards so others want in on the action.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 5, 2023 Nov 05, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#265 Are You (Even) Doing Data Mesh™? - Mesh Musings 55 Nov 02, 2023

Key takeaways:

If you want to say you are doing data mesh, go ahead. If you really want to know if you are, look to Zhamak's book and her talks - how much are you attempting and are you going with the thin slice model - not ignoring any of the pillars. Have some empathy for yourself.
If others want to say they are doing data mesh, it's not really a big deal as long as we aren't trying to emulate the ones fooling themselves or over-marketing.
I'd rather welcome more folks into the tent than gatekeep. Maybe that is just my own prerogative but I have empathy for people who see what they think the end journey should look like. But it's like that transformation picture of build a skateboard, then bike, then moped, then motorcycle, then little 2 door car, etc. with the end idea of building a sedan.

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#264 Will GenAI and Data Mesh Really Mix? - Interview w/ Madhav Srinath Oct 29, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Madhav's LinkedIn: https://www.linkedin.com/in/madhavsrinath/

In this episode, Scott interviewed Madhav Srinath, CEO at Nexusleap.

Overall, we are super early in the Generative AI cycle and hype is huge. This discussion is one of early impressions, not fully formed answers. It's far too early for that.

Also, FYI, there were some technical difficulties in this episode where the recording kept shutting down and had to be restarted. So thanks to Madhav for sticking through and hopefully it isn't too noticeable. Generative AI will mostly be shortened to GenAI throughout these notes. LLM stands for large language models which power GenAI.

Some key takeaways/thoughts from Madhav's point of view:

?Controversial?: An emerging best practice seems to be having layers of LLMs - one model where you might ask it complicated questions and the second model is trained specifically to vet the answers for correctness and governance concerns.
The cost of running many models in production is typically actually quite low, at least infrastructure wise. Instead of an always-on architecture, most organizations are leveraging a serverless architecture - or leverage APIs from others providing the models - so they essentially only pay a few cents per query.
?Controversial?: Use GenAI as a "scalpel, not a broadsword". Many are trying to use them in overly broad ways and getting not great results.
The ability to take a mountain of data and get something out of it in a structured way isn't a new concept. We've been trying to do that with data mining for years. It's just that it is finally maturing into something more widely useful/usable with GenAI.
People are generally still only trying to solve pretty shallow problems with GenAI, e.g. writing an article. Scott note: That's probably good because most aren't ready to do the work necessary to have GenAI be usable for much deeper use cases
?Controversial?: We may need human handlers for LLMs to do GenAI well. If we aren't sure of the quality of the answers and we need high quality answers, there needs to be guardrails and probably a human in the loop. Scott note: this might prove to be better than having the human just do the analysis or not, remains to be seen.
If you have the right guardrails in place, there isn't really any harm in starting to work with GenAI. But, you have to understand it's early days and there are definite sharp corners for you to cut yourself on - that human in the loop is important for a myriad of reasons and you have to be careful around things like privacy.
?Controversial?: It's better to start at domain level questions and focus on domain-specific problems right now with GenAI. That way, you can more easily control the inputs you feed it and it can help with more specific, targeted questions.
Look at machine learning use cases. Creating narrow focuses for each model has been proven to be a far better strategy instead of creating one overarching model to try to solve many problems. Why not try the same with LLMs, creating models specific to topics?
Relatedly, you can add more focus areas to an LLM as you train it. Trying to get it to understand everything at the start will likely overwhelm the LLM to a point where the quality of your answers will fall.
LLMs can be used to infer relationships between domains or data products. You still have to point them at high quality data and you need someone to check their work but they could be used to more easily find out where data products already are or should be interoperable.
A potential good use case is to have GenAI models focused on finding those potential relationships and then use a second GenAI model that's more targeted at finding information based on those relationships.
?Controversial?: You shouldn't be training GenAI models from scratch. Start with one of the many open source models available and train it on your specifics. Leverage work that others have already done for you. You can train them by having your business specialists share information with the LLMs.
Since everyone essentially has access to the same models, companies will differentiate on the information - and especially the quality of the information - they feed their LLMs.
?Controversial?: GenAI may be more useful for data producers than data consumers. They still need to focus on the fundamentals but GenAI can really make them more productive. Scott note: imagine being able to get 5 sample data models or be able to ask an LLM to figure out the best way to make your data fit with other data products to be interoperable.

For Madhav, a lot of what Generative AI has become is the concept of data mining with a personable interface. We've been trying to create a way to dig into data that is unstructured and get some insights or information - something that is structured - for a while. The concept isn't new but we've finally found something that might actually be able to do it well and make the outputs easy to consume.

Right now, in Madhav's view most of the emerging GenAI use cases have been pretty shallow, such as to help write an article. It probably can get far deeper but it's quite early days. However, we probably need to put a human in the loop to actually make sure the answers LLMs are giving are correct. That might be the best option in his view, something like a guide plus guardrails driven by a human to make sure these LLMs don't hallucinate. In a way, this isn't all that different from other machine learning work - black boxes tend to have unexpected consequences.

Madhav's view is that it's totally okay to start working with GenAI at your organization as long as you understand GenAI/LLMs have quality issues right now and are really only at the MVP stage in many senses - especially if you are using them internally on your own data. There will probably be good ways to put a wrapper around them to prevent improper data usage/leakage and prevent hallucinations as well.

Starting with domain specific questions/problems is where Madhav thinks people should focus their GenAI work. If you try to feed an LLM a ton of information from many sources, you can't really be sure of the logic it uses to generate answers versus keeping the inputs tighter and asking about more specific business areas. Keeping that tighter focus and having many LLMs across the organization gives you an ability to more tightly focus your models on specific topics. You can then attempt to add additional focus areas to those LLMs once you have the model performing well on one topic.

While LLMs aren't magic, Madhav is seeing an emerging use case where people point the LLMs at maybe two data products and ask it to infer relationships between them. It might discover something people haven't thought of before. You still need a human in the loop or you end up with something like the Pastafarian 'belief' that global temperature rise since the 1700s is caused by the decreasing number of pirates globally - correlation doesn't equal causation. It's not a magic wand but it could help people find more places where data is already interoperable or should be. Then, he's seeing once those relationships are discovered, a differently tuned GenAI model is used to actually infer some information based on those newly discovered relationships. Again, specialized models.

Madhav doesn't believe most organizations should be training their GenAI models from scratch. Instead, go and find the open source models and add your necessary information to the training - basically, why start at zero to train it to one when you can start at 0.7? Leverage the work others are doing so you don't need super expensive LLM training focused engineers. By starting with an existing base model, you can tune it based on your own answers instead of trying to feed it very specific data and supervise the base-level initial training. Leverage your business subject matter experts and get them to share their tribal knowledge with the LLMs as well.

Circling back to the idea of layered LLMs, Madhav talked about how some organizations are having models specifically tuned to answer questions about data but then there is a secondary LLM that is focused on checking the answers the first one gives for sanity/correctness as well as around governance, e.g. security and privacy concerns. Again, that separation of the work. And it's not nearly that expensive when it's all done in a serverless way - if your LLMs are becoming cost prohibitive, you are probably not running them in a cost effective way.

Madhav has a few strong feelings around what organizations should be doing with LLMs. The first is that most should not be trying to train their own LLMs - the time and cost just don't make that much sense when open source models are advancing at frankly remarkable speeds. The second is that GenAI is probably more helpful for data producers than data consumers. It really can make producers far more productive, e.g. letting them generate insights on their own data or helping them to find good interoperability points with their data and other data products.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 29, 2023 Oct 29, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#263 Panel: Applying Site Reliability Engineering Practices to Data - Led by Emily Gorcenski w/ Amy Tobey and Alex Hidalgo Oct 26, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Emily's LinkedIn: https://www.linkedin.com/in/emily-gorcenski-0a3830200/

Amy's LinkedIn: https://www.linkedin.com/in/amytobey/

Alex's LinkedIn: https://www.linkedin.com/in/alex-hidalgo-6823971b7/

Alex's Book Implementing Service Level Objectives: https://www.alex-hidalgo.com/the-slo-book

In this episode, guest host Emily Gorcenski, Head of Data and AI for Thoughtworks Europe (guest of episode #72) facilitated a discussion with Amy Tobey, Senior Principal Engineer at Equinix and Alex Hidalgo, Principal Reliability Advocate at Nobl9. As per usual, all guests were only reflecting their own views.

The topic for this panel was applying reliability engineering practices to data. This is different than engineering for data reliability which is focused on data quality specifically.

The overall concept is taking what we've learned from reliability engineering across disciplines but mostly in software, especially SRE/site reliability engineering, and bringing those learnings to data to make data - especially data production and serving - more reliable and scalable. Scott note: this is probably one of the most frustrating topics in data for me because it feels like it's basic foundational work yet most organizations aren't tackling this well yet if at all really. The best starting point for an organization is simple awareness and starting to have reliability engineering conversations around data. And you will probably feel like you're behind after listening to this. Everyone is behind on this 😅even most orgs aren't doing SRE well so applying it to data, that's no surprise.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Reliability engineering is one of the majorly overlooked aspects of what we need to bring to data - whether you are talking data mesh or not. People are starting to really move on product thinking and to a lesser degree bringing microservices and CI/CD approaches to data. But reliability engineering is still a somewhat distant afterthought if at all.
At the end of the day, reliability engineering comes down to observability first. If you can't observe what's happening with your systems, you can't really start to identify issues and work on fixes.
Observability in data CANNOT only be about the data quality. That is observing what is happening with the data versus the data applications/systems/pipelines/etc. The data itself may seem fine - no quality issues - but if your systems storing, moving, transforming, etc. your data are degrading, you are missing the forest for the trees. A broken pipe that delivers 5% of the water but it's clean is still broken even if the water passes quality checks.
Relatedly and somewhat in contrast - the measures used in operational systems reliability engineering like availability and latency are potentially the wrong areas to focus on for reliability engineering in data. Don't copy-paste those key metrics, think about what matters when it comes to the data applications/systems as well as the actual information you are sharing.
As Amy mentioned, there is too much binary thinking about data - whether it is quality, reliability, availability, etc. The idea of criticality is often missing from data discussions - and trying to measure by who screams the loudest is a really poor approach 😅 As an overall organization, we need to map what data is critical and why.
The best way to find out your SLOs for data related work is to have conversations. Just because you can, doesn't mean you should. Not all needs are created equal. Don't take consumers at their first word, dig in and find out what will serve their needs but limit the work to something reasonable. Remember that return on investment is what matters, not simply return. Most data work has value but does the value justify the costs?
Keep things simple. As Alex said, focus on "is this doing what my users need it to do?" Look to measure around what your systems need to do to serve customer needs. It's very easy to get caught up measuring the wrong things.
Currently, in many organizations, reliability engineering work is much more like archaeology than engineering as Emily noted. We need to focus on providing the context to those trying to ensure reliability and that takes forethought on the systems/architecture side as well as the organizational side. Observability - data or otherwise - isn't a switch you can easily flip, it's a practice you build and get better at over time.

Other Important Takeaways (many touch on similar points from different aspects):

You can make progress on reliability engineering in data just by making sure you have an incident management process. If you do, then look to improve it. You can iterate towards good, everyone starts somewhere :)
As Emily pointed to, can all your SLOs really be covered by SLAs in data? How do you have an SLA around semantic meaning for instance? Do we need to extend the SLO concept to really make it applicable to data?
Just starting the reliability concept discussions with people in data is often difficult if people don't have operational systems/plane experience. In data, far too often there is a either 'it's right or it's wrong' approach instead of how acceptable something is. Sometimes imperfect data is acceptable and the same goes for the reliability of your systems.
You can iteratively improve your observability and reliability practices. You can start to show value as you build more capability. Much like with anything, unless you have budget and full high-level support, start small.
Take Alex's simple definition of reliability: "Is this doing what it's supposed to be doing?" Then work to find ways to observe if your systems are actually doing what they are supposed to be doing. And only focusing on where you think things could go wrong is monitoring, focus on observing - the difference between those two is a rabbit hole you should go down yourselves.
Relatedly, you need to really consider expected behavior when building systems. This is of course around data quality but also setting your SLOs - what are you trying to achieve by building the data application/product?
As Alex said, "SLAs really exist for lawyers and financial departments, and SLOs exist for engineers actually trying to keep things reliable." Data contracts are often focusing on the SLAs instead of the SLOs but both are important when thinking about data. SLAs engender trust as a guarantee, SLOs are about making sure you're actually thinking about consumer needs.
?Controversial?: Don't start with formal SLAs - and maybe SLOs - unless your organization is mature enough to handle them. People under 18 - at least in many countries - aren't able to enter many legal agreements for a reason. Your maturity level matters. Potentially look to informal agreements or 'best faith effort' as you are starting to think about SLAs and SLOs.
People often think about data freshness and quality metrics but there are other important metrics like durability and availability. If you have perfect data but no one can access it or it gets lost, what value is it?
Does managing cost fall under reliability engineering? Efficient usage of resources often falls on the SRE team to some degree - how do we think about wrapping in cost management. As Emily noted, many companies have their data management on a different cloud from their applications and that creates a LOT of cost inefficiencies.
Cloud economics actually prevent good data usage and practices. Because cloud providers make so much of their money on data movement - especially egress - the economics of using best-in-class tools becomes untenable, especially if those are on a separate cloud.
Relatedly, many - most? - cloud data tooling seems to be focusing on driving usage and revenue instead of driving value taken in the greater picture. Instead of focusing on how to play together to create value, each is creating lock-in. That creates additional reliability and observability challenges.
In data, many organizations are still at the pre-crawl phase relative to observability. Something as basic as 'if you create an alert, have a runbook for when the alert goes off' is missing. We need to bring maturity in - but at a reasonable pace, not try to fix overnight - so organizations can get to actually reliable systems and data.
For a lot of organizations, they still treat data work as a cost center. Even in data mesh to some degree. If we can't prove out the value of the data work in general, it will be MUCH harder to get budget for something like reliability engineering work in data. But reliability is really crucial if you want to become data driven rather than simply say you're data driven.
At the end of the day, lack of reliability engineering practices in data are twofold: first, lack of awareness. Second, organizational. There simply needs to be a focus and investment if we believe reliability engineering in data will pay off. Or maybe it's not worth it for most organizations. Time will tell but you can bet that the most advanced organizations that really are 'data-driven' aren't skipping it.
Once we tackle the above, hopefully tooling will follow but data tooling in general just isn't in a good spot relative to reliability engineering - it's too hard, too manual, and too expensive in general to do reliability and observability well around data systems. Scott note: this is why I'm an advisor to Masthead Data because they are doing some really cool reliability engineering stuff for data.
One positive note is that industry challenges such as GDPR are driving organizations to realize they need to change. It may not be happening that quickly but the cost of keeping up with regulation in legacy systems is becoming greater by the day. We might see a bigger tipping point soon.
A key point of SRE is to understand the inherent complexity and interconnectedness of our systems. There's some understanding of that which exists in data but we generally don't look to solve for it by learning about how things are interconnected beyond lineage.
Good observability and reliability engineering practices have change as an inbuilt assumption. This is where monitoring often falls short - you have to constantly update your very explicit assumptions versus your understanding. Focus on understanding the needs of your users/customers and observe if the system is actually serving those needs appropriately.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#262 Setting the Groundwork to Become Data Driven - Interview w/ Corrin Shlomo Goldenberg Oct 22, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Corrin's LinkedIn: https://www.linkedin.com/in/corrin/

In this episode, Scott interviewed Corrin Shlomo Goldenberg, Senior Product Manager of the Data Platform at BigPanda.

It's important to note that BigPanda is not at the stage yet where data mesh makes sense but this is a story of getting production of data into the heads and hearts of the application development team, which is a crucial aspect to doing data mesh well, whether it's done pre data mesh or as part of the journey.

Some key takeaways/thoughts from Corrin's point of view:

When doing data work, it's easy to fall into the trap of trying to do everything. Go back to product basics - start from the why, why are you doing this? If not, we just are creating new forms data swamps.
It's not uncommon for developers to think of data simply as what's in the database, especially in B2B startups. Sometimes you have to work with them to get them to really understand they need to be creating and storing data to be leveraged for analytics. It's not even data exhaust, sometimes the data doesn't even exist!
Related, many B2B companies feel they aren't data oriented enough. You can work to change that of course but know that almost everyone else feels the same; we all start our data journey somewhere, get inspired to go forward.
It's hard to pinpoint the time for a growing B2B company when it's actually time to start collecting and analyzing a lot of their data versus when it would be overkill/too early. Scott note: for larger organizations, look to have the conversation early in the lifecycle of any product - build a data sourcing strategy even if it's not implemented from day 1.
Obviously doing data work isn't free - make sure you have the conversation about when to flip the switch. It's often driven by someone wanting a report or information, prepare ahead of that.
An indicator you need to be preparing more data is when the product managers are struggling to answer basic questions. Often it's the 'how many' type questions that shouldn't be hard to know.
?Controversial?: When getting started with development teams even understanding data work, it's far easier to have that data work centralized in a data team. You can decentralize over time but introducing them to the idea of data work and a data platform in general while trying to hand over ownership might be too much. Scott note: this probably isn't really controversial as much as an inconvenient truth.
Prioritization is key - not just what you work on but what is the incremental value of different aspects of work. Look to make sure you can justify what work you are doing - circle back to 'the why'.
Ownership isn't just about who owns the work but who owns the outcomes. Focusing on the work over the target outcome is not likely to end well.
Similarly, ownership isn't always black and white. While a team owns their domain, a central data team will often own the data related to the domain. Partnership is crucial, teamwork makes the dream work.
Use good product management practices - just building something won't automatically create usage. Talk to your constituents and help them understand what you've built and why to drive more usage.

Corrin started with the tale of BigPanda and how she started building out their data, ML, and analytics capabilities. When she came in, they didn't have the infrastructure or really the focus on a scalable platform for storing and analyzing their internal data. They were doing a lot of this for external clients but hadn't moved to doing it internally, which is pretty common in B2B startups. But BigPanda wanted to do a data driven transformation of their business model so they had to change the situation around their internal data.

There is always a balance for when you start collecting data at scale in Corrin's mind. At a B2B startup, you need to ask how early should it be for the company but the same is applicable for an early-stage offering at a larger organization. Most development teams aren't tasked with dealing with creating the necessary data until far later in an offering's lifecycle but it would be nice if you could include it at the start. But it definitely isn't free so there is always a balance and the conversations need happen, hopefully earlier than later.

Corrin's tipping point for when you should really start to press development teams on creating necessary data is when it becomes hard to answer simple 'how many' type questions. It is also an easier conversation than a hypothetical one. If it takes more than a day to get basic information on how your customers are using your product, that's obviously an issue that's only going to grow. It's also a pretty tangible place to start.

When they started to build out the data platform, Corrin said it just made sense to start centralized. If the R&D team wasn't really thinking about data, trying to upskill them enough to take over the work entirely was probably a bridge too far. Plus, if your data requirements aren't complex enough to require decentralization, decentralization is often just an extra layer of complexity. So they moved to a high communication model where people can see what data work is happening even if it's controlled by the central team. They can slowly upskill the development teams to understand data instead of trying to hand over ownership prematurely.

Corrin talked about working with the team to understand the product mindset to data. Start from the why - it's easy to fall into the trap of trying to do everything because it might have value. That's what happened with data lakes that became data swamps. Focus people on the why and you can bring them more and more into working with data.

Similarly, while Corrin and team didn't have a lot of pushback on getting things done, she was very cognizant of prioritization and cost/benefit. Again, focusing on 'the why': what is most important and when? Why are the requirements like this? Can we cut the cost down by storing for less time and/or refreshing less often? When you say 'real time', what do you actually mean? Etc.

Corrin has been seeing good results from having strong ownership conversations. While the central team still owns the data, they are partnering with the domains as the domains still need to own the concepts and the understanding of the information. While this might not work at a large scale, it's perfectly normal and functional at a 300 person company. Scott note: centralization isn't the enemy until it becomes a bottleneck 😎

As with all global companies, BigPanda has some challenges around communication, per Corrin. Time zone differences and of course differences in focus are just two of them. So she recommends spending a lot of time to communicate to stakeholders about what you are building and why. It's easy to assume that because you build out a data product, people will use it but you have to work with people to ensure they actually use what you built.

Corrin pointed to the fact that many companies in the B2B space feel they aren't "data oriented" enough. She gave a few tips for how to become more data oriented but also has empathy for people feeling that - it's pretty common, most B2B companies feels they aren't as data oriented as everyone else. Similar to data mesh, where everyone believes all the other companies are far down their path. It's simply optics - companies project a better image than the reality of their situation with data.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 22, 2023 Oct 22, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#261 Just What the Heck is a Data Product Container? - Zhamak's Corner 29 Oct 19, 2023

Key Points:

Containers in software abstracted away a number of very cumbersome tasks and encapsulated a lot of the dependencies software had to its environment. Combined, that meant developers could focus on delivering value instead of focusing on the infra. We need to do the same in data.
It's all about sharing data in a responsible and easy way. That means putting all the components together so you don't have to manage many versions. Just like microservices.
How do you manage to make this easy for the data product developer - bundle everything together. But centralized data products are creating a lot of potential issues/risk to scalability and flexibility. We just keep trying to centralize in data.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#260 Driving the Big Picture Forward - More on Northern Trust's Data Mesh Implementation - Interview w/ Jimmy Kozlow Oct 15, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Jimmy's LinkedIn: https://www.linkedin.com/in/jimmy-kozlow-02863513/

In this episode, Scott interviewed Jimmy Kozlow, Data Mesh Enablement Lead at Northern Trust. To be clear, he was only representing his own views on the episode.

Also, FYI, there were some technical difficulties in this episode where the recording kept shutting down and had to be restarted. So thanks to Jimmy for sticking through and hopefully it isn't too noticeable that Scott had to ask questions without hearing the full answer to the previous question.

There is a lot of philosophical discussion in this conversation but tied to very deep implementation experience. It is hard to sum up in full without writing a small novel. Basically, this is one to probably listen to over just reading the notes.

Also, Scott came up with a terrible new phrase, asking people to "get out there and be funky."

Some key takeaways/thoughts from Jimmy's point of view:

Start your mesh implementation with your innovators. Find the people who are excited to try out something new. You want to spend your early time on innovating and learning, not constantly driving buy-in. Find good initial partners!
It's okay to start a bit simplistic - with each domain and on your general implementation - because you better believe there are complexities coming as you scale. This is not going to be simple. But the ability to tackle that complexity effectively is what differentiates data mesh so that unavoidable complexity ends up being where you find lots of incremental value.
Data product complexity is often from maintaining interoperability across the data sets within the data product - not just to other data products. You want to build out a data product with more data sets if you can because that means there is much more information a consumer can easily leverage that is highly related instead of stitching things together across many data products.
?Controversial?: There is value in inorganic data product growth. A central team can see where there is a gap in the available data on the mesh. Even when there isn't a use case calling for a specific data product, filling in those important information gaps can create an environment where more use cases will emerge quickly. Scott note: I go back and forth on this a lot. Data for the sake of data is bad but if you identify gaps, it can accelerate a lot of new use cases.
Financial services is somewhat unique in that many of the desired use cases are already known - lines of business have had their use cases often for decades so it's about serving them well in new ways rather than inventing new use cases whole cloth, at least at the start of a journey.
?Controversial?: Start from serving existing needs instead of trying to invent those new use cases. That imagination around new use cases will be important but there's probably a lot of things your business wishes they could already do to serve customers and that's where your low hanging fruit is. Go after those first to create adoption momentum.
While data mesh might provide more initial friction to use cases, - you need to gather up more information early to embed the necessary regulatory/governance decisions - it means a better overall outcome. But getting people bought in that the initial friction is worth it can be hard.
The most important metric - at least for Northern Trust - is shown value. That demonstration creates momentum through excitement and it pushes adoption as more teams want to get some of that value and they can also have the fear of missing out.
What success looks like in data mesh will be different for every organization especially through the different phases of your mesh implementation. Scott note: this is so incredibly important. Consider what would make your implementation a success in each phase.
It's important to understand the difference between creating the ability for teams to leverage data mesh and actual adoption. One is much more tech focused and is very important but value comes from people participating and using what you build. Just because you build it doesn't mean people will use it. Focus on ensuring adoption.
It's crucial to constantly communicate why you are going this route with data mesh. Even if there is friction to getting something completed, it's "the right kind of friction." You are making your data work sustainable and scalable.
There is a steep learning curve in getting to critical mass with data mesh. Each domain has a different capability level and trying to keep teams collaborating and communicating well can be a challenge. Don't assume good communication will happen, you have to enable it.
?Controversial?: Leverage your central team to help domains early in their mesh journey. Help from the central team is an easy way to inject speed to getting that initial data product shipped. And at the same time, they help bring that domain's capabilities up to a necessary level.
When helping a domain, the central team's goal is to get them to good enough quickly and then give them the autonomy to do what's valuable. Things like internal sharing communities are then extremely helpful as domains can exchange deeper insights with each other and connect, creating potential collaborations.
?Controversial?: When first working with a domain, start simple. Don't go for an advanced use case or data product, build up their capabilities instead of trying to throw them in the deep end.
There are so many complexities in data mesh. Constantly consider trading complexity versus value, whether that is speed, a deeper use case, etc. The complexities will still be there when teams are better able to deal with them, no need to try to tackle everything at the start.
Take measurements - e.g. time to first data product production for a new domain - with a big grain of salt. If you are close to the domain, you can get a sense of how much friction is in your processes and platform but there are so many factors impacting time to launch an initial data product.
There's a hard balance to strike between data modeled to fit the use case and data modeled to fit with the rest of the data that's available on the mesh. It's especially hard when the domain really doesn't understand how to model data well yet. Be prepared to help domains out to make sure they aren't just publishing a data silo in data product form.
There are many places in your implementation where you want to reduce time and friction. Getting to a proper level of understanding isn't bad friction. Trying to rush people through their understanding of data and how their data plays into the organization is likely to bite you in the end. Same for trying to rush people through learning your new ways of working.
While it might seem obvious, focus is such a crucial aspect to doing data mesh right. Helping people figure out what to focus on when and keeping the lines of communication open to figure out what's most important right now is TOUGH but valuable.
Similarly, it's incredibly helpful to help people connect the dots, help them see what could be possible. If you have people focused on making sure others can understand the big picture of your implementation and the available data, they can contribute to that big picture so much more.
An organization starting out highly centralized will have a very different journey to one that is starting out highly decentralized.
?Controversial?: Your central team is there to mostly "coordinate, facilitate, and align people to the policies and practices" plus drive adoption. Look to have a guiding hand, a light touch.
?Controversial?: Be very careful cutting corners in your implementation. There are many ways to hurt your scalability. Governance is especially a bad idea to cut in financial services.
?Controversial?: Ask yourself who, at the end of the day, is tasked with making sure your data mesh implementation continues to progress. Not the specific tasks like building a data product but the overall implementation. If there isn't someone focused heavily on that, you might want to reconsider your approach.

Jimmy's role is a rather unique one in that he is literally tasked with enabling data mesh to happen and make sure they are focusing on the right things. That means a myriad of different things but a lot of it is ensuring the communication and collaboration happens where it needs to while still focusing on the big picture. It might be like an American football coach that coaches the entire team but is still calling plays and making the minute decisions during the game too. It's a big set of tasks to take on.

At Northern Trust, they started their mesh implementation with the innovators according to Jimmy. That's a common pattern for all tech innovation, not just in data mesh - find the people enthusiastic to try something new so you don't have to spend half your time driving buy-in. There obviously also had to be a need where data mesh could work and would be a differentiator.

For Jimmy, one big complexity factor he is seeing is around data products. He believes that you really start to drive value of a data product higher the more relevant and interoperable data sets you can include in the data product - within reason of course. But as you add that fourth or fifth data set, it gets complex to maintain interoperability and consumability even at the data product level. But that complexity is where a big part of the value really lies in data mesh - that it pushes organizations to take on that complexity but in a scalable way.

There is a bit of a push/pull in data mesh for Jimmy: organic data product growth from new use cases versus the centralized team pushing for the value from more and more interoperable data - basically creating data products that fill the gaps between existing data products to enable additional use cases. New use cases emerge the more data products you have that are well crafted and that link data across domains; but the question becomes do you push for new data products that don't have a specific use case - inorganic growth - in order to create a tipping point where those use cases can quickly emerge? It would mean less work for consumers and faster time to market if the data is already available instead of working with the producing team. But will the data products be made well enough to serve use cases? Are people ready to go and discover pre-made data products instead of ones tailored to their needs? Should we only apply this to new data products - how would a central team spot when additional attributes are needed to take an emerging data product from only serving a use case to being more globally valuable? It's still a question in data mesh as to whether to pursue inorganic data product creation.

At the start of their journey - and even though they are two plus years in, it's still relatively early - Jimmy and team are focusing on existing, known use cases as they build out the existing available data and improve their capabilities. Capabilities not just to deliver new data products but how to deliver incremental data that fits well into their already existing set of available data on the mesh. It's about building out the entire picture instead of focusing too much at the micro level. Scott note: balancing that micro and macro level is hard - extremely hard? - but the earlier you get good at figuring out how to add value at the overall mesh level while serving use cases, the more value you will deliver with each incremental data product.

Jimmy talked about with data mesh, even though we can deliver scalable data products relatively quickly, there can be more initial friction for new use cases. E.g. teams have to go and collect the necessary information to do the governance well instead of trying to add the governance at the end. And some might be frustrated or not bought in that the upfront friction is worth it. So he's trying to show the value far exceeds that initial extra friction but people will of course resist new ways of working. Such is the nature of working with humans 😅

When asked what does success look like for them, Jimmy pointed to showing value. It's important to note that isn't merely delivering value but being able to show that value. As Jerry McGuire said, "SHOW ME THE MONEY!" Being able to show value generation helps to build momentum and adoption. If you are proving value then it's not nearly as difficult to get incremental investment. Teams want to participate and capture value too. Excitement builds. But it's also important to note that what success looks like will change - maybe not wholesale but at least in part - in different phases of your implementation.

For Jimmy, there are two important aspects of your implementation, essentially the setup and the knockdown. You have to set up your implementation for success by building out the platform and capabilities but getting teams to actually adopt is still crucial. Just because you built something amazing, you still have to work with people to understand the shift in mindset and approach to get them to buy-in and adopt. A great platform that no one is using isn't really a great platform…

Trying to keep momentum of the whole mesh implementation until you reach critical mass is very challenging. Jimmy talked about how trying to get teams with quite different capability and speed levels to work together can be hard. It's not as though any organization is built to all move together as one - it's not a car that's built to move as one unit, it's more like a group of cars - so you need to really focus on the coordination, collaboration, and especially communication. ABC - Always Be Communicating 😎

Jimmy believes that the central data team in data mesh should be a key point of leverage. They can jump in to help a domain early in their journey to get something delivered while raising that domain's capabilities. A central team can bring repeatable patterns to find easy paths for new domains. That way, the domains still learn by doing but they don't have to learn by repeatedly failing. But once a domain is capable enough, the central team tries to move out quickly to give the domains autonomy to do what's valuable. They are also building a community of practice for practitioners to share insights with each other, providing even more leverage and discovering more repeatable, high value patterns.

When asked about bringing a domain up to speed and the question of complexity, Jimmy strongly believes that you should start simple. You don't hand the 7yr old who wants to help you cook the knife and have them go wild on day 1. Get them into a groove, get them confidence and understanding how to deal with data, then you can start to think about adding complexity. But dealing with data is complex enough, keep it simple for them to deliver value initially.

When thinking about success and things like a new domain's time to first data product, Jimmy believes you can learn where there is friction but the actual times vary quite a bit. So you might see lengthening times for new domains launch a data product but it's a good thing because you are dealing with domains who really aren't sure what they are doing and have to learn a ton about data in general and their own data - that means you are penetrating the less data savvy parts of the organization. All things equal, you want to go faster but just take things with a grain of salt.

There's also the fun of trying to thread the needle of data modeled for the initial use case - so fit for purpose - yet also modeled so it fits well and is interoperable with the rest of the data in your available mesh of data products. Jimmy said this is especially true of domains just getting up to speed with dealing with their data and data modeling so prepare for them to move slower and need more help.

When asked about where there is friction in the process of bringing on new domains that we _shouldn't_ try to reduce, Jimmy pointed to learning and understanding. People need to take the time to understand how to do data work and all that but also understand the new ways of working and why the organization is going in this direction. It's that old are you trying to get them to do the steps you say or achieve the target outcome you give them. Learning takes time, don't rush it.

Jimmy's role is pretty unique as far as other organizations telling their story. He's focused on helping people focus in the right areas but also on helping them connect the dots. There is such a big picture when you think about the entire information scape of an organization and helping people to connect to each other and see where they could enhance that bigger picture is highly valuable. This drives better value while also reducing miscommunications, duplication of work, and wasted time. Scott note: this is somewhat similar to the 'Data Sherpa' concept I've mentioned repeatedly that just about everyone looks like I'm a madman when I bring up.

The question of what to decentralize versus centralize is a tough one for every organization doing data mesh. Jimmy pointed to the fact that where

Weekly Episode Summaries and Programming Notes – Week of October 15, 2023 Oct 15, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#259 The Fellowship of the Data Mesh: The Journey to Meshtopia - Mesh Musings 54 Oct 12, 2023

Key points:

Every organization starts from a different location when heading to Data Meshtopia. Every journey will look different. Try to understand why they chose to do what when - often by just getting in touch and asking them.
Because the journey looks different, where you put your focus when will depend. Look to the fundamentals of what are you trying to achieve, how to achieve short and long-term value, and how are you making progress.
You can choose your own journey - there are different paths to getting to Meshtopia, just because you look like another organization doesn't mean you have to tread the same path.
Think about the old trope of playing to your strengths while shoring up your weaknesses. This doesn't mean ONLY play to your strengths, that is where I am seeing the most failures… Just because you're bad at governance doesn't mean you can ignore it…
Find your fellowship or fellowships. Find people to constantly stay in touch with and exchange information. They are also your best leverage points quite often: "Hey, we want to do XYZ approach, Scott here says this is a very standard approach from these 10 organizations and here are the pitfalls so we aren't making this up out of our butts."
Meshtopia isn't some utopia. It's not as if all our problems are suddenly solved - some of the old lingering ones might be but there are new problems to battle too. It's not as if life is blissful the second you get to a better place.

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#258 Data Mesh on Hard Mode: Learnings From Airtel's Early Data Mesh Journey - Interview w/ Sid Shah Oct 08, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Sid's LinkedIn: https://www.linkedin.com/in/siddharthin/

In this episode, Scott interviewed Sid Shah, Head of Product Data and Analytics at Airtel, a large India telecom operator. To be clear, he was only representing his own views on the episode.

Before we jump in, it's important to note that Airtel are doing data mesh on 'hard mode'. Because of regulatory requirements/restrictions, they are all on-prem. That means extra challenges when it comes to securing compute resources.

Some key takeaways/thoughts from Sid's point of view:

Sometimes you have stop/start in data transformation. It's okay, you can rebuild your momentum. It's okay to try to move when you aren't 100% sure.
One important thing about data mesh is that it can validate that you aren't the only organization facing a lot of the common challenges that come with high scale and business complexity/velocity around data. You are not alone. Scott note: there's a reason 500+ companies are doing data mesh :)
Similarly, you can leverage the stories of other organizations on a mesh journey to get buy-in internally. Their stories can help you explain to your organization that others are using this approach, that this isn't just a problem in your organization.
Many data issues in a large organization can probably be traced back to poor ownership in some form or fashion. Can the teams who should own data - the ones who know it best - even own the data if they wanted to? Do they have the tooling and capabilities?
What data related pains are universal to your organization? Does data mesh target any of those? Those universal pains in your organization can be a great selling point for doing data mesh.
Changing data ownership is very much not trivial. It's not about responsibility, it's about building up the capabilities to allow domains to truly own their data in a safe and scalable fashion.
It's crucial to mesh - forgive the pun - data mesh and the way your organization functions and operates. It's crucial for implementing but also driving buy-in. It's a great way to alleviate push back when someone says 'that could never work here' or similar if you align it to your existing ways of working.
The business doesn't want a sausage factory tour. Focus on speaking to their business problems and then actually addressing them. Data quality, velocity, reliability, etc. What can they do and what are you fixing, not how you fix them.
A potential good mesh selling point for line of business leaders is time to market. If the time between deciding to run an experiment and it going live can go from a few weeks to a few days, maybe even a few hours, what does that mean for them?
?Controversial?: Getting outside in perspective to validate your approach can be a good starting point. Look to bring in outside presenters doing data mesh to talk about how data mesh is working for them and help frame it for your organization. Scott note: Controversial because I know of no one else doing it…
?Controversial?: As long as the eventual ownership lies with the domains, it is okay to co-own data and hand it over gradually. Give domains the time they need to increase their data fluency.
If you can, get data mesh explicitly included in the engineering/data strategy for the organization. Scott note: Easier said than done and probably don't go for that until you're ready but it will make your job implementing much easier.
As many others have noted, do not try to onboard every domain at the start of your journey, that's a recipe for disaster. Sometimes you have to tell domains you aren't ready for them yet, whether that is capacity or because their use cases are too complex for your platform right now.
There's a temptation to go after the most 'value rich' domains but you will be far better served going with the domains that are leaning in, that are really excited to participate.
Failure at some scale is inevitable in data mesh. Learn from it and move forward. Try to make sure your stakeholders understand that failure is not the worst thing that can happen if you can iterate based on it and if you limit the blast radius.
Really consider what your focus should be in every stage of your journey. At the start, it's not about building the perfect solution, it's about building momentum, getting a few good wins to use to market internally, and learning where your friction points are so you can build a scalable approach. It's easy to try to get every aspect of data mesh 'right' at the start but that will overwhelm you quickly.
Make sure to keep all your stakeholders engaged. That might sound obvious but every persona - and oftentimes person - is quite different so you will need to focus on different aspects of data mesh when speaking to them to succeed in maintaining engagement.
You'll need to keep a lot of people updated on your data mesh progress and upcoming plans. But very few care about the nitty gritty details. What's changing that will allow them to deliver more value?
Consider whether you should immediately start billing lines of business/domains for using the central mesh platform versus making them aware of the costs first for a time. It can be a big shift in thinking and approach so switching the cost to domains if they aren't getting incremental budget could be a real point of friction/loss of buy-in. Scott note: this is a really important point
It will be pretty hard to lock down your return on your investment in data mesh or for specific use cases. How does the business value faster time to market, fewer incidents, better quality data, new use cases, etc.? Scott note: you can flip it around and make it something domains need to measure or assign a value to but it's not exact :)

Sid started by talking about some of the challenges of data and analytics in a very large organization that has grown through acquisitions - data is siloed across the organization and you can't get sight of the customer journey across lines of business. To address this, a few years ago they started landing data from across the organization into a central data lake. You can probably see where this is going - a central team getting more and more overloaded while there is an ever increasing demand for data. Add to that a data platform where the only teams sophisticated enough to use it being in the central data team and you have major bottlenecks, poor quality governance, the business teams weren't sure what to trust or what data really meant, all combining for a bad analytical experience. So that was the stage for why they started to look at data mesh.

Even before coming to data mesh as a potential solution, Sid and team thought the biggest problem was poor data ownership. The teams who knew the data couldn't own the data even if they wanted to - they didn’t have a platform they could actually use or the data fluency to own their data work. The other issues could really be traced back to that - poor quality, bottlenecks, low trust, etc., it all came back to poor data ownership.

According to Sid, coming across data mesh and all the other organizations that were implementing - including the stories from this podcast and the community - validated that they weren't the only ones experiencing these data and analytics issues. And it gave them not just validation but hope and a set of talking points and proof points to leverage internally for driving understanding and buy-in.

Sid and team understood early on that changing data ownership is not a "trivial exercise". It's not just changing the responsibility, you have to enable domains to actually be able to properly own their data or it's just the same problem of bad data and bottlenecks but just new people to point the finger at. They decided to not try to solve challenges at the small scale but come up with an approach that was going to really target the pain at the enterprise level, so that's why they arrived on data mesh.

Two things that helped drive data mesh forward as a concept at Airtel were a general understanding of data mesh and its aims at the senior leadership level in the organization and again feeling that pain for long enough that people were ready for a fix. A third aspect was creating a way for data mesh to actually happen at Airtel specifically - how could data mesh work in their organization without trying to change everything about the organization and how people/processes worked? Essentially, how can you explain how this would all work to people at their level of involvement - senior leadership doesn't worry much about what tech your platform uses - and that your mesh implementation won't disrupt the entire organization in a bad way while addressing a lot of the causes of data pain.

As a number of guests have pointed to, when speaking to the business, Sid recommends to speak to their business pains. Data quality issues, not being able to nimbly react to the market based on data, reliability issues of things breaking, etc. The business wants to deliver on their objectives, not talk about how you are building the platform. Think about time to experimentation as well - if you can take the time to deploy an experiment in the market from weeks to days or even hours, what can that do for them? The product and engineering orgs love the sausage factory tour so keep it to just them :)

When looking to drive buy-in and understanding of data mesh with the technical teams, the engineering and product teams/leaders, Sid brought in a number of people from other organizations doing data mesh. People got used to the idea of data mesh and also, if they are seeing many internal presentations on data mesh, they understood the momentum was building so they'd have to get on the bandwagon. Scott note: this is part of why Data Mesh Understanding exists - not everyone can easily bring in those outside speakers.

When they were just starting out their journey, a number of domains gave the quite fair pushback of they didn't have the skill sets to do data mesh, to own their own data. They also didn't have the resourcing to take on the additional work, both people and compute - recall, Airtel is all on-prem. So Sid and team had central resources - including people - to loan to those domains who wanted to own their data, who were bought in, but needed that help. Another aspect was to find champions and develop hero stories, where someone not that advanced around data was able to successfully build a dashboard or similar to inspire others and lower the perceived bar to doing the data work themselves.

Sid and team knew that if the domains couldn't successfully own their data, the work would fall back on his team. So they had a bit of an extra incentive to make it work 😅 So they made sure to only take on the amount of work they could handle instead of onboarding every domain they could/that was interested right at the start. Just make sure you communicate effectively/transparently. Scott note: It's absolutely normal and reasonable to push domains out in time if you don't have the capacity to work with them, whether that is resources or platform capabilities around their use cases.

At Airtel, Sid and team didn't succeed on their first attempt at data mesh - in fact they "failed miserably". They weren't fully prepared and didn't have a lot of the issues figured out. Luckily their company didn't throw in the towel but the failure helped them understand what really mattered. It was about making data mesh possible, making it a part of the strategy so teams could feel like it was okay to lean in, and building out the resourcing - platform, compute, and people sides - to make this an actual possibility.

Sid didn't see his early data mesh role as trying to convince every domain, win over everyone upfront. It was about getting out the first few use cases and working with a few domains. It was more about finding collaborators that could help drive to a scalable platform. It was about finding the friction points as domains look to own their data and then addressing those friction points. Really focus on what you need to accomplish and don't try to take on the world :)

It's going to be important - and somewhat challenging - to keep all your stakeholders engaged at the start of your journey according to Sid, especially those not participating early. It's important to keep engineering and product leaders engaged but also the people doing the actual work in the domains and the data team. And of course, you need to keep the business stakeholders engaged as they are the ones who will do a lot of the work but also get the biggest benefit. But it's also crucial to understand each group will require different approaches to keep them engaged. Focus on that incremental progress and find leverage points instead of trying to tell them about everything. For Airtel, the product and engineering teams were excited about faster product delivery and quicker delivery of insights.

For Sid, it was important to communicate to leadership that the data mesh journey would take a few years and to get their buy-in that this was a long-term approach, not a quick quarterly fix. But most teams really only care about what is going to be coming, what value will be delivered, in the next quarter or two. Again, focus on delivering updates about what the audience cares about and understand you will have multiple personas to cater to.

At Airtel, budgeting, especially around data, has been done centrally and not in each domain according to Sid. So, for the first year of their data mesh journey, the data team isn't even showing people exact budgets/costs for their domains around their data work but they intend to build a perspective on overall expected costs. This is a big change in thought for the domains from when the costs were all on a central shared data team. The data team didn't want to switch from essentially an afterthought to billing the domains too quickly but they do want the domains to consider costs.

The return on investment conversation has been somewhat difficult to nail down in many use cases for Sid. What value do you attribute to faster time to market? If something now takes 1 day instead of 4 weeks, there is a very clear business value BUT what is the measurement of that business value? Time savings is nice but ends up being pretty low in the grand scheme at Airtel. There are also new use cases that could not have been done before. But overall, it's hard to point to an exact return figure. Scott note: this is where you can flip this around on the lines of business and ask them to tell you how much value it generates for them to do a use case or to get to market quicker/easier and do smaller scaled, fine-tuned experiments.

Sid and team are still figuring out things like the domain onboarding model and many governance aspects in the platform and out. Should they have an embedded expert in things like data modeling go into domains to help them get up to speed? Now that sharing data is far easier, how do you make sure data is well governed across a number of dimensions like quality, security, privacy, etc.? It's not all figured out and that's okay :)

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 8, 2023 Oct 08, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#257 Panel: Doing Data Mesh Data Governance Well - Led by Andrew Sharp w/ Nicola Askham, Kinda El Maarry, PhD, and Jay Como Oct 05, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Andrew's LinkedIn: https://www.linkedin.com/in/andrewsharp27/

Nicola's LinkedIn: https://www.linkedin.com/in/nicolaaskham/

Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/

Jay's LinkedIn: https://www.linkedin.com/in/jaycomoiii/

In this episode, guest host Andrew Sharp, Principal Consultant - Data Governance & Data Protection at The Oakland Group (guest of episode #172) facilitated a discussion with Kinda El Maarry, PhD, Director of Data Governance at Prima (guest of episode #246), Nicola Askham, AKA The Data Governance Coach, an independent data governance consultant (guest of episode #129), and Jay Como, Strategic Advisor at Curate Insights (guest of episode #92). As per usual, all guests were only reflecting their own views.

The topic for this panel was how do we do data governance well including how do we get started around data governance in data mesh. There's a lot to learn about how to improve your governance but there are no blueprints unfortunately. You have to do the work specific to your organization.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

The single biggest misconception around data mesh is that because we call it doing decentralized data, it means decentralize everything. It's even worse when people think it means decentralize everything on day 1. Find your balances as to how far to decentralize different aspects but a central governance function/team will be crucial to doing data mesh right. They need to focus on leverage points and enablement, especially via guardrails, policies, and standards. Each of those three we want to - at least eventually - automate as much as we can. That's the 'computational' part Zhamak mentions.
If all we do is empower the domains, there will be chaos. We've seen this with improperly done decentralization in tech repeatedly. It's 'federated' governance and not merely decentralized. We need guiding hands provided by governance to enable the domains to do the right things.
In mesh, you have to balance individuality/freedom and consistency. The information encapsulated and how it is encapsulated must be free but why constantly reinvent the wheel where you don't need to? Try to make conformity easy, simple, and the best decision around those places where differentiation isn't a value add.
Data mesh creates a great path to _start_ to federate/decentralize data governance. If you try to rush into decentralizing everything upfront, it will lead to chaos. Domains need training wheels around a lot of governance aspects, don't throw them in the deep end.
Data ownership is such a crucial aspect to get right in data mesh, especially for governance. That can feel obvious but it's also one of the most common places mesh implementations are suffering or falling down. Be clear what ownership means and help owners understand and take up ownership.
Trying to standardize everything instead of create standards that make things easy but are flexible is something that has held back data governance for so long. Data mesh gives us the opportunity to try so many more things out. Take advantage of that and learn along the way.
Communication and culture. In data, we are used to thinking that the data communicates itself: look at this perfectly self-describing data model. It's just not the case. And you can't transform your culture and the way your organization communicates overnight. Invest a lot of time and effort into communication and culture if you want to have sustainable value delivery with data mesh.
Business people don't want 'data governance', they want to achieve business outcomes. Share with them how the work will help them achieve their goals instead of the specifics of data governance. Speak to outcomes and you will get them excited to do data governance. Focus on the why - the how is exciting and interesting to data folks but most business people don't care!

Other Important Takeaways (many touch on similar points from different aspects):

Your mesh governance setup will look different to any other organization's at the very low level and that's to be expected. Your organization is unique - fortunately or unfortunately 😅.
It's incredibly easy to get yourself in trouble by trying to be too prescriptive around mesh data governance and following it 'as written'. Zhamak didn't try to write everything as if it was the mesh bible. We're still figuring out how this works. Test what works and be prepared to change what doesn't.
Mesh data governance best practices are barely emerging at most. You will need to develop your own theories and practices in some regards. And be prepared for lots of evolution as we as a collective industry figure out how to do this better before we figure out how to do it well.
Almost every single organization struggles trying to figure out the mix of responsibilities and roles around data owner, data product owner, data product manager, etc. You will almost certainly not get it perfect to start - test and evolve. Scott note: please, for the love of all that is holy, go talk to say 5 mesh governance leaders and see where they settled on roles and responsibilities so you aren't all repeating the same mistakes.
Every domain will have its own quirks. Don't be afraid of changing things, especially roles/responsibilities depending on domain maturity/capabilities. Yes, everything being uniform would be great but it's not realistic.
Federate/decentralize at your own pace, don't get ahead of yourself. Some aspects of governance will remain more centralized than you expect as you test what works best for you all along the journey. Again, your equilibriums will be different from other orgs and will change along your journey.
Focus on maturity models to understand how to appropriately decentralize aspects of your governance. Not every domain will be as mature either. Again, data mesh is a journey, you don't flick a switch and everything changes.
We need mechanisms to at least enable - and potentially enforce - interoperability. If all we have is good data products that don't interoperate, we've only got high quality data silos. Consider how strongly you encourage interoperability and when.
Part of a successful data mesh implementation will be about ensuring the data products created are valuable. Highly related, it is important to ask if you are creating the right data products. It's hard to do that ahead of time with governance but governance is key to at least measuring if you are succeeding there.
Try to define what good looks like for your domains. It's easier to paint from a reference than from just your mind. It will also encourage more standardization across those things where again freedom and variation don't add value.
Nicola said, "You don't go and assign data owners, you identify and engage them." Instead of treating data ownership like a hot potato, you need to figure out who should be the owner and help them step up. That can be different in each domain. Yes, a blueprint that works for every domain would be great but it's not realistic.
Many people are excited about what data mesh can bring to their team when they take data ownership. Engage and flame those fires. Help them realize that data ownership can be a great boon, not just more work. Easier said than done of course 😅.
On the flip side of course, there are many who will be highly resistant to taking on data ownership. Look to high-level exec support to put the right incentives in place so people will do the right thing. It's also okay to leave some people behind rather than spend all your time trying to convince a brick wall.
Data governance in data mesh is harder than traditional data governance. If you are really struggling to do many of the aspects of traditional data governance, really consider if you want to take on doing data governance on hard mode by going the data mesh route. Data mesh isn't for every organization.
Conversely, data mesh can be something that forces you to really mature quickly when it comes to data governance. It can be the fire that forges much better practices but that's a risk and it will mean a lot of work if you don't have a great foundation. Scott note: I still go with the previous point that don't try to rush into this if you don’t have a strong foundation but they are interesting points to weigh against each other.
An absolutely crucial aspect of good governance is making the implicit explicit. Even make that explicit to your teams; tell them repeatedly to make the implicit explicit. Because that's where you will find friction and misconceptions that will undermine your work.
If you are trying to layer in a ton of new roles, it usually means asking for a lot more budget. Think if that will be a hindrance or can you get away with less than perfect ownership to start. It's a risk but if the initial consumers are understanding too that we don't have the budget to nail it perfectly at the start, it can mean you can move ahead with far fewer road blocks.
Historically, data work has been often bifurcated with the technical/technology data work in one group's hands and the logic/conceptual/non-technical work in another. Learning to blend the technical and non-technical will probably be harder than you expect. It's a lot about unlearning bad habits and that isn't easy.
As often mentioned on this podcast, data literacy in mesh needs to focus on enabling people to actually do the work around data rather than deal specifically with data tools. We need to lower the bar to working with data at the tooling/services/platform and the understanding levels.
You may see engineering people get more engaged in governance than they have historically or more than you’d expect. Obviously, invite them into the party to help out. They can add some really great additional expertise and perspective.
There is a great misconception around mesh data governance and maturity. Everyone seems to think other organizations are doing governance really well while they are struggling. That's just not true, everyone is struggling. That doesn't mean take it easy but also don't get discouraged.
Maintaining mesh momentum is a monumentally material mission. All jokes aside, it is really easy to lose momentum, keep to looking to deliver incremental value and take measured risks - don't swing for the fences at the start.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#256 How to Drive Towards a Data-Driven Culture with Data Mesh - Interview w/ Amy Edwards Oct 01, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Amy's LinkedIn: https://www.linkedin.com/in/amy-tang-edwards/

In this episode, Scott interviewed Amy Edwards, Formerly Director of Analytics and Product at Vista. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Amy's point of view:

A potential Northstar metric for how data driven you are: how often are people getting, interpreting, and actioning on the data themselves versus waiting for an analyst to tell them? Hard to measure but it's where you want to head, where more and more people are making the effort to interact with data.
If you want a data product/use case to succeed, the absolute most important thing is an engaged consumer stakeholder - someone who really, really wants the data for a use case and how they want it. If someone isn't leaning in, consider not building something for them. Scott note: this sounds controversial but it is reinforced in almost every data mesh conversation I have.
Similarly, if you are building a data product, you should make sure you are very aligned with your stakeholder. Don't be a request dumping ground, build iteratively together.
To understand your progress towards being data driven, you need to actually measure things to track changes over time, your progress. You almost certainly will not find the perfect aspects to measure at first and what you measure will evolve. But it starts with measuring something.
There's nothing wrong with starting with a using a success metric that you know isn't going to be something you focus on when a data product matures. As products go through phases, so too should your measurement. So start with number of users as a key early data product success metric - it's easy and somewhat correlated to value driven.
?Controversial?: The adoption curve to not just data mesh but each incremental data product is steeper than you expect. Try to get more people using each data product more so incremental use cases emerge. That will mean you need to do more hand-holding than you likely expect.
?Controversial?: Because that adoption curve is hard, especially across data products, your data team or data product owners might need to do more connective tissue work with other data products to enable people to more easily consume across domains. Always look to drive more and deeper engagement to create a more data-driven culture.
?Controversial?: When thinking about interconnectivity between data products, do you have the consumers create their own bridges or do you pull some of the data from one product into another? Will that create duplicated data and is that the worst thing? What is needed today for where you are in your journey to a truly data driven culture?
Sometimes you will probably build something that is pretty customized to a use case and you can't adjust your data product easily to fit a new use case, especially a use case combining data across multiple products. See if you can push some of your data into another data product to again make things easy for consumers.
!Controversial!: It can make sense to focus more on driving adoption than creating every data product to be highly reusable. If there isn't demand for data and/or if consumers aren't actually leveraging data, a highly reusable product will still not see much usage. People using the data is your data-driven Northstar metric.
Similarly, it's okay to take on data tech debt around your data products if it's a conscious decision. You understand what the work will be to make it - the data product or how it fits in your broader mesh - more scalable in the future and you made trade-offs now.
!Controversial!: duplication of work is better than not having the data available to consumers. It's okay to have the same data in multiple places for now as we crawl or walk. When we're running (or even flying!) we won't have the duplication. Scott note: how does this factor in to MDM?
Providing a lot of data analysis capabilities and flexibility to people who aren't confident in their data abilities can overwhelm them. Potentially look to offer them more static views/dashboards early as they get comfortable learning to consume analytics-driven insights.
You should want your data analysts to move beyond the work of pulling reports and answering basic questions but you might have to leverage them as change agents to teach others to do that. Would be great to not have to use them that way but you probably will need to have some group of people on the ground helping people up their data fluency. A YouTube training course will not be enough :)
When you think of managing data as a product, that means data is part of the product too. So managing the data should be part of the general product management/software development responsibilities of the domain. But that's an advanced state - you almost certainly won't start there early in your mesh journey.
When your data product team is too separated from the rest of the domain or the general product aspects of your company, there is a risk of that team becoming siloed and not really understanding the domain to the level they should.
When you start to build out your foundational data products, your data products might look more like data than products but you want to move more and more towards that product mindset and output if you can.
?Controversial?: When building out a data product team, the first hire should be someone with data engineering capabilities. Scott note: right now, strong agree. If we can get tooling/platforms to a better place, maybe that changes.
Because of the lack of a really strong UX, data products can't really drive demand in-and-of-themselves. Therefore, to see good usage that drives strong value, you need that really engaged stakeholder. It's incredibly difficult - impossible? - to create a data product so amazing, people flock to it simply because they have to use it like some consumer products.
?Controversial?: Crawl -> Walk -> Run. Build a solid foundation for your mesh in the long run. Scott note: this would be compared to only building to specific use cases. Should you build out a strong foundation of data products to support many use cases or only those use cases you are targeting?

Amy started the conversation around measurement in data. To become data driven, you have to actually measure things. It doesn't have to be the perfect measure but you need to track and understand your progress. In fact, it almost certainly won't be a perfect measurement and your measurements will evolve over time too as you want to focus on different aspects of a data product or mesh journey. In her own journey to get back in shape, she focused early on exercise quantity over quality because it was about forming the habit. After she saw success there, she started to focus more on performance and speed of her runs.

It's also important in Amy's mind to understand what are you actually trying to achieve in a certain phase of any implementation. For data products, there is the development phase, once it goes into production, maturity stage 1, etc. What looks like success or is important will change and evolve in each phase and as you progress in your mesh journey. That evolution is not driven only by that your measurement framework gets better, it's that what you want to even focus on changes. So, starting with number of users of a dashboard was a good initial metric = it was early phase, relatively easy to track, etc. So as a data product matures, what you track should change.

For Amy, even in a more data literate environment, it can be a pretty hard adoption/learning curve for consumers with each individual data product. So, you want to try to get more users early in a data product's life to encourage developing new use cases. But people don't just generally start using a new data product so someone needs to do more handholding and work than just creating the data product. Part of that might be the data team or data product owners doing more of the interconnection work between data products so users can understand and be confident - otherwise, they will probably stick to what they already know. Focus on increasing and deepening engagement, that's what drives your culture towards being data driven.

Amy talked about how there is a bit of a push-pull when thinking about those better interconnections between data products. A product owner or manager should be thinking about how can their data product better connect to the greater data landscape inside the company. And then should that be pushing data from their data product into someone else's or pulling in data from another data product to make their data product more valuable. Find those paths of least resistance for consumers to get useful information to drive increased adoption.

For Amy, there is an interesting challenge when it comes to balancing high reusability versus customization for data products. If you do not create the demand for data with the consumers, then even the most reusable data product won't see a lot of use. So they focused more on customizing data products to use cases than is probably general practice in data mesh. But that meant the users had an easier time adopting and leveraging the data products. You can't create too many highly customized solutions in the long run, you just can't support that many data products. But data products that are perfect in theory with very low consumption is probably far worse than a situation where you will need to refactor in the future but you have people eagerly consuming data. It was still manageable to use this approach quite far into their mesh journey.

At Vista, there was some duplication of work in Amy's view, where multiple data products might have some of the same data. But, she viewed that as a conscious choice and that it's better to have the data more easy to consume than to really put the hammer down on not have duplication of data. Again, they really focused on driving usage with an eye on improving things later. Scott note: this is an interesting note and balance to strike. I honestly don't know how I feel. If you're aware and it's communicated and people understand, as long as it's not causing issues, I guess it's okay?

One key metric for Amy along the early drive to becoming data-driven was how often people were self-serving their analytics. Essentially, how often were they going and looking at dashboards or outputs of some kind to make their decisions rather than leveraging a data analyst to answer questions for them or worse, waiting for a data analyst to come tell them the insights. It can be hard to measure specifically but you can get a decent idea of momentum around that kind of metric.

A really interesting insight Amy had was that when they first started trying to work with the broader organization, many of the people were not yet confident with their analytical capabilities and providing them a ton of flexibility around their analytics dashboards overwhelmed them. So, instead, they started by providing more static views to let people learn how to consume data and analytical information before giving them more advanced self-service tooling. And to get the general business users more comfortable, they paired those business folks up with data analysts to walk them through what data existed and how they could query it and leverage it. It let the business users get comfortable in the shallow end, not throwing them in the deep end.

Amy talked about how the role of the data analyst might not be the most glorious when you are in a transition phase. They were using data analysts as change agents, helping to train the business people to get better and better with data. It would be great if the data analysts didn't have to do that but there is that necessary transition period where people are improving their data fluency. After getting the general populace to a better data fluency, the analysts could focus more on higher-value, more in-depth analysis.

When asked about how a data product team should look and function as part of the greater organization, Amy talked about how an end-state, highly data-driven organization might look versus how you will probably start out. If we want to manage data as a product, we have to think of data management and generation at the domain level as just part of the product function/software development. That may still be a separate data product team within the domain but it's not a completely separate unit/function. But when you are starting your data mesh journey, you almost certainly will not be at that capability level from a platform standpoint or general developers being that data literate - there will probably be a need for a separate data product team that is led more from the data side than the product side. But with that setup, beware of silos even within the same domain, try to make sure the data product teams are integrated into the work of the domain.

Amy talked about the number one most important aspect to a successful data product or use case: an engaged consumer stakeholder that is strongly aligned. It's very hard to manufacture demand by creating the most attractive data product in the world. You want a stakeholder who is helping drive the development decisions and is chomping at the bit to get the data. Don't let the data product teams develop in isolation either. Make sure they are iterating with the stakeholder.

When building out your data mesh, Amy recommends focusing a lot on the foundation first. You will be building out some unsexy data products, really the core of what you'll build on later, as the early part of your journey. But that means you can focus more and more on building to value later. Of course you need the buy-in, funding, and momentum to go this route. Crawl, then walk, then run.

Quick tidbit:

When building out a data product team, Amy recommends your first hire being a data engineer. They are the ones who "build the rough plumbing", the pipes in the walls and without that, the water doesn't move anywhere throughout the house, in or out.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 1, 2023 Oct 01, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#255 Zhamak's Corner 28 - Generative AI and Data Mesh: The Start of a Long Road Sep 28, 2023

Takeaways:

This is just scratching the surface of generative AI and data mesh - we will have much deeper discussions in future episodes.
Zhamak believes generative AI has a ton of positive real world potential, especially in data mesh. Scott is more skeptical. But if things like GenAI are only able to be leveraged by a few large companies trying to collect as much information - especially sensitive information - as possible, there are some big potential societal issues that might come from that. We need to democratize the ability to leverage these types of tools.
ChatGPT set off a frenzy. It can be easy to want to move incredibly fast towards implementing generative AI. But companies don't have the vast amount of data where they can throw moderate - or worse - quality data and get something useful out. Garbage in, garbage out is a real concern.
Because they have less data than essentially the sum of the internet like OpenAI used for ChatGPT, companies need to focus on providing quality data into an LLM (large language model) in order for it to actually provide good results. Again, otherwise it is garbage in, garbage out.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#254 Easing Into a Data Mesh Journey - Ocean Spray's Pre-Data Mesh Preparations - Interview w/ Paul Cavacas Sep 24, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Paul's LinkedIn: https://www.linkedin.com/in/paul-cavacas-32a36158/

In this episode, Scott interviewed Paul Cavacas, Senior Manager of Data and Analytics at Ocean Spray.

Quick note before jumping in: Ocean Spray is just at the beginning of their journey - in their pre-implementation phase - and there hasn't been a lot of resistance yet internally. That might make a few people jealous 😅 but there's a lot of interesting things Paul is doing to ensure that they are ready to decentralize what makes sense to decentralize at the right time. There is a lot to be gained from not rushing in. Also, apologies that Scott's audio is a bit weird, he had yet to build his makeshift sound studio in the Netherlands.

Some key takeaways/thoughts from Paul's point of view:

As many have stated, asking the data team - especially one person - to become an expert on many different areas of the business just to complete data work for a project just won't scale. At best it creates incredibly concentrated tribal knowledge. Use this point to drive buy-in for decentralizing data ownership.
Having someone who really knows your internal IT application landscape well can really help in choosing which initial teams to start with for a data mesh implementation. That person already has good relationships and a deep understanding of your operational plane so you can pick good problem areas and partners.
Similarly, build your early buy-in momentum with people that are more likely to be excited to participate in a data mesh implementation. You don't need to convince the most difficult teams to participate at the start.
Central ownership isn't necessarily bad until things stop scaling. Having that central ownership means less flexibility and agility to react quickly to market changes or opportunities but also less cognitive load on teams. It's a trade-off.
Many of your domains really won't understand data ownership. Find ways to slowly transition them into understanding what ownership entails e.g. starting with documentation and SLOs. What data are they sharing and why does it matter? This isn't going to happen overnight.
If you aren't building overly complex data products, look to find people within the domain that are somewhat technically savvy - especially if they want to advance their careers - and start to prepare them for data ownership. Those might be your data product developers or data product managers. Scott note: Brian McMillan talked about a plan to do that in episode #26.
?Controversial?: Even if you aren't looking to move fast with your data mesh implementation, look to put at least a proto platform in place so people can at least test out ideas if they want to move more quickly.
Setting up your testing and data contract framework will probably be a big challenge. Talk to people about what they need and iterate towards it. But it will be difficult to figure out at the start and you'll start with something sub-optimal. Be prepared and go forward anyway :)
Focus more on finding the "right" partners - those willing to really partner and learn and try - rather than the "right" domain that has the most valuable data or use case. This comes up in many episodes.
If you are looking for the right partners, there are good signals to watch for such as who presents their internal results with more advanced data graphics because they are likely a domain that really cares about the data and want to dig in more.
Consider how much need there is to decentralize certain aspects and if there is enough capacity and/or work for a domain. Those might sound different but if there isn't a ton of data work, that's a lot of effort for a domain to learn to own data for a small amount of data work. Scott note: Yushin Son at JPMC mentioned they are creating a central team for small domains that can't justify a full data product team in the panel episode #233.
When creating a satisfaction scoring system for data products, consider what is the goal of the satisfaction score and how long that feedback is useful. So when a data product is in beta, harsh but useful scores are valuable but a 2 on data quality before v1.0 might not be relevant once it is actually a released product. It's okay to wipe past scores clean.
?Controversial?: When looking at your first data products, you don't want to lift and shift existing data assets to your platform. But, you can go after existing use cases as your early products if you are providing the same thing in a better/easier format.
Don't let perfect get in the way of good/progress. It's okay to put in solutions that won't work in the long run as long as you acknowledge they will need to be replaced and give yourself the room to replace them. It's taking on technical debt through a conscious choice and not locking yourself in.
?Controversial?: When you are getting started, make sure to work with a few different areas to prevent yourself from delivering a solution, even at the mesh level, that is overly targeted to one domain's needs.

Paul started off with a bit about why they are headed down the data mesh path. For a large internal project, Paul had to become an expert on so many aspects of the company and that's just not scalable in the long-term or if he's on vacation. So, he's started to decentralize the data capabilities - slowly - as teams understand what data they will likely need to own in the long-run. And he's not playing data ownership 'hot potato', he's making sure they are prepared in the right ways.

At Ocean Spray, Paul shared that until recently everything tech - including data - was very centrally owned. In some areas, the IT team knew possibly more about the business processes than even the business people in those domains. So the company is going through all of their software and applications to decide how that should look in the future. Data mesh plays well into that rethink because central ownership scales until it doesn't and limits flexibility.

As there isn't a rushed timeline, Paul has been able to put together a complete idea of what the data mesh roadmap will look like. But he also understands that it could look completely different as he learns more and starts trying to actually implement different aspects. There are some existing data sets/assets out there that could pretty easily become data products in the right environment so that is where they are targeting first. They are working with the teams to transfer some part of the ownership, especially around documentation of use cases and SLOs. The central team is pairing to take existing data assets, decompose those into their data products and help people get on the path to real ownership.

Paul recommends what Brian McMillan talked about in depth in episode #26: finding people within domains that are at least somewhat tech savvy and want to advance their careers. Work with them to get them more and more up to speed. Ownership is not something that gets transferred in a day - treat it with more respect than that. So that's finding receptive people inside the receptive domains. Yes, it won't always be easy but why make the buy-in complicated at the start if you don't need to?

Right now, Paul is building out some of the technical underpinnings of the platform they plan to build. If there are teams that want to move more quickly, they can start to test things out now. As long as those teams understand things aren't fully automated and they may have to change things about what they build now when the company starts to fully move to data products. One big piece he is anticipating is the need for testing and data contract mechanisms. But exactly how to do that is still a challenge and will be learned along the way. He's anticipating a workable but not perfect solution to start. Build to useful and then improve.

Paul circled back on the idea of finding the right partners over the right use cases/domains. Having engaged and excited partners, who know you can up their own data capabilities and drive value for them too, will make your early journey far easier than going for the most "valuable" data. You are also likely to get better feedback because they are bought in to collaborating with you! To find those partners, potentially look at how teams present their results internally. If they are presenting with lots of advanced figures and almost a flair around data, that is great sign.

How much data ownership/work gets decentralized and when is a key remaining question for Paul. He's aware that he'll have to test what works and iterate as he learns but there are plenty of domains that are too small to justify them learning a ton about how to own data when there just isn't that much data/data work to deal with. There will be a shared ownership model between the central team and the domains. Scott note: this works up to a certain scale and in certain types of organizations. Shared ownership in a very large organization rarely works that well for all that long - too much political infighting and challenges but it's an interesting pattern for smaller orgs that seems to be working well.

Paul's plan for assessing the quality of data products is to create a rubric scoring system - asking people to rate them across multiple dimensions like usability, data quality, SLA compliance, etc. And that the scores or how they are measured may change across time. At the start of a data product's life, when it's still in beta, those scores can be invaluable to iterate towards value but then consider throwing the historical scores out once it hits that v1.0. That's because there is a useful aspect of feedback depending on what you are trying to achieve and bad historical scores could hinder the success of a data product when it is now very high quality and valuable.

For Ocean Spray, their first few data products are going to be source aligned, combining a lot of important sales information. That way, those people who want raw data can still get at it but then they can build out more and more views/data products for the users on top of those. That way, there is still the scalable/productized underlying production of the raw data and then more fit-for-purpose outputs for the different users.

Paul is not letting perfect get in the way of progress. Data contracts have to get to a place where we aren't locked onto schemas as something that can never change. But no one has really come out with a better solution yet, so that's what he's doing to start. It's better than nothing so go with it while you figure out better ways.

Paul finished with a bit of advice around working with a few domains at the start of your journey. That way, you can take the learnings and understand the needs from multiple domains to abstract to a better solution for the organization rather than one overly tied to one domain's needs. Scott note: people seem pretty 50/50 split on working with one domain or 2-3 at the start of your journey. It's an interesting question.

Other unique factors of Ocean Spray:

The corporate structure is a co-op of growers so there isn't some massive pressure to grow at all costs.

Domains have been able to get access to other domains' data relatively easily for a long, long time. It hasn't been cleaned and prepared for them but there is an existing culture of sharing.

They are moving more and more to 3rd party applications rather than custom-built, which means data isn’t necessarily in an easy to consume format by default. (maybe not all that unique?)

Because many domains are quite small, the central team will likely still own most if not all of the data work for those domains.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 24, 2023 Sep 24, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#253 Data Mesh Implementation Success Metrics - Data Quality - Mesh Musings 53 Sep 21, 2023

Key takeaways:

As mentioned the last two times, at the start, it's more important to start measuring something than it is to measure the right things. Do NOT let analysis paralysis hold you back. Start measuring early to figure out what actually matters and that will also change over time.
Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring.
Use fitness functions. Episode #95 with Dave Colls covers a lot on this.
Data mesh really is a journey and your success measurement will be too. You will need to find small and simple ways to measure. Don't get bogged down. Your measurements will be rough and kinda depressing with the amount of challenges to tackle at the start. Just understand this is about how well you are doing, not how complete you are - there is always more to do!
Reflect back on how far you've come, we often forget to do that!
When it comes to data quality measurement at the implementation level, you need to think about what are you trying to accomplish. Many people go down the wrong path of trying to measure quality in a vacuum. It's about what are the expectations and why do we care about quality - to improve our decision making around data and to improve trust so more people feel they can rely on data. It's that simple. Now, measuring how well you are achieving those gets a bit harder… :D
So, what to measure or consider how to measure regarding data quality at the implementation level: how often are people in compliance with their quality SLAs, whatever those SLAs may be? How quickly are you detecting and resolving/recovering from incidents? How many incidents are you having and what is their severity? Who is actually discovering the issues - are there automated detections and is it the producer or consumers discovering them? How do you actually think about trust and the impact of trust on the success of your implementation? How do you measure and increase trust levels? How does that impact value creation? And finally, what is the quality of your metadata?

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#252 Designing and Building a Better Data Governance Approach - Interview w/ Lauren Maffeo Sep 17, 2023

Use code DATAGOV23 for 35% off ebook copies of Designing Data Governance from the Ground Up here: https://pragprog.com/titles/lmmlops/designing-data-governance-from-the-ground-up/

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Lauren's LinkedIn: https://www.linkedin.com/in/laurenmaffeo/

Designing Data Governance from the Ground Up (Lauren's book): https://pragprog.com/titles/lmmlops/designing-data-governance-from-the-ground-up/

In this episode, Scott interviewed Lauren Maffeo, author of the book Designing Data Governance from the Ground Up and adjunct Lecturer at George Washington University. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Lauren's point of view:

In governance, a very easy way to go down a bad path is to not automate your standards. Making governance an easy aspect of data work will go a long, long way.
?Controversial?: As an industry in general, data governance maturity is still at the infancy phase. And the pace of maturation is far lower than other aspects of software like security.
The majority of organizations are not mature enough with data governance to get a lot of value from things like ML or NLP.
Data governance best practices are hard to come by. There isn't really even a large community specific to data governance for people to easily exchange ideas.
If the number one cause of cybersecurity breaches is employees, is the number one cause of bad decisions made on data because of A) bad data or B) employees not being data fluent enough to make the right decisions? Does the second contribute to the first?
"You really have to embed data literacy into very strategic ways of communicating with the organization and educating them that way. Without that approach, I think very little progress can actually be made."
?Controversial?: It's not that hard to teach people to be relatively data fluent. Not to a data engineer level but investing a relatively small amount in your people can get them to a level they can at least typically spot low quality data.
?Controversial?: You can't just "do data governance later" and have a good outcome. Scott note: this absolutely shouldn't be controversial but still seems to be a common pattern.
If we actually want to treat data as a product, that includes practices around data too. A product manager couldn't decide to put out their first roadmap three months after the product is shipped.
It's hard to measure the ROI on data governance work but it's pretty clear that it's necessary when your data work is not driving results.
Despite governance being crucial, you can still roll governance out in a measured manner. You can create ways for people to get used to the idea before rolling it out and it isn't as if day 1 'everything changes!', it can be a gradual process that adds value, not (many) complications.
Embracing a fail fast culture in data will probably be hard but it's absolutely something everyone should do. Make the 'failures' small and contained - limit the blast radius - but still failing fast "the essence of innovation in tech."
Look to leverage sandbox environments to try out new approaches to governance and get data stewards used to anything new you will be asking of them. A new policy or tool doesn't have to be a sudden light switch flip.
"We also can't afford for leaders of any department to not know what quality data looks like for their teams, because their success, the success of their teams depends on having quality data that their customers trust."
?Controversial?: Data mesh is a risky endeavor, do it in a small-scale way and keep it very cost contained as you test if it will work in your organization.
For driving data literacy/fluency, gamification really can work. Look to make learning about data a fun and gamified experience.

Lauren started with a somewhat common refrain for this podcast: the pace of data practice maturation - around governance and data practices as a whole - is just not keeping pace with innovation in other aspects of software. Even the pace of conversation is not maturing as fast. Cybersecurity is maturing very quickly for example but we're just not seeing that in data. So companies are just not ready to really derive a lot of value from things like machine learning (ML) or natural language processing (NLP).

One of the big issues around industry maturity and data governance for Lauren is that there isn't even a large community around the topic. So there isn't a larger cohesive conversation around data governance best practices. Scott note: it's really hard to have a broader conversation too because approaches and practices vary widely and data governance has about 15 varied subtopics that each deserve their own focus rather than being lumped under a huge umbrella of 'other' that governance has become.

Most cybersecurity breaches, Lauren pointed out, are caused by internal employees making a mistake. So how do we think about that relative to data? Is that people creating low quality data and/or is it people not being data fluent enough to actually make good decisions based on data? In cybersecurity, there is a big emphasis on training people to see what an attack looks like - should we take the same approach in data regarding bad quality data? "You really have to embed data literacy into very strategic ways of communicating with the organization and educating them that way. Without that approach, I think very little progress can actually be made."

Lauren talked about how few companies are really going broad with their data literacy programs, training up a large number of their employees. There is a lot of talk about that as part of data governance programs but few are walking the walk. And she believes it's not that hard to get people to a relatively data fluent level - understanding SQL, being able to more easily spot low quality data, etc.

"We'll do the data governance later," is something Lauren has seen and heard in conversations. Governance is seen as something that can be layered on like a coat of paint at the end of a car being manufactured. But because good governance is intrinsic to data quality and matching to the actual business use case, trying to do it later rarely leads to good results.

When asked about selling the return on investment of data governance work, Lauren admitted that it's often quite nebulous but data governance is so key that people know they need it despite not being super clear on the specific value of the work. And you can roll out your data governance tech, policies, and processes at a reasonable pace, creating some definitions and a sandbox to show people how it will work. She is really big on the idea of a sandbox to get people used to new governance practices and tech. It isn't as though everything changes suddenly, it's that you're working towards better data practices that will drive value for the organization. Fail fast is "the essence of innovation in tech" so we need to embrace it far more - but still safely and sanely - in data.

"We also can't afford for leaders of any department to not know what quality data looks like for their teams, because their success, the success of their teams depends on having quality data that their customers trust," Lauren said. So we all need to be in this together and have domains really owning and understanding their data. That can't be on a central data team.

Gamification is one thing Lauren is seeing work for improving data literacy/fluency. It is a great pathway to creating a data-driven culture. Make it fun and give out rewards :)

Lauren wrapped on a simple message. Automate your standards. It is easy to have your tech and standards/processes quickly lose alignment if you aren't making things easy for people via automation.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 17, 2023 Sep 17, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #130 Making the Data Quantum Leap - Starting from the Data Quantum at PayPal - Interview w/ Jean-Georges Perrin (JGP) Sep 14, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #130 with my partner in our weekly data mesh roundtables Jean-Georges Perrin. There are a lot of interesting things to take away from this. A biggie is to have an early thesis about what to drive towards - what will drive value early? Doing data mesh doesn't simply create value. And you need to build momentum. There's a lot here to learn about how to apply good software engineering practices to data with data mesh.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Data Mesh at PayPal blog post: https://medium.com/paypal-tech/the-next-generation-of-data-platforms-is-the-data-mesh-b7df4b825522

JGP's All Things Open talk (free virtual registration): https://2022.allthingsopen.org/sessions/building-a-data-mesh-with-open-source-technologies/

JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/

JGP's Twitter: @jgperrin / https://twitter.com/jgperrin

JGP's YouTube: https://www.youtube.com/c/JeanGeorgesPerrin

JGP's Website: https://jgp.ai/

In this episode, Scott interviewed Jean-Georges Perrin AKA JGP, Intelligence Platform Lead at PayPal. JGP is probably the first guest to lean into using "data quantum" instead of "data product". JGP did want to emphasize that as of now, he was only discussing the implementation for his team the GCSC IA (Global Credit Risk, Seller Risk, Collections Intelligence Automation) within PayPal.

Some key takeaways/thoughts from JGP's point of view:

Data mesh as it's been laid out by Zhamak obviously leaves a lot of room for innovation. For some, that's great. For others, they want the blueprint. And it's okay to wait for the blueprint. But JGP and team are excited to innovate!
PayPal's 3 main initial target outcomes from data mesh: A) faster and easier data discovery, B) easier to use the data in a governed way, and C) increase data consumer trust in data.
PayPal's initial data consumers are data scientists so their platform and data quanta are built to serve that audience first.
Really consider what you want to prove out in your MVP. Is that minimum viable A) data quantum, B) data platform, C) data mesh, or D) something else? Only doing a data quantum probably sets you up for trouble and a platform only won't be tested until it has data quanta on it.
Data contracts are crucial to making trustability actually measurable and agreed upon. Otherwise, it's far too easy to have miscommunication between data producers and consumers, which leads to lack/loss of trust.
Producers, don't set your data contract terms too strictly when first launching a data quantum. There's no need to over-engineer - despite how interesting that can sometimes be...
For too long, we have tried to keep software engineering and data engineering overly separate. They are both just engineering with slightly different focus and data mesh really leans in to that.
We've also tried to keep operational and analytical far too separate. We should look to build out tooling where data can live that serves both operational and analytical workload needs. But we aren't there yet.
Analytical APIs, at least as far as we've seen them to date, are just not going to do what we need relative to accessing data from data products/quanta according to JGP.
Standardizing metadata access APIs across data quanta has made it very simple for data consumers to begin using new data quanta as they are introduced to the mesh. PayPal has observability, discovery, and control APIs.
Domain is an overloaded word. It can mean a very large high-level domain like Sales, Finance, or Marketing with hundreds or thousands of people in it or it can mean a smaller, 'two pizza team' level scale.
PayPal is doing only one data quantum per domain but that domain is really at the two pizza team scale - they aren't trying to have a single data quantum for Marketing.
It's crucial to understand data quanta and the use cases they power both have life-cycles. So really applying product thinking is crucial.
Most data engineering teams do work in a waterfall approach and that just doesn't scale well. However, moving to data mesh can mean additional cognitive load as it really requires an Agile mindset to do right and that shift in the ways of working is not trivial.
It's good to have smaller delivery requirements so you get faster feedback on what you are creating - a core tenet of Agile. Don't try to deliver everything all at once. Get it in users' hands early to get feedback.

JGP started the conversation talking about how in his team, he's really leaning into the idea that software engineering and data engineering are not that different. Zhamak has discussed this too. We should focus on sharing practices so we all create better software and infrastructure. For JGP, data engineering work in most organizations has followed a very waterfall approach. However, his team has been mostly working in an Agile manner. Therefore it wasn't a huge switch to their ways of working - like it is at many organizations - once they started doing data mesh. And luckily, there was already an appetite for changing the way they were tackling data challenges.

In the spirit of being agile and capital A Agile as well, PayPal set out on their data mesh journey. They wanted to do an MVP but what was the P? Minimum Viable Data Product/Quantum? Minimum Viable Platform? Both? Minimum Viable Mesh? JGP recommends looking at what you want to deliver as a minimum unit of value. PayPal already had extensive data platform expertise so they were able to focus on delivering data products/quanta (plural of data quantum) but they worked in parallel to build out their initial data quantum and mesh capabilities. As many guests have noted, it's dangerous to only do a minimum viable data product/quanta.

PayPal has been building data platforms for a long time. As mentioned by JGP, they were one of the pioneers of the self-service data platform concept. But data mesh offered a path to faster and easier data discovery, to making it easier to use data in a governed way, and to increased trust in data by the data consumers - their first consumers being data scientists. A big benefit of addressing those needs is those data scientists are able to better tell if the data they access is the right data for their use case.

One thing JGP emphasized that's significantly helping PayPal move forward is standardizing APIs across data quanta. Those are not data access - or analytical - APIs as JGP thinks those will just never work all that well. Instead, as their audience is data scientists only to start, everything anyone needs other than the actual 1s and 0s of the data is accessible via Python APIs. The metadata, the observability/trust data, etc. Then, the data scientists use notebooks to work with the data. But standard APIs means data consumers only have to learn one interface. This is similar in concept to what many are doing with data marketplaces - one standardized way to interact with the information about the data quanta.

PayPal is using the terms data product and data quantum as two separate things. A data product is simply a product powered by data and analytics. Those have been around for quite some time. But PayPal is looking at data quanta like side cars, used specifically to power more and more of their data products going forward.

PayPal have invested heavily in making data contracts work well per JGP and earlier PayPal guest Jay Sen. They've been building APIs to make it far easier to consume data contracts as people learn more about a data quantum. And as mentioned before, they can consume observability metrics via API as well. When asked about how are they setting their actual contractual terms, the data producers initially put out some contractual terms and then may adjust those terms as data consumers request. It's important for data producers to not set their data contract obligations too strictly unless there is a user-based need.

JGP made the good and often unspoken point: the term domain has lost a lot of its meaning. It can mean a very high-level domain like Marketing, Finance, Sales, or HR. Even in software companies, a domain could be Product. But at PayPal, they are being quite strict about what they mean for domain in data mesh: it is a small scale sub-domain - think two pizza team size - and they enforcing a strict 1:1 relationship of one data quantum per domain; and of course, not cross domain source data quanta too. That way, each small domain can focus on creating a great data quantum instead of worrying too much about how big each data quantum should be. The scope should never get that huge at a two pizza team size.

Back to APIs, PayPal is implementing an API-first approach. APIs for the data quantum control plane, observability APIs, and data discovery APIs. It's the preferred way of working for their initial consumers - data scientists. However, as mentioned previously, JGP does not believe analytical APIs - that is APIs designed to do things like filtering and returning many hundreds to thousands or more results - are really feasible. Definitely not now and possibly ever. So APIs are great for getting at the metadata but not the data for analytical use in his view.

JGP wrapped up in sharing how our tooling must evolve so we don't have to think about such a hard wall between analytical and operational. There will always be analytical and operational workloads but our systems can evolve to support both. We aren't there yet though.

Quick tidbit:

If you are just delivering data, the 1s and 0s, you are not delivering the necessary trust.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #65 What's a Data Contract Between Friends - Setting Expectations with Data Contracts - Interview w/ Abe Gong Sep 12, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #65 with Abe Gong all about how people are implementing data contracts in the wild. There are so many ways people can just do only defensive data contracts and I think that is such a missed opportunity. Maybe it's where you will have to start but there's a much better way and we talk a bit about why I think that is so distressing that people aren't talking to each other.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Abe's Twitter: @AbeGong / https://twitter.com/AbeGong

Abe's LinkedIn: https://www.linkedin.com/in/abe-gong-8a77034/

Great Expectations Community Page: https://greatexpectations.io/community

In this episode, Scott interviewed Abe Gong, the co-creator Great Expectations (an open source data quality / monitoring / observability tool) and co-founder/CEO of Superconductive.

One caveat before jumping in is that Abe is passionate about the topic and has created tooling to help address it. So try to view Abe's discussion of Great Expectations as an approach rather than a commercial for the project/product.

To start the conversation, Abe shared some of his background experience living the pain of unexpected upstream data changes causing data chaos / lots of work to recover from and adapt. Part of where we need to get to using something like data contracts is to remove the need to recover in addition to adapting and move towards controlled/expected adaptation. Abe believes that the best framing for data contracts is to think about them as a set of expectations.

To define expectations here, this would include not just schema but also the content of data, such as value ranges/types/distributions/relationships across tables/etc. So for instance, a column may be a one to five for rankings and then the application team changes it one to 10. The schema may not be broken - it is still passing whole numbers - but the new range is not within expectations so the contract is broken.

At current, Abe sees the best way to not break social expectations is via getting consumers and producers in a meeting to talk about the upcoming changes and prepare, such as with versioning. But, as tooling improves, Abe sees a world where we won't even need a lot of those meetings going forward - either because data pipelines can be "self-healing" and automatically adapt to changes upstream or because metadata and tools for context-sharing will reduce the need for meetings.

Abe sees two distinct use cases in general for data contracts or more specifically how people are using Great Expectations to implement data contracts. The first is purely defensively - put some validation on the data you are ingesting to prevent data that doesn't match from blowing up your own work; the second type is when the consuming team shares their expectations with the producers and there is a more formal agreement - or contract - with a shared set of expectations. The first often leads to the second, via an agreement conversation that happens after there was an upstream breaking change.

Abe also mentioned there is a third constituent on data contracts in the room: the data. Sometimes the consumers and producers may agree on what they expect, but if that’s different than what’s in the actual data, then it's hard or dangerous to move forward. The data has a veto.

There was an interesting discussion on the push versus pull of data contracts - should the producer team create an all-encompassing contract or should we have consumer-driven contracts? Would producer-driven contracts be too restrictive, preventing the serendipity insights data mesh aims to produce? Would consumer-driven contracts mean multiple contracts for each data product that the producer agrees to? Is that sustainable?

So, to sum it up, the idea of a set of explicit expectations around a data product that are the result of collaboration between producers and consumers sounds like where we should all head if possible. If the expectation set is only coming from the producer side, it might be overly restrictive and miss a lot of the nuance necessary to actually create consumer trust. And exclusively consumer-driven contracts don't sound sustainable or scalable.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #48 Overcoming Obstinate Organizational Obstacles in Data Mesh - Interview w/ Scott Hawkins Sep 10, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Scott Hawkins' LinkedIn: https://www.linkedin.com/in/scott-hawkins-8934393/

In this episode, Scott interviewed Scott Hawkins, Principal Data Architect at ITV.

Scott views data mesh as a mechanism for change. Your company culture and your understanding of it are crucial to establishing data mesh well, driving that buy-in. Ask yourself: what challenges does data mesh actually address and hopefully solve, how will it impact the business not the tech, and what does it change. Your organization might not be ready for data mesh. Or a specific domain might not be ready. And that's okay!

As ITV moved forward, they found a "good enough" solution via a global ID. It's not perfect, there might be some overlap - such as one person might have a different global ID for their online subscription versus their broadcast subscription - but it is far better than what they were doing. And it allows for interoperability/joins across the data. This is a big improvement - don't let perfect be the enemy of good or done.

One thing working for ITV is deploying a "team-in-a-box" to help domains move forward - similar to an internal consulting team. Each situation is different so each box they are given is different. The team-in-a-box concept also means it is somewhat easier to build common best practices internally. Coming to the table with defaults has really helped ITV.

Per Scott, there are 3 good ways to drive buy-in for the domain teams:

At the senior level - so it trickles down as the management for the team is bought in.
Via a strong carrot - solve a problem for them as a kind of quid quo pro / mutually beneficial solution. Trying to solve an unrelated problem will drive lower buy-in.
Work on realigning the team KPIs/OKRs with the senior leaders to actually realign incentives.

Continuing on the driving buy-in, Scott recommends working with the domain managers to generate a viable/valuable carrot for the entire team. Explain to those leaders why it matters, work with the leaders to revamp the KPIs if the KPIs are getting in the way of delivering a good data product. This is why exec-level buy-in is so crucial - it is pretty hard to start modifying team KPIs/OKRs without it! Talking to teams to understand who they (the individual and the group) operate is crucial to developing the right path for them.

Scott also talked about making failure an option. You can try to work with a domain and if it isn't working, it's okay to move on. You don't need to get everyone onboard on day one or sharing their data on day one. If you design incentives well, people will want to participate eventually. Until then, it's okay to walk away from that team.

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 48 with Scott Hawkins. I thought this episode was crucial because it is foundational to understanding a lot of good techniques and practices for getting lots of the organizational pieces of data mesh right. You will learn a ton on this one!

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 10, 2023 Sep 10, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #150 3 Years in, Data Mesh at eDreams: Small Data Products, Consumer Burden, and Iterating to Success, Oh My! - Interview w/ Carlos Saona Sep 07, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 150 with Carlos Saona. eDreams' approach is very unique and interesting because it was essentially all on its own so there are a ton of useful learnings to consider if they are the right fit for your own organizations.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/

In this episode, Scott interviewed Carlos Saona, Chief Architect at eDreams ODIGEO.

As a caveat before jumping in, Carlos believes it's too hard to say their experience or learnings will apply to everyone or that he necessarily recommends anything they have done specifically but he has learned a lot of very interesting things to date. Keep that perspective in mind when reading this summary.

Some key takeaways/thoughts from Carlos' point of view:

eDreams' implementation is quite unique in that they were working on it without being in contact with other data mesh implementers for most of the last 3 years - until just recently. So they have learnings from non-typical approaches that are working for them.
You should not look to create a single data model upfront. That's part of what has caused such an issue for the data warehouse - it's inflexible and doesn't really end up fitting needs. But you should look to iterate towards that standard model as you learn more and more about your use cases.
?Controversial?: Look to push as much of the burden as is reasonable onto the data consumers. That means the stitching between data products, the compute costs of consuming, etc. They get the benefit so they should be taking on the burden. Things like data quality are still on the shoulders of producers.
You should provide default values for your data product SLAs. It makes the discussion between consumers and producers far easier - is the default good enough or not?
?Extremely Controversial?: At eDreams, you cannot publish data in your data product that you are not generating. In derived domains (e.g., customer history), “generate” includes the derived stitching. NOTE: Go about an hour into the interview - not episode - for more specifics.
When starting with data mesh, there must be a settling period - consumers must understand that things are subject to change while a new producer really figures things out for the first few weeks to months.
You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic.
Be careful when selecting your initial data mesh use cases. If the use case requires a very fast time to market, while it has value, you likely won't have the time and space necessary to experiment and learn. You need to find repeatable patterns to scale in data mesh. Hurrying is a way to miss the necessary learning.
Look ahead and build ahead for obvious interoperability. E.g. create foreign keys for data products that don't exist yet but will.
Be clear about what early data mesh participation means - what will it net domains that are part of the early implementation? And be specific too about what your early implementation won't include or achieve. Don't over promise and under deliver.
Similarly, strongly emphasize that learning is a priority in your early implementation and that you are factoring in learning into promises and estimations. You can't promise you'll find the right solution to a challenge on day one, things need space to evolve as you learn more.
It's okay to not have everyone as part of your initial implementation - engagement or even buy-in - but set yourself on a path where their participation is inevitable.
Making data as a first class citizen doesn't just happen. There is incremental work to be done by the domains. Make sure you reserve time to actually do that work - the data quanta creation and maintenance.
It is not feasible to have your documentation be fully self-describing for everyone. eDreams chose to set the bar at documentation that is self-describing for readers that already know about the domain. For readers that do not know the domain, that introduction must happen somewhere else.
At the start of a data mesh journey, your central team will likely control all the use cases being served by the mesh. But at some point, self-serve needs to happen. Consumers need to be able to serve their needs without their use cases going through the central team.
With small-size data products and the data combination burden being on consumers, versioning tends to not be as much of a problem because the concepts of domain events don't change that often. Or when they do, the retention window has been typically short, making versioning easier.

To make data producers feel a better sense of ownership, 1) look for ways for producers to better leverage their own data; 2) maximize your number of consumers for their data quanta so there is a quicker time to identify issues with the data product - more eyes means more who can spot issues; and 3) create automation to easily/quickly let domains identify sources of data loss rather than searching: with proper setup, you can make it easy to identify if the data pipeline is the problem. If it's not, then the issue is in the domain.

When Carlos and team were looking at building out how to tackle their growing data challenges a few years ago, they were looking at request for proposals (RFPs) from a number of data consultancies around building out a data lake but just were not convinced it would work. Then they ran across Zhamak's first data mesh article and decided to give it a try themselves. Until more recently, Carlos was not aware of the mass upswing in hype and buzz around data mesh so their implementation is very interesting because it wasn't really influenced by other implementations.

When they were starting out, Carlos said they didn't want to try to create a single, overarching approach. It was very much about finding how to do data mesh incrementally. They started use case by use case and built it out organically, including the design principles and rules - they knew they couldn't start with a single data model for instance. But it was quite challenging iterating towards that standard data model.

When choosing their initial use cases to try for data mesh, Carlos and team had some specific criteria. They rejected anything that needed a very quick turnaround because it wouldn't let them have enough time/space to try things, learn, and iterate. They did plan ahead by creating foreign keys to data products that didn't exist to make interoperability down the road when they would exist easier. And they were very honest with stakeholders about what early participation meant - and what it didn't mean; that way, it was clear what benefits stakeholders could expect.

According to Carlos, while they had executive support and sponsorship for data mesh, that wasn't enough to move forward with confidence at the start. They needed to have a few key stakeholders that were engaged as well and wanted to participate. It was also okay to have some stakeholders not engaged but just informed of what they were trying to do with data mesh. You don't have to win everyone over before starting.

Five things Carlos thinks others embarking on a data mesh journey should really take from their learnings: 1) it's okay to not have everyone really bought in or especially engaged upfront but they will have to participate - make their eventual participation inevitable. 2) Really emphasize that you are learning in your early journey, not that you have it figured out - and factor in learning when doing estimations and promises. 3) Don't try to design your data model from the beginning; you need to learn via iteration - you will start to find your standards to make it easy to design new data products. 4) When treating data as a first class citizen, it's important to understand that will take additional time. Reserve the team's time to create and maintain their data quanta. 5) Let the use cases drive you forward and show you where to go.

Carlos' philosophy is, within reason push as much of the burden onto the consumer as you can. Obviously, we don't want consumers doing the data cleansing work - that's been one of the key issues with the data lake - but the costs of consumption should fall on the data consumers as they are the ones deriving the most benefit. So eDreams makes the consumers own stitching data products together for their queries and makes them pay for the consumption. This minimizes the costs - including maintenance costs - to producers.

One very interesting and somewhat unique - at least as far as Scott has seen - approach is how truly small Carlos and team's data quanta are. Thus far, they have really adhered to the concept that each data quantum should only be about sharing a single type of domain event and really nothing more in it. This again makes for lower complexity and maintenance costs for data producers. They are considering changes with upcoming BI-focused data products so that is to be determined.

Carlos believes - and Scott exceedingly strongly agrees - it is not feasible for your documentation for your data quanta to be fully self-describing. You can't know someone else's context. You need to write good documentation so people can still understand what the data product is and what it's trying to share but if you do not have knowledge of the domain, it would be a considerable amount of effort - essentially impossible to do it right - to fully explain the domain and how it works in the documentation of each data product. Getting to know how other domains exactly work is outside of the scope of the data mesh.

At the start of their journey, the data team was in control of all the use cases, who was consuming, and who was producing, according to Carlos. But, as they've gone wider and there is a self-service model for data consumers, more and more of the use cases are directly between the producers and consumers - or the consumers are consuming without much interaction with producers if they already know the domain. It could become an issue with people trying to understand data from lots of different domains for the sake of understanding but it hasn't been an issue so far.

To date, Carlos hasn't seen many problems around versioning. They thought they would have many more issues with versioning than they have which Carlos believes is from keeping their data products as small as possible and using domain events. When they have had versioning, the retention window for the data has been relatively short so the versioning has been relatively simple to move to the newer version. And because most people are getting their data from source-aligned data products, changes have a smaller blast radius - they won't affect data products that are downstream of a downstream of a downstream data product. Domain events have been enough because their main stakeholder has been machine learning. They are now working on a different kind of data quanta for consumers such as BI, and they plan to include more governed versioning there.

One of the biggest challenges early on according to Carlos was that domains didn't really feel the ownership over the data they shared. So to increase the feeling of ownership, they first looked for ways for producing domains to use their own data - as many other guests have mentioned. Second, they tried to maximize additional consumers of data products by looking for use cases. That led to faster feedback loops if there was a problem - more eyes on the data - so producers discovered issues sooner. And third, the platform team helped identify issues that might be in the system or in the data platform/pipeline process - if there was data loss, there is automation to help identify if it is on the platform side; if it's not on the platform side, then it is an issue with the domain. That one automation has led to a lot less time searching for the cause of data loss rather than fixing data loss.

Carlos and team built in a few different layers of governance. The first is a universal layer for standard metadata in each data product, like when something happened, who is the owner, the version of the schema, the existence of a schema, etc. These are enforced automatically by the data platform and you can't put a data product on the mesh without complying. Producers must also tag any PII or sensitive information like credit cards. Then, a second layer is policies for data contracts between producers and consumers. As many guests have suggested, they have found having default values for SLAs in data contracts provides a great starting point for discussions between data producers and consumers.

"You can have your cake and eat it too," using domain events per Carlos. You don't want direct operational path queries hitting your data quanta as they are designed for analytical queries - they will have a separate latency profile. But at eDreams, the pipeline that writes data quanta to the analytical repository is implemented with streams that can be consumed in real-time by operational consumers (microservices).

Other tidbits:

When launching a new data product, there must be a settling period - consumers must understand that things are subject to change while the producer really figures things out.

You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic.

Data products should have customized SLAs based on use cases. You don't need to optimize for everything. Let the needs drive the SLAs.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #133 Nitty Gritty From the Deployment Committee: Crucial Learnings on Driving Buy-in and Data Product Discovery - Interview w/ Ammara Gafoor Sep 05, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 133 with Ammara Gafoor. There is a ton to learn from this one and reflect back on.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

ammara.gafoor@thoughtworks.com

Ammara's LinkedIn: https://www.linkedin.com/in/ammara-gafoor/

Data Mesh in practice article series from Ammara and colleagues:

#1: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start

#2: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-organizational-operating-model

#3: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-product-thinking-and-development

#4: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-technology-and-the-architecture

In this episode, Scott interviewed Ammara Gafoor, Principal Business Analyst at Thoughtworks who has been working on a few client projects related to data mesh including one for well over a year.

Before jumping in, it's important to note that much of Ammara's learnings come from an implementation in a 100K+ employee company split into 21 high-level domains. So the definition of domain in this episode revolves around that context of a very large business unit, not a two pizza team size sub domain.

Some key takeaways/thoughts from Ammara's point of view:

There often is a hangup around data work, especially relative to data mesh, where people want to get it all right, all perfect the first time. That's never going to work. Get something decent out there, test, and iterate. Perfect is the enemy of done. No bikeshedding!
If you don't look to change domain's KPIs to align their operational work to data mesh "you won't prioritize it - you cannot prioritize it." Make it easy for domains to prioritize data mesh work if you want it to get done.
?Controversial?: source oriented data products should not be made available to business users within the domain or to almost anyone in other domains - at least by default - as they are difficult to understand for anyone other than the highly data literate people in the domain.
"Don't make things that you don't need yet." Build data products for use cases you've identified. Think of the target business outcome and work backwards to create the necessary data products.
Far too many people try to build customer 360 solutions. And they very rarely work. No one ends up happy.
Sometimes it seems like data mesh literature says get rid of many existing data roles - but the people in those roles are subject matter experts in how the organization and/or the domains do data. You want to leverage them as you evolve their roles and they can pick up new skills related to data mesh.
When driving buy-in, be prepared to repeat yourself multiple times. And then repeat yourself multiple times more... Learn to speak the language of the domain as well to drive buy-in with the business people.
Look for the 2 most obvious use cases inside a domain. Supporting those use cases, you will need 5-6 foundational source-oriented data products that will support many - most? - of the use cases for that domain. It will appear like you need many more source oriented data products but zooming out, you will see you don't need nearly that many.
Typically, using the median, use cases are powered or supplied by two consumer oriented data products and each consumer oriented data product is powered by three to four source oriented data products.
They planned for a single approach to work with all domains to start implementing and contributing to the data mesh. That proved to not work well and every domain needed it's own process to really make it all work. There are commonalities of course but there isn't a cookie cutter approach unfortunately.
It's very easy for people within the same domain to talk past each other, especially in a functionally siloed org structure. Data mesh can be a forcing function to find a shared, common language to drive to better results.
From Team Topologies, there are 3 streams: operating model, product stream, and technology stream. They should all be given equal weight. No, you can't just focus on the tech stream...
There may be push-back from both IT and the business side to having the business people collaborate closely in a data mesh implementation. But it's very important to overcome that - that collaboration with the business people is crucial to making data mesh work.
Business leaders may have - or at least believe they have - what they already need currently via shadow IT. It may be difficult to convince them to change their ways of working "for the greater good".
When looking at driving buy-in with domains, look at how they are incentivized. Often, short-term results are more rewarded than focusing on mid- to long-term results. If next year's funding is based on this year's results, they will focus on this year's results...
Typical personas in a data mesh implementation: the IT sponsor, the business owner, the "no stance/sideline", the "says yes to your face...", the product owners, and the data lake builders (aka those feeling the target on their backs).

Ammara started off the conversation sharing about how she and her team "had it all laid out" for the plan to standardize how they'd bring each domain up to speed on data mesh - from the introduction of new ways of working to being ready to participate in the data mesh implementation in just six weeks. And then reality struck. Each domain is different and much like trying to explain the benefits or implementation of data mesh, a single approach for all audiences just didn't work well so they adapted. Every domain is unique and required its own unique approach to make implementing data mesh in that domain work. There are of course some commonalities but each of the 13-14 domains that are part of the data mesh implementation thus far has had its own unique challenges.

So, Ammara shared some stories about working with different stakeholders. Often, the first stakeholder they encountered was an IT sponsor for the domain itself - either an IT leader in the domain or an IT counterpart for the domain. This persona typically wanted to bring them in and welcomed them with open arms. And while they were often bought in on data mesh, there was a push - from IT and often the business side - to only speak with IT. So Ammara and team had to work to get permission to also include the business people in the conversations about their proposed data transformation. Because without the business support and knowledge, your data mesh implementation is likely to fail. How many episodes have said tie your data strategy to your business strategy? But, the business people often have what they need currently via shadow IT. So why would they want to give that up? It's an emotional response to be asked to give up what you have for the greater good and the long-term.

There is the concept of immediate returns - you build a dashboard and there is immediate potential value - versus the mid- to longer-term returns from things like building your data platform and building out your data governance capabilities. Ammara has seen many times there is not any incentive to wait and focus on the mid- to long-term returns - if your funding this year is based on results this year, focusing on your results 2-3 years out is often doesn't feel like an option. They won't get rewarded for that long-term work. And most domains don't even have the capabilities to do said mid- to long-term high-value work. But to do data mesh right, we need to incentivize patience - and incentivize and provide the capabilities to do things right for the long-haul instead of just the short-term, low stakes wins.

According to Ammara, as part of a successful data mesh implementation, there is the technical stream - the Team Topologies meaning of work stream - but you must also work on the operational stream at the same time. And the product stream too. If you don't look to change domain's KPIs to align their operational work to data mesh "you won't prioritize it - you cannot prioritize it." You need to put a metric into place to measure progress - it doesn't even have to be a great measure! It's a way to start the conversation. There is too much of a hangup in data mesh around trying to get things perfect the first time. Get it done, measure it, iterate on it, and move forward. Don't let perfect be the enemy of done and/or good. Don't fall to bikeshedding.

The cost of change and the cost of failure in data historically have been very high per Ammara. But we have new economic models with cloud that make that no longer true. We now have "the privilege to be able to fail". Failure wasn't an option historically. But that's such a foreign concept to many, it will cause some to push-back. They have lacked the psychological safety to fail. And we have to understand why they are pushing back and work with them to understand that failure in a highly agile environment is incremental learning.

After picking the 2 most obvious use cases in a domain - again, the very large business unit concept of a domain -, Ammara believes it will reveal a 5-6 of the foundational source-aligned or "source oriented" data products of the domain that will be able to power most use cases. So just start building the MVP of those source-aligned data products because they will support other use cases down the road as well.

On Personas, Ammara laid out a few she and team have run into:

The IT sponsor - typically a Data Architect or Data/Analytics Lead; bought in to data mesh, likely after feeling the pain points as Zhamak has laid out. Trying their best to go wide on getting people bought in on data mesh and has some - but not a ton of - social capital to influence. Their social capital is more with the IT/data people and less on the business side of the domain. They are critical to get things moving.

The Business Owner - generally supportive of the data mesh initiative but doesn't have the time - or the incentive - to spend time on the data mesh implementation. You're trying to get their support by the promise of making their lives easier.

The Sideline Watcher - sees data mesh as probably 'yet another data trend'. Not pushing back but not taking a stance. Waiting for the tide to turn one way or another before making their own waves.

The "Yes to Your Face" - will say yes to you and then just go do whatever they were going to do anyway… These are inevitable - try not to take it personally.

The Product Owners - they are building the dashboards or the analytical solutions, desperate for the data. They really WANT to work with you but don't know exactly how - how can they get the resourcing and we're asking them to rethink the way they do their work. Help them figure out how they can partner where possible.

The data lake (or other historical data paradigm) builders - have spent so much time and effort to build a viable data lake/warehouse/etc. Often fight you because you're going against everything they've built. It's not personal against the data mesh team but it is personal if you put all their hard work aside. But they can build data initiatives very well, try to work with them and let them know you're building off the knowledge they've gained if not their direct work.

For Ammara, a lot of the data mesh literature and conversations feel like they say there are new roles and therefore there isn't room for many existing data roles, like the data warehouse or data lake builders/maintainers. But she thinks that's not a great idea - and Scott agrees. They are subject matter experts in how the domain's data flows and systems actually work and can be excellent guides to bringing more people into the data fold as they themselves pick up new skills. Trying to hire your way to a data mesh is not a great idea… No one is redundant, everyone has valuable knowledge for Ammara.

You need to make your IT sponsor successful in order for your data mesh implementation to go broad in that domain so that means learning the - and communicating in the - language of the business according to Ammara. That might mean you have to deal with the horror of PowerPoint Presentations. And as many guests have said, the selling points and implementation details of data mesh don't stick with the broader audience the first time. Repetition, reframing, holding of hands, etc. You won't succeed if you try to just message once. Be prepared to repeat yourself. And then repeat yourself again.

Ammara gave an example of why data mesh can really help improve communication and drive to common language. In manufacturing, there is the concept of "on time, in full delivery" as a very crucial KPI. And the domain had analytics teams constantly asking to build this for the different manufacturing lines while at the same time, the business side said they didn't have the information. How could that be when there were 10+ completed "on time, in full delivery" projects that had been funded? So once Ammara and team removed the data team from the picture, the business folks were able to talk with the regular IT team and they came to a shared, common understanding of what was actually needed and what was missing. It's pretty easy to lose sight of what the actual need and use case is when people are siloed by function.

It is crucial to understand the three streams of work model, per Ammara. The operating stream is "building the cadence for IT and business to communicate" in order to prioritize. This helps identify which data products will be built. The product stream is identifying the actual data products that need to be built, as in what are the scope and boundaries. The technical stream is about building the data product and the platform needs. Each of the three streams should have equal weighting. This is another way to think about your MVP thin slice, you must encapsulate some of each capability, each stream.

As previous guests have noted, many domains build data products that benefit themselves first in Ammara's experience. This obviously makes it easier because there is more buy-in and no cross-domain communication and prioritization friction. But that is just the initial stages of a data mesh implementation - still in phase 1 before going truly broad. More domains are moving to support use cases across domains so phase 2 might be up soon.

Ammara does not believe source oriented data products, ones that are difficult to understand outside the domain, should not be made freely available on the mesh; they should not be made available to business users within the domain or to other domains. And her reasoning is very sound: if the data products are difficult to understand, it's easy to misuse them and they are more likely to change with the source systems so breaking changes/versions are more common. Other domains can consume the information from those source oriented data products in specially designed consumer oriented data products instead of directly from source oriented data products. Data scientists are a bit of another story as they are data literate enough to do some spelunking but even then, data scientist beware.

Ammara is also seeing an interesting pattern relative to source oriented data products. When you really start to map out a lot of obvious use cases for a domain - and remember, the size of a domain in this context is quite large -, it might seem like you need a large number of source oriented data products. But when you zoom out further, it becomes clear that you can actually shrink those into a much smaller number, that 5-6 data products mentioned earlier for that domain.

The way things are evolving at Ammara's current client is 3 layers relative to data products and use cases. For each use case, there are one or more - typically two it sounds like - consumer oriented data products. Then each consumer oriented data product is derived from or powered by typically three to four source data products. So the domains are able to create multiple consumer oriented data products off the same set of 5-6 data products. But it's still early days and will likely evolve further :)

Rerelease of #177 - Zhamak's Corner 14 - The Data Can't Protect Itself Sep 03, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 177. As stated in the original show notes, this is one to revist often as it is a great level-setting on why are we doing what we do in data mesh.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

This is likely to be an episode to revisit. Zhamak explains a simple concept - data should not be copied unless it is owned by a data product - but the why is multi-layered and important. It might be one of the most important yet underestimated aspect of data mesh because when done right, it truly ensures trust in data - for consumer but also producer. There's a lot of nuance in how Zhamak is thinking about this but the actual application is quite easy :)

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 3, 2023 Sep 03, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #52 Data Mesh Data Governance: Getting Out of Your Own Way - Interview w/ Sarita Bakst Aug 31, 2023

This is a rerelease of a previous episode as we are on a hiatus related to Scott's recent health issues. This is a very important episode for many reasons so please do check it out again :)

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Data Mesh Learning Meetup Presentation: https://www.youtube.com/watch?v=7iazNKG8XQo

Sarita's LinkedIn: https://www.linkedin.com/in/saritabakst/

In this episode, Scott interviewed Sarita Bakst, Managing Director and a leader in Firm-wide Data Management at JP Morgan Chase. Sarita was previously on a Data Mesh Learning meetup and is helping lead the firm's data mesh governance charge.

Per Sarita, historically, data governance has meant controls and gatekeeping to most people - typically getting in way of innovation. So there needs to be a focus on changing that narrative, not just through words but actions to show it's not the case. In data mesh, you need to ensure that domains can make good decisions on governance and seek out subject matter experts when it makes sense.

Sarita covered that one of the key issues in the way governance has been done is the people making decisions - the central governance team - don't have the real understanding of the data. When those decisions are put in the hands of the people who really know the data, but with guardrails and guidance, the fear is lifted about can we actually use this data and how. This opens up lots of new opportunities to leverage your data.

Sarita strongly recommends starting with purpose-built data products. Find a use case and build data products to serve that specific. And data products MUST be about unlocking business value. You don't need to serve up all of a domain's data on day one, in version one of that domain's first data product - make it extendible and reusable so you can find additional consumers and expand over time.

Get out of your own way on data governance in data mesh. You are going to learn and your approach will evolve as you learn. It's okay to not know everything upfront, set yourself up to not get in trouble - put the proper guardrails in place - but you won't know everything. Think of designing your risk controls as toll-gates and make sure they aren't bottlenecks.

Have standards (not standardization) so people don't have to invent things from scratch. Standards for interoperability, naming, etc. - think of them as guiding principles instead of rules. Make sure domain owners know who to contact and when on governance subject matter expertise.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #44 A Pragmatic Approach to Getting Started with Data Mesh at Northern Trust - Interview w/ Khanh Chau Aug 29, 2023

This is a rerelease of a previous episode due to my health related issues. New episodes will begin again in a few weeks. Please enjoy this very important episode of Data Mesh Radio.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

In this episode, Scott interviewed Khanh Chau, Lead Architect for the Data Mesh Initiative at Northern Trust.

Khanh believes you have to be passionate about making data better to do a good job implementing data mesh. And it is DEFINITELY a journey so you need patience and vision. Also, each journey is unique, you can't just copy/paste from another organization. You need to make failure okay - but you should look to make it easy to fail fast, measure, and adjust.

Khanh talked about the need for exec buy-in before heading down the data mesh path. They got that exec buy-in by proving that the total cost of ownership of data was quite high as the consumers had to do a LOT of work to get the data to usable.

When speaking internally, the business people were very excited to participate if it meant they could get quality data. Some of the IT/data engineering folks were harder to convince. It was especially hard to get them to shed layers of not-useful technology.

Some IT teams were easier to convince - they had felt the impact of a few too many middle-of-the-night data downtime incidents. Other teams hadn't felt that pain so there were harder to win over. There was also the incentive of additional possibilities - data mesh meant they could do things they couldn't do before.

Khanh talked about making the platform the easy and right path for 80% of use cases. They focused on making things easy to configure; basically: what transformations do you want to do and then it automatically provisions the pipelines. Their goal was to make it easy to make good progress quickly; their time to initial deploy went from 2-3 months per data service to 2-3 weeks per data product and they hope to drive it down further.

Northern Trust has been moving forward with data mesh for about 7 months as part of their high-level digital transformation initiative. On the data side, they had previously focused on data virtualization and data federation but it was not delivering the results they wanted. It was not as scalable as they wanted - it was taking 2-3 months to launch each new data service. They also did not have great information on who was consuming the data and why.

For their data mesh proof of concept, Khanh and team set a timeline of 9 weeks. They needed to prove value by then or data mesh would be a very tough sell internally. Khanh talked about the need to sell data mesh as a paradigm shift in order to get people out of technology-focused thinking.

Northern Trust decided to take a pragmatic approach e.g. not pushing all aspects of data ownership fully left. Khanh and team were focused on finding a "happy balance" on data product SLAs and quality - improvement was necessary but the team preferred done to perfect.

A big focus and a key driver for Northern Trust has been building muscle and learning/evolving along the way. It's important to evolve quickly and not build muscle in the wrong way.

Northern Trust is still in the early days on figuring out interoperability between data products. It's more of an art than a science. Khanh believes bi-temporality is more important right now than interoperability.

There are a lot of great learnings to takeaway from the Northern Trust journey.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Rerelease of #25 The Most Key Data Mesh Takeaway: It's Not Just You - Mesh Musings 5 Aug 27, 2023

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this release of episode 25, potentially the most important episode of Data Mesh Radio.

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 27, 2023 Aug 27, 2023

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

New Episode Hiatus Announcement Aug 25, 2023

Due to lingering severe health issues and not trying to rush myself back, I will be doing a 3 week hiatus around new episodes. In that break, I will be rereleasing some old episodes that I think people really need to listen to or possibly reflect on if they haven't listened to them lately. Here's to better health in the near future!

#251 Panel: Tying the Business Strategy to the Data Work and Vice Versa - Led by Burce Gültekin w/ Ghada Richani, Beth Bauer, and Michael Toland Aug 24, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

#251 Panel: Tying the Business Strategy to the Data Strategy and Data Work (and Vice Versa) - Led by Burce Gültekin w/ Ghada Richani, Beth Bauer, and Michael Toland

Burce's LinkedIn: https://www.linkedin.com/in/burcegultekin/

Beth's LinkedIn: https://www.linkedin.com/in/beth-bauer-102449/

Ghada's LinkedIn: https://www.linkedin.com/in/ghada-richani/

Michael's LinkedIn: https://www.linkedin.com/in/mjtoland/

Michael's 'Super Chicken' reference talk by Margaret Heffernan: https://www.ted.com/talks/margaret_heffernan_forget_the_pecking_order_at_work

In this episode, guest host Burce Gültekin, Chief Data and Analytics Officer at FrieslandCampina facilitated a discussion with Ghada Richani, Managing Director, Data & Technology Strategy & Project Management Office at Bank of America (guest of episode #206), Beth Bauer, CEO at her own consulting company PosiROI (guest of episode #218), and Michael Toland, Senior Product Management Consultant & Coach at Pathfinder Product Labs. As per usual, all guests were only reflecting their own views.

The topic for this panel was how do we tie the business strategy all the way down to the data work via the data strategy and vice versa of making sure the data work is tied into the business strategy. It's a complex topic so a lot of this is about driving to good ways of working and keeping sight of the actual overall business goals.

Scott note: As per usual with panels, I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

It's easy to lose sight of the data work when focusing on a data strategy. Make sure to understand and layout your understanding of what work will need to actually happen - at a high level - to deliver on your data strategy. Essentially, make sure you can execute on your data strategy via work and not words. And don't you dare forget your governance 😎
In many organizations, there is some - maybe a LOT of - cultural change necessary before the data strategy influences the business strategy. You need to prove the data work drives business outcomes and is aligned to the business strategy for a while - maybe 2-3 years - before the data strategy can be a key factor in shaping the business strategy.
Think about layering your data strategy to include what you can accomplish, what value you can drive today based on past investments and what you are trying to build for the future, where additional investments could drive more value down the line. If all you focus on is areas for additional investment instead of reinforcing the value already generated, it can feel like past investments haven't really paid off. If you don't share about what capabilities you've built and the value, why would someone want to invest in future capabilities?
Well executed data work - at its core simplicity - helps people to make better decisions. Lean into how your data work will help people to be better at their job. If you are pitching data as the key decision maker, most business execs will at best laugh it off but probably feel threatened. Show them how data makes them better at making the key decisions, not making the decisions for them.
The business strategy and data strategy are frameworks for making decisions - help your organization understand how data will make them better at making decisions. Not just by being more informed but by having more focus on what information matters. There are so so many inputs to consider, people are information overloaded. The data strategy should be about finding what matters and elevating that so leaders can focus on execution, not sorting through and absorbing information from 150 dashboards.
To develop an effective strategy, you have to understand where you are as a business. How can you know where you want to go and how you want to get there if you don't? Look to assess your high-level organizational capabilities as well as at the line of business level so you can help people align on the work to be done. A low data literacy team trying to do advanced AI is a bad strategy 😅
To create a good data strategy, you need to actually understand what a data strategy is and how it applies to your specific organization. A data strategy isn't only about desired outcomes but the entire information lifecycle from data acquisition/generation through to management and then leveraging the information gained to drive business decisions and outcomes. And say no to data for the sake of data!
Data leaders should help their business leaders to get data fluent to a degree where they can leverage data and know when they are out of their depth relative to data and should call in the experts to help them. We don't need every executive to be highly data literate as long as they recognize where they have weaknesses and lean on the right people to help them with data when it is called for.

Other Important Takeaways (many touch on similar points from different aspects):

As Burce said in her wrap-up: "Communicate, communicate, communicate and make sure everyone has the right incentives in place so that their behavior is moving [the business and data strategies] forward."
Think of data as feedback from the real world about what is happening and what you are trying to do. If we purposefully stop people from thinking of data as this mystifying, all powerful force, what more could we do? How many more people would be willing to use data more frequently?
If the data strategy is its own wholly separate plan/approach, you're probably doing it wrong. It must be closely aligned to the business strategy or you are likely headed for trouble. A good data strategy is an enabling function to allow data and data work to support the business strategy.
If your data strategy isn't tightly aligned with your business strategy, your data work to execute on your data strategy is far more likely to be seen as a cost center. Make sure the rest of the organization understands how the data strategy supports their work and key business targets for data to be seen as a value generator.
Many executives think of data as mostly powering their applications and interactions with customers/partners. It's crucial to get them to understand how data can help make better decisions looking forward and not merely reporting on what happened. Especially because they need to help generate the data to use to make those forward-looking decisions.
Before doing data work, you should always ask what is the business justification. Things might not turn out how you expected but it's important to share with others and also make sure you are focusing on work that should drive business value. It also makes it easy to show the value you delivered instead of people believing the value happened elsewhere. Data work enables value, look to show that often.
If you want a data strategy of becoming more data driven and more data fluent to succeed, you have to take into account the organizational and cultural change aspects. You don't have to solve for everything but technology isn't going to solve your data literacy challenges for you!
When focusing on the data work, think about the small steps. You don't need to map every single step out but you can't try to take giant leaps and expect things to work out well.
When building out the data strategy, look at timelines to value and making people understand when value will come. If you only focus on quick wins, you will see your potential for data wins dry up over time. You have to balance near-term wins and building long-term capabilities to create far more capabilities for wins in the future. But only focusing on the 2-year out horizon probably won't win you any followers, friends, or funds.
A potential good leverage point to bring business execs in on the data strategy is around risk. What risks are they looking to take and how can data better enable them to understand and quantify those risks to ensure the potential rewards are actually worth the risk?
Throughout the data strategy and data work, always tie things back to specific business problems. Not having a good data platform isn't a business problem, not being able to handle your data at scale to reliably and quickly make decisions on certain challenges is a major business problem and not having a good data platform causes that business problem.
If you provide data without context, your business partners won't really care. Same for data work. Think about the business context of what you are trying to do and communicate that to drive alignment and buy-in.
Data leaders should certainly want to be part of the conversations shaping the business strategy. But unless you've got a proven track record in your company of data driving your business results, it might take time to earn your seat at the table. While unfortunate, that is totally normal.
Cultural change unfortunately typically takes a long time. If your company is not heavily focused on data, it may take a long time before the business strategy is influenced by data. You need to get more and more momentum and change people's view of data and their experience with leveraging data to do their jobs better before you can expect data to be integral to the business strategy. So start building the momentum now!
The data strategy is just half the battle. You need to incentivize people to actually execute on it, to do the data work. Incentives matter far more than we typically give credence to in data.
Find your advocates, your data champions. Those are your best tools to really drive cultural change, the people sitting alongside other business folks and showing them the power of embracing data.
If you are tying the data strategy to the data work, you need to understand how you already do data work. Coming up with a grand strategy perfectly aligned to the business strategy that completely changes how data work gets done is going to be a real mess.
If the people crafting the business strategy aren't data literate, are they going to create a strategy that relies on data to drive results? Probably not. Try to help them get to a decently data literate level.
When building your data strategy and focusing on the data work, you should consider what work ties to which parts of the business strategy. If the company isn't trying to do real-time data interaction and analysis, do you really need that streaming ML solution? If you can tie your investments directly to business objectives, it's easier to get funding and to keep the attention of business leaders around data initiatives.
It's okay - and probably a best practice - to iterate on your data strategy as you do more data work. Are you getting more or less aligned with the business strategy? Even if the business strategy is a once a year proclamation from the CEO and Board, you don't have to live with that time table. Test, learn, adapt.
The data work alone never adds value. If you deliver to an amazing data strategy and the organization can't act on, can't take advantage of the work, then it was a waste of time and effort. Don't get too far ahead of the organization while you improve what they are doing relative to data.
Well-functioning data teams do innovation work that may or may not pay off. Try to keep that in mind when you build your data strategy. If no one is trying and potentially failing, you are probably playing things too safe. Keep your blast radii small if things blow up, but create space for innovation.
When talking about data work and value, business value is not always in dollars. Make sure people understand that. Ask your business partners how much they value certain outcomes.
The world is changing so much faster than it ever has before. Execs need inputs at a much quicker pace than historically to stay on top of things and ahead of the competition. Work with them to understand their pain and align on data work to address it instead of focusing on the cool tech aspects.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#250 Staying Aligned on the Vision in Data Mesh - Lessons from Volkswagen's Journey - Interview w/ Christoph Spohr Aug 20, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Christoph's LinkedIn: https://www.linkedin.com/in/christoph-spohr-2a33b619b/

In this episode, Scott interviewed Christoph Spohr, Lead Architect of Big Data Platforms and the Product Owner of Data Mesh at Volkswagen Group. To be clear, Christoph was only representing his own views on the episode.

Quick Note: apologies that Scott's audio is a bit weird, he had yet to build his makeshift sound studio in the Netherlands.

Some key takeaways/thoughts from Christoph's point of view:

"There are not solutions, only trade-offs." Understand that nothing is perfect.
!Controversial!: Your biggest risk to a use case is choosing the wrong abstraction, the wrong architectural paradigm. When working with a domain, look for a way to achieve the required result that is as close to their existing way of working as possible. Scott note: this can become challenging on finding reusable ways of working if you go too far but greatly smooths domain-level adoption. It's all a tradeoff 😜
The second biggest risk is do stakeholders really feel and acknowledge the pain the existing approach/system is causing and want to move? The third biggest risk is about promising things you cannot deliver - obviously look to avoid that.
?Controversial?: A really key aspect of doing data mesh well is having and keeping perspective: doing data mesh is more about "aligning people on a shared vision and not getting lost in the overall process." Don't get so bogged down in exact implementation details if possible.
Keep people informed and aligned and start your conversations on clarifying what people believe/understand about data mesh so you aren't talking past each other. For many people in your organization, their understanding of data mesh will be shaped by vendor content/conversations. You need to check alignment many times as it can drift.
Advice to former data mesh self: Really focus on keeping that communication between all parties going and making sure people are aligned on the approach, complexities, trade-offs, and risks. And don't think people automatically stay aligned, stay on top of the communication and relationships.
In a large organization, if you want to implement data mesh, find sweet spots for it to succeed. Because you will have to make compromises/trade-offs. Everything in data and IT is about finding the right compromises but don't put yourself in a place where your compromises fully undermine your data mesh vision.
When putting together your enterprise-level vision for data mesh, you need to break it down into two parts: A) how would it work with and serve existing business processes and B) how to approach the IT architecture and assessing/acquiring the necessary capabilities to do data mesh.
Data mesh is one of the most - if not the most - complex data architectures and you need to have a strong understanding of the complexities and not underestimate them. Otherwise, scalability will likely be a significant challenge.
Data mesh does not mean all your data problems go away. As Zhamak says, it also creates new ones.
It's important to listen to stakeholders' challenges and deliver a solution that addresses those challenges. If you try to deliver to their exact wants, those often aren't scalable ways to address their challenges. Focus to the end result, not what they _say_ they want.
When it comes to data mesh, make sure stakeholders understand that there is a cultural change component. If they want things to work at the speed and scale data mesh can provide if done well, approaches and hearts/minds have to change. Data mesh is not a magic wand to wave at your data to make your approaches/solutions scalable.
?Controversial?: Your IT people need to work on their communication skills. And watch out for IT/data team members getting a big head about what they can do, that can create lots of challenges.
Plant your data mesh seeds early - have conversations with stakeholders, even when they aren't ready to take advantage of the data mesh approach/your implementation yet. They will be much more likely to come to you for help when they are ready and eager.
Try to understand your senior leaders' IT systems background and their favorite - or at least known - approaches. When you frame data work in a way they understand, even if some of the implementation aspects change, they are much more likely to be in favor of it.
?Controversial?: Instead of going to the business leaders, look to talk to the technical people within a domain first to find out the business challenges at the technical level. That way, you can bring the business leaders an idea of an approach that is more fully formed. Scott note: you can take this too far quite easily and try to come in with only a technical solution to the challenge.
?Controversial?: People think data platforms are mostly about doing more and more complex analysis on data but that's rarely the use cases that add the most value. Scott note: strongly agree, reducing friction to leveraging data is rarely about the most complex data and analytics practices.
"[My] loyalty to data mesh ends when complexity gets too big." Scott note: take this one to heart. Data mesh for the sake of data mesh helps no one - except maybe a resume 😅
Don't get bogged down by analysis paralysis. Look to work in a simple and speedy manner - if you have to backtrack, it is almost certainly still going to be quicker than trying to answer every question ahead of doing the work.

Christoph started the conversation with some perspective into the IT/data landscape at Volkswagen where they have 100Ks of employees and 100s of petabytes of data. So, when you think about what good data mesh practices look like in that environment, at that scale, it's much more about "aligning people on a shared vision and not getting lost in the overall process." And it's easy to lose alignment because so many people are getting their understanding about data mesh in general - or at least aspects of data mesh - from vendors. Those definitions are often very different, especially around how much is centralized or decentralized. So look to start your conversations with colleagues by clarifying everyone's understanding of data mesh or a specific aspect of data mesh to stay on the same page. Your understanding will change over time and so will theirs.

In data and IT work, at the end of the day, everything is about compromises and tradeoffs for Christoph. Nothing is the "right" answer, it is about the search for the least bad answer. When you are looking at data mesh in a very large organization like Volkswagen, he recommends looking for your sweet spots - there are places data mesh could thrive and places where it's not a good fit. Don't get too bogged down in trying to go organization wide early on. You couldn't support that anyway!

Christoph recommends before getting too deep into your data mesh implementation strategy, break down your high-level vision into two pieces. The first is: how would this align to and serve the existing business processes? You need a business side of your vision - what are we doing data mesh for? You also won't be able to change the way the organization works easily so find ways to align to how things already work. The second is the IT architecture and the necessary capabilities to actually pull off doing data mesh well. How do you get into a data-driven mindset? What are the kinds of technical implementation details you will need to accommodate - not even the use cases, just what types of domains and sources. How will you manage risk, how will you make sure things can scale?

Choosing the wrong abstraction, the wrong architectural paradigm is the biggest risk to a use case in Christoph's view. Each team has a different way of working and you should look to, still within the framework of data mesh, find a way of working that achieves the target results but is as close to their existing way of working as possible. This will likely smooth adoption friction and prevent some misunderstandings. His next two biggest risks were first, make sure you have stakeholders that really feel the pain of the existing approach to the use case and systems and are willing to move. If there isn't a desire to move, it's difficult to drag them along and succeed. The other - the third biggest - risk is about don't over-promise; make sure what you say you will deliver is realistic.

For Christoph, "there are no solutions, only trade-offs." When discussing approaches, let people know the trade-offs you are making because nothing will be perfect. That honesty will resonate well with most people.

In Volkswagen's journey specifically, Christoph expected the biggest pain point for customers to be that data was in silos but it wasn't - they were more concerned with fine-grained access control. So when he talks to stakeholders, he makes sure to understand their pain first instead of trying to pitch the approach they've landed on with data mesh. But it's also important to not ask people what they want in his experience, they will try to just add scaling to their existing approach when a new approach is often necessary. So, listen for the pain, pitch to the pain, but don't pitch to their exact desired approach if it's not a scalable solution.

When asked about advice to his former data mesh self, Christoph circled back to the challenges around communication. You need to constantly be staying in touch with people, aligning on vision and sharing trade-offs. You also need to focus on what are people's pains and look to deliver something scalable to that pain rather than asking them what they want - they often want their current approach to magically scale. You have to partner with them and make sure they understand that cultural change will be necessary to accomplish their goals - this isn't a magic mesh wand. And he also recommends watching out for any of the IT/data team getting too big of a head because they can do really interesting things with data.

Christoph recommends a few things for getting more buy-in for your data mesh implementation in a large organization. One is to plant your seeds by having lots of conversations about what you are trying to achieve because when people are ready, they will come to you. A second is to talk to a lot of people to understand where big challenges/opportunities are in the organization. That way, you can try to focus a bit more on high-value use cases. And the third is try to frame your approach and choose an abstraction that matches the senior leader/stakeholders. If they like one approach, try to frame things - and hopefully develop your approach - in a way that works well for them so you aren't trying to drive buy-in for the approach as well.

Volkswagen is potentially a different kind of organization to many others out there. In Christoph's experience, as a hierarchical organization, it often makes sense to talk to the technical people within a domain first to assess what challenges there are and come to the senior leaders with something more fully formed and targeting their actual business challenges rather than asking them to share their data related challenges. Be opportunistic in selecting use cases to go after. And he is also not loyal to the concept of data mesh - if it gets too complex at scale, he will look for something else.

As to ways of working, Christoph recommends to go for simple and speedy because even if you head down a path that isn't working, it will still likely be quicker to pivot and head in a new direction than if you had gotten bogged down with analysis paralysis. Look to converse with other experts in the industry on your journey - Christoph was unlucky in finding people who had done data mesh at Volkswagen scale but that's not the case for most organizations.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 20, 2023 Aug 20, 2023

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#249 Zhamak's Corner 27 - Creating Truly Scalable Interconnectivity for Data Aug 17, 2023

Takeaways:

A data product by the data mesh definition cannot by definition be a silo because it is meant to be consumed by the rest of the organization as a greater part of the picture of the organization. Whether that is true in practice remains to be seen :)
APIs are the way we've learned in software to interconnect between services, so we need to learn how to leverage the same approaches in data.
Within a single organization, we may be able to get by without having a great set of industry-wide interconnectivity standards - the company can fully control their own sharing. But, we know that to truly unlock the value of our organizations, cross organization data sharing - in a scalable and useful way - is necessary. So we need better ways to do that - standards and tooling/technology.
It remains to be seen which will emerge first: the standards or the tooling/technology - a chicken and egg issue there - but they must really must be intertwined to work well.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#248 Doing Data Quality Right by Building Trust - Interview w/ Ale Cabrera Aug 13, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ale's LinkedIn: https://www.linkedin.com/in/alejandracabre/

In this episode, Scott interviewed Ale Cabrera, Senior Data Quality Product Manager at Clearbit. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Ale's point of view:

A key part of understanding what data work will be impactful is a simple phrase: "Is my understanding correct?" Putting out there what you took in and making sure you're on the same page will save a ton of time and headaches!
Her advice to her past self: In data, far too often, we try to jump to solutioning instead of really taking the time to understand the problem. Start from understanding the problem and assessing it first.
It's very easy to make data say something that it's not actually reflecting. Quality isn't just about accuracy or similar metrics, sometimes there are intangible aspects around correctness that people get but usually can't measure.
In data work, many people miss two crucial aspects - the voice of the customer and the why. If you build the greatest thing ever but it isn't what the customer wants, it won't be used. Similarly, if you focus on the work and not the target outcome, your results are likely to be subpar.
If you want to prove data work return on investment, try to associate it to a key company metric and talk about how improving that metric will drive better business outcomes.
When you want to prove out the value of data quality, attach quality issues to direct business challenges or goals. It’s easy if you are a company selling data but you have to understand why bad quality data negatively impacts the company in order to gain influence to improve your data quality.
Far too often in data work, people try to exchange data - the 1s and 0s - and forget to exchange information. Get people together and make sure you are in alignment, get people actually talking to make sure you understand and have a good forward path.
The most crucial aspect of data quality is trust. If you can't drive trust, then all the 'quality' in the world doesn't mean anything because people won't use it.
As a data person, your job is not to do data work. It is to unblock teams/projects/people and add value. Yes, that is through data work but the work isn't the point.
The way to think about trust is a combination of "credibility, reliability, and intimacy by self-orientation."
Measuring trust is hard but a good way is through interviews with users.
Nothing will ever be perfect so consumers need to understand that. A mature and healthy conversation is to ask them to tell you if they see any data quality issues because issues always can happen. If they won't consume from data that might not be perfect, they can’t consume any data.
When you are quick to react to data quality issues, your consumers can get more value from the data because they can be more sure sooner whether the shape of the data changing is an issue or a genuine reflection of something changing.
In data, typically one-size-fits-all really ends up being that it fits none. You don't want to over customize a data product but still, your consumers are likely going to want something that fits their needs rather than something generic that they have to do the heavy lifting to be able to use.
How people consume the same data can have a large impact on trust too. While data people hate Excel, sometimes you have to deliver something they can use in Excel. Otherwise, they will struggle to trust the data enough to rely on it.
Is what makes something great the output or the impact? When applying product thinking, something doesn't really matter unless it is used to generate value.
A key aspect of product thinking is prioritization. Yes, it would be great if we could build to every ask but focusing on what will most likely deliver value and why will keep you more focused on generating value through your data work.

Ale started with a bit about her career and what led her to focus on data quality, including what led her to her current role. On advice to her past self: in one of her past roles, a lead engineer gave her enduring advice: think before you code 😎 Oftentimes, in data, people jump to trying to solution instead of really taking the time to understand the problem and look at the various potential solutions to choose one that will work better in the long run. She also said to respect the data when considering quality. You can get data to tell a story that it really isn't reflecting the reality of what is happening so you have to give it the respect to not shape it to tell "statistical lies".

Another important aspect of data work for Ale is focusing on the voice of the customer and the why. What is the user telling you, what are the pain points. You can build the best solution but if it isn't what the customer wants, they probably won't use it. Look at all the amazing data platforms that barely anyone uses. Focusing on the why, why are people looking at this data or trying to address this challenge, will keep you from trying to solution instead of meet the customer where they are and help them solve the business challenge, not just solve a challenge about data. As a data person, the point of your job is not to do data work. It is to unblock teams/projects/people and add value. Yes, that is through data work but the work isn't the point.

When trying to prove return on investment for data work, Ale believes it's important to tie it to a key business metric. Something like 'Our data quality improvement from 98% accuracy to 99% accuracy will mean a decrease in churn by 5%, netting the company X amount of revenue.' In her case, she is attaching the data quality work to reducing churn because it is a key metric that is intrinsically linked to data quality in a company selling data.

For Ale, an important aspect of establishing yourself as an internal expert on data quality is tying the data work to value - what actually matters for the company. Improving data quality for something that drives 1% of revenue so it grows 25% is not as much of an increase as improving something that is 10% of revenue so it grows 3%. But the business might favor one over the other. What matters most to the business and why?

Ale shared a story about why it's so crucial to get people actually in a space exchanging context and information instead of just exchanging the 1s and 0s of data. She was working on a project involving two teams and neither side had ever met each other despite being months into the project. The data quality was terrible, it was seeming like half the data was wrong or missing. So the other side asked about what was happening when they were pulling data from both servers. Her team only knew about one of the servers. So get in the room - virtual or physical - and actually communicate!

Trust is the most crucial aspect of data quality work for Ale. If you have the best quality data but people don't trust it enough to use it, then it's still not of value. And trust is often built more through relationships than anything else (see Beth Bauer, episode #218). What is the purpose of the work if not to drive an outcome. If there isn't trust, can you really contribute significantly to an outcome? The way to think about trust is a combination of "credibility, reliability, and intimacy by self-orientation." She uses interviews as a good way to measure trust as time goes along. And that also encourages users to actually say something if they see an issue, which also increases trust after an issue. Listening to people is just a fundamental building block of driving trust.

When asked about if you lose credibility when telling data consumers to immediately flag any issues, Ale was clear that in data, mistakes happen. Nothing is ever perfect. So it's crucial that we can have conversations and tell people to speak up if they see quality issues. Recognizing that there will be data issues doesn't mean you are sloppy with data, only that you are realistic. Being aware and reacting quickly to issues also means consumers can be more sure that an anomaly or change in the data is actually something real and react to that more quickly. Recognizing you will make mistakes and working quickly to rectify those mistakes builds trust.

For Ale, there is a tendency in data for delivering overkill and/or something the other party has difficulty trusting. Providing a dashboard for someone who wants to see the actual data is just not that helpful. Meet people where they are while you upskill them to be better, even if that where is in Excel or Google Sheets.

Ale talked about switching to the product mindset in data. If you are building 'amazing things' but no one is using them, are they really all that great? Is what makes something great the output or the impact? A good way to get to impact is focusing on acceptance criteria - what would make the user happy and what are they actually expecting?

In wrapping up, Ale shared a bit about how to become a good product manager and her thoughts on how crucial prioritization is to actually applying data product thinking. How specific should you get with a solution is a very tough question but you can start by asking what is the expected impact versus a more generic solution.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 13, 2023 Aug 13, 2023

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#247 Data Mesh Implementation Success Metrics - The Data Platform - Mesh Musings 52 Aug 10, 2023

Key takeaways:

As mentioned last time, at the start, it's more important to start measuring something than it is to measure the right things. Do NOT let analysis paralysis hold you back. Start measuring early to figure out what actually matters and that will also change over time.
Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring.
Use fitness functions.
Data mesh really is a journey and so will be how you measure success. You will need to find small and simple ways to measure. Don't get bogged down. Your measurements will be rough and kinda depressing with the amount of challenges to tackle at the start. Just understand this is about how well you are doing, not how complete you are - there is always more to do!
Good metrics to consider for the platform success metrics: satisfaction, time to deploy new products, time to update existing ones, ease of use - which kind of blends into all the others -, searchability/discoverability, ease of interconnection, mean-time-to-detect and mean-time-to-recovery from issues, governance, guardrails, and automation/helpful artifacts.
Look to measure friction and how you reduce it and if possible - I have no great ideas here - what is the extra value add to the business, something that the platform was the genesis of? Maybe that's suggesting data products?
Lastly, reflect back on how far you've come, we often forget to do that!

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#246 Making Federated Data Governance Approachable and Effective - Interview w/ Kinda El Maarry, PhD Aug 06, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

#252 Designing and Building a Better Data Governance Approach - Interview w/ Lauren Maffeo

Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/

In this episode, Scott interviewed Kinda El Maarry PhD, Director of Data Governance at Prima. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Kinda's point of view:

Always be assessing data literacy gaps and work to address them. Gaps don't have to be a bad thing, they are a chance for your people to grow! But ignore them at your own peril :)
A key aspect of your job as someone on the central governance team - or anyone managing a governance function - is to reduce friction, not add overhead. It can be hard to do that at the start of a journey but it should be a north star focus.
The phrase 'data governance' still has a stigma for many people from what often happens with a central-only approach to governance - processes and rules divorced from the actual business processes and the data which creates bottlenecks and headaches.
The federation of governance in data mesh helps address the bottlenecks but it introduces a new layer of complexity which can also cause some fear.
?Controversial?: If you are asking people to take on ownership of data, you must make sure their line of business/part of the org has the actual capabilities to do so. Otherwise, they will greatly struggle to own and are likely to push back.
Don't talk about governance in the weeds to people, the toolbox talks, the policies and standards. Focus on what are you trying to achieve with governance and then lean back on the toolbox if you need to explain the how. But many don't care about the how.
There is a sliding scale of how much is federated versus centralized based on domain capability and maturity level. You don't just push all responsibilities on to each domain at the start, it's a gradual process finding the right balance.
When you find a domain that has skill gaps necessary to overcome, set up a roadmap and have as positive of a conversation as possible to start along that roadmap - it's not what they lack but where they'll grow.
!Controversial!: Where possible, get rid of the central data governance committee. Collaboration is more key than trying to keep every stakeholder across the organization informed and involved in most decisions.
?Controversial?: Data governance people should stop thinking about federation as losing control. It's about enabling the decisions to be made by those that can best make them. That way, the governance team gets to focus on the guardrails, the standards, the blueprints - the things that make stuff easier for everyone.
No one is saying hand over control of very sensitive decisions without reason. It's about making sure the day-to-day decisions can be made so when there is an important/sensitive decision to be made, that's where the central team can step in to assist.
When trying to get teams to take ownership of their data, there isn't a silver bullet - every discussion and domain is different. But start with a clear understanding of what ownership means and start the discussion with what will you do to make them capable and ease the burden of ownership. Just dumping responsibilities in their lap won't go well.
Data culture is an often overlooked aspect of data governance. But it provides your scale - getting people to evangelize and share what's working instead of all knowledge dissemination coming from a central team/training is how you can get scale.
It's not exactly the data governance team's role to find the right reporting structure and career management aspects of a data mesh implementation but the team can help by creating good spaces for people to share and communicate to create a sense of comradery.
Centralized capabilities and coordination of certain aspects are crucial to doing data mesh right. You want to decentralize the decisions and the application of those decisions but centralize the enablement. A core, central data team is pretty important to making that happen.
?Controversial?: Interoperability between data products is important but interoperability often can get in the way of delivering data products. Don't focus as much on it as you probably think.

Kinda started off discussing the traditional view of centralized data governance and the stigma around the phrase. Because the legacy way were a team that didn't really understand many of the key business aspects around the data itself, the policies and guardrails were not designed to make creating data assets or projects easy. The decoupling of understanding from decisions led to bottlenecks. And data mesh is specifically designed in part to eliminate those bottlenecks.

However, as Kinda pointed to, the federated governance model of data mesh brings a whole new layer of complexity to data governance. So talking federated governance can create anxiety that there will be even more bottlenecks. But really, for many things, it makes governance far easier. A crucial point of data mesh is to unlock the ability to quickly react at the domain level by giving them the ability to make (most of) their own decisions - and easily lean on expertise when they need to.

Kinda talked about good communication around governance: focus on the why, what are you doing to drive value via governance, instead of the toolbox angle of policies, standards, etc. If they understand the why, they are more likely to lean in to the extra work data mesh puts on their plates. So start from listening and talking pain points - theirs and other people within the organization. If you are showing they are heard and that you're addressing their pain points to drive value, governance isn't so scary.

One aspect of data mesh that people seem to get a bit wrong in Kinda's view (and Scott STRONGLY agrees) is the federation versus centralization balance. You need to understand each domain's capabilities and overall maturity level. It's okay to start with most of the governance centralized if a domain isn't ready yet. It will be a bottleneck but it's a known bottleneck that is taken as a conscious choice.

Sometimes, in Kinda's experience, domains will want to move on pushing out a data product before they and/or the organization is ready. That's okay, just make sure they understand what they own and that they will be on their own - at least it's okay as long as it isn't breaking regulatory compliance or anything.

Kinda believes that you should look to do away with the central data governance committee where you can. You want stakeholders engaged of course but you can move faster and with better decisioning by collaborating, not committee-ing (blame Scott for that phrasing, not Kinda 😅). For example, instead of creating a global data quality standard herself and pushing it out, she created a hackathon around data quality measurement and the best solution - or really a combination of solutions - won.

Governance people should not look at federation as a 'loss of control' according to Kinda but instead, it's freeing the governance people up to focus on where they can add the most value, creating those blueprints, standards, guardrails, etc. It's helping drive data literacy and data culture. And it's acting as an advisor when necessary - you don't want each domain to implement GDPR compliance for instance. At the end of the day, the people on the ground in the lines of business know the data better than someone with only a centralized view really could so make it possible for them to make the right decisions and apply them easily. A big misconception here is that it's handing over all control but that's just not the case - when there are sensitive decisions or things that are of a bigger scale - e.g. regulatory related decisions - that's when the central team should be focusing on partnering the most.

There are a few parts to Kinda's approach for driving buy-in for a domain to take data ownership. The first is that you have to understand each domain and person is different. There isn't a single script that just works. These are people you're dealing with. Be prepared to customize the conversations and solutions. Second, have an understanding of what ownership actually means so you can communicate expectations clearly. Hard to understand exactly what's needed at the very start of a journey but get as close as you can. And third, when actually having the conversation, don't start from the expectations and new demands on them, start from what you're doing to make it possible so they actually can take ownership of the data. That includes tooling, training, and realigning their KPIs/OKRs so the data work is actually a key part of their role and measured results.

For any data governance effort, Kinda believes you should make sure to focus on all three of technology, process, and people. The people aspect is the most overlooked but your people are what provide scale by enabling the application of whatever governance work you are doing.

In general, driving data culture is crucial for data governance in Kinda's view. Data literacy and culture is part of data governance of course but in general, to find your internal key enablers and to inject data as part of the business processes, you can't just create mandates or platforms. You need to make creating spaces for people to share ideas and celebrate each other part of your culture.

For Kinda, when asked Zhamak's provocative question of "Will we actually need a CDO in an organization doing data mesh?", she and Scott agree the answer is yes. Because, at least at the start, there needs to be someone running that central coordination and helping to make all the decisions around what to decentralize and when and a very important one: doing the exec to exec buy-in and communication. So strategic direction and maintaining momentum and engagement is crucial, thus a data leader is necessary.

Kinda believes in both a top-down and bottom-up approach to doing data governance in data mesh. The top down is that executive buy-in and helping to change people's prioritization and then the bottom-up is focusing on the use case and making sure they have the necessary guardrails and processes as you continue to build those out.

In wrapping up, Kinda shared a somewhat controversial viewpoint on interoperability. While it is important, trying to focus on it too much will likely slow you down. She believes you shouldn't let interoperability get in the way of your work and you can't try to solve for all interoperability at the start.

Quick tidbits:

If you are doing data governance right, it will remove friction instead of add additional overhead. At the start of a journey, the scales might not be as far in favor of removing friction but always look to build that out - that's a key success metric.

For every project or initiative always, always, always start by assessing your data literacy gaps. If you don't, you are setting people up for failure.

You can show success through your data governance program by measuring things like time to actionable insights. Metrics in general will play a big role in proving out success so try to measure what is changing and you will probably be bad at measuring at first.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 6, 2023 Aug 06, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#245 Panel: Lessons From Doing Data Mesh Again: The Second Timers Club - Led by Samia Rahman w/ Khanh Chau and Sheetal Pratik Aug 03, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Samia's LinkedIn: https://www.linkedin.com/in/samia-r-b7b65216/

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

Sheetal's Links:

Sheetal's LinkedIn: https://www.linkedin.com/in/sheetalpratik/

First Paper at ICEB Conference: https://easychair.org/publications/preprint/qZ3m

Interview in Harvard Business Review: https://hbr.org/resources/pdfs/comm/BeyondTechnology_CreatingBusinessValuewithDataMesh.pdf

Medium Blog on Saxo Bank Implementation: https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f

In this episode, guest host Samia Rahman, Director of Enterprise Data Strategy, Architecture, and Governance at life sciences company Seagen (guest of episode #67) facilitated a discussion with Sheetal Pratik, Director Engineering and leading India Data Integration Platform at Adidas (guest of episode #24), and Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44). As per usual, all guests were only reflecting their own views.

The topic for this panel was a bit different with reflections and learnings from doing a second (or more) data mesh implementation from a practitioner standpoint - it's people who have the experience to reflect back on multiple implementations to give advice to their past selves and those earlier in their journeys after seeing multiple organizations implementing data mesh up close. Khanh Chau was the leading data architect for Northern Trust's data mesh implementation before taking up the same role at Grainger for their implementation. Sheetal was the head of data integration at Saxo Bank as part of their data mesh implementation before moving to Adidas where she is leading the India Data Integration Platforms and has also played integration lead for moving an on prem centralized integration platform to federated cloud integration platform. And Samia worked deeply on two implementations at Thoughtworks - including one closely with Zhamak - before starting at Seagen.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Prepare to take many concepts from abstract to something concrete and explain it to many people. Repeatedly. And your definitions will change over time too. A big part of leading an implementation is about communication and keeping people on the same page and informed.
Similarly, prepare for confusion. People will go to different sources for information - including a lot of vendor content - to learn about data mesh. So keeping people aligned and understanding key aspects of data mesh are crucial. "What is data mesh?" can be a dangerous and difficult question when it shouldn't be.
It's important to learn from other organizations' implementations but your journey - if you are going for success and not simply copying - will be unique in what matters most when. It's easy to get overwhelmed trying to manage every aspect perfectly. So look to focus on learning and iterating along the way instead of getting it perfect upfront. Stepping back and looking at the 7 journeys across the panelists, they really don't look that similar.
When choosing which domain(s) to partner with, highest value or ROI can look like the most important metric but really, finding someone who is willing to take a risk and will also champion your solution when it succeeds is the most important. You do need to find use cases that have reasonable ROI but having a true partner is the most important part.
Data mesh isn't a solution to the business people - and it really shouldn't be to data people probably too. It's a way to address an ever growing challenge and issue within the organization. The business people feel the pain, talk to the pain. They don't care nearly as much about the how as you assume. Make them a tasty sausage, not give them a sausage factory tour.
Don't underestimate the fear of change and loss of control. Data mesh is a new approach and many are sick of new data approaches 😅 but people's fear of getting left behind or not being in control is a natural human reaction. Make sure people understand how this makes things better not just for the organization but also for them.
Defining data mesh success metrics - at the micro level of data products and at the macro level of the entire implementation - is still hard. There are a lot of measures you can use that are helpful - and do look into fitness functions - but it will take some effort to find good success metrics. And those metrics will change over time.
As some past guests have also noted, a good way to drive buy-in for those actually doing the data work is to make it faster to produce data for themselves, not just for others. If you can make getting access to data and creating something useful for the data producing domain quicker, many will lean in to leveraging the platform quickly.

Other Important Takeaways (many touch on similar points from different aspects):

Faster time to business outcomes is likely attractive to business partners to get buy-in AND a success metric you can actually somewhat reliably measure.
For success metrics for your implementation itself, look at time to deliver new products. Slow delivery might be because a domain isn't ready or leaning in but repeated slow delivery is probably a problem with the platform in some way.
It's tempting to try to start from the infrastructure aspects first but you have to focus on the value of doing something like data mesh. What is the value proposition for the organization? What is it for the specific person you are talking to?
The needs of your organization will look different than other organizations. You probably can't know exactly what will end up being your big focus ahead of time. A lot of the learnings and realizations can only happen when you're in the midst of your journey.
Because every part of an organization matures differently, each domain generally has developed a way of working that fits to their domain but makes it very hard to then standardize the ways of working or sharing data. You need to build central frameworks for collaborating but still allowing domains to work (mostly) in their existing ways. Otherwise, you are probably fighting a losing battle.
It's easy to lose sight of how important the business partners are to your data mesh success. They can provide a ton of leverage to driving buy-in, budget, etc. Or, they can provide a ton of headwinds/resistance. Find your good partners and make sure you're aligned on making your overall data mesh approach a success.
Similarly, at its core, data mesh is about business transformation. Yes, transformation driven by changing how we interact with data and do data work but business transformation nonetheless.
To get funding/sponsorship for something beyond the first few data products, you have to show the value proposition, especially in a soft economic environment. You aren't going to do that by talking technology. It's important to show a good return soon for sure but also a broader vision.
Technological solutions don't equate to business value to most business people. Talk to them about what they care about.
With every success, you can build momentum. But that momentum only builds if you're selling the successes, telling people and proving out the value. Data leaders need to be as much about showing off the results as achieving them unfortunately.
It can be effective to mesh buy-in to point out how decentralized efforts will probably happen - e.g. shadow IT - but they become their own little IT fiefdoms if not at least coordinated in some way through a central mechanism. It doesn't need to be ownership but if you don't have a central framework and ruleset, you end up with silos, spending more money on many platforms, and worse quality data.
As Khanh said regarding governance, "So how do we bring this whole ecosystem of technologies [together] in such a way that is safe for people to do things faster and meet all the regulatory compliance?" Two key words in there are safe and faster, don't make governance about gates, make it about getting to data production more quickly but still safely.
While data mesh may not itself be cheaper than doing nothing, it often results in cost savings relatively quickly by removing duplication of work. If everyone is at least broadcasting what they are working on, then there is far less overlap that might cause confusion and two teams doing essentially the same thing.
Trying to build the entire platform ahead of time is a really bad idea but you also can't come to the business with an inferior, clunky, hard-to-use, immature solution. Make sure your organization has some platform capabilities to not make it a huge hassle to build out early data products.
Don't try to onboard every domain at the start of your journey. Start to seed the conversation with them, talk to the pain points, but you can't handle 30 domains coming onboard at once.
When driving buy-in, don't only focus on executives. You need people throughout the organization leaning in. If you need new skills to accomplish data mesh, you need people who will acquire those skills and level up their data fluency. Talk to the people doing the actual day-to-day work.
You potentially don’t need a huge budget to start a data mesh journey. You can start lean - yes, you will have leaner results but if you can prove faster time to business outcomes, you can generate more and more momentum.
Build momentum around your mindset shift. You aren't going to convince people overnight.
It's important for domains to take over ownership of data but that also doesn't happen overnight. And your initial ownership will probably not be as strong as you'd like. And it probably won't involve the operational plane system experts/developers as much as you'd like. But that will evolve and get better. You have to get started with okay, not perfect. Crawl, walk, run.
You probably need to apply DevOps principles to your data products where the data product developers also support the data products, do the operations aspect of them. But it might be hard to get people really bought in on that when your platform is still pretty nascent.
You want to build feedback mechanisms around data products to drive domains to make them better. But there are many concepts around data product feedback being touted by vendors but few people are happy with what they've built or are getting from vendors. The best feedback mechanism is probably a culture shift towards providing more/better feedback.
As Samia said, "It's not really on us as governors to govern and … force it on you but to empower you so that you are doing the right things for yourself and for your organization."
There's a ton of interest in how generative AI can play in data mesh but it's still far too early for it to have a big impact.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Data Mesh Roundtables - Join Us! Aug 01, 2023

You can simply use the link here https://us06web.zoom.us/j/87380091930?pwd=NVhUS3ZDOS9hSFFBbU1sQm9HQUJHZz09 or find a link on LinkedIn events: https://www.linkedin.com/events/weeklydatameshopenroundtable-da7091373106551734272/comments/

Link to past recordings playlist: https://www.youtube.com/playlist?list=PL9tZROTS_Pi7TjjhLR-sCtwQVeth6Jv7e

#244 Leading a Data Transformation the Empathetic (and Right) Way - Going Far Together - Interview w/ Benny Benford Jul 30, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Benny's LinkedIn: https://www.linkedin.com/in/bennybenford/

Benny's Substack: https://www.datent.com/p/elevating-data-to-a-profession-why

In this episode, Scott interviewed Benny Benford, the former CDO at Jaguar Land Rover (JLR) and who is currently building out a community around data transformation. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Benny's point of view:

!Controversial!: As a data leader, if you want to be effective, you will have to focus much more on what role data work plays in the organization rather than the data work itself. It's about partnering and communication, not SQL and LLMs. That can be a hard lesson but it's crucial to being successful.
?Controversial?: It's easy to try - and common to want - to separate data culture and organizational culture but it doesn't really work. "Data culture is … an aspect of your organizational culture that you're trying to build and create."
A guidepost for your transformation journey: "If you want to go fast, go alone; if you want to go far, go together."
The data team can be leaders in transforming the organization but they must do it in collaboration, not just going out as pioneers 'to set an example' or something. Absolutely partner with your transformation org if you have one.
When you align your data culture transformation to the overall organization transformation messaging, you aren't fighting for attention to a different message. You are using the momentum of the overall org's transformation strategy. Don't try to fight the tide if you don't have to. Try to get that data transformation as part of the org's strategy and messaging.
Given the emerging attention to data in many orgs, it can be tempting to try to go off and try to set culture. But that's almost certainly not what the company is asking a data leader to do.
?Controversial?: "Agile and data culture are two sides of the same coin." They are very close just approaching things from different angles. Scott note: I mostly agree. The big challenge is when people are overly rigid with their approaches to Agile.
If you try to force people to share data, it often stokes fears of 'can this data be used against me?' And that is a rational fear in an overly competitive org. You need assurance from senior management/the board of directors that trying to undermine other lines of business/domains will not be tolerated. If you want an open data sharing culture, you need to create clear consequences for internal political misuse.
Specifically at JLR, when they opened up a lot more access in their Tableau environment, it felt risky. But because of their well-defined and broad reaching initiative, it was nothing but positives, leading to far more/better collaboration.
Being frightened of change is totally normal. Don't try to solve every aspect ahead of time. Oftentimes you have to move with uncertainty and have a good framework for adjusting as you learn more. 'Bravery isn't a lack of fear, it's moving ahead despite your fear.' Scott note: this should be at the top of every data mesh strategy document…
Very few data leaders have strong transformational backgrounds and it can be hard to drive transformation without that. Look to educate yourself on how to drive effective transformation.
Champion's forums for your data efforts are very effective. It can inspire others but often, it can also draw out existing challenges you might not have known. Leverage those who will share their experiences to make it less scary and improve processes for those that follow in their footsteps.
All organizational choices essentially have puts and takes. Don't try to hide the weaknesses or challenges of whatever route you are taking. Be honest.
When decentralizing data work and ownership, many people in the centralized team may worry about their role and their future with the organization. Provide clarity and information on happy paths for them inside the organization including their career development.
Try to explain how the transition is happening "with you" not "to you". Easier said than done.
When thinking about data transformation, it can be helpful to think of it as splitting into projects and program. You need someone minding each aspect. One person trying to own both - the day-to-day execution and the long-term change in how your data org works - can be pretty difficult. It's a LOT.
?Controversial?: Transformation happens when a few people's main focus is leading that transformation. If you have it as a bit part of many people's roles, it's unlikely to amount to a cohesive change effort.
You don't need a huge training budget or permission to train everyone in the organization. As a data team, you can start a small training program and as that sees success, you can build more and more. There is a lot of great free content out there, make sure to organize it for people but you can get going now.
Get people used to the idea of imperfection in data - and then give them permission to do imperfect things. Don't be cavalier and sloppy but it's okay that things provide value without being 'perfect'.
EVERY organization is far more immature with data than they show publicly. Everyone feels like they are behind. Meet your peers and embrace sharing reality instead of the Instagram view of your data transformation journey. It will lead to better information sharing and probably also better sleep at night.
It's crucial for data leaders to find allies across the organization. You can't transform a 5K+ person organization by yourself. Other teams can provide you leverage and support. Partner with those willing to partner.

Benny started off talking about data culture and organizational culture. He doesn't believe you can separate the two but it is a common desire among data leaders to do so. Data culture is something you build and create as an aspect of your organizational culture. Otherwise, you will always be fighting against the tide of your organizational culture. It all needs to be integrated change/transformation. Otherwise the organizational changes will be in conflict with your data culture changes. And people are employees of the organization, not only the data team, at the end of the day.

"If you want to go fast, go alone; if you want to go far, go together." For Benny, that emphasizes the long-term transformation efforts you need to drive to a better data culture. You can go somewhere quick but if everyone is going somewhere quick, there is no cohesion, no real concerted change to the organization and thus the culture. If you have a general transformation team, align with them. Align on the overall corporate goals and leverage them to drive your data culture transformation. Yes, slightly harder than just creating your own but far more effective and long-lasting :) Yes, you want to get your data transformation initiatives as part of the org strategy but it doesn't have to be the main focus. And it might take a bit of time to make it a core pillar of the strategy but that's how you do effective change management: building to better over time.

Benny made the excellent point that while it can be tempting to leverage the attention data is getting in many orgs to try to drive major organizational culture change, it's almost certainly not what the org is asking a data leader to do. Just because there is some 'juice' there, it's not where a data leader should focus. Ask if your job is really leading the organizational change or is it about delivering on data related objectives?

While pressing for an open data sharing culture is great in spirit, Benny has seen it can stoke a lot of fears - what happens if someone looks deeply into my line of business/domain to try to find mismanagement or incorrect decisions? And that could happen in a very competitive organization. That is why the cultural guidance needs to come from senior management and the board of directors around what would be considered unethical internal use. Assuring people there will be consequences for trying to undermine instead of help other lines of business can reduce a lot of fears. When JLR opened up their data sharing, it felt like a risky proposition but really, because the guidelines were clear and it was about opening up much more information access, it went very smoothly and significantly improved collaboration.

Benny pressed on the idea that we can solve every eventuality, every challenge in a data transformation ahead of time. Of course, we all know that's not possible. People will be afraid of change, that's normal. But having a good transformation strategy incorporates that fear and addresses it by having good ways to uncover and address emerging challenges. Assure people that while you don't have every answer, you have their back.

It's also very common for data leaders to really lack experience in transformation initiatives according to Benny. While data leaders are really good at data, that lack of transformation experience can be hard. Lean into this isn't about going it alone, we're in this together :)

Benny is a big proponent - a champion you might say - of champion's forums. Essentially having a place where champions of your data initiatives can exchange information with each other and your data leadership team are crucial because you can find your existing challenge points much easier. If you take in their feedback, they feel seen and heard and lean in even more. You also have great points of leverage to inspire others based on the success of your champions.

When moving from a centralized to a decentralized structure for the data team, Benny saw a lot of fear from those in the centralized team. Would they even have a role in the changed organization? What was their career path? You don't want to lose your data talent - it's so expensive and hard to replace - so be clear about the path forward for them. There is always uncertainty but try to show them they are and will continue to be valued.

Another experience from JLR that worked well for Benny and team was that when he was hired as a data leader, he realized he probably couldn't lead all of the transformation of focusing on both the day-to-day execution on data projects and the overall change program. So he got in a second leader to lead the projects while he focused on the program aspect. Transformation doesn't happen in a vacuum, people need to be focused on that specifically. That's the program aspect. Just executing well won't transform your team and culture.

Similarly, Benny shared how they built a transformation flywheel around data - at first, it was training a few people as part of the central data team's time, training them to act on their own in their lines of business. They didn't even have budget for this training specifically in year one, two, or even three as it continued to grow to 100s of people. They started seeding the organization with data capable champions that pushed others to take the data training. And after a few years, the training was so much in demand, it was too big for the data team to own so they brought in external trainers. But you can start small and have a big impact.

Benny recommends when getting started with - or really any point along the way of - a data transformation initiative, a big benefit is to get people used to and comfortable with the idea of imperfection in data. You can capture great value even around something that isn't perfect. And it will never be perfect. That's okay. Nothing in this world that is complicated ever gets to perfect!

Deep partnering with other parts of the organization might seem obvious but in Benny's experience, it isn't all that commonly done by data leaders. The data team should have partnerships with HR, Finance, Sales, Marketing, etc. And if you have a transformation office, that should be your number one best friend internally. You need partners to move the organization forward. And that also means data leaders need skills that aren't just data skills.

Benny wrapped on that point - what is a data leader's role in a large organization? Their role is rarely to focus on the data work itself anymore and that can be a bit of gut punch for those who love data. But it's about building the bridges to the rest of the organization and helping them do their work better. That means you're still doing the data work, just with the lens of bringing it into your business partners' context. Still, it can be frustrating and hard to give that data work up. That's not unusual or unexpected, it's just part of what it means to take on that data leadership role.

Quick tidbit:

Data culture and Agile culture, Agile transformation end up being pretty similar in many respects. Look to how organizations are successfully implementing Agile transformations to inform your data transformation strategies.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of July 30, 2023 Jul 30, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#243 Zhamak's Corner 26 - The Fundamental Data Need: Autonomy with Interconnectivity Jul 27, 2023

Takeaways:

It's important to understand that we need enablement to do data mesh well. That is enablement through technology and enablement through organizational approaches/behavior changes. Doing only one will likely not work.
"...they need to move fast, they cannot be bogged down by centralization of any kind, organization or technology." Scott note: I will say we discuss later the need for centrally provided enablers but central bottlenecks are the speed and flexibility killer, look to prevent and remove them where possible.
People want to simply produce data as a normalized process of doing their job and make it consumable for the rest of the organization. How can we enable that? Why is it so tough? How do we make it interoperable - and more importantly interconnectable - too?
Right now, the missing core component to do data mesh well is an easy ability to create and manage data products. Everyone is having to cobble things together and then trying to layer on the observability, the access control, the interconnectivity, etc. But it's built on a shaky foundation.

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#242 Making Data Accessible Makes Your Data Work Successful - More on PayPal's Data Mesh Journey - Interview w/ Kim Thies Jul 23, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Kim's LinkedIn: https://www.linkedin.com/in/vtkthies/

Gemba Walk explanation #1: https://kanbantool.com/kanban-guide/gemba-walk

Gemba Walk explanation #2: https://safetyculture.com/topics/gemba-walk/

PayPal Data Contract Template OSS: https://github.com/paypal/data-contract-template/tree/main/docs

Start with why -- how great leaders inspire action | Simon Sinek | TEDxPugetSound: https://www.youtube.com/watch?v=u4ZoJKF_VuA

In this episode, Scott interviewed Kim Thies, at time of recording a Leader on the Enterprise Data Team at PayPal and now SVP, Client Innovation & Data Solutions at ProfitOptics. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Kim's point of view:

When talking about data mesh to execs, it's helpful to go back to basics: "these are the four main principles, and this is what we've built and why." Scott note: I recommend you slightly alter the phrasing, especially around "Federated Computational Governance" ;)
Look to Simon Sinek and "Start with the Why". Always investigate the why for the other party. What would be enticing to your business execs to lean in on data mesh? But data mesh for the sake of data mesh is not going to win over the business.
The communication and relationship building aspects of data work are often overlooked and will serve you better than just about any architectural or technology decision. Build the relationships so your great data work will address actual business challenges and be leveraged!
?Controversial?: Similarly, learn 'the art of the conversation' so you can extract from people their needs/wants and then see how you can help them meet those. What isn't going well where you can help is a great question to find new use cases where the stakeholder will be engaged - you're directly addressing a business challenge for them.
It's probably better to start from needing a solution and data mesh is a good fit rather than using data mesh as a hammer and looking for nails. You shouldn't want to do - or at least shouldn't propose doing - data mesh for the sake of doing data mesh.
If you have a clear business use case, it's much easier to get people engaged and keep them engaged/involved - around data mesh or data work in general. Look to a tangible benefit - e.g. cost savings is a pretty easy first use case to go after.
?Controversial?: Data teams - especially data engineers - need to spend more time "experiencing" how customers/users actually use data to deliver on their business objectives. It will lead to better outcomes and better relationships with the business leaders/teams.
Leverage the Gemba Walk philosophy: walk the 'factory floor' and talk to people far more often. Ask them how they get their work done. It doesn’t need to be overly formal, just collect information to help others do their job better with data.
You don't get to rest once you've gotten initial approval. Execs' attention will not last, they will start to focus on other challenges. Keep pointing to the business challenges you are addressing - not the data work itself - to stay relevant and near top-of-mind.
It's not as if every aspect of your business starts doing a data mesh approach when you start your journey. There will be compromises and other parts of the business will likely choose other approaches. That's okay and normal. Build your momentum and successes around data mesh but accept it won't be right for everyone, especially at first.
Data mesh will be received differently by every person or at least every persona. Each of the pillars might resonate differently. So be ready when speaking to focus on the aspect that is getting them to lean in. You need to balance the four pillars in your journey but not every conversation :)
There is probably far more transformation needed in your data practices than you expect. Even after hearing that, there is still probably far more than you expect. Processes and hearts + minds especially.
In the current business environment - likely headed for recession - you might be able to get people bought in on data mesh simply for time-savings for the data team. In downturns, cost cutting becomes far more attractive.
Once you get your mesh journey going and you have some interesting capabilities to offer, it's important to go out and find additional use cases. Even if you're proving a lot of value, people still probably need a little convincing or at least some additional understanding of what you're doing, they won't all come to you.
To speak the same language as your business partners "you have to listen first." It's pretty easy to assume you mean the same thing but even foundational phrases like data product and data contract often have completely different meanings for people within the same organization.
It's incredibly easy to overlook user experience in data. Don't fall into that trap! Scott note: we did a data user experience panel if you want to dig deeper, episode #190.
?Controversial?: Domain ownership is probably the most important aspect of data mesh because it puts the data back in the hands of the people who really know it best. Context loss is such a prevalent problem in data and data mesh solves for that quite well.

Kim started off with a bit about the PayPal journey to data mesh. They weren't looking to do data mesh, they had a specific business problem of disparate data sets across multiple domains that needed to be combined with decent governance and observability. It just so happened that data mesh was a great fit, so they went the data mesh route. And there actually was an engineering-led conversation of the engineering team looking for a use case to use data mesh but that didn't end up being the data mesh initiative that went to production. There was also a bit less scrutiny on a business-led journey because it was within a larger line of business/domain rather than it was the core data team strategy.

One thing Kim believes is that data teams, especially data engineers, do not spend enough time really understanding and "experiencing" how customers use data. They should go and pair closer to the business teams to deliver better solutions. That will lead to better outcomes but also better relationships. This is that aspect of data UX that is often overlooked. People want something to make their jobs easier or make them perform better, not just data. So how does data actually intersect with that?

To actually do that 'experiencing', Kim and team leveraged the Gemba Walk philosophy (see links above for a deeper dive). It’s a Japanese concept of going and 'walking the factory floor' to collect information. You don't need to do super formal interviews, meet people where they are and ask them questions about their day-to-day. Get a sense of what really matters and how they do work. If you just ask people what they want, they might give you the Henry Ford 'faster horse' answer versus you discovering their points of friction.

Kim discussed the challenges of not just keeping but retaining buy-in and attention to your data mesh implementation - or really any data work. Once you get the approval, there are probably things that are more top of mind for your business partners, especially compared to the specific data work parts. So keep circling back to the business challenges you are addressing in conversations to stay near top of mind. Kim and team accomplished that by getting to a really rapid, tangible proof of concept. They quickly had something to show from the work that made it clear this could work at scale.

Starting from entirely inside one business unit meant Kim and team had a lot of autonomy. As soon as they were brought more into the central data team, things changed and there was much more focus on communication and sharing - and compromising too - especially around technology choices and approaches. Many teams were taking different approaches to similar challenges with the same technology so how do you get to best practices?

One thing that really stuck out to Kim about data mesh and driving buy-in is that different pillars resonate for different personas. Data scientists or even data engineers embedded into a domain often love the self-serve aspect. Not even for consumption but being able to actually self-serve their own data production needs. They aren't afraid of the ownership principle of data mesh because they can own their own timelines and not get stuck in centralized bottlenecks. They might not even realize they are struggling pre data mesh because the central bottleneck/friction is so ingrained to their way of working, you can really make things far better for them but they wouldn't have considered asking.

Kim believes you should be prepared to market your mesh capabilities once you have them up and running. That doesn't have to be a sales-y approach but you want to go and find additional challenges people couldn't solve but now could with mesh to gain more momentum, converts, and funding. Learning the art of the conversation is crucial. Is there a way that you can advance their business goals, address their challenges? If yes, they are more likely to lean in and stay bought-in to the use case. Don't be afraid of a little sales and marketing tactics to get to better business outcomes for all.

It's very easy to use the same term and mean something different. Just look at all the what is data mesh or what is a data product content out there. So Kim believes when talking to your internal partners, start from listening first so you understand how they are using different terms. It's easy to jump to solutioning instead of understanding but your solutions will be lacking without the understanding :)

It's pretty easy to lose track of the customer experience in data mesh in Kim's view. There are so many things to focus on and honestly, UX design in data hasn't really been a thing. As an industry, we haven't talked about UX design for the platform or the data product much before data mesh started to force the conversation. If you have all the capabilities in the world but a bad user experience, are people really going to use what you've built?

Circling back to communication, Kim talked about the challenges of communicating with execs and getting bogged down in the work done. She said it's important to start from baseline communication around data mesh: "these are the four main principles, and this is what we've built and why." If you start there, then people can really start to connect the work to what might be beneficial for them. And as she stated earlier, still look to start from the listening and understanding aspects first and then always start from the why. Scott note: see the YouTube link above.

In wrapping up, Kim talked about the through-lines of the conversation. Establish and build relationships, not just data products. Learning the actual problems/challenges is key. Talk to people where they are before you try to build something for them. Understand what's actually going on and where their friction points actually are. And then also look for scalable business cases - the best way to discover those is by establishing the relationships :)

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of July 23, 2023 Jul 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#241 Data Product Success Metrics - A Kinda Deep Dive - Mesh Musings 51 Jul 20, 2023

Key summary points:

At the start, it's more important to start measuring than it is to measure the right things. Do NOT let analysis paralysis hold you back.
Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring.
Create a framework and tooling/platform capabilities - where necessary/useful - to make measuring and reporting against success metrics simple. That framework should be about defining the metrics and especially how to measure, not what success looks like for individual data products.
Use fitness functions
Good metrics to consider in order of usefulness: user satisfaction, user value, data quality, time to business decision, delivery to expectations, time to update (can be squishy), and usage

Please Rate and Review us on your podcast app of choice!

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#240 Driving to Better Healthcare Patient Outcomes Through Data - Interview w/ Smriti Kirubanandan Jul 16, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Smriti's LinkedIn: https://www.linkedin.com/in/smritikirubanandan/

Smriti's HLTH Forward Podcast: https://hlthforward.buzzsprout.com/

In this episode, Scott interviewed Smriti Kirubanandan, a Healthcare and Public Health Data Expert at a large consulting firm. To be clear, she was only representing her own views on the episode. Much of the challenges and opportunities discussed in this episode are more on the US side because of the not-so-well-functioning healthcare system there.

Some key takeaways/thoughts from Smriti's point of view:

In healthcare, it's easy to lose sight of the patient in the data - focusing solely on a condition, an area of the body, or a set of data instead of a person. It's vitally important to be focused on the data through a lens of treating the patient as an entire person.
!Controversial!: It can sound time consuming to interact with data "in a much more intimate format" much like a 1:1 conversation but it's very important to drive to better outcomes. Instead of automated decisioning, we can point our tooling to compile the relevant information better to make decisions faster without removing the care or the person. Machines making automated decisions leads to worse patient outcomes.
"Obviously, privacy is important. Ethics is important. How do we interconnect this data and how do we get to communicate amongst" the payers and providers? So physicians can look at a much more complete picture of the patient to treat them better.
There are many organizations collecting important health data about people. We need to rally around the patient outcomes instead of the financial outcomes and combine the data. Easier said than done of course.
?Controversial?: Companies with important health data need to lean forward at the table, to buy-in on collaborating around sharing data or we will continue to have suboptimal patient outcomes.
More organizations should make it possible to 'act local' relative to individual health. Instead of every decision being a very complex one, can we make things easier to simply make health progress if not 'fix' everything for someone's health. Basically, make it easier to make small decisions around more concrete and focused areas, much like a domain in data mesh.
It's very important to empower people to leverage their own health data so we have to focus on getting them access and then giving them the power to do something with their data to drive better outcomes.
There are 3 big issues we need to tackle simultaneously: 1) How do we give access to relevant and useful data to caregivers? 2) How do we ensure digital equity? And 3) how do we share data ethically?
Think about interoperability - can I pull data from one system to integrate into my system - and interconnectivity - a more two-way interoperability/integration. We need to focus on interconnectivity far more.
Especially in something as important and complex as healthcare, it's crucial for the data and engineering people to stay focused on target outcomes and not get lost in the code/work. A shared vision at the project and organization level are key.
Many data projects go wrong because we still struggle with communication. Not that we aren't communicating but keeping all parties to data work aligned and in sync as learnings emerge is very hard. And data work needs to allow gray areas, which it often doesn't do well currently.
Value-based care is a really important aspect of getting people the best care and data can help support that well. But it requires a lot of ethics and transparency to get there.
?Controversial?: Digital twins of actual people could change healthcare in major ways. If done right, it could greatly improve the ability to treat patients because of the ability to test against negative health outcomes and find more optimal treatment plans.

Smriti started out the conversation with a bit about her background and then jumped right into a key challenge in data around healthcare today: treating the patient as a person and not a set of data points and measurements. How do we look holistically at a person and focus on what would be best for their health AND life - the two are intrinsically linked. We still want to drive insights but personalized care, at least in the US, seems to be on the out and we can bring it back better with better data.

Can we actually interact with healthcare data at the person level instead of at the billing code level? Smriti believes we can - that instead of letting the automated tooling make the calls on important health decisions - such as if key procedures or tests are authorized by insurance - we can use the tooling to allow a more "intimate" interaction with what the person is going through and how can we serve them best. That we can better leverage tools to make more humane decisions for folks.

For Smriti, it's still a very tough challenge for how do we get data to interact across the various healthcare data silos, how do we smoothly exchange this data. And how do we tackle the governance of making the data interconnect? Right now, physicians cannot see a large amount of crucial patient data; but is that on patients to connect the data between offices and facilities so doctors have a more complete view? How do we maintain privacy if we are sharing information across systems? What about ethics, do we really want to give a lot of these companies very intimate health data? Scott note: see the recent acquisition of One Medical by Amazon - they are now supposedly requiring patients to waive their HIPPA rights to get care.

Because there are so many challenges around integrating healthcare data across so many systems/silos, Smriti believes that one company itself can't make that big of an impact to the overall system. BUT each company being better about doing their part can help achieve a data-driven aspect to healthcare that leads to better patient outcomes. There needs to be more of a concerted effort to collaborate in the right ways.

In Smriti's view, it's very important to empower people to make better health decisions for themselves driven by data. That means giving them access to more of their data, giving them the capabilities to leverage that data to make decisions, and then empowering them to actually act on those decisions. There are some pretty basic things we can do to improve the health of our fellow citizens and it's on a number of people to keep the pressure on to move forward on that. Not one entity can do it alone but we should all be pressing for better solutions.

Smriti talked about 3 big challenges to sharing our data and driving better patient outcomes. The first is how do we actually get data in front of our care-givers? How can we empower the individual to share that data and how can the physicians or other care-givers access it and drive better patient decisions? The second is how do we ensure digital equity? Many people don't have access to good internet. Many are not digitally literate enough to actually participate in data sharing. How do we empower them to participate in better health outcomes? And the final challenge is how do we actually share this data ethically and with empathy? All of these are being worked on but it takes a very large cross-org contingent to move things forward. Everyone can play a part but it will take a lot of collective work.

Interoperability versus interconnectivity is something Smriti is passionate about because interoperability doesn't really ensure that two systems can share information all that well - it might be that your data is in a proper format but your definition is way different than mine. Interconnectivity is about a two-way collaboration and easy integration between systems around data in her definition. That interconnectivity is necessary to really supercharge our health data revolution :)

Smriti talked about a challenge in data that many past guests have touched on - how do you keep people focused on target outcomes instead of the minutiae of the work, keeping them from getting lost in the code instead of what you are trying to achieve. It's key to have a shared vision about what is the goal and why are you doing the work. And if people lose sight of that, you need to bring them out of the weeds or you'll get very interesting solutions that don't solve actual important problem.

When asked why data work seems to not net expected results so often - the 80%+ of analytics initiatives don't meet expectations statistic - Smriti pointed to difficulties in communication. Not that we aren't communicating but the challenges around how do we quickly iterate together and share small-scale incremental learnings so reality and expectations are not constantly drifting apart. Basically, communication is hard and we need to place more focus on getting it right - and having a wider tolerance range initially of what 'right means - but there's no silver bullet. We need to be able to be vulnerable with each other and operate in gray areas :)

Smriti really believes in the concept of value-based care. But to get there, we need transparency around price and care. Individuals need to have access to their information but also need to equip themselves with the knowledge of how to leverage their data to get better care. It isn't all on the care workers.

Digital twins in healthcare is something Smriti is really excited about. A digital twin of a person gives providers an ability to potentially test reactions to different treatment protocols, optimizing positive outcomes and hopefully minimizing negative outcomes. Physicians can test a number of treatments simultaneously without experimenting on the patients themselves :) Healthcare digital twins are in their very early stages but she is quite excited about the possibilities.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of July 16, 2023 Jul 16, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#239 Panel: The Role of Data Product Management in Data Mesh - Led by Frannie Helforoush w/ Alla Hale and Jill Maffeo Jul 13, 2023

Please Rate and Review us on your podcast app of choice!

Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Frannie's LinkedIn: https://www.linkedin.com/in/frannie-farnaz-h-a7a11014/

Jill's LinkedIn: https://www.linkedin.com/in/jillianmaffeo/

Alla's LinkedIn: https://www.linkedin.com/in/allahale/

In this episode, guest host Frannie Helforoush, Technical Product Manager/Data Product Manager at RBC Global Asset Management (guest of episode #230) facilitated a discussion with Alla Hale, Senior Data Product Manager - Digital Capabilities at Ecolab (guest of episode #122), and Jill Maffeo, Senior Data Product Manager at Vista (guest of episode #151). As per usual, all guests were only reflecting their own views.

The topic for this panel was broadly data product management and the role of the data product manager in a data mesh implementation. Data Product Manager is still a very nascent role so there is still a lot of confusion around it :) If I were to sum up the feeling of the conversation very succinctly, it would be: it's early days, have patience.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

The role of the data product manager is pretty wide-ranging. It's easy to get overwhelmed and not focus on what really matters. We have to be patient as we learn best practices around data product management because it's still a nascent space.
It's crucial to focus on who the user of a data product will be - whether that is an intermediate user creating a report for another or someone directly consuming from the data product. It's very easy to lose sight of what is the exact use case and how will people use the data product - to consume information. Data for the sake of data is just expensive 1s and 0s.
There isn't even a common understanding of what a data product is, no standard definition. So data product management is even tougher to define in many ways. It is still a very emerging practice so everyone is still figuring it out. It's okay if things are a bit messy and muddled, they are for everyone else too.
Because there isn't _really_ a tangible UI (UX is also a bit muddled) to a data product, it's really hard to get a good understanding of the boundaries of a data product. People don't have an experience with data products like they do with most types of products, digital or physical, so you have to have some patience as people figure it out.
There are a ton of learnings we can bring from physical and digital/software product management to data products. Some things only need small tweaks to work well. But be prepared for lots of trial and error so make more room for experimentation than you would in software.
What is a data product team - or is there a data product team - is a question every organization has to ask. And the answer for each org probably evolves too. At first, as you are figuring out how to build and manage data products and your platform is immature, you probably have a team specific to data products. But eventually, for many data products/domains, there will likely simply be a data product developer that is part of the product team or data product development will be among the general team's developer duties and product management gets a bit more fuzzy.
Product marketing is a relatively foreign concept in data products. But it seems there needs to be far more interaction with existing and potential users of a data product to add value to existing use cases and create new use cases. As of now, that responsibility probably falls on the data product manager unfortunately.
The best path to developing a very valuable data product is 'consuming stakeholder' engagement. If the consumer isn't engaged, if they aren't giving the necessary information to really develop the data product to solve a use case, consumption seems to be below expectations across every org I'm seeing.

Other Important Takeaways (many touch on similar points from different aspects):

User Experience (UX) is such a crucial part of essentially all product management - how does that play in data products? Is it on the platform team to create the UX and the data products just plug into that UX? Is that just the UI? How do documentation, number and types of APIs, interoperability/interconnectivity to other data products, etc. factor in?
We need more adoption of forward-thinking/leaning product management practices in data. Instead of only being reactionary to requests, how are we going to extract what people want next?
Data products are so new and so many people have preconceived notions of what the phrase should mean or does mean. Be prepared for lots of work on alignment just around the definition of data product.
Be prepared for friction when introducing good software product management practices to data. Many data people aren't used to really interacting with product management/managers so they will require some time and hand-holding.
Getting buy-in for the budget to put the proper amount of data product management/managers in place might be a little difficult. Especially when you start to ask where that budget comes from. Make sure to have these conversations early in your journey and adjust as needs arise.
Similarly, a common issue in software product management is a PM is too overloaded to talk to customers or potential customers. That's even more likely to happen in data product management if you aren't careful. Really focus on making sure there is enough time for product discovery or you will be doing development by request only.
Defining data product success metrics - broadly and then specifically for each data product - is going to be a struggle. Usage is an indicator but not comparable across data products for instance. That doesn't even get into the challenge of then measuring the success metrics. But you just have to start and starting with mediocre metrics is better than not starting at all.
How the heck do we think about A/B testing relative to data and data products? Do we have enough consumers to test? What does it even mean to A/B test data?
Similarly, experimentation is key to doing data better. But can we experiment around the data itself? Or are we just using data to power and measure experimentation? Maybe experimentation around interfaces to data?
Data products have a lifecycle. If you are unsure of the value of a data product to users and you can't really get them to articulate it, sometimes shutting down the data product - at least temporarily to do the 'did anyone scream' test - is going to be the right call.
It's crucial to balance long-term product sustainability with time to market. As you learn more about building data products, many organizations are seeing decreasing time to market of new data products but especially at the start of a data mesh journey, it is really easy to create non-sustainable/maintainable products to match requests instead of needs in a productized way.
We can create data products that answer questions that 'should' be asked, but should we? If we are sharing insights that are not ready to be leveraged or are not what the stakeholders care about, what will adoption be like? Thus far, it seems like most data mesh implementers are saying adoption is below expectations for proactive data products not created to specific stakeholder-defined use cases. Hopefully that changes as we increase data literacy but important to consider.
It's crucial to consider interoperability and cross-domain usage/queries as part of your potential user flow but it's also very hard to try map ahead when you aren't sure of those use cases. We can easily lock ourselves in and reduce flexibility, which then might become like a data warehouse, just micro-warehouses everywhere…
It will be interesting to see where data product managers come from - if they come from traditional product management, they have to learn a lot of the hidden nuances of data. But coming from data, it's easy to get too tied into the data work instead of what is the data product supposed to do.
Figuring out where data product management reports will be important for organizations. It could be into the data team, it could be into product org, into the domain, or maybe even the CTO. There are definitely puts and takes to each. Personally, I think they should start reporting into the data org and eventually move to being part of the product org or reporting into the domain itself because data ownership and data products are just a part of a domain's mandate.
Frannie asked a great question: should the data product manager own the data product strategy? And what even is a data product strategy? I feel like the answer is yes but that data product strategies will be very immature as people learn how to build and manage data products. Who should own the overall strategy across all data products?
It all starts from the user and use case. To build a good data product, you need to understand how people are going to use it. But then, are we building one-off instead of universally usable data products? I think you start from a use case and build out, expand out but maintain as much flexibility as possible.
Product discovery - the process of finding out what products you should build and what are the useful features to add to those - is really important in product management. But it's far more challenging in data because everyone always says yes to 'do you want more data?' How can you determine what will be used and valued? Where is a real use case versus a 'that would be nice to have'?
Data product management should extend into what data do we want, what incremental data should we generate, not only what data do we have. As Stephen Galsworthy talked about, especially with hardware, you often don't get to change what info you are collecting. Data sourcing is going to be part of data product management.
Maybe good data product marketing is just being highly available and willing to explain - having office hours and show and tells. If trying to generate new uses cases from unengaged potential users isn't driving great value, maybe it's all about inbound marketing, not outbound.
Much of the value in data mesh is cross domain use cases and that value is captured by combining information from different data products. But it seems we are still early days in figuring out how to communicate some of the potential questions out there to design good data products to answer them.

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#238 Bringing Software Testing Best Practices to Data - Interview w/ Sofia Tania Jul 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Tania's LinkedIn: https://www.linkedin.com/in/sofia-tania/

Presentation: "Data Mesh testing: An opinionated view of what good looks like": https://www.youtube.com/watch?v=stNZQESndAA

In this episode, Scott interviewed Sofia Tania (she goes by Tania), Tech Principal at Thoughtworks. To be clear, she was only representing her own views on the episode. Scott asked her to be on especially because of a presentation she did on applying testing - especially important for data contracts - in data mesh.

Scott note: I was apparently getting extremely sick throughout this call so if I ramble a bit, I apologize. Tania's dog also _really_ wanted to be part of the conversation so you might hear us both chuckling a bit about her antics. And Tania has some really great insights so I probably asked her probably the hardest questions of any guest to date. She did a great job answering them though! A lot of the takeaways are about are we actually ready to do a lot of the necessary testing to ensure quality around data, which I don't think has a clear answer yet :)

Some key takeaways/thoughts from Tania's point of view:

We have to bring software best practices to data but we should do it smartly and not make the same mistakes we made in software, let's start from a leveled up position. Zhamak has said the same. The question becomes how but looking at how practices evolved in software should bring us a lot of learnings.
Just pushing ownership of data to the domains won't suddenly solve data quality challenges. The new owners - the domains - have to really understand what ownership means and what quality means for use cases leveraging their data.
A reasonably good way to measure if your data product is 'good enough' regarding data quality is to look at your SLOs (service level objectives) and SLIs (service level indicators). If you are constantly hitting the SLIs, you can probably focus more on new features. If not, you need to improve your quality.
!Controversial!: Consider almost a zero-trust approach to testing for data. Test as data is flowing through the systems, as data lands. Test in development as to what changes might impact data. And then consumers should be writing tests against source data to prevent issues. Scott note: that's a lot of tests but how important is certain data to your org?
In a decentralized ownership model, many data consumers are less likely to trust data - at least at first - so you need to show them why they can trust at the data product level. Leveraging proper testing and data contract strategies is crucial to being able to prove out data quality.
You should look to build out a robust testing and observability framework as part of your platform. Data product owners/domains shouldn't have to build it out manually themselves.
If you only have detection of data quality issues once data hits production, once something or someone is potentially already using it, that's an issue. Look to create ways to test data at the data product development stage, as part of the CI/CD. We can't rely only on lagging quality indicators if we want to up our data quality game.
Data for analytics and AI is even more complicated than on the operational plane. Generally, data hasn't been transformed multiple times on the operational plane so if there is an issue, it's either in the source application or an issue with the API call that was made. In data, we have to develop smarter tests as data flows through your pipelines.
Data producers need to define quality data in terms of what consumers actually want/need. Instead of arbitrarily setting quality levels, what do the consumers want?
Consumer-driven testing in data sounds wonderful. But it's hard to see teams being willing to do it :) We need better tooling and ways of working to make this easier.
Data quality surveys of data consumers are important for a number of reasons but are lagging indicators. They should be used to help develop appropriate SLAs/SLIs for data products and monitor if data products are generally meeting customer needs.
?Controversial?: Can a data producer really develop a custom test for their data product for each consumer or do the consumer owes it to the producer to develop tests to ensure the data product continues to serve their use case well? Scott note: this could start a LinkedIn war but it's an important question to ask!
If you push for consumer driven testing, don't be surprised at a lot of pushback. That happens still even in the API world where it's been more accepted for years :)
Are consumers ready and able to programmatically define what good data quality means for them for each use case? There are some tools that can help but practices and tooling are still mostly nascent.
?Controversial?: Many consumers still have the 'give me all you have and I'll sort through it' mindset. Trying to get them to lock into what they are consuming will be hard.
There can be a real chicken and egg scenario around data products, especially testing. Consumers don't know precisely what they want and what will best suit them until they see the data/data product. But building out a data product and having to change it a lot to customer feedback is also tough - producers want to build it once instead of 10 iterations. Just be prepared for this to be an ongoing issue in data and for it to lengthen times to data product release.
?Controversial?: Having your transformations handled by low-code/no-code solutions can easily hurt you more than it will help you. Be very wary. Scott note: this is coming up in A LOT of conversations recently and was featured in the Thoughtworks radar released in early May
In software development - including development for data - abstractions are crucial but can get you in trouble. Really think deep about your abstractions because it's easy to lose sight of what underlies the abstraction. And abstractions of abstractions of abstractions just compounds the issue :)

Tania started with a bit of her background, especially related to data mesh. She worked on one of the clients that was an inspiration for Zhamak's original data mesh blog post and spent 2+ years as the lead on the technical side of a data mesh implementation at another client. Her background as a developer and tech generalist have shaped her thoughts around bringing good software practices to data.

For Tania, the reason she originally put together the presentation on software testing practices in data was a client question (paraphrased): 'if we already have data quality issues in the centralized setup with clear ownership and people who really know data, how the heck are we going to _improve_ data quality by pushing data ownership into the domains?' It's a very fair question - just pushing ownership without the capability and buy-in to own the data is possibly (likely?) going to lead to worse quality. So we need tests that work and can be shown to consumers to help ensure quality and trust in that quality. Showing people your kind of data quality certification goes a long way towards trust.

In Tania's view, much of the existing data observability tooling and practices, while valuable, only really alert when there is already a problem that's hit production. Is there a way where we can shift testing left, not just in ownership but testing earlier in the flow of data? Earlier in the development timeline of a data product? So that is 3 potential ways to shift left, to test earlier. Think about detect versus protect - can we prevent data quality issues instead of only better identify and resolve them?

Tania talked about how data product producers need to start to shift their thinking around data quality. What specifically do my consumers want - and why? Quality is inherently subjective so extract from them what their needs are and look to serve those. And we should look to stop using _only_ lagging quality indicators like surveys. They are valuable in reshaping what SLAs should be and is a data product meeting needs and expectations but they are certainly not designed to quickly detect issues. But do consumers actually know what would make data 'high quality' for them?

Consumer-driven data quality testing is a good idea for many reasons in Tania's view. When we think about a single data product having 5 known, regular data consumers, does the data producer need to develop 5 different sets of tests to specifically protect against breaking changes or issues specific to each use case? Do they have to define quality metrics differently for each use case of the same data product? Do they have to be so familiar with each use case that they evolve their tests as use cases evolve? How much can we reasonably ask the data product consumers to do in the testing space to ensure quality?

But Tania admitted she hasn't led a client in doing consumer-driven testing for data. It's really hard to get data testing right in general, are people really ready for doing consumer-driven data testing? We don't really have the tooling or the general best practices to do it well yet. And there is also just philosophical pushback - being forced to programmatically say what good quality means instead of saying 'the data quality isn't good enough' is a tough pill to swallow for consumers. Do consumers really know precisely what they want? Things like the tools Great Expectations or Soda Core are a good start here but we need more. And many consumers are still in the 'give me all the data you have' kind of mindset so reducing the possible scope of data they get is not an easy mindset shift.

Tania also pointed to a persistent challenge in data that is a chicken and egg problem: data producers can't build exactly what consumers want until they get feedback from the consumers. But the consumers don't know exactly what they want until they've seen an early iteration of the data product. So you have lengthening time between conception to release because both sides need more from the other to move forward but can't until they get at least some information. A good way to press on consumers might be to ask them about bad-case scenarios - what has to be there and why? That will _possibly_ prevent kitchen sink feature requests.

As the conversation transitioned into low-code/no-code tooling, Tania lamented about the difference between ease of use and simplicity. While low-code/no-code tools can be very easy to use at the start, as scale/complexity of use cases increases, they often become extremely difficult to manage. They are focused on ease-of-use, their architecture isn't about maintaining simplicity of managing the solution as it scales. As you add more and more views, you might actually have 30-40 joins across many data products and performance comes to a halt. This was also mentioned in the Thoughtworks Radar that was released in early May 2023 (Tania contributed to that).

In wrapping up, Tania shared what she believes is a good way to measure if you are doing well enough with your data product, especially in regards to data quality. Look at your SLOs (service level objectives) and SLIs (service level indicators) - are you hitting those regularly? Then maybe you can focus more on new feature development. But if not, you might need more/better monitoring/observability.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of July 9, 2023 Jul 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#237 Zhamak's Corner 25 - We Don't Have to Jerk the Wheel - Making Smaller Correction Decisions to Get to Our Data Destination Jul 06, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Takeaways:

We should be thinking about how we can get out of the batch mode into the streaming mode. Yes, technologically but also think about
How can we get to making decisions based on smaller amounts of data more frequently - both automated systems like AI but also for our people. Instead of making adjustments or decisions based on big batches of data, we can make smaller course corrections.
"Data mesh is about building responsibility into data and the quality of the data you share and being explicit about that quality."
Make the cost of mistakes that much smaller by creating smaller decisions that add up to the bigger decisions - it's not one giant leap, it's many steps that can avoid more hazards as you come across them.
"Make decisions at the speed of the market" is crucial to being nimble, being able to react to opportunities or new challenges. To do that, we need to put data in the hands of those closest to the market, the domains.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#236 Driving Buy-in For Decomposing the Monolith; and Then Actually Doing It - Interview w/ Brenda Contreras Jul 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Brenda's LinkedIn: https://www.linkedin.com/in/brenda-contreras-9649a47/

In this episode, Scott interviewed Brenda Contreras, VP of Engineering and Architecture at Self Financial.

Some key takeaways/thoughts from Brenda's point of view:

"Iterate small and sell your solutions on a practical level."
It's kind of funny how often people in tech try to skip the communication. If you really align on communication and understanding, your business partners are far more likely to empower you to drive business value for them through engineering and data work.
?Controversial?: As an engineering/data leader, don't dictate: set the vision, explain the vision to business partners, but try to let your technical team leverage patterns that will work for them instead of only your favorite way.
Similarly, make sure your team understands which aspects of target outcomes drive value and why. They might have an approach you didn't expect but if they aren't focused on the key aspects of the outcome, even amazing feats of engineering won't create value if it's not tied to business needs.
Fail fast is very important to doing microservices right. How can we learn to adopt it in data and AI? "We need we need to be … able to experiment more, we need to be more flexible" to really drive to business value quicker and easier.
Before you start to decompose anything, it's crucial to understand what you already have. That can sound a bit obvious but if you start trying to do the work before understanding the 'before' picture, getting to a good 'after' picture is going to be very hard.
You have to understand the landscape of your systems but it's equally important to understand how those systems power the actual business. Not just services or products, but the business. What matters and why? Talk with and carefully listen to your business partners to find those.
It's crucial to align with your business counterparts on big picture visions and then what aspect of the big picture you will address and get clear on why and when. This can feel obvious but it's often overlooked. Over-communicate, share your vision often and far and wide.
Your business line leaders don't care about the how of tech - most of the time at least. They care how can you support them in tackling their goals and challenges. Make your communication about changes focused on that. They don't care how the sausage is made.
It's absolutely valid, even in a data mesh or microservices journey, to have a "sustain bucket". Things don't need to change for the sake of change. 'Legacy' doesn't have to mean bad/a headache.
When you start to talk about breaking down the monolith internally, it's important to sell people on the reality: managing smaller systems is easier. Having one group that owns a set of systems - and that it's clearly defined - makes it much easier to find the information you need. Microservices isn't easy but it does make things much easier to manage at the micro level.
Save talking deep tech or data for the people doing the tech or data work - when you talk to business partners, abstract that for them. They don't care about the particulars of your system for sending emails - they care that the emails are reliably sent and tied to the business aspect.
When you first start moving towards microservices, look to start setting up your domains around your products. Then you can start to break your systems down into microservices from there. You should think about data mesh the same - don't start building the data products without knowing the exact owner and why they own it.
Don't try to toss out the old in favor of the new. Look to net new on your new platform or approach. Just because something has been around for a while doesn't mean it isn't valuable as is. Look for where changes will make a big positive impact.
Be very ambitious in your vision for where you could go. But take it at a reasonable pace - don't be in so much of a rush that you create more challenges than you address.
You can sell the value of experimentation at the experiment level. It's hard to just get an overarching experimentation budget but constantly showing the value of fast experimentation will make it easier to get the time to do more experimentation in the future.

Brenda started with a little about her history, mostly on the app development and software engineering side before moving to Self where she's added data/analytics to her architectural focus. She learned a lot working in the office of the CTO at Charles Schwab about how to decompose a monolith and not become overly rigid around the philosophy of microservices at the same time - how do you still have shared services? How do split into product domains? How do you make sure everything plays nicely together internally and for the customer?

That learning about how to share data well on the operational plane is serving Brenda well in general at Self. When she first arrived, creating an inventory of what already existed and how it worked together was crucial to start decomposing appropriately. What were the systems yes but especially how they powered the actual business. What really matters and why? What are the key strategic initiatives and how can you positively impact them by improving systems? A 99% improvement in latency on something that drives no change to how people use systems is wasted if impressive work. It's okay for things to live in a "sustain bucket" as you focus on key business drivers. And you can only really find those key drivers by listening to your business partners.

When Brenda started to consider where to start with decomposing the operational systems at Self, she first looked to where they might "run out of runway". What were the systems that were likely to cause trouble driven by growth? Because you don't want to have to tell your business partners they have to stop growth 😅

A key aspect of decomposing a monolith is getting buy-in. Brenda had a lot of success with showing people the benefits of smaller systems that are easier to manage. Deploys are easier. Maintaining a smaller database is easier. It's easier to figure out who owns what - if someone needs information, they know who to go to. Etc. And extracting the information and then explaining your plan for change should be done in the language of the business. They don't care if you are switching your backend to Cassandra, they care about what impact changes will have on the business and what changes for them.

Brenda recommends that when you start to look at decomposing a monolith into microservices - so on the application side - you should start by breaking into domains first. Without clear owners and sets of microservices, you start to split things off without a clear forward path. Once you have the domains identified, you can start to move necessary capabilities into your microservices.

Differentiating baby and bathwater - what should get tossed out and what should be kept - is crucial in Brenda's mind when you are doing any kind of transformation. There are existing solutions that you don't need to migrate immediately because they're fine as is. Again, don't look to make changes for the sake of changing something. Look to net new use cases as you start to decompose and eventually those legacy use cases will get reimagined and replaced.

It's very easy to focus on the wrong things in Brenda's experience - instead, focus on the conversations with business partners and work backwards towards what really matters. Then the work follows from that. E.g. in an acquisition, integrating systems and/or migrating the acquired company to your systems seems to be a major focus instead of integrating what will drive value and keeping things again in that "sustain bucket".

Brenda's secret sauce to aligning with business partners comes back again to communication and understanding the big picture of the organization, how their part of the organization fits in, and what are their priorities. Essentially, it's taking the vision for where we want to go and breaking that into manageable projects to move forward on. Always be communicating priorities and timelines and working with people to realign at the small and large scale vision level. And, a biggie is to _really_ listen to your business partners and C-level execs when they talk challenges.

"We need to be … able to experiment more, we need to be more flexible," Brenda said relative to figuring out how to drive more business value around data. The fast fail approach in software is crucial to getting to business value fast - how can we adopt that in data? Partly, it's again going back to communicating well with your business leaders through good storytelling. She said, "iterate small and sell … your solutions on a practical level."

In wrapping up, Brenda again emphasized the importance of good communication and collaboration with your business partners. It's so easy in data and tech in general to let that drop to focus elsewhere. But when you are all at least reading from the same book, even if there is some disagreement in strategy, people can at least make informed decisions. Don't try to skip the communication with your business partners!

Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of July 2, 2023 Jul 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#235 Decom-mesh-ioning - Appropriately Decommissioning Existing Platforms in Your Data Mesh Journey - Mesh Musings 50 Jun 29, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

So here are the summation points of this episode:

Your legacy data platforms probably aren't going anywhere anytime soon. Those can have long lives but look to shut down unused capabilities
Your new projects, where applicable, should be data mesh data products. But a large percent aren't going to be that at the start. Figure out some incentives for people to push their new data products to your mesh platform early if possible
You should not look to lift and shift data projects/assets/products unless it is truly easy for all users - producers and consumers - lift and shift sounds great but it doesn't work well in all but the rarest of cases
Be prepared for people to get concerned about having to migrate early - communicate strongly that you aren't forcing migrations of existing data work

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#234 Doing Data Work That Matters: Perspective From a Line of Business Head - Interview w/ Iryna Arzner Jun 25, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Iryna's LinkedIn: https://www.linkedin.com/in/irinakukleva/

Mobey Forum: https://mobeyforum.org/

In this episode, Scott interviewed Iryna Arzner, Head of Group Customer Growth, Retail Banking at Raiffeisen Bank International (RBI). To be clear, she was only representing her own views on the episode.

Scott note: I mostly use the phrase line of business or LOB instead of domain in this write up but they are mostly interchangeable.

Some key takeaways/thoughts from Iryna's point of view:

As a line of business head, data has value but only in so far as they can use it. If it's not aligned to a use case or business need, data work can be more of a distraction than a benefit.
It can be very interesting for a line of business owner to know how much their data is worth to other parts of the organization - that could drive funding for additional data work inside their LOB or even more funding than that because the LOB is core to driving business value at the organizational level.
"You cannot be successful in your data strategy if there are no business leaders that understand the value of the data and are very much determined to uncover this value." Scott note: couldn't put it better
A good way to get your business leaders more data fluent is to very closely pair with them. Sitting side-by-side on a project will up their fluency far better than any training course ever could.
"How do we get these data insights that we actually need to fuel the business strategy?" It's crucial to understand the LOB business strategy and focus data work around that. Start from the business needs and work to the data, not the other way around. Scott note: PREACH!
{For many senior business execs, they've operated on gut and limited data for literally decades. You need to partner with them to show why data makes their decisioning better/quicker instead of that the data will now make the calls. Data is a tool for them to be better, not their AI robot replacement :)
You can't force a senior exec into leaning in on data. Start from their actual business goals and work backwards towards what could be better or what can't they do unless they have better data.
It's easy to say your data work should have business value but actually get specific around driving to how your data work supports the business strategy - corporate and line of business strategy. Scott note: if it doesn't have business value in some respect, why do it?!
Far too often, especially in advanced areas like AI, data people/teams want to do really cutting edge projects. But if they aren't attached to business priorities - or even if they are but just looking at the business challenge in a way that doesn't align to business leaders - it will likely be wasted effort.
?Controversial?: Please, please, please stop building platforms for the sake of platforms. LOB leaders don't care about what cool tech you use, they care about capabilities that drive business value.
?Controversial?: If your LOB execs aren't willing to share their strategies and use cases necessary to develop planned data work, you might have to resort to bringing things to them you believe are aligned and hope they bite. Scott note: if you can skip these execs at first, they'll come around if others are driving more value through data
Related but very different, talk to LOB leaders about if they are interested in those potential insights that aren't directly tied to current business strategy/challenges. Some (e.g. Iryna) like to be opportunistic and have the cognitive load for potentially interesting if tangential insights. The key is to ask.
!Controversial!: The "time is past" where business leaders can get away with not understanding tech or data. Business leaders need to understand the necessary data capabilities to drive to their use cases. They can't just wait for insights to be dropped in their laps.
Data people need to help business leaders understand the value of a platform approach. Just because something can be addressed 'tomorrow' doesn't mean that it's the right call. What is sustainable and scalable? It's crucial to communicate that.
?Controversial?: It's okay to focus your data work on the pioneers, the data advanced domains. Yes, they might get even further ahead of the pack but they will be easier to work with. Scott note: this is actually a pretty big debate in data mesh as it can lead to building a platform only for advanced teams.
Really closely partner between the business and data teams on use cases. If you requirements dump and the data team does work divorced from the business context, it will rarely drive good value.
Part of delivering a good product or service is giving your customer something relevant to them. In data, that seems to be missing from lots of work - what is actually relevant to someone's needs and wants?
There is a tendency by many data people to defer to the data first. Instead, think about leveraging your subject matter experts more to help define what data you need or what reasonable hypotheses to test :)

Iryna started out with a bit about her background and then jumped right into it. She said "it takes two to tango" relative to the business side and the data side collaborating. She understands the power and value good data work can drive for her line of business so she and her team are often pairing with the data team to figure out how to drive value.

For Iryna, a data strategy cannot be successful unless there are at least some business leaders that are leaning in - that understand the value of data and are also determined to actually capture that value. Specifically in her area, Iryna gave an example that while in the past, banking was often done via relationship with a banker - so insights about customers could essentially be managed in the head of the banker - as people move to mostly digital, there needs to be a lot more data insights to be able to offer them the best services to their needs at the right time.

Starting from the business strategy and the use cases and working backwards to what data is necessary to support those use cases is crucial in Iryna's view. It's proving to be the case over and over in data mesh that starting from the use case is key to getting value out of the data work because you need to know what would actually drive business decisions and actions and then drive into the fine details. But it's easy to get lost in the data work AND it's just as easy to get lost in the business strategy - you need to actually get specific about the data products necessary to support use cases.

Iryna discussed that most senior execs have been operating much more on gut than on data for years to decades - you aren't going to change that overnight and you need to partner with them to make their lives easier via data if you want them to lean in. And the best way to do that is to partner on their business challenges, looking for opportunities or challenges they can't address right now without better data. And don't start with the biggest challenges first, think small and quick to prove out value as you build to bigger and better.

A common if understandable failure point Iryna mentioned was data people's desire to do really challenging work - fairly often irrespective of if that really matters from a business priority standpoint. Or even if some AI work is aligned to a key business objective or challenge, if it isn't aligned to the way the business leader thinks about the business, it often won't leveraged because they have to change their mental model of their business just to "get" it. So again, start from the conversation with the business leaders and work backwards to significantly increase your chances of your data work driving large-scale business value.

Iryna mentioned she's "allergic" to platforms built for the sake of building a platform - 'it's what everyone is doing, so we should too!' Or wanting to use fancy tech instead of focusing on capabilities. Those are red flags that the data team aren't aligned to business value or learning from past mistakes - past projects didn't fail because the platform just wasn't cool enough, it was because the platform didn't make it easy to do what was needed!

Many execs in Iryna's view won't want to engage much on data that isn't tied to their strategic initiatives/challenges. But, there are still many like her that are happy to be opportunistic. Actually have the conversations and ask the leaders where they come down on that. Some want people laser focused on what they are trying to do but you can build interesting and highly valuable partnerships with the flexible and opportunistic ones like Iryna.

"This is the past," is something Iryna said about business leaders having no clue about technology or data. Leaders need to understand at least the basics and upskill to be better able to leverage the new data capabilities from new tools being developed all the time. The data team will be needed to identify - "in a nice way" - business leader skill gaps and help them close those gaps. And it's important for business leaders to understand the benefit of platform and product thinking in data. Just because a challenge can be 'solved' tomorrow doesn't mean it will stay solved! Sustainability and scalability of solutions in data are crucial.

Iryna believes it's good to at least start your data initiatives working with the "front-runners and pioneers". That's because then you will save time and hassle - they are already bought in and capable. Seeing those front-runners drive lots of value with data will lead others to want to follow. Scott note: there is a lot of controversy around this topic - will you only build for teams that are already advanced? How far do you go with this strategy?

As to where lots of data projects go wrong, as past guests have also noted, Iryna pointed to the requirements dump pattern - the business stakeholders and data team meet, information is exchanged and then the data team goes off and works on the use case and comes back a month or two later and it's not really what the business side wanted and it doesn't deliver the value. Instead, as Ghada Richani also pointed to, the business and data side need to pair closely throughout the process as the use case and needs evolve and as incremental value is created. The data team can actually get a big boost to happiness from seeing the actual business value get created from their work. Make this whole process mutually tied and mutually beneficial.

When thinking about products and good customer interactions, Iryna says it's important to bring your customer something that is actually relevant to them at that point in time. We should look to do the same in data. And that should honestly be easier in data because your customer can - and should - literally tell you what they care about and want. And you can prototype together instead of having to deliver a final product - you need to communicate that but it's more of a partnership than just a customer relationship. Start to work together on the vision of what success looks like.

Iryna gave a great recent example of when the data isn't going to necessarily give you the best information. They are looking at a specific financial product and how to sell it better based on certain life event triggers. So, instead of making guesses and random hypotheses, why not ask the people on the ground for what they look for, what are their hypotheses as to why they are selling when and what are the best triggers to use to sell the financial product? This is how data and business processes can build off each other - you don't have to treat your data as if it should tell you the information - extract context from subject matter experts!

While she might not be the most typical line of business head in a data mesh implementation - she's more altruistic than what we're often hearing - Iryna and her team are simply excited to understand what value their data has for the rest of the organization. That might even lead to insights that get shared back to her team for additional use cases. But proving out the value of her domain's data is also likely to lead to more funding for their data work so it should have a positive impact all the same.

In wrapping up, Iryna urged listeners to tie their data work more closely to the business strategy - yes at the corporate level but even the line of business objectives. Every bit of your work should be tied somehow to driving business value and you should have visibility to that. Yes, platform work can have that but you should understand how that platform work actually drives value. And also, you should sit side-by-side with your business leaders in your data work - that will up their data fluency far more than any formal training program ever could.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of June 25, 2023 Jun 25, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona Jun 22, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

#233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

Balvinder's LinkedIn: https://www.linkedin.com/in/balvinder-khurana/

Yushin's LinkedIn: https://www.linkedin.com/in/yushin-son-30362b1/

Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/

In this episode, guest host Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44) facilitated a discussion with Balvinder Khurana, Technical Principal and Global Data Community Lead at Thoughtworks (guest of episode #135), Carlos Saona, Chief Architect at eDreams ODIGEO (guest of episode #150), and Yushin Son, Chief Architect of Data Platform & Data Products Engineering at JPMorgan Chase. As per usual, all guests were only reflecting their own views.

The topic for this panel was an architect's view of data mesh, especially from an architecture lead standpoint. There are many challenges architects face in data mesh, managing the micro level minutiae, down to the data product output and input port decisions but balance that with crucial high-level decisions. Balancing the near-term and long-term vision and roadmap/North Star.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

Balvinder said, "…data mesh expects a lot more out of architects, there would be a lot of people and process and operation management that you will have to understand." It's important to understand that architects aren't just building out the systems but the entire org capability to do decentralized data well. It's a lot of responsibility but also a lot of interesting new challenges.
As keeps coming up in many episodes, doing decentralized data/data mesh doesn't mean everything is decentralized. That's what federated means in data mesh: building core building blocks that make the work far easier but not trying to chase into every corner on every use case - that's not scalable/repeatable. Creating ways for people to collaborate and interoperate easily. Doing the glue work so people can focus on the specific value add of the use case/data product.
It's absolutely crucial to understand that data mesh is not a complete vision just yet. If you are expecting to pick it up and simply run with it like it's a playbook, you're in for a bad time. The tooling isn't exactly there yet to do this easily and even if it were, we are still in the early days of learning the patterns to do it well. It's like microservices in 2013 more than 2023.
Every team will interpret data mesh differently - and many will interpret it in a way that lets them do what they want most :) Be prepared to step in to prevent people building everything themselves and also be prepared for teams that still expect a central data team to own everything. Communication and balance will be key.
Multiple years into your journey 'we're figuring it out' will still be a common refrain. Don't make flippant decisions but you'll learn and improve your processes so leave yourself maneuvering room but don't worry about getting things perfect. Optimize for learning and iterating to better, especially on the culture side.
Be prepared to compromise. You probably won't be able to decentralize major core systems - e.g. SAP ERP - immediately or maybe ever. While a perfect setup might be what you want, driving to value sooner is where to focus. Be practical and prepared for 'better but not good yet' kind of outcomes as you implement, especially early.
It's easy, especially early in your journey, to get overly focused on one domain or use case. That can be at the architecture level, the ways of working level, all sorts of governance aspects, etc. You MUST keep a balance between the micro - e.g. down to the use case or data product level - and the macro of can this decision scale and serve the broader organization.
Your organization is not a greenfield. Be prepared to not look exactly like any other orgs. You'll have lots of constraints. Frustrating yes, but every single implementation is messy behind the scenes. Everyone is trying to figure it out and cutting some corners and/or making missteps. Give yourself a break as you learn and iterate towards value.

Other Important Takeaways (many touch on similar points from different aspects):

We just plain don't really have data mesh best practices yet. They are starting emerge in certain aspects but if you aren't ready for some ambiguity and experimentation, you aren't ready to do data mesh.
Every organization will find different balances between centralization and decentralization for certain aspects of a data mesh journey. It's okay if your balance looks quite a bit different from others'.
Your balance between centralization and decentralization for many aspects of your implementation will shift over time. Constantly be measuring and assessing if things are 'good enough' and be prepared to change. But that also gives you the freedom to experiment :)
Pre data mesh, it was hard to conceive of doing decentralized data similar to how microservices works on the operational plane. It seemed it would cause chaos and data silos. But if done well, you can give teams significant autonomy and still have a larger internal data ecosystem where data from across the organization plays nicely and can be more easily used.
A centralized platform offering means more complexities than just one team owning a massive data lake setup - e.g. managing backwards and forwards compatibility across many domains - but also has lots of advantages. A biggie can be that with teams owning their own infra, there is no resource contention - especially important in financial services around month-end processing.
The old paradigm of central data ownership and control prevent fast releases so you don't really have any autonomous work by the producing domains. And you can't keep up with things changing - they all go through a backlog that means the time between something happening in the real world and the change to the table in the data lake can be weeks to months. That just doesn't work for the speed of change in the real world.
It's crucial to understand that data mesh is about building and using new muscles, ones around decentralization. Much like training, you will probably strain or pull some muscles. That's normal and to be expected. No one gets it perfect the first time around.
Building out the interoperability at the tooling level - not even the standards to interoperate the data but the tooling to make it possible for teams to combine data across data products or source data from upstream data products - is going to be harder than you expect. Not even policy/security governance, just the way data flows independently throughout the mesh rather than one giant pipeline per use case.
Early in your journey, you probably need to find domains that are excited to give data mesh a try. Rather than dragging a domain along, look for a true partner.
Some degree of failure is inevitable in a data mesh journey. But the cost of failure - and our ability to learn and iterate from 'failure' to 'not failure'/value - is so drastically different in data mesh. Fast failure, while possibly scary, is a crucial concept to embrace. Quickly learning and iterating is key.
Once you've built up the buy-in and excitement for data mesh, your job might become hype management :D People are used to data projects being built as fast as possible but with not as much focus on sustainability - data-as-a-product - or interoperability. Make sure to keep initial expectations manageable and share why doing it right is better than doing it right now.
When building out initial expectations, maintaining flexibility for breaking changes is very helpful. You don't want to make breaking changes carelessly but as you learn how to do data mesh, it's key to be able to scrap something that you find won't work rather than support a non-scalable platform for years to come.
Right now, you'll probably have to build a lot more tooling than you'd want, including some custom glue between services. Hopefully that gets far better but right now, there is still a lot of building to do if you want to do data mesh well.
If you weren't part of the microservices revolution and evolution, really study what worked and didn't. There are many lessons learned that you don't have to relearn :)
Yushin said, "Architects are not wizards. So we make a lot of choices and some of those choices will be wrong. And we need to experiment. And we need to pivot quickly once we find out that something's not going to work." That flexibility is necessary. If you don't have any flexibility to experiment and change, your org is probably not ready for data mesh.
It's still very difficult to abstract away the necessary capabilities for managing data from the tooling. It's very easy to get overly tied to specific tools. That is costly and difficult to find specialists for many tools and becomes a blocker to scaling. Look to build good abstractions where possible. Tools are hopefully coming soon.
Your room for experimentation is much broader, especially at the architecture/platform level, early in your journey. You probably don't only want to go with more 'legacy' tools, it's okay to try some bets on more 'cutting edge' offerings. There's balance of course but you can be a bit adventurous if you manage risk from that experimentation well.
Really focus on enabling self-service for the domain teams - there are certain aspects of governance that should be enforced automatically by the platform so teams don't have to reinvent everything and then it all becomes a mess because everyone is handling things differently. And it's okay to delay tackling use cases because you aren't ready to meet all their needs just yet.
It's going to take far longer than you'd like to sunset existing platforms and the data assets/data products using them. Don't press for lift and shift. Potentially look to the Strangler Fig Pattern as you replace existing use cases. And REALLY assure people you aren't going to decommission existing data platforms overly quickly.
Look to prevent copies of data that don't have a very clear reason to exist. If you can always get the data, do you need to persist it? There's storage costs yes, but a bigger cost might be the headaches around governance - including bad actor access risk - of keeping track of all those copies… As Carlos recommended, look to push the cost of that governance onto teams that want to keep copies and see if they still want to make copies of data :)
Potentially look to leverage tools that use open formats so you don't have to do custom integration around tools. Scott personal note: this is where I think data mesh can actually push the biggest industry changes. Imagine no more crazy proprietary formats and tools just work together!
You can drive data mesh participation buy-in from consumers by showing the general friction and pain points of your existing data setup. Almost every team has a lot of pain points. Can you really trust upstream data right now or do you want a contract that provides you a lot of assurances?
Cost will be a major factor in a lot of your architectural choices. Would that things were relatively equivalent but pricing across vendors is often extremely different. Be prepared to use more tools than you might want.
Different teams will have different capability levels regarding data. There is often going to be some needs that fall back to a centralized team, whether that is the platform team or another shared central team. In an ideal world, it would all be decentralized ownership but sometimes the cost of creating a full data function inside a domain isn't worth it. That's okay because that team shouldn't be a bottleneck if it's only handling a few teams but be careful 😅
Right now, a real value differentiation of data mesh is the ability to scale with nimbleness and agility while managing data quality. But for most organizations, at least right now, their data mesh implementation isn't super cost optimized or performant. Cost optimizations are typically second order concerns as you start out but will become bigger and bigger topics.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#232 It's About the Value, Not the Data - Effectively Partnering With the Business - Interview w/ Aaron Wilkerson Jun 18, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Aaron's LinkedIn: https://www.linkedin.com/in/aaron-wilkerson-81bb21a/

In this episode, Scott interviewed Aaron Wilkerson, Senior Manager of Data Strategy and Governance at Carhartt. To be clear, he was only representing his own views on the episode. Apologies for the lawn work sounds around the middle of the episode :)

Before we jump in, this episode contains a lot of really good framing on how data leaders can actually partner with business people to drive to what matters for them. How do you extract what matters to the organization and to each specific business partner? And then how do you tie the data work to that? So while this episode is not heavy on data mesh specifics, it's really important to really considering the business partner's point of view and how to work with them to drive value for the organization.

Some key takeaways/thoughts from Aaron's point of view:

?Controversial?: Aaron (and Scott) laid out a challenge for data leaders: have a conversation this week with a stakeholder and never mention data. We get too wrapped up in the data instead of listening and understanding stakeholder business challenges.
When thinking data strategy, you should first think business strategy. At the end of the day, it all comes down to how data can support the business in its objectives, not about doing data work for the sake of data work. What are the key business goals and target outcomes?
Business people very rarely care about the how of data, the sausage making. Don't try to communicate to them about the how, focus on the what and the why. Really drive towards what are they trying to accomplish and work backwards towards what data work you can do to support them.
When working with business partners, yes you can try to teach them to be more data driven. But most of the conversation should still be in the language of the business. Data work itself doesn't drive value, how it supports business partner value creation is crucial. Make sure you understand what matters and then communicate how you will help them.
Dig into what business people are measured on, their KPIs, to best figure out how to support them with data work. How others are measuring their success is a very important indicator.
?Controversial?: Data leaders really need to understand and take to heart that data just doesn't play that big a role in the day-to-day machinations for most business people. It can play a big role in supporting their work, it's just never going to be a top-of-mind focus for most.
?Controversial?: Coming to business people with the data first instead of starting from their business challenges and goals first - you are essentially asking them to do the work of figuring out if you've brought them something of value. Start from asking them what they'd value and work towards creating that.
Focus on the data work supporting your business partners where they're at - not where you wish they were - to move forward. Speak with them and _listen_ to their challenges. Then find ways to bring data in to support and improve their work-streams.
When tying data work to the business strategy, while many people might have the same target outcomes, they all have different responsibilities. Really dig into what aspect the person you are working with needs to happen to best support them. Scott note: really important nuance to consider.
When talking to an exec, be concise. "… here's the challenge we're facing, here's the outcome we want to get to, and here's next steps. And then are you all aligned to us?"
Try to focus your data work on taking away work from your stakeholders instead of adding more things to their plate.
Just because data might drive incremental benefit, it might not be worth it. Think return on investment, not just return. Data people have to work with business people to ask the question of what would actually get them to act. A 10% improvement on one aspect of their work probably won't get buy in. Find what would get them excited to move forward and focus on supporting those initiatives with data work.
By starting from the problems people have, their pain points, your business partners will be bought in that any insights you find are likely valuable. Otherwise, if it is an incremental opportunity they hadn't considered, you need to read them in much more on the details of why the problem matters before even getting to the insight. That's again more work for them.
You will probably have to build a track record of quick wins before business partners look to rely on you for bigger and bigger projects. And that's okay, you can get to know their general problem areas well through the quick wins.
While data people often want to share everything they find, it can be a double-edged sword. Just letting people know about something can be okay but if you identify a new challenge or opportunity that isn't closely aligned with their current focus areas, you are asking them to do more work to identify if it should be a new focus. Think carefully about whether and how to bring those up.
To be able to partner well with people down the road, first make sure you "stabilize, build the trust, and get everyone kind of on the same page around … what we need to do to fix [any] leaks." Lay that groundwork and start to build off a good foundation before trying to do large data projects.
Learn to say no and to prioritize. Don't stretch yourself too thin or you won't deliver as well as you could and you will likely not build the trust necessary to really drive the value you want.
Communicating what you're doing - really marketing your work and success stories - is crucial in data leadership. If all you are focused on is doing the work, you won't build the momentum and have people coming to you to partner on work.
Focus on learning the language of the business. Yes, it would be great if everyone were data fluent but the main value drivers for most companies happen at the business level.
RACI matrices are a very useful and important tool. Scott note: this comes up A LOT - get specific on responsibilities and ownership. Overcommunicate on both aspects.
It's amazing how important making your business partners feel seen and heard is, not even 'fixing' their pain points. People are far more likely to work with people that take the time to understand them. It sounds simple - and it kind of is - but it's crucial to laying the groundwork for a good partnership.
?Controversial?: When focusing on being a data leader, when you aren't in a conversation with a fellow data person, you need to lean in to talking in the language of the business. Too many data leaders are still entrenched in the ways of doing data instead of supporting the business via data work.

Aaron started with a bit about his background and how he's been upping his game around better partnering with the business and focusing more on the strategy aspects - data and non-data strategy. He was hitting a career ceiling from focusing so much on the data work itself and communicating by showing people the data. Since he started digging deeper into data strategy, everything points to the business strategy, so the last few years have been about getting smarter on aligning the data strategy to the business strategy and doing data work to support key business objectives. What are the target business outcomes and how can data / data work support those?

For Aaron, when he looks at companies' business strategies, it's pretty rare to even see the word data featured. Maybe it's to become data-driven but it's certainly not 'build a data warehouse' or anything at the tactical level. It's drive revenue, reduce costs, etc. So a key approach is doing the logic gap work - what do they care about and then think how data might support that. At the end of the day, they care that they get help on their key initiatives, not whether you built a data warehouse or leveraged some new paradigm or tool to drive that help.

To best partner with business people, Aaron recommends digging into their incentivization. Not just their goals but what are the measures of their success, their KPIs? How can you support them in doing a better job at what is most important to their role? Understanding what drives them will help you best understand how data work can support them, which in turn should support the business strategy. And it's crucial to really take to heart the fact that data just isn't that big of a part of the day-to-day work of most people inside the business. We want data to make that work more impactful but trying to get everyone to be a citizen data scientist or whatever is just not going to work or drive good business value.

Aaron talked about two different ways of bringing data to business people - starting from a data first approach or starting from the business pain points and working to the data. If you come with the data first approach, where you've done some analysis and ask them if this is of value, you are asking them to do more work to evaluate if it's useful to them. That's probably not going to win you many friends. That's the tail wagging the dog. The business people care about certain things, start from listening to them and then working to support them via data. You want the data to make their lives easier, not to add more work to evaluate what you've found and for them to figure out if it even matters to their strategic objectives and KPIs.

In a lot of the same vein but from a different angle, it's really important to meet your business partners where they are with data. Look at their work-streams and their goals/challenges and find ways for data work to better support what they are doing instead of trying to reinvent the wheel or push them in an entirely new direction. The more friction to change, the more likely even potentially very valuable insights won't drive value because it won't drive actual action.

Aaron made a really good point about while the organization may have target outcomes, each person working to drive to those outcome has different responsibilities - and measures of success - and it's important to figure out what aspect they own and how you can help them specifically with data. And be concise! He said a good way to think about it is in 3 slides to get your point across: "… here's the challenge we're facing, here's the outcome we want to get to, and here's next steps. And then are you all aligned to us?" Try to focus on making sure you are reducing workload for people in most instances rather than again making them do more work to get to value. Find ways to make it easier for them to focus on more and more valuable work and automate the important but rote stuff.

Understanding business partners' goals is only one aspect of partnering well according to Aaron. You have to understand the magnitude of an improvement that would get them to move. There is friction to change so a 10% improvement is often not going to be enticing enough. Think about Aaron's example of a rival cable company offering you a $2/month discount to switch. And you also need to understand how would you communicate opportunities to your specific business partners to spur them to action. How do we market our data-driven insights and improvements?

While we would probably all like to make major impacts with data, Aaron has seen it's more likely you need to put together a track record of small quick wins to gain the trust of many business partners. Once you've done that, they will be more ready to partner with you on bigger projects. And while you get to those quick wins, you can more easily understand what drives them and where there pain points lie so you are better prepared for a bigger project when they bring it to you.

Aaron talked about the difference between bringing up new challenges you've found and planting seeds for something that might be important down the road. Planting seeds - letting people know a few tidbits of information along the way - is great to lay the groundwork for future work together. But most people will _not_ thank you for bringing a new challenge to their attention if it's not a major unknown issue or opportunity. They are focused on what they are most aligned to, they have to do more work to assess if the new opportunity or challenge you brought to them is worth their time. So consider carefully whether to bring something like that up and if yes, how. Again, try not to give people more work unless it's of significant value to them.

To get to a place where you can work on high-impact, large data work, Aaron recommends making sure you build from a solid core. Look to stabilize what data work is already occurring, look for and address value leaks, and build trust. Then you will get to a better partner position. And look to stay away from constantly trying to generate ideas instead of listening to your business partners' ideas. They know their business better than you do and will almost certainly have better intuition about where to focus.

Aaron also believes it's really important to prioritize and learn to say no. Don't stretch yourself too thin. Again, focus on stabilizing and then start to work with people on future state but don't add too much to your plate. Saying no is an incredibly powerful tool.

Another powerful tool Aaron recommends is marketing your work internally. Many data people fall into the trap of trying to focus simply on doing good work and not sharing what they are doing widely enough. A lot of data people got in to data to focus on doing interesting work instead of heavy communication work :) But it's important for data leaders to constantly be sharing what's going on and just communicating in general. To understand where to focus your work, you need to understand people's needs. And you need people aligned with what you are doing and why. Talk to them about what are the top challenges, what you are doing to address them, etc. Eventually, you can increase the data fluency level of your business partners to a degree where they can help you make strategic decisions too but that only comes with constant communication.

On the topic of data fluency, Aaron believes it's probably better in most organizations for the data people to learn the business language and what drives the business rather than trying to teach a lot of deep data concepts to hundreds or thousands of business people. The business processes are what drive value - yes, data can significantly improve the business processes but without understanding which ones to focus on and how to improve them, does the data work really have much value?

Aaron, as many past guests have also noted, really recommends leveraging RACI matrices - responsible, accountable, consulted, and informed. Get really specific on who is responsible for what - get explicit and over-communicate to make sure there aren't misunderstandings or gaps.

While it gets said a lot, making people feel seen and heard ends up having a large impact on partnering in Aaron's experience. "You're more likely to work with folks who you feel understand you versus people who were talking about something … you don't at all care about." If you learn people's pain points and talk to them, even if you can't address them right now, they are far more likely to lean into the conversation and future partnership.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of June 18, 2023 Jun 18, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#231 Zhamak's Corner 24 - Can We Change Mistakes to "Happy Little Accidents" in Data? Jun 15, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Takeaways:

We can do better in data than what we did learning decentralization in services/the operational side: "We have to level up. We can't repeat the past mistakes. Let's not be silly and fool ourselves just because we have a schema, now we have an amazing system."
The services world has learned good ways of communicating between producers and consumers. We should look to learn more from them and look to adapt then adopt what works well.
We need to change our approach to measuring and reflecting on past decisions - we might have made a decision based on not great information, does that mean the decision was bad simply because it didn't work out? Probably not, but as Ari Gold said in Entourage "There are no asterisks in life, only scoreboards…" Can we really get to a place where we allow those asterisks?
Zhamak believes we can adopt many software development practices across data - that's pretty key to data mesh - but one area people seem to be skipping over are things like decision records - what were you thinking when you made a past decision, what did you know and what were your hypotheses. It's easy to judge results but it's better to judge the judgment :)

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#230 Getting Real About Data Product Management in Data Mesh - Interview w/ Frannie Helforoush Jun 11, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Frannie's LinkedIn: https://www.linkedin.com/in/frannie-farnaz-h-a7a11014/

Post on the Product Trio concept by Teresa Torres: https://www.producttalk.org/2021/05/product-trio/

In this episode, Scott interviewed Frannie Helforoush, Technical Product Manager/Data Product Manager at RBC Global Asset Management. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Frannie's point of view:

There is a difference between the product mindset and creating/maintaining data products but both are very important to exchanging value through data. We should be looking to apply the product mindset to all aspects of our data work, not just how it applies to data products specifically.
To do data product management well, you should look to software product management practices and recontextualize those to data. Many map well but some don't. It's not a copy paste, think through what should be applied to data differently.
The data product manager needs to serve as the bridge between data producers and consumers, making sure consumer requirements are satisfied much like with a software product manager where they are the bridge between software engineering and software users.
If product management is the intersection of business, tech, and user experience (UX), how should we think about that for data product management? Tech and business are easy but there isn't a user interface (UI) so think of UX in terms of data fluency, access, and documentation.
Relatedly, documentation around data products is more important than traditional software products because there isn't really a tangible UI for data products.
Data discovery platforms are very useful for data consumers because they are not only for discovering that data products exist but they are also great for discovering information about the data products - they provide a good initial understanding of data products before digging deeper.
In data mesh, you shouldn't think of one single data platform to rule over everything. Users care about capabilities and their workflows/UX, not about exactly how things are stitched together on the back-end. Don't have 100 platforms but don't also create a monolithic beast you can't evolve.
?Controversial?: Product discovery - the act of discovering what product(s) you should build - is crucial and often overlooked in the data product space. Many organizations are waiting for requests instead of data producers discovering what data products users might want.
While we should contextualize product management functions and needs to data product management, oftentimes the terminology gets all jumbled up. Be very explicit in defining terms to other stakeholders to prevent as much confusion as possible - but there will still be confusion :)
To help guide you in creating the right data product(s) to serve a use case, you really need to work closely with consumers - what are the day-to-day business problems of the use case? Then you can start to work backwards towards creating the right data products to serve the use case.
Make sure to dig into who will actually use a data product and how will it drive value for them. There are far too many business processes that are wildly inefficient or even unnecessary simply because people aren't asking - or asking again - those questions. Don't let your data product be inefficient, be intentional in asking.
Leverage the 'Product Trio' concept for data product management. While you won't have a design lead, find someone that can still help to optimize the data user experience (DUX) instead of the UI as you would with software. Again that UX is data fluency, access, and documentation.
Use product management frameworks to deep dive into potential data users' actual challenges. Get specific so you can find points of leverage to improve the process. To actually drive value, you have to drill down.
A 'fast fail' culture/approach is crucial to getting the best value from data. Producers and consumers can quickly test what works and iterate towards better instead of making data work all or nothing. Failure in data has to be something we are okay with.
Potentially the hardest aspect of data product management is defining success and the metrics to measure your success. Don't feel bad if you are struggling here too :) It's okay to start with less than great metrics and iterate towards better.
Enabling self-serve for consumers of data products should fall on the platform rather than on the data product owner. If the platform isn't robust enough, you shouldn't make every data product manager build out those capabilities to make up for it.

Frannie started off with a bit about her background and how that has helped her adopt the data as a product mindset, applying product thinking, when looking at creating and maintaining data products. A lot of it is about mapping traditional software product management to data product management as many things translate pretty well 1:1 but lot of things _don't_. And knowing the difference is a crucial point of leverage. The data product isn't the point, it's merely a way to exchange information to support a use case. In software product management, the product manager "serves as a bridge between software engineering and software users". In data product management, the product manager should serve as the bridge between data producers and consumers to ensure consumer requirements are satisfied.

Martin Eriksson's idea that product management is the intersection of tech, business, and user experience is foundational to Frannie. However, what actually is a user experience (UX) relative to a data product? Much of the user interface (UI) is actually owned by the platform itself rather than the data product so that makes UX a little more nebulous. She believes we should break down the UX into three categories: data fluency, access, and documentation. Access is the easiest because it's about making sure people can request access and can grant access easily - yes, automatic access is great too but that's not always possible. Data fluency is about how well the data product communicates its information to target audiences. Can people understand what the data product contains, why it exists, how things are organized, etc.? Lastly, unfortunately it seems documentation is harder and also more important in data product management than software product management. In software there is a tangible user interface but that's not really feasible for data products so the documentation is the way to guide people to understanding.

For Frannie, data discovery platforms are really valuable for consumers to understand what data products exist. And once they find data products that might be of interest, the platform allows them to learn more about the data product. Companies should make it easy to find the most useful information about a data product all in one place, e.g. the lineage, the purpose of the data product, sample queries, sample data, etc. as well as request access. Scott note: this is a partly a trapped metadata challenge. Many tools trap their metadata so it's hard to bring in things like SLA observability directly into the data discovery platform.

There is a big difference between data discovery and product discovery - or in this case data product discovery - in Frannie's terminology. In software product management, product discovery is about finding the needs users have and looking to fill that with product{s). Basically, discovering what products or features of products should exist. This is often overlooked in data product management because so much of the genesis for data products is defined use cases where the consumers come to the producers with their use case at least somewhat well scoped and understood. But data product discovery is a very useful practice to at least investigate if not lean more heavily into. Scott note: I've been saying this for literally over a year - communicate and spark the art of the possible with data consumers!

Frannie talked more about the process of discovering the right data product(s) to create. You want to dig in deep with the potential users. They might have ideas about what data they want but the most important aspect is what is the day-to-day challenge or opportunity they are looking to address? What is the actual business process and how do they want it improved? That will help tell you what data product(s) you need to create for the use case and how you need to shape it to address the business process they want to make better. Essentially, consumers are experts in what they are trying to fix, don't rely on them to be experts in what data to share and how, that's for the producers to own.

The 'Product Trio' concept from Teresa Torres has really been helpful to Frannie and team around data product management. While it is coming from the product management space where the trio is the product manager, the technical lead, and the product designer, it needs to be altered for data products since that designer role doesn't make as much sense when there isn't a direct UI. So you still have a product manager, a tech lead of some kind, and then someone that is more aligned to the user, maybe a data scientist or a data architect, to help ensure a positive data user experience (DUX). Again, Frannie breaks that DUX down into 3 components: data fluency, access, and documentation.

Frannie recommends looking at product management frameworks for leading product discovery sessions, e.g. the value proposition framework. Frameworks can help you get to actual tangible information about business processes and their challenges. Think about concrete ways to qualify and potentially even quantify aspects of the business challenges. Get pretty specific instead of a laundry list of potential use cases, drill into one or two and figure out how to help. Scott note: this seems to be a recurrent issue in data - lack of specificity around use cases and needs. It's part of why requirements gathering fails. Do not skimp on this process :D

Failing fast is a concept Frannie believes we need to adopt - and maybe adapt - in data. That's what prototyping needs to be better able to quickly get to value. Failure has historically been somewhat of a catastrophe in data because so many things have been essentially all or nothing with huge budgets. But with things like data mesh and in general with collaborative iterative data work between producers and consumers, failure doesn't mean forever fail, it leads us to places to improve. So while we might need new terminology around fast fail - data people seem to hate the concept of failing at all - it's a practice that will help us quickly iterate towards better solutions and more value. A fast fail culture with smaller blast radii can really highlight how people will use a data product instead of the pre-planning phase where you try to come to how both sides _think_ a consumer will use a data product.

Frannie shared what probably all data product managers feel but few hear: it's very difficult - and probably a bit squishy - to define success for a data product and then also define the metrics to measure success. E.g. just pointing to number of users doesn't necessarily equate to value. User satisfaction is a useful measure but obviously it doesn't cover every aspect. How well are you doing at hitting your SLAs? It's okay to start with non-perfect metrics as long as they are directionally valuable. Scott note: I recommend listening to episode #95 with Dave Colls on fitness functions if you want to dig deeper into success metric tracking.

Quick tidbit:

It's important to build out your data consumer self-serve capabilities to give consumers a better feeling of being able to actually get necessary access. And it's even more important to make sure that is handled by the platform and not the individual data products.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of June 11, 2023 Jun 11, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#229 Making 'Agile' Work in Data - Mesh Musings 49 Jun 08, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

My overall point here is that why do so many folks in data hate Agile? Because in data it's so rarely done well basically. Things are done because 'that is the way they are supposed to be done' instead of 'because this will make our teams happier and more efficient'. And quite honestly, Agile isn't for every organization. The spirit of Agile probably should be for every organization so maybe go read the Agile manifesto but in data, the one size fits all approaches are obviously breaking more and more. So work with your teams and talk about what you want to achieve and collaborate with them to get there. Yes, easier said than done but I believe in you.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#228 Keeping Your Eyes on the Prize: The Data Value Chain - Interview w/ Tina Albrecht Jun 04, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Tina's LinkedIn: https://www.linkedin.com/in/christina-albrecht-69a6833a/

In this episode, Scott interviewed Tina Albrecht, Lead Coach for Data-Driven Transformation at Exxeta.

Some key takeaways/thoughts from Tina's point of view:

Always start from your value chain - how do you actually generate value from data work? Any process or other tool you attempt to leverage that isn't focused on improving your data value chains will likely be ineffective in generating value. And why do data work if not to generate value?
Your two most likely reasons you are losing value in your value chain are lack of clear ownership/responsibility and bottlenecks. Look to regularly assess both.
When measuring if things are good enough, generally the DORA KPIs are good measures of data process maturity. But also look at two aspects: A) how happy are people (from customers, decision takers up to the team) with the current process. Satisfaction is a great measuring stick because it is highly correlated to effectiveness. And B) how much effectiveness is lost to bottlenecks and constraints.
The two ways most data mesh implementations seem to be going wrong are a misinterpretation of Team Topologies and lack of teams owning responsibilities. On the first, there are often breakdowns in how teams collaborate together and on the second, we need the platform team to own enabling domains but the domains keep trying to push work back to the central platform team.
It's important to regularly assess if aspects of your data transformation are good enough for now. But it's also very important - and easy to lose sight of - how are your teams feeling during the transformation. If you significantly improve capabilities but everyone is miserable, will they leverage those new capabilities? Scott note: interesting perspective and it factors into the rate of transformative change: constant seismic shifts are probably not great for morale.
In data mesh, you should consider rotating your data engineers between domains so you can have a larger group of data capable people that truly understand domain challenges. Scott note: we have to still make sure we give embedded data engineers a career trajectory or they will leave.
Many domains will truly not understand what data ownership really entails. Work with them and embed highly data fluent people that can raise the domain's capability and understanding of data ownership.
!Controversial!: Just having a simply community or guild for your data engineers and having them become part of the domain is not enough. There should be a central organizational home for data engineers and you should embed them into domains but the central team should handle their career trajectory.
Team Topologies is very helpful in data mesh but if you don't understand your actual value chain, it won't work well. You need to start from how you generate value.
When considering if data mesh is right for an organization, if the organization doesn't have a clear vision of what would be better if they had better data capabilities, that's a major red flag. Do they understand how data can drive value for the organization specifically? Would data mesh align to their business strategy?
Clarity - getting deep into how things actually interconnect - is crucial to doing a large scale transformation like data mesh right. Too often, things are left up to vague interpretation and balls get dropped. Drive to specifics.
?Controversial?: Overcommunicate. Set literal mandatory information exchange mechanisms - e.g. at the person-to-person level in workshops - between domains. Get explicit that domains need to be regularly exchanging information, not just the 1s and 0s of data. Scott note: this is an emerging pattern from teams who have cleaned up problem data mesh implementations. It seems like it really works.
?Controversial?: When trying to change your data processes, confusion is actually a positive sign. It indicates people are listening and trying to figure it out. If everyone thinks they understand all the changes without any confusion, they probably have wrong implicit assumptions. Scott note: human communication is silly sometimes but this is an excellent point.
While it can be hard to exactly measure the value of data, you should always be asking what is the purpose of the data or data work. What value does this provide, why are we doing this work?

Tina started with some of the big questions you will need to consistently ask yourself in a data transformation: is this good enough? Is this good enough for now? What is changing and how is it changing? What are the milestones we want to hit and, looking back, that we have hit? And often overlooked: how is/was the team feeling during the transformation? Those questions can help you narrow in on how well your transformation is going - and let's face it, data transformation doesn't stop. A good three layer system to think about when breaking this conversation down is: the surface layer of what happened in the past and where things are now; one layer deeper is where could they go if they had stronger data capabilities; and the bottom layer being okay, but how can we get there?

When actually measuring whether a current solution is "good enough", Tina recommends two measures or questions to consider. The first is: how happy are people with the current process? Happiness is a decent measure of effectiveness. If people are happy, working hard to improve the process probably doesn't make financial sense, the return isn't there. The other aspect is to think about how much effectiveness of the process is lost to constraints and bottlenecks. This analysis will give you a good insight into the process and provide the perspective on if it is currently good enough.

Tina talked about two ways most data mesh implementations seem to be going wrong. The first is how teams are interacting with each other: the team and domain setup. There are often breakdowns in how teams collaborate together so things that should be explicitly owned and bridged between the teams just keep getting dropped. The other is data ownership where domains won't take data ownership and/or don't understand data ownership. We need the platform team to own enabling domains to leverage the platform but the domains keep trying to push work back to the central platform team or not owning data well enough so the central team has to step in to help. If that happens, the platform team can't get the necessary platform work done to do data mesh and they become that centralized bottleneck again.

In data mesh, Tina believes there is still a lack of understanding by domains of what data ownership means, what they are actually supposed to be responsible for. You can help domains better understand data ownership by making sure they have necessary embedded data engineering talent within the domain to actually be capable of owning data as more domain members learn how to own data. And you need strong governance capabilities to help teams understand how to interoperate data between domains easily.

Tina talked about with one client, they are rotating embedded data engineers between domains - and the central platform team - so the domains become more data capable and you have a wider knowledge base about data across the organization as the data engineers share that knowledge with each other and the organization. And just having a simple community or guild for the data engineers wasn't enough, they needed to go with the embedded model instead. That central hub and a central team managing the data engineers' careers has been very important to keeping people happy.

Similarly, Tina talked about how while Team Topologies is a great tool for organizing your teams in data mesh, it's only a tool. If you don't understand your value chain, if you don't really focus on how you create value via data, it won't save your data mesh implementation. Start from value first.

Is data mesh right for your organization? When Tina is assessing that question for clients, a great starting question is simply what changes, what value would come from doing data mesh? If there isn't a clear vision as to what would be better and how that would drive clear value - and a large amount of value too, data mesh is not a light undertaking - then will data mesh really align with the business strategy and drive value? Scott note: I think these questions are REALLY crucial to answer. If you don't know what will change if you do this well and how that ties to business value, you shouldn't do it :)

For Tina, in general in data work, there are two big areas where value is blocked or lost along the value chain. As she mentioned earlier, lack of clear ownership and responsibility is a big one. People understand how value is generated but it's unclear who owns what and so major needs along the value chain aren't met - basically, no one thinks they own crucial aspects and so they don't get done. The other is again simply bottlenecks - where are things blocked or where are dots not connected? Once you identify issues in either aspect, you should have your areas to target for change to drive more value related to those specific processes.

A key aspect of transformation for Tina is deep clarity. There are so many things that are changing, who owns what and what outcomes do they own? What is actually being done and why? There needs to be strong governance leadership that lays out many aspects rather than leaving things to chance. It doesn't have to be heavy-handed governance but ensuring things will work together - and that someone _owns_ making them work together well - is the best way to ensure a successful data transformation, data mesh or otherwise. And you have to stay on top of things, it's not a one-and-done kind of transformation.

Intentionality around communication is also crucial to successful data transformation for Tina. Overcommunication is a virtue. Data isn't about the 1s and 0s, it's about sharing information. So you need to be explicit in setting expectations and creating mechanisms for people to exchange information, especially across domains. Have regularly scheduled workshops to actually get people exchanging crucial context - it's like a good relationship, you need to continue to work on your communication. Many think about that information exchange at the actual 1s and 0s level but we need people to exchange information with each other constantly too. Otherwise there are too many incorrect implicit assumptions and again, balls get dropped and value is needlessly lost.

Tina made a somewhat comical but very true point: when you are doing a large-scale change to how you do data work, if there isn't anyone saying they are confused, it's probably a bad sign. Because large-scale change is difficult and inherently change will be at least a bit confusing for most so if no one is speaking up, they probably have some bad implicit assumptions you need to address but you don't know what they are. At least with confusion, you can drill into where they don't get it. Lean into confusion because it creates the perfect situation to actually exchange context and drive people to the same page.

While it is incredibly difficult to provide an exact value of data and data work - it will be valued differently by different people - Tina still asks people what is the purpose of doing the work. Why do we care about this data? That will tell us the general value of it if not a specific dollar figure.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of June 4, 2023 Jun 04, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#227 Panel: Creating a Data Mesh Platform (1st Iteration) - Led by Paolo Platter w/ Manisha Jain, Jean-Georges Perrin (JGP), and Max Schultze Jun 01, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Paolo's LinkedIn: https://www.linkedin.com/in/paoloplatter/

Paolo's Medium (multiple data mesh articles): https://medium.com/@p-platter

Agile Lab's website: https://www.agilelab.it/

Manisha's LinkedIn: https://www.linkedin.com/in/evermanisha/

'A streamlined developer experience in Data Mesh' blog post by Manisha: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-platform

'A streamlined developer experience in Data Mesh (Pt. two)' blog post by Manisha: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-product

'Data Mesh Accelerate Workshop' blog post by Thoughtworks: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html

Max's LinkedIn: https://www.linkedin.com/in/max-schultze/

Max's Data Mesh Learning meetup presentation: https://www.youtube.com/watch?v=QwtTdP2wKFo

(he has many more on YouTube! https://www.youtube.com/results?search_query=max+schultze+data+mesh)

Data Mesh in Practice ebook he co-authored (Starburst info gated): https://www.starburst.io/info/data-mesh-in-practice-ebook/

JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/

JGP's 'Data Mesh for All Ages' book: https://jgp.ai/2023/01/20/data-mesh-for-all-ages/

JGP's website (lots of data mesh content): https://jgp.ai/

JGP's Blog Post 'The next generation of Data Platforms is the Data Mesh': https://medium.com/paypal-tech/the-next-generation-of-data-platforms-is-the-data-mesh-b7df4b825522

In this episode, guest host Paolo Platter, CTO and Co-Founder of Agile Lab (guest of episode #3) facilitated a discussion with Manisha Jain, Data Engineer at Thoughtworks (guest of episode #220), Jean George Perrin (AKA JGP), Intelligence Platform Lead at PayPal (guest of episode #130), and Max Schultze, Associate Director of Data Engineering at HelloFresh (guest of episode #21). As per usual, all guests were only reflecting their own views.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Scott's Top Takeaways:

As with every aspect of data mesh, you will need to focus your data platform work on what best drives value - doing so incrementally and both in the short and long-term. Look to start at common friction points in your org's data work, not just data mesh work, e.g. automated provisioning. That will give users a good first experience that is incrementally better to existing data work, potentially driving buy-in.
It's _really_ crucial to understand user personas and necessary capabilities for each persona. Who is using it, why are they using it, and what are their points of friction? That experience plane will become incredibly important as you move along but it probably shouldn't be your initial focus.
The word platform, especially in data mesh, can be thought of as a plural. Don't focus on only creating a 'single platform to rule them all', focus on delivering scalable capabilities that reduce friction and support scalability and reliability/trustability of data work by producers. Users don't care how it fits, create a system that can evolve and scale!
You must treat your platform as a product itself - think about how well it satisfies user needs, how good is the user experience, etc. But that also means your platform has a lifecycle where you add and eventually prune features. From the start, apply product thinking to your platform :)
Prioritization of platform capabilities will probably always be a challenge once you have more than a few domains using the platform. Balance being reactive and proactive with the need to generalize solutions to fit many use cases. Again, look to product management for advice on how to manage your roadmap and priorities.
REALLY think about how to handle breaking changes. They are likely inevitable and that's okay but you need to focus strongly on communication and limiting the impact / providing a gentle migration path. Don't break without really needing to and do so sparingly.
Data integration, especially in a highly regulated industry, is going to be a challenge you will face relatively early on. It's likely not going to be easy, be prepared for that.
You need to understand where there is friction in the data product creation/management process - that's where your platform should focus. That might feel obvious but it's not how data platforms have been built - basically, look to automate unnecessary friction first, even if that means focusing on things like templates and blueprints instead of cool tech.

Other Important Takeaways (many touch on similar points from different aspects):

There are many ways to potentially get started - probably too many to list. But think about getting to early necessary capabilities that deliver value. It's easy to get bogged down in technical aspects - instead start specifically by asking what creates value quickly.
If you aren't ready to build your platform incrementally, you probably aren't ready to do data mesh. You need to be comfortable with demonstrating value and building as you learn and as your needs progress.
When you are early in your journey, discoverability/usability is a characteristic many might overlook - you are building data products to support a use case. But if you want to drive buy-in and get incremental users of those data products from additional domains, discoverability might be a very important early capability. What is your adoption leverage point?
There's an interesting balance between MVP for the platform and easily enabling MVPs for data products. It's hard to say exactly where the line falls. Scott note: Glovo said in episode #139 that they wish they had focused a bit more on making it easier to launch and initially manage data products over other more advanced capabilities.
Consider when to start saying you actually have a platform - that can be a bit of a political statement. Potentially wait until you have started to build out the data experience plane for data producers. If there isn't a tangible way to interact, the platform may exist but users don't really know what it is and what it enables them to do. But the experience plane shouldn't be among the first capabilities you build either. :)
Your data platform work needs to focus on capabilities and enabling value delivery. But that value delivery needs to be visible to business users. Basically, find your value leverage points that are visible to users and focus on satisfying those when you can to drive buy-in that you're delivering value. It can be a political game unfortunately.
You can't treat domains as if they are the same or, especially early, often even similar. There will be different value drivers/needs and often very different capabilities so you have to make sure your platform has the necessary capabilities to drive value for that domain. And if it doesn't yet or won't in the near future, it might not be time to partner with that domain yet - and that's okay!
At the start of your journey, you will probably be focused on building capabilities to serve specific needs - of course, try to build those in generalized ways to serve many needs but still. As you evolve and grow the platform, you need to focus more on your overall suite of capabilities and look for where you have gaps. Don't only wait for capabilities requests - but also don't just build cool things because they are fun to build :)
Start to build out data product specifications based on early data products. Those will provide a much easier path for later data products, whether built by new domains or existing domains using the platform.
Charging models and who should own what costs are going to play a bigger role than you'd probably like. It's going to be a challenge you need to address at the platform level even while (presumably) others decide who should actually own the costs.
Do not focus too much on building capabilities to _launch_ a data product. The launch is just a single day in the life of a product. You need to build capabilities that also help domains maintain and evolve their data products.
Should the bounded context of a data product drive all the way down to the resourcing? Basically should there be zero shared resources between data products. I know this one is a bit controversial as it's a key aspect of Zhamak's vision but people worry about costs. If you believe in that complete separation, how do we make sure there aren't overlaps on computational resources and storage? It won't be super easy because it's not how most data services have been built historically.
Many data engineers will still be thinking waterfall. Be prepared. Agile can work very well if it's done well - but it's not done all that well in data all that often… SAFe seems to be less hated in doing Agile for data - it's pretty widely panned in software - but it _might_ provide a bridge to doing Agile in data. Something to investigate but definitely not advice/guidance.
Paolo essentially said (heavily paraphrased/edited for flow) 'We really need to pay attention, to optimize the ratio between the value that we bring with new features against the satisfaction of various users. This drives prioritization across phases, whether the focus of that phase is on adoption, productivity, satisfaction, etc. What is the outcome we want to achieve in each phase? Then we use that to prioritize features.'
Every mesh platform build out will be different, especially depending on what capabilities are most valued in your organization. For some, that might be security/privacy governance re regulators. For others, it might be usability. Find you value leverage points and also find what the people with the purse strings will find the most valuable.
Measuring the return on investment, even just the success, of a data mesh platform will be tough but it's important to start early and _start_ to get your arms around it. Productivity gains are a pretty easy measurement to consider as you get more sophisticated. Look to measurements of return on investment in software to get some ideas but be comfortable that things will always be a bit squishy.
If you are building your platform as a product, products focus far more on user experience (UX) than most aspects of data infrastructure have historically. Make sure you really understand the value drivers - if users don't adopt your platform, nothing else you do with it really matters :)
Producers being willing to own their data - and then understanding what that ownership really means - will probably be one of your biggest challenges. Hard to tackle that at the platform level but look to support those efforts as best as you can.
Data producers and consumers still don't generally talk to each other in many organizations. Look to potentially put more mechanisms in to foster communication. Scott note: see my mesh musing - episode #188 - where I believe all consumers MUST register their use case with producers.
You will have to decide things like do you isolate compute and storage resources at the domain or data product level. Data producers don't care, make that invisible to them. They key aspect to focus on is to enable and not block - basically, give domains the agency to get their work done but find ways to reduce their friction and obfuscate decisions like that from them. They are important decisions but irrelevant to users.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#226 Learnings From Implementing Data Mesh at a Large Healthcare Company - Interview w/ Mike Alvarez May 28, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Mike's LinkedIn: https://www.linkedin.com/in/2mikealvarez/

In this episode, Scott interviewed Mike Alvarez, Former VP of Digital Services leading the data mesh implementation at a large healthcare distribution company. He's now working on his own startup.

Some key takeaways/thoughts from Mike's point of view:

Lean in to the new value-creating possibilities that can come from empowering thousands of your colleagues to leverage data.
As an industry, we have to learn to do data work in an incremental fashion. It's not been the norm and it can break people's perception of data work but it's crucial to get where we want to go.
You can drive data mesh buy-in from domains by showing them the freedom they will have. Autonomy, empowerment, going at their own speed, etc. can get many to lean in.
Advice to past data mesh self: Early in your journey, you can share your vision until the cows come home and people will say they understand - and probably think they understand - but it's incredibly easy to get misaligned. Really focus on what you are trying to achieve. What are the target outcomes?
Similarly, it will be harder than you expect to drive buy-in. Many people say that but it's still going to probably be harder than you expect after hearing that :)
We need to move away from old approaches to data for large companies because the sheer scale of initiatives ends up creating bloat and risk factors unto themselves. Small and nimble gives us much quicker time to value delivery and builds to much greater outcomes.
Shadow IT develops to try to move at the speed of business for domains. But it's rarely scalable or robust enough to even support the domain in the long-run and it certainly isn't built to integrate well with the rest of the organization. Try not to hold past shadow IT decisions against domains.
Most teams - especially pre data mesh - don't truly understand the data they are ingesting. It's on consumers to get more information but if the producers aren't helping them, teams will ingest what they can even if they don't fully understand it. Data they don't understand well drives value but could be driving so much more value.
Start from the problem first: what am I trying to solve? Do I need a new approach or can I use something I already have? Don't reinvent the wheel but we might just have to reinvent doing data at scale a la data mesh.
Collect stories of past attempts internally with negative outcomes. What were the common reasons, the common patterns for things failing or not delivering expected value? They are useful for perspective and to drive buy-in.
Treating data as a product makes more and more sense the deeper you dig into it. But just doing data as a product can't survive on its own as an approach to doing data.
When trying to share information about data mesh, it's not like everyone will instantaneously understand or be on board. It will likely take a while in most organizations to build up the momentum to even consider starting on a data mesh journey. Have patience.
Data mesh really enables teams closest to the customer, closest to the day-to-day business, to drive more value through data. It allows them to react much more quickly as the world evolves and focus on the problems of the customer.
?Controversial?: The operating model change with data mesh is what drives the real value. And lots of domains can get bought in that they get to own their own destiny but be empowered to manage their data like a product instead in non-scalable and quickly deteriorating ways. Scott note: I think we do need better tech to fully leverage the potential value of data mesh but right now, I agree that most of the value is driven by operating model changes.
A shared vision of what you are trying to achieve is important. It lets people rally around something and start to build a community, which is crucial to delivering on a data mesh approach.
?Controversial?: Don't try to force your domains, your lines of business to leverage your centralized tooling and comply with optional governance (there is non-optional governance of course). In exchange for those who leverage central tooling, pay them back via automating away toil where possible. Community is about give-get. Be a good member of the community.
The three crucial dimensions of a product: viability, feasibility, and desirability. When adopting product thinking, you should think about does your product satisfy all three factors.
You need to communicate when something isn't feasible. Too often in data, people have just said no instead of 'no, and here is why…' Let people in on your thinking and prioritization process around what work to do when.
Good product management skills are necessary to understand data as a product - to transition us from creating and sharing data sets to sharing high value information exchanged via a data product. You need to delve into and understand the domain to figure out what would be most useful to share via a data product.
!Controversial!: It might be time to completely let go of the concept of "single source of truth." We've been chasing it in data for so long but the cost/benefit is starting to look it doesn't make sense. What are we trying to achieve - perfect data or a strong understanding of the world and how it's changing? Scott note: strongly agree and so does Zhamak.
New, more correct information about aspects of the business is not always welcome. Unfortunately, you might have pushback if you attempt to tackle a problem that changes people's view of their business. So choose use cases, especially early, well :)

Mike started off with the general need for large companies to change their approach to analytics at scale. We've been doing a lot of the same things for the last 30 years and they aren't quick enough to respond to changing business needs - 6+ months and $1M+ to get to your first query just doesn't make sense anymore - did it ever? And we can do better now. The business side of companies shouldn't have to wait for data and see the world change well before a solution is delivered. We need to move at the speed of business.

Regarding shadow IT, despite leading a central data/IT organization, Mike doesn't hold it against domains. The lines of business can't deal with the bottlenecks of going through a central team and try to build things themselves. However, it's rarely all that scalable and certainly isn't built with sharing to the rest of the organization in mind. Shadow IT just isn't built with a product mindset so it becomes brittle and dilapidated quickly. So the central team is a bottleneck but the decentralized approaches don't scale. Add in the teams generally not really truly understanding the data they are ingesting or even often producing and it's a recipe for data underperforming expectations. Of course, this is what Zhamak identified and why she created data mesh.

Mike talked about when considering a new approach to data, he didn't want to do data mesh for the sake of it. What was the problem they wanted to solve? Could an existing approach or platform do what was necessary? What were the organization's past failure modes or times when things didn't meet expectations and what were the common through-lines or patterns? And then he took those past unmet expectations and used them for understanding as well as driving buy-in. The definition of insanity is trying the same thing over and over and expecting different results. So if data projects were constantly not meeting expectations, shouldn't we change the way we approach data? And treating data as a product seemed like a great start, which led to selecting data mesh :)

While data mesh can feel like the right call to some immediately, it's not likely to be the universal reaction at any organization. Mike and team spent a number of months driving to how this could work and building up the buy-in and momentum to even start on their data mesh journey. This isn't an overnight approach, you really need to think deeply about how it could work and - back to those potential failure modes - how it could go wrong so you can prevent heading down bad paths as best as possible.

But what really drove Mike's interest in data mesh as a possible solution was how it could enable the teams closest to the customer to react to customer and market needs, especially changes in customer demands/wants/challenges. It is about empowering the teams to move at the necessary pace to stay ahead of the competition instead of waiting for a centralized team to give them access to leverage their own data or the data of teams close to them in the organization.

For Mike, the value of data mesh isn't about the technology shifts, at least not yet. It's about the operating model shift - giving domains the capabilities and empowerment to handle data. We are trusting them to own their data and giving them the ability to do so in a scalable way. We are giving them the ability to react in a much quicker and more meaningful way. All of these can get people leaning in to doing data mesh. But they don't care whether it's data mesh or any other paradigm. That's where data people need to connect the dots for them, how can this work and what benefit does it have for the domain. And what are the actual changes for them?

To get the most out of data mesh, Mike believes you have to have a strong vision of what are you actually trying to achieve. It's not an approach to take on lightly. You need to really think about aligning everyone around that shared vision and build as a community effort. How do you take the principles and new approaches and focus on delivering business value - for their own domain and the broader organization too?

Mike believes a big part of doing data mesh is kind of the social contract around enablement and empowerment. Sure, teams can go off in their own direction but if they give up some of their autonomy to stick to the centrally provided tooling - which makes governance far easier -, you need to give them something in return. In their case, Mike and team gave the gift of automating away a lot of the toil work :D

On advice to his past data mesh self, Mike talked about early in a mesh journey, people believe they are aligned on vision but they probably aren't. Holding all of data mesh as a concept and then contextualizing it to your specific organization is a massive amount of work and cognitive load. Trying to get someone to fully understand that upfront without seeing the progress, you will almost certainly have some misalignment and misunderstandings. Instead of the specifics, focus on the target outcomes, what are you trying to achieve? If people align on the benefits, you are more likely to gain and retain momentum. And it will take a lot of effort to get most people committed to the vision, just be prepared for that.

Mike talked about the three key aspects of a product: viability, feasibility, and desirability. Feasibility it a crucial aspect to consider in data - especially data as a product thinking - because often, something just isn't likely to work for a number of reasons. And when there is the desirability but not feasibility, you really need to communicate why it's not going to happen. With data mesh, there can be a misconception that the switch has been flipped and we can do any data work we can think of - and that we should! But that prioritization process and understanding - and then communicating - what is the current art of the possible is important. Always be communicating about what you are doing when and why.

Domain understanding is crucial to really understanding data as a product for that domain in Mike's view. How do we move from trying to serve data sets as if that is the product to creating the information that will be most useful to consumers about the domain in a productized way? And then iterating towards more and more value as you improve the data product or suite of data products representing the domain. Easier said than done of course.

Mike asked the provocative question of do we still want to seek the fabled "single source of truth." It can be a bit like the dog chasing its tail - when you catch the tail, then what? Are we trying to perfectly clean data or are we trying to drive value from data? Is the juice worth the squeeze or can we drive better value - and especially nimbleness - by taking a slightly different view? Scott note: Zhamak urges people to consider "the most relevant source of truth" because there are multiple perspectives on the same things that can all have value, you have to decide what is best.

Mike warned that some use cases are politically untenable or even toxic. Especially early in your journey, consider will participants actually want to know the information. Yes, in the abstract, we want everyone to be perfectly data driven but humans aren't and won't ever be. Don't ignore that and tackle something that will be more hassle than it's worth.

In wrapping up, Mike had two points. The first is learn to work incrementally. That has been somewhat of the antithesis to how data work has historically been done but it's incredibly important. The second point is to really lean into empowerment and the art of the possible. We don't really know what might happen when we empower thousands of our colleagues to be better able to leverage data. Be excited and open to the journey of finding out what value they create.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of May 28, 2023 May 28, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#225 Zhamak's Corner 23 - Driving to Resilient Data Value Now and in the Future May 25, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Key Takeaways:

Postel's Law: Be conservative in what you do, be liberal in what you accept from others.
We can do better in data than what we did learning decentralization in services: "We have to level up. We can't repeat the past mistakes. Let's not be silly and fool ourselves just because we have a schema, now we have an amazing system."
The services world has learned good ways of communicating between producers and consumers. We should look to learn more from them and look to adapt then adopt what works well.
Zhamak believes we have to learn to prepare our data for future use cases. Scott note: If she means reuse of data being generated for current use cases, most agree. If she means creating data that doesn't currently serve a use case, almost everyone else seems to disagree. Time will tell.

More on Postel's Law: https://ardalis.com/postels-law-robustness-principle/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#224 Building Out Scalable Automated Access for Data Mesh at Disney Streaming - Interview w/ Himateja Madala May 21, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Himateja's LinkedIn: https://www.linkedin.com/in/himatejam/

Himateja's AWS ReInvent presentation (link starts at her part): https://youtu.be/y1p0BGsPxvw?t=1991

In this episode, Scott interviewed Himateja Mandala, Senior Data Engineering Manager and Head of the Data Mesh Data Platform at Disney Streaming. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Himateja's point of view:

?Controversial?: Your existing data platform(s) might not be able to serve data mesh well, even with reasonable augmentation - especially if your data platform has become hard to change. You might have to build from scratch.
When the data platform's key users aren't part of the centralized team, you need to think about enabling automated capabilities by default, e.g. security the second data lands or easy to leverage and understand monitoring/observability.
?Controversial?: Data products serving different use cases often end up looking relatively different. Is your data product for dashboards and reporting/analytics; is it for serving a recommendation engine or machine learning model; or is it more for internal usage? Be okay with data products not being uniform.
Even if your data mesh platform operates outside the traditional paradigms, many data producers - especially data engineers - will still be thinking data pipelines. Be prepared for that, it's an ingrained way of thinking for many.
Data contracts are very helpful in defining and maintaining quality. If you set up good observability on your data products, owners can quickly identify when there are quality challenges.
When building out your platform, user conversations are crucial. Go and focus on pain points. The coolest capabilities in the world won't lead to good adoption if you aren't addressing real pain/needs.
Automation and blueprints are key to scalability in a data mesh data platform. Teams need to be empowered to easily do their work.
Don't only focus on creating the tooling to process and share data in the abstract, dig into how teams will share information with each other, how they will communicate. That isn't only exchanging data via data products.
Even if you have domains inside your organization that want to share data/information with each other, it is hard to get to a place where consumers can actually trust the data without a lot of explicit enablement of trust at scale. Enabling trust at scale is a key role of the platform.
Enabling teams to go at the speed of their business through owning their own infrastructure really drives good buy-in. It might take slightly longer to get something spun up the infrastructure for the first data product but they quickly learn and will often strongly prefer the visibility and control they now have.
When requests for new capabilities come to the data platform team, you need to consider how to generalize the capability to be applicable to more use cases if possible. And sometimes the right answer is the platform can't support that one-off need.
Centralize your governance capabilities in the platform but federate the decision making. There should be standard approaches to access control to make it easy for people.
!Controversial!: Disney Streaming has very strong RBAC (role-based access control) policies to make it very easy to delineate who should have access to what but if you have a certain clearance level for one domain, you have that clearance level for all domains. Scott note: this is about 35min into the interview, it's a really interesting approach. I couldn't see it working for highly regulated industries but it's working very well for them.
?Controversial?: Preventing data from leaving the mesh under any circumstances is an effective risk control - if someone somehow gets access to something they probably shouldn't have access to, the blast radius is quite contained.
If you have any data sharing agreements with partners/vendors, make sure to keep their access heavily contained. Create specific spaces - or cloud accounts - with strong rules to prevent them getting improper access to any other parts of your data.

Himateja started the conversation with the situation Disney Streaming was in that matched many organizations right now: many data platforms but not one that will really fit with data mesh, even with augmentation. So she and her team decided that because the existing platforms were too hard to change to meet the needs of a data mesh implementation, they'd need to build their data mesh platform from the ground up.

When you have new key personas leveraging the data platform, even if those are data engineers embedded into the domains, Himateja recommends rethinking how data work is done. What do people need automated and by default, like security? How do you create monitoring/observability that helps people easily pinpoint issues as they come up? How do you make data accessible by default at the data product and greater mesh level? Etc. In a decentralized, federated data approach, ways of working and needs will be different so dig into what are the actual pain points instead of solving the same pain points of previous implementations.

Himateja shared that while people may think data products are pretty similar, they end up relatively different based on use case. Audience also really mattered when trying to figure out what capabilities people required early in the journey - execs were often more focused on data privacy and security and data scientists were focused on data quality. It's hard to focus on the business context at the platform level because many people are used to doing that via request. The data products themselves need to own business context.

Data contracts are crucial to maintaining data quality in Himateja's view. While they are certainly helpful to data consumers, they are also very helpful to data producers because - with proper observability - data product owners can quickly identify and address quality issues as they emerge instead of waiting until consumers complain and downstream data is wrong. That proactive alerting and then response helps everyone better trust the data. However, data contracts are still a work in progress because not everything is easy to define in a contract, there are definitely gray areas that are improving but not great yet. Scott note: and that's okay, we can't get everything perfect upfront, we have to iterate towards better :)

Himateja then shared a lot about what the data platform team that she leads set out to do at the start of their data mesh journey. One aspect was to create a center of excellence approach, standardizing how data engineering work is done to create data products across the 15+ teams running on the platform now. They did that by starting to drill into pain points and doing lots of listening to potential users. They needed to take a different approach rather than just yet another data platform.

Preventing the central data platform team from becoming a central data engineering team was a worry for Himateja: how do you prevent being a bottleneck and empower teams to do what they need to do? Especially at the start of a journey? As many guests have pointed to, automation and blueprints have been crucial. Teams pushed back initially at the thought of managing their own infrastructure but they realized it gave them the ability to move at their own pace - no more waiting in a prioritization queue for necessary infra. Another key milestone was developing tools to make it easy for cross domain communication and data sharing. Domains at Disney Streaming actually wanted to share their data with each other and the data mesh platform/implementation made that possible - it was previously very difficult to trust data but now that quality metrics were clearly defined and tracked, data sharing and usage between domains increased significantly.

Prior to doing data mesh, Himateja shared that data engineers in domains had no real visibility into data infrastructure - provisioning timeline or any other aspect. They'd push a ticket and wait for things to happen. But with data mesh, since they own the infrastructure, they can go at their own pace and can understand much more about any delays. So they understand and better control their own timelines, which makes them far happier. And then once they’ve gone through the process of spinning up infrastructure the first time, the next time they can be that much faster. They are better able to move at the speed of their business.

Himateja shared about the process of evaluating new platform capabilities requests. As many past guests have noted, you need to establish a process to abstract away the requirements from individual use cases to find a generalized approach. Otherwise, you end up with yet another overburdened platform that you can't evolve. A specific example at Disney Streaming was enabling their Apache Kafka clusters to better communicate across domains which were leveraging individual cloud accounts. Instead of building a solution to share data for each technology they use in the platform, they built a system to better enable sharing across accounts with proper access control and privacy. And sometimes, the answer is that you can't support a unique requirement via the platform - that's okay and often is the right call.

At Disney Streaming, Himateja and team implemented a very interesting approach to access control via RBAC (role-based access control). There are a few levels of data usage clearance but if you are at access to PII level or access to financial information level in one domain, it's the same clearance level for all domains. This might not work for heavily regulated industries but it's working very well for them. The work to decide who has access to what data is done ahead of time instead of constant requests. They just think very carefully about each use case and who should have access and why but there isn't a need to manually grant access. And there is of course oversight to see how people are using data to make potential changes. Scott note: this is a really interesting approach and I'd love to hear people's feedback.

In wrapping up, Himateja shared how they are strongly limiting their blast radius around sharing data with partners/vendors. They have accounts that are not able to get access to any other accounts where they give those partners/vendors access so there is not a way for them to access data they shouldn't be able to see. It's a simple security pattern but others should consider adopting it in her view.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of May 21, 2023 May 21, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#223 What Does Your Data-Driven Org Look Like - Mesh Musings 48 May 18, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Important points to consider about being a data driven organization:

If your execs can't envision what would change if the organization were significantly better at data, you have some work on your hands at understanding their challenges first and then also evangelizing
Your domains should be leaning in and understand what being data driven means for their domain specifically. If they aren't you won't be able to really help them move forward. You can't drag a team to being data-driven.
Your execs should be aligned on what being data driven means for the organization, especially at the macro level - how does it all fit together instead of highly competent data silos? How do we focus on incremental value delivery via concrete use cases?
You should absolutely make sure your data strategy and your vision of the data driven organization ties to the actual business strategy and supports crucial priorities.
You need to make sure you have - or can build - a test and learn culture. If not, can you really be data driven?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#222 "Not All Governance is Created Equal": Data Governance as a Data Mesh Value Lever and Driver - Interview w/ Lynn Noel May 14, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Lynn's LinkedIn: https://www.linkedin.com/in/lnoel/

DAMA New England: https://damanewengland.org/

Correlation/Causation XKCD: https://xkcd.com/552/

In this episode, Scott interviewed Lynn Noel (pronounced Knoll), Data Governance Lead at AIM Consulting. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Lynn's point of view:

There is a strong data governance value proposition in data mesh around making data broadly "safer, better, and easier to use and share." Lean into that value and communicate it widely.
?Controversial?: Many organizations doing data mesh are moving from a very centralized approach and they have certain concerns and approaches they should take. While other organizations are overly decentralized and need to be using different approaches to migrating. Really consider these as two different approaches that land on a similar end approach.
You should look to return governance to its original meaning: guiding and steering. In data mesh, governance shouldn't be about control.
?Controversial?: "Not all governance is created equal." There are non-negotiables: infrastructure security and data classification. Don't over-index on sharing data at the expense of crucial governance.
Shift your governance left. Shift left the responsibilities - where appropriate - but also shift governance left in your timelines - it needs to be part of the build processes.
If you are doing data mesh, you have to understand the existing organization governance structure - whether that is called data governance specifically or not - and align with it. Legal, risk management, IT, etc. Make sure they are aware of your work and give them confidence you will keep things governed well.
Every member of a project team on a consulting engagement is obligated to deliver value. That means you need to translate your work to business value. Governance isn't only risk mitigation, it should have incremental value.
In data, we too often "confuse the pipes for the water" - meaning we focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data.
Consider telling business users you are providing them an ecosystem of capabilities instead of a platform. They can understand that they have their own "habitat" and can customize aspects to meet their own needs but that it all needs to work well together too with other habitats.
Federated computational governance is an overly scary term. Break that down for users and what it means for them and why it actually makes their lives easier/better.
?Controversial?: "If you automate something that users aren't expecting to have automated, they will be extremely surprised and disoriented by the platform." Make sure to communicate and ask people what they actually want automated. What you see as friction might be value-add work for them. Communicate!
You can plan out a lot of things but they just won't survive "first contact" with the data model and/or the business model. Be prepared to iterate, don't plan too much and be flexible; iteration is great, you are implementing what new you've learned.
If your organization is coming to data mesh from a highly decentralized approach, it's very likely there are pockets of data that never were put into the central data infrastructure. It will probably be hard to get people to trust sharing that data and giving up control of it because the previous approach couldn't handle what they were doing.
?Controversial?: Part of data mesh or any transformation initiative, you need people who have ownership over, the responsibility for, organizational change management. Driving buy-in at the personal and organizational level. Scott note: I kind of like this but it's also scary. Who has the responsibility of driving buy-in specifically?
Use the phrasing capability instead of feature when talking data platform. What capabilities are you offering to enable your use cases? And your capabilities should be very use-case agnostic.

Lynn started with her being the lead data governance person for a data mesh implementation at a client. It's allowed her to have governance be part of the value drivers for all aspects of mesh. She said data governance in mesh is about making data "safer, better, and easier to use and share." The value proposition of doing governance well in data mesh starts to emerge when you think that to really get the most out of data, you need teams to feel like they can rely on data, not just use it.

It's crucial in all aspects of data, but especially with data mesh, that governance is an enabler instead of a blocker according to Lynn. As a governance leader, you can't be trying to put up blockers but it's crucial for the team to understand why aspects of governance are helpful and add value. You should be building alongside - especially the building aspect, being part of the work - to keep things moving while also making sure you aren't setting yourself up for trouble later, whether that is quality, regulatory, or otherwise.

Lynn said "not all governance is created equal" - it's important to establish where you can and cannot compromise. Infrastructure security and data classification are both non-negotiables. Both wrap in to external threats and internal access control, including to regulatory requirements. Risk management and compliance just cannot be ignored so make them part of the build process and ensure everyone understands why they matter. It's a core of product thinking around data.

There are multiple ways to 'shift left' in governance in Lynn's view. The first is shifting governance left in the project timeline - it should be part of every aspect of building your data mesh from the start - platform, data products, etc. You don't layer governance on at the end. We also should be shifting some of the responsibilities left to the developers - while giving them the capabilities and knowledge to handle what they can and come to experts for what they can't.

Lynn has had some success with talking to business users about not providing them a platform but an "ecosystem". It lets them envision something on a bit more of a tangible level, that they will build out their own habitat but it plays into a larger ecosystem. So they are comfortable in their own habitat which lets them better deliver value to themselves and others. Of course, your ecosystem can't infringe on other ecosystems so it is a good analogy for how domains all interact.

When looking at what to automate as part of your platform, Lynn recommends strong communication with users. You can surprise and disorient them if you automate the wrong things. E.g. there are certain aspects of discovery analysis that look a lot like standard tests but one is incremental work where you want to dive deeper into the data and one is part of daily work without value add from doing manually. Automate the second but not the first :)

Lynn brought up that your organizational setup before moving to data mesh will have a very large impact on what you should focus on when in your journey. If you are coming from an overly centralized approach, it will mean that you understand the benefits of integration but might have some challenges with letting go of central control. If you are coming from a very highly decentralized world, you might need to focus much more on alignment because domains may have had freedom to do as they please with little thought for users in other domains.

In organizations doing data mesh coming from a very highly centralized approach, Lynn says be prepared for a lot of dark data. Because the officially blessed IT systems were overly rigid, many people were doing work outside of the systems, often in Excel or similar. Getting people to trust you that you'll take good care of their data and they can still maintain access will be hard. There are many fun governance challenges including naming conventions to tackle with data like this.

For Lynn and her team, it's been very important to focus specifically on answering what's in it for me from users. Me in the personal sense and me in the organizational and/or domain sense too. And having people whose role involves owning organizational change management is very helpful as part of a product team. There is literally a person or people responsible for getting alignment and driving buy-in.

Quick Tidbits:

In data, we too often "confuse the pipes for the water" - losing the forest for the trees. We focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data. Focus on the outcomes and business value.

It's easy to scare people by using the 'federated computational governance' term. Work with teams to show them where we can automate the past pains of governance so everyone can work on what drives incremental value not manually applying policies.

Regarding a data warehouse: "I never saw a plain vanilla implementation survive anybody's business data model." Your plan will not survive first contact with the business model as is. Be prepared to iterate on your plans in data mesh. But now, that doesn't break everything like it did for the enterprise data warehouse.

Similarly "no taxonomy survives first contact with the data." You have to build so you can update and iterate, prevent brittleness.

Don't use the phrase feature for your platform, use 'capability' instead. Make sure you are building the capabilities to support multiple use-cases and start to prioritize capabilities based on what use cases they can unlock.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of May 14, 2023 May 14, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#221 Panel: Building Your Data Mesh Roadmap - Led by Eric Broda w/ Elizabeth Calloway and Phill Radley May 11, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Eric's LinkedIn: https://www.linkedin.com/in/ericbroda/

Eric's Medium: https://medium.com/@ericbroda

Phill's LinkedIn: https://www.linkedin.com/in/redshirts/

Liz's LinkedIn: https://www.linkedin.com/in/elizabeth-negrotti-calloway/

In this episode, guest host Eric Broda, an Executive Consultant in the financial services space (guest of episode #38) facilitated a discussion with Liz Calloway, a Data Governance expert in Financial Services (guest of episode #92), and Phill Radley, Principal Data & AI Strategy Consultant at Thoughtworks. As per usual, all guests were only reflecting their own views.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward.

Scott's Top Takeaways:

It’s important to understand that when you decentralize, different aspects can - and should - move at different paces. Your roadmap needs to account for that but it should also take advantage of that. Domains can go at their own pace, including ones looking to quickly drive towards significant data value.
Everyone's roadmap, by their inherent nature, will be unique based to the organization's context. Trying to copy-paste from another organization will end badly. That said, there are some pretty core capabilities that all mesh roadmaps should have.
A roadmap should point you towards your target endpoint. Of course with data mesh, there isn't really an exact endpoint - you don't stop evolving, much like with microservices - but the idea is the same: how do you want the organization to operate relative to data production and sharing? Set your North Star to guide you.
Your roadmap really should have time built in for "socialization". If you don't, then there is no real extra collaboration or communication between teams and everyone executes on what they expect is needed - you optimize at the micro level, not the macro. It's crucial to human cognition to not only be 'doing' but thinking and relating to other people. That's how you can drive sustainable change management.
Be very clear on your value proposition for your data mesh journey but don't try to get your budget and set your value proposition as this long-term delivery of value. Look to the incremental value delivery and find value propositions for incremental work. Basically, deliver value along the way and don't tie your budget and value to something that is a massive delivery 3 years down the road.
For data mesh, absolutely be prepared to make compromises on when you plan to deliver capabilities. If you aren't prepared to make compromises, you unequivocally are not ready for data mesh.
Focus more on business capabilities over purely underlying technical capabilities in your platform roadmap. The tech is the most tangible part but it's also not the thing that really ends up driving value or being overly important to most users.
Governance might actually be the most crucial aspect to your roadmap. If you don't map out your planned capabilities around governance, you don't really know when you'll be able to tackle certain types of use cases. And it's also very easy to leave off crucial governance issues to later that will hurt you as you try to scale like policy as code, data product blueprints, and interoperability standards.

Other Important Takeaways (many touch on similar points from different aspects):

Try to delay your key architectural decisions until you know more. At some point, you have to make a call but don't try to lock in your architecture right away. However, this will REALLY frustrate many (most?) data engineers because they often care more about the tech than the capabilities.
Overall, focus more on what are you trying to do to support and advance the business strategy in your roadmap. Data people want to focus on the data work but it's very easy to diverge from the business strategy. So make sure you are keeping your roadmap aligned to business goals.
If you are going to create a roadmap, you should get clear on the definition of your roadmap. What are you actually trying to achieve with your roadmap? What is important to include and what isn't? Don't create a roadmap to create a roadmap - at the start of your process, define the goal around creating the roadmap.
A roadmap at the high level is a strong communication tool to drive alignment across many parts of the organization. It is a vision to share and work towards. You will have your top roadmap but you might have roadmaps for aspects of your data mesh like your platform too.
The roadmap isn't rigid. The target vision 'destination' is but focus a lot on what is our next 'pit stop', what is our next value delivery instead of trying to plan everything out years in advance.
If you're spending too much time to ensure you are 'right', being 'right' might not be actually valuable - you spent too much of your "time, talent, and/or treasure" as Liz said where the return on investment isn't there. Learn to fail fast and iterate to value with smaller bets.
Even when it comes to a roadmap, prepare people for one of Zhamak's mantras: think big, start small, move fast.
Your roadmap shouldn't be too ambitious and you should roadmap out your organizational changes too. Far too many try to focus purely on the technical and platform aspects. Do not try to escape Conway's Law - look to reflect its impact in your roadmap.
Don't focus overly much on exactly where you are going without reflecting on how far you've come. Think about your successes - showcase/celebrate and try to measure them as that will drive further buy-in and momentum. You can highlight the work of the team and inspires others. Make this part of your formal roadmap.
You want to think about when you will add your self-service capabilities. You have to make some compromises at first in a data mesh journey, especially around the platform. It can't do everything you want it to on day one or you will have spent far too much time building instead of doing. Be honest about the importance of capabilities and look to automate where you find the most friction.
Governance on your roadmap will really set a tone for your journey. Is this about better data processing or about better business outcomes through AI and analytics?
Does your definition of minimum viable product for a data product evolve as you mature in your data mesh journey? It's an interesting question to dig deeper into.
A simple, compelling value proposition could be 'the work we do to satisfy regulatory compliance should be reused for our analytics and AI. We should drive the cost and complexity down for both by focusing on reuse.'
Your roadmap could possibly even include developing a better decisioning process for future work. Focusing on tying business use cases back to data work. Even something like measuring the value of your data work. It won't be great from day one but think about business processes around data work you want to mature.
In your platform, potentially err towards building the business capability first and then building out the self-service aspect to take care of that. You need to understand what you actually want to build and an excellent (the best?) way to do that is to get real-world feedback on what are the actual pain-points of putting something into production.
It's okay to have either (or both) an org-wide, large-scale transformation roadmap or a much smaller roadmap focused on that incremental value delivery. Both could be a valid approach depending on your organization's situation.
Be real with assessing where you are in your data maturation journey, your data governance journey. No one is judging you on that. It's better to be honest to yourself than going for bravado. Again, no one externally is looking at you and especially not at your roadmap :)
There are multiple successful and sensible ways to start a data mesh journey. You shouldn't be overly focused on how others are getting going. Yes, learn from them but you can - and should - tweak to your organization. Don't go for easiest, go for most likely to succeed in the short and long run. Yes, easier said than done.
Getting your first mesh data product to production is a "momentous occasion" as Eric said. You had to really get a lot of things in place to achieve it.
There are a few parallel value delivery tracks in your data mesh journey. There is business value delivery - delivering on the use case. And there is also mesh capability delivery - delivering on the ability to do data mesh. When you first start out, the business value delivery will outstrip the capability delivery but the capability delivery is where you have your long-term value leverage, your ability to scale value delivery.
It isn't easy and can sound cheesy, but make sure to build empathy into your roadmap - how do we exchange context with each other and learn what drives value for each other so we can do better work that delivers more business value? Good empathy will lead to stronger communication and collaboration when it's valuable/necessary.
If you see something, say something. Over communicate. Think about what others might want to know and let them know, don't only wait for someone to ask.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#220 Building Your Early Mesh Data Platform and Data Product Capabilities - Interview w/ Manisha Jain May 07, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Manisha's LinkedIn: https://www.linkedin.com/in/evermanisha/

'A streamlined developer experience in Data Mesh' articles by Manisha:

Part 1 - Platform: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-platform

Part 2 - Product: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-product

Article on Lean Value Tree: https://rolandbutler.medium.com/what-is-the-lean-value-tree-e90d06328f09

Blog post on mentioned data mesh workshops: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html

In this episode, Scott interviewed Manisha Jain, Data Engineer at Thoughtworks.

Some key takeaways/thoughts from Manisha's point of view:

Manisha's top 3 pieces of early mesh journey advice: A) put together a specification of what a data product is, make it clear. Your definition will evolve/improve but if people don't understand the building blocks, it's going to be hard to build value. B) start to create standardized input and output ports because that is how data products - your units of value - actually exchange value. C) make it easy to discover your important SLOs to create trust; partner with consumers to serve what they need.
When working with your first domain, it's crucially important to make sure they have strong data engineering talent - whether that is existing people or someone embedded into the team. That way, someone inside the team building the data product can better interact with the platform team to communicate basic needs at the infrastructure level. This isn't as necessary for later domains - the platform already exists.
Both the platform and product teams need to really understand and align on responsibilities and necessary capabilities. That will help you streamline your developer experience, which is crucial to scaling data mesh.
People very often confuse data and data product. You should look to get crisp on what a data product means internally. Scott note: we still don't have a simple, on-paper explanation of a data product you can put in front of a team of non-data folks…
It's crucial to understand how data mesh can align with how your organization thinks and works. That will make it far easier to drive buy-in. "…only when they're comfortable with that concept … will [it] make sense to go ahead and explore more."
The platform team needs to focus on delivering capabilities to domains, not technologies. But they also need to think about mesh-level capabilities, e.g. supervision capabilities. Think about what capabilities are needed when, don't boil the ocean. Maybe certain use cases are too difficult to tackle right now. Scott note: that is the mark of a good org approach, when you can say "we aren't ready yet" and it's okay
It's important to remember that the data platform is there to make it easier to create, deploy, and manage/evolve data products. Your data products are the unit of value exchange in data mesh. Use that as a guide when deciding what the platform should offer.
?Controversial?: Boundaries around data products and data product teams are more important than most realize. We really do need teams to be able to act independently and not deal with the hassles of shared infrastructure. Scott note: I know people get worried about cost but time to market of information matters too as well as time spent dealing with untying infrastructure knots by the platform team.
Consider doing a series of workshops with a small group closely aligned to each domain to drive understanding and alignment. Don't try to do all the domains at once, each has a different set of needs and capabilities.
The Lean Value Tree is an effective method to break down what you are trying to accomplish into actionable pieces of work with explicit assumptions. You state the bets you are making, the hypotheses you are testing, etc. so people are on the same page.
When you start working with a domain in data mesh, really drive down to specifics. Don't just identify a use case and maybe the necessary data products. What skills and tooling are necessary to create and maintain those data products? What would a team look like to own those data products? How are you going to put that all together?
Doing data mesh, every new domain is different so the onboarding plan for each incremental domain will have to be adjusted. That doesn't mean everything starts from scratch but you need to assess gaps in each domain's capabilities to figure out how best to enable them.
To understand what capabilities domains need, the platform team needs to have strong communication and partner with the domain teams to build what they need, what helps them get the job done, not build the coolest platform :)
The platform team should focus on addressing three questions in the initial build: a) how do users create value? b) how can we ensure users trust the data? And c) how do you make data products usable and discoverable?
Your platform team probably won't recognize what all will be reusable components until they've brought multiple domains on to the platform. And that's okay.
?Controversial?: Most aspects of concepting and then building/deploying a data product are reusable, the data modeling and data transformations are the things that are unique to every data product.
Interoperability standards won't happen magically but you can do more custom mapping between data products very early in your journey to get use cases out. Start to look for places to create simple standards used to integrate data products.
?Controversial?: Offering automated modeling or sample data models is a double-edged sword. They can be helpful to get teams to something passable but it's rarely going to create a good data product without more work anyway. Basically, at best it's an outline but shouldn't be published like it's the article.
Look to build to the thinnest slice that delivers good value. Don't get ahead of yourself. But don't try to get by ignoring one of the data mesh principles.
As you build data products in a domain, you will learn more and more about that domain which will lead to enhancing/evolving the data products you do have and potentially creating new ones as use cases emerge. It's important to stay curious and be prepared to share new insights via data.
?Controversial?: Data people have to really learn to speak in the language of the business. Otherwise, we will continue to talk past each other instead of learning and iterating together towards value.

Manisha started the conversation with her thoughts on how to get going with data mesh, on-boarding any domain but especially your first domain. Work with a small team aligned with that domain to find how data mesh can align with how the organization works and thinks - this will be different for every organization. That alignment is crucial to getting people comfortable and driving buy-in. People have to be comfortable with how it will work and what are their responsibilities. As Manisha said, "…only when they're comfortable with that concept … will [it] make sense to go ahead and explore more."

According to Manisha, when you are bringing teams up to speed, it's really crucial to get on the same page on what you mean and what you expect from them. They often confuse data and data product for example. The differences can be subtle but are important to understand. As Chris Haas also stated in his episode, they are using the Lean Value Tree method to break down target outcomes into explicit assumptions and more manageable aspects of work. What are the bets you want to make and what are the hypotheses you are testing?

Your initial workshop(s) with a domain can also be a lesson in how to deliver value using a data mesh approach and prioritization. Manisha talked about how when working with a domain, you might identify multiple potential use cases. But you need to choose what is a priority to do now and why. This can surface what are the top one or two use cases and also show the domain how to prioritize as use cases continue to emerge in the future. The use case(s) they select to prioritize then directly lead to discovering the data products that needed to support the use case. And then you identify what skills and tooling are needed to actually execute and build and then maintain the necessary data products. Then, you can start to back into what a team working on the necessary data products (and potentially platform) look like. You can use that Lean Value Tree concept to really get specific because far too often in data work, things are left too vague. Scott note: Get specific, get explicit, chase away vagueness - but of course leave LOTS of room for experimentation and iteration as you learn and build.

When asked more about workshop dynamics, Manisha shared how they try to keep them from being too heavy on the domain - get a few people, maybe 2-4, who really understand the domain and can represent the business aspects, not just the data and/or technical aspects. Each workshop has its own goal as an outcome but it's important to first align data mesh to organizational goals, the business strategy. Then you can get into data mesh specifics. They call their workshops 1) accelerate, 2) discovery, and 3) inception.

Manisha shared some crucial dynamics when working with your first domain that do get easier as you bring on additional domains. In the first domain, it's crucial to really narrow in on understanding and definitions including roles and responsibilities. Data product owner is a new role, what does it actually mean? And there's the initial platform work too. But as you bring on your third, fourth, fifth, etc. domain, there is internal learning to share with the new domains. There is more clarity around what a data product is - they can even see already built data products - and roles/responsibilities. But you will need to definitely do a gap analysis to figure out how to best enable each domain as each domain is unique. So there is a balance - look to maximize reuse of platform, processes, organizational changes, etc. but don't look to force new domains to adhere to exactly how previous domains went through the journey.

For Manisha, it's very important for the platform team to think in terms of capabilities. Deliver capabilities, not technology, to the domains. Work with early data product teams closely and focus on what they are trying to do instead of how you want to solve the technical aspects. Focus on specifically what are they trying to achieve? Also, the platform team needs to consider what mesh-level capabilities are necessary when. Don't try to deliver a complete platform at the start - your platform is a product and minimum viable product, make sure you understand what minimum means and don’t go overboard.

The platform team can focus on a few simple things to drive to a good initial outcome/partnership with domains in Manisha's view: 1) how does the work create business value? What do the domains need to do to actually drive value? 2) How will users trust data, what does trust mean and what's needed? 3) How do we make it possible for domains to create and manage a data product that is usable and discoverable? By focusing on the task at hand and then mapping to capabilities to support that task, you can prioritize and deliver something useful and valuable without boiling the ocean. You don't need to try to include every capability at the start, that is a bad anti-pattern. Get close to the use case and find friction. You will also learn to recognize reusable components of the platform but some reusable components might not be evident at the start.

Manisha then went further into finding and identifying reusable components. The things that are most unique to each data product are the data modeling and data transformation in her experience. Almost every other aspect of spec-ing out and building a data product are reusable, merely customized to the data product itself. Finding the necessary SLAs and SLOs by working with consumers, that is a reusable process. How your SLAs are actually measured, the definitions around those SLAs are reusable. The infrastructure and CI/CD is reusable. The overall data product blueprints are reusable. So look to make these reliable as your organization learns how to build data products to make for easy reuse.

On data modeling and interoperability, Manisha shared that it's crucial to let domains evolve how they model their data as they learn. And interoperability, especially to support a use case, is of course important; but you will likely see a need for interoperability standards emerge when it's needed - basically, don't try to build all your standards ahead of time. That might be creating an enterprise data model with a different name :)

When asked specifically about sample data models and automated data modeling tooling, Manisha pointed to them being a double-edged sword. While they can be helpful, most (all?) data products need more custom data modeling to maximize their value. Essentially, the tools can get to a decent initial data model but domains should look to improve them. If Platform teams offer automated modeling tools, they should make sure there is a big caveat to their usage .

Manisha recommends you make sure your initial domain has strong enough data talent - whether existing or embedded - to communicate the basic needs to the platform team. Regular developers are often not going to be data fluent enough at the start to drive to exact data infrastructure needs like a data engineer could. But be careful not to over index towards tech too. Every domain will need people skilled in creating value through data modeling but you probably won't need people as advanced in data infrastructure later - the platform is already built by that point :D

It's important to differentiate what the platform should offer and what the data product developers should handle according to Manisha. The platform, at least the aspects around data product creation, should be focused on making it quicker, easier, and more reliable to create, deploy, maintain, and evolve data products. It sounds easy but it's actually easy to lose focus on that. Look for friction points in the creation and management lifecycle and automate what doesn't add incremental value. E.g. a data product developer shouldn't have to manually add data to the catalog so look to automate it - and yes, not everything should be built upfront :) Scott note: she added some good flavor around data product boundaries but it's very hard to summarize

Within the platform, Manisha believes it's very important to maintain team boundaries because shared resources become a bottleneck and pretty quickly can become very hard to manage. This is why Zhamak has been so clear on the data product as an independently deployable unit of architecture. Manisha gave the example of even the namespace for data products in the data catalog should be reserved for that one team so teams have a dedicated space to put all their data products.

Manisha gave some early mesh journey advice:

1) back to data product specification, you should create something that gives teams a very clear idea of what a data product is and encompasses. Scott note: still waiting for someone to open source their data product creation template…

2) if, as Zhamak says, data products are our unit of value exchange in data mesh, then making it easier to exchange value is crucial. Start to create standardized input and output ports so you can easily ingest and serve data. ETL shouldn't be a concept, it's ingest or serving only.

3) really focus on making it easy to discover and then implement SLOs and SLAs. Being able to understand and trust data is crucial to being willing to rely on it. That trust comes from good communication around SLAs.

Manisha believes learning the language of the business is crucial for data people. You need to extract the actual business value drivers and build to those so you have to be talking the same language - unfortunately for data people, the language that aligns to business value is usually the business language :) Look to ask more business user-focused questions than trying to get technical.

Quick Tidbits:

"… the data product spec should [at a] minimum talk about the data set ports, domain, service level agreements, how do I share my data, what does data sharing look like…" - Make your data product specification easy to understand what someone will create and what a consumer will receive.

Again, focus on a streamlined developer experience that keys in on autonomy. That's the way

Weekly Episode Summaries and Programming Notes – Week of May 7, 2023 May 07, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#219 Zhamak's Corner 22 - Increasing Resilience of Data Processes Through Software Best Practices May 04, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Key Takeaways:

We need to be better about getting on the same page regarding some semantics in data mesh. Otherwise, it's hard to work together internally and across organizations to move the industry forward.
There are so many things we've learned about how to break systems into smaller components on the services side, about preventing tight coupling but the data world has yet to apply those learnings. We're heading down some paths that we don't need to if we follow past learnings.
As an example of above, early data contract approaches are too tightly coupled around schema. We need to be a little less rigid there but how feels to be determined.
Postel's Law: "Be conservative in what you do, be liberal in what you accept from others." Learn it and think about how to apply it to data so we create more resiliency across our internal data ecosystems. Right now, there isn't much out there on the how.
Resiliency at scale is possible on the operational plane, why not the data plane? We need to be "very mindful and not naïve" around how we integrate in the data world to not make the same mistakes we made on the services side.

Postel's Law: https://ardalis.com/postels-law-robustness-principle/

Semantic Diffusion article Zhamak mentioned: https://www.martinfowler.com/bliki/SemanticDiffusion.html

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#218 Building the Right Data Strategy: Why Are We Even Doing This - Interview w/ Beth Bauer Apr 30, 2023

Weekly Episode Summaries and Programming Notes – Week of April 30, 2023 Apr 30, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#217 Pt 2 of Another One on Buy-In: Flipping the Script on Working with Your First Domain - Mesh Musings 47 Apr 27, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Here are 8 more leverage points to consider when trying to work with your initial domain

Make it EXTREMELY tied to their top priorities.
Find additional funding sources.
Find a true executive sponsor - think about being the one that brought a whole new approach that is driving far better data results…
Having their data be leveraged by other teams can mean more funding flows for their data projects and potentially their domain as a whole.
They control how things work for everyone in the data mesh implementation. They aren't just the trendsetter, the data mesh implementation leads will make it work for everyone but they are the secret favorite child and initial shaper.
Similar to moving at the speed of business, MUCH smaller experiments.
Better data quality.
Potentially easier regulatory reporting.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#216 What is Your Part in Doing the Right Thing in Data: Value, Ethics, Literacy, and More - Interview w/ Guy Taylor Apr 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Guy's LinkedIn: https://www.linkedin.com/in/guytaylor/

Deep Work by Cal Newport: https://www.youtube.com/watch?v=xJYlhhT7hyE

In this episode, Scott interviewed Guy Taylor, Director of Data Science and Analytics, as well as the Director of Experimentation at Booking.com. To be clear, Guy was only representing his own views on the episode.

Some key takeaways/thoughts from Guy's point of view:

In general, "people want to do the right thing." Look to reward people that do the good, ethical things as part of their work product.
Always be asking "what is my part in this?" Ask for the expectations and try to be clear in your expectations of others.
Literacy is not only about an ability to read but also write. So with data literacy/fluency, we need to be able to use data but also create and share it. It's about learning how to share information - not just 1s and 0s - and share it well.
There are still major communication gaps between producers and consumers in many cases with data. Part of that is just not getting on the same page, really making sure both sides close that gap. Scott note: as Andrew Pease said, both parties should go more than half way to ensure you've covered everything.
If you don't align on expectations, you're far more likely to have a bad time :)
Data people need to stop trying to jump to the tooling to address a challenge first. Get the information necessary - what are people trying to accomplish to create value? - then look to how tools can drive towards capturing that value.
In tech and especially in data, there is a strong tendency or leaning towards taking action now. Sometimes no action or no action right now is the right answer. Often times, you need more information to make a better decision instead of patching the hole in the bottom of the boat with bread.
?Controversial?: The way many organizations are trying to leverage data contracts won't lead to significantly better outcomes. There should be technology handling the interface but consumers still need to speak with producers to align on expectations and share their use cases.
?Controversial?: There needs to be more accountability and responsibility on data consumers to actively participate in data work to drive towards the most business value.
Ethics in data is always a challenging but interesting issue. Start by creating a set of principles and try to frame potential choices within those principles. Your company ethics will evolve, update your practices as they do. Try to "do the right thing" and encourage others to check in on past decisions to reevaluate too.
It's crucial to consider cognitive load - people are most productive when they have time to spend on thinking. Work to give teams, especially those new to owning data, the time to learn, not just the information they should learn.
When people know there are expectations on them but they don't know what they are, that's unnecessary cognitive load. Look to make expectations more clear on domains about what data ownership means.
On data ownership, be very clear about how far it extends. Data producers should own sharing the information but the consumer has responsibility of ownership too. Scott note: for me, you need to be clear in every relationship but it can be the producer owning the data, the insight, or the insight and the 'so what' - you just have to be very clear!
In most organizations, teams are overloaded with work and with cognitive load. You can't just easily pull apart the complexity and fix that overnight. And in a lot of cases, it will be hard to do it at all. Look to prioritize instead of do everything. Saying no can be your ultimate productivity tool.
Be like Marie Kondo - 'does this spark value for your organization.' Don't be afraid to shut things down that don't spark value and refurbish your orphaned data generating processes that are valuable.
Time for innovation is crucial. It's on leaders to enable their teams to prioritize innovation and experimentation.
Always be delivering value. Look to use incremental value delivery methods - how do you break down a big potential project and deliver over time instead of the high risk way of a big project?

Guy started off by talking about data literacy and how the analogy of literacy - it's not only the ability to read but also write - carries over to data well. Data literacy or data fluency, it's not just can someone consume data but can they also produce data, can they share information in a way that can be 'read' by others? After all, we aren't trying to share data, we are trying to share information but via data.

When Guy embeds people from his data team into domains it "is with the express purpose of doing education, making sure that we are having the conversations around what things mean, what our expectations of those things are." Instead of embedding people into domains to do most of the work, they are focused on helping other people get to a level they can handle far more of the necessary data work. Which is quite often not the deep data work but bridging the communication gaps and getting on the same page. That is especially important for expectations - mismatched expectations is one of the most prevalent and damaging challenges to data work. So Guy is asking data team members to spend a lot of time making sure the producers know how to manage those conversations and drive to what is actually of value instead of what was initially requested.

In his experience, there is tendency for data people to try to jump to the tooling to solve issues according to Guy. Going back to expectations, if you try to solve without the expectations setting and leveling conversation, you will likely not deliver what consumers expect. You may see it as solved but they sure don't. That's where you get the dreaded "the data is bad" feedback because there aren't clear metrics and expectations. If you align - and as Ghada Richani mentioned in her episode (#206) stay aligned through collaborative prioritization - then there is a much better chance of delivering value and making all parties happy.

A comment Guy made was that there is an over tendency towards action in tech and especially data. People see a problem and they want to jump to trying to fix it instead of getting the necessary information first. And it may be no action is the best answer too. Just because there is pain, that doesn't mean action is necessary immediately.

Right now, Guy sees the industry conversations around data contracts and data sharing agreements as slightly naïve: people seem to be thinking this is about data integration between systems instead of data sharing between two parties. And that producers should declare every aspect of what they are producing instead of consumers being part of the conversation. Consumers need to share what they are trying to achieve, how they will use the data, etc. so producers understand the value and what would disrupt that value creation. There needs to be accountability and responsibility falling on consumers too. The contract portion can serve as the technology interface but that doesn't replace the need for conversation.

Ethics in data is always going to be an interesting but challenging problem in Guy's book. A good place to start is the social contract aspect: how would this be viewed by society? As an organization, start down the ethics path by creating and agreeing to a set of principles. Create good ways for people to seek and receive useful feedback regarding ethics. And honestly, your company ethics will change and it's important to reevaluate your ethical choices, especially as you learn more - your organization will have made mistakes and that's typically not something to lose sleep over, fix it now and know you’re better. Basically look to "do the right thing."

While it can feel good to 'make progress', Guy believes in the Deep Work by Cal Newport type philosophy. People have the greatest impact when they have the time to really think and process. Yet, in today's work world, that is a rarity for anyone. If we are asking teams to really take on data ownership, we have to work to prioritize the time to learn - and that includes processing time. Yes, people learn by doing but not only doing :)

Guy talked about trying to clear the space for teams to learn something new, including the impact to the cognitive load capacity of teams, especially when it comes to data ownership on the domains. When people know there are expectations of them and their work but those expectations aren't explicit or clear, that's unnecessary cognitive load - domains need to have crisp and clear expectations - and if the expectations of them by consumers and the data team aren't super clear yet, communicate that. But try to get to more detail to make the implicit very explicit.

Another common friction point Guy pointed to is the lack of understanding of your impact on others. He believes again that most people want to "do the right thing" but they don't always know there is even a problem to address. How are your actions impacting downstream data consumers? Again, there is a responsibility on those data consumers to generate the conversation! Sharing that context allows people to do the right thing because they are aware.

Currently, most teams in most organizations seem to be overloaded with work and cognitively overloaded in Guy's view. It would be lovely to wave a magic wand and fix that but it's not possible. So, we have to work to pull out the complexity and give our teams the ability to do their best work but high value work has interconnected complexity so you can't take a machete to it and expect good results. Look to prioritize what really matters when you can and break things down into manageable chunks.

In talking again about data ownership, Guy believes that producing teams should not have to own too much of the downstream consumption. They should own the transformation and sharing of data after it's been transformed but the consumer should own the insight or metric - they asked for the data so they need to have some responsibilities and accountability too. Scott note: I don't entirely agree it should be that for every use case but I do like a very crisp line of ownership if it works.

Guy uses the Marie Kondo approach when looking at orphaned systems or processes - evaluate it and is this "sparking joy", is it creating value. If it is, great, let's get it into a good shape and put it into the right hands ownership wise. Not shove something to someone while it is still in data disrepair but get it functioning well and hand it over. But if something isn't creating value, shut it down. For too long in data, there has been a hesitancy to shut things down because at some point they might create value. Don't fall into that trap.

It's important to get to an experimentation - a test, learn, and then iterate - approach in Guy's view; he is the Director of Experimentation after all! A good way to get people to see the value of experimentation is how it provides far better incremental value delivery. Instead of huge projects with big budgets that take years and rarely deliver the value expected, what if instead you took the overall goals and broke it into manageable pieces and delivered value over time as you get closer and closer to the project vision. You'll be more nimble and get a return on investment far quicker.

A few tidbits from the end of the conversation:

Skunkworks can be a great approach to trying things out and seeing if there is value. Don't try to move the skunkworks directly to production but you can do some fun and useful innovation that way.

Good leaders set their teams up to innovate. They prioritize the time to try new things and let their people explore, let them go off the "paved road".

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of April 23, 2023 Apr 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#215 Panel: Leading a Data Mesh Implementation - Led by Kim Thies w/ Omar Khawaja, Ferd Scheepers, and Mike Alvarez Apr 20, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Kim's LinkedIn: https://www.linkedin.com/in/vtkthies/

Mike's LinkedIn: https://www.linkedin.com/in/2mikealvarez/

Ferd's LinkedIn: https://www.linkedin.com/in/ferdscheepers/

Omar's LinkedIn: https://www.linkedin.com/in/kmaomar/

In this episode, guest host Kim Thies, Director of Intelligence Automation at PayPal facilitated a discussion with Ferd Scheepers, Chief Information Architect at ING, Mike Alvarez, Former VP of Digital Services at a large healthcare distribution company (guest of episode #236), and Omar Khawaja, Head of Business Intelligence at Roche (guest of episode #96). As per usual, all guests were only reflecting their own views.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually.

Before we jump in, I think the main takeaway here would be a data mesh implementation leader's journey can be a lonely one. Find peers and exchange information. You can reach out to me (Scott) but there are also many leaders that want to exchange information with each other. The other is the meaning of journey: it's never done; be prepared to continue to push - it can feel Sisyphean but it's important to keep moving forward and expect to continue to drive buy-in.

Scott's Top Takeaways:

Everyone sees the 'Instagram photos' version of other organizations' data mesh journeys - it's not the reality. Everyone is struggling with certain aspects of data mesh because if this were easy, people would read Zhamak's book and be done with it. It's just not realistic to expect that, give your leaders (yourself?) a break :)
It's incredibly important to understand you will get things 'wrong'. Scratch that, you will get MANY things wrong. But 'wrong' in data doesn't have to mean wrong for good/all time. It's about trying, learning, and iterating to better. Really look into fast fail practices. If you feel the need to get everything right, data mesh is definitely not right for you right now.
A champion early adopter is crucial to drive a data mesh implementation. There needs to be someone who will partner with you that is "brave enough to try new things" as Omar said.
It's easy to focus on so many aspects of data mesh and forget that the mindset shift and cultural change are the biggest challenges for most organizations. They are the most squishy/least tangible but that doesn't mean they aren't absolutely crucial.
Teams need the autonomy and empowerment to move at the right speed for them. The more friction in their data work, the more time - and effort - it takes to deliver value and the world can have moved on. You have to give teams the capability to strike while the iron's hot. That's a good way to drive buy-in: we're going to give you the ability to move at the speed of business.
Be very clear on expectations. What do we owe each other? Focus on how data mesh drives value for them but also how it drives value for the organization. What is the output of their work, what value comes from what they did or will do? It will be quite difficult to change this kind of thinking overnight.
Similarly, be very clear on responsibilities and be very clear on target outcomes. It's easy to get lost in the work instead of what are we trying to achieve.
A good success story doesn't have to be a massive win. Time to delivery is really valuable so showing getting something into MVP in a few weeks will get a number of teams excited. Driving that lower is of significant value - and it's a tangible value for many business partners where they can reasonably share about the impact to their business.

Other Important Takeaways (many touch on similar points from different aspects):

There's a very good reason why we refer to them as data mesh journeys :) You can't treat this as a project. Think of your physical fitness and health. It's an ongoing journey with plenty of trials and tribulations.
It's okay if data mesh is not a fit for your organization. There's no shame in that. Be honest when considering if it's a fit or if you just want to do it because it sounds like a good solution in the abstract.
In general, look to collect reasons for past failures prior to doing data mesh. Not to throw others or past ways under the bus but to emphasize that the current ways aren't working as well as we might like. Proof points really help convince others.
It's always a challenge to maintain autonomy and empowerment without it leading to silos. There isn't a magic formula. It's something you will have to constantly keep a watch on but you're not the only one.
There is a balance between autonomy and interoperability. You'll probably get it 'not great' at first and THAT'S OKAY. Work to find the right balance - or at least an acceptable balance - as you move forward. And that balance will likely shift.
The focus of the platform should be on making users' lives simpler so they can actually own the data. It's easy to fall into focusing on the tech but the value is in friction reduction for users and the abstractions you offer them.
Be prepared to need to continually drive buy-in. Gravity around historical - legacy? - data management practices is STRONG. You will have people wanting to use an enterprise data warehouse or a data lake setup or not wanting to focus on integration with the rest of the organization. Announcing you are doing data mesh doesn't magically mean everyone is bought in.
Similarly, be prepared to explain concepts and terms repeatedly. Data mesh - and data work in general - is not the main focus of most people in your organization. Do your best to have the patience to work with people to drive towards a better understanding.
Don't focus overly on terminology - see my unicorn farts theory… - focus on what the change in work means for people and what outcomes you are trying to achieve. Business leaders and users don't care that you are doing data mesh - data mesh is a shared language for the data people working on the implementation. Especially don't try to explicitly define federated computational governance :D
As Ferd said, look to "bring governance back to the essence of what we want to do." Don't get overly focused on governing everything instead of the right things. That can mean not taking on really difficult challenges right at the start of your journey by passing on use cases. That's okay, think about your capabilities and honestly assess them. Don't bite off more than you can chew.
Incremental progress is the best approach to most (all?) aspects of data mesh rather than trying to take giant leaps.
Early in your journey, likely focus more on quick wins that will give you some good internal marketing material - market those successes to build desire for additional domains to join, build the momentum of success. Also, think about limiting chances for failure early. Fast fail in general is crucial but you don't want a full 'fast fail' on your first use case.
Many things will not work or go great in a data mesh journey. That's the reality of being on the bleeding edge, being an explorer. It's very important to reflect not just on wins but take the learnings from what didn't work. If you don't reflect on your mistakes, you're far more likely to repeat them.
Your early champions and business partners can come from unexpected corners of the business. Look for bravery and a well-defined - and valuable - use case with limited initial scope. Be open to the idea of surprising initial use cases and working with unexpected lines of business/domains.
Try to directly tie business value generation to your data mesh work. Understand the value drivers for the organization and prioritize the work based on those value drivers.
To get buy-in, you don't have to offer domains the perfect solution, just one that is a better option than whatever else is available internally for their use case.
Another good potential hook for potential business partners is that data mesh - specifically handling your data as a product - has less technical debt so it can quickly pay for itself in saved time especially for use cases that are mandatory like regulatory compliance.
Be very diligent in putting teams together, especially early - throwing bodies at the problem won't solve anything but not having the right capabilities in the teams sets those teams up for failure. I recommend reading Zhamak's book on this topic re necessary capabilities in teams.
Incentivization to become a data producer can be quite hard but it's extremely important to get right. How can we think of value creation through data without the perception of monetization? Because that can feel like you are selling data externally.
For your first use case, you need to have the business and technology/data team working as one team - one _actual_ team, not just a team in name. That doesn't mean a huge re-org but you need people constantly sharing context and partnering to drive to value. Otherwise, you have a high risk of falling back to the same old ways of doing things.
Make failure not a catastrophe. As Vanya Seth says, "contain your blast radius."
It's crucial for people to understand that doing decentralized data means a lot of changes. Get people comfortable - or at least acutely aware - that we'll need new ways of working and delivering value.
It's probably easier in most organizations to get funding/buy-in if you focus deeply on the business use case. Yes, building data mesh capabilities has an overhead to the initial use case or two as you spend time and effort learning how to do data mesh and it's okay to talk about that. But if you aren't talking direct, tangible business value, getting buy-in - and more importantly funding - will likely be harder.
Make sure you have buy-in from consumers on use cases. If they aren't bought in, will they even consume? If you don't have users, the best product in the world is useless.
Budgetary issues - why would one domain create value for another if they aren't getting compensated for it? - are extremely common. There isn't an easy answer here unfortunately.
As stated above, in general, it's often lonely being the person leading a data mesh journey. Who do you talk to? There's me (Scott) but find other leaders and network.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#214 Is Core Data the Way to Achieve MDM's Goals in Data Mesh - Interview w/ Marcie Stoetzel Apr 16, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Marcie's LinkedIn: https://www.linkedin.com/in/marcie-stoetzel/

DGIQ Conference (Marcie and Samia Rahman will be speaking in June; in San Diego, CA, USA): https://dgiq2023west.dataversity.net/registration-welcome.cfm

Square hole video Scott mentioned: https://www.youtube.com/watch?v=cvwN_O5ypTE

Article about the "leaf sheep" sea slug Marcie mentioned: https://www.bbc.com/travel/article/20210324-the-odd-sea-creature-powered-by-the-sun

In this episode, Scott interviewed Marcie Stoetzel, Principle Product Manager of Enterprise Core Data at Seagen. To be clear, she was only representing her own views on the episode.

Before we jump in, Seagen might be a bit of a special case in how much the domains can leverage each other's data around a key record type. There is a lot to learn but you might not be able to find use cases that are as broadly impactful to many domains at once.

Some key takeaways/thoughts from Marcie's point of view:

?Controversial?: In data mesh, building a data culture focused on engagement, learning, and upskilling might be as important (more?) as doing the data work - if teams aren't willing to engage, what's the point of doing the work? Scott note: Marcie doesn't explicitly say this but it sure feels like an undercurrent. It's crucial to make your data culture something that can embrace data mesh.
When going to domains, come with a target value proposition. Why would they get value from participating in your data mesh initiative? Scott note: If there isn't a value prop for the domain, you will almost certainly struggle with incentivization. There are very few 'good Samaritans' willing to do a lot of data work with no discernible direct benefit to that domain.
Seeing is believing. Don't just tell people what you are going to do, get demos in front of them and show them more about what you plan to do.
Make sure to solicit feedback as you build. It's far easier to pivot after a short period of work than after a longer period. You can find misaligned expectations far earlier and work on appropriate prioritization. Basically, you spend less time building things that don't matter and more time on things that do.
Look for ways for domains to share about what information they have that might be useful in the organization. Oftentimes, domains operate so independently, they aren't in the habit of finding ways to collaborate. Scott note: Communication is the easiest path to high-value, cross domain use cases - the mesh won't find them for us.
To actually get to cross-domain use cases, you need domains to consider things from another domain's point of view. So extract more information about each domain to literally show to other domains to spark conversations. Be someone that connects the dots for domains and then connect people across domains.
Once there is more visibility to cross-domain use cases, business goals can be realigned to focus on building to common goals. That can help with some incentivization and alignment. Scott note: I hadn't previously heard this one articulated but it really does map with goal realignment where many organizations are seeing success with data mesh.
When you are early in your journey, look to create a "thin slice of an end-to-end experience for [a] domain." Really don't try to do too much. Scott note: this is what pretty much everyone who has worked on at least two data mesh journeys circles back to. It's VERY important to understand the thin slice concept.
Interoperability requires a certain level of trust. You can decide by your use-case how strong that trust has to be. If you need it to be essentially perfect for regulatory reasons, that's far different from the needs of many other use cases.
Focus, especially early in your journey, is very useful. Again, thin slice. Don't take on too much while you build initial momentum and buy-in.
?Controversial?: Find ways to humanize your data work, add a bit of levity and humor. It will make people connect to it more and internalize it more. Plus, it's just more fun.
?Controversial?: Similarly, "explore, discover and mature data together." It's okay to be vulnerable, especially as we learn. Transparency is also crucial.
Community and things like workshops are crucial to see wider adoption and engagement in a data mesh implementation.
Consider if you want to use the phrase master data management (MDM) at all. Master has slavery connotations and MDM feels far too heavy as a phrase. It can scare off the exact people you want leaning in to dealing with Enterprise Core Data.

Marcie started off with a bit about her background as a teacher and as well on the commercial side of the healthcare space which has shaped her view of teaching/learning and also healthcare data needs. Part of Seagen's goal in using Enterprise Core Data instead of Master Data Management (MDM) as a phrase was the moving away from the connotations with slavery (master) and that this data is core to the enterprise, that this is crucial to the organization, not just a data management practice or task. Enterprise Core Data at Seagen is about creating a way to make data that many domains leverage the same to prevent lots of domains doing the same work and make interoperability FAR easier. MDM also is typically managed centrally instead of enabled centrally and managed at the domain level (federated governance) so you have to rethink a lot to do Enterprise Core Data instead in a data mesh setup. Trying to map 1:1 to MDM in a historical data approach won't work well.

At Seagen, Marcie and team decided to tackle one type of core data record first - healthcare professionals - rather than trying to unify every type of record across healthcare - don't boil the ocean or bite off more than you can chew. They are working on creating the platform for domains to manage their core data records which creates more sharing opportunities and even higher quality data - teams can better cross-reference information. While they are still pre-production - it's early days - even bringing this to domains' attention is sparking conversations between domains about potential collaboration and new use cases.

Marcie and team are winning converts by showing them what the platform will be able to do for their own domain but also keeping an eye on that interoperability and leverage provided to other domains. That means that domains get some value from participating even if no other domains participate. If other domains do participate, then everyone gets more value from each other. They started with a simple value proposition - this will make handling your own data easier - and then created a group collaboration incentive - the more domains that participate, the better the information and the less work everyone has to do to get to better outcomes :)

When asked about incentivization complications around domains wanting to focus on their own goals, Marcie mentioned that as the domains are starting to find cross-domain use cases, the organization can realign goals to be about focusing on those common goals where everyone wins. What drives the most value for the business and how do we incent that kind of outcome/behavior? Scott note: this is an interesting nuance that I haven't heard before.

How they found the cross domain use cases was also interesting. Marcie and team met independently with different business domains to extract what each team felt could be a good output of working with a common enterprise core data platform, as well as tying these individual value props back to the larger goal of helping more patients. The data team then literally took all those outputs and showed each business domain that uses HCP (healthcare professional) data the value across Seagen of having an enterprise core data platform. This sparked collaboration ideas between domains for how to drive even more value from the core data platform.

Marcie is seeing that domains have to spend a LOT of time to cleanse and match data. And downstream consumers of their data have to do the work too as they don't know it's already been done upstream. The quality requirements for most use cases are pretty high. So creating a way for domains to much more easily interoperate data will save them a LOT of time and effort. The core data platform will hopefully prevent lots of domains from having to do a lot of that quality checking work and it will also increase quality by having more sources of information to verify data is high quality / correct - basically, the checking of quality becomes far less arduous.

One thing that is working well at Seagen for Marcie and team is doing lots of demos and small proofs of concepts. Similar to doing sprint demos, teams are buying in because seeing is believing. They are showing the business real, realized value that will come from their participation. So teams are leaning in. Similar to what Karolina Henzel mentioned in episode #104, there can be a LOT of value in addressing data quality issues for domains.

Marcie talked about the value of maintaining focus on a thin slice. Instead of trying to get many domains bought in on the data mesh concept, there is a specific use case and they are only working with a few domains at the start. There are clearly defined and scoped benefits. Again, thin slice. And focusing on the end-to-end solution to working with this data for the domains has also helped to get and keep everyone on the same page.

Quick Tidbits:

Look for ways to share knowledge in fun and interesting ways. Upskilling can be a bit intimidating, make it more gamified and less high pressure. Humanize (or in Kye's case, dog-ize) it a bit.

Really embrace an attitude of learning - be vulnerable and transparent. "Explore, discover and mature data together." "Engaging conversations, exploration, curiosity, and a safe space" are crucial.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of April 16, 2023 Apr 16, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#213 Zhamak's Corner 21 - Reinventing Data Development, Not Data Processing and Storage Apr 13, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Key Takeaways:

We need to make the data product the first class primitive of information sharing - make them the basic building block of how we create our internal data/AI ecosystem. (repeat from ZC20)
"Tools reshape behavior" - if we want developers to change their relationship to data, we need to give them the capability to do so easily. Or even if not easily, at least not arduously :)
The industry sees the value of data mesh but we need to find much lower effort ways to create sustainable, large-scale change. We all need to be finding catalysts.
There is no reason to try to reinvent a lot of the technology in data at the physical layer. Lake storage, streaming technologies, ML libraries _at the physical layer_ are great. But we need new ways of accessing and leveraging them to make it far easier to create and manage data products.
While everyone seems to be talking data products, there seems to be so many different definitions. This has led to a weaker market pull on vendors to improve their tooling to make data mesh more easily possible.

Semantic Diffusion article Zhamak mentioned: https://www.martinfowler.com/bliki/SemanticDiffusion.html

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#212 Reflections on Building a Data Mesh Platform from Scratch - Interview w/ Jyotshna Karki Apr 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Jyotshna's LinkedIn: https://www.linkedin.com/in/jyotshna-karki-81a24038/

In this episode, Scott interviewed Jyotshna Karki, Data Engineer at Novo Nordisk. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Jyotshna's point of view:

In data, especially in data engineering, people need to be curious. There are so many new innovations that may really be majorly beneficial. Look to try out more approaches and technologies.
You can have happy data producers and consumers with a centralized data lake setup and still have data mesh be the right evolution. In the long-run, at scale, it isn't efficient to have a centralized data team coordinating all data use cases.
For many domain teams, the centralized data processing and storage can be a black box. Data goes in, it gets transformed and stored by the central team and then served out. This can create a high dependency on experts and technology.
?Controversial?: If your domain team consists of their own data engineers and data scientist with domain knowledge experts to manage their own data products, it's okay to work with multiple teams at the start of a mesh journey. Scott note: if you don't need to drive buy-in and your org can do this, I don't see it as a major risk. But probably at most a few hundred (tens maybe even) organizations are like this worldwide.
Don't try to enable every tool as part of your platform. You should focus and create a good experience on the most widely used tools rather than trying to support every tool available out there.
?Controversial?: Probably don't try to automate processes at the proof of concept stage. Wait until the need and impact is greater. But non-automated processes are typically tech debt you should look to pay down when it makes sense, don't ignore that or it will hurt more at later stage.
Around best practices or things like reusable components and data pipeline blueprints, look to create centralized community sharing mechanisms but with decentralized ownership and contribution. Try to enable that sense of community knowledge sharing and trust.
Create a process to assess if you need to make a change to your platform. What are the business needs and are you meeting them? Constantly look to evolve and improve your platform.
KPIs for your platform are important - it's a product - but it's okay to start out pretty simple and use low tech monitoring signals like number of support tickets and customer feedback.
Similarly, try to be more data-driven around building your platform. Even if that data is pretty raw and unsophisticated to start. Scott note: try to stay away from vanity metrics but it's okay to / you will probably start with vanity metrics until you understand what drives business value from the platform.
Democratizing data, especially doing data catalog well, has led to less data deduplication. Because people know where to find data and can reliably get access again, they don’t copy or build something similar to ensure they have data they need.
"…make sure that it is reliable enough for people to depend on this data." Trust is crucial, give them visibility to see how data is handled to give them enough trust to really depend on it, not just use it.
Look to community events like hackathons to drive additional experimentation and value. If you make innovation a part of your culture, good things will likely come.

Jyotshna started off the conversation with a bit about her background, especially in data engineering and the need to be and stay curious. There are so many new approaches and technologies that could provide significant benefit to consider. Think in that product mindset and look to evolve your approaches and tech stack to create more value.

Specific to Novo Nordisk's data mesh journey, Jyotshna and team saw the writing on the wall for their data lake setup. While their centralized data lake was doing well and people were happy with it, there were increasing consumer and producer demands and the central data team was still required to help teams create their data products. Having a centralized team in the middle of every use case just wouldn't be efficient. Then, they hit some cloud service limits which caused some major headaches as well. All this led to looking to decentralize via data mesh.

At Novo Nordisk, many domains already had significant data capabilities and there were people building data products anyway according to Jyotshna. What they really needed was a way to empower and enable teams to more easily create and manage those data products in an interoperable way and lower the bar. So the central data team was to focus on the platform but there wasn't a huge need to upskill all the domains. There is still another centralized team of data experts to help domains that aren't as data fluent. Scott note: while this is not super uncommon, most organizations are not this lucky :D

Specifically to the pharma industry, Jyotshna shared some of the pre data mesh compliance/regulatory issues that were better addressed with data mesh. Domains needed to work with regulators but it was hard for them to really see exactly how the data was stored as it was managed by the central team, which is part of compliance. It was all part of the central data lake AWS account and those teams didn't have the ownership or visibility they needed. But with data mesh, the teams now have the visibility to their own data storage and access to audit logs and data governance.

Jyotshna shared that at Novo Nordisk, there was so much demand to participate in their data mesh, the data platform team and any centralized data capabilities - to assist the domains that didn't have high data fluency - worked with multiple teams to start. This helped them to define the requirements for their data mesh platform to support multiple data domains. While this is a data mesh anti-pattern, it went well for them as many of the domains again were quite capable with data engineering and data analysis. There were also many domains that wanted to contribute aspects to the platform so there were good feedback loops between the platform team and many domains. Scott note: Don't go this route unless your domains are already highly data fluent/capable. Working with many domains at the start can create a high-risk scenario instead of thin slicing.

Jyotshna and team are focused on enabling proof of concepts more than trying to automate everything right at the start. She noted they are focusing on understanding the problem deeply and moving fast to get proofs of concept into people's hands and then circling back to automate when there is more need and things are slightly more stable. Basically, they are being agile. It also has led to more modular components and reusability - they can get things out in a prototype phase and then think bigger picture how to deal with similar problems instead of point solutions.

In order to prevent tight coupling and keep modularity, Jyotshna and team started to actually remove things like data pipeline blueprints, reusable components, and bootstrapping accounts from the data mesh platform. While that might feel counterintuitive, they wanted to create a community specifically around things like blueprints so the central team wasn't managing them, community members were managing them. Look to create central sharing mechanisms but with a decentralized ownership and contribution model. Community-led innovation is more scalable than centralized knowledge ownership.

When thinking about platform maturity and if they need to pay down any tech debt, especially around certain features, Jyotshna and team benchmark quality levels and compare those to the actual business needs. Being in a heavily regulated industry, some aspects of compliance just are non-negotiable, you must meet them. But there are places where 'good enough for now' is a completely acceptable and correct answer. Some signals they use are support tickets and direct feedback around different aspects of the platform. They are also starting to build KPIs but it's a work in process.

One interesting aspect of doing data mesh has been less duplication of work per Jyotshna. This is a target goal of data mesh of course but it came about naturally as now that people can reliably find and access data, they don't feel a need to build it themselves.

Jyotshna said "make sure that it is reliable enough for people to depend on this data". Part of your platform and your overall mesh is to make it easy for consumers but also producers to trust the data. If you have a black box process, can producers really trust it? And evolution of your data products plays a part in trust too - a consumer can trust that the way data is presented is still relevant to the business.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of April 9, 2023 Apr 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#211 Another One on Buy-In: Flipping the Script on Working with Your First Domain Pt 1 - Mesh Musings 46 Apr 06, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

3 use case aspects or patterns to lean away from when trying to work with your initial domain:

What I call the domain type 'Everyone else in the organization wants your data so give it to us'.
Who is going to fund this? Not having a clear path to exec sponsorship and what the value expected will be is not great.
How do I add this to my priorities - essentially 'great, but that's more work I can't take on.'

And the 4 you should look to lean into:

Find your initial use case that is self-contained in the domain.
Move at the speed of business.
Find a use case that is easy enough with enough return to pay for itself as you also invest in the other aspects.
We will be investing heavily in making your domain better with data.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#210 Organizational Scalability in Data Mesh - Interview w/ Chris Haas Apr 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Chris' LinkedIn: https://www.linkedin.com/in/christopher-andr%C3%A9-haas/

Article on Lean Value Tree: https://rolandbutler.medium.com/what-is-the-lean-value-tree-e90d06328f09

In this episode, Scott interviewed Chris Haas, Advisory Consultant at Thoughtworks. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Chris' point of view:

When starting a data mesh journey, you have to find teams that are actually willing to participate. If they won't truly take ownership of their data, it's essentially a non-starter. You can help convince many domains by showing them the investment you'll put into their teams on upskilling and other capabilities enhancements - it's not just new responsibilities and work.
In data mesh, look to drive domain-wide understanding and buy-in, at least to the vision of what a more data capable domain means, the benefits to the domain. Not everyone will or has to care about data mesh specifically but you shouldn't have data mesh by decree from upper management.
Consider lean value tree mapping - it helps everyone align on a single mission with goals to support the mission, what bets you have to make for each goal, and on down. It can help you stay focused on what are you trying to achieve.
For use case prioritization within a domain, look to 1) what are the business goals, 2) what use case will have the highest value, and 3) what is the easiest to execute on. Don't get overly focused on value if you aren't ready to actually deliver it. Scott note: don't forget risk too
?Controversial?: Choose a use case with a single domain if possible for your organization's first data mesh use case. It just limits the amount of challenges and potential failure modes.
It's common to want to first focus on the technical aspects of data mesh - it's the tangible part and easier to see changes - but you should be focusing on the organizational aspects just as much if not more early in your journey.
You can start your data mesh platform team in a central data team, that's a normal pattern. And it doesn't have to be huge, 4-5 people building and a product owner - the platform is a product too.
A domain-based data product team doesn’t have to be large or require a big reorganization of the domain. Make sure they have the capabilities - data fluency and domain subject matter expertise - to execute but it can be a small team carved out of the domain.
Data product teams tend to be long standing teams but they are typically responsible for developing multiple data products, not just a singular data product. Most data products don't need that level of day-to-day tending :)
?Controversial?: You should consider your data product team a new team in the domain and a new team means budget. If you don't have budget and you don't rework priorities, you are putting additional work on a domain and the likelihood of success falls - look to avoid that risk.
Doing decentralized data, doing data mesh doesn't mean everything is decentralized. Scott note: this is a persistent myth around data mesh. Not everything becomes decentralized or you will just create silos.
?Controversial?: Once you've proved out you can do data mesh and drive value, when thinking about adding new domains, consider setting up a transformation office to coordinate, upskill, and prioritize work across domains.
Look to build an internal community to encourage cross domain communication and hopefully collaboration.
Finding cross-domain use cases is likely to be more difficult. Leverage that internal community to create more conversations that result in new, valuable use cases.
When finding your data products for a specific use case, start from what would be the specific data product to support this use case. Then start to work backwards towards what source-aligned data products you need. And of course, look at existing data products for potential reuse first.
!Controversial!: Source aligned data products should be - or at least start off - "very specific and very small."

Chris started out with a rather blunt but crucial statement: when thinking about data mesh, you have to identify at least one domain that will actually take ownership of their data. A successful data mesh implementation can't be entirely IT driven. And domain data ownership and coupling that with data as a product are typically very much not the natural order for most large organizations. You will need to invest into domains to make them capable of owning their data - showing that you will invest in their success can help win them over - so you want to make sure it will be money/resources/time well spent.

For Chris, it's very important for there to be at least a domain-wide understanding - and hopefully buy-in too - for what the domain is trying to do around data. That can be about data mesh or simply how they are changing their relationship to data. It won't work well to do data mesh from just upper management buy-in and pushing that down as a mandate. And that requirement can lead to deciding that data mesh isn't right for a domain and that is perfectly okay and reasonable - not every data management challenge is a nail for a data mesh hammer.

Lean value trees are an important tool for Chris and team when speaking with a domain about data mesh. What are they actually trying to achieve - the mission - and work backwards from there. Break down the goals, then figure out the assumptions or bets around those goals. This helps you stay focused on what you are trying to achieve - is it deliver a data product or is it address the use case and create business value?

So the output of a lean value tree helps align the team on a mission or missions which align to the business goals. Then, when thinking about use case prioritization, you need to balance those business goals, the amount of expected value of a use case, and the expected amount of work to deliver that use case*.

* Scott note: it is kind of included in the amount of work and expected value but I'd also factor in risk - what is the risk of this use case not being valuable and what is the risk of the team not being able to execute well on the specific use case. Probably don't take a big gamble on your first use case ;)

At the start of a journey, Chris recommends to find a use case that benefits the producing domain if at all possible. Yes, we want domains to publish data to benefit the entire organization but if a domain is going to be the test subject and invest their money, their people's time, their people's cognitive load, etc., it will be hard to find a domain willing to do that for another domain without decently strong incentivization. And at the start of your journey, those incentivization and community mechanisms are likely hard to come by. If you don't have these challenges and domains are all happy to help each other, consider yourself very lucky.

Based on interactions with a number of clients and prospects, many organizations would rather focus on the technical aspects of data mesh first over the organizational and Chris wishes that weren't the case. While building out the technical aspects is no easy task, if you aren't ready to actually do the day-to-day work, what are you building the tech to support? Scott note: this is extremely common and is also a very common comment from consultants. The tech feels more tangible and it's easy to say yes/no than squishy operating model discussions. But they are crucial to doing data mesh right.

Chris believes an early data mesh alignment on the organizational model doesn't have to be - and shouldn't be - disruptive. You can have a domain start to carve out new ways of working without doing a major reorganization. There should be a team building the platform and a product owner for the platform but 4-5 people building is reasonable and they can live in a central data team, that's a normal pattern, no reorg needed. As for the data product team, you want them to be part of the domain if possible and it should be a mix of highly data fluent people and domain subject matter experts but again, it can be a small team. So a small team carve out but not realigning the entire domain. Scott note: remember domain is an overloaded term. Some domains are sub-domains that are 3-5 people but typically we mean the line of business which can be 10K+ people.

For Chris, when asked about data product teams, he said you should look to have them be long-standing teams. You need an owner per data product but often the development teams will be charged with creating multiple data products rather than only one. Especially as you build the knowledge of how to build good data products, that team can become more efficient. But do not treat your data products like projects, they should be managed/evolved by long-standing teams, otherwise they are more likely to fall to data disrepair.

As a consultant, Chris recommends working with a consultancy :D but the point is that your data product team in the domain should be considered a new team and that new team needs a budget. It might be reallocated budget but hopefully it's new budget. But the consultancy angle is that if the implementation doesn't deliver value, it's easier to cut the consultancy than reassign or lay-off employees. Scott note: I think this is a valid concern but many can't get the budget and many organizations are doing this work entirely internally.

Once you've proven value from and a capability to do data mesh, a typical pattern is to set up a transformation office that will assist other domains in their journey according to Chris. That way you can maintain coordination, find reusable patterns - tech, architecture, people, process, etc. patterns -, prioritize, upskill, etc. That office is helpful in getting teams up to speed but also setting expectations - domains won't magically transform their data capabilities overnight. And look to build an internal community function to encourage cross domain communication and hopefully collaboration.

For Chris, that community aspect is really crucial to identifying cross-domain use cases. The data product owners should be communicating with each other to discuss what they've created in case that sparks new ideas for new use cases. For finding what data products you need to support a use case, start from the mission, then the consumer-aligned data product that would best serve that mission. Once you know what you'd want to have as the end product, you can start to find the necessary source-aligned data products.

Chris wrapped up with a few useful tidbits:

Don't worry about trying to serve future, unknown use cases with data products you are building now. Build for reuse and build for evolution and then evolve when those new use cases emerge.

Source-aligned data products should be - or at least start out - "very specific and very small." Don't try to cram too much in. Scott note: I think this can create some discoverability issues but it is a pattern I am seeing more. See Carlos Saona episode #150 for how that looks in an implementation

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of April 2, 2023 Apr 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#209 Panel: What is BI's Place in Data Mesh - Led by Ammara Gafoor w/ Elif Tutuk and Ryan Dolley Mar 30, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ammara's LinkedIn: https://www.linkedin.com/in/ammara-gafoor/

Ammara's Articles on data mesh Part 1: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start

Part 2: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-organizational-operating-model

Part 3: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-product-thinking-and-development

Elif's LinkedIn: https://www.linkedin.com/in/elift/

AtScale article on data mesh: Principles of data mesh and how semantic layer brings it to life: https://www.atscale.com/resource/data-mesh-principles-semantic-layer/

General AtScale article: The Semantic Layer’s Critical Roles in Modern Data Architectures: https://www.atscale.com/resource/the-semantic-layers-critical-roles-in-modern-data-architectures/

Ryan's LinkedIn: https://www.linkedin.com/in/ryandolley/

Super Data Bros Blog: https://superdatablog.substack.com/

Super Data Bros YouTube: https://www.youtube.com/@superdatabros

In this episode, guest host Ammara Gafoor, Principal Business Analyst at Thoughtworks (guest of episode #133) facilitated a discussion with Elif Tutuk, Global Head of Product at AtScale, and Ryan Dolley, Independent Data Consultant and one of the SuperDataBros (guest of episode #183). The focus topic area was what role does Business Intelligence (BI) have in data mesh - e.g. where does it sit and who owns it - and how do we enable BI to really drive significant value in a data mesh implementation.

A few other episodes that would be good to get a broader picture here on related topics in addition to Ammara and Ryan's episodes are #199 with Brent Dykes and #192 with João Sousa.

Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward.

Scott's Top Takeaways:

It's really important to actually define what business intelligence even means to your organization. If everyone doesn't have a clearly defined picture, it's one of the easiest things to have a major mismatch on expectations - everyone thinks they know what we mean by BI but we often mean different, sometimes vastly different, things.
In data mesh, it's okay to have multiple org groups/layers of BI across the org - say BI teams embedded in the domains and a central BI team too - as long as responsibilities are relatively clear and teams _communicate_. If everyone is working on similar or overlapping goals, it's going to create BI sprawl - e.g. dashboard and report sprawl with weak ownership.
BI must focus on enabling business users, not just data analysts. How do we make it so regular business users can actually drive their own analysis and then own the output? How can they then share their insights back to others.
It's important to focus on enabling BI capabilities over tooling. In BI that feels especially hard because the interface to the data for most people is literally the tools themselves so of course tools feel like they are most important. How we make that distinction, especially to users, is hard and seems to be to-be-determined.
Really consider if BI is only a consumer or, more likely, is it also a crucial producer to the mesh. Are data analysis outputs going to be made available in something like an analytics catalog? Whether you call that part of the mesh or not, how are people reliably sharing the insights they create and how are others discovering those?
Ammara brought up the complexities of how do we go from providing raw data in these fundamental data products to actually doing BI - the tools are not designed to ingest from many sources and the BI professionals are typically not super technical. Do we create domain data marts or mesh aggregated data products? I like the recipe angle Mahmoud Yassin mentioned in episode #103. Data virtualization technologies also probably play a key role here.
Elif made a good point about there is a difference between making the data analytics ready and business ready. How do we bridge that gap? How do we even define that gap so we can recognize it before addressing it?
Design thinking and product are truly crucial to doing BI right, whether data mesh or not. For too long, all aspects of data but especially BI have been focused on outputs - kind of like # of widgets produced - as the metric. It can feel like accomplishing something even if it's not creating value. We need to focus much more on how people will use what we build to drive specific value. How do we create "delightful experiences" as Zhamak always says for consumers of business intelligence.

Other Important Takeaways (many touch on similar points from different aspects):

BI is about "translating the things that happen in the business … into the format that's going to enable people to make informed decisions on it…" as Ryan said. It's easy to get bogged down in tools and techniques but at the end of the day, we're trying to provide intelligence to make better decisions, that enables people to understand and take reasonable actions.
Data mesh gives us a chance to reinvent how we produce BI dashboards and reports. Instead of creating single tables to support each dashboard, we can easily source reliable data without creating rigid and quickly deteriorating or abandoned data assets.
Balancing user experience - especially letting users use their tools of choice - and reliability/scalability is very hard in BI. How do you track lineage into Excel? How do you _govern_ data that's been put into Excel? But if you try to get everyone into Power BI, will you have user disengagement? Scott note: you won't even be able to pry Excel from my dead hands. It's not for reporting but it's the best data poking tool around for small to moderate data sets
We need to start to think about how our mesh data products interface with business users. At the mesh experience plane level, how can they have what Zhamak calls in her book "a delightful experience"? In general in data, we often overlook the rank and file users to focus on the highly technical ones - those are more 'our people' - but we will miss out on a ton of business value if we do that with data mesh.
Is a centralized BI team a mesh anti-pattern? I agree with Ammara and her colleague Emily Gorcenski: nope! You need someone really focused on generating and owning cross-domain use cases or we focus on preventing data silos for nothing. If we don't actually combine the data across the domains, what's the point of data mesh?! Just make sure to share information with other BI teams to prevent knowledge silos.
There is a lot of focus on who sits where and exactly how things interact, which is important in some respects. But it's more important to focus on what are we trying to actually accomplish and then who owns that. Clear responsibilities win, the org chart doesn't! :)
Governance is a key enabler in BI. Not just defining who does what but creating common and easy to leverage interfaces to our data. That can't be done individually for each data product - it would be a ton of work on data producers and consumers would have an awful time. So you have to consider how far does the mesh experience plane extend. Is it into BI or not? If not, how do we achieve scale if business consumers have a disjointed experience and can't easily share with each other.
Mesh consumer-aligned data product owners need to consider if BI users are a target consumer. What output ports are you creating to make it easy for BI users to leverage your data product?
How do we think about the semantics and metrics layers relative to BI in data mesh? BI users want to be able to trust data without deep-diving into transformations and lineage. How do we make that simple and easy? I really don't have good answers here.
Zhamak has talked about her disdain for layers specifically for data in a data product and the large-scale data pipeline approach. I don't believe this extends to the concepts of a semantics layer or a metrics layer if you use this as something data products need to produce their metadata into. It's another aspect powering data discoverability. It's kind of a complicated analogy for how this might work and I am working on the wording…
Governance has to be focused on enabling BI usage of data more than restricting. Of course, controlling who has access and what people can do with it is crucial but with mesh, if we are so focused on producing great quality data across the org, we have to think about how we can enable usage at scale. That's table stakes for data mesh.
We need to create good and low friction ways for people to go from generating an insight to sharing that insight. It's often that insights get locked in at the BI layer. How do we share those insights back to others on the mesh to leverage? Can we easily push insights into the mesh to create a new data product? This is a cultural AND a technology issue that needs to be addressed.
In BI, there is often too much of a bias towards 'give me the complete picture and I'll go from there' which leads to complicated and non-effective customer 360 approaches. Just sourcing a bunch of data without a specific business objective. Humans are curious creatures! Do we need to focus on what information are you trying to understand and why first rather than produce all this data and I _might_ have an interesting insight? I think yes.
One thing that _feels_ like it is often missing from a BI strategy is bringing the business knowledge to the data instead of only bringing the data to business users. How to actually go about bringing the business knowledge to the data, I have no idea. Maybe this is the consumer-driven data modeling and feedback involved in data product creation but it feels deeper than that. It's a concept that is brewing in my mind…
Do we want to have some org-wide defined metrics? Taxonomies and ontologies can easily become overly rigid but having optional adherence can make things much easier when you think about standardized metrics and meanings. It's all a balance and it feels like you should be constantly assessing it because it can easily go the route of too many metrics and your mesh becomes littered with metrics that all mean slightly different things.
BI people are likely to be skeptical of data mesh. Trying to get access to tons of systems and cross-correlate and consume data from multiple sources has historically been a pain - probably an insane pain to be honest. Be ready for pushback even if this is a better solution in the long-run - and likely short-run - for them. Show them how BI in data mesh can be better, that their focus is on value.
We need to get BI people on board with data mesh and what it offers and how it will be better for them in the long-run. BI people are often the data gateways to execs. If the BI people aren't on board, are your executives going to see the benefit of data mesh? Not just buy-in wise but are they literally going to see an improvement in their business operations from data work if the BI people aren't leveraging it?
In general, we need to get better at translating the business need into the BI need then into the data need. Data work is often not as valuable as expected because it gets divorced from what people actually care about. Far easier said than done of course but think about user experience more relative to the entire data consumption and analysis / BI process.
Churning out reports and dashboards is not a good end-consumer user experience. What do business users get out of it to drive their business? We need to tie more BI work directly to what would drive change, what would drive action if we found out new information. We can start by just looking at who is using reports/dashboards before getting more complicated.
Product thinking in BI isn't just about the consumer experience, we need consumers to be part of the conversation and giving more feedback on usage and importance of the BI work. Instead of just having information pushed at them, it needs to be a collaborative push/pull.
Ammara asked about data mesh and BI teams: "…how does data mesh not add to the complexity? How do we not add more strain to the teams? … How do we not add to more to their cognitive load?" How do we make BI feasible in data mesh? It's not as obvious of an answer as it feels initially. We want to get more value leverage, not more work, from the teams.
Data virtualization might be a key aspect of doing BI well. Ghada Richani (episode #206) raved about what data virtualization has allowed them to do as well because it's crucial to easily be able to ingest data you are supposed to have access to. And many tools aren't designed to take data from lots of sources. I have lots of places where I am seeing data virtualization misused but this one seems safe :)
Some people are better served by feeding them reports and dashboards than giving them advanced self-serve analytics capabilities. Do we need to level them up? Do we want to move past "first wave" type practices? I honestly don't know. Some people like to cling to ways of the past but it might also be serving them well. If it's scalable and reliable, do we need to push to 'modernize'? Probably? A bigger BI question than just data mesh related :)

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#208 Making Prioritization a Priority and Focusing on Delivering Value - Interview w/ Srinivas Paluri Mar 26, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Srinivas' LinkedIn: https://www.linkedin.com/in/spaluri/

In this episode, Scott interviewed Srinivas Paluri, CTO at Rentbase. Srinivas was previously part of a data mesh implementation as the Senior Director of Data Engineering at Zillow.

Some key takeaways/thoughts from Srinivas' point of view:

Data mesh advice to former self #1: Ambiguity is inevitable. Don’t be afraid of ambiguity - it's often better than binary thinking - but also be as clear as possible on responsibilities even if the target outcome is ambiguous. Clear responsibilities at least drive things forward.
Data mesh advice to former self #2: Involve product management way earlier. Every product owner needs to understand product ownership, prioritization, and what is the value to the business. And then communicate the value of the work to the business. If that value's not clear, why are you doing the work?
Data mesh advice to former self #3: Create a small team, maybe 5-6 people, focused on enabling new domains to learn how to own their own data and create data products. Scott note: see episode 48 for how ITV is implementing this pattern
Prioritization - and communication around prioritization - is probably one of your most useful tools in data mesh. If you get good at that, teams will often buy-in more quickly. Data producers see changing priorities, not more work. Consumers have a clear understanding of what work is done when and why instead of silence or a link to a Confluence or JIRA page.
Good data mesh product management isn't only focused at the data product level or the platform. You need to look at the bigger picture of how everything fits together to drive even more value. Make sure you have thought coverage of mesh-level product management.
Make sure everyone is aligned on how data engineering work supports business value and the organization's goals. The exec team should understand it of course but don't skimp on informing the data engineers - if they know more, it can help with better prioritization.
Data ownership - and buy-in on that ownership - is crucial and can have immense value. You can drive that buy-in by showing people the consequences of centralized data ownership creating bottlenecks. What more value could have been created if things weren't stuck in the JIRA logjam?
To communicate the cost of data issues, it's very hard to try to put dollar amounts on them but you can show people the business impact and they can start to calculate it themselves - 'this report takes 5 people 3 days to clean up at the end of the quarter' - that has a big impact on business people who can calculate costs far better than the data team.
Work with teams to assess and document wins with data. You might not get to exact numbers but you can show value with a little bit of partnering. Then share those wins and attract others that want some of that data-enabled value creation :)
Look to business partners for setting prioritization around data work. See if you can get them to seek the budget to do the work with you. What are the company's priorities and where should you be investing to support those? How does data work play into that?
Set yourself up to learn and iterate when it comes to architecture. Prepare yourself to fail, and fail fast. It's way too hard to try to get everything right up front, you'll expend too much energy. Embrace that and get moving.
Engineering teams need to be part of key business decisions. Not necessarily as a decision maker but engineering finding out about changes once systems break - this is a pattern that has to change. If you want to be data-informed, data-driven you have to consider the impact to data as part of the product decisions/changes.
You should make sure you support new data owners with technology and capability building. You need to make them capable of owning data, not just give them the responsibility. And that won't be free.
In data mesh, seek out teams that will give you blunt and honest feedback and are willing to work with you to improve your data mesh implementation. You want people who will push you to do better and share in the pain and the success.
Focus on communicating what data mesh changes for your organization, what value it unleashes. Really talk with your central data team and explain what it means for them and that it's not a threat to their jobs, they get to focus on cooler things than building pipelines.
?Controversial?: Don't look to rebuild your entire tech stack to do something like data mesh. See how far you can get with a lot of what you have already.

Srinivas started by saying how important architecture is, yes, but how it's too hard to try to get everything right up front. Instead, set yourself up to be able to change your architecture as you learn more. Fail fast is crucial, you need to be able to get moving sooner rather than later. Otherwise, there is too much risk in building something that doesn't fit needs or even more likely, never building anything because you're always waiting for the perfect technologies.

To do engineering - especially data engineering - well, Srinivas believes engineering needs to be part of the decision making process. If not providing input, at least then being aware of strategic shifts and working with key stakeholders to shift systems to better align to new changes. You can't change your business model and not expect it to not require a shift in what data you need and how you work with it! And you could have started collecting data sooner for the business model shift too :)

As data mesh aficionados know, centralized data engineering can create bottlenecks - and almost certainly will at scale. Srinivas recommends you show people the impact of these bottlenecks through specific examples. You can use that to drive better buy-in for decentralized, domain-based data ownership. It can be hard to quantify the exact impact but it's important to try to at least communicate the outcome of those bottlenecks, how did it impact day-to-day business operations and capabilities.

When trying to push data ownership to domains, it's crucial to make sure you give them the support to actually own the data per Srinivas. That's technology and that's capabilities/understanding. If you are saying data ownership has value and you can show the value, then the organization should support that value, it's not free, it's worth investing in.

Srinivas believes you should focus on making gradual changes rather than sudden shifts in a data mesh or other large-scale data implementation. There needs to be commitment to making change to do it right. And as part of that, look to create and foster close collaboration with users. You need teams to be blunt and honest to help you get to where you need to be, to get to a valuable outcome. That feedback will help you improve your processes and platforms.

If Srinivas could give 3 pieces of advice to his former self about data mesh:

#1 On ambiguity: it can be helpful rather than binary right or wrong type thinking. So get comfortable with ambiguity, the world is rarely black and white. But ambiguity should be more about the ways of achieving or the expected end result - make sure to set responsibilities as clearly as possible because ambiguous ownership rarely works out well.

#2 Get product management involved earlier. They should be there from the start. Understanding product ownership is so crucial to getting data mesh right. You need people who are focused on prioritization and focusing on things that are valuable to the business. If it doesn't have a clear value, should you be doing it? And then when you establish value, communicate it!

#3 Look to create a data mesh/domain enablement function. A team that is specific to helping additional domains figure this out. It's not easy and you will find great advocates that can really help teams get moving quickly. A team of 5-6 people would probably be a good enough size. Scott note: see episode 48 with Scott Hawkins for a great example of this internal consultant team in a box approach

Another thing Srinivas learned looking back on his time at Zillow is to focus on communicating to people what does data mesh change for the organization and especially for them. There is a vague sense of data mesh changing the way we work but what is the actual value we expect to drive. Not a specific dollar amount but what's the vision of the organization of once you reach a relatively data-informed, data-driven state? Faster reactions to market changes? Better identification of new opportunities? Significant cost savings? And again, talk to people about what changes for them, driving value and also responsibility/role wise.

If you want a somewhat visceral approach to showing people why the central data engineering team has become a bottleneck, Srinivas recommends asking how long would it take to clear your current backlog with your current team if no new tickets came in. What about how big of a team would you need to actually clear your backlog based on the number of tickets coming in - is anyone's backlog actually decreasing? What could your organization be doing if you didn't have that bottleneck? What more value could you be creating by really enabling teams to understand and leverage the organization's data? Yes, there will be a cost but we have to invest to create value :)

When driving data mesh buy-in, Srinivas again went back to the need for product management in general. Talking with the producers about why this matters; how you are shifting their prioritization, not just adding additional work; how they can actually achieve this and what are meaningful milestones / incremental value deliveries; etc. Prioritization - and communication around prioritization - are likely to lead to your easiest, happiest paths.

Data mesh product management should not only focus at the micro level - the data products and the platform - but also at the mesh level according to Srinivas. To really drive value at scale, you need interoperability and things working in harmony. Product management is not just about managing the product but the entire suite of products. Think about your data products as an overall suite of information to serve many use cases and drive a huge amount of value.

For Srinivas, when looking at setting your priorities for data work, start with what are the overall organization's priorities, what are the priorities of your business partners? If you want to do data work that isn't valuable to them, will it be valued even if it drives the expected value? Look to find data work that are both valuable and valued and that support the overall organization's priorities. Scott note: this is especially key as you are getting going on a data mesh journey.

Everyone on the data engineering team should understand how their work supports the company's priorities and drives value for Srinivas. And the exec team should understand how the data engineering work maps to value creation too. Sometimes, that can be harder with platform work and the like, but it's important for everyone involved to understand how the data engineering work drives value. That helps map to prioritization decisions as well.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of March 26, 2023 Mar 26, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#207 Zhamak's Corner 20 - Crossing the Data Value Chasm Mar 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Takeaways:

People are seeing early value from data mesh but we can see where it could be a much greater amount but we are still held back by the organizational challenges and even more by the tooling.
In many cases, the tooling isn't good enough yet to change developer behavior - if we remove friction, they will likely want to lean in more on data mesh.
We have to find catalysts at the micro level in data mesh to make massive shift at the macro level. We can't try to change everything through pure force of organizational will. But we haven't found these catalysts yet.
It's easy to get lost in the vastness of change in a data transformation around data mesh. Try to focus more at the micro with a goal of creating cascading reactions to drive the macro.
We need to make mesh data products the first class primitives of information sharing - make them the basic building block of how we create our internal data/AI ecosystem.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#206 Learnings from Delivering and Then Measuring Value of Data Mesh Work - Interview w/ Ghada Richani Mar 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ghada's LinkedIn: https://www.linkedin.com/in/ghada-richani/

In this episode, Scott interviewed Ghada Richani, Managing Director, Data, Analytics, and Technology Innovation at Bank of America. To be clear, Ghada was only representing her own views on the episode, not that of the company.

Some key takeaways/thoughts from Ghada's point of view:

"What I know today is going to change tomorrow." Data mesh is a journey, don't try to get too comfortable, we should always be trying and iterating. Be an explorer. Scott note: strong agree and so does Zhamak
Speed is always a challenge with data mesh - some want to move too fast but others want to boil the ocean to make everything that comes after extremely fast. Work with people to take them along the journey and be part of the decisioning process, don't get ahead of yourself.
Take your stakeholders lockstep along your journey, keep them very informed. Let them control prioritization. That way, they can see what changes are happening and why delivery timelines are extending - they made those calls!
Expose the evolution of the data product to stakeholders. They can then understand tradeoffs, especially in regulatory or other governance challenges. And again, they own the prioritization :)
Don't start with generic requirements, start with stakeholder deliverables. The requirements will emerge from that conversation - stakeholders don't necessarily know what is required technically or as a product structure, but for sure they know what they want to achieve from a business perspective.
In data mesh, data producers should be treated as a stakeholder as well. Make sure they are engaged and that they are getting something from the process. That can be credit/visibility for value creation, true ownership instead of demands/requirements, additional insights about their own domain, lots of information about why this matters, etc.
When driving buy-in and/or getting approval, you have to know your audience. That might seem obvious but it's really not. What do they care about? Have you actually talked to them about what they care about? How can you win them over? How can you make it make sense to them and excite them?
When trying to get approval for a big project, break it down into tangible pieces. If there are 20 things that could be improved around a high-value process, look at them discreetly and deliver more and more of the 20 over time. You get a budget to fix 1 or a few and then prove out value and get a budget for more.
If something is a priority for your team but it isn't for another team that you need to partner with, lean into that friction. Why is it a priority for your team and not their team? Should that be pushed up the chain to align better? You can't make it a priority by pressing them, you need to work the right levers.
?Controversial?: One lever you can pull to drive data producer buy-in is the opportunity cost of not acting. There is a reason for a use-case, something that will improve a process, drive value. If you ask the reluctant data producer to own the risk, the cost of not acting to improve that process, most will say no way and will participate.
Measuring the value of data work is very hard and pretty imprecise. It must be a collaborative process with stakeholders - they are the ones who derive the direct value from the work. How much value is there in speeding up time to deliver data products by 40%? Your data/platform team can't know.
It's crucial to build an environment where data failures fall on multiple stakeholders. And that a data product or other data work not meeting the expected value isn't necessarily a bad failure - your hypothesis maybe simply didn't prove true. Limit the costs and size of those failures but if you aren't failing, you probably aren't taking on enough risk.
?Controversial?: For high profile, high visibility projects/data product builds, check-ins 2-3 times a week is normal and is often helpful. You can identify and attempt to mitigate challenges and risks as they emerge.
?Controversial?: Create a highly visible accountability model for stakeholders and make sure they are aware. It's not about calling people out but holding them accountable and if they aren't doing what's necessary, the executive sponsor should know and can address it.
Virtualization at the query layer has allowed Ghada and team to mature the underlying data products over time while still presenting a mature and complete data product experience to users. Without virtualization, data consumers would not be nearly as happy with the data mesh implementation.
A data virtualization layer has made interoperability easier as well. Connections between data products still need to be found but then they can be codified and offered as a view for others to use.
Discovering and establishing domain boundaries is crucial in data mesh. Sometimes, "orphaned" data that isn't really owned by a data domain might live there temporarily but you should always be looking to find/create a true owner if people are relying on that data.
Your domain boundaries will change and that's okay. Be ready, be vigilant for measuring if it's time for boundaries to change. There isn't a silver bullet way to approach this but with experience, necessary boundary changes will start to become more obvious.
It's okay to have more than one data product per data domain but make sure they are truly incremental to each other. The boundaries and the governance are more important than the number of data products in a domain.
Every data product should have a specific purpose. Not serve only a single use case, we need data reusability. But don't add too much scope to a data product.

Ghada started discussing balancing speed, structure, and control in your data mesh implementation. There are those that want to build everything upfront and boil the ocean but there are also those that want to get to value as soon as possible without taking the product mindset to heart. Work with both sets of people to keep them deeply informed and show them why a balanced approach works better. If stakeholders are very close to the journey, they won't be pushing back on timeline - they can see where prioritizations are changing and the learning is happening as data products - or other aspects of your mesh - are being built. In fact, let them control prioritization where it makes sense so they are the ones causing timelines to stretch and they made the tradeoff decisions.

Keeping stakeholders closely informed also has benefits around control in Ghada's experience. They can understand the tradeoffs relative to governance challenges like regulatory compliance. Exposing the actual evolution of the data product itself to stakeholders helps stakeholders feel comfortable with the process and that compliance/regulatory concerns are addressed.

For Ghada, starting from requirements for a use case doesn't work well, people aren't sure and they get stuck in the details instead of the big picture. Instead, work with them to focus on what they are trying to achieve, what their deliverables are, and then work backwards to figure out what they need to meet their own deliverables. And those deliverables better be tied to value somehow :)

While driving buy-in from data producers, Ghada recommends making them a clear stakeholder in the process. She's found that really deeply informing them of how their data will be used and the value it will drive often gets them excited to participate. Of course, you need to work with them to prioritize the work but showing them the value - or potential value - of a great use case often helps set that prioritization. You also want to make sure to highlight their work, either for them or preferably making the stage for them to present the value delivered from their data, giving them credit and visibility. When you do those things and give people true ownership, not just requirements, many data producers are far more willing to get involved.

In general, when trying to get approval for data work, while Ghada recognizes it can be hard, she has a few good approaches. One is to look at different aspects of what you are trying to improve. Say a process or product line drives significant value for the company. What could you do to tangibly improve the value it delivers? Not as one giant project, break it down into more tangible improvements and seek a budget to tackle one or a few so you can prove out value, getting a budget for additional improvements. Another is to know your audience. This might seem simple but really, you have to learn what drives your counterparts and find a way to communicate the benefits in their own language and address something that matters to them. Make it digestible and hard to resist wanting to tackle the challenge. It's definitely more art than science.

One way Ghada has found to drive buy-in from reluctant data producers is to assign the cost of not doing something to them. Essentially, there is a benefit, a value to doing the proposed work - whether that is increased revenue, decreased cost, decreased risk, increased speed, etc. So, there is a negative of not doing the work the proposed work and you ask the reluctant data producer to officially own the cost of not doing that, own that business risk. Many have become far less reluctant to participate :)

For Ghada, there are two ways in general to measure the value of data work - economic value and impact value. Economic value is slightly easier to conceive if not that easy to measure - if you make improvements to a process or say create a new product line, you measure the incremental revenue it drove or the amount of cost savings. Impact value, the team(s) impacted by the changes have to give the value measurement - what is the value of speeding up a process, improving the data quality, lowering the associated risk, etc. Neither are exact measurements so it's crucial for stakeholders to understand that it's about triangulating and assessing value, not an exact amount of return. And the stakeholders again have to be the ones that assess value. Only they can say what an impact would mean for them, the data team doesn’t have the context to do that. And you need an organizational environment where the forecasts are seen as forecasts, not commits.

It's okay to have failures in data work, according to Ghada. As many past guests have also noted, experimentation is about trying, learning, and iterating. Sometimes the learning is that this won't work or isn't worth the effort. Getting to that learning quickly and iterating to value - or stopping work when that's the right call - is crucial to driving significant value from data work. Your culture must allow for failure or you just won't take on initiatives that are higher risk but higher reward and where the reward justifies the risk. You need to see getting to value and getting something directionally right as a win so you can iterate towards more value.

In Ghada's experience, for high profile, high visibility, high intensity projects/data product builds, it's not unusual to check in 2-3 times every week with all the stakeholders. While it may feel like overkill, you can find miscommunications or friction early and even more importantly you can identify and work to address challenges and risks as they emerge, e.g. if someone is disengaging. Instead of the data team going off and doing a bunch of work to deliver at the end of a months-long project, it's tight feedback loops and iteration and changing priorities through close collaboration. And have a highly visible accountability model - if someone isn't delivering, that should escalate to the executive sponsor to figure out prioritization and an appropriate response.

On the platform side of things, Ghada is very happy with their use of data virtualization for their virtual query layer. As teams have learned how to build and mature their data products, data virtualization has meant they can expose what a mature data product looks like even when the underlying data product is not yet mature. The underlying data creation and curation process is not fully productized or robust in many instances but consumers don't have to care. The views presented to users are controlled by subject matter experts and serve as a type of interface or output port of a sense.

More on data virtualization:

1) Sometimes, that virtualization layer can lead to query performance challenges but usually, that's tied to someone trying to do too large of a query all at once instead of breaking it down appropriately.

2) Data virtualization has made exposing connections between data products much easier. It's just creating another virtualized view. Connections need to be discovered/surfaced manually but beyond that, it's quite easy to do interoperability if the data fits well together.

Discovering and mapping domain boundaries is really crucial in data mesh according to Ghada. And it will get easier as you go along. You really want to consider what are you trying to accomplish with a data product and not have many things loaded into one data product, or it will become overloaded and be hard to evolve/improve well. Data owned by a team that are not the subject matter experts is a likely occurrence but you should look to rectify it quickly. Teams building data products that consume information from upstream data products should not take unnecessary dependencies. At BofA, they create a virtual view that combines the upstream data from the source data product with the data product rather than that downstream data product taking a dependency. They are also having many domains that are represented by one data product but some have more than one data product. The boundaries and the governance are far more important to get right than trying to match a certain number of data products to domains.

Every data product should have a defined purpose in Ghada's view, that's how you can find your data product boundaries. But a data product should also not take on additional purposes, that's scope creep. That doesn't mean it can only serve a single use case, reusability is crucial but when someone tries to find the right source for accomplishing a goal with data, it's best if they have to consider fewer options but still get all the data they want/need. Yes, easier said than done :)

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of March 19, 2023 Mar 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#205 The Gartner Drama Files - Mesh Musings 45 Mar 16, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#204 Driving Towards Data Driven - How to Add More Data to Your Org's Decisions - Interview w/ Stephen Galsworthy Mar 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Stephen's LinkedIn: https://www.linkedin.com/in/galsworthy/

In this episode, Scott interviewed Stephen Galsworthy, former Head of Data at TomTom. Obviously, given he's no longer with the company, he was only representing his own views.

Some key takeaways/thoughts from Stephen's point of view:

Just because the products you sell are made from data, that doesn't mean you necessarily have a good process for leveraging data to make your products better. You need to embed information collection into how you create products, how customers interact with your offerings.
The AI Flywheel: If you can get good information on product user experience, you can feed that into AI systems that generate incremental insights to improve your user experience even more. Hopefully, that generates more users which generate even more information to improve even further.
If data collection on usage is not part of your business model - for a number of reasons - it can be hard to convince customers and/or partners to enable that data collection. Even if it's simply to improve the user experience. Try to add it in to your product development as early as possible.
If your organization is data hesitant, look for existing success stories from data. Look for something that couldn't have happened without the data. And then share that success internally to drum up more interest.
?Controversial?: Data should rarely be THE deciding factor. Data should be a touch point that can strongly inform or give clarity. Use it for giving clarity or measuring as you iterate. Help execs understand it's not magic and that it's not all right or wrong.
It's easy to trust data when it confirms your intuition. How do you use it when it doesn't? How much credence should you give the data?
?Controversial?: High performing companies tend to be those that use data to help make adjustments to their product strategy, using it as a tight feedback loop. It's not as though the data decides but it constantly informs and confirms (but yes, not absolute confirmation).
Don't fall into the trap of collecting all the data you can. But do think about what you could do if you had additional data, what might that inform. Work ahead, you don't get past data the day you implement.
There is a big difference between producing data and producing data as a product. But if we don't incentivize and assist teams to produce data as a product, few will and then your data practices will remain fragile.
Data producers need to be able to understand who has taken a dependency on their data but it's too hard in general right now for them to understand. Better technology offerings here can help.
A great - possibly the best - incentive for a team to produce their data as a product is what their data consumers can provide back to them in the form of insights. How can a team managing an app get more insight to improve their app by sharing their data? That's a cohesive org-wide data practice instead of domains only focusing on themselves.
'In production' should not be seen as the end or the key goal, that's a project mindset. In production just means the start of a new phase for your data product.
To get a team to data-driven, find a data ambassador within a team - and if there isn't one, deploy one into that team. Really encourage someone to be the lead of wrangling their data. Work to develop a passion for data in someone in the team to spread to the rest of the team. Scott note: staying away from infectious analogies
The most data-driven teams have a mindset of always be experimenting. It's not about being right at the start, it's about constantly trying, learning, and getting better.

Stephen started off with a bit about his background from working for companies consolidating information into a product they sell, such as TomTom taking in tons of data to create maps to sell. However, using data as a key aspect of your products doesn't necessarily translate into collecting and analyzing information about how your products are actually used which would allow you to drive improvements like new features or even new products. Organizations that are focused on obtaining and analyzing information of usage and other market dynamics are likely to be those that win in the market more often going forward. Their user experience knowledge is going to be a much tighter feedback loop.

The AI flywheel is something Stephen mentioned that can create a virtuous cycle. You are creating data around interaction points with your products that feed AI to make the products better. The better they are, the more people interact with them generating more interaction points, meaning you can make your product even better. Essentially, collecting and analyzing data to make your product better -> you make your product better -> more people use the product -> more data to analyze to make your product better. However, if it's not inherently part of your business model, trying to pivot to that information gathering practice can be a tough sell for customers and/or partners. Oftentimes, partners aren't even allowed to hand over data. Think about how you'll effectively collect data as early as possible even if you don't start collecting it then.

A good way to build momentum around the importance of data to an organization is start small according to Stephen. It might sound great to try to convert the entire organization to being data driven at once but Rome wasn't built in a day. Show some successes from working with data, find something that couldn't have been done without leveraging data like a new product, and then share those successes internally and encourage more parts of the organization to try leveraging data.

Stephen believes that data should rarely be THE deciding factor in decisions. The human in the loop is crucial. And it's crucial to make that clear to execs - you don't want them to think data is a magic wand / silver bullet but they also don't want the data making the decision. Data is a touch-point in decisions, it's used to create better feedback loops as you iterate towards good solutions. Leveraging data well often isn't really about making the right call, it's about finding better and better ways.

It's very important to frame the role of data in your organization well - it's what differentiates high performing organizations in many cases based on Stephen's history. Again, it's not a silver bullet but data makes it so you can make smaller bets to quickly get to a better solution via tight test, learn, and then iterate loops. Again, you want to make data a companion to execs instead of an either/or to their intuition and experience. Data can help them arrive on the right decision.

Stephen laid out a data maturity journey for an organization or a team: you need to start with collecting data - it seems obvious but you need mechanisms to actually collect data as noted earlier. Once you start collecting data, you don't need to become the most data driven team in the world overnight; start to process and analyze the data - find bits of information and insights to assist as you decide if there should be a formal data analyst/data scientist or not. From there, drive to faster experimentation and improve. The faster you can get solid information and iterate, the better. That will naturally lead to more data democratization as people see the value of the data and want to get involved. That can easily build from a team level up to larger organization level as well. But it's a journey, it's not a switch to flip.

Being cognizant of the cost of data acquisition has been relatively easy for Stephen since his past employers have been involved in hardware/IoT and there was a distinct cost to pipe the data back into the organization. But it's easy to lose sight of the cost of data collection for many organizations. He recommends looking at what data you want to collect by specifically what you want to figure out and why it might inform a decision. Just collecting data for the sake of data isn't good. Scott note: you really want to have these conversations as early as possible because the earlier you have the data, the more you can shape your decisions. If XYZ metric will be crucial for a decision in 6 months, you don't want to start collecting the data for XYZ in 6 months.

For Stephen, the challenge of data producing teams understanding downstream dependencies on their data is getting better but is still not fixed. But we shouldn't focus on just understanding who is using our data - it's the difference between producing data and producing data as a product. If you are producing data as a product, you should be actually interacting with your consumers so it's inherent that you should know who is consuming - and crucially, you know why they are consuming. But there is of course a cost to producing data as a product, lots of engineering time, especially if you don't have the tooling and capabilities, so that should be incentivized whether you are doing data mesh or not.

The best incentive, the 'best carrot' rather than stick, for getting teams to really work on sharing their data as a product has been the insight flywheel from Stephen's point of view. The team sharing their data, if the consumers then generate many useful insights that are helpful back to that producing team, that's how data at an organization-wide scale hopefully works. The 1+1+1+1 = 10 kind of approach but it's also hard to make sure that will happen and that teams make sure to give information back to the producing team as appropriate. You want to try to foster an org culture of 'if another team wins from my data, that is a win for our team too' but easier said than done :D

In wrapping up, Stephen talked about how can you get a team that isn't data driven to start heading down that pathway. There was the data maturity journey mentioned earlier but this is about developing a passion in someone on that team for data and then letting their enthusiasm for - and results from - data get everyone else on board. You may have to deploy someone into that team but make sure there is someone in the team driving them to be more data driven because of passion instead of simply try to use the stick.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of March 12, 2023 Mar 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#203 Panel: Making Privacy Practical and Scalable in Data and Data Mesh - Led by Debra Farber w/ Samia Rahman and Katharine Jarmul Mar 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Debra's LinkedIn: https://www.linkedin.com/in/privacyguru/

Debra's Shifting Privacy Left Podcast: https://shiftingprivacyleft.com/

Katharine's LinkedIn: https://www.linkedin.com/in/katharinejarmul/

Katharine's book: https://www.oreilly.com/library/view/practical-data-privacy/9781098129453/

Samia's LinkedIn: https://www.linkedin.com/in/samia-rahman-b7b65216/

Quick acronyms to know: PETs - privacy enhancing technologies; SMEs - subject matter experts

Scott Note Warning: there is some nerding out about how awesome it could be if some advanced privacy approaches and PETs were implemented at a broad scale across the industry to protect individual's privacy. It's pretty early days so warning about getting your hopes up :)

In this episode, guest host Debra Farber, privacy expert and host of the Shifting Privacy Left podcast facilitated a discussion with Katharine Jarmul, the author of the upcoming book Practical Data Privacy and Principal Data Scientist at Thoughtworks (guest of episode #157) and Samia Rahman, Director of Data and AI Strategy and Architecture at life sciences company Seagen (guest of episode #67).

Scott note: given this is a newer area, I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward.

Scott's Top Takeaways:

Privacy has been dominated by risk compliance historically but it's starting to move past the defensive governance aspects. Privacy has mostly been at the tail-end of the development cycle across systems and data but is starting to shift left across the board much like aspects of data mesh.
Regarding data mesh, given how many additional aspects of development we are asking domains to own, is it fair to ask them to own privacy as well? How can we train people to understand when to use privacy enhancing technology and then make it easy to implement those decisions? Much like other aspects of the self-serve platform.
Privacy tech is emerging and maturing at a very significant rate. What was once a pipe dream or was prohibitively expensive is much closer to being available to the masses. Much like data mesh in general couldn't really have been done before cloud-native tooling/technologies started to mature, privacy is in a similar wave forward. If you want to hear more about specific tech, there's some talk in this episode and Katharine's interview (episode #157).
With the explosion in upcoming privacy-focused legislation across the world - much of which is at least slightly different from each other - we will see a large increase in the need for organizations to do privacy well/better. Shifting left is really the only way to do this scalably or we'll potentially see organizations _stop_ doing a lot of currently valuable data work because the cost of privacy and risk compliance becomes prohibitive. Upcoming legislation may be the thing pushing privacy forward more than anything else.
Privacy is extra crucial when dealing with data leaving the organization, whether in partnership or for those selling their data on a marketplace. Mozhgan Tavakolifard, PhD (episode #154) talked about this where many companies are opting to merely package and sell insights because they can't track usage deep into those partner or data purchaser systems. It's a bit like what Zhamak discussed re data going into the data science area of an organization right now and all governance and visibility at best gets hazy in Zhamak's Corner 18 (episode #195). Will cross-org data mesh help address that? Probably but it's 5+ years away.
Paraphrasing Debra: At the end of the day, privacy is not about compliance. Privacy is about respecting the humans behind the data, not just the data itself. Protecting the data itself is about risk to the organization, that's compliance. We need to encourage a mindset of how do I respect these human's choices and their desires in this context of collecting the data, that's essential.
If you don't do privacy well, there are risks to the company of course. But a big one is that people will still look for - and usually find - ways to get access to sensitive data. People will seek out the value. If you make data easily accessible with the right privacy levels, you can unlock many high-value new use cases in compliant and low-risk ways. Organizations should start to look at the rewards of doing privacy well, not only the risks of doing it poorly.

Other Important Takeaways (many touch on similar points from different aspects):

New ways of doing privacy are going to mean "measurable, quantifiable, verifiable, and auditable tools and capabilities."
We need to think of privacy like any other tooling in the development lifecycle. It's about providing abstractions to make the easy/right calls to the domain experts and having a central support structure when things are more tricky.
Privacy isn't just about the data, it's not simply a metadata-like concept people are trying to add back privacy to data at rest. Much of the important aspects of privacy are about how the data flows through systems and privacy in those flows and each of the systems, not just the end place it gets stored.
In data mesh, the self-service platform will need - currently needs? - to provide privacy-as-code capabilities and so people can easily build data products with privacy built-in instead of added at the end of the process. We can't expose the tech, that's far too complex. How do we provide the good abstractions to make this easy and thus scalable?
We need an ability to almost have a privacy capability as an ingest mechanism - point at a data source and say "we need this anonymized" and it's not a super custom build by the producers. We're just at the start of developing those types of capabilities but we need to make it so it's not all on the producer, consumers can consume with privacy on demand.
We need policies as code or other easily digestible forms of policies - and compliance - and need to train our people well on what privacy means, why it's important, when to apply, etc.
We need tooling to help with federated privacy because otherwise, there is too much privacy context/knowledge AND technology to learn and it won't be scalable. It appears there are some tools emerging but it's still seemingly early days.
Anonymization is often pretty easy to overcome if you just add additional datasets. This is especially a risk in sharing data with other organizations. Anonymization isn't a wand you wave and all your privacy risks get taken care of.
How can we still derive the value of anonymized datasets? It's often much harder so will companies do the ethical privacy aspects or only the required aspects of privacy? We need better, easier PETs to make it easier to still extract value from anonymized data.
How do we balance enough privacy training and not hit information overload? It's hard to get people to learn what's necessary because privacy is such a big topic. We need global and domain-level policies that can again be actually digestible.
Can we measure time to compliance, time to privacy, time to 'doing the right thing' ethically? That would be best to understand where we need to improve but we're probably just at the start of that. This is an interesting fitness function area.
Subject matter experts (SMEs) have so much specific knowledge that you need to leverage them to discover privacy risks and privacy rewards too. Much like any aspect of governance, trying to have the central team make decisions just isn't scaling so we need to make the people in the domain capable enough to handle privacy.
Privacy rewards: in many organizations, there are very high value sets of data that cannot be leveraged for specific use cases due to privacy and other compliance restrictions. Getting to a place where we can easily leverage that high risk but high value data will potentially unlock large amounts of business value.
There are lots of instances of teams finding those high value data sets and using shadow IT to get at them. If that's the only way people can get access, many will completely skirt any compliance and privacy. So getting to a place where they have access but it's according to policy and tracked is crucial to lower organizational risk - from compliance and ethics wise.
From Katharine: "But if you build easier ways to get access and safer, more responsible, more ethical ways to get access, then you have a win-win situation and people are not going to find shortcuts."
Companies are starting to loosen the shackles on data and focus on maintaining privacy but also enabling innovation around privacy-sensitive data. That's a great mindset but there are still many questions on how to do that specifically.
Too many, especially in blockchain, conflate privacy and confidentiality. Keeping something confidential is a security aim. So if you focus on confidentiality, you can't actually use the data you have - no one is allowed access, it's on lockdown.
We need to get far better at risk modeling for privacy. What are the potential harms by the humans? We need to move beyond only thinking about if there might be a breach, what data might they get. We can free up data for far more uses if we do this right but ethics around data usage is just not a common thought. We need to train people to think ethically and about potential harm.
There are multiple issues with anonymized data. Are you taking away the utility? Are you fooling yourself into thinking it can't be de-anonymized? That's a pretty common outcome with adding additional data sets - especially a risk if sharing data externally. Don't treat anonymization as your hammer and everything looks like nails.
We need to teach developers about differential privacy which is about "bounding the probability of someone learning" a specific thing. Differential privacy "got a bad reputation" but we can add noise and maintain accuracy now. It is the "gold standard for anonymization".
Healthcare patient data is one of the biggest challenges in privacy because you want to maximize the efficacy of care but also maximize privacy. And then how do companies take the data of the individual to extrapolate further to see broader trends?
We need to get people upskilled so they can understand when to transform data in privacy preserving ways - and then the self-serve platform needs to make it easy for them to do that. But we don't have great industry-wide understanding on how to do either of those that well yet.
Self-sovereign identity, while very interesting, is probably a long way away from being widely adopted. There needs to be a lot of industry collaboration and agreement and it's not really a big benefit in many areas based on the legal requirements versus cost. It would be great for company-to-company interoperability with privacy but who will build it? The 3 panelists were very excited about it though :)
Privacy and data sovereignty are going to be intermingled in interesting ways in data mesh. Querying data where it is instead of piping it all over the world* will help maintain privacy and comply with laws - many countries don't allow data to be exported as is.

* see Zhamak's Corner 13 episode #173 that covers some of what querying data where it is means and that's not necessarily about source systems but it does mean not moving it without necessity

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#202 Creating a Balanced, Sustainable Approach to Your Data Mesh Journey - Interview w/ Kiran Prakash Mar 05, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Kiran's LinkedIn: https://www.linkedin.com/in/kiran-prakash/

Kiran's article on the 'Curse of the Data Lake Monster': https://www.thoughtworks.com/insights/blog/curse-data-lake-monster

In this episode, Scott interviewed Kiran Prakash, Principal Engineer at Thoughtworks.

Some key takeaways/thoughts from Kiran's point of view:

?Controversial?: You MUST have exec sponsorship to move forward with your data mesh implementation. You need the top-down push for necessary reorganization when the time comes. Scott note: only kinda controversial, really more often ignored :D
?Controversial?: Data mesh, if done well, doesn't need to have a huge barrier to entry. That's a misconception. If you think about gradual improvement/evolution, you'll be on the right track.
"The Curse of the Data Lake Monster" was like the data field of dreams - there was expectation that if you build a great data lake, value will just happen. If you ingest and process as much as you can, the use cases will just happen. And it really wasn't the case. So we should apply product thinking to data to focus on what matters.
The 'Curse' was a manifestation of Conway's Law - the strong separation between IT and the business led to mismatched goals and subpar outcomes. With microservices, that started to be much less of an issue on the operational plane so why not try with data?
It's easy to lose sight of Conway's Law and aim for distributed architecture first but the organizations doing data mesh well are changing their architectural and cultural approaches and patterns together. Don't try to do the architecture first, you really don't know your key challenges yet.
It's very important to have a target operating model and get clear on your organizational vision and purpose around data - how will you use data?
Once you have an organizational vision and purpose, domains should start setting goals aligned to that vision and purpose.
As others have noted, don't get ahead of yourself, work in thin slices for your data mesh implementation. Stay balanced at an overall level between the data mesh principles as you add more and more thin slices but don't try to solve all problems up front.
If you modernize your legacy software but don't change the organization, expect to do the same type of modernization in about 5 years.
To really get to a scalable approach to data mesh, you should look for organizational and process reuse as much as tech/architectural and data reuse.
Move from measuring outputs to measuring value outcomes. Sounds simple - it isn't - and it's crucial to changing your mindset about how you approach data.
?Controversial?: Use the 4 key metrics from DORA (https://dora.dev/) to measure how well you are doing in your software engineering. And a key aspect of data mesh is about applying good software engineering practices to data after all.
If you want to measure the value of your data work, you need to break it down into tangible objectives. Ask the owners of those objectives to provide the value of meeting the objectives. Then look to measure how much data work contributed to achieving those objectives.
Think of a use case as a value hypothesis: you are making a bet that something will have value. It's okay to be wrong, that's the nature of betting. But limit the scope of your mistakes so you learn and adjust towards value instead of making big mistakes.
?Controversial?: If you don't have a culture where it's okay to fail, it will be very hard to do data mesh well. Scott note: it will essentially be impossible in my opinion
Many times what people consider minimum viable product is neither minimum nor viable. This is often due to a culture where you can't test things with users when they are still very rough. That will limit your success with data mesh. However, most people are reasonable so if you read them in that this will be a rough sketch/first iteration, they usually are on board to help you iterate to good.
"Architecture is about the important stuff. Whatever that is." - Ralph Johnson (via Martin Fowler)
Always think about necessary capabilities and build to those. The most important are those capabilities you need now. Don't get ahead of yourself.
The 'data platform' is really a misnomer, there will be multiple platforms. Users care about services, not if you have one platform or five or more. Don't have platform sprawl but don't over-centralize, that usually leads to scaling and flexibility challenges.

Kiran started off by talking about a blog post of his with a colleague from 2019 called "The Curse of the Data Lake Monster" - lots of clients were building big data lakes and it wasn't providing the expected value. There was an expectation that if you ingested and processed as much of your data as you could, it would create great use cases and lots of value. But it didn't happen. Value doesn't just happen without concerted and concentrated effort. So Kiran asked why aren't we applying product thinking to data to figure out and focus on what matters to drive value. What would happen if we focused on outcomes instead of platforms? What if we measured value not how many terabytes were processed and stored?

A key reason Kiran feels the 'Curse' happened was the strong separation between business and IT. That separation meant both were seeking different goals instead of collaborating. IT was focused on building things instead of solving business problems and the business side was focused on doing what they could, not building out a scalable and robust data practice. Conway's Law in action. We saw microservices really help to tackle those same issues on the operational plane so the data side of the house was definitely ripe for some product thinking-led innovation.

Data mesh avoids a lot of the issues of past data approaches by not leading with the technology - the first two principles are not tech focused. For Kiran, many (most?) of the organizations that are getting data mesh right are respecting Conway's Law and shifting their architecture and organizational approaches together. But in thin slices so as to not put too many eggs in one basket and make reasonable progress. And they are getting the exec sponsorship because while you don't want to reorganize your entire company upfront to do data mesh, you do need some top-down pushing to actually drive the necessary org changes when appropriate.

According to Kiran, while many people think data mesh has a high barrier to entry, that shouldn't be the case. There should definitely be a target operating model at the organizational level and organizations need to keep that in mind but it's not as though, again, you reorganize the organization all upfront. Organizations also need to really answer the question of what are they trying to do with data and what would doing data mesh well drive for them - if that's not crisp, they probably aren't ready to do data mesh because their business strategy isn't aligned to or contingent on doing data well.

Once the organization has the target operating model and vision down, Kiran recommends that domains should start to set their own specific goals aligned to the broader organizational vision. They should work on some hypotheses on how to achieve those goals and how they plan to measure their progress towards those goals. Start to build out your thin slice approach to making progress towards your vision and goals. Don't get super far ahead of yourself, look to progress at a meaningful but reasonable pace and tackle as little as is necessary now while still making sure you are aligning with the organizational vision. Keep your eyes on the prize, don't take on too much now. And yes, easier said than done.

Kiran pointed to a quote he read about if you modernize your legacy software stack but don't change your organization, you will need to do the same modernization in about five years. The same goes for data - if you are taking on data mesh from a tech-first approach, you'll just have to do all the same modernization again and you won't get a lot of the potential benefit from data mesh. Decentralizing the architecture will only really change things if you change the organizational aspects too. People, process, technology.

For Kiran, we need to really start to focus more on measuring value outcomes instead of inputs - how many terabytes or operations per second isn't directly tied to value. Teams need to have it made clear what is actually valued and valuable. In many large organizations, there are often less clear links between data work and business value so you have to educate and incentivize teams to do the high-value data work.

When thinking about trying to measure the return on investment in data work, especially data mesh, Kiran recommends starting by breaking it down into more tangible goals and measuring the value of achieving those goals. Then you can start to say how did the data work contribute to achieving those goals. But a data team can't really know the value themselves, whether inside the domain or not. And by breaking things into smaller goals and objectives, you can more quickly iterate towards value with tight feedback loops. Instead of large-scale projects, you build to larger and larger objectives through breaking things down and achieving meaningful micro progress that leads to large macro value.

Kiran talked about thinking of use cases as value hypotheses: you believe it will have value and thus you are making a bet. And it's okay to get things wrong, just limit the scope of the mistake so you can use the missteps as learning so the larger macro bet has a much higher chance of paying off. This is iterating to value, those tight feedback loops. If you don't have a culture where it's okay to be wrong, okay to fail, then data mesh is potentially (Scott note: almost definitely) not right for you.

Minimum viable product is often neither minimum nor viable in Kiran's experience. If you can't put something pretty rough in front of stakeholders, you waste far more time and effort building in wrong directions and are less likely to hit on success. But that's often out of the control of the product team. So it's a catch-22: do you put in a lot of effort to get it well past MVP or do you risk losing face? So we need a culture where we can actually do thin slicing well to really derive the most value out of building software, whether that is apps or data products. That incremental value delivery is really crucial to maintaining nimbleness as you scale. If you can't actually deliver in thin slices, it can significantly increase risk as you are making larger bets. But Kiran's seen that if you spend the time to explain the need for thin slicing and that what they are looking at is the 'sneak peak' and you just want feedback, most people get it and are reasonable. But you need to communicate about it :)

Kiran used a phrase Martin Fowler uses often from Ralph Johnson: "Architecture is about the important stuff. Whatever that is." When thinking about decisions that are hard to reverse, spend a lot more time and care, but the ones where it's easy to reverse, those probably don't make up the core of your architecture. When building your architecture, it's again important to build incrementally instead of trying to get it perfect from the start. Think about necessary capabilities, not technologies. In data mesh, that is about data product production and then monitoring/observing, data product consumption, mesh level interoperability/querying, etc.* Start to map out what you need and then think what level you need from a capability standpoint and when. You don't need to build out capabilities for when you have 20 data products when you have 1-2 data products.

* Scott note: Kiran went on for quite a bit here about necessary capabilities for data mesh late in the episode if you want to listen.

Quick tidbit:

Leverage the 4 key metrics from DORA to measure how well you are doing your software engineering as applied to data - 1) Lead Time to Changes (LTTC); 2) Deployment Frequency (DF); 3) Mean Time To Recovery (MTTR); and 4) Change Failure Rate (CFR) https://dora.dev/

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of March 5, 2023 Mar 05, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#201 Choose Your Blast Radius and Other Lessons Learned Across 10s of Data Mesh Implementations - Interview w/ Vanya Seth Mar 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Vanya's LinkedIn: https://www.linkedin.com/in/vanyaseth1809/

In this episode, Scott interviewed Vanya Seth, Head of Technology for Thoughtworks India and Global 'Data Mesh Guild' Lead for Thoughtworks. To be clear, Vanya was only representing her own views on the episode.

Some key takeaways/thoughts from Vanya's point of view:

Data mesh is at a similar inflection point to where microservices was a decade ago. Let's not relearn all the hard lessons they already learned. We should adapt/contextualize to data of course but we can skip a lot of the anti-patterns.
Similarly, many people are stuck thinking "there's no way that could work" regarding data mesh like they were when people suggested development and operations be combined in DevOps. It's understandable - it's hard to imagine a post monolithic world when all you've known is monoliths.
?Controversial?: We should try hard to prevent creating the fear of missing out (FOMO) for those not doing data mesh. If data mesh isn't right for your org, especially if it isn't right at this time, that's perfectly okay. Don't take on the overhead cost of data mesh if it won't bring more value than cost. Scott note: PREACH!
?Controversial?: Some CDOs or CAOs, their organizations don't really get the value of data so they are implementing data mesh to try to prove out value and make their mark. That can obviously create issues if their organizations aren't ready.
A few indicators an org is ready for data mesh (see below for expanded context): A) data/AI investments are not delivering the promised/expected returns and/or it's hard to point to the value delivered in general from data/AI investments; B) the organization is attempting to throw more people at centralized data management and it's not working (platform included); and C) there's extremely unclear ownership around many aspects of data, especially who owns aspects of hand-offs or who owns the end data asset - how can a consumer actually ask a question about the data with no clear owner?
"Innovation in queue syndrome" = your innovation agenda is "in queue" and keeps getting deprioritized because you are dealing with everything else first just to keep your data practice flowing.
Use value stream mapping to understand how your organization drives value from business processes and where there is value leakage. Especially useful if data work isn't driving value.
We should take a lot of learnings in how microservices service discovery evolved, especially the tooling, for data mesh. There is no need to reinvent the wheel on this.
Some existing tooling from the microservices space is just fine for data mesh too. We don't need to invent new tools when existing ones - which are already robust and mature - can be extended or even used as is.
Platforms aren't about the tooling, they are about the holistic user experience - how do you stitch things together to automate the toil and let users focus on what matters? The tooling is under the hood, not the main interface.
Users of your various data platforms should not be directly interacting with tools for the most part. It should be about abstracting away the tools and making it easy for them to interact with data, not the tools of the platform.
!Crucial!: "Choose your blast radius." Far too many are looking to change the entire organization at the start of a data mesh journey instead of limiting scope to a reasonable level. Find one "courageous" domain to move forward.
"Nothing succeeds like success itself." Get to a data mesh win that you can tout quickly so others will get bought in and see the value and want to participate. Incremental value delivery builds interest and momentum.
Build your platform at the same time as you're building your initial data products. Far too many platforms are built with tools as the focus instead of automating away toil and focusing on necessary capabilities.
!Crucial!: Evolvability should be a first class concern when building your platform, just like with any product. You must be able to continue to improve and change to meet needs.
Focus on the abstractions and the ubiquitous language - e.g. business people don't care what the technical underpinnings of a data product are, they care about what it means for them and how they can access/leverage it.
When starting your data mesh journey, look at the use cases to decide how much of each pillar you really need. Don't overbuild early. If you only need a minuscule amount of governance, great. If you don't actually need the producing team to be overly involved in ownership, awesome. Don't go for full data mesh at the start.
What you should focus on relative to your early journey is unique to your own situation and use case. Don't worry about competitors or how others are starting - their circumstances are their own.

Vanya started with a bit about her background and how deeply entrenched she's been in the microservices space - that played into the overall conversation a lot. Both Vanya and Scott agree if we want to do data mesh right, we really should take learnings from microservices and DevOps so we don't have to relearn what they already did the hard way.

For Vanya, data mesh is at a similar inflection point to where microservices was a decade ago - people were extremely skeptical that developers and operations could even work together, much less around combining them in a singular approach with DevOps. It's hard to imagine a post monolith world when all your career and experience are with monoliths. We have to be somewhat kind to those people in understanding that change is hard and scary :)

But, as a counter, for data mesh Vanya believes (and Scott agrees) we must try to prevent creating the same fear of missing out (FOMO) that microservices had. For many, if your organization wasn't doing microservices, it wasn't seen as a cool place to work and that all the best developers were at companies doing microservices. We don't want that in data mesh because it will lead to lots of wasted effort for companies that shouldn't be doing data mesh now or potentially ever.

According to Vanya, there are a few really good indicators an organization might be ready for data mesh. Before we get into the 3 she listed, a few things that might be indicative of indicators (Scott note: I know, I know, silly Scott phrasing) are constant displeasure of the kinds of initiatives they've been doing in the data and AI space - there is a constant pressure to prove the value of data and AI investments but really, an inability to do so. Long and lengthening cycles to return on data work/projects. A biggie is an ever-growing platform that is trying to do too much and hasn't been delivered - trying to boil the ocean.

So the 3 indicators data mesh could be a good fit that Vanya listed were:

1) Investments in data and AI aren't delivering expected value and it's hard to actually point to the value that is being delivered. Users aren't getting "the right data at the right time with the right quality".

2) Large and growing central data teams where trying to scale is done by throwing more people at the problem and it just isn't working. When automation would be better, they add people.

3) Confusion around who owns data when and why. Who owns the handoff between systems? Who owns the documentation and metadata around data? When someone has a question, how hard is it to find who owns the data?

Vanya highly recommends using value stream mapping to understand how you drive value with business processes and especially where are value leakages; this can be data or not, and should be applied to both analytical and operational data processes. You can understand better your business processes and expected outcomes - if something didn't meet expectations was that because expectations were wrong or did something happen along the way to lose value? Value stream mapping gives you an objective and neutral starting point and helps identify problem areas - value leakage - where you can prioritize what to tackle first.

In microservices, Vanya pointed to how challenging service discovery started to become until tooling came along - specifically mentioned Consul - so we really don't have to reinvent everything in data mesh. The tools out there, especially those in the open source space, are really making nice progress - specifically mentioned DataHub - compared to where they were 2 years ago at the infancy of bleeding edge data mesh adoption. Overall, we should 1) look to existing tools to see if we can use them as is; 2) look to extend existing tools where possible to cover incremental needs specific to data mesh; and then 3) look to create new tooling that is required for data specific challenges. Again, don't reinvent the wheel.

For Vanya, one thing many organizations struggle with in data mesh is the self-serve platform - what is the goal? Circling back to an earlier point, it's not about building the most amazing, ocean-boiling platform. It's about stitching tools together to automate the toil away - how can you create a holistic user experience to focus on doing the value-add? The value of the platform to the users is the abstractions away from the tools that make it easy to focus on what needs to be done to drive value from data, not play with the shiny tools. Focus on enabling interacting with the data, not the tools of the platform.

"Choose your blast radius" is a key phrase for Vanya. Think about scope appropriately and don't try to bite off more than you can chew. You don't have to reorganize your entire organization on day one to do data mesh, that is far too much of an upfront cost and makes failure a massive cost. Look at how it was done well in microservices: thin slices, not taking a sledgehammer to the monolith. Gradual evolution is sustainable, a revolution either succeeds or it doesn't - don't take on risk that isn't actually beneficial!

"Nothing succeeds like success itself," was another line from Vanya. It's crucial to get to an early win or two to show off to the rest of the organization proving data mesh delivers value and getting them interested in participating. 'Hey, we did this and it was a big win, who's next?!' It's not just about showing value, it's about showing there was a reasonable encapsulated timeline, not just promises. That incremental value delivery creates momentum and the more momentum you have, the more you can get people on board.

As many past guests have noted, it is a pretty bad (Scott note: fully terrible) idea to build the platform and then bring it to the users when it's done in Vanya's view. There are far too many unexpected friction points and finding those and tackling/automating away the actual friction is where the platform adds value, not bells and whistles. You want to find those as they emerge and work on tight feedback loops - that's product thinking! And if you don't make evolvability a first class concern, you are not building your platform as a product either.

For Vanya, it's pretty easy for tech people to focus on the tech, whether that is in data or not. But the overall organization doesn't care about the tech, they care about what they can do. So it's crucial to find the ubiquitous language and make your implementation and platform about what are people trying to do. The user isn't accessing S3, they are accessing the Inbound Marketing Conversion data product. S3 is simply a mechanism to accessing the data and insights.

When considering your thin slice early in your data mesh journey, it's okay to have a very unbalanced slice in Vanya's view. This has been mentioned before but it's important to reiterate. If you only need a bit of one of the pillars but you do need more capability in another of the pillars, that's absolutely okay. Don't build today for all the problems of 6mo from now. You want to focus on tackling the toil of today.

Quick tidbits:

Vanya's phrase "innovation in queue" is when an organization keeps putting off their innovation agenda for more immediate concerns - everything innovative ends up getting deprioritized in the queue.

Most data mesh journeys are taking six to seven months to really prove out doing data mesh and value. Scott note: this seems to be standard for larger organizations but a complete POC means faster follow-on for additional use cases. It's a balance!

In some organizations, if data is not really valued the CDOs or CAOs are looking to implement data mesh to show the value of data but their organizations often aren't ready and trying to do data mesh just creates more challenges than benefits.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

{Bonus} Zhamak's Corner 19.5 - How Does AI/ML Change When Trust is Automatic? Mar 01, 2023

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

This episode is part of the greater AI/ML conversation I had with Zhamak but it's super important to emphasize the importance of trust - enough so that I created a separate quick episode on it. Not just trust in the data itself but that there is easy access and there will be going forward. A lot of the things we have done in data historically has been defensive in nature - especially grabbing a copy of the data now because who knows when you'll get access to it again.

What if we can implicitly trust that there has been care and foresight in preparation of the data I find, that there is an owner I can ask if I'm confused or curious, that my access won't suddenly go away or that what's there won't suddenly change without my knowledge? In ML/AI, the data scientists have done things in ways that made sense to their situation and challenges. What happens when we make trust inherent? What incremental value does that drive?

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#200 Zhamak's Corner 19 - An AI/ML Future Without So Much (Needless?) Complexity Feb 28, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

This episode is part of the greater AI/ML conversation I had with Zhamak. To start, Zhamak recognizes we aren't where we want to be in terms of capabilities - ways of working or tooling - to make this a reality just yet. But, if we can make it so data scientists can trust and easily consume from data products - that we create data products that don't care what use case type - regular analytics or AI/ML - can we remove a lot of the complexity they face? Do they need feature stores for data they aren't transforming? If they can get continued access and know the quality, why create a separate process that has fragility instead of trust the data product owners upstream?

I wasn’t smart enough in the moment to talk about do we need to have a copy of the training data itself for reproducibility but folks smarter on ML than I am can answer that one, probably in the affirmative. But overall, there is a lot of complexity in the way we do AI/ML because data scientists can't trust the sources of their data and they feel the need to take control because if they don't, their models break. So we need to earn their trust and show them a better way. But again, we aren't there yet, so let's work to make this a reality in the future.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#199 Finishing Your Data Marathon - Driving to Action from Data - Interview w/ Brent Dykes Feb 26, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Brent's website and book: https://www.effectivedatastorytelling.com/

Brent's LinkedIn: https://www.linkedin.com/in/brentdykes/

Brent's Data Analytics Marathon Forbes article: https://www.forbes.com/sites/brentdykes/2022/01/12/data-analytics-marathon-why-your-organization-must-focus-on-the-finish/?sh=2af698743c3b

In this episode, Scott interviewed Brent Dykes, Chief of Data Storytelling at his own firm, AnalyticsHero. Scott asked Brent to be on after João Sousa pointed him to Brent's content.

Some key takeaways/thoughts from Brent's point of view:

Focus on the so-what, what should people take away and do from the insights, not the sausage making of the insights. Execs want to eat the dang cake, not hear about how you made it!
!Controversial!: You want to get to a place where you remain more neutral until the data informs your view. It can cause more cognitive load to update our views instead of waiting for the data to speak first.
Many organizations lose steam in actually driving action on analytics, they don't drive change with data - they fail at least one of the following: generating actual insights, communicating their insights well enough to drive action, and/or actually acting on the insights.
The analytics marathon: data collection -> data processing -> data visualization/reporting -> data analysis -> insight communication -> take action.
Many companies stop the data marathon after getting to the visualization/reporting step. They aren't driving the results they want so they focus more time on collection and processing instead of finishing the race and get caught in a loop.
Really consider why are you doing data work. It's not to simply do analytics, to build the dashboards and reports, it's to take action on the data and affect change through more informed decisions.
It's easy to like the data when it supports your narrative. A strong analytics culture bends decisions and thinking to the data instead of the other way around.
Companies with good analytics practices, that take action on data, typically have at least an executive sponsor around analytics if not buy-in from the entire leadership team. There is an understanding that actions are driven by data when possible. And there is a test and learn culture.
Executive support and a test and learn culture are what drive results from analytics - many companies buy the same tools and have vastly different degrees of success with data.
If your organization isn't doing analytics well, the best way to drive towards doing analytics well is get to wins from analytics and build momentum to drive to higher and higher exec sponsors.
?Controversial?: Sometimes to drive necessary understanding, in documentation - or other ways of bringing data users up to speed - you really need to show the lineage all the way back to how the data is even collected.
The more you share data, the more likely information is to be misunderstood. Beware the difference between what a metric means and what people _think_ it means.
?Controversial?: To get to scale, we use passive communication - read mostly documentation - about our data. But to truly drive to understanding, we also can't shy away from active - read person-to-person - communication. Scott note: Episode 150 w/ has some interesting insight on how far documentation should go.
?Controversial?: Storytelling is often the easiest way to sway people with data because human brains have evolved to accept information via stories. We've been doing it for 1000s of years.
Mature organizations understand the data can be wrong and prepare for that. Move fast and make incremental moves instead of a big bang approach. You learn more and can do better the next time even on actions that weren't as valuable as expected.
?Controversial?: Business analyst roles should evolve to be more like personal trainers - helping people learn how to do good analysis and then communicate their insights. They won't work themselves out of a job, merely get to a place where they focus on the bigger, harder, deeper, more valuable questions.

Brent started with a bit about his background and why he titled his book "Effective Data Storytelling: How to Drive Change with Data, Narrative, and Visuals." There are a few places many organizations fall down in driving change via their analytics whether that is a failure to generate actual insights, a failure to communicate insights well enough to drive action, or a failure to actually take action on the insights - he's most focused on the communication of insights, a place often overlooked. You can find the best insights in the world but if you can't communicate those insights well enough, no one will understand them and/or understand the potential impact of acting on them. Communicate well enough to drive change!

The analytics marathon is one of Brent's big analogies for explaining where organizations fail along the path to taking action on their insights. There is data collection, which pretty much all organizations do. Then data preparation into data visualization. But this is where many orgs fall off because they are simply reporting on what's happening, the descriptive analytics and not actually driving to diagnostic analytics. Instead of doing deeper analysis, they believe their problems lie in what data is collected so they try to collect more data, thinking it's simply a lack of information instead of a lack of analysis. And then of course, once you do the analysis, you still have to communicate and take action.

For Brent, a few common indicators an organization will likely have a good analytics practice include: 1) an executive sponsor for being or becoming data driven. Possibly the entire leadership team. 2) a general commitment to driving actions from data where possible. It's "how we do things." 3) A test and learn culture in the organization that's supported by data.

If an organization isn't yet data driven, isn't doing analytics that well, Brent recommends getting to wins and slowly moving your executive sponsorship up the ladder. It might start at a Director level and then after you build momentum, people will take notice and you can climb to VP level and then C-Suite level. It's about showing the value of analytics and plugging along so you have proof points when you move the conversation higher in the organization. Rome wasn't built in a day and neither is a good, organization-wide analytics practice.

As data initiatives have become more ambitious, it's often meant ownership has become more murky according to Brent. What was once data that was essentially only for the generating team is now a potential core value asset and driver for the organization. And that opens you up for much more misunderstanding. Focusing on making sure information is understood - not just data is made available - is crucial to making good decisions with your data. There's often what the metric means and what others assume it means.

Brent shared his views that we need both active and passive ways of sharing context around data. Passive is the metadata, the documentation and the like. If we want to scale, passive is crucial. Self-service can't just be a pipe dream. But too often, people in data want to only do passive and ignore the people-to-people conversation. But often that's key to nuanced data or crucial to working with key people making big decisions based on data.

For 1000s of years, humans have been passing information via stories - human brains have evolved to share information via stories. We inherently want to know where the story goes. For Brent, mastering that storytelling with data and about data is the best way to convey the information we generate and discover with our data. If you don't communicate insights to those who can take actions, in a way they can understand, they won't take those actions :)

For Brent, execs rarely want to hear how the sausage was made via data. You want to show them what you've discovered and what they should do with that, not how you came upon that. It can be important to show people the sausage making isn't that hard though, especially trying to enable a team to do self-serve analytics. Really consider which is more appropriate to the situation.

It's pretty easy to like what the data is saying and point to the data as backing you up when you agree with it in Brent's experience. But a truly data-driven culture will focus on updating their thoughts and processes based on what the data says, improving their understanding via the data instead of trying to bend the data to support our hypotheses. It's about getting to a place where you try to remain more neutral until you hear what the data says and shape your vision around that.

On data-driven versus data-informed as semantics of what we're trying to get to, Brent likes the idea of data-driven. For him, it means really leaning into the data. Part of that is understanding and accepting that sometimes the data is wrong or we didn't ask the question in the right way - that's just getting to data maturity. And data-driven companies recognize the value of learning - there is still value from experiments and moves that didn't have as much benefit as expected when digging into the why. You have incremental understanding even if not direct incremental business value. And it sets you up to do better on the next iteration.

When asked about where will business analysts fit in data storytelling in the future, Brent sees them like personal trainers - they won't be doing the work but showing people how and assisting them until they can get to a level the BAs aren't as needed. That's for the analysis and especially the insight communication. Pure self-service analysis is nice in theory but you need a way for people to get help and make sure they aren't hurting themselves :) If more people are far more capable, that means the BAs can focus on the more valuable, large-scale questions.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of February 26, 2023 Feb 26, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#198 How Do We Make Data Contracts Easy, Scalable, and Meaningful - Interview w/ Ananth Packkildurai Feb 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ananth's LinkedIn: https://www.linkedin.com/in/ananthdurai/

Schemata: https://schemata.app/

Data Engineering Weekly newsletter: https://www.dataengineeringweekly.com/

In this episode, Scott interviewed Ananth Packkildurai, Author of Data Engineering Weekly and the creator of Schemata.

Scott note: we discuss Schemata quite a bit in this episode but it's an open source offering that I think can fill in some of the major gaps in our tooling and even ways of working collaboratively around data.

Some key takeaways/thoughts from Ananth's point of view:

!Important!: Collaboration around data is crucial. The best way to get people bought in on collaboration around data is to integrate into their workflow, not to create yet another one-off tool in yet another pane of glass.
?Controversial?: There is so much friction between initial data producers - the domain developers - and data consumers because they are constantly speaking past each other. The data consumers have to learn too much about the domain and the data producers rarely really understand the context of most analytical asks.
Data creation is a human-in-the-loop problem. Autonomous data creation is not likely to create significant value because the systems can't understand the context well enough right now.
As Zhamak has also pointed out, there is far too much tool fragmentation. It made sense with lots of readily available VC money and finding how to approach things with cloud but we need holistic approaches, not spot approaches to things like data quality, observability, lineage, etc.
!Open Source Product!: Schemata was created to enforce certain rules in a cooperative platform around data sharing specific to data schemas to help alleviate much of the above friction.
Data needs to really take a lot of learning from the platform engineering for microservices space. They make it easy for teams to test new services or changes, deploy, etc. In data, we are asking domains to own their data without giving them the tooling to easily do so. It's too much of an ask. Scott note: PREACH!
?Controversial?: In general, we need better ways to share about what data is already available and what data we expect. This is where data contracts as a platform instead of tooling only becomes important.
!Scott Controversial Note!: Many get data contracts woefully wrong. Data contracts aren't about _only_ the contract. They signify a relationship that has contractual terms but think about a vendor - is your only interaction, communication, set of expectations, etc. the contract or is a part of the relationship with strong guarantees?
Consumer-driven data contract testing is important. It is defensive in a way - if my upstream changes, I don't necessarily want to consume from it. But consumer-driven testing can also be a great part of the conversation around how consumers are actually using the data - it's a programmatic way to describe usage to producers. Scott note: if we can make consumer-driven testing easy, great. But we need to reduce the burden on producer and consumer to ensure data contract compliance.
We need to be able to take consumer requests and translate those to producers as well as give guidance to producers about effective - cost or otherwise - ways of meeting those requests. E.g. including a user ID might seem easy to a consumer but it could be very expensive for producers in their standard way of calling user ID. How can we put a prescriptive way in front of data producers to make it easy to meet requests?
?Controversial?: Pull requests > requirements gathering. A consumer can show exactly what they want and a producer can either approve or deny but it generates a better conversation.
Teams need to figure out coordination and communication in a decentralized data modeling world. That is the federated aspect of data mesh - if it's all decentralized, nothing works with each other and you end up with garbage data at the organization level despite domains having well modeled data for themselves.
!Controversial!: Schemata believes there needs to be a core data domain for data that links most other domains. Scott note: While not rare in data mesh, a core domain can become a bottleneck and may not give you the flexibility required. Adevinta (episode #40) discusses leveraging a core domain in depth.
It's very, very valuable to provide automated feedback via your platform to people considering creating a new data asset - whether that will become a data product or not - including how well it fits in the organization's data landscape. Are you creating something that can actually be leveraged for other use cases? Does it integrate well with existing data assets/products?
It's crucial to have something that gets all the parameters of a data contract on paper. Think about negotiating an agreement with a vendor - is it all just verbal or are you starting from something concrete and working from there? Have your platform provide the basic parameters that people can adjust.
Far too often, the first conversation a data producer has with their consumer is once something breaks for the consumer. These silent or stealth data consumers create expectations without ever telling the producer. That causes many, many issues.
Schema should be immutable - the only way to change your schema is by creating a new version.

Ananth started by sharing a bit about his background. Despite writing the Data Engineering Weekly newsletter, he sees his experience as somewhat between a data engineer and a data analyst. That gave him the ability to see the full end-to-end journey of how data was handled at many different organizations. He consistently saw that analytical data outside of the application scope was an afterthought because developers were focused singularly on their application, not how it fit into the greater scheme, especially on the analytics side.

For Ananth, the data marketplace is a useful concept for many organizations when thinking about data contracts. It might be a bit more of a data bazaar than like Amazon in certain ways as there can be a bit of collaborative negotiation - 'oh, you have XYZ to offer, what about ABC, could you do that?' We need standardized ways to discuss/document data to make it far easier to share data, or at least start the conversation off from an informed standpoint when collaborating to get the most useful data created and shared. We need programmatic ways for producers to share what data they have available including their expectations like SLAs and consumers to request data they want with their expectations.

Scott note: It's crucial to understand that data contracts are less about the actual contractual terms and more about the establishment of a relationship that is covered through the contract terms. There are expectations but the contract isn't the entire relationship between the data producer and the data consumer. Essentially, the relationship includes the contract but just having SLAs will not resolve many of the issues people have around data contracts/sharing.

Similar to something Chris Riccomini mentioned in episode #51, Schemata is looking to provide feedback to producers about what broke downstream when they made a change. Or more valuably what will break before a commit is deployed. Data producers haven't had much of this feedback historically - e.g. "if you make this change, it will break your data contract expectations on the schema front because of…". But, Schemata is also designed for producers to see how well what they are offering fits with what other domains are offering when thinking about how well does my domain or potential new data product integrate into the overall organizational data sharing landscape.

On consumer-driven testing in data contracts/agreements, Ananth thinks there are two aspects: structural and behavioral. Structural is what you'd expect and what most people discuss in data contracts - mainly schema validation, is it backward compatible, is it strongly typed, is the required metadata complete, is there a registered owner, are the SLAs defined and complete, etc. The behavioral is similar to what Abe Gong talked about in episode 65 about what are the expectations, does the data behave the way people expect such that it can actually be leveraged for their use case. A key, widespread reason we need consumer-driven testing is producers rarely really understand how data consumers will use their data or are using their data already. Thus, that behavioral testing can inform the producer - along with actual human to human conversations - about how consumers will be/are leveraging data.

One general issue many teams have according to Ananth is the consumer doesn't really understand the cost or complexity of doing something around data creation. E.g. the producer of one domain might not store the user ID so to get every user ID is an expensive database call. So a consumer creating a pull request instead of a demand/request for data means you can start from a deeper conversation about what the data will be used for and why it's structured like it is proposed in the PR. This is also much more in a developer in the domain's workflow of using git. It's all just far less vague even if the initial proposal is infeasible - a producer has far more information about how the data might be used to start to iterate towards a workable solution.

According to Ananth, many people looking at Schemata have seen the need for years but there hasn't been a great way to implement what Scott calls "making the implicit explicit" around data sharing/data contracts. And this isn't a typical problem at a small company but once you get to a certain scale, the need for decentralized data modeling starts to become very evident. But with decentralized data modeling, it's pretty easy to put yourself in a bad spot because there is no collaboration layer so you create data silos / things that just don't interoperate well. Much like thinking federated governance versus decentralized governance in data mesh.

Schemata has a concept of a core domain that then every incremental entity or event you model, it will automatically assess how well the new event or entity is connected to the core domain. The theory is to quickly figure out how well what you are building will connect into the greater whole of the organization through the core domain. It gives you quick feedback on what is in process and you can easily add more fields to better match the core domain if a producer wants. It isn't a blocker, it's giving feedback to someone creating a pull request - data producer or consumer - about how well the resulting data model would fit in the organizational data landscape.

Ananth discussed how data creation is really a human-in-the-loop challenge - autonomous data creation is just not very valuable now and might never be. We need a collaborative platform to create data that is truly valuable and understandable but especially usable. The crucial aspect is to make a tool that integrates into people's workflows instead of yet another screen that further fractures the data management experience. Schemata is trying to be like Snyk - automatically scanning and giving people actionable advice but with little effort on their part. Where are your likely pain points? How could you address them? You can more easily set a goal of remediation/improvement and figure out how well you are doing. What are the top 2-3 things you could focus on to make the data you share that much better/more valuable?

A big thing many overlook in creating data contracts is about defining the value and/or cost of something happening according to Ananth. It's about getting people to the table to discuss something concrete and make sure people are on the same page. Instead of requirements, it's a collaborative discussion. Alla Hale in her episode #122 talked about every conversation, you should have something to show the other party, whether a full prototype or a post-it note with a little drawing. So getting to clear contract/agreement is far easier if you have a system that defines an owner, defines the parameters you need, makes sure the implicit aspects are explicit so both parties can fully agree, etc.

One thing Ananth - and Scott - keep running across are stealth data consumers creating one-sided data contracts. Essentially, the consumer has created their consumer-side testing and is consuming but the data producer has no idea they are consuming their data. Or many don't even really do the testing/contract model to protect themselves at all. The first the producer hears about their consumption is when something breaks for the consumer. With Schemata, at least there is a contract in place and stealth data consumers just have to inherit existing contractual bounds. Scott note: I hate stealth anything in data, let the producer know or they will potentially make breaking changes that could be prevented if they were just aware.

According to Ananth, we can really learn a LOT from the DevOps movement that has become more the platform engineering movement on the microservices side. If we try to push ownership to domains/data producers without the tooling to help them verify they comply with governance and that things are working okay, that's a lot of extra work on the data producer end. It's why we are seeing so damn much pushback from domains about not wanting to own their data - it's just way too much of an ask. Data producers just don't have enough information about what might be an issue when they try to make a change and it causes unnecessary friction. We need to make both the producer and consumer more productive, so that people can develop and deploy without tons of manual intervention.

Far too many teams are using tooling to solve single problems and while that one-off tool helps address a singular issue, it creates an even more disjointed data management workflow in Ananth's view. It's easy to focus too much on the spot challenge instead of the overall challenges in data management, the holistic process. Tooling fragmented with cloud and it made sense as we figured out new approaches and patterns - and VCs were quite free with their money - but we need to think about the whole process as one again now. Zhamak has mentioned this multiple times as well.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#197 Explorers Needed, Experts Not (Yet) - Mesh Musings 44 Feb 20, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

All about why we need more explorers in our data mesh implementations and that it's too early for experts :)

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#196 Data is a Team Sport - Learning to Collaborate Through Data - Interview w/ Andrew Pease Feb 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Andrew's LinkedIn: https://www.linkedin.com/in/andrewpease123/

In this episode, Scott interviewed Andrew Pease, Field CTO of North Europe at Salesforce. To be clear, he was only representing his own views on the episode.

Some key takeaways/thoughts from Andrew's point of view (mostly written by him):

Sensitizing people to data and improving their data fluency can be a challenge. Lots of people have had some less than perfect past experiences - perhaps a dry, abstract class has given them "statistics trauma". It's important to make it digestible for them to get started.
Organizations typically evolve into silos so IT systems/approaches often evolve into silos too - Conway's Law. The bigger those organizations and silos are, the harder they are to bridge / the deeper the divides.
Much as we'd like one, there is not a single silver bullet architecture for all organizations to overcome these silos.
Without relevant IT architectures and processes, it can be challenging to put relevant and timely data and actionable insights into the business people's workflows. You won't get it "perfect" the first time, but get started and learn to improve through experience.
You should reiterate to people that data is there to augment their role, not to replace it. It's there to help them be more efficient and successful in their work. That's a key part of data fluency, not just understanding how to use data but where data can help.
Feedback loops are very important to increase data quality levels and data value. It's important to build in these loops to make end-users feel like they are a part of a constant and never-ending improvement exercise. It shouldn't be a big burden but data quality is a team sport.
It's important for data consumers to understand not only the potential of data and analysis, but also the limitations. E.g. you can't reliably score lead quality from simply a person's name and their email address. The data needs to be representative enough to find useful patterns.
AI should be perceived in the enterprise as augmented intelligence - it is there to make the human in the loop better, not replace them.
It is crucial to inform operational teams, the data producers, about what data might be needed in the future, not just now. And then incentivize those data producers to actually create and maintain quality data products. If all we do is ask, it likely gets lost in the operational "priorities".
Anecdotal feedback on what data is being used and is useful is great. But it's not going to tell the actual full story. Make sure to create ways to track usage and measure impact of data work.
Data hackathons can be great ways to set up some cross domain collaboration and improve data fluency but also knowledge of other domains and the organization as a whole.
It's vital to figure out how to get people excited about data, in combination with incentivizing them to do so in appropriate contexts.
IT and the business side need to meet and collaborate in order to make data a crucial and embedded aspect of everyone's roles.
As always, communication is crucial, especially around reorganization of data teams and competencies. A clumsy reorg will certainly alienate - and possibly infuriate - people.
"The most complex system that we have in our organizations isn't a computer, it's the people who are operating the computers." When we think about the composable enterprise, we need to think about the humans in the loops and how and where they interface.
Look to have a standardized way to bring people to better data fluency. Many different roles have budgets for ongoing training in their field, everyone should have that for data and it should be part of any organization's new employee ramp period too.

Andrew started off by discussing the general way that organizations evolve. It's pretty natural for most to evolve into silos and the larger the organization, the deeper the divides between the silos and the harder it is to bridge those divides. With Conway's Law, IT systems/approaches also then often develop into silos. There is a lot of required intentionality to prevent evolving into silos or lessen divides that have already formed. And there is no "silver bullet architecture" to overcome the challenges silos create or undo the silos.

One of the big dreams of being data driven is putting timely and actionable data - "what do you want to tell them?" - in the workflows of business people. But, according to Andrew, many organizations attempting to do that look at it as an all-or-nothing kind of goal and that's just not reasonable. You won't get it "perfect" at the start. And that's okay, it doesn't make it not worth doing. As part of that process, it can be very important to reiterate that data is there to help not replace people - AI should mean augmented intelligence, it's there to help the human in the loop be better.

There are two major opposing forces re data quality in Andrew's view. First, you never get a second chance to make a first impression so your data quality has to be up to a certain level before showing to potential consumers. But conversely, the only way to get to actual quality data - essentially what matters, why it matters, and what quality levels are acceptable - is to get data in front of consumers and then iterate towards the required quality. Feedback loops are crucial to actual data quality so you can optimize for what matters. Your data consumers must understand that data quality is a team sport so they need to participate too.

Andrew brought up his concept of "statistics trauma" when discussing improving people's data fluency - essentially, many have a bitter taste from past statistics/math and/or data related work/school. So to get execs more data driven, you need to sensitize them to data but in a careful approach. That falls to the CDO and it can be challenging but is quite rewarding when it works. It's as much about communication as anything else in data.

In data, Andrew believes there needs to be far more bi-directional conversations. Data consumers need to tell data producers what they need and that can include data that doesn't exist yet so the producers need to start capturing it. So the earlier a data consumer can tell a data producer about their needs, the more likely they will get what they want down the line. Data mesh helps there because it's not the central team trying to understand and take requests to the producers. By cutting out the data team in the middle, you have a better chance to get to what data consumers want more quickly. But we can't lose sight of something that many seem to overlook - we can't just inform data producers of what we want them to produce and maintain, we need to properly incentivize and enable them to do so.

In Andrew's view, there is obviously value in collecting feedback on what data is viewed as valuable. But it's going to have bias - essentially it's valued but might not be valuable - so you should develop more concrete ways to measure what data work is useful and valuable. We should track what is being used but also how well what we thought would be valuable actually performed - that way we might better know what additional data might drive incremental value. Your feedback loops should include both quantitative and qualitative measurement where possible.

If you make ultimatums around data usage - you're either with us as a data user or you're against us - you won't get buy-in per Andrew. Mandates just don't get the buy-in some people believe. So you need to work to figure out why someone is not leveraging data. Again, make it less intimidating and make it rewarding. If you threaten people - do this or we'll fire you - you will simply get people adhering to the letter instead of the spirit - we want to make using data useful AND fun. Gamifying learning about data and data hackathons are two great ways to accomplish that.

Around data, if we want a "yin yang synergy" between business and IT, both parties have to meet the other _more_ than halfway in Andrew's experience. Both sides have to be willing to partner to improve. There isn't a silver bullet way to accomplish it but embedded IT in the business and vice versa can certainly help. You could rotate people across different business units. Etc. However, it's very important if you are in a decentralized organization to make sure you share best practices.

Andrew said, "the most complex system that we have in our organizations isn't a computer, it's the people who are operating the computers." There is a major change in the way our brains work between learning something and trying to get a point across. Some people are good at switching between those quickly - e.g. in a meeting - but many aren't and it's important to not leave them behind. So communication is crucial to get right and think about the broad group you are trying to work with. Sometimes data should be brought in to the discussion to make a point but sometimes it should purely be about increasing data fluency.

It's easy to try to focus on hiring for data skills in many roles but really, every organization should at least consider data training as part of a new employee training according to Andrew. Obviously don't forget existing employees but immersing people in data, especially the data of the organization, from the start pays off in the long run.

Quick tidbit:

Data consumers need to understand what is actually possible with data. E.g. lead scoring based on a person's name and email address is not a reasonable request.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of February 19, 2023 Feb 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#195 Zhamak's Corner 18 - Fixing Unnecessary Complications in Serving Data to AI/ML Feb 14, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

So, continuing the conversation about AI and ML's place in data mesh, we start the episode with Zhamak discussing an unnecessary complication we've created in data - why do data sets/assets only have to serve one user or even user persona? Yes, product thinking is about creating reuse but are we thinking reuse across regular analytics and ML/AI at the same time? We need to make it easy to give access in the language of, that native mode of access of, the data consumer. We shouldn't have to care what it is used for, regular analytics, ML, or anything in between.

There's also this very painful bifurcation between upstream data production and data science where the second data enters the data science realm of influence, it's copied over and you lose sight of it for discoverability, governance, security, quality, etc. They pull it in and then it's essentially impossible to track. That creates all kinds of problems. So why don't we extend data mesh into what they are doing? Do they need to make copies of the data in the feature store? If they have a trusted source of access to the data, do they care?

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#194 The One Where Scott Goes off About Data Contracts (Part 1) - Mesh Musings 43 Feb 13, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#193 The Hidden, Pesky Persistent Challenges in Data-Intensive Applications/Service/ML - Interview w/ Ebru Cucen Feb 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ebru's Twitter: @ebrucucen / https://twitter.com/ebrucucen

Ebru's LinkedIn: https://www.linkedin.com/in/ebrucucen/

In this episode, Scott interviewed Ebru Cucen, Lead Consultant at Open Credo. To be clear, Ebru was only representing her own views on the episode.

Some key takeaways/thoughts from Ebru's point of view:

It's far too hard for data producers to actually reliably produce clean, trustworthy, and well-documented data. We need to give them a better ability to do that, whether that is tooling or ways of working remains to be seen. Scott note: It's no wonder it's been hard for many teams to get their domains to own their own data ;)
There is a hidden challenge in data-intensive service/application development. The version of the data - the schema, the API, and the data itself version - need to be understood and coordinated as the developers don't control their own data sources unlike software development of the past. But we don't have good ways of doing that right now on the process or tooling front - data product approaches help but fall short.
We are lacking the tooling to easily manage data quality for producers. While there are so many data related tools, there is a real lack of things that make it easy to manage the quality. We are getting there on observing or monitoring quality, but not managing and maintaining quality.
Fitness functions can help you measure if you are doing well on your data quality/reliability.
As the speed to reliably ship changes on the application side increased - microservices and DevOps -, that just made the data warehouse, the data monolith that much harder to deal with. Instead of slow-changing inputs and gentle evolution, it simply became more and more of a data exhaust model that breaks the warehouse.
Large data monoliths are just far too hard to maintain, especially as the speed of change of application and the world increases.
However, monoliths aren't 'the enemy', even microservices advocates say sometimes a monolith is the right call. Look to figure out what's the right solution for the now and future. Don't distribute, don't decentralize without a specific reason.
?Controversial?: Data really needs much better version control systems and practices. Yes, there is the versioning of the data product but the actual versioning of the data - that immutability factor, when did this data change and what was it before that - is the most important versioning for data/analytics.
Versioning means safety - safety for consumers but especially for producers to be able to roll back. We need those better safety features so we can test much more thoroughly in data but right now, we don't have great ways to do that.
It's hard to fight Conway's Law. If we don't fix our ways of working together, it is extremely difficult for consumers and producers to align well enough to get the most value from our data. Communication issues will be reflected in the data as well.
The tools we have for data are so specialized, you might need to use 5+ tools just to properly manage a simple ingestion process - it's just not there to support the producers well enough.
How can we observe and validate data before writing very specific testing - testing shouldn't be the only line of defense. We need a way to define and create our quality gateways much more easily.
With fast feedback cycles and close collaboration around data, especially with data science, it makes everyone so much more productive. E.g. people aren't building on deprecated data sources and you can get to initially testing a hypothesis in a day or days instead of weeks.
It's important to think of your data like a garden instead of a single project - you must tend to it and improve it further. Your garden is never "done" and weeds can creep in quite easily. Get that green thumb.
To build to good models in data science, you have to ask what questions can we ask of the data, can we get enough data, is it high enough quality, etc. You need to answer will we be able to achieve a likely positive outcome and then iterate towards good - and then make it better - instead of making things all or nothing and making a static model.
Not all questions can be asked of the data you have and you need to measure how well what data you have can answer the questions you want to ask. Be realistic about what you are trying to do and what you actually can do based on what you have.
How do we create psychologically safe environments where people can fail safely and learn from that - we need iterative communications, interactions, learning, and development.
Inject more empathy into your teams and communications. We need a better way of understanding the challenges, what are we achieving together, instead of what is each person's role. The sum of the parts is the purpose.
?Controversial?: As we increase the amount of data we have and the number of people attempting to leverage that data - and let's not forget the increasing complexity of the world at large - we are likely to see it get harder to communicate relative to data. We have to try harder than ever to get it right, or at least to an acceptable end outcome.
?Controversial?: Similarly, our understanding of certain questions or sets of data will change more frequently than historically and communicating that - and why our understanding has evolved - is going to get more complex as well.

Ebru started by sharing her background where she was a software engineer and trainer including training people on SQL before moving into data/data science. As a software engineer, it was crucial to at least model and understand data well enough to ingest and store it for the application side. The big challenges for software engineers really came in integrating that data into the monolithic data warehouse and then keeping it well integrated as the application evolved. The monoliths were bottlenecks on the software side and the integration into the data monolith was just becoming too much of a major bottleneck for all. As the DevOps and microservices movements picked up steam, the speed to reliably changing the application significantly increased. That increased speed created more and more challenges in integrating into a monolith - the applications drifted far too quickly to easily work with the data monolith. But monoliths aren't necessarily the wrong choice for all, just at scale and especially scale of complexity, they become a massive bottleneck.

When talking about versioning, Ebru talked about the many copies of data challenge - which one is the right one to use and can I trust it? There are people doing incredibly important work on data where they can't reliably trace it to source and know they are working on the right version. And with no clear ownership of data, nothing ever gets cleaned up so finding a reliable, repeatable source of data is very hard. So people copy the data they do find to their work area lest the source goes away, creating more copies. So we've figured out how to do versioning relatively well on the software/microservices side with APIs but we haven't figured it out for data - whether that is versioning the analytical API or the data itself. It's far too hard to make our data assets maintainable right now, thus the big push to data mesh.

For Ebru, when asked specifically what is the most important aspect of versioning in data - code, schema, API, or the data itself - she chose the data itself. This is a somewhat controversial choice but her reasoning was traceability - what actually happened to the data and when did it change? She expects that the code versioning, we'll have more version control systems and many people already manage their data related code work in git or other systems.

Another point Ebru made was that software development hasn't really had a focus on aligning itself to what version of data it is using. When you do a production deployment, the database is the database, it's tied to the application. But when we start to think about how we actually deploy software going forward, if it is referencing external data as part of that, the version of the data source it's leveraging obviously matters far more and we need far more coordination to ensure the software is referencing what we need it to. There is not enough tooling out there to easily manage this coordination and it's causing far too many issues.

Scott note: this is a really incremental thought here but VERY hard to explain. Historically, most services have been more or less wholly contained in what data they use or they access information from other services via a versioned API on the operational plane. So the coordination is less challenging. We have not really figured out well how to do that for data intensive applications - this is partly why everyone is building data products, whether data mesh or not, but it's still challenging if you don't think about providing a steady access mechanism and a way for a consumer to know what they are accessing hasn't suddenly changed without their knowledge. See the episode on my rant on data contracts and how it's not just schema and constraints.

We just can't escape Conway's Law according to Ebru. While many people have applied it to the operational plane, we really need to think about how Conway's Law applies to data. The way we exchange information can't only be the data itself, we need to get better at how we actually communicate and collaborate internally or gaps in how we communicate will be reflected in the data and our data integrations. Without fixing the way we work together and communicate, the producers and consumers will not collaborate well enough to leverage our data to the fullest extent.

Ebru believes that right now, it's still far too hard for producers to reliably publish clean, trustable, and understandable data. We haven't developed great ways of working and the tools are definitely not there yet. So if we try to push ownership on them too quickly, it will not go well. They have historically published what they want and we need to make it far easier to publish what consumers specifically want or they won't likely want to participate.

Data mesh is a sociotechnical approach but for Ebru, there is a lot of talk about the social and the technical is still lacking. There are so many tools but they don't work together that well natively and most only do a few very specific things - you could need 5+ tools to accomplish just the ingestion part of a use case. There is also a major challenge on the testing side - can you observe what changes would occur before writing the tests?

In general, we need to change our ways of working in data to enable much faster feedback cycles in Ebru's view. She was working on a project where everyone was in close collaboration and you could try things out and get feedback in the same day, meaning there was far less time spent building toward a solution only to find out the data wasn't available or there were other challenges. With better data ownership, we can go from idea to ingestion to testing in a short period of time, significantly improving how productive data science team members are.

Scott note: if you listen to early data mesh presentations from Zhamak, she talks more about data science/machine learning/AI than regular old analytics. This is that data bazaar/data marketplace kind of concept in action.

Ebru believes we need to take more learnings from microservices, especially the concept of Lego pieces. In data, we haven't really built incrementally to really achieve good value - it's often been all or nothing. But cloud means we have a chance to do things differently. That iteration means we can fail faster too - if we have an idea but we can't get the right data or even get enough of the right data, instead of building for weeks, we can change course. It's important to realize you can't ask any question to any data as well - sometimes you have a question that just can't be answered with the data you have or can get and that's okay.

To do data well/better, Ebru believes we need to create psychological safety and an ability to fail safely. That means we will have to train data consumers far better on how we work with data - a 95% confidence interval doesn't mean what most believe. And our understanding of data evolves too so consumers must learn to evolve their understanding. Human interaction is far more crucial than many want to believe in doing data well.

In data, as Zhamak has mentioned this trend towards super fractional roles, Ebru believes there is far too much focus in many organizations on what specifically is "my role" instead of what is the team's role and how can we make sure we accomplish our objectives. This fractional thinking of course creates more friction and challenges and handoffs - handoffs are always a place of lost context. So work to have teams focused on accomplishing team goals instead of individual ones.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of February 12, 2023 Feb 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#192 Diagnosing the Analytics Gap - All About Diagnostic Analytics - Interview w/ João Sousa Feb 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

João's LinkedIn: https://www.linkedin.com/in/joaoantoniosousa/

João's Medium: https://joao-antonio-sousa.medium.com/

Brent Dykes' LinkedIn: https://www.linkedin.com/in/brentdykes/

In this episode, Scott interviewed João Sousa, Director of Growth at Kausa.ai. To be clear, he was only representing his own views on the episode.

The "four types" will often be throughout this summary. The four types refers to the types of analytics: descriptive - what is happening; diagnostic - why is it happening; predictive - what might happen in the future; and prescriptive - what actions should we take.

Some key takeaways/thoughts from João's point of view:

Of the four types of analytics, diagnostic analytics is VERY underserved. The other three - descriptive, predictive, and prescriptive - are where most organizations are focusing more so there's a "diagnostic analytics gap."
?Controversial?: Of the four, diagnostic analytics requires the most domain/business expertise.
Tips for improving your diagnostic analytics: 1) show the value of drilling down in to the why - find a few use cases and communicate the value well; 2) promote a closer collaboration between data and business people; 3) improve your definitions around data roles; 4) very clear communication of expectations and who does what; 5) don't get into firefighting mode, have a structured approach to diagnostic analytics; and 6) automate the repetitive parts.
There are 3 levels of diagnostic analytics immaturity: getting "stuck in the what" instead of the why; "the usual suspects" where you look at things from the same angle, same slice and dice and only leverage a small portion of the data available; and "need for speed" where you have the right data culture of drilling down to the why bet often get stuck in the trade-off between speed and analysis thoroughness.
Signals you are "stuck in the what": 1) in reviews, teams are just reviewing what happened instead of data-driven recommendations or data-driven hypotheses. 2) You lack impactful stories when asked "what real data-driven insights have you shared recently that drove action?"
Signals you are stuck in "the usual suspects": 1) few to no real new major insights or hypotheses. 2) Lack of incremental data work / no new slices and dices to further analyze data. 3) Indirectly, data teams become more disconnected from the business.
Signals you are stuck in "need for speed": 1) continuously cutting corners on thorough analysis in the name of speed. 2) Juggling too many tasks / priorities with constant 'hair on fire' type requests.
Culture and people are the biggest levers in data but the hardest to change. We need better processes and tooling to enable accelerating diagnostic analytics. Tooling and processes specifically for diagnostic analytics are few and far between.
Many companies do not put much value and/or effort in to diagnostic analytics - that is highly correlated to analytics maturity.
Diagnostic analytics work can be seen as boring compared to predictive and prescriptive work. It's typically not as technically challenging and many data people are not as interested in the business aspects.
?Controversial?: The best teams segment their diagnostic analytical questions into strategic, tactical, and operational. High performing teams also adjust thoroughness versus speed to best suit the specific need. They also automate as much as possible to reduce burden on the human in the loop.
!Very Important!: Decentralization presents a big potential risk to diagnostic analytics. Analytics within the domain seems covered but many questions are cross domain...
Scott note: There isn't a clear owner of diagnostic analytics in data mesh - if domains know their own data well, they should be able to do diagnostic analytics on information internal to the domain but it will be far harder cross domain. And that is end-state, not mid mesh journey. Diagnostic analytics likely falls to where you have your business analysts, whether that is embedded only, centralized only, or a mix.
Insight definition criteria (from Brent Dykes): 1) provides a shift in understanding; 2) is something unexpected where the organization was not previously aware; and 3) it's relevant and/or aligns to what stakeholders care about. João added a 4) it must be delivered on time and communicated effectively.

João started the conversation discussing the four types of analytics: descriptive - what is happening; diagnostic - why is it happening; predictive - what might happen in the future; and prescriptive - what actions should we take. Most of analytics work over the last 30 years has been the descriptive and both descriptive and diagnostic are typically owned by the analytics team. Data science, ML, and AI have moved the needle for doing predictive and prescriptive analytics the last few years. But diagnostic analytics remains underserved.

That diagnostic analytics gap exists for a number of reasons in João's view. On the people side, diagnostic analytics requires two sets of skills/knowledge: the analytical + technical and the business + domain. Without the domain knowledge, it is far harder to connect the dots around the why - a key concept in data mesh in shifting data ownership left*. Yes, we know sales in this region are falling, but why, what changed? João believes diagnostic analytics requires the most domain knowledge of any of the four types.

* Scott note: I always think of the Pastafarian - or Church of the Flying Spaghetti Monster - figure as to why they dress like pirates and how the number of pirates is strongly inversely correlated to global temperatures. See here. Correlation doesn't mean causation, relevant XKCD here

On the tools and processes side, João believes diagnostic analytics is far less developed than any of the other four types. Dashboards are great for descriptive analytics - what is happening - including some exploration but they are difficult to use to actually understand the why, drilling down to the root cause. Culture around diagnostic analytics is another large issue for many organizations - there are many varied approaches and lots of differing views on the actual value of doing deep diagnostic analytics.

João has three different diagnostic analytics immaturity stages before getting to a well-functioning approach. The least mature is "stuck in the what," where the business stakeholders are the ones trying to do diagnostic analytics with low data fluency* to drive to the why. They are only reporting the what, the descriptive analytics. The second maturity level is "the usual suspects" - essentially, the team builds lots of slice and dice dashboards and then just monitors things using that, they don't think to keep adjusting their angles and dig in. The third maturity level is "need for speed." The domains have the capabilities - usually via embedded analysts - to analyze their own data but are almost always siding with speed versus comprehensiveness of analysis. The world and business are changing fast but it takes time to do good analytics well to generate an actual insight.

Scott note: this brings up the question of where diagnostic analytics lives in a data mesh implementation. If domains have high data fluency, then presumably they can do their internal analysis but what happens if the information to drive to the why is cross domain? This is why I believe many domains are likely to have their own business analysts but organizations will still have a centralized business analyst team too.

On the question of who does the diagnostic analytics in most organizations - an empowered and highly data literate domain or a centralized analytics team - João said it depends. In a low data maturity organization, it's typically the analytics team - hopefully pairing closely with the business. In a higher data maturity team, it's about upskilling the subject matter experts in data and providing the right tools so they can do the analysis themselves.

João shared two signals you might be "stuck in the what", that you need more diagnostic analytics maturity. The first is in your weekly or monthly review meetings, you are talking about what is happening and there are only some high-level guesses as to why - "oh, that's _probably_ because we changed the website" - and not much more. Nothing is data-driven answers or even hypotheses. The second is reflecting that you haven't taken any real data-driven actions with a large impact recently. If you aren't driving your actions from your data, it's likely you aren't answering the "why" questions.

It's a lot harder to detect if you are in "the usual suspects" phase of maturity per João. The data teams aren't getting lots of additional requests. The business people are generally happy because they have dashboards that show a lot of information sliced and diced in how they typically look at things. But they are only testing existing hypotheses and not really coming up with fresh/new insights. So two signals are that there aren't really any new insights or hypotheses and there aren't many requests from the domain to the data team. The third signal, one that's indirect, is that because there is that lack of incremental data work and requests, the data teams start to become more disconnected from the business.

Teams that are stuck more in the "need for speed", João said while it's a better place to be, it's still frustrating. Teams are always trying to balance thoroughness of analysis versus speed. So some signals you are there is the pressure to cut corners on thoroughness of analysis in the name of speed - that actually happening is another signal - and constant high-priority interruptions for diagnostic analysis, juggling too much and putting aside the long-term work to take care of the fast turnaround requests.

When teams break past the immaturity stages for diagnostic analytics, João pointed to a few things high performing teams do well. The first is to segment questions/requests into tactical, strategic, and operational. Strategic questions are typically more big picture so they change less frequently and thus are typically less urgent than operational or tactical requests. Strong teams also adjust their thoroughness versus speed depending on what the situation calls for. Lastly, they automate as much as possible - there is still a human in the loop but repetitive tasks aren't value-add tasks for someone to do.

João shared Brent Dykes' definition of an insight, which is probably much more strict than many use. First, it must provide a shift in understanding - so not "we found this anomaly", it changes what people know. Second, it must be unexpected - so those teams stuck in "the usual suspects" won't meet this because they are only testing against the expected. And third, it must actually matter, it must be relevant and/or aligned to what stakeholders care about. João added his own criteria of it must be on time and communicated effectively. These are all necessary to actually drive the right action.

João wrapped with a few tips for improving your diagnostic analytics: First, show the value of drilling down in to the why - find a few easy initial use cases to really show the value, not your most difficult questions that will take months to really answer. Second, have the data and business people collaborate more closely so the data people can better understand requests and business people can start to think about new analytical approaches. Third, really get clear around your data role definitions: who does what and why and what _aren't_ they supposed to do. Fourth, start to get very clear on expectations, improve that communication so everyone is on the same page. Fifth, plan ahead and don't get stuck in firefighting mode grasping for straws - it's too easy to approach diagnostic analytics in an unstructured, reactive manner. Finally sixth, look to automate away the repetitive parts as much as possible.

Quick tidbit:

Beware the 'boring' label for diagnostic analytics. Many data people want to focus on the more technically challenging predictive or prescriptive analytics. Show people diagnostic analytics is valued.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#191 Zhamak's Corner 17 - AI/ML's Place in Data Mesh Feb 07, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Humans by our very nature categorize things - otherwise how can we really differentiate? How can we learn about new ideas and experiences if not finding a way to store them in our mental models. And in data, we've been treating diagnostic and descriptive analytics as an entirely different category to the predictive analytics of AI and ML. The way we partition the world in data is around how data will be used and then prepare the data as such, to be very fit for purpose. What if instead we partition around the data domain and don't really care about who or how things are used - we want to serve all consumers - what changes? Can we create data that is simply usable by many? Does that actually reduce complexity overall by not owning data production designed to specific purposes? Do we really need to treat AI/ML as if their consumption is all that different or special?

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#190 Data User Experience (DUX): An Introductory Panel - Led by Karen Passmore with Alice Parker and Wannes Rosiers Feb 05, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Karen Passmore (CEO at Predictive UX) led this discussion with Wannes Rosiers (Product Manager at Raito) and Alice Parker (Data Engineer at DNB). This panel was held in partnership with Data Mesh Learning - you can see a link to the video here: Panel: Data User Experience - An Introduction (Data Mesh Learning and Data Mesh Radio)

Alice's LinkedIn: https://www.linkedin.com/in/aliceparker/

Wannes' LinkedIn: https://www.linkedin.com/in/wannes-rosiers/

Blog post 'The Importance of UI/UX - and why Raito’s first hire was a designer': https://www.raito.io/post/the-importance-of-ui-ux-and-why-raitos-first-hire-was-a-designer

Raito blog: https://www.raito.io/blog

Karen's LinkedIn: https://www.linkedin.com/in/karenpassmore/

Predictive UX: https://www.predictiveux.com/

Some key takeaways from panelist Wannes Rosiers:

DUX handles user experience using data for all your users (data producers, data engineers, data analysts, data scientists, report users, ...), hence it is not - and certainly not restricted to - data platform user experience.
Data products are typically chained from producer oriented data products up until consumer oriented data products. Downstream product and UX requirements are completely different than Upstream, yet you should be considering all of them. This friction between global UX and local UX is the biggest challenge to scale Data User Experience.
Much more than typical digital products, you have no idea how people interact with your data products. We are moving more and more to self-serve analytics, which means that while developing a data product, you don't know what will end up on the screen of your user. You can wireframe fixed dashboards, you can't for certain other data products.
As always, accessibility of your data product is a huge part of your user experience. Next to this, it is important to monitor usage to continuously evolve your product. (Not surprisingly the two key elements we work on at Raito)
Domain thinking exceeds source thinking in your product. You should abstract away the concept of data sources, users are interested in valid and bounded insights, typically domain-bounded.
And last but not least. You need top-level decisions on data product granularities. To make sure you can connect certain data products, they should be interoperable and preferably on the same granular level. Which level to pick is mainly guided by company KPIs, hence you need management support to pick these.

Scott note: I am pretty new to thinking and dealing with UX in data so I took an opportunity to write down some of my own takeaways that may or may not agree with any or all the panelists. Hopefully you'll find them useful.

The most key takeaways from Scott's view and learnings:

If data is not usable, is it useful? How many data projects fail simply because no one really made the data usable?
UX KPIs are really important because it's such an easy thing to overlook.
If you have a bunch of fragmented UXs in your data value chain, your overall UX will suffer. Think about a dish - you might prepare all the ingredients perfectly but if they were all cooked separately, will there be the right flavor harmony? Probably not, it will be disjointed flavors that don't blend well.
Sometimes we think about the target business process in user experience so we don't always have to think about directly using a platform - we can think about how things integrate into a business process, a workflow. If I don't have a UI but interacting with data is part of my day-to-day role, that's a user experience.
Data projects and data work need three things: business requirements, technical requirements, and user requirements. Many skip the user requirements at their own detriment. The best way to improve user experience is to actually talk to the users about their wants and needs 😅
Empathy is crucial to data UX - make sure empathy is baked in to what you do :) stop prioritizing your technical requirements and business requirements over your user requirements.
It's very hard to figure out the user experience path between 'I create a data product' and 'user self-service drives an action from the data product'. It can feel a bit underpants gnome-like of step one steal underpants, step two ???, step three profit. So get really in depth talking to your team about what is necessary to actually get to an actionable insight from a data product.
UX technical debt, while not insurmountable, will be a bigger hindrance than many expect because the UX is typically intrinsically tied to the underlying implementation. So to improve the UX to a great degree, it can also require improving a lot of the underlying implementation. Definitely not always the case but it's something to watch out for.

A lot of additional takeaways from Scott:

User experience is often overlooked in data because historically, most data manipulation or analysis has been performed by experts in their own domain - e.g. a business analyst is an expert in SQL so all they need is SQL or data engineers have handled ingestion and transformation so there hasn't been a great UX.
It's easy to get trapped in the idea that a "data user" is just a data consumer but if we aren't designing and creating a good user experience for producers and anyone else involved in dealing with data, we are looking for trouble.
Each user persona has different requirements and it's important to not try to design one-size-fits-all experiences because it will probably be one-size-fits-none.
User experience is about lowering the bar to working with data to make it actionable, whether for a producer or consumer or anyone else. How do we remove friction from data to action?
Data consumer user experience spans a lot of things from finding then accessing then understanding data and then often transforming into their own purposes.
Data user experience can be improved at any point in a data project's lifecycle but it's always best to start at the beginning if possible.
How can we make it easier/quicker/more reliable/etc. to get to an initial answer on is data valuable?
Part of ensuring a good data user experience is enabling the data consumers to easily communicate their expectations/requirements to data producers. It's not just the UX at point of interaction but also the UX to make the data better.
It's important to understand focusing on UX isn't about right or wrong, it's about improving the user experience over time.
The data producers and consumers are often an afterthought when designing data architecture. The UX is something at best smeared on top to try to make it a little easier to deal with when it should be a crucial aspect of the architecture itself.
When many people think about data UX, they think about someone consuming from a dashboard, not all the steps that data had to go along to get into that dashboard.
Metadata is crucial to helping people understand the data they are accessing - especially when you consider being able to trust data as part of the user experience. If you can't trust it, you won't use it!
It's important to think about UX as a lifecycle because there is a data lifecycle - information is created and then acted upon creating more information of a sort. So we can't think of it as "data is consumed" and that is the end of the line for the business when it comes to broad user experience of data.
It's crucial to think about user experience at the data product level but even more so at the informational/question level. How can data be combined from multiple sources? Does UX extend to data interoperability standards even? Yes, but that shouldn't be managed by the UX team necessarily 😅
A very big part of UX is just encouraging conversations because then UX ideas will emerge. 'Oh, you're trying to do X, that makes so much sense, let me make this change to make that less painful.' But too often users take what is given and don't pipe up - give them room and encouragement to share feedback.
Similarly, open communication seems really integral to getting to a good UX. You need people interviewing the users but users also need to have space to discuss too. UX knowledge transfer shouldn't only happen because someone asked.
Information transfer can be really hard, especially if we do it all as pure documentation. Look for ways to enable better sharing of information - lower the friction to creating simple information sharing mechanisms, whether that's sharing the data or the metadata.
A huge benefit of data mesh is reliable and scalable data sharing yes but it's also about the art of the possible - 'oh, we have this data, what about XYZ use case, can we do that?' How can we embed curiosity and an ability to explore more in our data systems while still protecting privacy/PII/etc.?
It's easy to get myopically focused on the platform UX and not the total data UX. Some of that is cultural too. You want to try to design for it but the platform team can't own developing the culture. But making it loud and clear what would drive more value might help you design a better overall data UX.
It's very hard to know just how a user will interact and experience a data product. So have some conversation with potential users and maybe give them a private tutorial so they don't go off the rails. It's hard to embed 'understanding guardrails' into data we share.
It's easy to lose sight of cost when considering user experience. How do we make it so we can enable great experiences in a cost-effective way?
It would be great to have data usability design patterns and methodologies emerge for creating usable data products but we are VERY early days there.
Usability testing is pretty crucial because while they aren't your enemy, the saying "no plan survives contact with the enemy" is still apt with data work in general. Don't design in a vacuum, work on constant communication and information flow.
It's crucial to think about UX at the micro and the macro level. If you only think about it at each data product, you are missing the information of the organization and how it all interoperates to tell stories. But it's also pretty easy to focus on the grand platform and then dealing with the data products is a bit of a nightmare. So again, it's not easy :)

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of February 5, 2023 Feb 05, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#189 Our Data is In the Cloud… Now What? - Interview w/ Vikas Kumar Feb 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Vikas' LinkedIn: https://www.linkedin.com/in/vksnov9/

Vikas' Twitter: @vikaskumar9 / https://twitter.com/vikaskumar9

Vikas' email: vikaskumar9 [at] gmail

In this episode, Scott interviewed Vikas Kumar, AVP and Head of Data, AI, and ML at CNA Insurance. To be clear, he was only representing his own views in this episode.

Some key takeaways/thoughts from Vikas' point of view:

In data mesh, make sure to keep focused on bringing the business domains along. You aren't building for the sake of building. If users can't derive value from the data work being done, why is it being done?
The 2010s through the early 2020s have been about moving data to the cloud but we are starting to see people really leverage that data to generate value. The cloud unlocks many new possibilities around data due to flexibility, scalability, and unit economics.
With moving to cloud, there is much less focus on specifically managing the data and more focus on getting value from the data. SaaS data product offerings really unlock people's time to focus on driving value.
Cloud gives us the scale and data availability but there is still a long way between having the data available and leveraging the data for significant value.
Cloud can be a double edged sword - it gives you flexibility and scalability but without good controls, you are likely to do a lot of duplicate work. Be careful that ease of data product creation - or at least PoC creation - doesn't create chaos and data product overlap. Make sure to have good governance here including strong communication.
?Controversial?: We aren't very good yet at making it easy for business domain users to leverage data in many of their decisions. Where do we fall on the spectrum between we need to teach them how to do everything data and we need to curate everything for them?
?Controversial?: It's easy to focus too much on the short-term quick wins in data. You need to think about your overall data landscape and build a foundational approach so you can go after big picture, big impact bets with your data work. You should think about building every data product from a foundational approach too to make them more extensible.
We need to get people out of their functional silos with business people only speaking business and data people only speaking data.
To do data mesh well, we have to focus on the operating model of the organization around creating and maintaining data products. There is too much focus on the technical aspects instead of how does this actually get done in a way that fits with the organization's ways of working.
Data producers must assess data consumers' data fluency levels. If they aren't very strong with data, should you really be delivering them raw data instead of curated insights?
For any data product, you should start by mapping it to a target outcome. But it shouldn't stop there because with reuse, new outcomes may emerge that drive additional value.
Data product owners are crucial to building good data products. It's their job to identify and then satisfy the objective of building the data product. What are you trying to achieve?
We shouldn't focus only on the data product - the work to create that data product is what makes it valuable, the data product is merely the vehicle for delivering the value, the output of real product work around data.
?Controversial?: Many companies doing data mesh appear to be trying to leave data governance until 'later' and that is likely to bite them. The governance meaning the security/access control but also the interoperability. You might not need to implement all of your data governance upfront but you should plan out your general governance strategy very early in a data mesh journey.
Access control is a really hard problem. Many organizations don't have good communication or visibility into who is using what data and especially how/why. We need to be asking these questions and then setting access policies that expire too - we should check in to see if people still need access, that's just good governance.

According to Vikas, 2010 through the early 2020s the focus has been on moving the data to the cloud to better drive value. And now that more and more of our data is in the cloud, we are starting to see much broader adoption of things like ML and AI. The cloud gives us the promised but under-delivered scalability of the "big data" technologies along with the flexibility to move quickly and experiment. Cloud can also mean it's easier to bring non-data people into the mix to drive better collaboration between the data people and the business people/domain. So cloud gives us this massive scale and data availability but we still have to learn to better leverage our data, drive value from it - we are still in pretty early days there as an industry.

A big outcome of the mass movement of data to the cloud is how much time is spent on data management versus getting value from the data according to Vikas. DBAs used to spend 60%+ of their time just managing the data but data people's time is now focused on getting value and probably only 10-20% is spent managing the data specifically. But cloud can be a double-edged sword too - if it's very easy to create new data products or beta data products, you have to be very careful to not create overlap/duplicate work/data products. It all comes down to governance and your operating processes to prevent that.

As an industry, we are getting much better at serving data reliably at scale according to Vikas but we still struggle with the gap between the data is available and the data is able to be used by consumers in the business domains. We are still working on figuring out where to meet in the middle between handing people reports and maybe dashboards - a kind of old school approach - versus upskilling them to very high data fluency so they can build everything themselves.

When asked that question - do the data people have to learn all the business context or vice versa - Vikas gave the very data mesh answer of "it depends." But that makes sense because there shouldn't be a single prescribed method, you have to look at how your organization works and fit with that model. And you probably want to meet somewhere around the middle. Otherwise, you will cause unnecessary friction. So look to your general ways of working, cross train people, get people exchanging context about what they are trying to achieve and instill a culture of feedback and collaboration. That's how you can actually execute well on a data mesh strategy.

Vikas talked about your data strategy north star being about getting value from your data, reliably and at scale. So, you need to be realistic about where you are in that capability journey right now. As a data producer, you need to assess can your data consumers do everything necessary if you give them raw data or should you be curating it for them so they can actually leverage the insights. Work to find the high value return data work early instead of trying to do the most complicated aspects of data. It's okay to start small, no shame there.

A data product should always map to a target business outcome according to Vikas. But that shouldn't be the only factor. The reason for creating a data product should be trying to achieve that outcome so use that as the north start for the data product but we must build in a way where data products can be reused - sometimes with some additional work - for additional use cases. And it's really crucial to have a data product owner that is discovering and focusing on the objective of the data product. How can you provide the business meaningful data that meets their objectives, that should be a key objective of every data product.

When asked how do we balance focusing on the long-term wins instead of the quick - but typically small - wins, Vikas talked about the need to create a holistic view of your data and build a very strong foundation for how you will deal with data in general. That makes it so you can jump on the quick wins when you find them but you also have a steady foundation for making much bigger bets going after long-term big wins. But with a shaky foundational layer for your data, those long-term big wins are much less likely to pay off. And that foundational aspect comes in at the data product level too - build data products that can be easily extensible when it makes sense because they are built to be extensible from the start. Kent Graziano in the recent data modeling panel railed against having to rebuild every time you extend a data product, don't do that :)

For Vikas, there are many value streams for a data product - most people focus on the data set itself but it could be the governance work or the collaboration conversations between producer and consumer. We need to focus less on the data product as the exact output instead of the data product being the vehicle for delivering value but the overall product work itself significantly enhances the value of the data product.

Data governance seems to be the part of data mesh that confuses a fair number of organizations so they ignore at their significant peril according to Vikas. While you might not have to build every aspect of your governance upfront, it's crucial to think about how you will apply governance. And to truly get to the ideal of a self-serve platform, governance needs to be a simple part of the ways of working. Saving that for later is not going to end well for many organizations. And while access control is hard, we need to get far better at understanding who is using what and _why_. How long should someone get access to data? Forever access should be a non-starter. And how do we make it easy to grant that expiring access?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#188 Controversial Opinion: You Must Register Your Use Case With the Data Producer - Mesh Musings 42 Jan 30, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

I share my strongly held belief that every data product consumer MUST register their use case with the producer and why it is so crucial and important. I sum up with these points:

Data isn't ever going to be self-describing, at least not deep, context rich data and it definitely won't be as we focus on data democratization. So, you need to register your use case to make sure you and the producer are on the same page for what is truly being shared
Data producers don't know what data consumers want - and especially don't know what potential data consumers want - without more information/conversation. Data producers kinda suck at doing that right now so help them out and improve the information flow. Who knows, they might have something way better for ya
You might actually be able to get at a heck of a lot more useful information if the producer 1) understands your use case and knows they can give you access to the PII/sensitive stuff because your use case is fine on ethical/legal/regulatory checks and/or 2) you are taking on the risk around if you violate ethical/legal/regulatory bounds.
You need to communicate what would most effectively help you achieve a target outcome and have a reasonable, collaborative negotiation between data producers and data consumers. Just consuming from a data product as-is might not meet your needs well enough.
Kinda similar to #1 but just getting a much better understanding of what you are really consuming. Does it mean what you think? Did you create an amazing insight, how can you collaborate with the producer to show that off? How can you make sure you will continue to get what you need as the data product evolves. I'll be digging somewhat deeper on the next episode with a rant on data contracts.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#187 Maximizing the Value of Your Data Through Data Products - Interview w/ Bruno Aziza Jan 29, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Bruno's LinkedIn: https://www.linkedin.com/in/brunoaziza/

Bruno's Medium: https://medium.com/@brunoaziza

Bruno's YouTube (Carcast videos): https://www.youtube.com/@brunoaziza

In this episode, Scott interviewed Bruno Aziza, Head of Data and Analytics at Google Cloud.

Some key takeaways/thoughts from Bruno's point of view:

The end goal of your data strategy should be to reliably and scalably turn data into value. The best way to do that is by creating data products. How you get there might be different but don't lose focus on turning data into value.
"The number one barrier to your ability to drive value of data is not your technology, it's your people and how you organize your team."
Focus on the point of what you are trying to deliver, not the actual output. It's not about delivering a dashboard, it's about creating a sustainable way to explore, share, and consume information/insights, whatever form that takes.
!Controversial!: There are 3 phases to getting to data driven; 1) is building a data lake or ocean, 2) is data mesh, and 3) is getting to a data product factory equivalent.
It's easy to try to put the cart before the horse in data. Before doing something like data mesh, you have to think how you will develop data as a function in your organization.
Understanding the data product manager role and leveraging data product managers well is crucial to building an effective data product strategy and practice. They are your data product CEOs.
A CDO's effectiveness depends on if they have a true seat at the exec table - can they create the necessary change - and how many people in the organization are "devoted to the data opportunity."
?Controversial?: "Really smart data leaders hire for the business department." Understanding if someone cares about data and if you can work with them is important. If that potential head of marketing hire doesn't care about data or has low data fluency, will it be possible to work together?
The companies doing the best on data literacy are making data a crucial part of their culture. Their daily practices increase data fluency across the entire organization rather than in a centralized data team. It's not about just training, it's about changing habits.
Data driven companies are "162% more likely to surpass their revenue goals."
"We're seeing people kind of rushing into migrating and not thinking about governance."
?Controversial?: There are two components that significantly increase the chances of successfully transforming to being a data driven organization: 1) a true organizational mandate to become data driven and work on data products and 2) does your organization have the attitude and aptitude to drive towards being data driven.
There isn't a clear pattern yet for the best way to find your data product managers - teach the business aspects to the data people or vice versa. But it's clear that understanding what value data products drive - not just the ins and outs of the data product itself - is crucial.
More and more, the centralized data team model is getting swamped in large organizations. But too many are too happy to fully decentralize, which also causes many issues. Federated - decentralized control and work but centralized collaboration and practices - is the approach seeing the most success.
?Controversial?: There are 3 types of data products: 1) internal domain-focused (data on the inside, not very reusable), 2) core, centrally managed, and 3) everywhere in between :)
A major issue in the organizations that focus on empowering domains without interoperability is that - surprise, surprise - there are different semantic meanings and so the data becomes very difficult to integrate/interoperate.
Focusing too much on quick wins will mean you miss out on the places where data can add a lot of value. Quick wins are typically not big wins or you find the big wins fast and then all your following quick wins are moderately sized at best.

Bruno started off with something you don't often hear a vendor say: "The number one barrier to your ability to drive value of data is not your technology, it's your people and how you organize your team." So while you can't buy your way to a data mesh, you also can't just flip a switch and be doing data mesh. You need to build your organization's capabilities to a degree they actually can derive value from their data.

It's also not easy for a data leader to necessarily create the necessary change per Bruno's conversations with data leaders. Many don't get a true seat at the executive table. And even if they do, if there aren't enough people "devoted to the data opportunity," it will be a very hard road to drive the data function to where it can add significant value. Bruno also dove into what he's seeing that makes for a high data literacy rate at customers - changing the day to day interaction, the habits of working with data. Making data part of many more people's roles and making it an intentional part of the company practices/habits builds an incredibly deep bench of data talent across the organizations. So his three components to positive change in your data approaches as a company are a strong data leader, the proportion of people committed to data work, and daily practices involving data.

While we know being data driven has an advantage - data driven companies are 162% more likely to surpass their revenue goals per a study - Bruno sees a few reasons why only 27% of companies are actually data driven now. To be data driven, you need to reliably produce data at scale, hence creating data products. And to do that, you need to build out the capabilities to handle data at scale - and not skip the governance :) But the end goal is to provide a reliable way to create value from data. That's really it. The best way to reliably do that is via data products in his view.

Bruno is seeing people go through three phases in getting to a reliable, scalable way to turn data into value. Phase 1 is the data ocean - it's not a lake, that's landlocked. The second stage is data mesh, allowing people to autonomously innovate with data but relying on central resources. And the third stage is a data factory. Scott note: the factory analogy might be rough because 1) feature factories are a very bad software pattern and 2) factories are notoriously about producing the same things at scale. And while we want scalable ways of creating data products, they should be more fit for purpose to use cases (but of course reusable as well) in my view.

The data product manager role is crucial to getting data products right according to Bruno. You need someone to be the CEO for you data product, that is focused on the actual value the data product drives and how reliable is the data product creation/maintenance. What more should be added to the data product? How is it used? To drive that cultural shift, you need a strong leader of the data organization that is empowered to make the right changes.

For Bruno, there are two factors that significantly increase the chance of an organization successfully becoming data driven. The first is an organization-wide mandate that data matters and that people must participate in the change and leverage data. Especially if the CEO is bought in on the data opportunity and the need for more and better data for themselves especially and the organization more broadly. The other is the attitude + aptitude to actually go out and build a scalable capability to build data products. And that's far easier said than done. That can be driven centrally or in a distributed way but you need people to step up and own the data.

The centralized data team model is becoming harder and harder for companies to scale according to Bruno. The team needs to be constantly ahead of the curve and they don't have the ability to learn all the necessary context so they quickly get overwhelmed by requests. This was a key factor in Zhamak creating data mesh as a concept. But the teams that are just fully decentralizing are creating data silos and making it increasingly hard to answer cross domain questions. So the organizations that are doing a federated approach with a strong sense of overall collaboration are winning - there are things that are centralized and things that are decentralized and each organization needs to figure out what works for them but balance is crucial. Find the right approach for the job.

Bruno talked about those companies that are focusing more on empowering domains than on the bigger picture of how domains can also work together. No major surprise but it creates data silos because everyone has different definitions so nothing is easy to integrate/interoperate. This is leading to the rise of the idea of the universal semantic layer.

Quick tidbits:

Data leaders should be involved in the hiring process for business people. That way, you can start to build a relationship early and help select someone who values data and has a decent data fluency. You don't want to be left out of the process.

It's absolutely okay to have domain-only data products that are very specialized to that domain - basically data on the inside in a data product. It's also - per Bruno - okay to have very centralized data products that are pretty core across the organization. But look for places to build reusable data products to get the most leverage from your data work.

To do data products and data product management right, you can't only focus on the data product launch. Maintenance and growth/evolution are crucial aspects of product thinking.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of January 29, 2023 Jan 29, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#186 Zhamak's Corner 16 - An Interpretation of Zhamak's Call-to-Action Jan 24, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

We missed our window to record so I am interpreting what Zhamak is saying in her Medium post about why did she creator her company and the general state of the tooling market around data mesh. I also added on a the full 50min recording of our second recording which I had broken up into episodes 4-7.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#185 How the Heck Do We Do Federated Computational Governance Part 1 - Mesh Musings 41 Jan 23, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

I will have much more to say on federated governance in data mesh but to reiterate my points:

Federated doesn't mean decentralize everything. Really focus on enabling fast decisions with the people who have the understanding
Shared ownership of responsibilities or oversight but not owned by a central team can work. Stop thinking that this shoves all the ownership to the domain on day 1, they aren't ready for that
Start to plan out your governance roadmap pretty concretely at the start. But don't be worried about trying to have your ideal governance setup upfront. I've talked about CYA or cover your butt - think about what makes sense and what you can build as a skeleton upfront rather than trying to build a fully functioning body
Start to think about interoperability at the concept level, that semantic level but you don't need to have your ultimate interoperability plan from the start. You can enhance and develop it as you go. And go talk to people about what they are doing!
Blueprints. Do all the blueprints. Please, find friction and find a way to make easy buttons on this. There are 10s of episodes touching on this. Go look at what AgileLab is doing especially.
Leave automated access control stuff for later. Focus on fast requesting and granting access
Create data quality standards of measurement. That way everyone can understand how much they can trust a data product and why.
Data contracts are about a producer/consumer relationship. This ain't a transaction. Get to know both sides and build to an understanding of what information is actually shared.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#184 Ontologies Don't Have to Be Scary: An Ontology Primer - Interview w/ Neda Abolhassani, PhD Jan 22, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Neda's LinkedIn: https://www.linkedin.com/in/neda-abolhassani-ph-d-61354329/

OSDU Ontology: https://github.com/Accenture/OSDU-Ontology

In this episode, Scott interviewed Neda Abolhassani PhD, R&D Manager at Accenture Labs. To be clear, she was only representing her own views in this episode.

There's some very specific language about ontology in this episode but I think it's quite approachable for most people as a good understanding of ontology, the difference with taxonomies, and some specific insight into developing and applying an ontology.

Some key takeaways/thoughts from Neda's point of view:

When starting developing an ontology, it's best to start from the business questions you want to answer. It is okay to choose bottom up or top down, but the business applicability is the main point.
You can convince people ontologies and knowledge graphs aren't scary or that hard to learn and leverage with a small demo of what they do and how to use them.
Look for open ontologies that have already been created around your domain or area you are trying to model. They can usually be easily augmented and extended but there's no reason to reinvent the wheel.
Data people need to learn enough about the domain to build the right ontologies and data models but data people learning domain knowledge can "discombobulate" them :) Get the data people with the subject matter experts to learn what's necessary.
Try to keep your ontology as generic as possible but still encapsulate what you need; that way it is much easier to apply the ontology to other domains/departments.
Set your ontology up to evolve as you learn more about your organization and as your organization itself evolves too.
You don't have to change your ontology simply because there are new use cases. If you designed it generically enough, it should be able to handle most new use cases whether as is or with a few additions.
A good way to measure if your ontology is good enough and is still meeting your needs is to look at the business questions you want to answer. Are you able to answer them with your current setup?
It's okay to have a global ontology and then ontologies that are more specific inside the domain if that is necessary to extract more value from the domain.
?Controversial?: Knowledge graphs need an ontology as well as a data model. So if you plan to leverage a knowledge graph for data mesh, you will need an ontology.
When is best to actually start to deploy a knowledge graph and develop your ontology for data mesh - if you go that route - is still somewhat up in the air. The earlier the better in general but it will mean more work as things are unsettled early in your data mesh journey. Basically, it depends.
Constant schema changes can make designing and updating your ontology more challenging. Whether that means you want to delay adding something immature to your ontology versus waiting until the schema settles a bit more, it's hard to say. Just, it can create additional challenges.
?Controversial?: To really have high-value interoperability in a data mesh, you need a way to capture not just the metadata of the data products but also the semantic meanings in the domains. And that should be done via a knowledge graph.
Ontologies are richer than taxonomies because while both have definitions, ontologies also have description logic. This gives people an ability to define how data technically fits together across different aspects.
OSDU, or open subsurface data universe, is a specific open source data platform for the oil and gas subsurface data. Neda developed an ontology to go along with that.

Neda started off with a definition of ontology: "So literally, an ontology is a formal explicit specification of a shared conceptualization. I know that it has lots of jargon, but I'm going to explain it to you. So it is an abstract model of concepts, properties, relationships, and it is standardized, it is machine readable. And it is not just the instance level data, it doesn't include the instance level data, but it includes the schema and the type level information and how stuff should be connected in your domain."

When asked if it's best to start top down or bottom up when thinking about building an ontology, Neda said either is acceptable but the main advice is to start from the business questions you want to answer. After all, this isn't an exercise for fun, there needs to be a business purpose. And look for open ontologies for your problem statement or industry. There are a number of ontologies that have already been created that you can leverage, extend, and/or use for inspiration. There is no real reason to reinvent the wheel.

When building your ontology, Neda recommends keeping it as generic as possible. That way, you can apply it to multiple domains with no conflict. But it still has to meet your needs obviously. There are ontology editors to make things easier as well but it's important to set your ontology up to evolve as your understanding of your organization evolves and as your organization itself evolves. You can even do version control of your ontologies to make collaboration far easier as multiple parties look to improve the ontology simultaneously.

For Neda, ontologies and knowledge graphs go hand-in-hand. It's okay to have an ontology for the global organization and another one for a specific domain if that's of value. Ontologies are typically about communicating externally from the domain or whatever grouping you are representing. And knowledge graphs are for integrating data from different sources or domains. And for knowledge graphs, you need an ontology and a data model for it.

Ontologies are richer than taxonomies because while both capture the definitions, ontologies also have description logic. That description logic gives you a better ability to define things like unions, intersections, restrictions, and equivalences. So ontologies are broader than just concepts and terms.

Neda then discussed the OSDU or open subsurface data universe - an open source data platform for subsurface data in the oil and gas space. She specifically saw a gap in OSDU where companies loading their own data into the OSDU format was pretty challenging. It required a lot of subject matter expert time to match schemas to the OSDU format. So Neda and team developed a technique using a knowledge graph and AI techniques to try to automatically match and map data in a company's own schema to the OSDU format. And as stated earlier, a knowledge graph needs an ontology :)

As Neda worked to build out the OSDU ontology, she looked at the OSDU canonical data format and reviewed the schemas to understand what embedded choices were made so she could ensure she added that in to the ontology. She looked at the ontologies specific to related spaces or even some that were part of the OSDU area of interest like seismic data. However, the ontologies that existed for oil and gas were mostly outdated and didn't really cover what was really useful and interesting. An aspect of OSDU that made developing the ontology easier was that the schemas were not changing very often so there wasn't a constant remapping and versioning challenge.

So, circling back to data mesh, Neda believes it's important to leverage a knowledge graph to really ensure good interoperability between domains and data products. A data catalogue - or other mechanism for discovering data in data mesh - that only has information about the individual data products and not how they interconnect won't have as much value.* When should you actually start to develop and deploy your knowledge graph is again something that requires more study and feedback. All else equal, the earlier the better, but of course as things are developing and changing rapidly early in your data mesh journey, trying to _also_ update your ontology will be a ton of extra work. Time will tell.

*Scott note: yup. Isn't that just high quality data silos? Even if they interconnect, if people can't easily understand and find the interconnections, you likely lose a LOT of the value of data mesh. Whether knowledge graph and ontologies are the best approach remains to be seen however.

Neda covered an aspect that is really important for all things data mesh: how to measure when things are good enough versus they need updating :) for her, it's about what she said at the start: what are the business questions you are trying to answer? If you are still able to answer those well enough, you probably don't need to change your ontology. But if those questions have changed considerably and your current implementation is not able to answer those questions well, your ontology will need to be updated - maybe some new concepts will be added and some old concepts deleted. You do want to be careful to try to keep things backward compatible as you deploy a new version of your ontology.

Evolving ontologies is a challenging thing but if you designed your ontology well enough at the start, you probably don't need to do it all that often according to Neda. You should design your ontology in a generic enough way so it can handle new use cases without every little new aspect needing a whole new ontology version. However, that doesn't mean your ontology should never evolve. Things change or need clarification and you should be willing to be adaptable. Scott note: this is where Zhamak sees challenges with ontologies: if they are overly centralized and overly rigid, they prevent people from expressing real meaning at the data quantum level because they are trying to fit the definitions of the data quantum into the ontology.

In wrapping up, Neda shared her views on how to really get started on building out a good ontology and knowledge graph. It will require your data people to learn enough about domains from the subject matter experts to develop the ontology. Be prepared for that to be a bit confusing as sometimes learning a lot of domain knowledge can "discombobulate" your data people. And it won't be a super quick exercise. But Neda believes it will pay out in the end and add a lot of value.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of January 22, 2023 Jan 22, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#183 Business Intelligence's Place in Data Mesh - BI-gin With the End in Mind - Interview w/ Ryan Dolley Jan 19, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ryan's LinkedIn: https://www.linkedin.com/in/ryandolley/

In this episode, Scott interviewed Ryan Dolley, an Independent Business Intelligence (BI) Consultant.

Before we jump in, lots of things in the key takeaways are marked as potentially controversial. Because much of what Ryan covered hasn't really been stated by a lot of people. So until we have more consensus, of course things _could_ be controversial.

Some key takeaways/thoughts from Ryan's point of view:

Begin with the end in mind. It's easy to lose focus on what you are trying to accomplish instead of what steps you are taking. Focus on what the target outcome is and use that as a North Star to measure if you need to course correct.
BI people need to brace themselves for a wave of innovation coming. There is so much - hopefully positive - change coming up the stack and BI people can embrace it or get washed over by the wave. Embrace and ride that wave and upskill!
?Controversial?: Data mesh - and just about every other paradigm - does not focus enough on the last mile of analytics, at least not explicitly. So we need to get a lot more specific about what is necessary to actually take advantage of upstream improvements in data to deliver better analytics.
!Important!: We need to get more specific on who does cross-domain BI in a decentralized world. Otherwise, we have interoperable data but no one specifically leveraging that interoperability for improving our understanding of the overall organization.
BI as a practice needs to be much better at understanding and implementing iterative feedback. It's not been part of the playbook to date.
?Controversial?: If domains develop BI capabilities, how will they interact with a central BI team if one exists? Historically, they haven't gotten along well...
?Controversial?: Your central BI / Data warehouse team should get converted into an insight generation engine team.
We need to figure out if BI is a primary role inside the domain or if it is a skill people inside the domain need to develop as part of their roles. Which is preferable is likely dependent on the size, complexity, and data value of the domain.
BI can't be a data mesh afterthought - we'll get to it when we get to it isn't going to cut it. It might not be literally part of the data mesh implementation but it must be part of your overall data strategy. Otherwise, why are you doing data work if not to generate and take action on insights?
!Controversial!: The semantic layer is likely to play an important role in data mesh. Scott note: depends what you mean by semantic layer. See Zhamak's Corner #12 for more on the dangers of layers.
BI teams must become part of the insight generation and resulting action conversations. That means implementing product thinking instead of simply responding to requests. But most are not ready for that mindset shift just yet.
?Controversial?: Domains should own an element of BI and not just share their data via data products but should generate and share their insights as part of data mesh.
Product oriented thinking is not widely prevalent in the BI world - e.g. most BI teams do not understand the usefulness/usage of their BI assets or even have monitoring setup to track usage.
!Controversial!: The BI analyst in the board room as merely a live question answer mechanism is a bad approach. It often leads to wrong answers with no meaningful quality control.
BI people need to adopt product thinking and create meaningful relationships with users so they can evolve what they produce to continue to suit user needs. Blindly producing the same report for 4 years into the void needs to go the way of the dodo.
!Important but Annoying!: We will probably have to meet some people where they are when it comes to sharing insights. Trying to upskill everyone to being an SQL whiz isn't realistic. Allowing people to export into Excel for further manipulation isn't going anywhere...
Part of product thinking is designing to the user. That's pretty rare in data. What are they - the data consumer - specifically trying to do and how can the producing team support that? Instead of what does the producing team want to share, which is the common current method.
Dashboards aren't dead, they are the best way to answer known questions in an oversight and management kind of way.
?Controversial?: BI is possibly the most important part of your data culture simply because for the vast majority of your organization, they will interact with data mostly at the BI level. But too many organizations leave BI as an afterthought in data culture because it will be where the most people need the most help and thus is the most difficult aspect. People change is hard :)
?Controversial?: In a successful data mesh implementation, there will be less work on the BI team because a lot more will be pushed down to the data product. The more the data product can share the insights inherently without an interpreter - the BI team - in the middle, the better.

Ryan started with a bit of inspiration and/or a call to action to fellow business intelligence people: data mesh and a lot of other movements in data are creating a massive wave of innovation further up the data stack / data and analytics process. So BI people can either get with the wave and ride the changes to greater value or get washed over by it. Ride that wave :) Upskilling can be painful but it can mean a major upgrade in capabilities and value. And being highly tech-capable in BI is a great career move right now.

According to Ryan, the last mile of analytics where BI really fits has been somewhat ignored in most modern data approaches and technical innovations. Tableau defined an era of BI and while Tableau isn't going away, that era is coming to a close. So he believes things like data mesh need to focus more on the last mile of analytics - or at least specifically call things out more often as to what changes and how in BI. Whether we call that self-serve or anything else. What does the upstream changes from data mesh mean for BI? How can we take advantage of these improvements in data processing, ownership, quality, etc. to deliver actual insights to those that matter?

Ryan asked how should we actually do corporate BI in a data mesh type setup? Right now pre data mesh, most organizations have centralized BI teams. And while domains will probably need some BI capabilities, a recurrent theme in the podcast is that most exec questions are not single domain questions. So we build this huge amount of interoperable data, who is in charge of bringing that together at the corporate level? Who actually owns creating and maintaining C-suite and board-level dashboards and reports? Are we expecting execs to build those themselves? 🤣 And historically, central BI / Data warehouse teams haven't really loved embedded BI teams, often calling them "shadow IT". Can they play nicely together? If yes, how?

There is often too much of a 'we'll cross that bridge when we come to it' relative to BI as part of most modern data strategies according to Ryan. While BI may not actually be part of the exact data mesh implementation at some organizations - there's all kinds of definitions and each org is extending the self-serve platform to encompass different aspects of analytics - BI definitely should be part of your data strategy. Otherwise, what is the point of doing data work if not to, you know, generate actionable insights? This is especially an issue when trying to embed insights and analytics in general into other applications.

An important aspect of a data mesh implementation for Ryan is what is left to the BI tool and what is pushed down into the data product*. There isn't a lot of guidance or discussion happening exactly on what is a best practice or how people are implementing embedded insights, just that it's the future. There needs to be far more conversation about who owns data throughout the data and analytics process lifecycle. Is it all on domains?

*Scott note: episode #40 w/ Xavier Gumara Rigol covers this well

For Ryan, there are a few reasons why the general data warehouse has not been as successful as it could be. To actually interact with the data warehouse, people need a decent level of technical and BI skill. So when it comes to data mesh, we don't want there to be an even higher skill level required. So domains likely need to own some of their own BI, sharing insights from the domain in a consumable way to the rest of the organization. We can't only have domains sharing data, we need to be explicit about sharing information to a purpose. Is that to get the data, the insight, or the action that should be taken based on the insight? Scott note: there will be people in your organization that are only information consumers - taking info from dashboards and reports made for them. It's up to the organization to figure out if that is acceptable or to try to upskill those people against their will.

While product oriented thinking is picking up in more areas of data, it's still not very common in the BI world according to Ryan. Most BI teams have no real idea of how useful any of what they produce is to the organization. Sometimes that is they don't have the systems set up to even monitor it but if they do, they typically don't understand how usage translates to value or honestly even look at the usage metrics. That means the BI teams lack the insights to make their BI assets as valuable as possible - what are the SLAs people need to make the most use of this data? How timely or accurate or complete does this need to be to drive the most value and be trustable? If BI people develop those relationships and that information flow, they can evolve what they produce to continue to meet needs. We need to stop producing work into the void; who is using this BI asset and why and how can we make it better and/or less costly to produce?

A really important aspect of designing usable BI is thinking about the actual user in Ryan's view. It's often overlooked too and is part of product thinking: what is the product market fit for the BI asset / data product you are developing? How do people want to interact with it? How far along the pathway from raw data to insight to "so what" - the action that should be taken - is it the producer's responsibility to own and do the actual work? Does the consumer understand and agree? How can we generate things in data to create understanding and trust with the minimal amount of effort? Dashboards are great if you're monitoring a known question but we also need to get far better at answering how do we persuade someone with data.

Ryan believes BI people need to move past the idea that the output is the end of the process. That dashboard or report or whatever is a communication mechanism about what is currently known. But it's not as if there is a single answer that stays the same. How many customers do we have is not a one-time answer question. There needs to be a lot of work to bring more iteration into BI, especially collaborating with consumers as both sides learn more about questions they are trying to answer. So BI teams need to move past being order takers and move to being part of the productization process around data and insights. Iterating in BI is not common practice but needs to be!

BI is probably the most important aspect of your data culture according to Ryan. That's simply because it's the place where the vast majority of your organization will interact with your data, at the BI level. Whether that is in generating insights or merely consuming them, BI should not be left until the end of your data strategy. But it often is left to the end specifically because it is where there is the most human interaction and change management and that's the hardest part of change management, dealing with those pesky humans. Don't try to put it off until the end as it will just create more pain than is necessary.

Ryan said "begin with the end in mind." It's quite easy in data work to get lost in the how instead of the why. The how is the fun/interesting technical challenges part. What are you trying to achieve? BI at the end of the day is not about generating insights, it's about supporting actions that help the business. So how do you build your BI strategy into your data mesh implementation strategy so you are not just making data available and trustable? How can you move past just generating insights? How do you think about the end - making smarter business decisions driven by trustable data - and work backwards? Don't lose sight of what you are trying to achieve.

Ryan wrapped up the conversation by returning to the crucial points of integrating in product thinking to BI and upskilling. We can go further than we ever have before with data and BI but we have to embrace new ways of working, both on the strategy/data culture, and the technology. But just trying to advance your BI practice with technology is going to miss a whole lot of the value.

Quick tidbits:

We have to figure out in the domain if BI is a primary role or is it part of the necessary skillsets involved but not a separate role. It will probably be domain dependent, not even organization dependent, based on the complexity, size, and data value of the domain.

Right now, most organizations want to embed insights into their general flow applications - e.g. the CRM for the sales team - but they aren't doing the back-end work to make that possible.

Putting a business analyst in the boardroom simply as a vehicle to live answer questions usually causes more issues than it solves. It leads to a high pressure environment where you can't be sure of quality. Scott note: fully agree. There's a long rant I have about this…

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#182 Zhamak's Corner 15 - A Post Pipeline World? How to Build Data Products - and a Product Creation Ecosystem - That Inherently Create Trust Jan 17, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

We wrap up another recording of Zhamak's corner talking about how do we actually start to look to build data products in a post pipeline data world. Data tools right now are kind of duct taped to each other and duct taped to the pipeline - how do we rethink starting from the end product - that mesh data product - and hook the tools to that to make interacting with it better. If you build a system that truly focuses on intentionality and responsibility that people can see, it creates trust. Away with the data black box!

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#181 Learnings from BlaBlaCar's Early Data Mesh Journey: Positive Transformation for the People and the Organization - Interview w/ Kineret Kimhi Jan 15, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Kineret's LinkedIn: https://www.linkedin.com/in/kineret-kimhi/

Kineret's Blog Post 'Do’s and Don’ts of Data Mesh': https://medium.com/blablacar/dos-and-don-ts-of-data-mesh-e093f1662c2d

In this episode, Scott interviewed Kineret Kimhi, Analytics Lead at BlaBlaCar.

Some key takeaways/thoughts from Kineret's point of view:

!Interesting Decision!: BlaBlaCar reorganized their data organization but did not fully decentralize by embedding people into domains. Instead, they kept a central team but combined multiple functions into a squad around domains - a key domain might have a data engineer, data analyst, data scientist, and a software engineer.
!Scott Mantra Too!: Sharing your experience - data mesh or otherwise - early and often with the broader data community means better and quicker feedback, not just internal experience. It's okay to be vulnerable about what didn't go well, you can get better info and help save others the same pain.
?Crucial?: It's very important that when you split up your teams from functional data role teams, people keep in contact with functional role peers. If not, it can be very lonely as the only data engineer inside a domain. There is a significant turnover risk and a risk to not having scalable learning and knowledge transfer of data work if not handled well.
Data mesh will lead to a lot of potential changes to people's ways of working, especially with each other. Don't shy away from that, people need to know you aren't forgetting they need career development and that you'll support them as they learn and get used to new ways of working.
Communication really is the most important aspect of getting data mesh right. You need to get feedback and keep people informed, aligned, and in sync. Change is less painful if people are told why it is happening and are informed instead of discovering the results of change.
Look to align as much of your data team at the start of a data mesh journey - get everyone involved in the plan as best as possible. But don't completely change your data team structure at the beginning, start small.
On that note, starting small means there isn't a huge disruption to your ways of working and data org so a team/domain can get comfortable with new ways of working. You can get great feedback to make it easier/better for the domains that follow if you keep in close contact.
When doing your data mesh PoC, look for simple use cases. Set your PoC up for success and focus on learning how to do data mesh rather than tackling the hardest challenges first.
Reorganizing your data teams can be frustrating to non-data team stakeholders. You can't have everything drop to the wayside while you learn to do data mesh. There needs to be a balance between learning a new way of working and continuing at least some semblance of business as usual around data work so you don't cause major disruption to exec's ways of working.
It's really easy to go wrong with governance early in a data mesh journey. Getting people on the same page and on the same tooling is crucial so there is a better shared understanding, e.g. lineage, observability, catalogue, etc.
If you do not have key core aspects of data governance in place, get those in place before starting a data mesh journey or you will make it much harder on yourself than it should be.
?Controversial?: Data documentation needs to get disrupted because it is still far too manual and too difficult for consumers to really understand without just experimenting with the data themselves.
?Often Overlooked?: It's important to recognize data mesh isn't the right fit for every organization. Why are you looking to do data mesh? Does data mesh even address your data challenges?
!Important!: Make certain to give your PoC the resources - including people - needed to succeed but also set it up to succeed. Give the space and limit the pressure on your PoC to really learn if data mesh is for you and if your organization is ready to do data mesh.
As you get domain teams to start owning their data, it's not a switch to flip. It's a process, work with them to get them capable and don't ask them to do overly complex things as they learn. Crawl, walk, jog, run.
Make the team participating in your PoC feel like the pioneers and the potential heroes. Make it as easy on them as is practically possible and try to keep as much load off them as possible. Look to make participating in the PoC as beneficial to them as possible and not a burden.
Set up a strong governance process to prevent schema changes from causing unnecessary downstream pain. But that can't be on the centralized team to make the changes, that's a bottleneck and doesn't scale.
Don't expect data mesh to suddenly solve all your data challenges :)

Kineret started off the conversation saying she was previously running data engineering at BlaBlaCar but with their move to data mesh, that isn't really a necessary role anymore so now she is the Analytics Lead. This was part of their greater reorganization of their data org where they are organized around domains into squads instead of by functions like data engineering. So crucial domains might have a data engineer, software engineer, data analyst, and data scientist in one squad focused on their data. So BlaBlaCar has a central data team but each squad is essentially attached to a domain. They kept their chapters around functional roles to keep knowledge sharing high and promote more camaraderie between similar roles attached in a squad to the different domains. BlaBlaCar sees the value in sharing their experiences early and often with the general data community so they can take in external feedback and also help out others looking to do similar things with data. Scott note: If I had a nickel for every time I tried to preach this… :)

On advice as to how to start a data mesh journey, Kineret was relatively insistent that you need to form a group of people to partner with on the transformation. No matter your title, you need to have people to lean on and get feedback from. You can't drive it simply by force of will. Some things BlaBlaCar did that she believes helped make their journey (more) successful were: getting broad alignment around their data mesh journey, including around planning. It wasn't just a small team of people, all the data team worked together. Second aligns well with Zhamak's advice too: start small. It allowed them to get people used to a new way of working instead of trying to shift the entire data approach of the organization at once. They focused on collecting feedback from everyone involved in the PoC so they could see how well it worked and so that future domains could replicate the successful parts and avoid the ways of working that didn't go as well.

As part of their 3mo PoC, Kineret and team took in a LOT of feedback from that single team. The reorg made a number of the data squad attached to the domain feel lonely and disconnected from others in a similar functional role. Again, back to the chapters approach to keep people connected around their role functions. Kineret said she believes if they tried to move all the domains at once to data mesh, their journey would have failed and they likely wouldn't have kept their data people nearly as happy because they might not have implemented the chapters approach early enough.

To keep your overall organization on board with your data mesh journey, it's important to think about how stakeholders interact with the data team and keep that stable while you are in transition according to Kineret. If key stakeholders across the organization have to go through an entirely new and different process with each domain, those stakeholders are not going to be happy. So plan ahead and communicate what changes and have things well documented, not just at the data product level but at the business process level. If only you could do data mesh as if it were a separate thing unto itself but it's part of the business strategy, you can't keep it in a bubble.

For Kineret, there have been some data mesh transition pain points especially around people moving into different roles or day-to-day responsibilities but communication is the key to keeping everyone aligned and limiting the unnecessary pain that often comes with change. If someone is used to talking with Alice in the central team about challenges with data and then Alice is suddenly in another domain, there is some frustration and concern by that data consumer as to who to talk to and how can they get to a good relationship with their new contact. But a key goal of data mesh is to make data consumers' overall experience better. So while it might be a bit challenging as things change, keeping data consumers informed of change and making sure there are low friction processes to get what they need are crucial.

Kineret believes data documentation as a whole needs to be disrupted. We generally have some necessary pieces but it's still overly manual to properly document and consumers still generally can't really understand the data without digging into it themselves. The documentation generally isn't capable of doing what's necessary to get someone up to speed on a data product. Even though her team is doing great, it's still a challenge to find the right mix of important to in-depth. And it's still very manual work to create the documentation. Scott note: very much true. Data documentation is still an incredibly difficult task to get right and is probably far more tedious than it should be.

When asked for some general data mesh getting started advice, Kineret had some beginning questions instead. What does your general data governance look like? If it's not robust, you should look to set that up before decentralizing. You'll save yourself a lot of unnecessary pain. Second, what is your buy-in for data mesh like and what is the reason for thinking data mesh is the right choice to solve your challenges?* Data mesh isn't right for every organization. Third, are you really ready to do your PoC and give it the resources necessary to succeed? Are you setting your PoC up to succeed by not putting too much pressure or trying to tackle too hard of a problem? Lastly, can you find a PoC use case that is relatively contained so this doesn't have too much outside influence and too many stakeholders? Can you clear the space to make it possible to succeed?

*Scott note: so much this; if centralization isn't you bottleneck, decentralizing is far more likely to cause more issues. Be realistic about what data mesh can change and what it can't. Don't use an excavator to dig a 3 inch deep trench in your garden…

A non-standard approach Kineret and team took was by separating ingestion and putting that on the data platform team. So the data domain teams could focus on the cleaning and transforming data instead of setting up the extracting from databases or other data stores. This was part of BlaBlaCar's capability building and data ownership transition strategy. They didn't ask or expect the world from the data domains as they were learning. Find relatively simple things for them to do instead of the most complex data engineering tasks. Crawl, walk, jog, run.

Heading into the end of the conversation, Kineret really emphasized how important it is to get the people aspects of something like data mesh right. Make sure people can feel seen and heard, keep people informed, keep people in touch with those doing the same functional roles so your technical folks don't get too lonely, etc. Really make sure you focus much more than most technical people probably want to - the tech is cool! - on making this a positive transformative experience for the people in your organization, not just the organization itself.

Quick tidbits:

Governance can be really difficult but very crucial early in a journey. You want everyone on the same page relative to ways of working but also on the shared governance tooling, e.g. observability, cataloging, and lineage.

Make sure to treat your initial PoC domain teams like pioneers and give them the support and guidance necessary. Keep undue burden off them as best as possible, keep in constant contact for feedback, and look to make it as beneficial as possible to them. Celebrate the PoC domain team because they went and did the big, kinda scary thing that could transform your organization.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of January 15, 2023 Jan 15, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#180 Shared Understanding Leads to Data Value That's Outstanding - Interview w/ Chris Dove Jan 12, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Chris' LinkedIn: https://www.linkedin.com/in/charles-dove-b4715723/

In this episode, Scott interviewed Chris (Charles) Dove, Data Architect at Endava. To be clear, he was only representing his own views.

Some key takeaways/thoughts from Chris' point of view:

Data used by only one use case in one way is not how you make money by leveraging data, it's too expensive. Set yourself up to reuse data and make sure the organization is aware of what data is available.
?Controversial?: Tooling around data, especially metadata, has gotten better. But is it good yet? There are still some major fundamental gaps that seem like basic blocking and tackling around sharing data.
Data isn't the point, it's merely a vehicle for exchanging information.
Far too often there is an implicit understanding of a taxonomy/shared terms in different business units that is actually incorrect which leads to misunderstandings and mismatched data being treated as the same. But it's not easy to make all aspects of all parts of data explicit and easily understandable, we have to invest and find good ways to do that.
!Incrementally Important!: Business people in domains often don't understand their own data because it's embedded in an application. So they only experience their data in a context that is already framed for them by the application. So they don't think about someone else not understanding the data inherently when those others _aren't_ experiencing it through the same application.
Getting to a 'good enough' level of documentation is crucial to prevent misuse of data based on misunderstandings. But every organization has to figure out what is good enough and how to get there, what level of documentation is required, there isn't a blueprint.
A constant challenge in data is implicit assumptions by producers around 'they work here so they know this' instead of documenting nuance. This leads to misunderstandings based on incomplete documentation.
Beware the TLA - three letter acronym - in data documentation. It might have a lot of embedded context for those with domain knowledge but it's not helpful for those without the understanding. Focus on explaining the concepts at a level an outsider can understand.
?Controversial?: The point of data literacy isn't to teach everyone technical skills, it's to get to an understanding of their own data and how to share the context of their data so others can get a decent understanding of their data. If a good use case emerges, we don't need everyone to be able to create and maintain a data product but we need an organizational understanding of what data might be available and what it means.
Tribal knowledge is a double edged sword. It's great your organization has that knowledge but it's a massive risk point. Get it out of people's heads as much as possible. Scott note: find low friction ways like interviewing people to extract their context. Like this podcast...
Most organizations don't have a great data documentation strategy or practice. It's better to get going with learning how to share information about your data than try to make your data documentation or documentation strategy perfect upfront. Get something in place and recognize technical debt but something is better than nothing if it's not the end state.
Trapped metadata - where tools try to enforce a closed system instead of easily offering up critical information for ingestion by other systems - is a persistent problem that doesn't seem to be getting better. It's even worse when you write custom code to do transformations because most people aren't creating the necessary metadata at all.
We need vendors to be bought in that publishing their metadata is the right move and head towards metadata standards to make creating a more complete picture via metadata easy/feasible. Some vendors are moving in that direction, notably Atlan and data.world.
Truly getting people to change the way they think and feel - not just do work - is an incredibly difficult challenge that most companies don't ever really address. Make the change in ways of working a value-add to actually change hearts and minds.
To do data mesh well, we need to figure out scalable and highly effective ways of communicating changes - just offering data product versions won't cut it. To do that, data producers need to know how their data is used as well.
Companies need to really see and understand the business benefit from their data before they are likely to change their ways of working around data. That can be a chicken-and-egg issue though.

Chris started with his view that while tooling is getting better in general, most tools are still very lacking in how their metadata plays into the greater organizational view of data which means we can't do some pretty basic things. Or at least there aren't comprehensive tools that make easy sharing of context easy across teams because of many metadata incompatibilities/challenges. So we need to get to a way to show the semantic context-related metadata as well as the transformation metadata in one place that is also understandable by the business users. A hard order to be sure but fundamental to enabling the vision of companies being actually data-driven.

There is a very common problem in organizations that comes from an implicit taxonomy and homonym problem according to Chris. The classic example is the definition of a customer but it goes far deeper than that. Some bit of data often has a very specific meaning in a source system and/or domain but then a different business unit looks at it with their own interpretation of what it means and misses the nuance, the differences. So do you have an enterprise taxonomy or do you try to document the exact meaning differences or do you not let people have access to data in case they misunderstand? Not as easy of a choice as many would like to prevent these misinterpretations or misaligned data mixing.

An interesting and very crucial nuance Chris mentioned about data sharing: the business people in a domain are often consuming their own data through a different lens. The data is embedded into an application for them so the interface makes much of the nuance, the meaning explicit. But that meaning isn't included in the data by default - the column title in the table doesn't have that meaning. So those business people - the data producers - often struggle to understand why people are confused or don't get the nuance. So it's important to make sure the data producing domain understands the interface others use to consume their data. Scott note: this is benefit I hadn't considered of having domains consume from their own data products. If nuances about the data aren't explicit, if the documentation isn't good enough, will they get confused about their own data? Will that force them to do better in building their data products?

Chris hit on a common problem many are having in data mesh - and data in general: what depth of documentation and explanation is necessary for data to be useful and not misused? People automatically assume some level of knowledge of the domain simply because others are in the organization. 'You work here so you understand my domain' type of attitude. So we need to make sure people can at least understand what they don't know and give them a way to get up to speed on what they need to know about a domain. Self-service can be a recipe for disaster if people can't understand when they are missing the necessary understanding/meaning.

While domain-specific acronyms can have a lot of embedded information in them for people with knowledge of a domain, they are often a major hindrance to those trying to learn about the domain in Chris' experience. Instead of focusing on exactly what your team calls everything, focus on the concepts and why they matter. As Shakespeare said "what's in a name", don't be enamored with sharing context via domain specific language. Referring back to Vlad Khononov's DDD episode, the internal domain language is the ubiquitous language but the published language is what is used to share with the rest of the organization. Focus on that published language - how can things be understood easily by those outside the domain?

For Chris, the point and meaning of data literacy isn't what most think - it's about getting people to understand what data they have and the general meaning/context so it can be communicated with the rest of the organization. It's understanding how data can be used and shared, not the exact technical aspects. It's about getting to a capability to share context and understand other's context around data without getting overly technical. When there is a use case that emerges, not every single person in the company needs to be able to create and maintain a data product. Basically, the concepts matter far more than everyone learning SQL.

In Chris' view, tribal knowledge is a very dangerous place to be. You have amazing and extremely valuable knowledge but it's trapped in people's heads. What happens if they leave? We all know about tribal knowledge but it's especially important in data because again the context and nuance, not just the column name, matters :) So extract that valuable tribal information, get it into a consumable format for the entire organization. It frees up the time of your most knowledgeable people too as they aren't answering questions all the time - extract once but leveraged by many :)

Good documentation, good knowledge sharing isn't about anticipating every challenge and writing out the fix or entirely preventing it according to Chris - that's not feasible. It's okay to get things into a knowledge base rather than the perfect metadata tool at first. You want to improve but if you are waiting for the perfect solution, you won't ever move forward with your data documentation. So get something out, put it in front of others, ask for feedback, and improve. And as stated, documentation doesn't have to answer all questions - it's something to make sure people generally understand a certain set of data and if they have a deeper question, they have a clear question escalation path for who to ask.

It's easy to get lost in data by focusing on 'data as the point' in Chris' experience. Data is merely a vehicle for exchanging information. But there are lots of interesting technical challenges in dealing with data so data people often lose the plot. Without the context around the data, it's useless too, so we have to focus on delivering it as one packaged unit. Scott note: this is what Zhamak keeps referring to as a data product container or a unit of data - that it isn't merely the 1s and 0s but the context, the user experience, the lineage, etc. wrapped in one package so it is usable as is.

For Chris, the biggest issue right now in data, especially for something like data mesh, is the trapped metadata problem. It's something Scott has mentioned repeatedly: most tools that touch your data in some way at best generate metadata that is trapped in that tool or it's extremely difficult to extract and integrate that metadata into other tools. And when people write custom code to do transformations, they often don't generate the metadata at all! So trying to get the full necessary picture of what's happening around our data is extremely time-consuming and difficult.

Chris called out the need for specification around metadata so we can at least bring it all into one place. Only a few vendors have moved to making it possible to even extract most of the metadata they create - he noted Atlan and data.world - but hopefully more vendors are pushed - or dragged - into doing the same. OpenMetadata or other early projects may provide a good way to start developing some standards for how things are described, shared, and/or stored. But again, trapped metadata is a lock-in pain that vendors are seemingly unwilling to let go of unless their hands are forced.

To really move forward with how we all approach data - as an industry and at the organizational level - we need to change the way people think and feel about data according to Chris. But change forced upon people only _might_ change their way of working and usually doesn't. So we have to focus on changing hearts and minds or the behavioral changes won't actually net the necessary care changes to the ways of working and understanding we need in data. Easier said - to change hearts and minds - than done but actually changing how we work requires empathy not mandates.

Chris finished on two points: the first is to really change the way an organization does data, they have to understand how data fits into their overall strategy and how treating it as a product impacts the work. Something may be valuable now but that value might fade. It's okay - and even very healthy - to end of life any data work that is no longer valuable. And the second point is that reuse is really key to generating strong business benefit from data. The cost of getting data to a point you can leverage it is typically high, look to make it reusable and find valuable ways of reuse as much as possible.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#179 Reliability Engineering for Data and Data Mesh - Mesh Musings 40 Jan 09, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#178 Data Modeling in Data Mesh Panel - w/ Juha Korpela, Kent Graziano, and Veronika Durgin Jan 08, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Juha Korpela (Chief Product Officer at Ellie Technologies) facilitated this panel on data modeling in data mesh with Veronika Durgin (Head of Data at Saks) and Kent Graziano (The Data Warrior, former Chief Technical Evangelist at Snowflake). This panel was hosted by the Data Mesh Learning Community in partnership with Data Mesh Radio.

Veronika's Links:

Veronika's LinkedIn: https://www.linkedin.com/in/vdurgin/

Data Vault North America User Group: https://www.meetup.com/dvnaug/

Kent's Links:

Kent's LinkedIn: https://www.linkedin.com/in/kentgraziano/

Kent's Website: https://kentgraziano.com/

Kent's Twitter: https://twitter.com/KentGraziano

Data Vault Alliance: https://datavaultalliance.com/

Juha's Links:

Juha's LinkedIn: https://www.linkedin.com/in/jkorpela/

Ellie Technologies' website: https://www.ellie.ai/

This write-up is from Scott Hirleman's point of view:

As someone without a ton of depth in the data modeling concepts, here are some of my key takeaways that should be taken with a grain of salt :) I decided not to write up everyone's opinions but more what are my takeaways:

Start from the business concepts instead of the technical (a data mesh theme).
Focus on enabling people to do the data modeling instead of trying to do it centralized but do have a centralized understanding.
It's crucial to not only consider data modeling at a data product level, that's the route to data silos.
Communication is the most important data modeling skill.
Share early and often to get feedback and work together on that fast feedback cycle to quickly iterate.
Limit your blast radius, only bite off what you can chew and limit what can go sideways early as you learn - do not cause lots of downstream damage.
If you don't stay connected to each other and communicate well, you'll likely have MDM (master data management) style nightmares in data mesh.
Centrally define the standards and rules and have a clear way to get help and settle questions/disputes.
Starting data modeling from technical integration is about what can we do right now. We should focus on what we need to do to drive business value, not what is possible based on the existing solution.
In data modeling, far too often people don't look at what data consumers want. User requirements, business requirements, and technical requirements all must be met.
Not everyone needs to be a data modeler but everyone should understand how information is communicated internally via a data model.
Alla Hale method (episode #122): show up to every meeting with something to discuss. Don't show up to the first meeting with it built but constantly show something to get feedback on.
Shared understanding is crucial and can ONLY be reached via strong communication. Stop trying to shy away from communication.
Make the implicit explicit - "what do you mean by that?" is one of your most crucial tools to doing data modeling well.

Data modeling in data mesh will probably be far more similar than dissimilar to data modeling in a more centralized world. The focus on the business concepts is crucial. Far too often we try to start from the technical instead of what are we trying to achieve and it's crucial to not fall into that trap. Getting the technical aspects for interoperability wrong can be a pain but if things work together technically but not at the business level, that's a lot of sound and fury signifying nothing - essentially that's a lot of cost for work and compute that doesn't lead to actual business value.

One thing I'll note is every one of the guests is a Data Vault proponent. I'm not sold that it's the right way in the long run for data mesh - I feel like we need to evolve data modeling concepts for a more distributed, federated organizational approach. But from what they said, Data Vault does sound like a very solid base to start from - start from the business concepts first and focus on what you are actually trying to accomplish. Data modeling for the sake of data modeling is not something anyone should want to do.

As with just about everything else in data mesh, data modeling should be about limiting your blast radius of potential negative impacts as you get to fast initial and incremental feedback. Get to that iteration, take on things that matter but don't make it a big bang. Fail fast and all that :) This is not about taking requirements and going off to your own world at the domain level. It's even more crucial to have overall communication/cohesion as we enable more and more people/domains to own their data.

The most important aspect for developing lasting interoperability via data modeling is the business concepts. It's not that hard to do the technical interoperability once you figure out how things should work together. Starting from the technical feels easier but is a recipe for losing the value of the bigger picture. Typically, technical-based implementations are less extensible because the technology decisions are embedded into the solution instead of an enabling factor. Veronika said something like "focus on the words and meanings and not the data types." She also said that starting with technical integration focuses far too much on what can be done with the existing implementations instead of what do we need to drive value and how can we improve the existing implementations to create more value.

As with most things in data and software engineering, it's okay to build your data model in an opinionated way but maintaining flexibility, especially as you are doing your initial development, is crucial. Major change can be quite costly - especially if you have to change the entire foundation of what you are doing. Kent railed against the need for rework. Build your data model and subsequent data products so providing another view or angle isn't nearly as difficult and requires only the work to do that, rather than changing everything else you've done to also accomplish the new data view - in data mesh, this would often be a different API or a new table sharing the data for a different use/perspective. Again, look to prevent rework and ensure flexibility.

A massive concern with data mesh is data silos, the worry that if you have a bunch of domains doing their work separated and not in communication, nothing will interoperate. So you probably do need some kind of centralized group - whether that is their main role or part of their other responsibilities - helping domains do their work in the context of the greater organization. Note, that is what loose coupling from microservices means. Fully decentralized would be no connections versus things work together but can scale independently - that is decoupling. Having people who are there to help is the key to federated instead of fend-for-yourself so data architects are a crucial aspect of data modeling in data mesh.

While there is no centralized data model in data mesh - they aren't flexible and mean you lose a ton of context from trying to force things to comply with that model - there obviously can be centralized guidance and direction, a standard set of data models, etc. Think about a well-functioning federated government - maybe not the US… There are people doing work in the centralized function but it's about enabling those at the more local level to do things the right way. Juha quoted someone with something like "governance is not about leading people to do things right, it's about setting them up to do things the right way". That centralized team can't know what's right for specific situations - because they lack the localized context - but they can specialize in enabling doing things the right way. Kent claimed there is an enterprise data model and that can quickly go the wrong direction but if you interpret it well - that there are clear relationships across the business that are crucial to model well in your data, that are fundamental business truths you should reflect in your data - it can mean much less learning of deep domain specific context because you understand how domains fit the organization.

A number of people believe data modeling must be all about one view or perspective to rule them all. That is where data mesh fundamentally pushes back. You can have one view you agree on as an organization - such as revenue - but others should still be free to publish something that is similar in meaning but from another view. Much like in Domain Driven Design (listen to Vlad's episode for more, #171), there should be a 'language' (broad definition, think interface and terminology) of the domain to maximize the context of information shared in the domain and a separate 'language' used to communicate to the rest of the organization. That way we can still maximize context for business value locally but communicate globally in order to also maximize global data interoperability, which is a crucial organization-wide business value driver.

Kent mentioned another worry of data mesh that is often closely aligned with the data silo worry: master data management-related nightmares. While we absolutely have to reinvent MDM for data mesh - look for a few panels on that in the near future? - it's pretty clear it's bad to potentially have 10 different definitions that might filter to someone who doesn't understand the nuance and differences. Especially if that exec asks a simple-seeming business question and gets 5 different answers. Data trust is gone. So we have to be clear in tackling that problem and strongly communicating. Maybe not mastering data but mastering ways of answering typical questions?

Overall, I think you will learn a ton just like I (Scott) did :)

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of January 8, 2023 Jan 08, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#177 - Zhamak's Corner 14 - The Data Can't Protect Itself Jan 03, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#176 Measuring the Value of Data Work Part 1 - Mesh Musings 39 Jan 02, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#175 Ethical Data Usage - Informing and Educating Consumers - Interview w/ Esther Tham Jan 01, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Esther's LinkedIn: https://www.linkedin.com/in/esthertham/

In this episode, Scott interviewed Esther Tham, Experience Designer at Thoughtworks. Scott reached out to talk about data ethics based on a post Esther made on LinkedIn.

Some key takeaways/thoughts from Esther's point of view:

When designing your UX (user experience), companies should aim for as little friction as possible when signing up or transacting. For an ethical company, that means collecting as little information as possible to still maximize value of the service to the user.
Companies: if you don't need it, don't collect it! It isn't ethical but also it increases your attack surface for a data leak and potentially lowers consumer trust.
We don't have the proof points of many companies doing the right thing and disclosing potential issues of sharing information with them in an understandable way. But that would likely increase consumer trust. Is that trust worth more than the hassle to a company? We need companies willing to try being more ethical to really know but it's a cost with a very uncertain upside so not too likely.
People need to learn that their personal data has value - and risk - associated with it. Don't give it over without thinking about how it might be used/misused. But most people are nowhere near that thought process yet. Right now, most people are only worried at most about getting scammed, not should this company have my data and how might they misuse it.
Ethics isn't just about collection or even usage, protection is also crucial. If you can't protect sensitive information, you shouldn't be collecting it.
How can we encourage the general population to really care about ethical collection and use of their data? Is it just better explanation of how it's used? With greater understanding, will most people actually care?
The question of who is responsible for ethical data collection is an interesting one. On the one hand, companies should be behaving ethically. On the other hand, they often don't so how much responsibility to protect sensitive information is on the consumer not handing it over in the first place?
A designer's role is to advocate and build for the user. But we still don't really know exactly what most users want when it comes to data ethics. Do they really care about ethics around their data or are they willing to trade their data for services? Is it about more education/communication or do users genuinely not care? We need brave companies willing to test.
How do we press companies to be more ethical in the data they collect, how they protect data, and how they use data? Have many companies suffered damage - reputational or otherwise - from ethics breaches? We clearly can't trust every organization...
On the flip side, how do companies that are actually doing the right things ethically communicate that to consumers? Is there a value to consumers where they will seek out companies with high ethics? Is the cost of behaving ethically worth it, does it result in a tangible benefit? We assume there is an additional cost to behaving ethically too so there needs to be an upside for companies to consider it.
It's easy for consumers to have a false sense of security online relative to their data. While identify theft and similar issues are on the rise, companies are still asking for - and consumers are regularly freely giving - sensitive PII.
Very few people really think about potential misuse of data we give to private companies, often with little explanation by those companies of what they will use our data for. Can we really expect companies to fully explain their projected use of data when that might simply confuse more people? We can press them to do it but likely not expect them to willingly.
However, when companies do try to explain their use of data, does anyone read it? Are EULAs actually useful? Do we need something that is in addition to a EULA to explain how data will be used and what will be collected?
Most people seem to not really be all that concerned with the data they share until it seems it was used improperly, especially if a company sold their data to a partner or some scammer got ahold of it. And most don't expect a scam to happen to them so they only look to react after the data is already out there. Can we change their approach and view?

Esther started the conversation with her background of growing up in Singapore, one of the safest large cities in the world. But that kind of environment can also lull people into a false sense of security when transacting online. Even the government was not really thinking about misuse as birth certificates, passports, and national ID numbers were all the exact same number until 2006. And until 2016, companies could ask for your full national ID number and many people gave it out without a second thought to get access to services.

The Singapore government has pretty stringent requirements to only collect information for legitimate purposes and store it securely. The few incidents of people in their government accessing the private information of others has led to strong repercussions. Basically, it's not like many of the other countries out there with potentially strong laws but little to no enforcement. But when it comes to private companies, even in Singapore a number are still asking for a significant amount of data without giving clear justification or expected usage.

For Esther, creating a low friction user experience is a bit of a double-edged sword sometimes relative to ethical collection. Do you store someone's credit card information to make transacting easier next time? Do you try to collect as much information as possible to show them relevant ads? Let's be honest though, that second one is about selling ads not a 'better ad experience'… So, how do companies effectively balance how much information to collect in order to provide a great experience but not too much where you are getting things you don't need?

A fantastic point Esther brought up is that data ethics isn't only collection and usage, though that's often where we focus. Data protection, especially around sensitive information, is a major ethics challenge/question. If you can't protect the data, should you really be collecting it? If you are in possession of sensitive data, what level of duty do you have to protect it? Are you salting and hashing your passwords and other sensitive information?

Again, Esther believes we have to think about who is the burden on to protect information. If a company collects it, presumably the burden should be on them. But if people willingly give up very sensitive information to companies, what level of responsibility do we place on people to be informed and smart about giving their data away? Who is the 'shame on' for the second time something happens? How do consumers protect themselves from companies like the US credit bureaus that collect information without consent? Especially when they haven't been ethical in their level of data protection.

Another interesting point Esther raised is how do companies properly explain what data they will collect and how will they use it. Be honest, show of hands, how many actually read through most of the EULAs you agree to? Is that because they are insanely tedious or because we genuinely don't really care or have resigned ourselves to data being misused anyway? Is that just a friction point in the onboarding experience?

In Esther's view, most people seem to not really be all that concerned with the data they share until it seems it was used improperly, especially if a company sold their data to a partner or some scammer got ahold of it. And most don't expect a scam to happen to them. So consumers need a better way of understanding their information attack surface and to not give information as freely but that's mostly on companies to not ask for it. And if they can profit off it, can we really expect companies to stop asking? Is this a chicken and egg scenario where neither side is really going to move first?

If a company does really do a good job of disclosing potential risk of handing over data and exactly how they are going to use it, that presumably increases customer trust according to Esther. Unless people just don't want anyone to use their data but they are happier when they don't have to think about it. And then, for a company, is that increased trust worth the hassle or even the potential friction and scaring certain users? It might be the 'right' thing to do ethically, but how many companies are _really_ focused on acting ethically? So we aren't sure if this is a high cost for companies and we aren't really sure it leads to anything all that positive and we aren't sure consumers really care… so why would a company do it unless they have to or feel the need to act ethically? Are any companies brave enough to test it out?

Esther wrapped the conversation up with a call to action: understand that your data has value and consider if giving away your data is worth the value a company is providing you. And how could it be misused. That doesn't mean be paranoid but also, don't give out information quite as easily.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of January 1, 2023 Jan 01, 2023

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#174 Measuring the Impact and Value of Your Data Products in Data Mesh - Interview w/ Pink Xu Dec 29, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Pink's LinkedIn: https://www.linkedin.com/in/pink-xu/

In this episode, Scott interviewed Pink Xu, Change Manager of Business Impact of Data Products at Vista.

Before we jump in, there are a few specific examples in this to Vista but I think it is incredibly relevant when looking at measuring the impact of your data work. As Pink says, set the objective/goal for the data product and then measure if it met that objective/goal. It isn't the impact framework's job to specifically measure if the objective of the data product is valuable, only to provide an objective way to measure how well did the data product meet its goal.

Some key takeaways/thoughts from Pink's point of view:

Look to standardize the way you measure impact for data products. Much like data observability/SLA metrics, a centralized team shouldn't be the ones focused on measuring or defining the target impact of a data product, only providing the way to measure it.
Again like data observability, an impact measurement framework/methodology means people can trust exactly how impact was measured without having to dig into every measurement decision. It's not like grading your own essay, which is a problem with a not impartial measurement.
Impact measurement can only go so far. It shouldn't be the only consideration in valuing a data product but without a fair, impartial framework, measuring the value of work becomes all the more difficult.
A data product "enables business impact", it cannot create the impact itself if no one uses it. Think about who gets "credit" for the impact - is it the data product creator or the team that acted on the insights from the data product? Look to reward/credit both parties if possible to generate a better collaborative culture.
Data products don't inherently have value - they only have value if used. So look to interview data product users - and potential users that aren't leveraging that data product - to help assess how valuable a data product really is.
It's crucial to have comparability across data products regarding impact measurement to make it easy for people to understand what certain terminology and measurements mean. A general methodology framework for measuring means people can better understand and compare.
It's easy to only want to focus on direct financial impact of data products but this ignores value to the organization that isn't only increased sales or cost savings. If you don't focus on other factors, you will only create data products to directly drive sales or cost savings instead of helping make better long-term - or even short-term - data-informed decisions.
When considering creating a new data product, think about the impact you expect it to have. What is the objective for creating the data product? If you are successful with your data product, what does success mean and why would that justify the investment?
An impact measure that isn't directly financial is a decision influencer model. Does the data product drive key business decisions and is it a key contributor to those decisions or just one of many aspects used? It's more qualitative but still valuable/useful.
Another impact measure might be customer satisfaction / impact on lifetime customer value. This is a much more nebulous impact to measure, especially if customers have a long lifetime. But it doesn't make it not valuable.
Impact measurement is notoriously difficult - just ask anyone in marketing about their attribution model on in the non-profit space. Focus on creating an impact measurement framework and working with teams so they can leverage the framework; but the central team doing the measurement work itself can create a bottleneck and they will often lack the business context.
!Useful Outcome Insight!: More data products' objectives at Vista are business decision impact rather than direct financial impact.
They have not yet reached the point where impact measurement is used for project investment decisions - that is a harder problem and okay to tackle down the line as you learn to measure impact first. They are in the descriptive analytics phase for the impact framework, not predictive or prescriptive. Basically, that is a running problem, it's okay to not take it on when learning to crawl and walk.
Impact measurement is not about a yay/nay decision or a good/bad score for a data product; the impact measurement framework is only part of the value measurement of a data product.
Don't only focus on short-term impact data products - often the immediate financial impact. That will lead to small wins instead of making better long-term decisions. Focus on measuring both. Your impact framework should align to your business strategy, not only your next quarter's income statement.
At Vista, the goal is to get to a place where they can reasonably compare financial impact, business decision impact, and customer value impact against each other. But it's still at best apples and oranges and they are still early days.
Surviving versus health: when you are focused on 'surviving', impact measurement is a champagne problem. But impact measurement is important to your long term 'health' in your data ecosystem to make better decisions on data work.
Trying to measure data product usage in a vacuum is a bad methodology. You should instead look at what would be the expected usage if it were successful, if users leveraged it a lot. Otherwise, something that drives key decisions but rarely touched will be valued much less than something people touch quite often but they don't really leverage to make decisions.
Much like any aspect of data mesh, it's crucial to communicate with users as you build an impact framework and methodology. What are their concerns? How can you make it easy for them to adopt? It's easy to lose sight that an impact measurement methodology is also change management which requires strong communication.
When building a data product, think about who would want to consume information about its impact. Then make sure you leverage the parts of your impact measurement framework and align on how the data product will be measured. Figuring out how to measure after the fact creates friction and can lower trust.

Pink started the conversation on what her focus is: setting the framework, the methodology for measuring the impact of data products. At Vista, they wanted to create a way of measuring the impact of data products that was as standardized as possible so people looking at impact don't have to learn a new way of measurement for each incremental data product. With data observability in data mesh, most organizations are focused on centrally defining and describing data quality/trust metrics instead of centrally doing the measuring or setting the SLAs - that is on the data product owners. That way, there is comparability across data products, which means less work for someone to understand the quality of the data. Vista is taking the same approach with impact measurement, making it easier on data producers to measure their own impact and report on it and making it FAR easier on people consuming that information to compare across data products instead of customized metrics for each data product.

According to Pink, the most common impact people want to look at for a data product is direct financial impact. And that is completely understandable and what she initially thought she'd focus the most on. But as Zhamak and others have also mentioned, this can lead to only creating direct bottom-line impact data products, things that directly drive more sales and/or cost savings. Often that is quick, small scale, not interconnected wins. That means missing the big picture of what is happening relative to your organization to set long-term strategic direction. If there is something that helps to reduce time to hire for HR or leads to better retention, that is a much more difficult financial impact or business impact to measure. Then, think about the decision to enter a new market, the go/no go decision - is that really able to be measured in financial impact terms versus smart business decision terms? So at Vista, they are also looking at more qualitative measures like what impact does a data product have on making key business decisions - is it the core driver to making a key decision or is it one of 100 aspects of a decision? And then what about impact on customer experience?

So, Pink is spearheading developing a framework that includes what type of impact people want to measure. That obviously can be direct financial impact - be it sales, margins, etc. - but also a number of other impacts as mentioned above. The key to the framework is making it so people can easily measure the impact and then communicate it in a language others - mostly execs - understand. But the central data team - notably Pink here - does NOT tell data product owners what impact a data product should have. Nor do the central team do the measurement. The central team's role is to make sure people are measuring appropriately within the framework and help data product creators adopt the measurement framework.

As an example at Vista, Pink talked about a data product measuring the time between when a customer starts using a certain template to completing their order as well as the order abandon rate. If you look at how quickly someone is actually able to do the work they want and get to a good final quality, you can recommend better templates to users or even decide what types of templates you might want to create more of. And you can measure reorder rates as well. So the impact measurement is focused more on the customer satisfaction with a long-term, long-tail financial impact. Much harder to measure than the impact of a pricing change.

A really interesting insight is that even after a year, Pink still believes they are pretty early in their impact measurement journey. They are not using impact measurement yet to drive decisions on creating new data products versus measuring what has already been created. Essentially the impact measurement is in the "descriptive" phase of analytics and they have not yet moved to predictive or prescriptive. That doesn't mean the data products themselves are only descriptive but the measurement is currently at that stage. Scott note: this is VERY unsurprising to be at descriptive stage. Talk to anyone in non-profit about impact measurement. It's incredibly difficult. Pink and team are pretty far ahead of the general curve here; as an industry, we are just learning how to effectively measure data work impact/value in general.

Pink made a great point circling back on financial impact measurement. If you are focused only on measuring immediate value, you won't build data products for better long-term decision making. But you can't only focus on the promise of what impact a data product might have in the future. So you want to make sure any impact measurement framework considers all aspects of value and aligns with a long-term vision but can point to value creation in the short term too. As part of this, they are trying to work towards a way to equate financial impact, business decision impact, and customer value impact to make it more comparable between the three. But that takes a lot of data to measure long-term versus short term outcomes :)

According to Pink, a few missteps that many might make relative to impact measurement, some of which are covered above but important to reiterate:

1) creating a data product without a specific objective you can measure against. If the data product is successful, what does success look like and would that success justify the investment?

2) focusing too much on direct, measurable financial impact when deciding which data products to create/fund.

3) focusing on short-term or long-term impact only.

4) not interviewing data product users - and potential users - to assess their value.

5) creating a framework that doesn't make impact measurement comparable - at least in some form - across data products. It's really only qualitative at that point even if it involves numbers.

6) creating a framework focused on valuing data product usage in a vacuum. They are essentially vanity metrics - how much usage does the data product get compared to expectations and how much usage does it get relative to what a successful data product for that purpose/objective would get?

7) forgetting to involve users in the framework process creation - what are their concerns? If they were concerned it would value business decision impact well below financial impact, do you think they'd want to use it?

8) skipping communication about what people consuming a measurement would want to know. If the CEO wants to know XYZ metric from a data product, you want to build that measurement in upfront.

9) not helping data product owners adopt the impact measurement framework. They need to be able to understand it to want to use it, especially if it is measuring essentially how well their data product is driving value.

Quick tidbits:

When considering creating a new data product, think about the impact you expect it to have. What is the objective for creating the data product? If you are successful with your data product, what does success mean and why would that justify the investment?

A single framework means everyone is looking at the same things and makes it easy to understand. Are they reporting increased sales, contribution margin, gross margin, etc.? It also means people can trust how it was measured. Scott note: as a former financial analyst, this is incredibly important…

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#173 Zhamak's Corner 13 - Preventing Unnecessary Data Movement - In Spirit and in Reality Dec 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

In this episode, Scott and Zhamak covered the misconception many have when she says "leave the data where it is" - it's about leaving the ownership with those who should have it, not leaving the data in the source system. If you only do source system, all you have is current state, so your data isn't even immutable or bi-temporal! They also discussed the need to be smarter about processing data - should it be at the source or should it be on query? There isn't a universal approach but we also shouldn't have to move the data around just to process it, bring the processing to the data. Lastly, we need systems to get smarter around efficient processing. We have far too much manual work on making data processing efficient.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#172 Data Governance Evolutions and Revolutions for Data Mesh - Interview w/ Andrew Sharp Dec 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Andrew's LinkedIn: https://www.linkedin.com/in/andrewsharp27/

Blog post "Data Mesh – Is this the evolutionary trigger to reinvigorate Data Governance?":

https://www.theoaklandgroup.co.uk/data-mesh-is-this-the-evolutionary-trigger-to-reinvigorate-data-governance/

In this episode, Scott interviewed Andrew Sharp, Data Governance Lead at the consulting company The Oakland Group based in Leeds in the United Kingdom.

Some key takeaways/thoughts from Andrew's point of view:

Data Governance changes that are necessary for data mesh are both a threat and an opportunity. Are you throwing away the good with the bad? Or can you reinvent your practices broadly to do governance better in general and clean out bad habits/approaches when moving to a federated model?
!Controversial! Of the four pillars of data mesh, governance is the most challenging and least mature.
There are no roadmaps to doing federated governance for data mesh well - and likely there can't really be a specific roadmap for all as every organization is different. People are just starting to find their way on how to do governance in data mesh well and people will need to explore for their organization.
?Controversial? Data mesh will likely require a seismic - or maybe tectonic - shift in the way we approach data governance. That doesn't mean organizations have to completely change it all at once - that is overly high risk - but it probably won't work if we just try small shifts to our governance approach instead of small steps leading to a large movement.
Many organizations are already not doing data governance very well. So adding in the complexity of data mesh and finding entirely new ways of working when the current state isn't great? That is going to add additional challenges for many organizations on their data mesh journey.
Try to build momentum around positive organizational changes for governance. And make sure people are bought in to the test, learn, iterate model or change is going to be much more difficult than it needs to be, data mesh or not.
Data ownership: people generally don't really understand why data ownership is important. But often, it's hard to get domains to truly take ownership. And many don't understand what good data ownership means/entails. It's necessary to explain why they are the best owner and what owning data means.
There's already a shortage of people who are actually data governance capable and that's likely to get worse as data mesh creates more demand. It remains to be seen if we need embedded governance professionals in each domain or if making existing people in the domain governance capable will work - basically, full time roles or capabilities/responsibilities as part of other roles, we don't know which will win.
Existing data governance professionals will need to learn and adapt to keep pace in a data mesh world. Governance won't be exactly how it has been for years but there are lots of opportunities for governance capable people who also understand more technical aspects. You can teach the required tech aspects to willing governance people but they have to be willing.
?Controversial? There has been a shift - sometimes a large shift - towards more technical aspects in many people's governance roles. But that is likely more of a pendulum swing instead of permanent shift to more tech than non-technical. Scott note: Focusing on the technical and not all the other aspects will result in VERY subpar value realization of your data. It's where data engineering often goes wrong.
?Controversial? There is a misconception that the central data governance team goes away completely when everything is federated - data mesh or not. But that's more purely decentralized. The reality will be somewhere in between full central control and fully decentralized - and that balance, we still have to discover what works best.
In physics, there is distribution of load. Instead of one central team doing all the heavy lifting, you distribute it out to far more people and they each have a smaller load to lift. The central team shifts to focusing on the coordination and bigger picture. Just because you've been lifting heavy as a central team and can do it doesn't mean you should!
It's a valid and common strategy to not rush into data mesh. Many are testing and learning at a smaller scale rather than jumping in with both feet. They are testing, learning, iterating, and then reflecting. There isn't some arbitrary time table for doing the large scale changes like data mesh requires.

Andrew started the conversation off with a potentially controversial - but probably often agreed with - statement: of the four pillars of data mesh, federated computational governance is the most challenging and is the least mature in ways of working/patterns. Organizations are starting to learn and make their way forward but it's still a major challenge. Be prepared to explore and find the right path for you and your organization.

According to Andrew, most organizations are already not doing that well with data governance in the traditional sense so trying to figure out how to do it in a federated approach will be tough. And in data mesh, the computational aspect of federated computational governance means things are automatically applied where appropriate. That's very hard to do when you know exactly what needs to be done so it will be doubly hard in data mesh where we are still figuring it out. But changing your data governance approach can be an opportunity, not just a threat to existing status quo. How can we leverage the change we are doing to governance to be better than we ever were before? Far easier said than done but it's not only challenges.

To do data governance right in data mesh, Andrew believes it is more likely to require a major shift to generally how the industry approaches data governance; organizations will need to make big changes - over time - rather than just a few tweaks to better align with data mesh. But, it is very early days and that all remains to be seen, just a prediction. Scott note: I strongly agree with this belief. I think people are looking for ways to not invest effort in aspects of data mesh but I think many have noted the automated/scalable governance work pays significant dividends as your implementation goes wider.

But, Andrew wanted to stress that while we need major shifts, it's almost more like tectonic shifts than seismic shifts which often result the volcanic eruptions and earthquakes. Large but not moving quite as quickly - the big bang change approach to governance is overly risky. Why put all your eggs in one basket rather than try incremental improvements? Data mesh is all about trying, getting feedback, and iterating to improvement and governance shouldn't be any different. Build up the momentum around your changes and work with people to communicate where you are headed and why.

When discussing evolution of data governance and sort of traditional data governance roles and people that have been working in governance for a long time versus new people moving into the space, Andrew believes it is crucial for those doing the traditional type of data governance to grow and adapt their skills, especially technically. Will roles require additional responsibilities? Will domains have embedded data governance-focused people as their main role? Or will most of data governance at the domain level be split to responsibilities handled by roles not exclusively focused on governance? He doesn't expect widespread redundancies but do prepare for some changes. That said, it can be very much of a pendulum action instead of a shift that stays - so potentially look for an overly technical focus for a year or two before it settles into a better equilibrium.

"Turkeys voting for Christmas" is a phrase Andrew used relative to perception of the work many governance teams are doing in data mesh. Essentially, if turkey is a traditional Christmas dinner, are these governance teams that are helping lead the work to federate governance eliminating their own roles? He doesn't believe so and Scott STRONGLY does not believe so. Look at federated government - it isn't fully decentralized, that is just silos. Data silos are bad. So you need central coordination points and planners. Where the balance falls for governance responsibilities remains to be seen.

Historically, the central governance team has been doing all the heavy lifting because they are the ones trained to do so according to Andrew. But if we use a fishing analogy, we can see why central teams are happy to participate - if we have to fish and provide food for everyone in the organization, that's a LOT of fish you need to bring in. Instead, give them rods, teach them to fish. You can still focus on the big value fish - e.g. going and catching a swordfish or a tuna - but by breaking the work load down into manageable chunks, everyone can move faster and focus on creating more value where they have the best context. The less coordination we need across teams, the less unnecessary friction there is.

Other quick tidbits:

Most understand why data ownership is crucial. But many domains are not willing - or not capable - to take real ownership of data immediately. So gradual capability building and ownership handover is probably necessary.

The role of data governance professionals in data mesh is still in flux. Will there be embedded roles in domains or will it merely be skillsets as part of broader roles? Either way, there is likely to be a significant shortage of highly capable data governance people while the need for those people is greater in data mesh than traditional approaches.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 25, 2022 Dec 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#171 Creating Scalable Interoperability - Not Only Systems - Via Domain Driven Design - Interview w/ Vlad Khononov Dec 22, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Vlad's LinkedIn: https://www.linkedin.com/in/vladikk/

Vlad's book Learning Domain-Driven Design: Aligning Software Architecture and Business Strategy: https://www.oreilly.com/library/view/learning-domain-driven-design/9781098100124/

Before we jump in, the phrase DDD is used a LOT in this episode. It stands for Domain Driven Design. We've had past episodes on it but I want to make that clear. It's also important to note, there are some very specific terms used in DDD and it is easy to get overwhelmed. Look for the meaning and ignore the terms - we are designing how things work together, whether business capability, software systems, or general data flows.

There's also the importance of the difference between published language and ubiquitous language in DDD. Essentially, the ubiquitous language is the language of the domain and the published language is the language used to share information from the domain externally to other domains. So ubiquitous is internal facing between the business and software engineers in the domain and the published language is external facing to the rest of the organization.

In this episode, Scott interviewed Vlad Khononov, Author of Learning Domain Driven Design (DDD) through O'Reilly, Senior Cloud Architect at DoIT International, and independent consultant on DDD and distributed systems.

Some key takeaways/thoughts from Vlad's point of view:

DDD is a big topic to learn - you don't have to learn every aspect to take good value from it.
Make the implicit explicit. The more something is explicit, the easier it is to manage and manage people's understanding of it.
It's important that the published language doesn't change as often as the underlying implementation model within the domain. But we also absolutely need to be able to evolve our published language. In data historically, we have trained consumers to believe we won't evolve the published language and have tied that too closely to the implementation model.
?Controversial?: The concept of a domain in data mesh is very different from in Domain Driven Design (DDD). In DDD, you can really only identify business boundaries but in data mesh (or DDD for data), you can actually have an impact on - and potentially change - those boundaries.
Iteration is a key aspect of DDD. It's not about getting it perfect, it's about getting to okay fast and improving. Learning to do that in data will be tough but important.
It's crucial to understand the difference between the ubiquitous language - the language of the domain used inside the domain - and the published language - the language/interfaces used to communicate across domain boundaries. In data mesh, data products should be in the published language to make them useful outside of the domain.
DDD helps you to learn more about the business domain and provides patterns to help design software systems. But it's 'biggest benefit' is that it provides a bridge between the business needs of the domain and the software systems.
The reason organizations exist is not to ship software. They exist to execute on a goal - typically driving shareholder value - and software is merely a vehicle to that. Think about data the same way.
A key aspect of domain driven design is communication across boundaries, whether machine or human to human. So finding and defining those boundaries becomes key to prevent - or at least manage - business friction.
In DDD, it's important to not mix up domains and business capabilities. Domains are about finding coherent sets of use cases for software systems. Business capabilities is the hierarchical structure many think of when speaking about domains.
A model is a cornerstone concept in DDD - it's how we look to solve a problem in software rather than try to reflect the real world in software.
It's crucial in DDD to get the software engineers to speak in the language of the domain - the 'ubiquitous language' - in order to design systems to appropriately solve the business problem. In data mesh, we need to learn to embed the business context in the data we share in a similar approach of finding a language of the domain but balance that with the published language.
In DDD, it's okay to have one implementation shared via multiple published languages, especially as different versions when the implementation model changes. We have to figure out how that versioning works with data, especially when the change is outside the hands of the producing domain - e.g. purchasing external data sets that then change.
In DDD, it's okay to have more specific integration contracts for implementations for the job at hand instead of one "jack of all trades" integration contract. In data, this could be multiple views of the same general information instead of a single data model to share all aspects.
To get to a good published language in DDD, ask the people consuming from your domain, the counterparty in your integration contract. It makes sense in data mesh too to ask people what they want. Take that feedback and look for general purpose ways of structuring and sharing data but you have partners, you don't have to design everything yourself.
The anti-corruption layer in DDD is important - it is created by someone consuming from other domains and is a protection against ineffective models - specifically something that would contradict their internal model. In data mesh, this is made all the more important if we don't have shared taxonomies.
Pain isn't necessarily bad. In our bodies, it's a signal something is wrong. In software, friction/pain signals are often ignored but shouldn't be. In data, we need to get WAY more explicit about when pain comes up and where it's actually coming from.

Vlad started off with a simple Domain Driven Design (DDD) definition: "Software design should be a function of the system's business domain." But it once took him writing a 90 page book to really unpack what that meant. Domain driven design feels simple until you start to really dive deep and you realize 5m meant five miles, not five meters.

So why do we care about DDD? Per Vlad, DDD helps you learn more about your business domain, identify its requirements and strategic needs, and identify what makes a software system you are developing valuable. DDD also provides a set of design patterns and architectural styles to design your software solutions. Putting those two together, it bridges the divide between the business requirements and the actual design of the software solutions so your software solutions are designed to explicitly meet the direct needs of the business rather than through layers of abstractions and miscommunication.

For Vlad, the term domain is very complicated because it means a very specific thing in domain driven design but is often used interchangeably to mean what DDD would potentially call a subdomain but probably more correctly a bounded context. As an example, in episode 133, Ammara Gafoor mentioned her client has 21 domains across 100K+ people but in episode 130 with JGP, he used 'domain' to mean a team of 3-6 people. So getting specific around what the term domain actually means is important. In DDD, it's also important to note that subdomains will not cover all aspects of a domain because certain parts do not have software systems covering them. A key aspect of DDD is finding your boundaries between the subdomains which are essentially sets of interrelated use cases.

A business capabilities model is actually what most people are thinking of when discussing DDD subdomains according to Vlad. He noted in an email "The business capabilities model is a more precise way to map a business domain than DDD's subdomains. It gives much more insight, though usually the subdomains model provides just enough insight needed for designing software systems." This is where you can have a hierarchy so Marketing may be a tier 1 business capability and that might be broken down into Digital Marketing versus Brand Marketing versus etc. And then Digital Marketing might be broken into Lead Generation and Demand Generation. Or by Paid versus Organic. Or many other ways.

So, DDD is again about finding the boundaries between your use cases or a coherent set of use cases that work together. Those boundaries are used to drive the software system design decisions, how things work together, what architecture to use, etc.

Vlad believes the concept of a domain in software is very different from the domain in data mesh. Essentially, when attempting to do DDD for data, we misuse the term domain to mean line of business or business domain. In DDD, the boundaries that the team can decide on is how the software components work together across boundaries but they cannot really change the business capability boundaries or the subdomain boundaries. But in data, you have the ability to actually change the business capability boundaries and how things are grouped into bounded contexts, to impact the flows instead of simply identifying them.

We can't really reflect the real world perfectly in software - or even all that closely - according to Vlad. So instead, DDD uses the concept of a model, where the model is designed to solve a problem. The model should contain the minimum knowledge to solve that business problem - this significantly limits scope, which means preventing needless complexity.

A key aspect of DDD - which translates extremely well to DDD for data / data mesh - is finding the ubiquitous language. You must get to a level of communication where the software engineers can understand the domain well enough to speak in the language of the domain to the subject matter experts (SMEs) - typically referred to simply as 'the business'. Scott note: In data mesh, data product developers will need to speak in the ubiquitous language of the domain to understand how to share the domain's context with the rest of the organization. It remains to be seen how much data products should be specifically in the language of the domain versus the language of the broader organization AKA the published language in DDD.

Vlad discussed how important iteration is in DDD - it's not about getting it perfect the first time, it's about getting something out there and improving upon it as you learn more and as the world and domains/subdomains evolve. This applies well in data mesh because iteration and open sharing are crucial to getting to a data driven culture. However, in DDD, evolution of models creates friction. The way the systems were integrated is now changing so that is why there is a published language - essentially the integration language for how domains/subdomains communicate across boundaries. This is decoupled from the business systems so the business systems within the domain can evolve without it impacting how others interact with the domain. In data, the data warehouse has typically been quite tightly coupled so changes are extremely painful or in the case of the data lake, it is often raw data with no real published language or version controlling. This is a big friction point data mesh aims to address but exactly how companies address it will be varied and we are still figuring it out as a collective industry.

In DDD, Vlad doesn't like the concept of a single integration contract for all purposes - this is that mythical unicorn type approach where it's good enough for every scenario. Instead, he recommends communicating with the other party in your integration contract to find what works for them. It means less work for the producer and the consumer 1) has a say so more buy-in and 2) gets what they want. Of course, this can mean more overhead so look to balance that. For data mesh, it remains to be seen if this will create too much overhead - a big issue in data is overly specifically engineered solutions leading to little reuse so we must balance usability for your key consumers with usability by all while considering overhead.

Vlad then went into some of the patterns for actually sharing information in DDD if you'd like to dig deeper.

An anti-corruption layer in DDD provides protection for domains consuming information from potentially ineffective models to your use case according to Vlad. That doesn't mean the integration contracts themselves are ineffective but it might mean that some terminology difference - we always use the example of what does customer mean - that would invalidate some of the work you are doing because it contradicts your own internal model/use case. It's crucial to understand how to implement these protections for data consumers but in data mesh, data consumers should also make sure producers understand when things are ineffective or counter to their internal model. It doesn't mean things will necessarily change but it makes the friction point public/explicit, resulting in better understanding at the least.

For Vlad, pain in software can be good. It helps us to at least identify where there is friction and try to address it. But many teams try to ignore pain in software. It's a signal your boundaries are ineffective. That can be simply because things have changed and they were effective before, not a sign you had it wrong. But being on the lookout for where change will be effective and value-add is crucial to doing DDD right.

It's important to keep an eye on what we are sharing by thinking about what could be more effective as well in DDD according to Vlad. As we learn more about a domain, or as the real world changes, we can share better information. So change can be a value-add. We obviously want to balance that with constant change but if you can more effectively share information, do it. We have written language now, we didn't 15K years ago. So we are far better at sharing information. When you develop a better way, look to implement it.

Vlad finished by saying it's okay to dip your toe in the water relative to DDD. You don't have to learn everything and have an all or nothing approach. You can take parts of it and slowly adapt your ways of working. Important for data mesh too - you can take pieces and it's not going to be perfect, just keep making progress and don't delude yourself of the work yet to be done.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#170 When Should You Reorganize for Data Mesh - Mesh Musings 38 Dec 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#169 Sharpening Your Competitive Advantage With Data - The Solution Is Not Simple - Interview w/ Alexa Westlake Dec 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Alexa's LinkedIn: https://www.linkedin.com/in/alexandrawestlake/

Alexa's Medium: https://medium.com/@westlakealexa

In this episode, Scott interviewed Alexa Westlake, Senior Data Analyst at Okta. To be clear though, she was only representing her own views.

Some key takeaways/thoughts from Alexa's point of view:

Data can be transformational but it is expensive to do the work. "Without literacy, all your analytics is is expensive." So do the data literacy work to make your data work actually valuable.
?Controversial?: Don't focus your transformation initiatives around a negative, focus on a goal/aspiration. It is hard to maintain momentum around pain, especially as it starts to ease with early wins. While pain points can pique attention, shared goals and collective outcomes will keep people bought in and motivated.
?Controversial?: Data is "not going to make or break you [in every case], it is there to help you be better, it is there to help you unlock your full potential." Use it to sharpen your competitive advantage.
As you scale your organization, if you do not prioritize data - the way you manage the people, processes, and tech around data - you will generate a ton of friction. It will create a "feedback loop of pain."
"Never jump and hope." You need to make sure you have the support to get your initiatives going and then maintain momentum.
!Important!: Data work is often not the number one priority for the stakeholders you serve. Understand that and keep close enough to make sure you are working on analytics to support their top priorities.
Alignment is one of the hardest things to do organizationally and is far more crucial in data work than most believe. Be prepared to repeat yourself - repeatedly.
The sunk cost fallacy very often results in throwing good money after bad in data, especially platform work. Look for signs you need to shift when the investment rises and the return continues to fall.
People, especially in analytics related initiatives, often don't know exactly 'where the pain is coming from'. Work with them to understand their pain but that the data team and/or data work is not some magic wand that is waved and it's all better. It's tough but valuable work.
It can be hard to get executive sponsorship to generate new data explicitly for analytics, but not doing that results in only analyzing the information you already have. How can we incorporate information capture into part of the business processes? What data is valuable - and reasonable/ethical - to generate?
Get more specific when talking about data quality. "The data is bad" is essentially meaningless. Really drive towards specifics and then get people on the same page on measurement/monitoring of quality.
It's easy to focus on the data you have instead of the insights you can generate. When talking to business stakeholders, how often do they care about the data versus what it means? Focus on communicating in what matters, not the speeds and feeds aspects of data.
A lot of data tech debt / challenges in data come from decision makers not understanding their internal data ecosystems. Adding urgency to that often results in band-aid solutions that degrade quickly.
You can't turn the ship too quickly. Trying to bet all your data transformation on one aspect or a big bang won't work. Look for small wins to build momentum.
It's pretty easy to burn out your data team - make sure to give them interesting, important, and valued work.
Communicate with the business about what matters. As a data person, that usually means more listening than speaking so you can figure out what matters to them. Applicable Scott phrase (to vendors): no one cares what you want to tell them, they care about what they want to hear about.

When asking people generally what do they think of data, Alexa has seen many fall into one of two camps: either data is transformational or data is expensive. And in truth, both are probably right. There is massive investment in data but most organizations are still struggling to scale.

"The data is bad" is such a common refrain across the industry but Alexa believes that is like data without context - essentially meaningless. What aspect is bad? What makes it bad versus good? Why is it bad? Do we need better collection processes in place? Do we need more expertise in transforming and analyzing data? And when answering those questions, it's often very difficult to figure out what are the overarching problems that you can tackle instead of building point solutions.

Alexa has seen it can be a bit of a scary proposition to try to get exec sponsorship to generate new data specifically for the purpose of analytics. So most companies start with what data they have today, and don't get overly ambitious, at least until they are proving they can deal with what they have now well. But, at the same time, many fall victim to the sunk cost fallacy of 'we've spent a bunch on this platform already, we have to focus on scaling it out', often throwing good money after bad. You can't just constantly change your platform but ignoring problems simply because the decision was already made is a recipe for disaster.

According to Alexa, a lot of the challenges in data come from the decision makers not really understanding their internal data ecosystems so we need to make it easier for execs to make better decisions. That ML model might have 40+ pipelines in some form or fashion that feed into it, of course it's likely to degrade. And there is often an urgency to solve the problem of today with a solution that addresses that challenge today for that specific use case - fighting the symptoms instead of tackling the cause of issues.

While something like data mesh - or any other large scale data transformation initiative - is a big change, Alexa believes we shouldn't make that a giant leap instead of small steps. You have to get to small wins as you're turning the ship to keep earning the right to steer the ship. And you shouldn't revolve the entire transformation around a negative, a pain point. If you do, you'll end up focusing on the pain too much instead of the goal - it's hard to sustain focus and drive around pain. Look to focus on the motivation behind why you are doing data transformation work but also what are the incremental results.

A few ways to burnout your data team Alexa mentioned: not providing them work they feel is meaningful; putting low priority on data work in general but especially on the interesting insight generation; data not being part of the critical path to business success. Essentially, if people aren't working on interesting, important, and valued work, they will leave.

Alexa believes it's crucial to focus on your communication when working with the business - they aren't well versed in data terminology or often even data concepts. So focus on communicating to them what matters and why instead of using data jargon. Most decision makers make so many decisions across many contexts, work to make it as easy as possible for them.

"Never jump and hope," Alexa said about measuring and maintaining momentum. To do data work right, you can't be an order-taker - taking the requirements, going away to build, and then presenting the 'thing' at the end and it's done. Get closer to the decision makers, understand what their expectations and needs are. Are they shifting? Are they expecting too much? Have the constant flow of information bi-directionally to make sure you are still doing valuable - and valued! - work.

There is often a lot of pain around data for business stakeholders but they typically can't directly identify the exact source of the pain in Alexa's experience. Collaborate with them to figure out the actual pain so you aren't treating symptoms. And work with them to explain that just like in medicine, exercise, diet, etc., there isn't a miracle cure or improvement. Data work takes time. Explain why it will take time and drive to what actually matters to address. The common example is 'what does real-time mean for you and what value does real-time drive?' It's often '2 hours is fine, just sick of 24 hour delay.'

For Alexa, it's very easy to try to build out governance centrally because it maintains a feeling - if not a reality - of control. The reaction of most humans to the unknown is fear. But you can make slow improvements to build trust and momentum - as Laura Madsen also talked about in her episode. If you don't invest the time to empower and enable your employees around data, you won't get good returns from it.

To do any data work, but especially something like data mesh, right you need champions in the domains to help. Both to help you move things forward but also that constant communication loop as the world - and thus requirements - changes. Focus on them being your partner and treating them like a customer of your product.

It's very easy to fall into the trap of putting too much emphasis on the data work, according to Alexa. It is not likely to make or break your organization but it can be a significant competitive advantage. Data can unlock your value potential and sharpen your competitive advantage.

Quick tidbits:

"Without literacy, all your analytics is is expensive."

Alignment is one of the hardest things to do organizationally and is far more crucial in data work than most believe. Be prepared to repeat yourself over and over.

It's easy - and often fun - for data people to focus on the raw data you have instead of the insights you can generate. When talking to business stakeholders, how often do they care about the data itself versus what it means? Focus on communicating in what matters, not the speeds and feeds aspects of data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 18, 2022 Dec 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#168 Zhamak's Corner 12 - Your Data Is NOT a Cake - The Dangers of Layers Dec 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

What tech is already available that could be used for data mesh? There are so many amazing approaches and technologies in data but they've been used for the pipeline approach only. We need to think more like developers - not accepting the grunt work or death by a thousand cuts of data - and take a hard look at what we've done historically in data and what should be replaced.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#167 1 Year Anniversary - What More Have We Learned So Far? - Mesh Musings 37 Dec 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#166 Capital One's Data Mesh Journey and How They Created a New Product Along This Journey - Interview w/ Salim Syed Dec 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Salim's LinkedIn: https://www.linkedin.com/in/salim-syed-11981521/

Capital One Software: https://www.capitalone.com/software/

Capital One Slingshot: https://www.capitalone.com/software/solutions/

Capital One blog post on their use of Snowflake for data mesh: https://www.capitalone.com/software/blog/operationalizing-data-mesh/

In this episode, Scott interviewed Salim Syed, VP of Engineering at Capital One Software.

Some key takeaways/thoughts from Salim's point of view:

!Contrarian!: "Data mesh works with central policy, central tooling, but federated ownership." Work to federate your infrastructure ownership, not just data ownership.
No matter what size of domain or LoB, look to have the same responsibilities owned by each domain. They may fall under different people but the role they serve should look the same across the domains.
Get your roles and responsibilities established first, then look to start tackling other challenges.
Do your best to hide the data engineering complexity from your self-serve platform users. Help them do their job, not learn data engineering. Focus on creating a great experience for them in their workflows.
Think about data risk from an opportunity standpoint: if you have strong risk controls, you can feel more comfortable giving more people wider access - you know what is allowed so you can potentially give them access to more data than if you didn't have tight controls.
Doing federated anything right isn't about just giving people the tools/patterns/policies, you should focus on a usability layer. Should every team have to integrate and manage their data tools or should they be able to focus on doing what matters? What is their experience?
Design your data platform to your personas - who's getting something done and how do they do their work? Also, always ensure your infrastructure and your governance are in sync. No one likes to manually update the data catalog or even worse when it drifts.
Discoverable data doesn't have to mean everyone can access everything by default or that access is automatically granted. If people can find data and understand what the data is about plus then easily request access, that is enough. Also make granting access as painless and low risk as possible.
Look at users' most common actions that cause major friction and focus on tackling those, whether they 'feel like part of a data platform' or not. For Capital One a few were reviewing/monitoring risk and supporting manual updates of production data.
Cost controls are a crucial but often overlooked or ignored aspect of your governance - it's VERY easy to overspend in the cloud :)
!Commercial Offering!: Capital One launched Capital One Software and its first product, Slingshot, after seeing how difficult it was to properly manage costs around data mesh, especially given the separation of storage and compute with Snowflake. Lines of business don't typically have cloud cost managers so they wanted to automate the controls and cost tracking/management/forecasting as much as possible.
Don't make people in the business learn exactly how to tune to prevent or rectify cost inefficiencies. Again, back to experience, give them knobs and the right questions to assess their needs so they don't have to be an expert.
Cloud bill surprises are so common, it's a meme. Build out tooling to help people forecast their cost and then track and alert against it. Slingshot helps Capital One teams forecast and control data costs. Don't wait until the end of the month to let someone know they went 50% over budget.
?Contrarian?: Your central team will likely be reluctant to give up some authority, especially in deciding how infrastructure is deployed. Look to retrain central teams to build self-service capabilities that create the guardrails to give them comfort but still federate infrastructure ownership to the lines of business.

Capital One is on its own data mesh journey and what they learned and built internally inspired them to build an offering called Slingshot. When asked about where he would tell others to start their own data mesh journey, Salim mentioned that we can't solve our data issues - whether data mesh or anything else - through just technology. At Capital One, they started on the organizational aspects, breaking into discrete lines of business (LoB) and then creating units of data responsibility with a hierarchy. However, the data hierarchy wasn't the same in each, they left it up to LoB to determine - a large domain having 3-4 and a small domain having 1.

According to Salim, a big reason why they have so many people focused on risk is similar to what Sarita Bakst mentioned in episode 52: you want to give people as much access to data as you can while minimizing risk. So set yourself up to give people access when it's valuable / necessary for their job. There are even instances at Capital One where you need to complete training to get access to data. And they have active risk monitoring constantly in place as well to make sure they don't miss anything. Again, doing that gives you more piece of mind to leverage more data.

When you start federating your data ownership, you can't only give people the tooling and the authority, that won't work well in Salim's view. So, in addition to all of that, you need to focus on a usability layer. Think about email - it's pretty easy to work with an email client but what if you had to put all the plumbing and add your headers and all that yourself. Just giving people tools, patterns, policies, etc. and expecting them to be able to handle it all isn't realistic, make it easy to leverage, think about their user experience.

As an example, Salim talked about the process for creating a data product. Instead of interfacing directly with 6+ tools, there is a workflow with automated processes to make it a smooth process. As many past guests have noted, reducing friction to sharing data is a key element of driving value from data mesh. It's not just called a data platform, it's called a self-serve data platform for a reason :)

Salim shared some advice other recent guests have touched on: really focus on the job to be done, who is doing it, and ensuring your infrastructure and governance stay in sync. Focus on the job to be done, whatever that job is, instead of the tools. Again, reduce friction. Focusing on the persona of who is doing the work is also crucial - Audun Fauchald Strand and Gøran Berntsen from NAV in episode 37 talked about building a great data platform no software engineer would want to use. Use product thinking, who is using it and how do they typically do their work? And lastly, ensuring your governance and infrastructure stay in sync - if there is an update to the data product, does that automatically update the data catalog? How do you prevent drift between systems or kludgy manual fixes?

Data discovery done right is about a few things according to Salim. Ensure people can find relevant data easily is baseline but also get them to as much understanding as possible. Even sensitive data, what are the quality metrics, the general data shape, etc.? And make it easy to immediately request access when you find data you want to use which immediately triggers a request to the data owner with a business justification. And the relevant policies for that data product are automatically part of the approval process so the data owner doesn't have to remember the policies themselves. Again, reduce friction to getting the job done for all parties.

Salim talked about a few things they overlooked at the start, one for personas and one that was causing a lot of friction. As mentioned earlier, at Capital One there are a number of risk managers but the early iterations of the platform didn't cater to their experience. Which meant access requests were delayed and risk monitoring was tougher to do. So make sure to consider all your personas that will be using your data mesh - ignore at your business value peril. The other aspect was how much manual effort was involved in patching production data so they addressed that as well.

To be able to federate the actual infrastructure management to the domains, Salim and team knew they couldn't just hand over the tools. Again, the personas in the LoBs wouldn't have the expertise to manage the infrastructure. So they focused on exactly what Salim mentioned throughout: the experience. How could they empower the lines of business to own their data infrastructure without the domains having to manage their data infrastructure? So the team built out a platform with capabilities with experience at the core but with additional aspects like DBA best practices, guardrails, and cost management as part of the platform.

Scott Note: from here forward, there will be some discussion of Capital One's Slingshot offering. This is not an endorsement of the product at all but it is interesting and germane to the conversation (read: Salim was not selling, only telling, which is a-okay on the podcast). Cloud cost is near and dear to my heart and it's important - whether you build or buy - to not overlook cost management.

So, all these challenges of how they addressed their cloud cost management via their platform led them to believe there was a market for this type of solution per Salim. Capital One has a history of creating cloud cost tooling as they were the creators of Cloud Custodian (https://cloudcustodian.io/). Scott Note: unpredictable and/or high costs have been a major concern/pushback to data mesh since early 2021.

One general issue with on-demand / cloud computing is cost inefficiencies. There is always a lot of waste and it's often actually more cost effective to ignore than chase it down unless you know where inefficiencies lie. So Salim and team found it useful to not try to automatically clean up cost inefficiencies - that pretty much never works - but to highlight them relatively quickly and offer potential recommendations and/or help. And they lowered their own Snowflake costs quite a bit in the process.

The bigger benefit according to Salim was the team put proactive questions in place for when teams were provisioning their data infrastructure. Often, it can only take a few minutes to save a large percentage of money if only people know the knobs to turn. But you don't want everyone to have to be an expert - extract the information from them based on their needs and create a recommendation system. This is just yet more on the experience side - don't build cloud cost experts in every LoB, make it so they can make the right decisions as often as possible quickly and easily. You should also look to build in cost forecasting tools as part of your experience so people aren't hit with a surprise bill - the surprise Cloud bill is so common, it's a meme on Twitter.

From what Salim is seeing, most companies - or more correctly, central data/platform teams - are pretty reluctant to federate ownership of the infrastructure to domains. He believes that is because of things the LoBs don't understand like cost controls and best practices but that if you allow the central team to set guardrails and best practices, they will be more willing to give up control. Remains to be seen.

Salim finished with "in our experience, data mesh works with central policy, central tooling, but federated ownership."

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 11, 2022 Dec 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#165 Getting Your Data Mesh Implementation on Firm Footing - Early Learnings from SMG's Data Mesh Journey - Interview w/ Amy Raygada Dec 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Amy's LinkedIn: https://www.linkedin.com/in/amy-raygada/

In this episode, Scott interviewed Amy Raygada, Senior Director and Analytics Product Manager at Swiss Marketplace Group (SMG).

Some key takeaways/thoughts from Amy's point of view:

When signing up to do data mesh, you won't exactly know what you are signing up for - it will be different than your expectations in a number of ways so be prepared to be flexible. And it's a marathon, not a sprint. Have patience, you don't need to get it perfect on day one.
It's crucial to have a data product owner, someone who can ask the uncomfortable questions and make sure you are actually treating your data as a product. It's easy to get lost in focusing on the technical but that's not what drives the business value.
Look to start with a reasonable scope use case instead of a massive challenge. Make it so the teams can learn and iterate because you aren't solving the hardest challenges - yet.
Consider starting with a single domain and with a key consumer being that same producing domain. As they learn more about owning data and see value from owning their data, they will be better able to serve other teams too.
Spend the necessary time ideating and brain storming before beginning your journey. You want everyone to be working together so a clear initial vision - at least for first steps - is key. But keep updating your prioritizations as you learn more.
Data mesh requires major changes for organizations. People aren't good with very abrupt changes so look for the incremental changes where viable. It's okay to take a bit longer if you maintain people's happiness instead of trying to rush to the finish and burn everyone out.
Domains need help getting to data capable, being able to own their own data. You can't just shove ownership on them, you have to onboard them, help them understand how to work with data. You will definitely need to "babysit" them as they learn and that's expected and okay.
You will probably communicate more than you expect even though everyone tells you to expect this :) pair well and make sure people understand what is to be done and why it's crucial.
It might be hard at first but a central data team should be focused on enabling domains to own their data. There might be some desire to do the work for them but then you end up with central data ownership again.
Data contracts are relatively easy to understand conceptually and give a scope to what data owners must adhere to instead of something like "give me high quality data". And the data owners can set better boundaries/SLAs or push back on requirements more easily.
When working with a domain, make sure they understand the reason you are asking them to take on new types of work - what's the outcome? Show them what happens when they break a data contract so they can understand the impact. And build a good relationship so you can use them as your internal success story too.
Make schema contract validation - before deploying changes - as simple as possible so software engineers can check if changes will break data contracts. If yes and the change is necessary, then set a versioning strategy for that data product.
If you set teams up to understand the why of data mesh first, then what changes for them, it leads to a much easier conversation. Talk about what is the current state and why that's not where you want to be.
Leverage your first movers to be your advocates. Once you have one success story, pair with them to bring other people on board. The water is fine, jump in!
If a producing domain - or stakeholders in the domain - is hesitant to engage on something like data mesh, work with them to show initial value. Once they see it has a benefit and isn't some monumental task - and that you are there to help - they are much more willing to participate.
Look to cross bridges when you come to them. Obviously plan ahead but don't get too wrapped up in what might happen.

Amy started by talking about something many other guests have probably felt but few have said: when signing up to do data mesh, you won't really know for sure what it will be. And that's okay, your journey will take you places you didn't expect and have hurdles and obstacles you can't see or predict. But that's all okay, you can learn and iterate along the way. Expect the unexpected.

When Amy started interviewing at SMG, the team was not as familiar with data mesh. But for the past six months it's been a key part of her focus. She paired up with the head of data engineering and worked to brainstorm before moving forward - that pre-work lasted about 2-3 months. There was also a lot of other change happening in the general technology and data landscape at SMG - moving from on-prem to cloud, moving from monolith to microservices, moving off legacy technology, etc. - but that meant they were able to get everyone together for a 2 day workshop and really look at things from a fresh perspective. At the same point, people are not used to abrupt changes and with data mesh, there will be a LOT of changes - look to implement that over time, don't be in a rush.

SMG decided to start with a single domain - leads - instead of multiple domains. The initial use cases were useful for the leads domain as well, especially by significantly improving data quality. As part of enabling that domain, Amy and team are working closely with them to teach how to handle data, what data ownership actually means.

The leads domain was chosen because they had a significant need for help with data quality and had moved more to cloud and microservices than other domains. It was also a smaller, more manageable problem than say sales, which is in a major transition to Salesforce. They had three non-mesh data products that were impacted from poor leads data quality so there was a lot of downstream issues that they could address, a lot of incremental business value to drive by fixing the data quality issue. The other domains they considered were in big transitions so it would be harder to get where was necessary to drive value in a PoC with a lot more work and risk.

Data contracts have been a key goal and key driver for Amy and team. The goal is to better define what you are trying to do with data, what do you as a data owner need to actually do and deliver and what can the data consumer expect. The driver aspect is that data owners have something more concrete so they are willing because they only have to deliver what they say they will. It gives a limited scope to their data work.

A focus for Amy and team is to be the enablers only, building out the platform and teaching people how to own but not do/own as central data ownership was causing issues in the first place - it's what data mesh specifically moves us away from. The domain teams will certainly need "babysitting" but that is expected and can mean more information flowing to the platform team to make improvements too. Data ownership isn't a one or a zero, it's a process.

Amy believes it's important to really pair with domains to share the logic with them behind data mesh - why are we doing this change - and not make it feel like you are changing their ways of working instead of adding new, value-add capabilities to the domain. This close relationship has allowed them to do a data contract demo where you can show the domain what happens when they make a change that will violate their contract. That way, they understand what happens to downstream consumers - possibly themselves - when they make a breaking change. And that the platform alerts them to a breaking change too so they have a better chance of preventing issues themselves.

Similar to what Chris Riccomini mentioned in episode 51, Amy and team are implementing automated schema validation checking at the pull request level. This prevents breaking changes from going through with consumers being the first to know about an issue. It also kicks off a conversation about should this change be made and if so, how will they do versioning. And Amy knows this can overwhelm some people but the team she is working with understands the pain so they are eager to prevent that pain. They are also looking at data reviews - similar to architecture reviews - to assess if operational system changes will impact the data. Abhi Sivasailam in episode 9 mentioned they are doing a similar process.

Amy believes - and Scott agrees - patience is crucial. Getting the first domain into a really good spot and then enabling them to share their story and their learnings will be crucial when they try to go to additional domains. Not being in a massive hurry means teams have the time and space to learn how to own data instead of piling a huge additional workload on top of an overburdened team.

Educating the general company about what they are doing with data mesh has gone well. Amy created a Miro board using Barr Moses' old joke of data mess to data mesh. So they are working to explain the what and the why to everyone involved but in a simplified way. Talk about what changes - the new responsibilities and what those mean and drive. Talk about how you can bring everyone to the table and especially what benefits data mesh has for them. Really focus on the practical of what are they being asked to do and why.

While many stakeholders in their initial domain were anxious to engage at first, now that there is proven value, according to Amy those hesitant stakeholders are much more willing to pair up. They are providing test cases to the data team so they can quickly validate value and iterate together. They are already seeing the benefit of the work with other stakeholders and it's getting those previously hesitant stakeholders excited to team up. So if you want buy-in, look to provide some value first and then show/prove that value. Yes, easier said than done.

Quick tidbits:

Make sure to team up with any domains that have had success so they can help you sell other domains on working with you. They can help educate and also show that you are actually providing the business value you claim.

It's super easy to get bogged down by metrics. Push back on big metrics requests - why do you actually need this? As Alla Hale mentioned in episode 122: "What would having this unlock for you?" If it doesn't unlock value, why do it?

Having prioritization meetings weekly keeps everyone on the same page, heading in the same direction.

It's easy to get wrapped up in what might happen. Focus more on what's in front of you and what you are trying to do. Don't cross bridges before you come to them :) there are countless bridges in data mesh, focus more on the now.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#164 Zhamak's Corner 11 - Thinning the Veil Between the Data and Operational Universes Dec 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

What do we need to develop to provide a better experience for data product developers and consumers? How can we make it easy for consumers to move from discover to trust to lean to use? We still have a long long way to go with analytical APIs, they are basically only for sharing raw data at the moment - how can we better share the embedded information? And how can we give data product developers the ability to do the right thing by default where we optimize for ease of use and also for more generalist developers to use - instead of hyper-specialized tools experts? There have been two very separate universes of operational and analytical planes - how do we change that and leverage software engineering practices in analytics?

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#163 Improving the User Experience for All Parties - Early UX Learnings from Data Mesh at DNB - Interview w/ Alice Parker Dec 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Alice's LinkedIn: https://www.linkedin.com/in/aliceparker/

No Silver Bullet: Essence and Accident in Software Engineering by Fred Brooks: https://www.cgl.ucsf.edu/Outreach/pc204/NoSilverBullet.html

IBM Research paper mentioned: https://dl.acm.org/doi/10.1145/3290605.3300356

Microsoft Research paper mentioned: https://dl.acm.org/doi/10.1145/2884781.2884783

In this episode, Scott interviewed Alice Parker, Data Engineer at DNB.

Some key takeaways/thoughts from Alice's point of view:

It's easy for people to confuse user experience (UX) and user interface (UI). But UX is far deeper than most understand. We need to design systems and experiences that make working with data - as a producer or a consumer - far easier and more delightful.
People are very willing to talk about their challenges - show some empathy and give them the space to talk about what is holding them back and what they could do if you worked with them to address those challenges.
Data consumers need three major things to work well with data: 1) domain expertise, 2) time, and 3) to be able to "converse" with their data.
Ensure your data quanta - or really any aspect of your data mesh implementation - are documented for all your user personas. There may be different needs for each persona type. A data scientist probably doesn't need as detailed of explanation of lineage if they can see the transformations compared to a business analyst.
As part of designing our UX, we need to focus on "how to help users achieve their goals effectively, efficiently, and with satisfaction, how you can minimize risks, and how you can enhance the maintenance of tasks to be completed effectively."
To get UX right, you have to understand "the limitations of the humans using it." Because you have to design your experience and your risk tolerances around those limitations.
"Systems and technologies evolve incredibly quickly, but unfortunately, humans don't." Have empathy for that. The human brain’s processing power does not evolve at the same exponential rate as Moore’s Law, so take into account the limitations of cognitive load.
It's easy to fall into the trap of building one-size-fits-all experiences. But in data mesh, especially as we raise people's data literacy, their data capabilities, we need to make sure we understand persona needs and desires so we can design for them.
Interviews are crucial to understand what your personas actually need. Speak with them, reflect back their pain, understand what they are trying to accomplish, and then work to build something that addresses their needs without being tied to any one use case, domain, or person.
Your user experience will change and - hopefully - improve. Communicate with your users what you are doing and why, bring them in to the conversation. It's important that they understand you don't have a magic wand but you do have a sympathetic ear :)
Prioritization is key to delivering on an improving user experience in data mesh. There will always be a deluge of requests, figure out what are actual requirements instead of nice-to-haves and then assess your capability to deliver and the cost/benefit of delivering. Then communicate your prioritization.

Alice started by talking about her recent Master's thesis which was studying human computer interaction, specifically around data mesh at DNB. And it's a huge topic. To start though, she emphasized the difference between user experience (UX) and user interface (UI). Your experience is impacted by much more than just the visual buttons you click - or knobs and levers you move if on a more industrial setup - so user experience is much deeper than many think. Zhamak has mentioned user experience - in multiple contexts - as a crucial factor in getting data mesh right. It's a rarely explored topic in data, especially designing your architecture to provide a better experience for data producers and consumers.

According to Alice, there is even an ISO standard focused on experience. It's about "how to help users achieve their goals effectively, efficiently, and with satisfaction, how you can minimize risks, and how you can enhance the maintenance of tasks to be completed effectively." And breaking that down into each piece can be a pretty deep conversation. How do we ensure satisfaction of data users? What is risk when it comes to data? Risk of interpretation? Misuse? A crucial aspect of UX is understanding the capabilities and limitations of the people using it.

As Alice noted, "systems and technologies evolve incredibly quickly, but unfortunately, humans don't." And we can't think of every person as being the same with the same needs and capabilities. Which unfortunately means, one size will not fit all when thinking about building your platforms for data mesh. We have to design to serve persona needs. And to actually understand persona needs, we need to speak with them.

When listing out the different personas, just on the data consumer side Alice mentioned data analysts, data engineers, data scientists, data stewards, and business owners. On the producer side you have data engineers, data scientists, software engineers, and business owners. Then in kind of the in-between, other personas you have platform engineers, data governance people, etc. So you have to think about what each persona needs and then design for each persona. The personas even have different terminology, different ontologies they use. If only this were that easy :)

What this all leads to is going and actually talking to your potential users to find their needs - which is what Alice did with interviewing data consumers for her Master's thesis. Similar to what Jen Tedrow mentioned in episode 98, you need to go and listen to their pain points and then abstract away the use cases to see what are the bigger needs. And you can do that in a very informal way too. But you can really only get that feedback through conversation and explicit effort. Scott Note: this is EXTREMELY true… the number of times I've asked for feedback and gotten crickets is… yeah…

Alice noted it's important to let people know when their requirements won't be met, or won't be met on their timeline. That open communication will get them to trust you. It's okay to say no to a change to user experience, especially if you have good reason for it that you explicitly communicate to them. You can work with them to maybe find the quick wins that gets them additional value now. Prioritize what can be done now and explain why for your prioritizations.

Documentation is one crucial aspect that has been lacking in data according to Alice. Yes, data mesh calls for data quanta to be well documented but we need to ask who are we actually documenting for? If you have 5 different consumer personas for your data product, is it documented so all of them can actually use it? And then how do we make it as easy as possible for data producers to actually create and update that documentation? Does documentation only have to be written? What are some low friction ways to share the context of what a data quantum is all about?

It's crucial to think about incremental progress - and showing that incremental progress - on your user experience in Alice's experience. Every system everywhere will have frustrated users. Such is life. And you can't solve most challenges in a day. But look for ways to iterate towards a better UX and circle back and show people you are improving it - if that's data product creation cycle time, show them your prioritizations that sped up that cycle time and show them the improvements, show them you listened and reacted. They might still be grumpy but at least they have reasons to be less so.

As other guests have noted, Alice has seen most people are very willing to share about their current challenges. If you go with the right attitude - to listen and empathize - you can learn a ton. People want to feel seen and heard. And it might help more with your prioritization than you'd expect. As Alla Hale said in episode 122: "what would having this unlock for you?"

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of December 4, 2022 Dec 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

{Special} Scott's 2023 Predictions - Mesh Musings 37.1 Dec 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#162 Creating Data FOMO and Keeping Close to the Business - Interview w/ Dacil Hernandez Dec 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Dacil's LinkedIn: https://www.linkedin.com/in/daciluhernandez/

In this episode, Scott interviewed Dacil Hernandez, Director of Data and AI for Northwest Europe at Nagarro.

Some key takeaways/thoughts from Dacil's point of view:

At the end of the day, tech is the easy part in data. Creating value though data is the hard part. And why do data work if not to create value?
It's incredibly easy and pretty common for IT and the business to get disconnected - or never be connected in the first place. Focus on creating and maintaining relationships with a steady flow of context between both sides.
Dig into if your business partners actually understand what data ownership means, what it entails. They may be willing but not capable to own data at the start. Work with them to up their capabilities and understanding.
Data ownership should not be treated like the "hot potato", being passed to "anyone other than me".
Tell your business counterparts "I need your help to help you." You can unlock far more value through collaboration than waiting for requests.
Your data strategy should give people FOMO - fear of missing out. Give them incentives that makes it feel like they can't miss out on the value you're creating.
!Interesting Idea!: use gamification to find data quality issues. Don't make it a shameful thing, find it and get it remediated and celebrate the people who found it. Look to drive positive energy around your data.
Talk to potential data consumers before creating a data quantum or anything in data. You won't know what you can offer that will be valuable to them until you know what they want. So have the conversation.
Bad requirements and/or bad requests lead to bad results and data. Ask why someone wants work done - what are you trying to accomplish, let's focus on the outcome.
It's easy to lose sight that optimizing to turnaround time doesn't typically optimize time to actual value and you create hard-to-support data assets.
Dig into what data consumers are trying to achieve and why. Their idea of what they want may not be the most complete or best way to structure the data or insights. Again, have the conversation.
When looking at data quality, tie your metrics and measurement to value. What aspects of data quality matter to that use case?
Data trust is obviously crucial. Data quality issues will happen. When they do, remediate and put rules in place to prevent that same issue from happening again and show business partners that you fixed the cause. It was a data downtime incident, treat it like a software incident.
To achieve actual sustainable change, you don't have to make big, sudden shift changes. You can and should break change into much smaller pieces.
Centralized governance might not be your data bottleneck. Or even if it is, if you aren't mature enough to do federation and decentralization right, data mesh might put you in a worse spot. Really consider your challenges and maturity level.

Dacil started out by calling herself a purple person, meaning not red or blue, not only technical or only business focused but a combination. As part of that, she echoed many past guests - the tech is the easy part, creating value is the hard bit. It's easy to lose focus on creating value and playing with the cool toys, building really amazing things. But do they create business value?

It is crucial to have business counterparts as key partners, key stakeholders, in your data mesh journey according to Dacil. It is so easy for IT and the business side to get disconnected, for use cases and needs to change but the data getting shared doesn't change. And with everyone saying business first, we are still focusing on tech far too often. Dacil likened data practices to being a teenager - we keep hearing business first but don't listen, we do what we want :) And we need to really get crisp on what "business first" actually means.

Dacil believes it's crucial to get out of the IT desk request -> data loop we've been stuck in for so long. The business needs to come to the table and we need to bring them to the table. If the business side isn't part of the conversations, you can't get them to understand what data ownership means. They may say they will take on the responsibility but they can't really grasp how to do it well. So we need to partner with them to get what we need and we need to work with them to improve their understanding and their capabilities around data ownership.

"I need your help to help you" is something core to Dacil's work with the business. Too often, IT is left to wait for requests instead of working together to get what people need. It can unlock a number of additional use cases too.

When creating or improving a data strategy, Dacil recommends including incentives that mean people see/feel the need to participate. Leverage a fear of missing out or FOMO. Make sure people understand what will happen and the rules of engagement. Find the incentives to get them to participate by adding value back to them. Of course, easier said than done.

Dacil mentioned an interesting idea a company is implementing: using gamification to find data quality issues. So instead of it being this really bad thing to discover data quality issues, the people who found the issue get rewards. So that also will drive better data consumer literacy and drive up trust - they are checking data for "does this make sense" instead of just consuming data and learning in the process. Where else could friendly competition/gamification work in data? Look to create friendly and positive energy around your data work and show how it contributes to company value too.

For far too long we've had a vicious and costly cycle of bad requirements and requests leading to bad results and data according to Dacil. So, we need to ask far more questions - why do you really need this. As Alla Hale mentioned in her episode, not in a push back way. She said "what would having this unlock for you?" So take what they are looking for and why and repeat it back to make sure you are on as close to the same page as possible. And then keep communicating while you are building. Drive to that small prototype to make sure you are driving towards value together and aligning on expected outcome.

There's a maturity level to differentiating between what people want and what they say they want in Dacil's experience. And it is also often hard to differentiate between what they say they want and what they are trying to achieve. Always dig into what are they trying to achieve or you will create lots of wasted work. Again, have the conversation.

It's very easy to measure the wrong thing in data quality according to Dacil. She brought up an example where phone number was 100% complete for every record but most of them were not real numbers. So someone said they had perfect quality based on "there was something in the field" but it was unusable, wrong information. So when you look to data quality measurements, tie to value. What is actually valuable here? If it's operational data plane, that might be speed. If it's mailing address for sending out holiday cards to all your customers, accuracy is probably better than completeness. If a few clients don't get a card, that's probably better than sending out lots to fake addresses.

In Dacil's experience, business is typically the first team to notice data quality issues. And that means their trust is broken. Trust is hard to build but much harder to rebuild. How can you be data driven if you don't trust your data? People need to understand that issues will come up but put the rules in place - and show the business - to prevent that same issue from happening again. It was a data downtime incident, treat it just like you would with a software incident.

Other tidbits:

Data ownership is often like a hot potato, no one wants to catch it. You can't throw that responsibility to the business and expect them to know how to handle it.

Talk to potential data consumers before creating a data quantum or anything similar in data. You won't know what you can offer that will be valuable to them until you know what they want. So have the conversation, drill into what is of value and why, then collaborate together to drive to that value.

It's easy to lose sight that optimizing to turnaround time doesn't typically optimize time to actual value and you create hard-to-support data assets.

Break your changes into much smaller pieces. Big changes are more prone to failure and are harder. Make incremental, small-scale progress.

Measure if centralization is actually your challenge before you look at implementing data mesh. If it's not, will data mesh be worth it for you?

Really think if you are mature enough to really do federated governance and decentralized data ownership. Centralized governance is a bottleneck but that will likely be far better than chaos if you aren't ready.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#161 Announcing 3I (Implementers Interviewing Implementers) - Mesh Musings 36 Nov 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#160 Empathetic Upskilling and Data Literacy - Get Your Data Bootcamp Going - Interview w/ Alex Bross Nov 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Alex's LinkedIn: https://www.linkedin.com/in/alex-bross-33853837/

In this episode, Scott interviewed Alex Bross, VP of Data Engineering at Fifth Third Bank. To be clear, Alex was only representing his own views on the episode.

Some key takeaways/thoughts from Alex's point of view:

Start from empathy. Being able to empathize with someone will give you a far better chance of understanding their context. And analytics is really about understanding what the data shows with the applicable context.
There are 3 main barriers to change: logic, credibility, and emotion. Don't skip any of them.
Microservices can teach us a LOT about how to distribute data, especially from the people angle. When moving from the monolith with single branch development to API-driven microservices, how did people feel? How did we get them to the right place mentally and capability-wise? We should learn from that and leverage it for data mesh.
Don't try to do all your disruption at once. And data mesh is a disruption to the status quo. The business has a cadence, look to fit to that as best as possible.
When looking where to begin with something like data mesh, look at need and/or desire. Is there a team that is really struggling and needs change? Or is there a team very willing to try it out?
It's very difficult to have positive change with a team that is already struggling. If you are going to work with them now, make sure to be there to help instead of demand change.
Catalysts lower the amount of energy needed in a chemical equation. Be a catalyst for change - make it less difficult to change. Focus on that and many - most? - domains will welcome you with open arms.
Business users are, by and large, much more digital and data literate than historically. With a moderate amount of upskilling, they're able to leverage low code / no code platforms for scalable/repeatable data consumption.
If your data consumers can see the value data literacy can unlock for themselves - their careers - and their domains, they will likely be very hungry for training.
Failure doesn't have to mean disaster in data any more. With fast feedback loops, you can quickly adjust. Make sure people understand that - perfect is the enemy of good/done.
If you do a bootcamp, look for practical applications over theoretical - how can someone take this and apply it to their job, to their domain today? Actually doing something now - while training - makes it more tangible and likely to stick.
That practical work can have a positive impact now. Set milestones for trainees around real contributions to real use cases as they learn and gain the skills and confidence to contribute back.
From Allen Holub's Death of Agile: Training a team for 2 weeks slows them down for 2 weeks. Not training a team slows them down forever.
Starting change discussions around handing over data ownership is a big potential friction point. If you can give people the capability to own their data and the desire to learn and use their new capabilities, the ownership conversation becomes far easier.
It's easy to fall into the trap of trying to level up data literacy by hiring. If you want to upskill your people instead, look to the incentives of who might approve a data literacy program and align as best as you can.
If you want to get a data bootcamp approved, write a business case for it. People invest in what makes business sense. Make your data bootcamp make more business sense.

Alex started the conversation with an important message that often gets lost in data related discussions - let's start from empathy. Almost all people want to help each other and work together. Most will want share their context with each other so it's important to start from that. Being able to understand someone's context is crucial to sharing your own context with them.

So when thinking about data mesh, Alex thinks it makes sense to look at microservices and how people felt and behaved and learned during that transition. How did we upskill people for DevOps? What were effective patterns and what were the not-so-effective ones? And as Zhamak has discussed with her approach to data mesh, we need to decontextualize those approaches and learnings from software development and DevOps and then recontextualize them for data. Easier said than done - and not that easily said - but still, we have a lot of good and well documented history to learn from.

A focus on quarterly results is another aspect Alex feels is important to recognize - there is a real sense of urgency. But there is also a desire to not rock the boat too much - we can't do data mesh in a quarter. Combining those two aspects, we need to break data mesh work into more manageable and understandable pieces - those nice milestones. You can't do all the disruption at once as it will create chaos.

When looking for a place to get started with data mesh, Alex recommends looking for domains that either are having real business pain and pretty urgently need to change or domains that are very willing to take on change. If a team is having lots of issues, you can quickly add value and alleviate pain. If they are very willing, it obviously makes things easier. And then of course take the stories of successful change and use them to entice other teams :)

But, according to Alex, if a team is having issues with change, you should look to what's blocking them. It's easy to try to throw additional change at them but that's just more burden when they are already struggling, already underwater. Go in with empathy to help them, that will win them over. Anchor to the idea of being a catalyst for change, something that lowers the effort needed to achieve change. And look to manage feelings because it's scary to think that what you are doing as part of your role won't be relevant. Watch out for that and work with them to see it as the less valuable pieces going away, not their importance.

Alex is quite excited about some of the new data technology on the data consumption side - it's starting to feel like the tech is finally catching up to the software side in terms of tooling. Data consumers are much more digital and data literate than historically and with low code and no code offerings, with a bit of training, business users are able to actually get to and leverage data for generating insights. Note: Zhamak has said she is skeptical of low code/no code tooling but that has typically been on the data production side.

When Alex and team looked at where they were having the biggest challenges around data, it was consumability. Data consumers weren't comfortable trusting the data, couldn't really understand the data, couldn't always find the data, etc. And the data changes were centralized with the data team itself. While that is a common situation - the data team owning changes when they don't have the context of the use case or the context of the source systems - it wasn't scaling well for them.

Because of the data issues, data consumers were the hungriest for change and Alex and team decided to set up a bootcamp to level up data consumers. With more knowledge / higher data literacy, data consumers were excited to be able understand the data more, make changes on their own - thus not relying on the centralized data team -, trust the data more, etc.

Pushing the idea of finding your failures fast - and being okay with issues/failures - in data is something Alex is pushing. Perfect is the enemy of good so putting something out and then stress testing it is far better than theoretically sound but untested work. With a fast feedback loop, they were able to quickly adjust their data bootcamp when they heard learning SAS before SQL made it very difficult to understand for instance.

Alex shared some interesting learnings from the bootcamps they are doing thus far. 1) Focus on practical application and let people get their hands dirty now rather than teaching only theory. Time shouldn't only be spent in the classroom. 2) Look for those fast feedback loops. 3) Work to ensure people can take the time off necessary to actually focus on learning. It's easy to add more things onto a schedule, not put things off. 4) People really do want to learn to use data, they just aren't sure how without some structure. 5) Similar to #3, cognitive load is a very dangerous thing for bootcamps - don't overburden people. 6) Let people know they don't have to retain the exact specifics versus the knowledge on how to leverage data. You won't remember every function by heart and you shouldn't have to.

According to Alex, steering the conversation away from ownership towards getting someone excited about learning a new skill has worked well. Instead of 'I need you to do X. I guess I will train you,' instead they are starting from training new skills to unlock additional capabilities and people are far more willing to take on using those new skills. People are so excited to go unlock more value and do cool things with their new skills. And they'll layer in responsibility and ownership as those people learn more and apply their skills.

Getting to viable, valuable use cases is not hard according to Alex. He recommends just asking people what's frustrating them and then synthesize those into use cases that you can potentially address. That's actually what they are doing in the bootcamp - having people find use cases before coming into the training and then spending half their time during the bootcamp working on tackling those use cases. Getting them some real world situations to try to apply their new skills to. And if something already does exist around that use case, then they contribute by augmenting it or writing documentation or something similar. Create real-world impact milestones.

It's pretty easy to feel like you might be left behind if your team isn't framing data work the right way according to Alex. Instead of pointing out exactly where to fix or that things are changing or they need to upskill, he looks to create the environment for them to realize things aren't great instead and have the realization that they want to go fix it themselves, regardless of the issue.

Alex believes there are 3 main barriers to change: logic, credibility, and emotion. Most people try to appeal at most to two of the three and in data we often forget to manage emotion. And it's really important. And there is a heck of a lot of change involved in people's roles and work when it comes to data mesh in the long-run. As stated, it doesn't all change at once but you need to work with people to work through the change. Don't look to skimp on hitting all three.

To get a big data literacy program approved like Alex did, look to whose incentives align with it and work with them. Always start from how is this beneficial to the person who gives the green light. You can give them the choice of do we try to hire our way to data literacy or do we train and upskill our current people. It might feel faster to try to hire but it probably isn't. And it's probably way more expensive. For Alex specifically though, the team was pretty bought in so he didn't have to go to a heroic effort.

Other tidbits:

If you want to get a data bootcamp approved, write a business case for it.

Currently, Alex is encouraging a building block approach to everything. It's easy to fall into trying to scale too early but you do have to prepare for scaling if you're successful.

"Training a team for 2 weeks slows them down for 2 weeks. Not training a team slows them down forever."

Basic blocking and tackling questions, those FAQs, are probably the biggest stumbling blocks for most people in data mesh. How to get access to X or Y system. Look for patterns in those questions and try to address them through automation.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 27, 2022 Nov 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#159 Focusing on the Problems - And Business - at Hand in Your Data Tool Selection Process - Interview w/ Brandon Beidel Nov 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

LinkedIn: https://www.linkedin.com/in/brandonbeidel/

In this episode, Scott interviewed Brandon Beidel, Director of Product at Red Ventures.

Some key takeaways/thoughts from Brandon's point of view:

Be willing to change your mind, especially based on new information. Be willing to measure and iterate. It's easy to get attached to tools or tech because they are cool. Don't! Stay objective.
It's crucial to align on what problem(s) you are trying to solve and why before moving forward on vendor/tool selection, no matter build versus buy. If it doesn't have a positive return on investment, why do the work?
Beware the sunk cost fallacy! It's easy to not want to shut something down that you've spent a lot on. But don't throw good money after bad.
When requirement gathering/negotiating, have a 'maniacal focus' on asking "what does this drive for the business?" You can quickly sort the nice-to-haves from the needs and you can have an open and honest conversation about cost/benefit of each aspect of a request.
When thinking about maximizing value, there is always one constraint that is the bottleneck. You can optimize other things but they won't drive the value. Find and fix the value bottleneck.
A simple two axes framework when thinking about use cases and requirements is value versus complexity. Look for high value low complexity first.
Be open and honest in discussions around expected costs of work/tools - which can be considered part of the complexity. The data consumers understand the value and can weigh the return on investment.
It's very important to understand data consumers' incentives so you can collaboratively figure out what is best for all parties.
Look to create - in the open - a decision journal relative to build versus buy / vendor selection. It will create an open environment and get your thoughts better organized.
Your decision journal will make it easier politically to say you have new information and should consider a change. And you can better measure against if your assumptions were right or it's time to reevaluate if a tool or solution is still working for you.
It's very crucial to look at potential major success of a tool selection - what happens if our use is 10x, 100x our initial expectation? That can lead to really poor unit economics in the future for certain selections so it shouldn't be overlooked.
It's easy to over innovate. Think of having a certain number of innovation tokens. The cost of change is real and also hits people's patience. Look to see if existing tooling or capabilities support most of your use case first.
Total cost of ownership - not just initial purchase cost - is crucial. How much of your team's time will be spent managing and maintaining the tool? Look especially at skills, governance controls, and ability to measure if you are successful.
Perfect is the enemy of good in choosing tools. Use a well-defined process to avoid really bad decisions but spending time to find the absolute best solution when any one of six choices will do just fine is rarely worth it.
Having your reasoning and process written down and in the open drives trust. Trust towards the initial decision and trust for when it's time to reevaluate at tool. It also makes it easier to spot if something relative to your initial assumptions has changed.
Seek out those who might be the most against your decision. Take the time to understand their pain points and concerns; try to incorporate their concerns and align their incentives if possible.
When adding a new tool or serving a new use case, focus on how you will measure if you are successful now and in the future. It doesn't have to be perfect but otherwise, you don't know how well you are doing and will miss out on a great learning experience to do better in the future.
When you select a vendor, there is a logical time to reevaluate your choice and if it's right going forward - the contract renewal. And there are easily defined economics in play. You should do the same for anything you've built - set an artificial time to reevaluate, don't wait for things to go bad first.
Consider using the anti-corruption concept from microservices in data. You can avoid a lot of data integration costs and you are more easily able to rip things out of your platform. But it's okay to leverage proprietary solutions too, just be cognizant it may become an issue.
Involve the data consumers early in the process around serving their use case. And it helps for them to have skin in the game so they are focused on driving to the most business efficient outcome.

Brandon started off with a theme he'd hit on multiple times because it's so important: before proceeding on selecting a tool/solution, agree on what needs to be done and why. What will this drive for the business? It's easy to lose the forest for the trees - or even the leaves - in building out data platforms. The first part - agree - is necessary because you need alignment to move forward with the proper understanding of the problem to be solved. The what needs to be done and why part means there is a clear roadmap and that you have a specific problem you are trying to solve when doing your tool evaluation instead of focusing on the tool or feature.

Having a maniacal focus on 'what does this drive for the business' will mean you can align better on what is needed for a use case versus "a Christmas list" as Brandon put it. Having clear and open communication about what is a requirement versus a desire and the cost of each potential item on a data consumer's list has led to very efficient prioritization for him.

A key way of working when embarking on a new use case is to involve the data consumers early on - and make sure they have skin in the game, according to Brandon. The data team's engineering time being on the data consumer's P&L means the data consumers are more focused on driving to key results than cool features or nice-to-haves. And having open and honest discussions about the expected costs to deliver on each really helps them weigh the benefits. An important part of getting to a good outcome in these discussions is understanding and attempting to align on everyone's incentives.

Brandon mentioned how when discussing cost/benefits and different platform approaches, it's very easy to get overly complex. But that hurts the conversation and often devolves into technical discussions with people who care about the business output, not the tech. Brandon has two axes that he uses - complexity and value. Don't overcomplicate it. It's pretty easy to start with use cases that are high value and low complexity when you start to look at it through this lens. High value but high complexity use cases are tough but can obviously provide very significant value when you've taken care of the low hanging fruit.

One thing Brandon mentioned - and Scott recommends more broadly for data mesh journeys - is a decision journal. Having a place in the open where you write down the criteria for a decision makes it so people can feel more comfortable with the decisions made. What were the capabilities needed, what was the problem, what was the expected value, etc.? When getting down to the decision itself, how viable is the solution, what are the alternatives, what is the likely cost, what are the failure scenarios, etc.? It helps you to reevaluate in the future as well and have empathy with past decisions. Brandon has a list of many more crucial questions.

A really interesting point Brandon brought up regarding writing out your decision criteria is what happens if it's wildly successful. What happens if the tool/feature you choose, whether build or buy, has 10x the expected usage? 100x? Are the unit economics going to be good or will this potentially cause issues and how do you plan to adapt?

According to Brandon, looking at total cost of ownership - not just the short-term or initial purchase cost - is crucial when selecting a tool. Do you need training to actually leverage the tool and manage it appropriately? Does it integrate well with your existing platform/tools? Again, this circles back to value versus complexity. Costs should be factored into the complexity discussion.

Brandon emphasized perfect is the enemy of good. There is rarely a good return on finding the absolute best choice - the real benefit is in avoiding the wrong choices. If there is a 5% better return on tool B versus tool A but you had to spend months figuring that out - or what if it's 6 tools…? - that's not worth doing.

As part of Brandon's decision journal recommendation, he circled back on a few other benefits. A big one is that people are more likely to be aligned with the decision if they can follow the logic. If it's just a choice instead of seeing why the choice was made, there's often more friction and pushback. Also, it's easier to monitor if things have changed relative to your assumptions when you have your assumptions explicitly stated :) Having these assumptions on paper also gives you better buy-in to make changes because again, people can follow the logic.

When it comes to driving buy-in, Brandon recommends seeking out the people who are most likely to be detractors to your potential solution. Use collaborative negotiation. At the very least, go and understand their context and pain points. Try to incorporate that into your solution and look to align incentives where possible. Too often people don't feel seen or heard.

As many guests have mentioned, look to set your success criteria and especially ways of measuring before you start implementation work. It doesn't have to be perfect but otherwise, are you able to measure when you are doing well? And you can learn from things that don't go to plan much better if you can measure against an actual plan.

Brandon discussed how when you make a choice to go with a vendor, the contract renewal - or specifically a few months before the renewal - is the time to evaluate if it was a good choice and if you should continue forward with that choice. You should set up an artificial timeline to do the same for anything built internally. Instead of waiting for signals that you've made a wrong choice, regularly reevaluate. It's important to reflect back and see if it's actually solving the challenges you wanted it to solve.

Beware the sunk cost fallacy! According to Brandon, it’s very common to want to chase things where you've already spent lots of time and/or effort. Or if it had a lot of promise and isn't meeting the expectations. Don't throw good money after spent money. Take it as a learning opportunity and move on.

Circling back on tool stewardship and total cost of ownership (TCO), Brandon uses a framework of three main things to consider: skills, governance controls, and ability to measure. Do you actually have the people who can leverage a tool? Do you have the governance in place to use it properly? How will you measure if the tools are successful and being used as expected? He had a lot of good examples in the episode.

Brandon recommends people look into the anti-corruption layer concept from microservices in your data platform. It can lower the integration costs and also make it far easier to rip things out. You don't want to focus too much on this though and never leverage proprietary features. You don't need to build every capability from scratch but also don't unnecessarily lock yourself in.

Some other tidbits:

Your business counterparts probably won't care much about which vendor or feature versus what it gives them. Start at the high level mapping what's needed.

You need to define the problem you are trying to solve, not the vendor.

People are only willing to deal with so much innovation. Think of having tokens that people collect from you when you try something innovative and new. That's not an easily renewable resource. Look to what you have already to see if it will work.

When thinking about maximizing value, there is always one constraint that is the bottleneck. You can optimize other things but they won't drive the value. Find and fix the bottleneck.

"Knowledge has a half-life, decisions have a half-life." Don't get analysis paralysis, look to move quickly.

Be willing to measure and iterate. Be willing to change your mind, especially based on new information. It's easy to get attached to tools or tech because they are cool. Don't; stay objective.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#158 Zhamak's Corner 10 - Blazing Trails not Blazing Saddles - Setting Yourself Up For Success Nov 22, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

What can we do now relative to data mesh with what we have? People want to move, not wait for the tools to evolve. We can start to shift in anticipation of tooling getting better. It might not make things a ton better now, but when tools start to emerge, then we can jump ahead quickly. Learn from what happened in the API revolution and don't compromise on interoperability - that will just lead to high quality data silos, which is not a great outcome. And we need to get to a place with data where consumers have a delightful experience going from discover to learn to trust to use with little friction.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#157 Getting Practical with Data Privacy - Interview w/ Katharine Jarmul AKA K-Jams Nov 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Katharine's LinkedIn: https://www.linkedin.com/in/katharinejarmul/

Practical Data Privacy (Katharine's book in early release): https://www.oreilly.com/library/view/practical-data-privacy/9781098129453/

Katharine's newsletter: https://probablyprivate.com/

'Privacy-first data via data mesh' article by Katharine: https://www.thoughtworks.com/insights/articles/privacy-first-data-via-data-mesh

danah boyd [sic] website: https://www.danah.org/

In this episode, Scott interviewed Katharine Jarmul AKA K-Jams, Principal Data Scientist at Thoughtworks.

Some key takeaways/thoughts from Katharine's point of view:

Increasing privacy around data does NOT mean you have to give up value.
Instead of data privacy being a blocker, it can turn nos to yeses because there is a better ability to restrict illegal/unethical use. Regulatory and legal people want to say yes, so give them the ability to do so.
There are lots of tools available to enhance your data privacy now. This isn't a pipe dream. That said, don't look to replace person-to-person conversations and decisions with tech. You'll learn when to use what on your journey, it's okay to iterate :)
Empower the people who know the data best with privacy tooling. Don't make them build it themselves either. They will know best most of the time - but obviously provide them a path if they have questions/concerns.
Privacy is a sliding scale, not all or nothing. You can start off pretty not great and still make progress as you continue to assess where you can be better.
Use privacy as a lens for how valuable your sensitive information actually is. Can it be used appropriately? If so, what's the value of leveraging that data and is it worth the privacy cost?
When thinking about what level of privacy to enact in systems, think about your comfort level if it were you. Would you want your location history shared at all times? Using a multi-cultural approach is important too as different cultures have different norms around privacy.
There are some basic stakes privacy tech you should use as part of just general information security to protect sensitive information. It might even be required by law. Then, you should look to layer on privacy-enhancing technology on top to do things like actual anonymization.
Privacy choices should be broad organization-driven decisions, not something one person decides to implement. But if there isn't buy-in, it can sometimes be tough to show people the business value of data privacy.
Look to the privacy/utility trade-off - how can we maximize privacy but also maximize what we need to do the task at hand?
An emerging practice that has a big potential impact on privacy in data mesh is federated analytics and distributed queries - can we do analytics on data where it is without moving it to a big central place to do the analysis?
A lot of data scientists don't want to work on things they feel are problematic. So work to prevent problematic use cases and the problematic data practices that can give rise to them.
Privacy can be about first order problems like how do you anonymize your data. But it can also be much broader, like understanding the impact of what work we do on people and society as a whole.
We need to make privacy more transparent and obvious, otherwise people feel tricked. People want to know what they get out of sharing their information.
In data mesh, look to offer the easy ability to adjust privacy - privacy 'knobs' - so the domain expert can easily make choices without having to implement the tech. Enable privacy via the platform.
Typically, data privacy is applied at the data source. We are still learning how to do data privacy well in a federated setup with any cross domain data combination restrictions.
If you're constantly rejecting use cases, that will just create shadow IT. As an organization, figure out how you can get to a yes where you can.
Scott Note: K-Jams is writing her book to fill the gap in information about privacy technology between very basic info and academic research.

Katharine shared how she first started looking into privacy. She was doing natural language processing (NLP) and was working with lots of customer data that was supposedly clean and anonymized - but it wasn't. It raised a red flag for her as that can mean working with some problematic things. So she went looking for solutions and discovered there was a lot of existing privacy technologies - and even more now. We can look at first order privacy questions like how do we actually anonymize data or bigger and more philosophical questions about how do we prevent harm on society from our technical work.

While privacy can be a bit of a nebulous topic, Katharine recommends starting from the gut check, basically: what would you be comfortable with? Would you be okay with your chat history being shared with others? What about your location at all times? If no, look to prevent that from happening in what you are working on. But it's important to also use a multi-cultural lens on what is acceptable as what's okay in the US - or at least tolerated - is not in Europe. And this of course can extend to fines from GDPR but also reputation. Think about how data can be misused and look to prevent that. Scott Note: see episode 143 for more about data ethics and what we can do to prevent misuse.

Given that privacy is very contextual to the individual, K-Jams believes it's far too often that when we translate privacy decisions to code, we lose the context. And part of that is making privacy decisions and options obvious. If you are collecting people's locations, are you giving them an easy ability to see why or opt out? It's very easy for people to feel tricked.

Katharine gave an example of how digital native people - mostly teenagers on Instagram about 10 years ago - had different accounts - called 'Finsta' accounts for fake-Instagram - with different levels of anonymity and privacy to better navigate the opaque privacy settings and share data more granularly with who they wanted. It gave them an ability to control how they were sharing their data and with whom. But most people aren't capable - or willing - to do something like that.

To drive trust, K-Jams believes we need to show people what they are getting from sharing their data, what is the benefit. Then they can make an informed decision. But privacy settings are often opaque at best. She believes that companies leaning into privacy conversations with users will create a better relationship with customers. How are you delivering value back to customers from them sharing data with you? And increasing privacy doesn't mean you have to give up on the value of your data either.

A few privacy techniques where tooling already exists Katharine mentioned are pseudonymization, tokenization masking, and format preserving encryption - she recommends using these as some basics to protect PII (personally identifiable information) or other sensitive information. These are just basic stakes information security best - maybe even just 'not bad' - practices. Then you want to look to potentially layer additional technology on top like differential privacy. We can even leave data where it is to do federated analytics and federated learning, which has implications for data mesh and machine learning.

When looking at the value of privacy, it can be tough to drive buy-in internally - people assume cost with no additional value. But according to K-Jams, you can look at the privacy/utility trade-off. How can we simultaneously maximize privacy while still not inhibiting the work we need to do? And how, in data mesh, do we actually find those sweet spots. Katharine believes it's through giving the data owners the ability to tune privacy - think knobs - to the specifics of the need/use case. That's part of doing federated computational governance through a self-service platform after all. Try saying that 3 times fast…

K-Jams believes it's easiest - at least with current technology - to apply privacy at the data source. But when thinking about something like data mesh, there may be additional challenges like data from domain A and domain C should not be combined. So we are still learning how to do data privacy well in a federated environment. Scott Note: Jesse Paquette covered this in healthcare data in episode 10 where certain anonymized information could be joined with other anonymized data to make it personally identifiable. Many people are saving those "tricky" use cases for later or not trying to automate privacy and cordoning off those data products except by request.

When leveraged well, Katharine believes data privacy technology can actually add more value. If a data producer is not sure how data consumers will use sensitive data, they are very unlikely to share it. But if they can lock down the data in certain ways but still give them access, that is a win-win. The data consumers get access to information they wouldn't have gotten otherwise and data producers can still sleep at night. It can turn a no to a yes. Sarita Bakst mentioned something similar in episode 52. And you can also get past legal and regulatory barriers if you do data privacy right - your legal and regulatory people want to say yes, so give them the ability to turn their no to a yes. Offer up potential privacy concerns offsets - say only using it in a sandbox to start - to see where their issues are.

For K-Jams - and Scott - the desire to remove the people from the technology aspects of things like privacy ends up being silly. We can't make decisions only via the tech. Stop trying to replace conversations with technical solutions, sometimes people just need to collaborate to get where we need to go. Don't make it a yes or no decision for someone like legal, exchange context and look to collaborate on a positive outcome instead of 'can I do this?'

Katharine gave a good overview of how to move up the privacy ladder about 47 minutes into the interview (not the episode; likely ~55 minutes into the episode?). How do you move from not so great to okay to pretty good to good (but using meh -> eh -> heh -> hah because of Scott…). Privacy isn't all or nothing and you can improve and iterate.

Quick tidbits:

Second-layer privacy enhancing techniques mentioned: differential privacy, data minimization, federated analytics, federated learning, distributed querying, encrypted computation, and secure multi-party computation.

You probably won't get your privacy perfect on your first try. That's okay. Look to prevent regulatory/compliance issues but much like all aspects of data mesh: try, learn, iterate.

Think about what you can and cannot show in a data catalog about potentially sensitive data. You can share descriptive statistics and information about use cases without exposing the sensitive data until you know a new use-case is allowed/ethical. Look to share as much information as you can - where appropriate - instead of locking down anything related to sensitive information.

Empower the people who know the data best with privacy tooling. Don't make them build it themselves but they will know best most of the time - but obviously provide them a path if they have questions/concerns.

It's very easy for privacy concerns to become overbearing. If 90% of the time, you reject use cases, you will create shadow IT and that is far more dangerous for legal and regulatory reasons. Look to exchange context and work towards a viable solution.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 20, 2022 Nov 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#156 Zhamak's Corner 9 - A Vision of the Data Product Developer Role in Data Mesh Nov 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Who will be the data product developer in data mesh? There has been a misconception in Zhamak's view that the application developers should be the ones focused on building the data products as well - but she thinks they already have a full-time role :) But, we need someone applying software engineering practices and data know-how to building data products. Right now, to do data work, you need way too much tool knowledge instead of data understanding. We have hyper specialized data roles - ML engineer, data engineer, etc. - when we should have data developers that can tackle these challenges better.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#155 Phase Shifting - Preparing for Data Mesh Adoption Going Wide in Your Org - Mesh Musings 35 Nov 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#154 How Can Data Marketplaces Help Realize the Most Value from Our Data - Interview with Mozhgan Tavakolifard, PhD Nov 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Mozhgan's LinkedIn: https://www.linkedin.com/in/tavakolifard/

In this episode, Scott interviewed Mozhgan Tavakolifard, Data and AI Lead for the Nordics at Accenture. To be clear, she was only representing her own views on the episode.

Before we jump in, most of the conversation was about external data marketplaces rather than internal data marketplaces within an organization. It's also important to note that data marketplace technology and implementations are still in the relatively early stages - it's quickly evolving and maturing.

Some key takeaways/thoughts from Mozhgan's point of view:

Data marketplaces - internal and external marketplaces here - significantly lower the bar to data consumption because of standard metadata and user experiences. You should be able to easily see quality metrics, who owns a data product, access documentation, etc.
Data marketplaces, when done right, significantly lower the time to value realization for both data producers and consumers/purchasers. And standard quality measurements and metadata make it easy for consumers to understand how much they can trust data to make purchasing decisions easier.
Practices and tools are emerging for tracking data quality all the way to source to increase the trust data consumers/purchasers can put on data, especially for data marketplaces.
For external data marketplaces, trust and security are still major pain points. How can data producers trust consumers will protect the data they acquire and use it legally and ethically? What is their risk to consumers behaving improperly?
?Controversial?: Mozhgan believes smart contracts and blockchain/distributed ledgers can provide for compliant use by others purchasing data. Some marketplaces are already doing this.
For data producers, they also want better ways to ensure data consumers/purchasers are only using data in an agreed way so they can charge for any additional use cases. So they will be heavily incented to work with marketplaces that have tracking mechanisms in place.
"Data ethics is a nightmare," even before we think about data marketplaces. And that's not just data bias.
We can use a number of the techniques and guardrails used to ensure ethics around bias in AI to apply to data marketplace ethics around bias.
In a data marketplace, ethics falls much more on data producers than most people realize/expect. You should not sell data that can be misused! One way to prevent misuse is to sell insights instead of data itself.
Look to focus much more on the business returns of data work. Far too much is focused on the value generated without looking at the costs.
It's crucial to see organizations as living, breathing ecosystems. Design your organization and ways of working to be able to adapt.

For Mozhgan, data mesh is a perfect fit with data marketplaces as a data marketplace makes it simple for producers to easily share data in a standardized way and consumers to easily find and consume data with standardized metadata and access. Simply put, data marketplaces are the most sensible place and mechanism for sharing data in her view. They significantly lower the barrier to getting access to data and being able to understand data - including how much they can trust data.

So data marketplaces are good for internal data sharing but even better for being able to monetize your data externally according to Mozhgan. Again, the standardization and clear rules about what is allowable use means a faster time from discovery to value for both data producers and consumers/purchasers. Data having clear and concise SLAs means consumers can quickly go from discovery to trusting the data, meaning they can quickly leverage for their own use.

However, major pain points for external data marketplaces are trust and security - for data producers, they must create the trust in their data for others to use it but there is also a big risk to how data consumers/purchasers actually use data producers' data. Is it compliant/legal use? Is it ethical use? Will those data consumers properly protect the data they consume? If not, what is the risk to the data producer? How can we ensure proper behavior - whatever that may mean to the data producer - by the data consumer/purchaser?

Mozhgan believes blockchain/distributed ledgers might provide a good answer to be able to track compliant usage - are consumers meeting their contractual terms? Smart contracts are supposedly able to track this. However, ethical concerns are still not addressed in smart contracts, at least in a simple and repeatable way. The ways of doing this are still evolving. And she believes we can't really get to large scale data marketplaces without something like blockchain. Note: Scott is much more skeptical given there are few examples he is aware of where blockchain is really working for trust and security - can you really track usage in someone else's systems? What about their security capabilities to not have a data breach? Can we actually track ethical use in data?

Another aspect Mozhgan mentioned is that data consumers can only use data they purchase in ways allowed by the contract. Sarita Bakst mentioned this when talking about externally purchased data - data producers want to maximize monetization so data purchasers have to pay for each individual use case. So data producers want to track that consumers/purchasers are actually adhering to that part of the contract. There are a number of recent examples where data sellers will have wildly different prices for the data in PDF form versus an API. The API probably actually costs less to maintain but there's a strong correlation between consuming via API and getting a lot of value from the data consumed.

When it comes to data consumer trust - can they actually trust the data? - Mozhgan believes we are seeing better ways of tracking data quality all the way up to source. That independent verification is crucial. If data consumers/purchasers understand the exact quality dimensions, that typically makes the data immensely more valuable. Stolen credit card numbers on the dark web go for pennies because you can't really trust the source for example.

Mozhgan gave a really interesting example of where data marketplaces can take us. Utilities need to monitor trees and proactively trim them where possible so they don't disrupt powerlines or phone lines. But each utility typically does not have a great information set internally - often from a lack of the amount of data to actually be good at proactive tree trimming. So utilities are trying to get to a place where they can jointly share information with each other to improve their predictions for where to trim. However, a lack of a standard way to share data is really making it quite difficult to actually achieve the desired results. So how can we learn to quickly share information across organizations without a long and complicated process to do things like design a standard data model? Could a marketplace help?

"Data ethics is a nightmare," even not related to data marketplaces according to Mozhgan. This is not just AI model ethics with bias and the like but there are often unethical ways of presenting the data. Then of course, there are many companies collecting and using data unethically. And we don't necessarily always want to remove all bias - it may have predictive power. But we need to focus more on the impact of our decisions on the input and output/impact side with data. And she believes we can use a lot of the guardrails we use around AI to ensure ethics in data marketplaces.

Mozhgan recognized that ethics will always be a bit messy when sharing data outside the organization. One suggestion to prevent ethics issues is to only share the insights instead of the actual data used to generate the insights. Or you can share pseudo-anonymized data as well. But at the end of the day, ethics falls much more on data producers than most expect. You have a duty to not sell data that can be misused!

For Mozhgan, there is too much of a focus on the value generated from data work instead of the actual return on investment. This happened in AI with massive hype and it's happening more in analytics recently - everyone needs to be data driven, right?! You need to create a business case and look at what the expected costs will be for data work. We don't have really easy paths to predicting exact value but we can get better at that and be realistic about expected costs.

Quick tidbits:

Knowledge graphs will be crucial to sharing data with other organizations and internally for data mesh.

It's crucial to see organizations as living, breathing ecosystems. Design your organization and ways of working to be able to adapt.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 13, 2022 Nov 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#153 Federated Data Governance Through Changing Minds and Hearts - Interview w/ Mariana Hebborn, PhD Nov 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Mariana's LinkedIn: https://www.linkedin.com/in/mariana-hebborn-phd-118035117/

In this episode, Scott interviewed Mariana Hebborn, Lead of Data Governance for the Healthcare Sector at Merck Group Germany (not Merck, the pharmaceutical company).

Some key takeaways/thoughts from Mariana's point of view:

It's crucial to answer why are you doing data governance. Is it for improving data quality? Better data security? Know what you are trying to achieve to best focus your efforts.
Make it easy for people to understand how and why to share their knowledge with the rest of the organization. The people mindset really is the most important aspect of successful digital and/or data transformation.
Most everyone knows we need to go to federated data governance but the big question is how. How can we do it safely? How can we evolve? It isn't a simple switch we can flip.
To drive buy-in for moving from a centralized data governance approach, we need to show the benefits of federated - when done well - versus a monolithic approach.
At the end of the day, governance is about conversation and missioning - why should you care about governance? What value will it drive for your organization? Answer those questions first.
We need to find ways to organize closer to the source to capture far more domain knowledge when sharing data. Centralized teams just can't understand the context in a large and complex organization.
?Controversial?: Most data access should be to packaged insights - the computational result - rather than raw data itself. Most people consuming information want the insights, not the raw data.
We need to take learnings from operations and microservices so we drive to clear boundaries, clear responsibilities, and easy access to the information people need. That will prevent data silos and keep us agile.
Who should have access to what data is far less cut-and-dry than we'd like. It is much more about what's in the data and the specific usage. So domains need to understand some clear rules but make the decisions because they understand the data itself far better.
"The best data owners and data stewards are found, not made."
We need to get to a place mentally where data governance is so ingrained, there's never a question of if we should be doing it.
Start with looking to better govern the data that is already generating good value - or is otherwise important - first. Don't try to govern all data at the start.
If data consumers aren't sure if it's allowed and/or appropriate to use data in certain way, it is on their shoulders to ask. Data owners and stewards can only set rules that go so far. Otherwise, data owners and stewards cannot feel comfortable giving others access to their data.
Everyone should be able to browse what data is available even if they cannot have access by default. You can find more use cases that data consumers wouldn't have thought were possible or available.
Lack of knowledge of the law and regulations does not protect you from the law and regulations. Don't be naive.
"Centralize the knowledge within the domain".

For Mariana, when talking about data governance, the general industry consensus is that we need to get to federated governance, but the big question is how to actually do that. And governance needs and the pace of change are very different depending on the industry. Many industries are already adopting the federation mindset but are still struggling to do data governance well - it's either centrally managed or it's a bunch of silos. How can we get past that?

And according to Mariana - and Scott - centralized data governance is a pragmatic approach. Until it isn't. At scale, centralized data governance is breaking in most - if not all - large organizations. So we need to look for ways to organize closer to the people with the knowledge about the data so they can share the domain-specific context far better than anyone else. But we still want to lean on governance experts to keep domains aligned with the greater organization. Mariana believes we can win over people by clearly comparing federated versus the monolithic approach - for domains and for the greater organization. Show them what data governance means for them, why it matters, and why it benefits them, instead of trying to show them exactly what to do.

We have already figured out how to do cross domain information sharing on the operational plane in Mariana's view. That prevents the silos but is also not a centralized way of working. So, we need to figure out how to do the same with data for analytics, taking a lot of the same learnings from moving to microservices. We need clear boundaries and accountability - it is needlessly confusing when we don't know who is responsible for what. In data, we need to focus on getting people access to the information they need to better the organization. And Mariana knows the idea of central access and control feels like a good one - the central team knows governance best right? But, it just doesn't work well at scale.

According to Mariana, much of doing federated governance well is about changing your mindsets. To get started with federating your governance, you need to find your data owners and data stewards. And per Mariana, "the best data owners and data stewards are found, not made." There needs to be people on the ground in the domain ready to clean up before federating governance to that domain to tidy things up as best as possible. And we want to move towards more/better governance as well. You can't go from no governance to governing everything. Start with what matters and what drives value.

In many long-evolving industries or companies, people often ask if we should even do data governance in Mariana's experience. Really data driven companies don't ask that - we need to get to a place mentally where data governance is so ingrained, there's never a question of if we should be doing it. But we also can't boil the ocean. So start with looking to better govern the data that is already very important first, whether that is because they are generating value, are sensitive, are widely used, etc. And then figure out what you are trying to get from data governance - is it quality, better security around your data, something else? If you don't know, figure that out first.

Mariana believes crisp policies about compliance are very helpful to lower the amount of effort people need to be in compliance. We can't have every domain team learn all applicable laws and regulations. Well-crafted policies mean there is less work interpreting what is and what is not allowed. Of course, this is far easier said than done.

At the end of the day, governance is about conversation and missioning according to Mariana - why should you care about governance. It's very easy to fall into decentralized instead of federated governance. Decentralized is where you end up with data silos. And there are information silos to overcome as well - knowledge is typically trapped in people's heads instead of disseminated through the organization.

For Mariana, when doing federated governance, it's best to centralize the knowledge within each domain. And bring the rules to the data. Make it easy for people to understand how and why to share their knowledge with the rest of the organization. The people mindset really is the most important part of a successful digital and/or data transformation.

Quick tidbits:

Most data access should be to packaged insights - the computational result - rather than raw data itself. Most people consuming information want the insights, not the raw data.

We need to identify who are the people who are already working with data, who knows who should and who already has access to the data.

Everyone should be able to browse what data is available even if they cannot have access by default. You can find more use cases that data consumers wouldn't have thought were possible or available.

Accountability for data owners and stewards only extends so far. They need to own who should have access and what is proper use but if someone misuses data, that is on the person or team misusing the data, not the data owner or steward. Data consumers should ask if they aren't sure if something is allowable/appropriate.

Lack of knowledge of the law and regulations does not protect you from the law and regulations. Don't be naive.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#152 Zhamak's Corner 8 - Are We Using Tech Gold as a Paperweight? Nov 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#151 Driving Interoperability via Taxonomies and Tagging to Power Personalization - Interview w/ Jill Maffeo Nov 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Jill's LinkedIn: https://www.linkedin.com/in/jillianmaffeo/

Developing Interoperable Channel Domain Data (blog post): https://vista.io/blog/developing-interoperable-channel-domain-data

In this episode, Scott interviewed Jill Maffeo, Senior Data Product Manager at Vista.

Before jumping in, Jill gives a lot of very useful examples of outcomes they've been able to drive that could be abstracted to apply to your own organization's business challenges. Outcomes like better customer segmentation, faster time to launch new offerings, etc. If you are having difficulty with stakeholder buy-in, especially for someone in marketing, this episode could help you frame things in their language.

Some key takeaways/thoughts from Jill's point of view:

"When you're thinking about interoperability, it's just playing nice, right?" If you think of interoperability as a key part of your culture, it's easier to implement. Let people know why interoperability is good for them and the whole company.
Taxonomies help drive interoperability because there is already an established language even if things don't fit perfectly. New concepts can emerge and your taxonomies should change. But it makes the interoperability discussions have at least a common starting point.
Taxonomies are a living thing - make sure they aren't overly rigid and be prepared to continually evolve and improve them.
Within your taxonomy structure, if there is a reason for things to be unique for a domain or use case, that is okay. Look for potential ways to also convert that data to best fit your taxonomy but you don't want to force a square peg through a round hole.
Taxonomies really start to add a lot of value at scale. They are somewhat costly upfront with likely moderate return on investment early; but, if you do them right, will pay back a lot as you move forward. They make historical analysis - especially with interoperability - far easier because you've done the work ahead of time.
Taxonomy, when done well, is about balancing standardization and flexibility. Much like most things in data mesh, it's about finding the right balance for your organization.
Most customer journeys are cross domain. The more you make domain data interoperable, the more insight you have into the customer that can drive better business with your organization as a whole instead of only trying to locally optimize value by domain.
Similarly, many executive questions and desired insights are not exclusive to a domain. Can we get to answering their questions much more quickly and completely? What value would that drive?
To get funding for longer-term initiatives with a payoff down the road like taxonomies, look to directly attach your work to exec's strategic goals. Make it easy for them to see why this will be of value to invest in now rather than later.
If a line of business isn't ready to engage on a key long-term strategic goal like very rich metadata, look for places to make progress that you control. That way, you can capture more value from data being generated now so when you look to reengage, there is more incentive.
When re-engaging with a team that declined to work with you the first time, look to use as much empathy as possible. Yes, they said no but now is the time to say welcome to the party and invite them in. No 'I told you so', only 'great, how can we help?'
It's easy to lose sight of differentiation in metrics. With a good taxonomy and metadata strategy, you can differentiate better between actions - e.g. not how many emails did you send but what type and why? 5 marketing emails in a week is probably bad but 4 order-related emails and 1 marketing email isn't.
It's okay - and probably advisable - to have your taxonomy and tagging be able to serve multiple use cases. A postcard and a wedding invite look the same to manufacturing but are marketed very differently.
A potential way to entice people to participate earlier in a data mesh journey: let them know they will have more say and influence on the general choices. However, you still want domains that are just starting to participate to know that you want their feedback and that their voice also matters.
Think about building interoperability like building a structure. You need everyone coordinating - you want the electrician and the plumber to do their work before the drywaller. Redoing the work instead of setting your build schedule in place doesn't sound like a good idea.
Focus on getting your standards in place for interoperability. Scott Note: no one is publishing their standards and this NEEDS to be done. People need to share their standards explicitly as everyone is reinventing these standards.
When picking early use cases, potentially look for complementary use cases where having information needed for each individual use case can power even further use cases.
Good - even if only rudimentary - taxonomy and tagging lets you easily see how things have changed over time without having to manually stitch data. It also lowers the barrier to external domains leveraging your data.
A key benefit of taxonomies is the ability to tell richer stories. You have data across many different domains or business outcomes but you can see how they interplay.
?Controversial?: Standard tagging and taxonomy can remove - and even prevent - tech debt, partially because it prevents some manual stitching of data.
A good check for teams helping manage a data product portfolio is to take a few stakeholder questions and use what's available to try to answer them. How was the user experience? Is the information easy to understand? Etc.

Jill started off by talking about her current role where she manages a "team of products that span the gamut between data ingestion, data curation, metadata curation and creation, and also [the] upper funnel portfolio around measurement and insight from a demand perspective." So, essentially she is managing a suite of data products around marketing that covers a wide variety of needs, some more technical and behind the scenes and many on the front-line of powering analytics.

Taxonomy can be a significant help to driving interoperability. Taxonomy is ultimately about finding the right balance between flexibility and standardization according to Jill. Much like many decisions in data mesh, it isn't black or white but it's about finding the happy balance somewhere on the spectrum of grays. Having simple tagging and taxonomy has allowed them to see how something like marketing materials - e.g. postcards and flyers - has evolved over time without having to manually connect the data each time - when a new product is launched, it is added to the tagging and product hierarchy.

Jill also believes that good tagging and taxonomy remove - and prevent - tech debt. It means you have an easier time generating new insights without having to do manual stitching on a one-off basis and it creates a much lower barrier to people leveraging the data from outside the domain - if they can understand the general taxonomy, they don't have to be as deep in the context to leverage the data effectively. And taxonomy can help more effectively share metadata to let other domains really understand what a domain is doing.

What Jill and her team saw early in their data journey was that each team was doing a good job tracking their metrics but all the metrics were siloed. Even though they were run independently, obviously the organization needed information to be able to span the different marketing channels. So Jill and team started to really classify and say something like 'hey, it looks like you are doing what you call X and team B is doing something they call Y but we'll map it to Z.' That way, it's far easier to look at a customer journey across domains and silos. You can tell much richer stories.

Where Jill really sees the biggest value with taxonomies is at scale. To start out with, they are helpful but as you add more and more complexity to your data mesh implementation, instead of combining data by manually integrating data products with each other, you can quickly see historical trends across multiple domains to get a much bigger picture. The integration work, at least at the concept level, was done for you. But taxonomies are also not a magic bullet.

A key strategic goal at Vista over the last few years has been personalization according to Jill. So being able to see the big picture of customer journeys and being able to intersect with them with the right offerings at the right time - that sounds like business nirvana. But if you can't see everything that is happening across your many offerings, is that really possible? Or are you preparing yourself with historical data for a potential ML model down the road? Getting the right metadata in place early, that early investment, set them up for value down the road.

Jill talked about how after doing her PoC around what she was trying to do with taxonomy and metadata creation and curation, some lines of business weren't really ready to engage. But instead of putting this taxonomy project on hold, she looked for additional places where she could make progress for when those lines of business were ready, that the value creation at that stage wouldn't have missed all the necessary data in the meantime. Essentially, she looked to avoid the loop of 'it's not of immediate value but we'll get to it eventually' then six months down the road 'oh, if only we had this data over the last six months, then this would be of value.'

Once what Jill and team implemented started to show early wins, they circled back with the business to say 'hey, this is generating value, here's how it saved you some effort', so the business people were more ready. And the teams they had helped already started to advocate for more of the work too. Which gave Jill's team proof points when going to execs and other stakeholders that investing time in metadata and taxonomy is driving value and their participation would drive more value for all.

A key win for Vista according to Jill has been moving from a metric like how many emails were sent to someone in a certain period to what kind of emails - was it all promotional or were some of them based on an order? So getting more granular about the what has helped them reduce email opt-out rates for example. The next phase of that is to look at specific interactions a customer had and develop a next best action model - what can the company do to drive more business that is highly relevant? Again, that personalization. Jill gave a number of additional useful examples of what tagging and taxonomy is driving for them.

Jill also discussed how there shouldn't be one overarching taxonomy that everything should adhere to - you want to look at data from multiple angles. At Vista, customers may be looking for different things - like a postcard versus a wedding invite - but for manufacturing, it all looks the same. And you again want to apply personalization so you don't ask someone if they'd like to create another wedding invite a year later… But, if they are a restaurant that orders menus every 3 months with a change in the season, you want to 1) send them promotions ahead of time and 2) if they are late in their order pattern, potentially escalate to do something additional.

Especially when looking at taxonomies, Jill believes you can entice people to participate earlier by pointing out they will influence the overall choices more than those that come later. So they get more say. But, it's important to balance that as you bring on new teams: their voice matters and their feedback is important to continue to improve and evolve your taxonomies.

Jill gave the example of building a home on why we need to be coordinated in how we generate our data, with things like a unique ID: do you want the drywaller to do their work before or after the electricians and plumbers have done theirs? So getting the coordination done upfront on how you will look to combine data, even if not perfect, can save you a fair amount of time, money, and headaches down the road.

Most customer journeys are cross domain. Jill showed how the more you make domain data interoperable, the more insight you have into the customer that can drive better business with your organization as a whole - globally maximize customer value - instead of only trying to locally optimize value by domain. And many - most? - of your executives, their questions and desired insights are rarely only in one domain. So how are you working to really answer their questions and have that ability to answer cross domain questions?

Jill talked about phase one versus phase two of a data mesh journey. In phase one, you are focused on creating data products to meet specific use cases and it's pretty easy to end up with some overlap. So when you get to looking at your data products as a full suite of data products, those overlaps look like extra cost. So early in your journey, make sure domains are communicating about what they are building to prevent doubled up work. And it will still probably happen and that's okay. But part of product thinking and product management is portfolio management.

Quick tidbits:

A good check for teams helping manage a data product portfolio is to take a few stakeholder questions and use what's available to try to answer them. How was the user experience? Is the information easy to understand? Etc. That will inform future platform improvements and new data products.

"When you're thinking about interoperability, it's just playing nice, right?" If you think of interoperability as a key part of your culture, it's far easier to implement. Let people know why interoperability is good for them and the whole company.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of November 6, 2022 Nov 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#150 3 Years in, Data Mesh at eDreams: Small Data Products, Consumer Burden, and Iterating to Success, Oh My! - Interview w/ Carlos Saona Nov 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/

In this episode, Scott interviewed Carlos Saona, Chief Architect at eDreams ODIGEO.

Some key takeaways/thoughts from Carlos' point of view:

eDreams' implementation is quite unique in that they were working on it without being in contact with other data mesh implementers for most of the last 3 years - until just recently. So they have learnings from non-typical approaches that are working for them.
You should not look to create a single data model upfront. That's part of what has caused such an issue for the data warehouse - it's inflexible and doesn't really end up fitting needs. But you should look to iterate towards that standard model as you learn more and more about your use cases.
?Controversial?: Look to push as much of the burden as is reasonable onto the data consumers. That means the stitching between data products, the compute costs of consuming, etc. They get the benefit so they should be taking on the burden. Things like data quality are still on the shoulders of producers.
You should provide default values for your data product SLAs. It makes the discussion between consumers and producers far easier - is the default good enough or not?
?Extremely Controversial?: At eDreams, you cannot publish data in your data product that you are not generating. In derived domains (e.g., customer history), “generate” includes the derived stitching. NOTE: Go about an hour into the interview - not episode - for more specifics.
When starting with data mesh, there must be a settling period - consumers must understand that things are subject to change while a new producer really figures things out for the first few weeks to months.
You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic.
Be careful when selecting your initial data mesh use cases. If the use case requires a very fast time to market, while it has value, you likely won't have the time and space necessary to experiment and learn. You need to find repeatable patterns to scale in data mesh. Hurrying is a way to miss the necessary learning.
Look ahead and build ahead for obvious interoperability. E.g. create foreign keys for data products that don't exist yet but will.
Be clear about what early data mesh participation means - what will it net domains that are part of the early implementation? And be specific too about what your early implementation won't include or achieve. Don't over promise and under deliver.
Similarly, strongly emphasize that learning is a priority in your early implementation and that you are factoring in learning into promises and estimations. You can't promise you'll find the right solution to a challenge on day one, things need space to evolve as you learn more.
It's okay to not have everyone as part of your initial implementation - engagement or even buy-in - but set yourself on a path where their participation is inevitable.
Making data as a first class citizen doesn't just happen. There is incremental work to be done by the domains. Make sure you reserve time to actually do that work - the data quanta creation and maintenance.
It is not feasible to have your documentation be fully self-describing for everyone. eDreams chose to set the bar at documentation that is self-describing for readers that already know about the domain. For readers that do not know the domain, that introduction must happen somewhere else.
At the start of a data mesh journey, your central team will likely control all the use cases being served by the mesh. But at some point, self-serve needs to happen. Consumers need to be able to serve their needs without their use cases going through the central team.
With small-size data products and the data combination burden being on consumers, versioning tends to not be as much of a problem because the concepts of domain events don't change that often. Or when they do, the retention window has been typically short, making versioning easier.

Other tidbits:

When launching a new data product, there must be a settling period - consumers must understand that things are subject to change while the producer really figures things out.

You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic.

Data products should have customized SLAs based on use cases. You don't need to optimize for everything. Let the needs drive the SLAs.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#149 Data Mesh and Community: The Future? - Mesh Musings 34 Oct 31, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#148 It's A-Okay to Solve for Today: ANZ Plus's Early Data Mesh Success - Interview w/ Adelle McDonald Oct 30, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Adelle's LinkedIn: https://www.linkedin.com/in/adelle-mcdonald-79a9a2139/

In this episode, Scott interviewed Adelle McDonald, Customer and Origination Lead at ANZ Plus, a bank in Australia and New Zealand.

Some key takeaways/thoughts from Adelle's point of view:

To drive buy-in, the 1:1 conversations with domain owners - the business leaders - you will have to tailor your conversations to each person. Listen to their pain and reflect it back to them.
Focus on an ability to quickly pivot with low cost. That can mean things aren't as product-worthy to start but it means you can evolve towards value more quickly.
Addressing domain owners' pain points gets them looking at you as a partner. They will be much more willing to work with you, especially as you partner to provide actionable insights.
?Controversial?: ANZ Plus is embedding data leads into domains to handle the data quanta for the domain and also build the team what they need from data. As part of that, they are slowly building up the domains' capabilities to handle their own data. This minimizes friction and creates buy-in but is likely not long-term sustainable - ownership will need to be transferred.
Very important to tie the data quanta to use cases - driving value for users means focusing on use cases.
Developers or software engineers owning data is complicated. Make it so they can start to make small changes and learn in a safe way instead of dumping all ownership on them at once. Ownership and knowledge aren't a switch you flip.
Using a git-based, pull request approach, developers can attempt data work without manual stitching so they learn to do the work themselves; but it can still be easily overseen by someone with more data expertise.
One way to potentially drive executive buy-in is joint/collaborative KPIs. So it's not just about their domain's results but how well they work and drive results with another domain.
?Controversial?: It's okay to have a data asset with murky long-term ownership at first. If usage picks up, you want to convert it to a proper data quantum but we need to be able to test the waters with things and see if they are actually useful first. Clarity comes with usage.
When creating anything data related, use a software development lifecycle (SDLC) approach. Domains may create something exclusively internal to the domain but once you look to share externally, you have rules and standards and best practices. Move from the pipeline approach to the software approach to data.
Automatically generated documentation can considerably help with governance. You have it in the same repo and it can handle a large part of explaining what is happening with the data to make other decisions easier.
Automate your governance checklist as much as possible so you prevent the manual work of governance. No gates, simple checks, that's a winning, low friction way to govern.
When you don't automate your governance checklist, domains often feel they need to invent or buy the tooling to comply with governance. Making it just no friction to check through automation means far less complexity and fewer issues.

Adelle started the conversation on driving buy-in and how important it is to tailor your message. As prior guests have also noted, the easiest way to drive buy-in is by helping out the person you are trying to get bought in as part of the process. So find their needs and help drive to a positive outcome for them with data first.

When it comes to getting buy-in from domain owners, Adelle has seen finding their pain points and finding good ways to address those pain points will get them to see you as partners to leveraging their data. They will be much more willing to work with you rather than you simply putting new responsibilities on their plate. You can work with them to ensure their information - especially purchased data - is providing value and gives people actionable insights, not just interesting insights. It may be a tough pill to swallow but you need them to see you as that partner in the long run.

At ANZ Plus, they are embedding data leads into the domains to be the main point of data contact to external domains, according to Adelle. Those data leads are serving the domains by helping to really address their internal business needs with data while also creating the data sharing mechanisms - the actual data quanta in data mesh terms - for sharing that domain's data across the rest of the organization. With this work falling on the data leads, ANZ Plus is not generally asking for domains to take on too much responsibility relative to data - at least not yet. This minimizes the work the domains have to take on but still significantly accelerates the time to getting to business value within the domain for new data use cases. This means the domain owners have been very happy to work with the data leads because there isn't much incremental work they are responsible for - at least at first.

Adelle and the data mesh fans/leaders at ANZ Plus are aware that their data ownership model is probably not the right fit in the long run. But it's working well for them right now and that's what matters to them. They have found a setup that doesn't add a ton of overhead process and will evolve as the capabilities and resources to hand over actual data ownership get built out more and more. When they evolve, they are focused on maintaining the ways of sharing context rather than trying to keep the exact data quanta as is. But if your organization isn't very clear that things will constantly evolve, this could be a very hard setup to maintain in the long run.

As part of their plans for evolution, Adelle mentioned that they are focusing on maintaining the ability - with a low overall cost - to pivot. As the world around them changes and evolves and as they learn more by taking actions on the actionable insights they generate, to drive more value, the team needs to make sure they are evolving along with the world and their markets. They are focusing on getting to the right insights as fast as possible in sustainable ways.

Thus far, Adelle and team are finding it's very important to tie use cases to data quanta. To drive value for customers, they need to focus on the use cases. This is especially relevant when looking at customer journeys - you want to set yourself up to collect the right data to analyze to understand what's going on with that customer journey.

Adelle emphasized the need to create an environment for safe evolution by developers relative to data. At first, developers won't know how to deal with data in general as an entire concept but most will understand at least some aspects of working with data. So how can you get them more and more used to dealing with data and learning more? By providing them a way to make changes when they need in a safe and easy way. Easier said than done but teaching developers and software engineers to deal with data isn't a switch you flip, just like handing over ownership. It's best done as a gradual process. Easier said than done of course.

According to Adelle, it's okay to have somewhat murky data ownership at first for a new potential data asset. If it starts to get broader use, you need to lock-in who will own it, how, and why but you don't need to get ahead of yourself and drive towards a perfect data quantum each time you look to share data. Have high context exchange with other users, let them know how much they can trust things, but also be in a mode of trying it out and seeing where it might go before investing the time into data quantum creation. This gives domains more freedom to play with their data but data consumers also must be flexible around what they are consuming might evolve as it gets molded into something more scalable, usable, trustable, etc. Without an ability to evolve quickly, this model will likely not work.

It's crucial to think about data like any other software development for Adelle. Your software development lifecycle (SDLC) needs to have things like governance and API interfaces as part of the development. And, as an example on the operational side, domains can build small internal apps that aren't good for external domains as they figure them out and test. But once other domains need access, then you have to start treating your sharing access like a product. The same goes for data. You should have standard practices internally to make this low friction.

Automated documentation has been a big win for governance according to Adelle. While you still need additional documentation, if you can have the base level documentation auto-generated in the repo with your data quantum code, that's very helpful for sharing what you are doing with data and why. It also means it's easier to make other governance decisions because people can see what is happening, what is the information about. They've automated much of their governance checklist as part of their software development lifecycle as well so people can test against if they meet governance requirements as they are developing, not as a gate at the end.

A few tidbits:

Only collect data for a specific reason. If you don't know specifically why you are collecting it, why are you collecting it?

Making incremental data work requests a pull-based system (think pull requests in git), you can work somewhat asynchronously and developers can learn to attempt data work in a safe environment.

Before investing in doing data work, ask if it is recurring and what is the value, why are we doing this. If it will be recurring work, look to automate as much as possible first. It might not yet be data quantum worthy but it sets you up for when that time comes.

One way to potentially drive executive buy-in is joint/collaborative KPIs. So it's not just about their domain's results but how well they work and drive results with another domain.

When you don't automate your governance checklist, domains often feel they need to invent or buy the tooling to comply. Making it just no friction to check against governance requirements, as part of the SDLC, means far less complexity and fewer issues.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 30, 2022 Oct 30, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#147 Mapping Out Your Data Product Suite - Building Your Roadmap to Maximizing Business Value - Interview w/ Gunjan Aggarwal Oct 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Gunjan's LinkedIn: https://www.linkedin.com/in/gunjanaggarwal/

Gunjan's Medium: https://gunjan-aggarwal.medium.com/

In this episode, Scott interviewed Gunjan Aggarwal, Head, Digital Data Products and MarTech Strategy at Novartis. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Gunjan's point of view:

Set your overall data product strategy - for when you are in stage 2, going wider with data mesh - earlier in your journey than many may think. It's easy to focus only on use cases instead of the bigger picture.
Make sure to align early on who owns what - what are the clear boundaries between roles. Otherwise, with the amount of change data mesh drives, there will likely be unnecessary chaos. Get specific.
Don't fall to the 'Data Field of Dreams' - "if you build it, they will come." Focus on building to actual problem statements. Involve people early, make them accountable, give them skin in the game and they will care.
"The more you ask why, the more clarity you will get." Really dig in deep into the reasoning for creating new data products or ARDs (analytics ready datasets). If we have this data product, what will it unlock for us?
It's crucial to avoid the trap of building data products specifically to use cases. You must have the bigger picture in mind and focus on reusability instead of only solving one set of challenges. Can you extend an existing data product?
Data people should have domain knowledge where possible. That way, they can push back on requirements that don't make economic sense, that don't maximize the return on investment.
4 part approach to designing data products: 1) find clarity on the problem statement; 2) assess who are the personas that will benefit from it; 3) dig into what you already have available; and 4) focus on serving value to the problem statement in a way the persona can use.
?Controversial?: scalability is more important than time to market when it comes to data products, especially as you develop a broader set of data products. Tech debt around scaling is hard to combat just as you are delivering strong value with the need for scale. And it limits additional use cases leveraging existing data products if they can't scale.
Look to provide as many easy paths as possible for new data products. Templates, blueprints, standard schema, a global taxonomy, etc. They don't have to use them but they are great starting points.
You need to be proactive in partnering with the business. Data people have historically waited for requests/requirements. That won't lead to fast feedback loops and quickly iterating towards value.
It's easy to end up focusing on the single use case instead of the bigger picture. But this will likely result in business disruption from a half-baked product.

Gunjan is in phase 2 of a data mesh implementation at the moment, the going wide phase. As part of that, she's looking at how do you create a suite of data products to serve the needs of a broader set of use cases and look at putting things in place to more easily serve more ad hoc querying. She recommends setting your data product strategy for what will be your long-term needs earlier in your journey than most might think - what is the real business strategy for your mesh as a bigger entity than just individual data products? How will they actually work together so 1 + 1 = 3?

According to Gunjan, it is very important to clearly define boundaries and responsibilities for roles. It's easy to get confused about what is a data product owner versus a product manager, for example. Look to the RACI model for defining things clearly. If there is a lot of change and unclear responsibilities, that can cause lots of challenges and chaos. If you don't have alignment early, it's very easy to go wrong. So make sure you spend time before moving forward with data mesh to really focus and align on why are you doing this and who will own what. Start with the end goal in mind and march forward together.

It's crucial to make sure you involve people early in every data product you develop in Gunjan's experience. If you build something for them instead of building it with them, they are far less likely to buy in. Make people accountable, that their data product "is their baby". Make them part of defining success for a data product and work with them to make sure they can scale it up when it succeeds.

For Gunjan, when considering new data products, always start with asking why. Why do we think this will drive incremental value? Why is this the right time? Dig in layer by layer to understand why is this a good use of our time and what is our expected business value from doing it. It's easy to miss the forest for the trees. This method also makes finding reusability more likely. Why can't you use what is already built? And if there is a good answer, make sure to build your data products so they are reusable for other use cases down the line.

On digging deeper into reusability and extensibility, when you look at new use cases, consider if you need new data products to support it or if you can modify and extend existing data products instead. It's quite easy to build data products to try to support each use case individually but it will quickly overwhelm your teams. Look at the greater whole for how you can support your needs with a suite of data products.

Where possible, Gunjan believes it's best if your data people have domain knowledge - they can push back on the cost/benefit of choices far better than someone without the specific knowledge. Do you actually need real-time? What is the impact of different SLAs to the return and the cost to create/maintain? It's far easier to maximize return on investment if one person understands both sides of the equation.

Gunjan and team have a four part approach to building out new data products: 1) "Find the clarity on the problem statement". What exactly is the use case you will be serving? 2) "Who are your personas you are serving and what will they require?" Get specific around who you are trying to serve. 3) "What is available?" Dig into what you already have and evaluate if what exists can serve the use case(s). 4) Focus on providing the value to serving the problem statement in a way the personas can benefit.

In the long run, Gunjan believes the ability to scale up is more important than speed to deploying a data product. When you rush to create data products, you will inevitably create a lot of tech debt, especially to scaling when the time comes, so focus more on ability to scale. It might not feel like that at first but it's especially important when you have many data products. It will be very frustrating when you have additional use cases and cannot easily scale because you wanted to release your single data product 2 weeks earlier. And focus on prioritization as well. Some use cases will have to wait and that's okay.

According to Gunjan, you need to get pretty close to your domain business partners. Embed yourself in more business meetings discussing use cases and problem statements. You shouldn't be waiting for requests, you should be extracting that information in regular discussions. Fast feedback cycles leading to fast iteration is crucial.

There is no such thing as a future proof tech stack. So Gunjan recommends 1) accepting that and 2) preparing yourself for graceful evolution to meet needs. You should always be asking what are the risks to your platform and how can you mitigate them. Modularize your tech stack so you can easily add and/or replace when necessary.

In wrapping up, Gunjan talked about the importance of not focusing on the single use case but how it plays into the bigger picture, the long-term. It's easy to go down that path of focusing on the single use case, especially as you start out, but it will cause disruption to the business from a half-baked product. Product thinking, not project thinking.

Quick tidbit:

Easy data product templates and a good centralized catalog are very important. Having a good catalog can help you identify gaps in your data product coverage as well.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#146 False Dichotomies and Baseless Binary Choices - Why We Need New Thought Approaches in Data - Mesh Musings 33 Oct 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#145 From Failwhale to Massive Scale and Beyond: Learnings on Fixing Data Team Bottlenecks - Interview w/ Dmitriy Ryaboy Oct 23, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Dmitriy's Twitter: @squarecog / https://twitter.com/squarecog

The Missing README book: https://themissingreadme.com/

Building Evolutionary Architectures book: https://www.oreilly.com/library/view/building-evolutionary-architectures/9781491986356/

In this episode, Scott interviewed Dmitriy Ryaboy, CTO at Zymergen and co-author of the book The Missing README.

Some key takeaways/thoughts from Dmitriy's point of view:

Organizational design and change management is "like a knife fight" - you are going to get cut but if you do it well, you can choose where you get cut. There is no perfect org, and there will be pain somewhere but you can influence what will hurt, and make it not life-threatening.
There is too much separation between data engineering and software engineering. Data engineering is just a type of software engineering with a focus on dealing with data. We have to stop treating them like completely different practices.
When communicating internally, always focus on telling people the why before you get to the how. If they don't get why you are doing it, they are far less likely to be motivated to address the issue or opportunity. This applies to getting teams to take ownership of the data they produce, but also to everything else. There is often a rush to use tech over talk. Conversation is a powerful tool and will set you up so your tools can help you address the challenges once people are aligned. Paving over challenges with tech will not go well.
Build your data platform such that the central data platform team is unnecessary in conversations between data producers and data consumers. That way, your team won't become a bottleneck. Try to reduce cognitive load on the users - they shouldn't have to deeply understand the platform and its inner workings just to leverage it.
"Data debt is debt forever." You can certainly 'pay it down' but data debt typically has a much longer life than even the initial source system that supplied the data. Take it on consciously.
Looking to hire or grow full-stack engineers for an ever-growing definition of stack (backend, frontend, security, ops, QA, UX, data...) is probably not a great idea, we can't keep piling new domains on people and expect them to be good at all of them. Instead, look to build full-stack teams, and tools that look and feel sufficiently similar that e.g. "data engineering" becomes very close to "backend engineering".
Look for needless delays in work as a sign your organization isn't well aligned with what you are trying to accomplish. The cost of coordination should not be a huge bottleneck, especially at a smaller size organization.
Any agreement like a data contract or API needs to be agreed to by both parties. Consumers can't just expect things to not change, especially if they don't let producers know what, why, and how they are consuming their data.

Dmitriy shared some of his experience leading the team building and scaling Twitter's initial data platform. They had to transform from a relatively simple model that just wasn’t scaling to one that was far more scalable but obviously more complex. E.g. their MySQL setup, they couldn't add any more columns so they treated certain columns almost as a CSV within the cell. You can imagine how difficult that was for analytics… They also had many hidden dependencies as well so when changes were made, it would break all kinds of other systems. So part of his role was helping the team untangle that mess.

One thing they got right according to Dmitriy was moving to structured logging so there were at least standard columns that made it possible to combine data across applications. Dmitriy didn't have any specific advice on exactly how to standardize but mostly about the discussions around standardizing - always share what you are trying to accomplish, focus on the why much more than the how. The how is based on decisions once you've realized the why. This advice came up throughout the conversation

Twitter had a small central data team that had to either try to scale massively and still not be able to know all the business context necessary or - more sanely - build their data platform to keep themselves from being a bottleneck. They chose the second option. So when building their data platform, there was a focus on building tools to make the central data team unnecessary to most data conversations, getting the data producers and consumers to talk to each other instead of playing interpreter and doing the work themselves.

So, to sum up some advice from what Dmitriy learned from Twitter: 1) go for structured logging, 2) make dependencies clear and known, and also limit where possible, and 3) build your data platform tools so your data team doesn't have to be part of the conversations between data producers and consumers to enable scaling.

According to Dmitriy, when focusing on change management, again you have to focus on explaining to people the why of what you are doing. When you give people the reason, the pain you are trying to solve, most people then just want to know what you need from them in his experience. And be prepared to reiterate the why a lot. It's better to state it too often than not often enough.

When asked what we can take from software engineering to apply to data mesh and data engineering, Dmitriy is very passionate that data engineering and software engineering really shouldn't be overly separate. Data engineering is just a type of software engineering and we need to create the tooling to make software engineers capable of doing data engineering tasks. We can take a lot of learnings from software engineering and apply them to data but it shouldn't be seen as very different.

When thinking about data mesh especially, we are asking domains to pick up additional responsibilities. Dmitriy believes - and Scott has said it multiple times too - you can't give a team more responsibility without giving them more resources. That can be people - especially with incremental capabilities - or something like a platform that reduces the cognitive load and workload of the new - and existing - responsibilities. Give people a platform that they don't need to be experts in what it is doing under the hood to actually leverage. And full-stack engineers, especially if we add dealing with data, are just going to be overloaded. Look to full-stack teams where people within the team have some specialty areas.

Dmitriy shared some thoughts about how Zymergen's data capabilities and team has evolved. It was a central team that had ownership over everything which hampered the teams from talking to each other as much as needed. That introduced needless delays because the coordination cost was so high. So they reorganized and had a number of the data engineers move into the domains. Those data engineers were better able to leverage the data platform because they had built it and could teach other domain team members how to leverage it better too.

A lot of issues can be handled with just a conversation, not a technical solution, according to Dmitriy. This might seem obvious but so many people try to not go down this road. But technology is obviously also important. Think about when to use tech and when to use talk. Your team should build out the platform to make it easy to have better conversations because the tech is handled.

"Data debt is debt forever," according to Dmitriy. Meaning that data has a habit of sticking around for a very long time, even well past when the initial source system has been replaced. You can pay down that data debt, but you have to address it intentionally. A simple refactor usually won't do it. Right now, it's far easier to update an API, we need to get there with data where things like versioning aren't such a manual, pain-in-the-butt task.

Dmitriy ended with a few things: 1) indirection is your friend but don't use indirection until you need it; 2) a good place to learn about building your architecture to be able to evolve is the book Building Evolutionary Architectures (kind of well titled, huh?); and 3) organizational design and change management is like a knife fight - you are going to get cut but if you do it well, if you are a pro, you will choose where you get cut.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 23, 2022 Oct 23, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#144 Zhamak's Corner 7 - Oh, The Places You'll Go! How Can Imagination Lead Us Forward in Data Oct 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#143 Data Ethics is FAR More Than Just Bias - Mesh Musings 32 Oct 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#142 How to Leverage an Empathetic Ear to Add Value Through Data Governance - Interview w/ Karin Håkansson Oct 16, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Karin's LinkedIn: https://www.linkedin.com/in/karin-hakansson/

In this episode, Scott interviewed Karin Håkansson, Data Governance Lead consulting on a large data mesh implementation. To be clear, Karin was only representing her own views on the episode.

If there is one theme that resonated throughout the conversation, it was have more and deeper conversations with your data governance teams. Don't make assumptions and as a data governance team, don't make decrees - explain what you are trying to accomplish and why. And look to help where possible instead of telling people how to do things.

Some key takeaways/thoughts from Karin's point of view:

Data governance teams need to get closer to both data producers and consumers; but when doing that, emphasize it isn't about control or oversight. You are there to help.
Data producers and consumers need to share with the governance team about their pains and challenges - the governance team in most orgs is there to help. So get specific about where you need help. Emphasis on specific!
To get closer to data producers and/or consumers, identify a few obvious data problem areas and just reach out to those involved/impacted. Focus on asking not telling when digging into the pain points. And involve all stakeholders in assessing potential solutions to get them bought in to the eventual solution.
A control-based data governance approach doesn't add value to your data. We need new ways of working in data governance focused on adding value, not adding roadblocks.
Data mesh creates an environment where we need to look at things differently. And it provides the excuse to look at them differently too. Challenge your base assumptions.
If your organization isn't open to finding and implementing new ways of working, it will be challenging - at best - to implement data mesh. Really ask if you are ready for the change before implementing it.
Data governance teams need to focus on finding long-term solutions and balancing those with temporary fixes. Be extremely clear about what is a temporary fix and why it's temporary and must be replaced.
Always keep people informed as to progress - even if it is "no progress". Humans aren't good with uncertainty.
Data governance teams can easily be overwhelmed if you don't develop good ways to prioritize. Find the most important - not necessarily the most obvious - pain points and assess the value of fixing them.

Karin started the conversation sharing her background - before being "pulled" into the world of data governance - as a software engineer, data engineer, and more. She's always worked with data but only recently started focusing on data governance. Her prior background has given her the perspective needed to understand problems from the perspective of both data producers and consumers and how to best help them as the data governance team.

So, with that perspective, Karin is focused on finding new ways to get closer to both the data producers and consumers - but focusing on how that closer relationship serves as a helping hand, not in an oversight or control standpoint. And now seeing things from the data governance team's perspective, she and her team are there to be helpful and add value.

When asked how she recommends getting closer to data producers and consumers, Karin said finding a somewhat obvious data challenge - where are people complaining loudly? - and initiate a conversation as someone who might be able to help. And when you do get to speak with the stakeholders, focus on listening - ask, don't tell. Really dig into the pain points instead of trying to jump to a solution as quickly as possible. Usually the pain points are pretty well known. Start from a blameless perspective - choices were made and we need to fix the situation, not assign blame. And by involving the business people in the conversation - especially in assessing the solutions - they are more bought in to the outcome because they helped pick it.

In data mesh, we have to do data governance a bit differently according to Karin. No longer the ever watchful control tower - or Sauron's Eye -, we need to find ways to make data governance agile and flexible - Laura Madsen focused on this as well in her episode. A data governance approach based on control won't actually add value to your data - and why have governance if it isn't adding value? Historically, a lot of data governance teams have focused on creating a standard way of working instead of standard outcomes. Governance people need to involve stakeholders in determining ideal outcomes, not have a checklist-type approach. Data producers and consumers running into issues should be able to go to the governance team for help; and they should ask what the governance team is trying to achieve and they might find better ways of working together.

For Karin, data mesh creates an environment where we need to - and have an excuse to - look at things differently. We must challenge our base assumptions and, where necessary, look for new ways of working. But we also have lots of existing approaches that will work as well. We need to evaluate and be open to change when it is useful. Before embarking on data mesh or evaluating a major change in ways of working, you should ask if you are really ready for that change.

While the implementation Karin is working on hasn't deployed a considerable number of data products, what she's seen so far relative to data product ownership is promising. When they are really giving people full trust in owning their data, she believes they will step up. But you need to make sure they understand the new responsibilities, the new potential roles, and the value the data products bring to the organization. And drive more buy-in by helping them find additional value in understanding their internal data consumers - it might lead to additional insights for their own domains.

In wrapping up, Karin shared her thoughts on implementing temporary fixes - it's very easy for a data governance team to run around trying to do temporary fixes but it won't scale. Temporary fixes can have value but you must look for longer-term solutions to replace them. And be explicit with stakeholders about what is and what isn't temporary. As you build towards long-term solutions, always keep your stakeholders up-to-date on progress, even if that is "no progress". Humans don't do well with uncertainty. Data governance teams can become easily overwhelmed - you can't do it all. You must find a good way to prioritize - find the most important pain points and assess the value of fixing them. And really consider value, not just how much pain it seems to cause.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 16, 2022 Oct 16, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#141 Effective Data Storytelling to Secure Funding - Pop Your Data Bubble and Stop the Data Babble - Interview w/ Scott Taylor Oct 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Scott's LinkedIn: https://www.linkedin.com/in/scottmztaylor/

Scott's Website: https://www.metametaconsulting.com/

Scott's Book: https://technicspub.com/data-storytelling/ (save 20% off with code DATAWHISPERER)

Scott's YouTube playlist: https://www.youtube.com/playlist?list=PLashWxBySOAOKUtvn2NTBQeENJTLB8NoS

In this episode, Scott interviewed Scott Taylor AKA the Data Whisperer.

From here forwards, "Scott T" will be used to represent Scott Taylor to differentiate from your host. It's also important to note Scott T differentiates 'data management' from 'analytics', so things like data governance and infrastructure fall under data management.

Some key takeaways/thoughts from Scott T's point of view:

To actually reliably successfully obtain funding for data management initiatives, you need to articulate the value of the work to the business - why does this matter? Storytelling is key.
The quickest way to lose your shot at securing funding for data initiatives is to focus on the how instead of the why. Most execs do not care at all how it gets done. They hired you, the data leader, to handle that.
To get funding, focus your messaging on tying your data strategy and proposed data initiatives to the business strategy. Why does this matter - how will this drive the business strategy forward?
Be prepared for cynicism from the business side - many have heard about how this or that data/technology initiative will be the silver bullet for far too long.
Digital transformation is an enormous opportunity to change how your organization deals with data. Use that as a good point of leverage for funding your data management - it's a necessary foundation for a digital enterprise.
'Story' is about constructing a narrative and 'telling' is about effectively communicating. So storytelling is about effectively communicating what is the target outcome and why will this drive value.
Data storytelling is an art, not a science. You need practice to get good at it. Don't make your 'practice' be overly high stakes. Talk to a lot of folks on your side to hone your talking points. You don't publish the first draft of a book do you? Find your editors.
Focus on empathy and listening to find your audience's aspirations and pain points. Then tie your messaging about your data initiatives to those aspirations and pain points.
When you get to a yes, stop talking. Is your point to get the yes or is it to tell them all you planned to tell them?

Scott T shared a bit about his history and seeing the same patterns repeatedly, especially the need effectively communicate to get funding for data initiatives. To actually obtain funding for those data management initiatives, you need to articulate the value to the business of the work. And many data people do that pretty darn poorly if at all.

Where do things typically go wrong in getting funding for data management? Per Scott T, it's when the person asking for funding focuses the conversation on the how - e.g. how exactly are we going to implement data governance? Generally, the audience cares only about the why. Why is this important? They want to trust you to execute on the how. The how is crucial, but that's why they hired you - to take care of the how.

So Scott T recommends again starting from the why. Show them how this data management initiative will enable the business strategy. Because why have a data strategy if it doesn't drive the business strategy? Scott T has seen a number of times where data folks don't focus on how the data strategy ties to the business strategy and the perception - and often the reality - is that the data folks don't really understand what the business does and/or what drives the business results. That's obviously going to hurt chances of getting funding.

So what actually is data storytelling and how does it work? According to Scott T, there is the storytelling with data - which is not his focus - and storytelling about data, that data management side. Breaking it down into 3 component parts. Data - you are talking about data initiatives. Story - constructing a narrative. Telling - how you actually communicate what you are doing and why. It's an art not a science unfortunately but gets much better with practice. You need to get your audience emotionally attached to the outcome to get them bought in to funding it. This isn't a movie or a novel, you don't get a chance at exposition - it's a pitch, get their attention. And be prepared for - and welcome - the interruption. It means they are at least engaged if not bought in just yet. Your objective is to not get through all your slides because they were too interested.

To prepare your pitch for your data initiatives, Scott T recommends you really focus on what the decision makers are talking about - both their aspirations and their pain points. Then look to tie your data initiatives to those. What are they trying to do and where are they trying to take the business? How can your initiative get them there faster/better/cheaper/etc.? What are the problems they are having? Talk to those exact pain points and how your data initiative addresses them. What are the strategic intentions of the organization? This will take a fair bit of empathy and learning how to really listen and then, again, practice to tie your pitch back to their aspirations and pain points.

Scott T shared his 3 'V's of data storytelling - playing off the 3 'V's of big data -: vocabulary, voice, and vision. You need to combine all three to effectively tell your story. Don't talk in data language, use the language of the business. Make it an interactive conversation, not you talking at them. Show them why this matters and why doing your initiative will drive business results. Again, your data strategy must be about the business strategy.

Scott T finished on his mantra: Truth before meaning.

Quick tidbits from Scott T:

Be prepared for cynicism from the business side - many have heard about how this or that data/technology initiative will be the silver bullet for far too long and they haven't delivered.

Digital transformation is an enormous opportunity to change how your organization deals with data. It's a major potential accelerant for the business. Use that as a good point of leverage to pitch your data initiatives.

There are 4 ways data can add value according to Scott T. 1) grow the business, 2) improve the business, 3) protect the business, and 4) sustain the business. Be clear on which ones your initiative focuses on.

There are a number of sales and marketing techniques people can learn to become effective at data storytelling. You want to take your audience from "what are you talking about" to "we can't live without this". But it takes practice.

Data management is the less attractive side of data and analytics but it's the foundation. You can't build a stable building, a stable home without a good foundation.

When you get to a yes, stop talking. Is your point to get the yes or is it to tell them all you wanted or planned to tell them?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#140 Zhamak's Corner 6 - Bypassing the Binary Bias and Crucial Learnings from the Operational World Oct 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Inverse Conway Maneuver Definition: https://www.thoughtworks.com/radar/techniques/inverse-conway-maneuver

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#139 Reflecting on Learnings from Glovo's Early Data Mesh Journey - Interview w/ Javier Granda and Pablo Giner Abad Oct 09, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Glovo's meetup group: https://www.meetup.com/glovo-tech-talks/

Javo's LinkedIn: https://www.linkedin.com/in/javiergrandag/

Javo's Twitter: @JavierGrandaG / https://twitter.com/JavierGrandaG

Pablo's LinkedIn: https://www.linkedin.com/in/pabloginerabad/

In this episode, Scott interviewed Pablo Giner Abad, Global Director of Data and Javier "Javo" Granda, Senior Data Manager at Glovo.

From here forward in this write-up, P&J will refer to Javo and Pablo rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from P&J's point of view:

It's okay to not fit the exact or complete picture of data mesh in your early journey. Focus on what matters to your org and implementation and focus on learning over trying to be perfect. Iteration is possible and not too costly with data mesh. That's sort of one of the main points of data mesh.
When selecting your first use case, look for high value and low dependencies. The less cross-team coordination work needed to actually get to an initial end data product that has value, the better. And buy-in is much easier if the producers are one of the consumers too :)
When starting out, really look at how thin of a slice you can get away with for your MVP. Be prepared to make some hard compromises. Make them with your eyes open. It's tech debt but taken on consciously.
Focus on solving your problems of today instead of trying to solve all your future problems. Fixing the challenges of today will set you up to fix the challenges of 6 months from now in 6 months.
Focus on reducing cycle times to creating and iterating on data products more than you probably think you should. It's easy to get focused on delivering new data products instead of the capabilities to deliver new data products but that will cost you more and more as your data mesh implementation matures.
An important quote to remember re product thinking: "If you aren't embarrassed by the first version of your product, you shipped too late." Your data products don't need to be perfect when launched.
Just using your domain mapping from your operational/DDD side as your data domain map is likely to lead to some big challenges. Look to how your data flows to figure out good data domain mapping.
Misaligned domain maps between the operational side and data side can also cause issues because a team may need to own the data domain but they don't necessarily own the operational system or domain.
Glovo's biggest data pain point pre data mesh was that data quality was often not great and people spent huge amounts of time trying to check the data. If that's a challenge at your org, you can probably get funding to fix it.
If data trust is a key pain point, when you make compromises early, do not compromise on quality - or at least very clear communication on what quality means. That is the only real way to gain back people's trust - actually provide them high quality, trustable data.
Domain ownership is likely to be the most challenging data mesh principle in many organizations. Partially because it is the one that is most centered on change management.
It's easy for a central data team to fall into prioritization by loudest escalations. When moving to data mesh, make sure you don't fall into the same trap anywhere. Make conscious decisions based on value not on loudness.
It will be important to define your minimum viable data product. Is that the data shaped into a format consumers can use? Does that mean if target users are not that data literate, ownership extends into the visualization tooling? Hard to say what is right for every organization.
Only produce a data product if there is a known use case. But once a data product is created, owners should think about how they might serve additional users.
There needs to be more specific examples of what people's early platform builds look like. It's really tough to think about every capability and what might fit where.

P&J started out the conversation sharing about Glovo's history with data - they have always had a lot of data and been data heavy but how they handled data was not very structured; they didn't focus much on the data architecture. They treated the data warehouse like a data lake, dumping things in with little to no data modelling. People didn't trust the data so they spent huge amounts of time checking data quality - and the company made some crucial decisions on data that wasn't all that great.

Their data architectural choices were creating more and more bottlenecks as well according to P&J because everything was reliant on the central team to build and fix. They clearly couldn't scale to meet the company's needs. They built out the tech to try to support needs remove bottlenecks based on technical throughput/scale but the central team itself was never going to scale to meet their needs.

Per P&J, in a way, their problematic setup was helpful for data mesh buy-in because they had so much tech debt, it was easier to convince everyone they needed to move to something different rather than try to incrementally improve. Escalation by who screamed loudest just wasn't working and the central data team was falling more and more behind. There was a strong demand for understandable reliability - what was the actual quality level? - and clear ownership of data.

Rather than trying for a short-term fix, they looked to build their long-term data strategy according to P&J. Where did they want to go with data. Data mesh came at the right time and gave them the structure they needed to start really identifying their issues and setting their forward vision. They knew they needed the agnostic platform to serve the broader company instead of point solutions. They wanted to really apply product thinking to data. Federated governance and domain ownership of data were more new to them so it has been harder to really figure those out.

P&J shared their initial plans which fits a common theme in Data Mesh Radio episodes: make a thin slice of every aspect they'd need - but make some hard compromises. They knew they needed a platform that could at least read and process data. Their first few data products were more shared ownership between the domains and the central data teams. Etc. It's okay to not fit the complete picture of data mesh when you get going.

The business leaders didn't really care exactly how it got done, only that it got done according to P&J. So they chose security and quality as their main focuses. Quality was very crucial to regain data consumer trust so they created tests to show what were the expectations and that the data products were actually meeting those expectations. Security was basically to protect PII - they didn't need anything too complicated so they only built to what they needed.

On picking their first use case, P&J and team looked for a use case that had high value and low dependencies. The more dependencies, the more possible complications and ways to fail. So they chose customer interactions in the app - what were people actually doing in their app? Previously, to get at the data, it was very difficult because of many complicated combinations of data. The producers for that first use case really didn't have to do too much, the data was already being created in a format that was close to usable. And the producers were also the consumers, which obviously made it much easier to drive buy-in.

At Glovo, every data set, every data quantum must have a clear owner but they did save a lot of that change management pain until later. They didn't force the data producers to really own their data products and had the central data team really as the owner or at least co-owner. They are now pushing that ownership on to the producers and there is a fair amount of friction.

P&J brought up the quote from Reid Hoffman, who founded LinkedIn: "If you aren't embarrassed by the first version of your product, you shipped too late." So what is a minimum viable data product at Glovo? It is a data set, a group of tables, that has some amount of enrichment and structured to be used by the target data consumers. But they found that business leaders didn't really care about data products as units of business value until they connected the visualization tooling. So if it's not really usable by the target customer, is it really product-ready? How far into how data consumers use the data products does ownership extend?

As other guests have mentioned, P&J agree you should only produce a data product if there is a known need. But prior to data mesh, teams were only serving the needs of the teams closest to them in the organization. Now, data producers are thinking about who all could use the data and finding interesting new consumers, uncovering new potential use cases.

When asked what they wish they had been told when they started their data mesh journey, P&J shared a lot. One is to focus on the problems you have now and set yourself up to focus on the problems six months down the road when you are six months down the road. This was especially an issue with the platform team not serving what was needed now to try to build out the capabilities to support future use cases instead of the current use cases. Another is to focus a lot on capabilities - Glovo was focused on delivering data products but they wish they had focused on reducing cycle times to creating new data products. Another few: building cross-functional teams is crucial but always challenging; you need to be prepared to communicate more frequently than you probably expect and repeat yourself often. And lastly domain ownership will likely be a challenge in many cases.

P&J discussed how the data team "inherited" the existing domain map and they are still struggling somewhat with mapping out their data domains and how they differ to their operational domains. Just following the operational domains caused a number of challenges as it often didn't align with how data was flowing through their systems for analytical use. Mapping out your data flows is crucial to establish the right data domain ownership. But then shifting data ownership to other domains is also a challenge. Not an easy thing to solve unfortunately.

Domain ownership is causing a lot of issues currently - data ownership is typically not an expected responsibility for domains per P&J. Changing ownership from outside the operational domain to the team - because it is in their data domain, is also very challenging. They are struggling to really define each data domain and why these data domains are needed instead of just using the operational domains. So P&J are asking for more people to create content around domain ownership.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 9, 2022 Oct 09, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#138 Getting Started with Data Mesh by Collaborating with the Business - Interview w/ Darshana Thakker Oct 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Darshana's LinkedIn: https://www.linkedin.com/in/darshana-thakker/

In this episode, Scott interviewed Darshana Thakker, Architecture Director at BCG Platinion.

Some key takeaways/thoughts from Darshana's point of view:

Data can be the enabler to achieving your business goals but only if you actually tie your data work to your business strategy and goals.
Speaking data jargon when talking to the business stakeholders makes it hard to actually communicate. Focus on speaking to business outcomes and business value.
When selecting your first use case(s) for data mesh, evaluate domains on four different metrics: business value, capabilities, eagerness, and feasibility. There isn't a golden formula but it's likely some domains/use cases will rise to the top pretty quickly.
Look to - as best as you can - quantify the incremental business value from something like data mesh. That can be at the micro level - the single use case - or the macro level. But getting specific instead of "let's be data-driven" will lead to better buy-in and partnering with the business side.
You need to prepare to evolve your architecture. You can't have things set in stone. But you also can't just throw things against the wall and see what sticks. You need a balance between overly rigid and overly flexible.
If your centralized data function isn't a bottleneck, isn't the cause of your data challenges, data mesh probably isn't right for your organization - or at least it doesn't directly address your challenges. Time between identifying useful data and making it reliably available is a good place to look to measure if the centralized team is your bottleneck.
A good buy-in driving question for data mesh - or any data initiative - is "what's the cost of doing nothing?" Will you miss opportunities? Lose market share? How much are you "paying" on your existing tech debt? If we don't move, what is the cost to our future business prospects?
When looking at buy-in, you need to drive from the top down - so you can actually make necessary large-scale org-wide changes - and bottom up - so you are working well with the people doing the actual implementations.
Data mesh feels a lot like the Agile movement in that people expect it to be a silver bullet instead of a framework for thinking - and re-thinking - about how you approach your work.
As also recommended by previous guests, look to a vertical thin slice for your data mesh MVP. Make sure you include all necessary high-level capabilities - think the four data mesh principles - but don't boil the ocean, pick a manageable use case.
Keep technologists from focusing on the tech instead of the outcome - why would we use this and what is the business value? And what is the cost to launch and maintain it? Is it actually worth it?
If you are prioritizing you technology backlog, it's like "serving food to customers they didn't order" - you are putting together technology solutions that aren't what they want.

Darshana started the conversation admitting to being a technologist at heart - but argued we can't become over enamored with technology in data. Focus on the business outcomes! If you are prioritizing your technology backlog, it's like "serving food to customers they didn't order" - you are putting together technology solutions that aren't what they want or need. We have to move away from the concept that technology will solve your problems - it's very easy to listen to the vendor BS because people don't want to do the hard parts of solving data challenges. And the truth is, every situation is different and we can't copy-paste solutions. But that means signing up for hard work :)

To keep yourself from not going down the tech-focused rabbit hole, Darshana recommends asking more layers of "why" to people selecting technology. Okay, you want to use the latest and greatest tech - why? Why is this better than another solution? Why do we need to solve the issues it supposedly addresses? Why can't we use what we already have? This can questioning can drive technologists to use common vocabulary between tech and business by focusing on why are we actually doing work - what is the target business outcome? Every data team should be asking "what are you building for the business leaders?" Because if you can't answer that, are you actually adding value to the business?

A good question Darshana believes every company should ask - what are you trying to achieve when working with your data, especially something like data mesh? Is it just having the cleanest data possible? Or are you focused on serving customers better? Data can be the enabler to achieving your business goals but only if you actually tie your data work to your business strategy and goals.

For data mesh, it is typically the CTOs and/or CIOs that are the biggest advocates and thought partners to the teams driving implementations according to Darshana. But, you can't just move data mesh forward without buy-in from other execs. You need budget - whether that is directly from the CFO or from the head of a business unit. To get that budget, you need to talk about what would doing something like data mesh do for the business. You need to consider how to actually quantify the benefit to them - is the first use case going to drive better revenues, lower costs, etc.? But also, what are the added benefits of being more agile and scalable with data? Can data mesh lead to better talent retention too? Talk about unlocking additional use cases and capturing opportunities far quicker. Asking the business leader to "imagine a world where…" is probably not going to go well so get as specific as you can - easier said than done of course.

When working with the business to get buy-in for something like data mesh, Darshana recommends answering the question - before it's asked if possible - "what's the value of doing this?" Again, if you can get specific, that's great. Talk about total return - what are the upfront costs and the ongoing costs compared to return? But you can also ask "what's the cost of doing nothing?" Is that lost market share, lost opportunities, etc.? You can get the business counterparts to tell you what they are concerned about and you can help them address concerns with data. You can also point out the ongoing costs of your legacy stack / existing tech debt - you are incurring increasing costs just to keep it running.

When asked about driving buy-in for data mesh whether that should be from top-down or bottom-up, Darshana answered "both". You need the top level support to be able to actually change the organization, upskill people, create new roles, etc. You need a holistic view of your entire implementation or you are likely to build something overly use case specific. But if you don't have domains - especially the people in those domains doing the work - understanding and bought-in to the changes needed to do data mesh well, it won't go well. So you need to drive buy-in from top-down and bottom-up simultaneously. Sorry, no short cuts :)

In thinking about data mesh, Darshana has seen one big failure mode many overlook: many don't consider the ongoing costs of running something like data mesh. That's part of why it isn't for smaller organizations. Data mesh will probably result in higher costs than value for many organizations. It's hard to measure if your centralized data team is actually causing your data woes - every company has data woes, but it's more likely your centralized team is an issue if it takes a considerable amount of time from identifying data that would be good for a specific use case and then actually making it reliably available. Another potential challenge is around your "data transparency" - is your data well documented, understandable, etc.? If the responsibility for that is falling on your central team - who can't really have the context necessary to fully document the data in most cases - data mesh might be a good approach to consider.

Another failure mode Darshana has seen for data mesh is not having a mindset focused on evolving architecture. You should focus on the themes and principles and not the exact architecture. Especially because it will change. You just can't future-proof your platform - build it to evolve. You will incur tech debt, but with this approach it is a conscious decision to take on that tech debt. But you also do not want to go too far to the other end of the spectrum - you don't want to set things in stone but you also want to have some intentionality. A 'just try and see what happens and then react' type method won't go well in data mesh…

For actually getting going with data mesh, Darshana recommends an MVP via vertical thin slice. If this sounds familiar, it is becoming a recurring concept from many guests. To do your MVP for data mesh, you want to make sure you are not taking on too much, hence the thin slice. But you also want to make sure you incorporate all the high-level necessities of a data mesh implementation - so you can't just skip governance for example. You want to go after a small enough use case to not boil the ocean but also work on testing every necessary capability to go wide once you have built out your platform and figured out your repeatable ways of working. And it's okay to not get it perfect - the cost of change is far lower in data mesh - just drive towards value quickly.

As for where to start on a data mesh implementation, Darshana recommends looking at all your key domains and evaluating them on four different metrics: business value, capabilities, eagerness, and feasibility. Business value is about how valuable are the use cases the domain might participate in - and you want to think about net return and how quickly that return will likely happen. Capabilities is about how capable are the domains of actually delivering on a use case or use cases that will drive good business value. You absolutely need a central team supporting them with certain capabilities but a low data maturity domain with complex use cases is not a great place to start. As past guests have also mentioned, sometimes you need to focus on the domains where they are eager to participate instead of trying to fight tooth-and-nail. And lastly is the feasibility of the actual use cases - can they actually be accomplished? The business value may be through the roof but will it take 18 months to get a use case out the door? Will you have to invent new technology to handle it? There isn't a golden formula but you should look at these four criteria when making a selection for your initial data mesh use cases and domains.

Other Tidbits:

Speaking data jargon when talking to the business stakeholders makes it hard to actually communicate. Focus on speaking to business outcomes and business value. They don't care if it is data mesh, data fabric, data unicorn farts, etc.

For Darshana, data mesh feels a lot like the Agile movement in that people expect it to be a silver bullet instead of a framework for thinking - and re-thinking - about how you approach your work.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#137 Even More on Driving Buy-in in Data Mesh - Mostly About Domains - Mesh Musings 31 Oct 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#136 Building Your Data Platform for Change and Reusability via Modularity - Interview w/ Alireza Sohofi Oct 02, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Squirrel (OSS data platform) GitHub: https://github.com/merantix-momentum/squirrel-core

Alireza's LinkedIn: https://www.linkedin.com/in/alireza-sohofi/

In this episode, Scott interviewed Alireza Sohofi, a Data Scientist focused on building the data platform at Merantix Momentum.

Some key takeaways/thoughts from Alireza's point of view - some written directly by Alireza himself:

Where possible, look to build your platform in a loosely coupled way. It will make it easier to extend and evolve; and domains can replace pieces, mix and match components, or even extend the functionalities when it makes sense.
It's easy to fall into the trap of building a platform that is hard to evolve and support. Be very conscious about what you want to include - and not include - in your platform. Don't try to solve every challenge with a point solution.
To effectively share data - and the information it represents - software engineers / domains need to really understand their own data, including data modeling. That can't be easily outsourced. A platform team's job is to build the tooling so those domains only need to deal with the data, not the data engineering.
If you want a scalable platform - in many senses of the word scalable -, your platform should be relatively generic. It must also be easy to extend and augment. Focus on providing flexibility and ease of customization. One size definitely won't fit all.
Packages and templates are both useful but templates are typically more user friendly and easier to customize - start with templates when possible.
If there is a need for customizing or extending a package or template, it's better to first build it within a domain (with the help of the platform team if necessary). The generalized version of the new feature is then contributed to the platform. This leads to a more integrated domain-platform, more robust first release of new features in the platform, knowledge sharing, and avoiding bottlenecks that may arise if only relying on the central platform team.
Platform teams need to A) dog food the platform - you will learn far more by using it; B) provide good methods of communication for domains to give feedback and requests; and C) find better ways to exchange context with your domains regularly, e.g. pair work and scheduled informal chats.
The platform consists of several tools that should not only work well together, but should also work well with a wider ecosystem of open source tools. Solutions that try to offer end-to-end coverage usually fall short when it comes to flexibility and changing requirement and business environment. Composable components that can work together is the way to go.
Tools should be opinionated, i.e. encode the best practices, but at the same time hackable to the very core. Layered design where domain teams can choose the abstraction level which is appropriate for them is a good choice.

Alireza started by sharing a bit about how Merantix works with clients - often, their clients are not that deep into machine learning and want to outsource that. So Alireza and team are essentially building a data platform that is use-case agnostic across many different data maturity levels and modalities, both on the production and consumption sides, that is scalable and cost-efficient. Sound like a familiar challenge? While their platform is specifically for machine learning, it's a good approach to dig into, partly because they recently open sourced their platform so others can dig deep into the implementation aspects.

The most difficult challenge of the platform and working with customers, per Alireza, is the data ingestion. Clients are using a vast array of source systems and formats so they had to focus on each customer's specific challenge. Unfortunately, this means there is a custom-built driver for each use case/dataset for ingesting customer data but they have created a number of templates for ingestion and transformation to make the custom development - whether initial development or incorporating changes - relatively lightweight; the customization - and subsequent coupling - is typically only related to the business logic related to the use case. So, while it's not ideal, it's a scalable approach that is serving them well.

While customers are outsourcing their machine learning, that doesn't mean they are not data literate according to Alireza. To be able to leverage the Squirrel platform / service and get good information out for their applications, those domain teams still need to really understand their own data. Unfortunately, Alireza does not have the silver bullet to training generalist software developers to really handle their data - they must be able to model the data properly themselves. The platform team's job is to make tools so they can deal with the data but not the data engineering - but a centralized team handling the data modeling can quickly become a bottleneck.

When talking about scalability - both pure throughput scale but also scalable across many use cases, Alireza believes you must build a platform that is generic enough that it isn't tied to use cases. It must be relatively easy to extend or augment the platform as well. You need to provide people the flexibility and ease of customization if they want to own the complexity themselves. Easier said than done but still important to repeat. Blanca and Pablo at Plain Concepts said similar things in their episode.

According to Alireza, it's important to think about templates versus packages. Use templates where possible for more simple things because packages, by default, have some choices embedded in them. Templates are starting points that more often point directly to the choices people can make with defaults versus people discovering the choices made in packages. But both can be useful. And when a template or package needs to be extended, the work should be done by the domain team - otherwise you have centralized work that can become a bottleneck. But do have the domains contribute those extensions back to the platform as well.

Because Squirrel is open source, Alireza and team had to really think about how to make things loosely coupled, even within the platform. So the drivers, computation, storage, etc. are all able to be extended or even swappable. This means each domain team can replace things if they truly have reason to and they can still get good leverage from what is already built.

Alireza had some direct advice: when developing your platform, be very conscious about what you want to include. You don't want to do one-offs. What can you reliably abstract because it's a repetitive need? The first time you see a new pattern, don't rush to build supporting that in the platform. Otherwise, it's very easy to build a platform that is hard to evolve and support. Analyze a diverse range of challenges to find your patterns and then abstract.

It's crucial for platform teams to develop good communication and ways of working with your domain teams - or whoever your users are - and really learn how they use the platform according to Alireza. You should provide good ways for them to provide feedback and make requests but also more informal ways of teaming up and exchanging context, e.g. pair programming and scheduled informal chats. That way, every communication isn't an ask - think water cooler chat. Your platform team should be users of the platform too - the best information and feedback often comes from being a user yourself.

Alireza wrapped up on a major challenge that is still yet to be well addressed: how can we embed the semantics into the data. We need to figure out how to "solve ontologies that don't align across domains".

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of October 2, 2022 Oct 02, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#135 Iterating Consciously - and Quietly - Towards Data Mesh Capabilities - Interview w/ Balvinder Khurana Sep 29, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Balvinder's LinkedIn: https://www.linkedin.com/in/balvinder-khurana/

In this episode, Scott interviewed Balvinder Khurana, Principal Data Architect at Thoughtworks.

Some key takeaways/thoughts from Balvinder's point of view:

Data mesh is NOT a silver bullet and not everyone is ready to do data mesh - others have stated that but it's crucial to repeat.
A data mesh doesn't happen in a vacuum - you need to assess if you are really ready and does it align first to your business strategy and second to your data strategy.
If you decide to move forward on a data mesh implementation, really consider how you will measure progress and success against business goals.
To evaluate data mesh appropriately, consider what business value having better data practices would bring to your company and is your company aligned into lines of business or would you need to reorganize your business. Are you prepared to extend your line of business practices to data?
A common failure pattern in analytics has been not looking at the Intelligence Cycle - changing your operational systems and processes as a result of insights. Don't just generate insights, insights must generate action! Data mesh must avoid this too.
Even if existing centralized data team setups have significant bottlenecks, data consumers typically eventually get their needed data. Those data consumers can see something like data mesh as a risk - will they still be able to eventually get the data they need? Is eventually getting to faster access to new data worth the perceived risk?
If you have resistance to data mesh, look at delivering necessary capabilities to your data producers and/or consumers in a small-scale, incremental way and not specifically tying that in their mind to data mesh. Tie those incremental capabilities to business value.
Look to constantly communicate the what and the why of improvements to your platform to drive engagement. What is the purpose? Why should they care?
It's easy to fall into the trap of trying to iterate everything constantly. There is a concept of "good enough for now". Don't be focused on getting everything perfect, that juice is not worth the squeeze.
A good signal to reevaluate your domain boundaries is: a dashboard is still used but the owner no longer uses it. They don't want to own it any more so you need to find a new owner. That probably means changing business boundaries if the original user doesn't use it anymore.
To drive buy-in for data transformation, especially something like data mesh, you should look to drive it from both top-down and bottom-up if possible. It can work with only one but support from both makes it much, much easier.
It's okay to start out with lots of manual processes, just be conscious about where you want to automate. Take on tech debt consciously.
It's easy to get caught up in the trap of trying to optimize for time-to-market on every use case. But you need to balance time-to-market against quality and scope. You don't get to pick all three and might not even get to pick two. Look at what's really crucial.

Balvinder started by discussing the big question: is data mesh right for an organization - right in general and especially right for them right now? You need to start from questioning what are your key strategies - business strategies first before data strategies - and where are you right now. Data mesh isn't your data strategy and implementing data mesh doesn't happen in a vacuum - the real world is messy and ever-changing. So you need to line up your data strategy with your business strategy and then you get to the fun of "okay, how do we measure our success against our business goals?"

Once you have your business goals and how you will measure against them in place, Balvinder believes it's time to look at how can data help you reach your specific business goals and/or measure your success against your business goals. Again, have those business goals in place first. Then you start to look at your data operating model to see if it will help you achieve those goals and measurement. That's when you start to ask if data mesh is right for you.

When evaluating if data mesh is right for you, Balvinder recommends a multi-pronged approach. First is again to look at the value cases, what does your organization want to achieve with data? The second is to ask what challenges are you facing right now? Will data actually help you address those challenges? What are you trying to do with data and how would a more mature data practice help you achieve your goals? And the third is to look at how you are structured, do you have boundaries already set up around domains/lines of business? If these are aligned, then you can start to consider data mesh.

Not paying attention to the Intelligence Cycle is one way Balvinder believes analytical approaches have failed in the past - we need to keep an eye on preventing that failure mode in data mesh. The Intelligence Cycle is about taking information, analyzing it, and then pushing the results of that analysis - either directly or indirectly - back into your operational systems and processes. Essentially, the so what of doing data analysis - is there an actual impact to it? Actionable insights are only valuable if you actually take the action after all.

Balvinder shared the story of an existing client with a centralized data team and an existing data platform: a request for new data would go to the central data team; the data team would reach out to a number of potential data producing teams to try to figure out how to service the request. It took 2-3 months to get data from request to delivery. But, it was a known quantity even if it was slow. They knew who to reach out to. So, they were hesitant to rethink their approach because they can eventually get the data they need. Will it be the same in data mesh? That's a risk many might not be willing to take.

In order to implement data mesh in that kind of environment, to alleviate that perceived risk, you should focus on business continuity per Balvinder. By moving more slowly and only making small incremental changes to the existing platform and ways of working, they were able to move more and more towards data mesh without having to get everyone bought in ahead of time. Adding capabilities and explaining the business value of each capability - instead of trying to sell the data mesh concept as a whole - meant they could continually deliver business value. A thin value slice for each incremental capability with the selling point not being the tech. They created smaller milestones that led to a better path for their client.

And it paid off according to Balvinder - the client team felt empowered to do things for themselves with data. That team wanted to launch a new KPI and they already had all they needed to do it with the slow feeding method of adding capabilities and resources through a better platform. So they didn't need to go to the central team and quickly launched their new KPI.

Constant communication was crucial for Balvinder and team to drive engagement and buy-in. And it's important to not communicate just once as things will inevitably evolve as the real world changes. And it means you can deliver continuous improvements instead of trying for a big bang approach. Your domains will continuously change, your platform will need to evolve, etc. There is a concept of too much iteration and change as well - it's okay to assess if something is "good enough for now". You don't need things to be perfect, look to change where there is the most benefit.

For Balvinder, it's really easy to get overly focused on "I want this in production yesterday." But, when a data consumer says that, then they must understand they need to compromise on quality and scope. It's not necessarily even you get to pick two, there are aspects of all three and if you focus on time-to-market, it will mean it's not the quality they want and the scope needs to be limited. So what really matters most to the business? And if the answer is always time-to-market, Scott believes you might need to talk to your data consumers about what really matters.

Balvinder wrapped up on a theme many guests have touched on: data mesh is not some silver bullet. It won't solve all your challenges. You need to really think about if it's right for you. And there are many companies that aren't ready for data mesh. And that's okay too!

A few quick tidbits:

A good signal to reevaluate your domain boundaries is: a dashboard is still used but the owner no longer uses it. They signal they don't want to own it any more so it needs a new owner. That probably means changing boundaries because the business need that drove the creation of the dashboard is no longer with the team that created that dashboard.

To drive buy-in for data transformation, especially something like data mesh, you should look to drive it from both top-down and bottom-up if possible. It can work with only one but support from both makes it much, much easier. Having management support that the data initiatives/strategy align to the business strategy is important but so is buy-in from the people actually doing the work :)

It's okay to start out with lots of manual processes, just be conscious about where you want to automate. Take on tech debt consciously. When it's time to improve that capability in the platform, invest in it. But don't try to build everything ahead of time.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#134 Zhamak's Corner 5 - From Pipeline to Product, a Crucial Evolution in Data and Data Tooling Sep 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#133 Nitty Gritty From the Deployment Committee: Crucial Learnings on Driving Buy-in and Data Product Discovery - Interview w/ Ammara Gafoor Sep 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

ammara.gafoor@thoughtworks.com

Ammara's LinkedIn: https://www.linkedin.com/in/ammara-gafoor/

Data Mesh in practice article series from Ammara and colleagues:

#1: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start

#2: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-organizational-operating-model

#3: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-product-thinking-and-development

#4: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-technology-and-the-architecture

In this episode, Scott interviewed Ammara Gafoor, Principal Business Analyst at Thoughtworks who has been working on a few client projects related to data mesh including one for well over a year.

Some key takeaways/thoughts from Ammara's point of view:

There often is a hangup around data work, especially relative to data mesh, where people want to get it all right, all perfect the first time. That's never going to work. Get something decent out there, test, and iterate. Perfect is the enemy of done. No bikeshedding!
If you don't look to change domain's KPIs to align their operational work to data mesh "you won't prioritize it - you cannot prioritize it." Make it easy for domains to prioritize data mesh work if you want it to get done.
?Controversial?: source oriented data products should not be made available to business users within the domain or to almost anyone in other domains - at least by default - as they are difficult to understand for anyone other than the highly data literate people in the domain.
"Don't make things that you don't need yet." Build data products for use cases you've identified. Think of the target business outcome and work backwards to create the necessary data products.
Far too many people try to build customer 360 solutions. And they very rarely work. No one ends up happy.
Sometimes it seems like data mesh literature says get rid of many existing data roles - but the people in those roles are subject matter experts in how the organization and/or the domains do data. You want to leverage them as you evolve their roles and they can pick up new skills related to data mesh.
When driving buy-in, be prepared to repeat yourself multiple times. And then repeat yourself multiple times more... Learn to speak the language of the domain as well to drive buy-in with the business people.
Look for the 2 most obvious use cases inside a domain. Supporting those use cases, you will need 5-6 foundational source-oriented data products that will support many - most? - of the use cases for that domain. It will appear like you need many more source oriented data products but zooming out, you will see you don't need nearly that many.
Typically, using the median, use cases are powered or supplied by two consumer oriented data products and each consumer oriented data product is powered by three to four source oriented data products.
They planned for a single approach to work with all domains to start implementing and contributing to the data mesh. That proved to not work well and every domain needed it's own process to really make it all work. There are commonalities of course but there isn't a cookie cutter approach unfortunately.
It's very easy for people within the same domain to talk past each other, especially in a functionally siloed org structure. Data mesh can be a forcing function to find a shared, common language to drive to better results.
From Team Topologies, there are 3 streams: operating model, product stream, and technology stream. They should all be given equal weight. No, you can't just focus on the tech stream...
There may be push-back from both IT and the business side to having the business people collaborate closely in a data mesh implementation. But it's very important to overcome that - that collaboration with the business people is crucial to making data mesh work.
Business leaders may have - or at least believe they have - what they already need currently via shadow IT. It may be difficult to convince them to change their ways of working "for the greater good".
When looking at driving buy-in with domains, look at how they are incentivized. Often, short-term results are more rewarded than focusing on mid- to long-term results. If next year's funding is based on this year's results, they will focus on this year's results...
Typical personas in a data mesh implementation: the IT sponsor, the business owner, the "no stance/sideline", the "says yes to your face...", the product owners, and the data lake builders (aka those feeling the target on their backs).

On Personas, Ammara laid out a few she and team have run into:

The Sideline Watcher - sees data mesh as probably 'yet another data trend'. Not pushing back but not taking a stance. Waiting for the tide to turn one way or another before making their own waves.

The "Yes to Your Face" - will say yes to you and then just go do whatever they were going to do anyway… These are inevitable - try not to take it personally.

Encourage people to think business need first instead of data first according to Ammara. Think about what business outcome you are trying to achieve and then work backwards to what data you need to address that. If we are just sharing information without intention, it can lead to misuse of data - will people really understand it? Will it be compliant use? Sarita Bakst had really good thoughts on this in her episode. Don't make things that you don't need yet.

Data Mesh Radio is...

Weekly Episode Summaries and Programming Notes – Week of September 25, 2022 Sep 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#132 Zhamak's Corner 4 - The World Is Changing, Are You Accounting for That? Sep 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#131 More On Driving Buy-In in Data Mesh - Mesh Musings 30 Sep 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#130 Making the Data Quantum Leap - Starting from the Data Quantum at PayPal - Interview w/ Jean-Georges Perrin (JGP) Sep 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Data Mesh at PayPal blog post: https://medium.com/paypal-tech/the-next-generation-of-data-platforms-is-the-data-mesh-b7df4b825522

JGP's All Things Open talk (free virtual registration): https://2022.allthingsopen.org/sessions/building-a-data-mesh-with-open-source-technologies/

JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/

JGP's Twitter: @jgperrin / https://twitter.com/jgperrin

JGP's YouTube: https://www.youtube.com/c/JeanGeorgesPerrin

JGP's Website: https://jgp.ai/

Some key takeaways/thoughts from JGP's point of view:

Data mesh as it's been laid out by Zhamak obviously leaves a lot of room for innovation. For some, that's great. For others, they want the blueprint. And it's okay to wait for the blueprint. But JGP and team are excited to innovate!
PayPal's 3 main initial target outcomes from data mesh: A) faster and easier data discovery, B) easier to use the data in a governed way, and C) increase data consumer trust in data.
PayPal's initial data consumers are data scientists so their platform and data quanta are built to serve that audience first.
Really consider what you want to prove out in your MVP. Is that minimum viable A) data quantum, B) data platform, C) data mesh, or D) something else? Only doing a data quantum probably sets you up for trouble and a platform only won't be tested until it has data quanta on it.
Data contracts are crucial to making trustability actually measurable and agreed upon. Otherwise, it's far too easy to have miscommunication between data producers and consumers, which leads to lack/loss of trust.
Producers, don't set your data contract terms too strictly when first launching a data quantum. There's no need to over-engineer - despite how interesting that can sometimes be...
For too long, we have tried to keep software engineering and data engineering overly separate. They are both just engineering with slightly different focus and data mesh really leans in to that.
We've also tried to keep operational and analytical far too separate. We should look to build out tooling where data can live that serves both operational and analytical workload needs. But we aren't there yet.
Analytical APIs, at least as far as we've seen them to date, are just not going to do what we need relative to accessing data from data products/quanta according to JGP.
Standardizing metadata access APIs across data quanta has made it very simple for data consumers to begin using new data quanta as they are introduced to the mesh. PayPal has observability, discovery, and control APIs.
Domain is an overloaded word. It can mean a very large high-level domain like Sales, Finance, or Marketing with hundreds or thousands of people in it or it can mean a smaller, 'two pizza team' level scale.
PayPal is doing only one data quantum per domain but that domain is really at the two pizza team scale - they aren't trying to have a single data quantum for Marketing.
It's crucial to understand data quanta and the use cases they power both have life-cycles. So really applying product thinking is crucial.
Most data engineering teams do work in a waterfall approach and that just doesn't scale well. However, moving to data mesh can mean additional cognitive load as it really requires an Agile mindset to do right and that shift in the ways of working is not trivial.
It's good to have smaller delivery requirements so you get faster feedback on what you are creating - a core tenet of Agile. Don't try to deliver everything all at once. Get it in users' hands early to get feedback.

Quick tidbit:

If you are just delivering data, the 1s and 0s, you are not delivering the necessary trust.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 18, 2022 Sep 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#129 Iterating Data Governance for Data Mesh: Lessons Learned from 'The Data Governance Coach' - Interview w/ Nicola Askham Sep 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Nicola's website: https://www.nicolaaskham.com/

The Data Governance Podcast: https://www.nicolaaskham.com/podcast

Nicola's LinkedIn: https://www.linkedin.com/in/nicolaaskham/

In this episode, Scott interviewed Nicola Askham, a data governance consultant known simply as The Data Governance Coach and host of The Data Governance Podcast.

Some key takeaways/thoughts from Nicola's point of view:

The key point of data governance: ensure the data we use is the right data for the right people to better address business challenges. Everything you do with governance should circle back to that.
In doing data governance right, you need to set yourself up to take in feedback and iterate. You absolutely won't get everything right upfront. It's crucial to set expectations that your data governance approach will evolve as you learn more.
If you see data mesh as being about making better data more accessible to your current data consumers, that's a very big opportunity wasted. Aim to significantly expand your pool of data citizens. Not everyone should be a data scientist but data should play a role in much more people's jobs.
To get going in data mesh, you need to get your data governance to "good enough" and start moving forward. Think about what you need - is it the very complicated standard to last the next decade or is it about getting people to understand and trust the data they can now access? Probably the second...
To drive buy-in for data governance, you should tailor your message to the audience. It's very hard to have universal appeal around a specific selling point of data governance but data governance can - and should - drive value for everyone.
Every data governance approach should be tailored to the organization but it should start from a few building blocks: A) policy; B) processes and standards; and C) roles and responsibilities. (More info below)
A centralized data governance team making decisions about what to do with specific data will not scale - they just can't have the context/knowledge needed. So federated governance has been the sensible approach for a long time, it's just not necessarily easy to do right. Or at least it's quite easy to do wrong.
Central governance teams are crucial - they make it easy for federated teams to do what's necessary to comply with regulations and internal standards but with as little friction as possible. The central governance team should be a value-add, not a gatekeeper.
Make sure teams understand data governance can add significant value to them. Participation is not just some mandate, it has a benefit. Then make sure you are actually providing that value.
Data governance has a bad name because of those few using it to put up obstacles. Data governance needs to be about lowering friction, not creating it. Easier said than done of course.
In data mesh, you will likely need new roles to handle new data governance needs. Where previously, there were some vague ownership requirements, they should be much more explicit in data mesh.
Tying to the point above, you will likely have different sets of requirements under roles across your domains. That's okay. Look to create a standard model for roles and responsibilities and adjust where it needs to be adjusted. It's okay to have non-uniform roles but there needs to be a starting point for domains to go from.
When starting out in data mesh, look for a relatively simple first use case but don't only stick to simple use cases early in your journey. It will make it much harder to tackle the difficult use cases later. You don't want a mesh that can't tackle hard but high value use cases!

Nicola started the conversation sharing her thoughts on "normal" data governance. What does it even mean? Is that federated, centralized, decentralized, etc.? What she's seen is that functional data governance - at an organization of scale - is not centralized in day-to-day decision making. The central team just can't have the context, the knowledge to make good decisions quickly if at all. But there absolutely needs to be a central team to provide support and knowledge and set federated teams up to succeed. The central team needs to focus on friction-reduction and value-add work. To do that, you need to create standards and processes. But she emphasized keep your frameworks, processes, and standards as simple as possible - no one single, all-encompassing standard please!

As an example of functional governance, Nicola talked about how you can't just have universal data quality standards. Every use case may require a different combination of data quality - why optimize for completeness if it's not needed? A central governance team should be focused on helping business stakeholders define aspects of data quality and how to measure it - that way, data consumers can understand the quality of what they consume without learning different standards for each new data source but we aren't setting data quality requirements that aren't helpful or useful.

On data mesh specifically, Nicola agrees with most guests: data mesh is very much about the people side. That means the central governance team needs to collaborate with people outside the central team to iterate and improve upon your data governance approach. Feedback leading to improvements is necessary, the governance team can't issue decrees from on high. A part of data mesh that excites her is trying to solve for the age-old challenges of ensuring the data is the right data and that we get it in front of the right people to answer questions about the business - lowering friction to leveraging our data. What "right" means is always somewhat open to interpretation of course.

Whether you are doing data mesh or not, Nicola believes data governance can't be about obstacles. That is how data governance got a bad reputation. The phrase should spark joy, not fear or revulsion. Instead, it has to be about making it easy for data consumers to find the right data and then being able to find the right people and documentation to help them understand that data. Governance is about providing low-friction ways to provide access and drive understanding of your data and how to properly use it.

One thing Nicola has learned working on a data mesh implementation is that while in data mesh, there are a few new responsibilities that are called out explicitly, that might fall under different roles in different domains. Some responsibilities may fall under a data owner in one domain and under a data steward or mesh data product owner in another. The differing role types she recommends are data owner, mesh data product owner, and data steward. Find a standard setup for roles and responsibilities and then let the domains move responsibilities around as needed - don't make the domains come up with everything from scratch but don't hold on to your standard setup closely either. Everything in data mesh is about iteration and evolution!

Scott asked what actually is "good enough" data governance to get moving in a data mesh implementation. Nicola pointed out that no matter what, you won't get your data governance perfect when starting. Especially with something as immature as data mesh is right now. So have clear indications but nothing set in stone. And use her building blocks framework below. Also, think about what capabilities are needed early to drive value: is that some complicated interoperability standards or some data quality definitions/measurement to enable people to understand and trust the data? Probably data quality definitions.

According to Nicola, every data governance approach should be tailored to the organization but it should start from a few building blocks: A) policy - as it mandates who will be required to do what and why? Domains just don't do data governance out of the goodness of their hearts; B) processes and standards - lay out what you are trying to achieve and why and then give people an easy way to achieve that. That drives consistency and reduces friction, a win-win; and C) roles and responsibilities - it's very crucial to assign ownership and layout who exactly owns what; we've all been to meetings with no clear next steps and they are almost always a waste of time. Who owns driving things forward? Be clear about it.

Some additional data mesh governance advice Nicola gave: 1) Look for a relatively simple first use case. What has a high chance of success where you can also get some momentum and learnings? 2) Don't only look for the simple use cases early in your journey. That can lead to not being able to actually face the hard parts when they come. And with data, of course the hard parts will come. 3) Communicate early and often that you want to collaborate with people and that things will change. Solicit feedback and make constituents part of shaping your governance. 4) Make it clear the central team is there to help and not control - help around compliance, help with reducing friction, etc.

A general sentiment that has worked well for Nicola in the past is telling people outside the governance team: if you don't get more value from data governance than you put into, we'll change our data governance frameworks. The governance team may feel the pressure but if you aren't adding value with your governance, outside of regulatory compliance, why are you doing it? Teams will want to participate if you give them a reason to so find the value-add reasons to.

Scott asked about a particularly difficult question in data mesh: who owns downstream data products - data products combined from data of upstream data products? For Nicola, if the data isn't transformed, the ownership of the data, even in those downstream data products, should still be whoever owns the originating data product. Essentially, the ownership still flows if the data isn't transformed. But that can cause issues as well if the upstream mesh data product owner doesn't have direct control of how data is used downstream. High communication, trust, and good lineage are necessary.

According to Nicola, some people are looking at data mesh as only making data more accessible and usable to existing data consumers. But that's a big missed opportunity. Data mesh can make data more accessible to more employees, driving better decisions. We need data literacy to get to that target outcome but implementing data mesh will lead to a lot of wasted potential if we don't expand the data consumer pool.

In wrapping up, Nicola talked about how you can actually drive buy-in for data governance. While trying to sell everyone on upping your data governance game with the same message is not likely to succeed, data governance really does have value for all participants. If it doesn't, as Nicola noted earlier, you need to change your governance approach. So drive that home; tailor the message and speak to - after listening about - their pain points and how you can help them address the pain.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#128 Data Mesh Disillusionment Delusions and Dichotomies - Mesh Musings 29 Sep 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#127 A Quest for Readiness: The Importance of Capacity to Change in Data - Interview w/ Winfried Etzel Sep 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Winfried's LinkedIn: https://www.linkedin.com/in/winfried-adalbert-etzel/

MetaDAMA podcast (episodes in both English and Norwegian): https://podcasts.apple.com/us/podcast/metadama-en-helhetlig-podcast-om-data-management-i-norden/id1572639573

In this episode, Scott interviewed Winfried Etzel, Information Strategy Consultant at Bouvet, Board member of DAMA Norway, and host of the MetaDAMA podcast.

Some key takeaways/thoughts from Winfried's point of view:

Transformation teams - a team that helps domains to transform and upskill by embedding in the domain for a time - is an exciting pattern for data mesh.
Transformation teams can collect information more easily and find patterns as they are closely collaborating with more domains.
Think of transformation teams like a personal trainer - they help you get in shape so you learn what exercises to do and how but then you can work on your own. And you can engage them again if you need more help.
How can we give guidance and enable change at the same time - the transformation teams shouldn't do the work for the domains but need to work with them. Can we go broad in the organization if we only have a limited number of transformation teams? Do we need to be in that much of a hurry where that's an issue?
There's definitely some vendor washing going on around data mesh. It's often difficult to determine what is new and what they've recycled to sound new.
In data, we should look to take many learning from software engineering but also other disciplines. History, political science, law, manufacturing, etc. all have something to teach us to better approach sharing our information with each other.
"Data is a message to people in the future." What do you want to tell them? Data products are how we communicate with those people in the future.
Data maturity models are crucial so you can self-reflect - are we really willing and capable to do what we need? And how can we measure how well we are maturing our capabilities?
We need to adopt better change management principles in data if we really want to create data citizens. And you should explain how data is important to their role to drive buy-in.
Potentially look at using a big bang approach to organizational changes if it is to something like upskilling or even ways of working. Doing a big bang approach for new responsibilities though is probably more challenging and less likely to work.
Data ethics as a practice is relatively immature. Companies often don't see a purely business value to training people to act ethically. At most, the return on investment is indirect. Maybe we can figure out how to discuss it relative to retention? How many people want to be creepy and overstep with data simply because they can?
Many data mature domains not only want to build everything themselves, they also want to set their own definitions and standards for things like data products. It may be a challenge getting them to participate in data mesh.
When finding initial domains to work with in a data mesh implementation, look for domains with the capacity to change - willingness and ability. They will understand what they need to do and why even if they can't do it yet.

Winfried shared that in the data mesh meetups for Norway, most of the discussion thus far has been about domain driven design for data, data discoverability, and data as a product. If you are not sure exactly how to approach those topics or what they mean for your organization, you are not alone. And no one is quite sure what federated computational governance means either.

In order to do data mesh, Winfried believes people should consider the emerging pattern of Transformation Teams. These are teams that help domains by collaborating to build out their first data product - or possibly more - while also upskilling the team. Scott Hawkins at ITV (episode 48) discussed something similar. The transformation teams also are a great source for finding reuse as they are working with the domains and can more easily recognize patterns. They also can have an excellent perspective on how your implementation is going thus far. We need people that can enable change while giving guidance. A concern though is that the number of transformation teams is limited - how can you go broadly in the organization quickly? Or do you need to be in a rush to do so?

In Winfried's view, we need to take far more concepts from software engineering in general in data. Zhamak has stated this multiple times as many, many software engineering practices inspired parts of data mesh. Both mentioned Team Topologies as a crucial tool for change management. But we also should look to other areas outside software too. Winfried sees history as providing a lot of inspiration for data literacy. Political science can be a good way to think about organization design and communication. Law has millennia of examples of good ways to present arguments / one's context. Manufacturing can give us good insight into how we think about products including lifecycle - as Alla Hale discussed in her recent episode.

There are many data maturity models, most of which are not that differentiated, according to Winfried. They are still useful when assessing your readiness to do something like data mesh - at the macro and the micro/domain level. Are you organized correctly? Do you have the capability to do what's needed? Do you have the capacity - both meanings: the amount of time and the capability - to change? Are your teams ready and willing to share? Etc.

Per Winfried, in general in data, we have not adapted and adopted change management techniques all that well. We need to focus on providing strong learning paths so individuals can change to being "data citizens" as we evolve the overall organization. To inspire people to want to change and get better with data, we need to share with them how data is important to their role. Otherwise, it is homework with no purpose in their heads.

Should we go for a big bang approach or incremental, domain by domain? Winfried thinks it shouldn't be only one or the other. When it comes to things like upskilling or even changes to ways of working, big bang might be a better approach so you aren't having to constantly fight against the inertia of the historical patterns and working across so much time to get everyone to a new way of working. But Scott asked about trying to do a big bang approach to new responsibilities and both agreed that can lead to issues.

And we need to make sure we keep ethics as something top-of-mind when people learn about data - just because you can doesn't mean you should... Winfried doesn't believe data ethics as a practice has really matured yet. There is of course AI ethics but that is typically about biased inputs, not as much the "should we really even be trying to figure this out?" Companies don't really see the business value in data ethics - which is a challenge that everything has to be about driving business value. There are many cases, especially in the US now, where we need to think about potential harm much more than just potential risk of data exposure.

Winfried shared his view on a few things regarding data mesh and domain data maturity. The most data mature domains typically want to do everything themselves - that includes not just building but even things like defining data products. That can lead to challenges when trying to integrate their work into the broader mesh. The domains you should look for are the ones that "understand what they need to do even if they can't do it yet." Those domains have the capacity - the willingness and the ability - to change.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of September 11, 2022 Sep 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#126 Evolving from Data Projects to Data As a Product - A Data Platform Six Years in the Making - Interview w/ Blanca Mayayo and Pablo Alvarez Doval Sep 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Blanca's LinkedIn: https://www.linkedin.com/in/blancamayayo/

Pablo's LinkedIn: https://www.linkedin.com/in/pablodoval/

In this episode, Scott interviewed Blanca Mayayo - Product Manager, Data Platforms - and Pablo Alvarez Doval - Lead of Data Platforms and Principal Data Architect - at Plain Concepts.

From here forward in this write-up, B&P will refer to Blanca and Pablo rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from B&P's point of view:

It's easy to fall into adding fit-for-purpose capabilities to your data platform but don't. Stay focused on managing your platform as a product - all aspects of it have lifecycles - and you can't try to fit every use case, especially before there is a need.
If transformation, especially data transformation, is not tied to the business strategy, that is a major recipe for failure. You likely won't deliver good business outcomes.
"Beware the proof of concept" - too many try to do a proof without the actual concept. What are you trying to prove and how will you decide/measure if you proved it?
You can have everything necessary for a data initiative to succeed lined up - the sponsors, the will, the budget - and still fail. Nothing is 100%.
Three common data initiative failure modes: 1) focusing only on the technology aspect and not does it meet needs and can we maintain and pay for it; 2) only treating it as an urgent tactical needs instead of playing into the broader data strategy; and 3) not considering how to actually do change management.
Your platform is a product too - data as a product isn't just about mesh data products - think about capability lifecycle and how you communicate upcoming changes - especially deprecation - and help users migrate to the new capabilities.
Acclimatize people to change and evolution. Most people in data aren't good with - or at least used to - evolution and preparing for said evolution because the cost of change for data has been so high.
To do data as a product, you need the right balance of curious developers, risk/risk management, and capabilities.
The most likely places to find reusability in your platform will be the mechanisms around data product production and maintenance - the lineage, CI/CD, data quality monitoring, security/compliance, etc.
Reuse is crucial for a data platform - look to have data transformation and storage reuse of course but also really focus on providing templates and then letting users create their own templates. The transformations and handling data don't need to be overly exposed to users.

As important background for the conversation, B&P discussed Plain Concepts' journey from a consulting company towards a product company - they had been working with their clients for years, building out a reference architecture for each client's own data platform implementation. They were doing fit-for-purpose, solving very specific challenges with point solutions, not managing the evolution like a product. So the two pillars from data mesh that resonated with them most were data as a product thinking and the self-serve platform - platform thinking is key. They started to push back on their previous practice of putting in additional capabilities to the platform without a key business reason - it's cool tech! - and began to manage the platform much more as a product.

Per B&P, part of switching to product and platform thinking was starting to focus on "what to remove" from and what not to add to the platform. Now, they think about where they need to go to support all the users - and align everyone developing the platform on the same vision - instead of trying to support every possible use case. And it's okay for people to use technologies or services that aren't part of the platform if necessary. Shadow IT is only really in the shadows if it's not known about.

Another big learning has been the transition of pieces of the platform to keep up with the best available offerings. For example, they started using Hadoop and now mostly use Spark. The platform team was asked to support very specific use cases and did a temporary solution with an eye - from the beginning - on deprecation. And they recommend you start communicating deprecation as early as possible too and work with users to migrate when the time comes. What - typically homegrown - pieces of the platform are not up to the best practices in the industry? One they mentioned specifically was replacing their own data testing and validation system with GreatExpectations.

B&P discussed a major failure mode around data transformation in general: your data transformation not being a business strategy transformation. On the micro level, why is the use case you are developing good for the business? Then keep thinking about that up to higher and higher levels of transformation. Look at data mesh - if making a major transformation like data mesh isn't part of the business transformation, will you be able to even make necessary organizational changes? And you must constantly communicate about the transformation - the why and the what - and how the transformation will evolve too. You will learn along the way - leaving no room for evolution clearly doesn't work well.

And in general, people in data aren't used to evolution per B&P. In a few of their customers/clients, every other team except the data team has been good with the new ways of change management. And it's understandable, the cost of change in data has been high historically, especially with the data warehouse. To get the data team on your side, thinking evolution is good, you need to collaborate with them and get them to understand the reasons for change. Scott note: I know most listeners, you are the data team that wants to drive change but you are the bold leaders in data :)

When talking about change, B&P pointed to a common thread in interviews on this podcast: most data teams have not adopted modern software engineering practices - we're still essentially doing the same thing we were 20-30 years ago. The data architects and data engineers like to play with the bleeding edge technologies but they often aren't trying to adopt them for the right reasons. In B&P's view, instead of bleeding edge, it's better to use more proven tech most of the time unless there really is a capability that's necessary and only available in something emerging.

When asked about keeping an eye out for what should be added to the platform regarding reuse, B&P said they were seeing the same general patterns repeatedly, and even there is some parallelism into what other more general software or operational excellence initiatives require. It wasn't that it was exactly the same but once you zoomed out, challenges started to look pretty similar. And more often than not, they were mechanisms around data product production, not the exact data transformation and storage technologies; so lineage, CI/CD, data quality, security, etc.

Plain Concepts' data platform is split into three layers of reuse: the first is the technical layer, which they made too technical at first to easily use for people who weren't extremely data engineering capable - don't fall down the same trap. The second layer is all about templates that they've built out through constantly watching for patterns of use. There is even a template catalog. The final layer is all about enabling customers to create their own reuse-focused templates with an SDK.

B&P shared a few major underlying issues that will likely cause failure for your data initiatives. First, if your data strategy is only focused on the technical aspect, only on choosing a cool technology that you can't maintain and is too expensive whether that is time, cloud cost, license, etc. Second, trying to react to everything as an urgent tactical need or a short-term change instead of having it play into the broader vision/strategy. And third, not really focusing on change management at all - you need to align incentives, get your early wins to drive momentum, etc. It's a process to drive change.

"Beware the proof of concept", B&P said - many companies do proof of concepts (PoCs) but they don't have a specific goal or plan. What are you trying to prove out? Why will that drive value? Far too often, organizations cut corners, don't have enough team allocated to the PoC, don't have a strategy or goals, etc. Again, what are you trying to actually prove in your proof of concept? And how will you measure if you proved it? What will you do next if you prove it?

When determining if data mesh or anything similar is right for clients, B&P start from the pain points. What are the pain points and then, much more importantly, what is causing those pain points? Much like Scott frequently says: if data team centralization isn't your pain point, don't decentralize your data team! If you are looking at a new approach or technology, what are you really trying to even solve? And even if you line everything up, the sponsors, the budget, the will, things can still go poorly. Nothing is 100%.

Other Tidbits:

A few additional failure modes mentioned: 1) not continuing investment in your platform - it needs maintenance and innovation. 2) if you aren't having broad-scale collaboration and sharing.

Really consider how is IT seen? Is it just about cost optimization or is it a value driver? Can you move to a product mindset?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#125 Zhamak's Corner 3 - How Should You Choose Your Initial Use Case(s)? Sep 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#124 Is CDC an Anti-Pattern in Data Mesh - Meshing Musings 28 Sep 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#123 Reflecting on Multiple Data Mesh Implementations: Iterating Your Way to Success - Interview w/ Sunny Jaisinghani and Simon Massey Sep 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Get in touch with Simon and Sunny: data@esynergy.co.uk

A webinar (info gated) Sunny and Simon did on data mesh: https://events.esynergy.co.uk/data-mesh-experimentation-to-industrialisation-on-demand

Simon's LinkedIn: https://www.linkedin.com/in/simon-massey-82718a3/

Sunny's LinkedIn: https://www.linkedin.com/in/sunnysjaisinghani/

In this episode, Scott interviewed Sunny Jaisinghani and Simon Massey who are both Principal Consultants at the consulting company esynergy. They have been involved in multiple data mesh implementations including at a large bank. This episode could also have been titled: Aligning Incentives, Reducing Friction, and Continuous Improvement/Value Delivery but it doesn't roll off the tongue very well.

From here forward in this write-up, S&S will refer to Simon and Sunny rather than trying to specifically call out who said which part as that leads to confusion.

Some key takeaways/thoughts from S&S's points of view:

We are all still early in our learnings about how to do data mesh well. There is still a ton left to learn. Which is why people should share what they are learning more broadly. Helping others will help you.
Data mesh, whether it's your overall implementation, your platform, your data products, your ways of working, etc. is all about evolution, incremental improvement, iteration, etc. You don't have to get it perfect upfront to drive immense value in the long run. Acclimatize people to iteration and lower the pain of change.
Don't go for the big bang approach, find ways to continuously deliver incremental value. That builds the momentum necessary to drive data mesh broad in your organization.
"Data mesh is 25% technology and 75% ways of working."
Once you start to get into a groove with the organizational ways of working, that's when the value force multiplier in doing data mesh starts to take off. Fast time to market with new data products, quick iteration cycles, low-friction cross-domain collaboration, etc. But it takes time to really figure out how to do data mesh in your organization.
Your data mesh will inherently have a huge scope. Try to keep that scope as limited as possible as you are getting moving - especially the near term. It is very easy to try to "feature stuff" your data mesh implementation, especially the platform and doubly especially early. Keep complexity out where possible and focus on moving the ball forward, not spending a lot of effort preparing to move the ball forward.
If you can't show value from your early data mesh implementation work in 1-2 quarters, you are quite likely to lose any momentum and thus at least a portion of your funding. Find ways to deliver continuous incremental value.
For less "data mature" domains, you can use the allure of working with more cutting edge technologies - and upgrading skillsets - to drive buy-in, get software engineers interested in participating.
For driving buy-in with moderately data mature teams, ask them how quick iteration on data and reliable data would improve their day-to-day lives. Is there a good use case for your domain you can't handle right now?
For the high data maturity/capability domains, they might be reluctant to participate because they like to build things themselves. But data mesh means far less red tape/friction for data consumers, so they will likely pressure the domains to move to data mesh to reduce the friction.
In S&S's experience, the most technologically sophisticated domains wanted to do things themselves and were often among the last domains to participate in the data mesh. Don't be shocked if this happens in your organization.
Lastly on buy-in, seeing is believing. Many domains will be skeptical. Find those willing and work together to deliver great value. Many of those initial skeptics will now want to participate.
Cloud economics and scale are really crucial to achieving value from data mesh. The cost of failure and the scale of failure are much smaller so we can iterate quickly and take more risks. On-prem data mesh is probably not worth the effort in most cases.
Similarly, data mesh means pursuing high chance of failure but high reward opportunities around data is much cheaper/easier. Teams don't have to be nearly as worried about an analytical data product being a failure - it wasn't months of time and huge cost, it was a few weeks and not expensive.
Think of your mesh data product lifecycle from the start. In alpha mode, you are creating something only for your own domain; it's okay to make drastic changes and if you don't develop it further if there isn't value. Beta is more you bring on one external user and are still early in figuring your specific data product out. Don't push a beta or ESPECIALLY an alpha analytical data product out to the broader mesh.
In the same vein, you can treat your data as a product but not arrange it into actual products. Data consumed within the producing domain only doesn't need the same affordances. Don't overcomplicate things.
Handoffs between teams, especially where context is crucial, are massive friction points / bottlenecks. Have the producers and consumers directly work together, don't have highly specialized teams like a dedicated ETL team.
Use the phrase "analytical data product" or equivalent. If you only say "data product", people often think of other types of data products and aren't as focused on delivering something for your data mesh.
If you can really get away with a skunkworks data mesh proof of value, S&S recommend it - Scott is more cautious*. But for most organizations, it will be a top-down mandate, which will make finding willing participants slightly harder.
Focus on getting everything out of the way to make technology the easy part. The technology aspect is hard but it's the easiest part of data mesh. No one said data mesh is for the faint of heart or weak of will.
Big list of antipatterns below as well.

*Note: Scott doesn't recommend a skunkworks approach because there are likely extremely few organizations where you could get far enough to prove value without explicit funding

S&S started off the conversation with a very key point that should be said more often in data mesh: we are all still learning how to do data mesh well and where the sharp edges are, even those who have been doing general distributed data and/or data mesh for multiple years. As Scott said, everyone feels a bit like they are behind the curve but no one is the expert on this yet, it's too early.

Trying to lead a data mesh implementation - or even just thinking about data mesh - with a technology-first approach is big mistake many people make per S&S. Data mesh should be close to 75% the ways of working and 25% technology if you are doing it well. If you don't get to a good place with the organizational aspects, those ways of working, your data mesh implementation won't work. Once you start to really find your ways of working, that is when the value force multiplier of mesh really kicks in. But it takes hard work to get there. And, of course, old habits die hard. Change is painful and it will be difficult to change the ways of working.

S&S have helped lead multiple data mesh implementations and the initial getting going is always unique to the organization. In their first implementation, they had the budget and freedom to drive innovation enough to find the early adopters, then prove out the value; they were able to go to senior management with proof points when asking for funding to go broader. Most organizations, people won't have that luxury and it will need to be more of a top-down mandate with an executive sponsor. But that might make it harder to find use cases because people see it as a mandate instead of a collaboration at first. And you are likely to lose momentum and at least some funding if you aren't able to show value in 1-2 quarters so get to delivering continuous incremental value early instead of a big bang, back-end loaded approach.

On finding the "coalition of the willing", those domains willing to be the early adopter/guinea pigs, S&S recommend looking for people who can understand that having better, more reliable, more high-agility data practices will make their life better. Those tend to be the teams that are semi-mature with data. For the teams that are less mature with data, you can win them over with the allure, especially for those using more legacy technologies, of upskilling and getting to use some cutting edge technologies. On both of those types of teams, S&S have seen people get excited by the rate of iteration possible with data. For the data mature teams, instead of building everything themselves and having lots of checks by governance, legal, etc. to ensure they met every bit of compliance, data mesh means much less red tape - it removes lots of annoying friction for them and they can focus on driving value.

S&S recommend in general, in driving buy-in at the domain level, look for valuable use cases that are within the domain itself. That way, you can do much more trial and error because the producer and consumer are one-in-the-same. It also means they will be more incentivized to make it worthy of a mesh data product. And don't push out data products for broad consumption too early - after an alpha stage where it is only consumed within the producing domain, for the beta stage, start with a single use case outside the domain. Once you have the hang of that, think about offering broader access to more consumers serving more use cases but don't push data products out too early.

Another incentivization mistake many make - especially the exec sponsors - is focusing too much on the end-state big picture, per S&S. So the CDO or CIO saying "when we have all of our data organized into these beautiful data products with amazing self-serve, think about all the value. Domains, make that happen." But that misses the point of what about now? Why are domains incentivized in the meantime? If all the value is going to be in 2, 3, 5 years, why will they be excited to participate now?

So, according to S&S, you need to focus on quickly unlocking business value with each analytical data product as well as with improvements to the platform or other aspects of your implementation. And constantly be iterating towards delivering continuous incremental business value - always be improving what you've got when the work has a good return on investment. Also, believing is seeing. When other domains start to get a positive spotlight from their data-informed results, more domains will want to participate.

Back to the data product creation process, S&S emphasized how important "continuous improvement" and iteration to value are in data mesh. You don't have to get it perfect out of the gate. A huge benefit of data mesh is the ability to add - and adjust - incrementally as you learn more to drive more value. Consumers also have to be on board that things may change but it's far better. As stated, the analytical data product creation process S&S recommend is to start with an internal-to-the-domain use case. That way, it doesn't need to be excessively governed or even documented that well. The producer is the owner so they know the needs and the use case. Once you've moved past that alpha stage, don't look to push the data product out wide, find a single use case from another domain and work with them to iterate. Then you can look to publish it to the wider mesh once you really know what you have and what you want to share. And always look to improve and manage your data product like an actual product. "If you don't do the product lifecycle management, it's just a data mart." Data marts didn't go that well when we tried them in the 80s for a lot of those reasons.

Incremental improvement and iteration aren't just for your analytical data products according to S&S. Historically, our data setups have been sort of "all or nothing". You either got it right and it created a lot of value or you didn't and it was a flop. The ability to - and the cost of - change in data was very high. Monetary cost, delayed time to insights cost, etc. But with data mesh, it is inherently about iterating, evolving, making improvements, building incrementally, learning and changing, etc.

On some anti-patterns, S&S pointed to people trying to do far too much upfront before they start to deliver value. No battle plan survives contact with the enemy. The inherently messy nature of data is our enemy - you will learn far more after you get moving forward than trying to plan and build everything ahead. Set your North Star and start traveling. Focus on CYA - cover your...butt... - governance. Keep out the layers of complexity - no feature stuffing. It's incredibly easy to overcomplicate.

Other anti-patterns S&S discussed were: 1) Not ensuring your technology and business sides of every use case are collaborating. Otherwise, you deliver the cool shiny thing that the business side didn't want or need. That's wasted effort. Spend the time to communicate better to prevent that. 2) Going for width of data on the mesh instead of making sure it is ready for consumption. Don't put data on the mesh if there isn't a specific use case for that data. 3) Having a messy data strategy. Becoming "data-driven" isn't a strategy. 4) Moving forward without the organizational maturity to really succeed. 5) As stated earlier, focusing on the technology to the detriment of the organizational aspects. 6) Calling everything an analytical data product. If it's a report or an insight, S&S don't think that should be called a data product. It muddies the water too much. 7) Not adding a clarifier to data product to make it clear exactly what an analytical or mesh data product is. The phrase "data product" is more often misunderstood than understood. 8) Playing on number seven, trying to mix in operational concerns - live transactions - into analytical data products. You will optimize for operational concerns and analytics will become a second class concern. Making analytics a first class concern is pretty crucial to effectively doing data mesh. 9) Chaining too many data products off each other. Downstream of downstream of downstream data products can become a real issue. Try to push use cases upstream where possible.

Per S&S, data work has had a high cost - it's been very difficult to get everything right historically and on-prem hardware purchase cycles certainly didn't help. You wanted to make sure something will have a big benefit before looking to invest in it. So there was much less innovation - just going after the low hanging fruit where the chance of success was high. And yet, many projects still failed. However, in data mesh, we can lower the incremental cost to experimentation and new data production very significantly - so you can actually test out more experimental, high chance of failure but high reward use cases. And it means you have the fast time to market capabilities to go after so many analytics opportunities we couldn't in the past.

Moving to the cloud means we can fail quicker and cheaper according to S&S. Data mesh is going to be extremely hard and expensive and likely not worth it if you try to do it on-prem in their view. And we iterate from failure to success. Historically, as mentioned, the cost of failure in data initiatives was high - monetary, time, etc. Now, if we fail, we can fail in much smaller ways much more quickly - each analytical data product has a much smaller investment in it than most historical data initiatives. That also means faster feedback, iteration, improvement, etc. So much smaller blast radius and cost of failures means we can be more experimental.

S&S referenced one of Zhamak's old figures from early in sharing about data mesh where there are three separate groups, the data producers, the central data team, and the data consumers. The producers have smiley faces, the consumers have a neutral face, and the central data team are all frowning faces. That's because of far too many handoffs between teams - that central team has ownership without context and are a bottleneck/overworked. Jesse Anderson mentioned going outside your team takes 12x as long for work to get done. So there is a big inertia - because it is how many orgs have worked - around trying to have teams specialize in things like ETL or the consumer self-serve platform. But for your own sanity, stay away from handoffs where possible around capabilities.

Scott asked about ownership of derived data products and S&S, as referenced earlier, emphasized that many analytical outputs should not be called analytical data products in data mesh. They delved into the concept of treating your data as a product but not necessarily organizing it into actual products. The differentiation was capital p...

Weekly Episode Summaries and Programming Notes – Week of September 4, 2022 Sep 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#122 Unlocking Your Data Value Through Good Product Practices - Interview w/ Alla Hale Sep 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Data Product Manager Role on Alla's team: https://jobs.ecolab.com/job/15770737/data-product-manager-barcelona-es/

Alla's LinkedIn: https://www.linkedin.com/in/allahale/

In this episode, Scott interviewed Alla Hale, a Data Product Manager at Ecolab. To be clear, she was only representing her own views rather than anything on behalf of the company. She is hiring in Barcelona as well, see here

Some key takeaways/thoughts from Alla's point of view:

The most useful question in your quiver as a data producer is "what value would having this unlock for you?" It's not about pushing back, it's about skipping to collaborative negotiation. How can you work together to unlock business value?
It's important to remember this when there is a data request: "they are coming to you because they need your help." Act accordingly - with empathy and patience.
You need to really consider your data user experience (DUX) for your data products. How can you quickly get people past figuring out "what the product is" to leveraging the data product to drive value? You want users to enjoy using your data product.
User stated requirements often do not match actual user needs. To maximize the return on your data work, look to exchange context to find the needs instead of just taking requirements at face value. And do so with patience and empathy.
No prototype, no meeting = having something tangible - even if that is simply a process map on a Post-It note - for people to react to. Otherwise, what will the conversation be about? How do you prevent the meeting from being a waste of time unless there is a specific topic to address?
We need to take lots of learnings/practices from tangible/physical goods product management when thinking about data products. We have users who have needs. How can we best serve those needs and drive value through that? It's always about serving the users' needs.
Another product management learning - how can we do fast prototyping? Prototypes have an actual cost, even in data/software. What gets us to value quickly? How can we capture value early as we iterate towards product quality?
All products - and their features - have a lifecycle. Be prepared to prune features or an entire product if they no longer drive more value than the cost to develop + maintain.
Discussing sunsetting and pruning should happen with users even in the development process. Far too often, data consumers are used to data assets being available in perpetuity - even as they degrade. We need data consumers to be part of an active conversation about if they are still using something and how to create more value or lessen costs to produce data.
How can we avoid the data product manager having to do literal sales and marketing of their data product internally to drive usage? How can the organization make internal communication easier instead of literally marketing data products internally?
To really create the most value from data products, there must be bi-directional communication. One, the data product owner should create demand for what is available. Otherwise, why build for reuse if multiple parties won't use your data products? But second, we also need to react to data consumer demand to drive incremental value through new data products or features.
You should never develop a new data product without a very specific use case - Scott note: this is a common question in data mesh conversations. It's about 75/25 split on only building to specific use case versus not from conversations thus far.
It's crucial to understand the data doesn't make the decision. And only look to measure things that will change your view depending on what the results are.

Alla started with what should data product management take from general product management based on her time managing products in a number of tangible goods spaces. Start from the basics: you have a user and they have needs that you want to try to meet. And that you are responsible for discovering and summarizing those needs, not that the user should understand all their needs upfront - that just leads to requirements that often don't actually address needs... Just be careful to extract needs and push back on requirements with patience and empathy. But it all starts with needs, not the data.

Alla gave some thoughts on needs versus requirements and expectation setting. Consumers understand in a physical goods space that you don't just magically have the prototype or product - but with data, there is often the misconception that "you already have the data, why can't you just share it?" So you often have to start conversations with setting realistic expectations - realistic expectations about next steps, timeframes, etc. Helping data consumers understand that you aren't deprioritizing them and what's actually possible and when - including why that is - will lead to a better relationship.

There is an understanding that prototypes have a real cost in the physical goods space, we need to make consumers understand it's the same in data and analytics, per Alla. And that different grades of prototypes mean different costs and time to develop - how high of quality do you really need this to be? How high quality do you need for the prototype versus the end-state? And exchange context around what requirements/needs drive what challenges so they might deprioritize needs for you.

Alla's key phrase is, "what would having this unlock for you?" Instead of pushing back by asking "why do you need this?", her framing gets the other person on the front foot, leaning in to the conversation to share what this could mean for them. It gets them to what Scott calls "collaborative negotiation", a process where you can quickly iterate to what's really of value instead of a list of requirements that might not actually serve the needs of the use case. This question extracts more context from the data consumer where you might even be able to add incremental value that wasn't part of their ask. Again, this is not about deprioritizing or pushing back, it is about driving to the business value. Remember, they are coming to you because they need your help.

A general rule Alla has is "no prototype, no meeting." That prototype can be just a small drawing of a process map on a Post-It note. The goal is if we are going to have a meeting, we are meeting about something - a useful conversation will be about something, something people can react to. This prevents the dreaded update but no actual reason to meet meeting - the epitome of this meeting could have been an email in Scott's words. Instead, concrete aspects of a prototype elicit a response and that response means you can decide where to focus next or if what's been developed thus far will meet needs.

Alla emphasized that in product management, you are designing for people and developing specifically for the user. People aren't robots and we need to give space for emotional responses instead of purely logical ones. And while you are looking to serve user needs right now, those needs might change - or you might fully serve the need with your product so the need doesn't exist anymore. Thus, you need to understand that products have a lifecycle. Don't be afraid to sunset things that are no longer valuable.

As well as sunsetting a product, the concept of pruning should be discussed with data consumers at the start of development. You want data consumers to understand that things will change and some of that is removing "features" of the product - which might be certain parts of a data set or certain ways of accessing the data product - when the costs exceed the value to maintain. You want data consumers to be active in the conversation about what is still useful and what isn't but sometimes you'll probably have to resort to turning something off and seeing if people scream. Hopefully the mesh tooling can give data producers good insights into what is being consumed but even that is often automatic consumption.

One thing that is quite dissimilar in data product management from her prior product management roles is previously, she had partners in R&D, marketing, and sales in the process. Many parts of what each of those roles added to the product were crucial in driving value. How can we make sure we are having the conversations internally to drive usage but without the data product owner and/or manager literally marketing and selling? Alla sees it as somewhat on the data product manager but also a much larger degree on the organization - how are data products marketed internally? How can people learn about new products or changes instead of consumers having to discover every data product themselves?

Alla shared her view that you should never develop a new mesh data product without a very specific use case. This is a common question in data mesh: what should be your reason for creating a new data product - a specific use case or the domain sharing information they believe will be useful?

An interesting concept Alla brought up is that all domains should think about who can benefit - especially who will benefit most - from the data you have. That way, you can reach out to drive towards collaboratively finding a use case to develop a mesh data product to serve that use case. Really consider what you've got and how that could be leveraged internally instead of simply waiting for requests. And it means there is collaboration from the start so consuming domains might be a better partner if you go to them. If your organization is going to encourage this, you need to find a way to incentivize/reward domains finding good use cases for their data.

For Alla (and Scott), a truly under appreciated need in data is the data user experience (DUX). You should aim for consumers actually enjoying using the product. Zhamak has mentioned similar things, especially her love of the book The Design of Everyday Things. Data can be intimidating to many, how can you make it so people don't feel stupid when first working with your data product? As Alla said, look to get people quickly past "what the product is" so they can focus on what the product can do for them.

Alla wrapped up by sharing her view of data-driven - it's crucial to understand the data doesn't make the decision. We need to use data to inform our decisions but at the end of the day, people still make the decisions. And use that thinking to decide what you want to measure and why. Katie Bauer mentioned this too in her episode: if you won't change your behavior no matter what the result is, why should you spend the effort to measure that?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#121 Zhamak's Corner 2 - Are You Ready for Data Mesh? Aug 30, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#120 Applying ML Learnings - Especially About Drift - To Data Mesh - Interview w/ Elena Samuylova Aug 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Elena's LinkedIn: https://www.linkedin.com/in/elenasamuylova/

Evidently AI on GitHub: https://github.com/evidentlyai/evidently

Evidently AI Blog: https://evidentlyai.com/blog

In this episode, Scott interviewed Elena Samuylova, Co-Founder and CEO at the ML model monitoring company - and open source project - Evidently AI.

This write-up is quite a bit different from other recent episode write-ups. Scott has added a lot of color on not just what was said but how it could apply to data and analytics work, especially for data mesh.

Some key takeaways/thoughts this time specifically from Scott's point of view:

A good rule of software that applies to ML and data, especially mesh data products: "If you build it, it will break." Set yourself up to react to that.
Maintenance may not be "sexy" but it's probably the most crucial aspect of ML and data in general. It's very easy to create a data asset and move on. But doing the work to maintain is really treating things like a product.
ML models are inherently expected to degrade. When they degrade - for a number of reasons - they must be retrained or replaced. Similarly, on the mesh data product side, we need to think about monitoring for degradation to figure out if they are still valuable or how to increase value.
Data drift - changes in the information input into your model, e.g. a new prospect base - can cause a model to not perform well, especially against this new segment of prospects. That data drift detection could actually be a very useful insight to pass on as an insight - has something changed with our demographics? If so, what? When? Do we know why?
Concept drift - the real world has changed so your model is not performing as expected - is a crucial concept in data and analytics too. Are we still sharing information about the things that matter? In a way that is understandable? Are we encapsulating what's happening in the real world in our mesh data products?
Concept drift feels similar to semantic drift in the analytics world. So we can look to potentially take deeper learnings from how people approach and combat concept drift from ML and apply it to data mesh.
How can we monitor degradation in mesh data products and prevent that degradation our data and analytics work? Historically, reports drifted further and further from reality with no intervention because the pain of change was so high. Are we fully reliant on the domain to know? Can we use software to help us detect semantic drift? Very early days on that one.
ML models are designed to do one thing very well. Unfortunately, we don't have a good framework for reuse at the model level in ML. Maybe at the ML feature level?
ML models have expected performance metrics. Those expectations need to be set through conversations between the business team and the ML team. Measure using KPIs. Can we use a similar approach to expectations - at least for some specific use cases - for a mesh data product?
When building an ML model, you need to consider scope, business purpose, expectations, measurement against expectations, etc. Similarly, when doing any data work, you should consider the same. It is somewhat hard to measure the impact of most mesh data products but it doesn't mean you shouldn't try. What are you trying to achieve with the data product and is it meeting those expectations? Is the business need still relevant or has it changed?
Regarding graceful evolution and preventing breakages due to changes in sources or downstream breakages from changing the ML model and/or its outputs, ML unfortunately does not have any answers that we aren't already using on the data and analytics side. Good communication, contracts, monitoring/observability, etc. No silver bullet or MLMFD - ML Magic Fairy Dust.
The concept of a feature in ML - a smaller component of the model that might be reusable across multiple models - could be interesting to consider in data mesh. It would likely break with Zhamak's view of each data product owning it's own transformation logic but could create almost proto-transformed data. Almost like a service bus to easily serve data products. Probably has lots of drawbacks but interesting to consider.
Guardrails on ML models help to keep the models from doing things like reacting to data that is out of the norm. As Elena said, if an ML-based recommendation on a website is a bit off, the conversion rate falls but that's not the end of the world. But what if you are dealing with big dollar decisions? Should we look to proactively put in guardrails into our data products? Probably yes, if they are driving crucial decisions - consider failure modes and what to do in those cases.
Getting to fast incremental value is crucial when developing ML models. There needs to be very good trust and communication so people understand the initial quality level might not be great as you iterate towards a better model - or mesh data product. This is becoming a common theme - how can you release a v0.1 or v0.0.1 of your mesh data product and still drive value now while getting it to v1.0?

Elena started by sharing a basic definition of the concept of drift in ML. Drift causes model degradation -so the model is not as effective as expected - and can be generally split into data drift or concept drift. Data drift is typically something about the source of data you're using for your model has changed. That doesn't mean using a new source, more like you are interacting with a different set of prospects or customers than you were previously so your predictions as to their behavior are going to be wrong - you built a model to react to a different set of people. Concept drift is more aligned - at a very high level - with semantic drift in data and analytics - it is that some aspect of the real world has changed. If you look at spending habits, especially in ecommerce, between February 2020 and April 2020, as the global pandemic started to take off, the real world changed a LOT. That was an extreme example but the real world is ever changing - how can we make sure we are still measuring and sharing the most meaningful information in our mesh data products?

A very important aspect of ML model drift, per Elena, is that it is entirely expected. Drift, and it's resultant model degradation, is part of ML model reality. There is a cost of dealing with drift but when an ML model is negatively impacted, it is no longer making optimal decisions. So when you detect said drift to a certain degree, you would retrain the model, or shut it down, or replace it with a new model. It's also hard to say how long a model will be in production before it seriously degrades. Or what is the degradation threshold when you should retrain or replace. Similarly in data mesh we need to think about how we evolve our data products to prevent degradation. ML models are purpose-built to do one thing but start to degrade over time. Often in data and analytics, we've used data assets the same way - we kept using the same reports as they degrade but don't replace or evolve them. We need to do better in data mesh.

According to Elena, good ML practice means each model is designed to do one thing very well, not to do many things instead. There is sometimes misuse of ML models in organizations as people try to make use of the same model for multiple use cases. This is similar to the way a number of people use data assets - created to answer one question but leveraged to try to answer another. If there isn't a good understanding of exactly what the data asset addresses and how, it often leads to bad/incorrect conclusions on answering other questions.

So, how do we measure if an ML model has degraded? And how do we fix it if it has? Per Elena, you should measure your model against a certain set of expectations, typically via KPIs. If the model is no longer hitting expectations, it has likely degraded. Then you would look to retrain it - use the same steps as before to train your model but with the most recent data - or replace it.

Elena believes the most important aspect of building an ML model is communication first. What are you trying to actually do? What is the business reason for creating a model? When the model is created, what are reasonable - and how can we stay away from unreasonable - expectations? What are the business metrics to create the KPIs around? How will you track performance against expectations/KPIs? These same types of questions can be applied to a mesh data product. Why are you creating the data product? What is the target use case and what is the expectation for the use case? Is the use case meeting expectations? If not, is that because of the data product or the use case itself?

Scott asked Elena about graceful evolution of ML models - how can we set ourselves up to deal with upstream changes more easily and how can we manage to not break things for downstream consumers. Her answer was unsurprisingly familiar: lots of good communication, using (data) contracts, using monitoring/observability tooling, set guardrails, etc. Similar to the data mesh concept, Elena believes you should really think of each model like a product.

When asked how the ML and analytics sides of the house can better collaborate, Elena hopes that in many organizations, they aren't overly separate. Embedded ML engineers are similar to the embedded data and analytics capabilities/teams model many are using with data mesh. And she hopes once the super fast pace evolution of data stacks slows down, maybe both sides can start to consider using the same tooling. But the biggest driver will be good communication - it pretty much always comes down to communication...

As mentioned earlier, Elena strongly believes you should not try to use the same model for multiple purposes. But in ML, there is the concept of a feature. Essentially, it is a part of the model that might be used for multiple different models, producing a subset of the model's data input. So a feature could be reused across many ML models. This feature concept might be interesting to explore in a kind of proto mesh data product way. That way, we prevent multiple data products from doing the same work. An ideal way to prevent this reuse is communication - as Omar Khawaja discussed in his episode - but a company-wide source data catalog or repository could be a way to ensure everyone knows what data is being transformed and who owns it. Especially when new data products are in development so there is a much smaller chance of teams doing the same work.

We can learn a lot from ML monitoring/observability, per Elena. In ML, you need to monitor the overall ongoing quality of data ingested into the model, the quality of the output of the model, and also the quality at the point of ingestion. Often, that last part - quality at point of consumption - is managed by guardrails. If data is not within a certain specification, it is not passed into the model. Or the model doesn't react outside of certain bounds. Or if some metrics about the model relative to historical norms are very off, the model essentially gets paused and there is a failover to a less rigorous solution. Sometimes, passing bad data into the model is not the worst outcome - your Amazon recommendation is for buying another toilet seat... how many of these do you think I want Amazon? But ML models can power very big-dollar decisions. And these guardrails could be very useful in data mesh if you are driving decisions with a fast turnaround. Alert that something is unique and see is there a new normal or was there something funky upstream.

Elena mentioned ML development sometimes has a perfectionist issue: people try to get to a perfect model before deploying something instead of getting to fast value - putting something into production that incrementally increases value quickly - while you improve your model through quick iteration and tight feedback loops. This is becoming a very common theme in many interviews: how do we get to incremental value very quickly while we improve the long-term mesh data product through fast iteration?

Elena wrapped up on two thoughts: 1) Maintenance of your ML models isn't "sexy" but it's probably the most important aspect. Maintenance is proactive maintenance, monitoring, setting up good feedback channels, communication in general, etc. It's not just the model in a vacuum - is it having the impact you expect. And be prepared to pay for the maintenance. Which plays into 2) "if you build it, it will break." Set yourself up to detect the issues and make sure you budget people's time to keep things running and fix it when it breaks. And don't be surprised when it breaks.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 28, 2022 Aug 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#119 Cautionary Learnings From a Startup Doing Data Mesh: Orfium's Journey to Decentralized Data Success - Interview w/ Argyris Argyrou and Konstantinos Siaterlis Aug 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Argyris Argyrou's LinkedIn: https://www.linkedin.com/in/argyrisargyrou/

Konstantinos "Kostas" Siaterlis' LinkedIn: https://www.linkedin.com/in/siaterliskonstantinos/

In this episode, Scott interviewed Argyris Argyrou, Head of Data, and Konstantinos "Kostas" Siaterlis, Director of Big Data at Orfium. There is a ton of useful information on anti-patterns, what is going well now, advice, etc. in this one.

From here forward in this write-up, A&K will refer to Argyris and Kostas rather than trying to specifically call out who said which part in most cases.

Some key takeaways/thoughts from A&K's points of view:

On a data mesh journey: "It's not a sprint, it's a marathon." Pace yourself. It's okay to go at your own pace, don't worry about what other people are doing with data mesh, do what's right for you.
Really focusing on the why and showing people results was a far better driver to buy-in and participation than any amount of selling about data mesh as a practice. Calling it data mesh when trying to explain it to people outside the data team didn't go well either...
Orfium's "Data Doctor" approach - a low friction and low pressure office hours for a general staff data engineer - has really helped people help with data challenges and in spreading good data practices but without the "Doctor" becoming a bottleneck.
The Data Doctor's role is to answer questions and provide guidance but not do the work for people. Then, take what was discussed and the best practice and document it for others to learn from - providing good leverage for scaling best data practices.
In a smaller company like Orfium (~250 people), it's hard to justify a lot of full-time heads to implement data mesh. And trying to treat a data mesh implementation like a side-project also creates issues. There isn't a great answer here on exactly what to do except possibly take things slower than most startups are used to. Your data will still be waiting for you a few months later.
If you are having difficulty driving broad buy-in, showing people what data mesh can do in action really helped at Orfium. Once they saw the approach delivering value, they wanted to participate.
When trying to drive buy-in, specifically talking about data mesh didn't work well with non data folks. It's very easy to get confused around data mesh for data folks - just imagine it for non data folks.
Trying to use Zhamak's articles as the optimal early state - where you need to be just to get moving - requires far too much work. Get to a place where you can try, learn, iterate, and repeat on your way to driving value. It's a journey!
It's probably not a great idea for your first use case to be your most advanced or complicated - you will build your platform to focus on serving those needs instead of general affordances. Jen Tedrow's episode covers this quite nicely.
Really assess how much additional work your data products will be for a data product owner. For Orfium, it was something to add to the existing product managers' plates as it wasn't a huge incremental burden just yet.
Consider splitting your mesh data product ownership between business context ownership and technical ownership.
It's okay to head down the data mesh path while learning what domains really mean. Orfium was not doing Domain Driven Design in any sense before starting to do decentralized data.
Don't try to get all your teams to start developing data products at the same time. It is very hard to work with and upskill that many teams, even in a smaller organization.
A&K recommend to start slower than you probably want on building out your platform; and start small in general, don't take on the biggest challenges or too many challenges at once.
It's okay to have a very high-level concept of a mesh data product. For Orfium, that is: it's a product that is designed like any other software product - it solves problems for customers and is delivered via API. That helps the non technical people understand what they are delivering.
"Really think about what you are doing and why. Why complicate it more than that?" - said about data products but pretty universal in data mesh. It's easy to overcomplicate...

About 2.5 years ago when Argyris joined the company, Orfium - a growing startup in the music industry royalty business - was really starting to see a big uptick in data requirements to serve their customers and offer new features/capabilities. They had 3-4 people doing the data engineering and data science work and another centralized BI team but the need for more advanced ML and AI was becoming clear. The centralized data capabilities were becoming a bottleneck so either they'd need to significantly scale their number of people in centralized data functions or look to decentralize in some way. They decided to try data mesh because they were feeling the exact challenges Zhamak so clearly laid out in her articles.

For Kostas, he was really bought in to data mesh's self-serve aspect. Pretty much all the data in the company for analytics was flowing through him and his small team and that was understandably draining. Data mesh is more of a cultural approach than a technical one. He said, "I will build something interesting either way" so the loss of certain data engineering tasks meant he could focus on building the platform, which was just as or more interesting.

At first, for A&K, data mesh was definitely a bumpy road - they tried to do data mesh as a side project while keeping up with everything else they were doing. They couldn't justify treating the data mesh implementation as a main focus at the expense of the many things in production. They didn't have the spare headcount to do it either. As they built the first version of the data platform, they were having difficulty explaining data mesh to non data folks. So, the team most bought in and willing to try using a data mesh approach was the central data engineering team. Thus, their first version of the platform was advanced data tooling for data intensive use cases with the central data team as the main users. It worked well for the central data team but once they tried to get other users, the platform wasn't really built for people who weren't highly data literate with intensive use cases so driving buy-in with other teams was hard.

On trying to drive buy-in, A&K worked to explain what they were doing and what data mesh was internally. They started from the why, why would this matter and that got the exec team excited. But when they tried to sell data mesh to the engineering managers and engineers, it fell flat. The non data folks didn't really understand the nuances of data mesh - and they didn't really need to in most cases to participate and benefit from the data mesh implementation. It's very easy to overwhelm people with all the aspects of data mesh instead of what matters.

What ended up driving buy-in quite well, per A&K, was seeing the output of treating data like a product in action. Once there were tangible benefits and people could see what value a mesh data product could deliver, they were much happier to move forward in participating in the data mesh implementation.

Per A&K, a few of their missteps - so you can avoid doing the same - were: 1) trying to sell data mesh by the principles and using the phrase "data mesh" instead of the why and what changes for who they were talking to; 2) building the platform to serve the most data intensive use cases owned by data engineers so it was quite hard and not really suitable for others to use; 3) trying to get to advanced maturity in all parts of the data mesh implementation up front - e.g. they don't have fully automated access control but it's not really a big pain point; 4) thinking that Zhamak's articles or book are the blueprint for where you have to be at early stages instead of an inspirational goal multiple years into a data mesh journey; and 5) trying to get all teams to move together at the same time with data mesh instead of working team by team or mesh data product by mesh data product.

So, what _did_ work for A&K at Orfium in their data mesh journey? Again 1) showing people the results and the value the approach could deliver for a use case; 2) not hiring new people into roles - when they looked at the additional workload, with good support and upskilling/partnering, the domains could handle it as they were - per Scott, this tends to be the case in smaller companies with smaller domains doing decentralized data; 3) focusing on the why: why does this matter? If we get this right, what will that get us? And 4) using data mesh as an enabler to change people's hearts and minds about owning and using data; it's now more a core part of teams' responsibilities and they are taking it seriously.

What does good look like for Orfium in their data mesh journey right now? Per A&K, their teams understand the difference between operational and analytical data and are starting to manage their analytical data as a product. It has changed the role of data engineering and how people perceive the central data team internally to being the enabler, not the team that does the work for them. Their ML/AI teams are able to get quality data reliably so they can build out new use cases. They still don't have fully automated access control but focused on making it far easier to request and grant access and that's a good enough point for them at the moment. They are at 10-15 products - hard to say exactly what constitutes a product, hence the range - with far happier data consumers.

Orfium wasn't doing Domain Driven Design (DDD) on the operational or data side of the house prior to starting on their data mesh journey. And A&K think it's totally viable to not be doing DDD at all before starting. Other guests on episodes more centered on DDD for data have said similar things. This is permission to move forward.

A&K gave some insight to how attitudes and understanding relative to data have changed at Orfium. Previously, someone would want to do some analytical work against a team's data and the product manager would get them some kind of DB access to the data, so they could only access the data as it was stored for the operational system. There wasn't a clear separation between operational data and analytical data. So a lot of the evolution was just getting them to understand how their data might be used for analytics and produce and own data to be used for analytics. Far easier said than done but still achievable. And what also really helped was splitting business and technical analytical data ownership.

So, with all this learning behind them, what are a few bits of advice from A&K? Start slower on the technology build out; it's exciting to build cool stuff but that can wait - what do you need to get to the low-hanging fruit value. Start small, don't try to have all the domains (or if you are not doing DDD yet, your teams) move all at once. Don't mention the phrase data mesh to people outside the data team - it typically just generates confusion - speak to the value of the approach and not the technology or what it changes for the data team - what does it change for them?

When working with teams to understand the concept of a mesh data product, it's easy to overcomplicate things per A&K. For Orfium, there is a technical definition of a mesh data product and a business one. The business definition is quite simple: it's a product designed like any other software product - it needs to solve a problem for customers and is delivered via API. Really think about what you are doing and why, "why complicate it more than that?" The data products create a platform of information for teams to build data-informed applications on top of. Oh, and don't forget really good documentation for your data products.

Orfium has a really interesting concept they are using internally: "The Data Doctor". Essentially, people go with their "data symptoms" and the data doctor gives them a "prescription" - advice on how to address their challenge using best practices. It's a low pressure way to have something like a staff data engineer hold office hours to help people with their data challenges to ensure people follow best practices but also learn how - and have the confidence - to implement the recommendations themselves. And then, the Data Doctor needs to work with whoever implemented their advice to document the process and put it in a central repository so others can easily follow the same practices if they hit the same or a similar challenge.

Per A&K, on your data mesh journey: "It's not a sprint, it's a marathon."

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#118 - Zhamak's Corner 1 - Is Data Mesh Right For You? Aug 23, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Sponsored by NextData, Zhamak's company that is helping ease data product creation.

For more great content from Zhamak, check out her book on data mesh, a book she collaborated on, her LinkedIn, and her Twitter.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Data Mesh Radio episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#117 Data Mesh and Fight Club - How Should We Discuss Data Mesh Internally - Mesh Musings 27 Aug 22, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#116 A Startup's Early Journey Towards Decentralizing Data - Iterable's Analytics Evolution - Interview w/ Riya Singh Aug 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Riya's LinkedIn: https://www.linkedin.com/in/riyasingh1/

In this episode, Scott interviewed Riya Singh, Business Insights Manager at Iterable.

Some key takeaways/thoughts from Riya's point of view:

~4 years ago, Iterable was in essentially "spreadsheet hell" with lots of manual data work and no standard way of storing or sharing data across domains. While domains had good data capabilities, the integration and coordination between domains was very difficult at best.
Most exec questions can't be answered by the data from a single domain so cross domain data integration became a key factor in Iterable continuing to grow. How could they make crucial decisions informed by data if there was so much manual work to try to integrate ad hoc? Could they really trust something done manually each time?
Fast time to market for simple, base level capabilities of their data platform was much more valuable than trying to nail every feature upfront. Data consumers understood it wasn't perfect data at the start but it led to much faster exploratory data initiatives which led to valuable insights sooner.
You might have a much higher ROI buying tools than trying to really get by on low-cost but not feature-rich tools. If you build a very cost-efficient data platform that no one wants to use, is that actually valuable? How much time will you spend managing the tools or is it worth it to outsource that to a vendor?
Combining data across sales, marketing, and product meant Iterable could tailor marketing messages and find better prospects, measure marketing return on investment (ROI), and cost optimize their operations and product among many other new insights.
As teams that previously weren't directly interacting start to have more conversations, gaps in your data - whether in data created/collected or data shared - will emerge. Filling those gaps will mean you can answer more high-value questions to drive the business forward.
At Iterable, when there is a specific use-case identified for cross-domain data integration, the central data team takes over ownership of what would be considered a consumer-aligned data set in data mesh terms. With only 4-5 domains, Iterable doesn't need to decentralize the data team yet. The cost of decentralizing is far greater than the benefit right now.
Iterable found the most value by doing exploratory data analysis then quickly moving to minimum viable consumable form. Then, they work to continue to improve the data set. This approach means a fast time value by grabbing the low hanging fruit while continually driving to better data and incremental value. But to do this, consumers must be very aware of what they are getting when :)
A key way to keep stakeholders informed and bought in is by constantly keeping them updated on progress - Jen Tedrow talked about this in her episode. Keep people informed of progress or ongoing investigations so you can stay coordinated and all parties understand decisions along the way.
At Iterable, conversations between domains are happening weekly. That way, there is always space for people to keep each other updated on upcoming changes or new information. It makes it easy to keep each other informed. Far harder for organizations with many domains but still useful general advice - much like in any good relationship, schedule the time to exchange context!

While Iterable is too early in its life to fully look to implement data mesh, they are taking some major inspirations in how to enable domains to appropriately share data with each other, per Riya. When Riya joined, they were under 100 people and had a very fractured, one-off analytics approach built on a lot of disparate Excel spreadsheets. It was very hard to make data-driven decisions even within many of the domains but especially on cross-domain questions - which are often the most crucial to companies. Each domain had a relatively high data maturity, enough so that the centralized BI/data team thought they could and should still own at least some of the responsibilities to their data.

But, per Riya, to move from a fractured environment, they needed to at least consolidate sharing information into a single place or tool and to start to standardize how information might be shared and combined/integrated across domains. An example Riya gave was how the Product Team data was stored in such a different way than the sales and marketing information but Sales and Marketing both wanted to consume metrics from how customers and prospects were using the service. But again, integration across domains was extremely hard to do. They also needed a standardized ownership model across domains - what would the domains own and what would the BI/data team own?

So, what was the driving factor to push Iterable out of essentially the dreaded "spreadsheet hell"? Riya mentioned that the executives were not able to easily and repeatably/reliably get answers to their questions in a timely manner. Many crucial, exec-level questions are not domain specific and require data from multiple domains. And most exec questions aren't a single point-in-time question so you need to set up reliable processes to support answering those questions now and into the future. So exec pain at not being able to quickly make decisions backed by data - especially data they could trust - meant there needed to be a change.

Riya gave a few specific examples of big questions they couldn't answer in the fractured setup. 1) What makes for a good target prospect? You need to combine sales and marketing data about company type, size, etc. with purchase history and combine that with the product usage data to see what types of companies were actually using which features. And then combining in the renewal rate and how much each customer expanded. It could also give them an ability to target pitches based on which features were used by which type of companies. 2) What is the actual ROI on marketing spend? Scott, coming from FP&A, agreed this is an insanely difficult question without really good data. But it's notoriously difficult without really clean data. 3) What features or business programs should we kill or reduce investment? To cost optimize, you need clean data on what is actually happening with the business and product down to quite granular levels so you aren't making crucial decisions based on gut.

Per Riya, one very important outcome of the work to combine data across domains: identifying additional gaps in their data and analytics. Were they collecting the information to answer these new questions? If the domain had the information, could they share it? Once basic questions were more or less answered, they could see where they could do better on what information they collected and shared to drive deeper, high-value insights.

When starting to build out the cohesive, company-wide data platform, Riya and team looked at a number of tools. She said they made a few good choices and a few things that she would do differently. They focused a bit too much on trying to provide a really simple UI/UX instead of just getting data sharing and analyzing capabilities in people's hands and then working to improve from there. They eventually saw really big value in getting people doing initial exploratory work quickly - reducing the time to market of people being able to use even base-level features was very valuable.

They also went with something that made ETL far harder to manage than it should have been in Riya's view. Sometimes those more expensive offerings like Fivetran will have a much better return on investment - ask yourself how much of your time will be spent managing a system instead of value-add work and if that would be better spent on buying a tool. As Doron Porat mentioned in her episode, it's rarely a super easy choice but too often, people opt to try to roll their own when it's more valuable to focus on higher value-add work.

They are now using Snowflake, Fivetran, Looker, and dbt.

Iterable saw a lot of value again from the initial exploratory work between teams driving insights - once they found a good use case, the teams came to Riya and team to build out the data models that could easily combine that data between the domains that would allow for high quality, trustable data in a format that was easy to use. Essentially, once there was a clear use case, the central data team took over ownership to ensure and maintain quality. As seen in many previous interviews, this central ownership model scales until it doesn't. Right now for Iterable, centralized ownership of what might be considered consumer-aligned datasets in data mesh makes sense. Data mesh, as Zhamak has envisioned it, is not for all companies and a hybrid ownership model like Iterable is using can scale quite well for organizations with not that many domains.

So v0.1-1.0 of Iterable's data platform was Snowflake + Stitch + Looker, then v2.0 was Snowflake + Fivetran + Looker + dbt. Riya and team are starting to work on v3.0 of their data platform to support some initial data science / ML use cases. What they've found is far too much of the central data team's time is spent on manual tasks so they will be focusing on these new ML use cases as well as building in more automation and optimization.

A few interesting things from Iterable's approach: 1) by starting with exploratory analysis first, they could discover low hanging fruit insights while working to elevate the data set to production quality. Some things were just obvious in the data even before it was high-quality. Getting to an early initial consumable form and iterating towards higher quality drove value sooner; 2) the domains are constantly in communication with weekly check-ins. This gives them scheduled time to keep each other informed. Easier to do in a company with 4-5 domains but it means fewer surprises and more high-value collaboration; and 3) something that has worked well for Riya's team is constantly keeping stakeholders updated as work around data progresses. By keeping people in the loop, there is a tighter feedback cycle if expectations aren't aligned or diverge, meaning far less chance of wasted work.

Riya wrapped up by mentioning how crucial high context conversations really are to making your data strategy work no matter your data management approach. If the domains were just trying to drive their own data, the overall company would be flying blind. So find good ways to keep each other informed and exchanging context!

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 21, 2022 Aug 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#115 Understanding the Data Value Chain - Your Key to Deriving Value from Data - Interview w/ Marisa Fish Aug 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Marisa's LinkedIn: https://www.linkedin.com/in/marisafish/

Obeya method: https://obeya-association.com/what-is-an-obeya/

MIT course on "The Science of Intelligence": https://cbmm.mit.edu/education/courses/science-intelligence

John Duncan paper on brains executing series of programs: https://web.mit.edu/9.s915/www/classes/duncan.pdf

In this episode, Scott interviewed Marisa Fish, Director of Information Management at American National Bank. To be clear, Marisa was only representing her own views on the episode.

Some key takeaways/thoughts from Marisa's point of view:

Understanding your data value supply chain - the way you derive and deliver value from your data - should be the crux of data and analytics work. The data value supply chain breaks down into sharing the data itself, sharing analytical insights about the data, and managing the data. All three are crucial to creating value from your data.
Intentionality is crucial - instead of being reactive, stop and ask what are we trying to accomplish and what value will it drive. Then you will focus much more on high value-impact work.
Similarly, think about system engineering work as "mission engineering" - what is your mission in doing your work? Does the work you are prioritizing serve the mission?
When sharing information, start from: what is the point, what am I trying to drive with this information exchange? Are you trying to share one person's way of thinking or insights or give others the capability to derive their own insights from the new information? Both are very valid and useful but it's easy to talk past each other if you're not on the same page.
So much of the way most organizations work with data is about the known knowns - the data consumer knows what data they want and what questions they want to answer with the data. We need to enable people with questions to find the right data to address them and people to also do data spelunking with data they aren't sure what it might tell them. Look to the Library and Information Sciences space for how to approach that.
We need data librarians, not data publishers. Data publishers are about putting data on the shelf and serving only the known knowns. Data Librarians are there to help people find the information they need to address more of the unknowns - the value of curiosity in driving incremental valuable insights.
There is a major mismatch in most organizations between what insights the business units are producing and the key questions the C-level execs care about. Consider creating a Chief Data Analyst type role to pair with execs to make sure insights are produced to support their initiatives, not just answer their questions as they come up. Think ahead, build ahead.
Data teams need to take far more practices from general engineering - not just software engineering - so we learn how to better understand requirements.
When requirement gathering, expecting the data consumers to know all of their requirements upfront can lead to data consumers asking for the world and a bad mismatch between asks and needs. Look to new ways to exchange information about requirements including the Japanese Obeya technique.
Spend the time to ensure you understand how data consumers will derive value from the information you will share with them. That will give you a better understanding of how best to serve them and what stated requirements might not be quite so required.
Anytime you are sharing data, it's easy to get bogged down in the 1s and 0s. Ask yourself: what information am I trying to share and why? How is this driving value?
It's very important for data producers to really dig into use cases to prioritize the work but also to make sure you aren't over-optimizing or under-delivering on the value. What is the point of the work? And what do the data consumers want: analysis/insights or data?
Data producers/owners are often not willing to openly share all of their data. A big reason is compliance - with internal policies, regulations, etc. So a high context exchange of how a data consumer will actually use the data can lead to more data openly shared - the producer can be assured there won't be non-compliant use.

Marisa started the conversation sharing that she is hard of hearing and how that has strongly impacted the way she interacts with the world. With often missing certain words in conversations, she - and her brain's linguistic processing - have to work in a far different way to fully comprehend the meaning of what is being discussed. And we need to think about how we can use a similar approach to data - we won't always have all the context, how can we apply neuro-linguistic approaches and human data processing frameworks and apply them to data to better expand our understanding of the data?

For Marisa, any time you are sharing information in the form of data, it's important to understand that it isn't just machines communicating with each other. In all forms of sharing information and knowledge, you should ask: am I trying to mimic and share one person's way of thinking or am I trying to augment the way of thinking of the audience? In other words, am I trying to share one person's understanding or am I trying to give someone else the information to create insights and deepen their own understanding? Both are very valid and helpful but really focus on: what is the actual goal of the information exchange?

Marisa recently moved from a 25+ year career in the US Department of Defense into the financial sector. So she is learning a completely new "language" - actually several. The terminology, the business terms, the ways of exchanging information, the way information and requests flow, etc. From her years of working with very high-impact information exchanges in the DoD - intel drives foreign policy and can put many lives at risk - she understands the cost of data producers and consumers not aligning. So she recommends to start from a conversation of what is the point of this data request or work. And as Jean-Michel Couer said - not in a combative way but it's crucial to ask.

It's really crucial to dig into the use case and business need for a few reasons. That context exchange is crucial to driving any data initiative forward including prioritization. The data producers asking the consumers what they are trying to achieve means they have the context to better serve their needs instead of the data consumers having to know every requirement upfront - they have far too many known unknowns about the data. And it also ensures there is a business reason for the ask - how are you, the data consumer, going to derive value for the business from what we plan to share? If it isn't going to drive value, is there a benefit to doing the work?

Marisa and Scott discussed how difficult it can be to openly share data internally without really knowing what the downstream use cases are. As past guests have noted, without understanding the exact use, domains will not share as much of the data in most cases. Why? Because it's very easy to get into non-compliant use. And it's also easy for people to misinterpret and misuse data if you don't give them the context to truly understand what it means. So to have a more open sharing environment, especially in an industry or organization where data policies are stringent, sharing the context with each other is crucial. Data consumers must share about target use and how they will prevent misuse.

The data value supply chain - which is the core way to think about how you derive and deliver value from data in Marisa's view - breaks down into three parts: the data itself, the data insights/analytics, and the management of the data. If we look at data mesh with this lens, we are asking domains to at least take on sharing the data and the management of the data - with a lot of help from the platform. But as many guests have noted in the past, data mesh practitioners are somewhat split on how far the mesh data product extends - do you want to prepackage the insights for consumers or package the information up so they can derive their own insights? I think we are coming to an early pattern of it's both, where possible and valuable. Look to Xavier Gumara Rigol's episode for an in-depth dive into this.

Marisa shared about her work in the Library and Information Sciences world and how it applies to data - when it comes to exchanging data, so much of what we do currently is about the "known knowns". The consumer knows what data they want and what questions they want to try to answer. But that is only one of the four quadrants of information. While it can be truly difficult to grasp the unknown unknowns - not knowing what information sources or data you want and what questions you want to answer - the known unknowns and unknown knowns are crucial to expanding our understanding of what is occurring with our organization. Known unknowns is I have questions but not sure what data can help me answer them and unknown knowns is I know I want to analyze and leverage this data but what can it help me answer or what is it telling me.

There is a big difference between a publisher of data and a data librarian in Marisa's view. And you should look for the data librarians. A publisher of data is someone who just provides the 1s and 0s and not the real information. A data librarian is someone who facilitates finding information. So how can we do that at the organizational level? It's difficult enough to get people to be that data librarian for their own domain, how can we do that across domains? Does that need to be at the platform or human level? Is it we want both to overlap and serve as much as we can?

Far too many organizations treat the data team as a service-based model, per Marisa. This has been a common theme across many articles over the last few years. A service-based model makes you likely to be run as a cost-center instead of a profit center. So how do you switch that perception? Part of that is doing work that is directly tied to the data value supply chain. For every bit of work, ask: how will value be derived?

But, on moving away from a service-based model, Marisa and Scott agree that there still needs to be someone pairing with C-level executives to make sure there is information gathered and collected to support their key initiatives. A kind of Chief Data Analyst. Rather than ad hoc questions and "I'll get the information for you", it's someone aware of the key initiatives to the company that is ensuring insights are being manufactured to support those initiatives. There is a major mismatch in most organizations between what insights the business units are producing and the key questions the C-level execs care about.

For Marisa, to identify what to do to satisfy a data consumer's needs, you need to really understand their way of working. Do the operational business process mining to figure out how to best serve them as you work to build out the use case.

Marisa shared her feelings about getting overly focused on the small picture - the micro - and how that plays into the big picture - the macro. If you spend too much time focusing on individual use cases, especially gathering requirements, will you be able to scale up sharing data internally? Possibly look to the Japanese requirements gathering technique called Obeya - or open room. A data producer can lay out a considerable number of possible requirements and the data consumer can opt in to those requirements. And then the data consumer can avoid some of the unknown knowns - what don't they know about the data? Then, there can be an open exchange about those requirements between data producer and consumer.

In data, so much of the work most organizations do is a direct reaction to a request or question instead of focusing on "how are we going to derive value from this work?", per Marisa. We need to take practices from engineering - not just software engineering - on requirement gathering and understanding. How many people in the data space have really trained in proper requirement gathering? Is this mismatch between gathered asks and what the project actually needs to succeed the reason why so many data initiatives fail? Stop simply reacting to requests, ask why this matters, what value will it drive. Scott's favorite data mesh word: intentionality. Think of it as "mission engineering" - why are you doing this? What is our mission and does this support our mission?

Per Marisa, when we as humans collect information ourselves - not in our systems - we mostly do that through hearing and vision. So, how do we think about our electronic information collection systems? Can we get our systems to better mimic the way humans collect and process information? Should we try to mimic them exactly since humans have inherent bias? What can we learn from the way humans collect and process information and then improve upon those? What is the science of intelligence and how can that impact the way we build systems? Our brains execute a series of programs for information processing. How can we leverage multiple frameworks to do the same?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

A Call to Action - Please Consider Being a Guest! Aug 16, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, and/or nevesf

#114 Protecting the Meaning of Data Mesh - Mesh Musings 26 Aug 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#113 Data Governance In Action: What Does Good Governance Look Like in Data Mesh - Interview w/ Shawn Kyzer and Gustavo Drachenberg Aug 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Gustavo Drachenberg's LinkedIn: https://www.linkedin.com/in/gusdrach/

Shawn Kyzer's LinkedIn: https://www.linkedin.com/in/shawn-kyzer-msit-mba-b5b8a4b/

Data Governance In Action: What Does Good Governance Look Like in Data Mesh - Interview w/ Shawn Kyzer and Gustavo Drachenberg

In this episode, Scott interviewed Shawn Kyzer, Principal Data Engineer, and Gustavo Drachenberg, Delivery Lead at Thoughtworks. Both have worked on multiple data mesh engagements including with Glovo starting 2+ years ago.

From here forward in this write-up, S&G will refer to Shawn and Gustavo rather than trying to specifically call out who said which part.

Some key takeaways/thoughts from Shawn and Gustavo's point of view:

It's very easy for centralized governance to become a bottleneck. Make sure any central governance team/board that is making decisions has a way to quickly work through backlog through good delegation. Not every decision needs deep scrutiny from top management.
To do federated governance right, you need to enable the enforcement - or often more appropriately the application - of policies through the platform wherever possible. Take the burden off the engineers to comply with your governance standards/requirements.
Domains should have the freedom to apply policies to their data products in a way that best benefits the data product consumers. So if there are data quality standard policies, the data product should adhere to the standard for measuring completeness as an aspect of data quality but might be optimized for something other than completeness.
The cost of getting anything "wrong" in data previously has been quite high because of how rigid things have been - the cost of change was high. But with data mesh, we are finding new ways to lower the cost of change. So it is okay to start with policies that aren't complete and will evolve as you move along.
If you have an existing centralized governance board, that will sometimes make moving to federated governance ... challenging at best ... so you will need a top-down mandate to reshape the board. Look to meet the necessary representation across your capabilities (e.g. product, security, platform, engineering, etc.) but not create a political issue if possible.
Look to add incremental value through each governance policy. And look to iterate quickly on policy decisions where you can. Create a feedback loop on your policies to iterate and adjust. It's okay to not get your policies perfect the first time, you can adjust them.
Really figure out what you are trying to prove out in your initial proof of value/concept. If it's full data mesh capabilities, that can easily take 4-6 months.
An interesting incremental insight: Zhamak has warned about organizations trying to scale too fast as an anti-pattern that may result in lots of tech debt or a failure of your implementation.
An interesting incremental insight: in all of the data mesh implementations S&G have worked on thus far, the initial data product has not had any PII as that adds significant complications probably beyond what the value add of including PII would be in most cases.
Your data mesh implementation team should be 1-2 people from every necessary capability.
Data mesh is a large commitment - resources, time, focus, etc. - so you need to be prepared to fund it for the long-haul. This isn't an initial big-bang approach. But this is also why you should focus on continuous incremental value delivery once you get to delivering data products to keep up momentum.
You will get things wrong as you move forward with your data mesh implementation. Look to limit the blast radius but it's absolutely fine and expected that you will learn and improve. Data mesh gives people flexibility and flexibility allows for making changes. Set up fast feedback loops and look to iterate rather than trying to get it perfect the first time. Perfect is the enemy of done.

S&G started off giving the four general states of data governance in most organizations: none, centralized, decentralized, and federated. Many organizations, even quite large ones, have little to no major data governance oversight. As previous guests have mentioned, many get fed up with data governance only being a cost center - especially if it doesn't even offer much risk mitigation or regulator compliance - and essentially do away with their data governance. Decentralized data governance is an anti-pattern in general with each domain or line of business coming up with their own approaches, making cross domain boundary collaboration difficult at best - it's like each domain is speaking a different language entirely. Many companies move to a centralized approach but that often quickly ends up becoming a blocker without pretty specific controls in place. Rigid plus low throughput isn't great. Hence, why data mesh pushes for federated governance - governance with a central group to make necessary decisions and policies but where the people who understand best actually apply the policies to their work - namely the data products.

So, per S&G, the federated governance structure in data mesh in general should be a centralized board or team representing many different constituencies throughout the organization necessary to make smart and informed decisions about policies. Then the policies are codified - and coded - into the platform for domains to easily apply the policies to their data products. The centralized team should focus on making quick decisions by delegating developing policy researching and development to people within each of their own constituent groups - e.g. software engineering, platform, product, security, legal/compliance, etc. That way, the leaders on the centralized board don't need to have all the context themselves to make smart decisions as the people they delegated to can ensure their constituent group's needs are met. And the application of policies to data products at the domain level is made easy - or at least far easier - through automation. This setup gives the domains more freedom in how they apply the policies to the data products.

On speaking about greenfield versus brownfield for data governance, of course greenfield - meaning little to no data governance in place - is typically far easier according to S&G. It can be disconcerting to see large organizations with very large data practices and little governance but it's at least easier to only have to focus on creating and training instead of evolving and unlearning too. Either way, to move forward, look to build out the CYA - cover your...butt... - aspects of data governance first and work to build a minimum viable data governance board. Then you can start to ask about needs and create a backlog to start working through. But again, make sure to focus the board on making decisions and impact, not as a political entity. Easier said than done but showing them how to make decisions quickly and efficiently is great. And, with data mesh, policies can be changed or enhanced later - you don't have to get it perfect at the start.

If you are in a brownfield deploy of your governance board, it can be a political minefield, per S&G, as there may be overrepresentation of certain teams. But you need to work to have the right representation of needs, the right diversity of capability. There needs to be a top-down mandate to really reshape the way your board is composed so you can get to that fast decision making capability. As a reminder, this is somewhat counter to what Laura Madsen recommended in her episode but aims for the same outcome. Possibly look to disrupt your governance if it ever becomes too slow and the bottleneck.

So, you've got your governance board together - how do you get going for something like data mesh? According to S&G, you should first focus on policies that positively impact the technical people, e.g. that all output ports on your data products should be registered in the data catalog. And it's okay to not get your policies 100% correct upfront, you can adjust. Use a feedback loop to take in information about missing policies or currently deployed policies - are they meeting people's needs? Every policy should have incremental value. Security is obviously a policy area that could be considered as cost-only but it's still quite important to address and risk mitigation is a value-add.

For S&G as previously mentioned, in order to keep things moving, delegation is crucial. If there is a truly important decision with major implications, possibly the leaders of the different capabilities represented on the board need to get more deeply involved. But for most policies, those heads should delegate as much as possible to people they trust to represent their interests and move forward. We don't need the end approvers to be overly involved in routine decisions. Much like when purchasing a solution, the CFO typically doesn't need to be involved in specifically deciding which data catalog to use if it is a small portion of the budget - the experts did the work and made a selection, you delegated to them for a reason presumably.

S&G gave some advice around getting started in your data mesh journey. When they were working with Glovo, company management gave the team the time and budget to really build out the platform in tandem with the governance and the first data product. That took 6 months. And coordinating across all four pillars moving forward simultaneously was certainly not easy. If you don't have that amount of time and budget, you can do a relatively smaller proof of value/concept in probably 3-4 months; but Zhamak has warned of premature scaling causing a fair bit of issues for a number of companies trying to implement data mesh so trying to rush your proof of value/concept might not be the best idea. They also mentioned a pattern of the first data product at their clients not having PII as that complicates your initial platform needs for governance. And to pick a relatively simple source-aligned data product use case as your first data product.

As to who you should have on your data mesh "tiger team" if you are lucky enough to have some full-time heads to staff it, S&G recommend having 1-2 from each necessary capability so 1-2 data engineers to help build out the platform and upskill your domain team, 1-2 folks on the governance team, 1-2 from product or elsewhere to do the data product management, etc. and then obviously the domain you are working with needs to be heavily involved. And whoever is on the team, prepare to do a lot of data product/data mesh evangelism.

It's important to understand that committing to data mesh is a big long-term commitment, financial and otherwise, per S&G. Your implementation can't be a skunkworks approach, you have to be committed to moving forward together so you can drive the necessary buy-in. And it isn't just the initial implementation, you have ongoing growth of your implementation and maintenance. This is partially why so many guests have mentioned delivering continuous incremental value to make it easy to secure additional necessary funding.

When asked about what parts of your federated computational governance should be in the platform versus at the data product level, S&G believe you should always look to create the affordances and the easy path in the platform. The application of policy via the platform is the best way to ensure compliance and also standardization, which makes it easier on data consumers. But any decision relative to the specific context or needs of the explicit product should be made at the product level. So, the decisions about how to measure data quality characteristics would be at the platform level but the SLAs to meet for a data product would be set at the data product level itself by the domain team.

In wrapping up, S&G wanted to reiterate that data mesh isn't easy if you want to set yourself up for long-term success. It is going to take a lot of effort to get it going and deliver your initial data product and platform and governance policies. But by spending time to do it right, you set yourself up for gaining a lot of momentum. Don't get discouraged. And be prepared to get things wrong and then fix them, that's totally okay. Play, learn, iterate, improve.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 14, 2022 Aug 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#112 Driving Buy-In and Finding Early Success - Kiwi.com's Data Mesh Journey - Interview w/ Martina Ivaničová Aug 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Martina's LinkedIn: https://www.linkedin.com/in/martina-ivanicova/

In this episode, Scott interviewed Martina Ivaničová, Data Intelligence Engineering Manager at the travel services company Kiwi.com.

Some key takeaways/thoughts from Martina's point of view:

The most important - and possibly one of the most difficult - aspect of a data mesh implementation is "triggering organizational change".
Driving buy-in for something like data mesh is obviously not easy. As you are getting started, look to leverage 1:1 conversations to really share what you are trying to do and why and how this can impact them and the organization. These 1:1 conversations are crucial to developing early momentum.
On driving buy-in for data mesh, really think about how to limit incremental cognitive load as much as possible on developers/software engineers. If you can keep cognitive load low, you are much more likely to succeed - succeed in driving buy-in and succeed in delivering value.
When sharing internally about data mesh, it's important to focus on what it means to the other person. Using "data mesh" as a phrase can lead to a lot of confusion for people not on the data team. Make it clear what you are trying to accomplish - the what, the why, and the how. Using data-as-a-product as the leading concept resonated and worked well.
Kiwi.com started driving buy-in by working with the engineering upper management, then found a few valuable and achievable first use cases to move forward. And they have kept cognitive low on the engineering teams while they learn how to deliver data as a product.
If possible, the easiest way to drive buy-in is by finding a use case that is beneficial to the producing domain. If not, then look to spend the 1:1 time to really share why this matters.
Kiwi.com is getting software engineers in domains to commit to simply sharing their data, not even really structuring into data products. So the software engineers in most cases are really only focused on maintaining high-quality data sharing mechanisms - read: pipelines. That is a relatively low initial cognitive load/low workload ask.
Analytics engineers are creating the data products from the sourced data to satisfy consumer needs. Martina and team want to move to software engineers handling more of data product creation/management over time but it's a process. They plan for analytics engineers to upskill the software engineers by pairing with them closely.
It might initially be more important to find a way to evaluate and iterate on what data is shared and how than getting to the most complex or valuable data product. You want to build the muscle around sharing data first before trying to go too big too soon.
It's important to know what you are trying to prove out in your initial data mesh related deployment. It's okay to prove out you can produce data products before proving out you can build out the full mesh.
A key success metric for a data mesh journey could be how many direct conversations and then actions come from data producers and consumers speaking without data engineering involvement. At Kiwi.com, these conversations are still usually driven by analytics engineers but that might change in the future.
Data governance centralization didn't happen overnight. When you look to decentralize and federate your governance, you should look to be patient instead of trying for an overnight revolution.

Martina started by discussing how historical - legacy might be too harsh - data approaches like the enterprise data warehouse haven't kept up with the mass proliferation of data sources. When we were taking data from the monolith or monoliths, it was far easier to think about what data you might have and try to arrange it into something consumable. But now, with data coming from so many microservices and from external vendors and partners, it just isn't possible to use the same historical approach - too many things are changing. The centralized data team trying to own hundreds of pipelines flowing into one central lake or warehouse that they also own - it just wasn't scaling. So when the Kiwi.com data team ran across data mesh, it was very exciting - it was a way for the people with the business context to conceivably own and manage sharing their data in a reliable way.

The historical general approach to data governance - one centralized team trying to make context-dependent decisions for all the domains - just never made sense to Martina. They just could never know the context well enough to make good choices, especially good choices in a timely manner. She noted that if you are moving from that approach, centralization didn't happen in a day, it evolved. Your move to decentralization should also evolve - think thin slicing and decentralizing more and more rather than pushing all ownership to all domains at once.

Martina then talked about driving buy-in, a topic Scott circled back on frequently throughout the conversation. She noted - as many have, notably Khanh Chau in his episode - how hard driving buy-in can be when people haven't felt the specific pain you are speaking to. So she and her team worked to really have deep conversations with the software engineers about how important treating and sharing data as a product can be and how the data team will work to maintain low cognitive load on the software engineers.

So, how did they start driving buy-in? First, Martina and team worked with engineering upper management to make sure that as they moved forward with domain teams, they would have support. Then, they focused on finding good first use cases. What could be a use case that would drive significant value if they got it right where they could also limit incremental cognitive load on the software engineers? And what had a high likelihood of success to start to build out proof points and momentum?

Martina mentioned how truly crucial the low initial cognitive load aspect was to driving their data mesh journey forward. The central data team wanted to spend at most 2-3 days with software engineers to teach them how to share data. Is that going to be them creating actual data products on their own? Quite frankly, no. It was about teaching them how to share data and probably more importantly in the long-run, how to think about sharing data - that data product thinking. Then, analytics engineers structure the data shared into actual data products. This setup means it is easy to evaluate and iterate along the way.

Scott asked what were they trying to prove out initially - a dataset had value, they could build data products, or that they could build a data mesh? For Martina and team, it was more about building out a reliable way to share data, so their proof of value was focused on proving they could build data products. One really crucial aspect they wanted to test was could they bring the data producers and consumers together with a good outcome without the data engineers - so the producers, the analytics engineers, and the end consumers working together. And the answer is yes, they are seeing great results there! The direct relationships between data producers and consumers is spurring the data producers to rethink how they share and what they share - and very importantly what more data could they share.

Martina, like many other guests, brought up the general industry need for redefining data contracts - they just don't do a ton of things we need. We don't have a good way to detect semantic drift or often even to prevent changes before they break something. It's even difficult with lots of existing tooling for data producers to see who is consuming their data - and almost non-existent on data producers knowing specifically how the consumers are using their data. And so many more issues that should be wrapped into contracts.

In circling back on buy-in, Martina talked about how in a brownfield deployment, there are puts and takes. Negatives include dealing with the issues of existing tech debt, it's difficult to get prioritization, etc. But a positive is that you already have an existing backlog of requests where you can find some interesting use cases to try out for your data mesh proof of value/concept.

As part of driving buy-in for the proof of value/concept, Martina and team had to do a lot of 1:1 conversations. It can be frustrating to have to do so but these conversations are crucial to building initial momentum. Martina had some issues when she tried to explain they were doing data mesh to non data people internally; it's so easy to get confused by data mesh, especially for those same people. So she created a 1-pager focused on the data-as-a-product concept to help people understand what they were trying to accomplish. Focus on informing people the what, the why, and the how. Data mesh is more of an implementation detail to them. This is what Scott keeps referring to as "unicorn farts" - in every bit of internal documentation for consumption outside the data team, copy+find+replace "data mesh" with "unicorn farts". Because then you will delete every mention of data mesh - and unicorn farts - so you can focus on what actually matters to the other party.

Martina shared about the current role structure of their data mesh journey: data engineers focusing on the data platform, analytics engineers building data models on top of source aligned data to create consumer aligned data products, software engineers focusing on sharing source-aligned data, and data consumers producing aggregated data models across different data products. It is difficult to say they are building full source-aligned data products as of yet as they train their software engineers to really work with data and use data-as-a-product thinking. Remember, they are training them on sharing data for 2-3 days total - you can't bring someone that far along in learning how to handle data in half a week! Their goal is to embed the analytics engineers further into the domains to really upskill the software engineers more around data but it's early days.

It might feel a bit obvious but it's good to say out loud, per Martina: triggering organizational change is the most important part to getting your data mesh journey moving. It will be difficult to get moving but trying to build out your platform early or trying to get teams to create data products without the organizational support is very likely to fail. You need to get that organizational change going.

A few other interesting points to highlight:

A key initial success criteria was seeing software engineers start to consider what additional data they could share that could be useful and how they could share it reliably.

Buy-in for building a data product is obviously easiest when the domain will be the consumer but when that isn't the case, the 1:1 conversations and having upper engineering management buy-in made it possible to get domains to do the work.

Figuring out who owns data brought - or bought - in from the outside can be extremely difficult. Who wants to own often low quality data from an external source that needs to be cleaned and made into a product?

Kiwi.com leveraged their existing stack for their initial data mesh implementation. There are of course lots of missing capabilities but they can still deliver good incremental value without every piece of the platform in place.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#111 Applying Data Mesh Principles to Your Real-Time/Operational Systems - Mesh Musings 25 Aug 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#110 Disrupting - Not Destroying - Your Data Governance to Drive Incremental Value - Interview w/ Laura Madsen Aug 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Laura's book, Disrupting Data Governance: https://smile.amazon.com/Disrupting-Data-Governance-Call-Action/dp/1634626532

Laura's LinkedIn: https://www.linkedin.com/in/lauramadsen/

Moxy Analytics website: https://www.moxyanalytics.com/

In this episode, Scott interviewed Laura Madsen, CEO at Moxy Analytics and author of the book Disrupting Data Governance.

For the purposes of this write-up, when discussing data governance, it refers to the way many large organizations handle data governance at scale - a way that is very rigid and causes bottlenecks. We all know we can't stereotype or group every org together but general trends can be observed.

Some key takeaways/thoughts from Laura's point of view:

A big issue with today's data governance is that the concept of data stewards - the people who own the data concepts - is from 30 years ago and hasn't changed much despite the demands and scope changing dramatically.
The data governance committee/council structure most organizations use is inherently inflexible and ineffectual. Those making the decisions don't really understand what's happening with the data under the covers and those who do understand have little ability to influence the wider committee outside their own domain. And thus, they become a major bottleneck.
Data governance committees can be quite useful if they focus on communication and context exchange rather than driving decisions and work forward.
To drive change in your data governance practices, you need to disrupt but not destroy. Start to break down the big picture into much smaller, bite-sized chunks that when you improve on them will incrementally drive value - Agile provides a good framework to approach this.
You will absolutely have to throw out a LOT of your current data governance practices - over time - as you replace them with better ways of working. You will need to really evaluate each practice and assess if it will drive value or should be replaced.
"Marie Kondo" your data governance practices - really look at your processes one by one and ask "does this spark value?" Reference: https://storables.com/storage-ideas/marie-kondo-method/
Current data governance practices does no provide incremental value to most organizations, they are about compliance and risk mitigation. If you can drive value creation, you can more easily drive change. People want to enable value creation or at least are hesitant to stop it in most orgs. Look for small ways to drive incremental value to build momentum.
The current data steward and data ownership model essentially rewards innaction more than action. Action has risk and risk mitigation is a large part of the data steward and data owner's role. We need to change that relationship and reward enabling valuable use of data but within compliance.
Laura is a fan of the hub-and-spoke model for data governance - and in general. To make hub-and-spoke work, 1) everyone has to really work on strong communication and 2) the central governance team cannot fall into the trap of trying to fix the data themselves, they must empower and enable the teams to fix their own data.
Data governance teams must stop writing policies -> compliance and InfoSec should be doing that. Policies become something that can be audited - don't give regulators the path to fining you.
It's crucial to understand that there is a "good enough" in data governance and it's often good enough only for right now. Find that line of good enough for now, look to reevaluate, and find places that aren't good enough or are just barely good enough to focus on. There will assuredly be lots.
It will probably be very uncomfortable disrupting the way you do data governance - at first. Start to build that muscle memory with small, incrementally valuable changes.
There must be balance between flexibility and rigidity in data governance. Too much flexibility causes chaos. Too much rigidity causes the pain you are probably feeling right now.
Laura wants data governance professionals to know that she understands how difficult your role is and the work you do is very valuable. She sees you :D
To start to change your data governance ways from woes to "woah, this is working", first start by rethinking who is accountable.

Laura started the conversation with her big question when it comes to data governance: How did we end up here? How did most organizations end up with a very rigid, not scalable, non value-add data governance approach? How are we doing data governance essentially the same way as 30+ years ago? How can that make sense given all the changes and advancements in tech/software in the last 3 decades? For Laura, it really doesn't make much sense and we should disrupt that model.

With the data steward model as it currently "works", the steward is someone in the business that has subject matter expertise but often has a hard time driving incremental change because of politics. So we need to work on flipping the incentives and role goals to drive incremental value from governance instead of becoming a costly bottleneck with data stewards and owners preventing use instead of encouraging it.

For Laura, the biggest issue with data governance right now is the governance councils and committees. They typically have worthwhile goals, ones every organization should strive for. But committee structure almost inherently means they will be ineffectual in driving high-value work. The data owners have no real line of sight to what's going on with the data and the data stewards - who do have that line of sight - can't move forward without approval. And after a few meetings of nothing really getting done, the required decision makers often stop attending. So the committee holds useless meetings instead of actually pushing work forward.

Committees and councils can have a benefit in Laura's view if they are focused on communication instead of direct action. They can be great ways to share context internally, especially among key stakeholders.

Laura is a big fan of the hub-and-spoke model of organization to drive things forward with governance. The key to leveraging a model like hub-and-spoke is strong communication touch points between the centralized hub - the data governance team - and the spokes - the domains. A common failure point with data governance teams, especially in the hub-and-spoke model, is that the governance team tries to fix the data instead of enabling the teams to handle their own data. As Jay Sen reiterated: empower people, don't try to do their jobs. Hub-and-spoke can probably work relative to data governance in data mesh but you have to be careful about what is centralized in the hub.

Look at the roles you have that support data governance. Reconsider who does what with a simple “RACI” model of Responsible, Accountable, Consulted, and Informed. Flip the script by removing accountability for activities when the role has no ability to impact the work - i.e., making a data steward responsible for an entire data domain when they have almost zero ability to change workflows, impact data quality, or align expectations, etc.

Laura recommends for people to stop "putting their heads down", working so they can drive something. Instead, really think about why you want to change what you're doing - what is your reason for wanting to disrupt your data governance? If it's just to shake things up, that's probably not going to go well. You will need some major force of will and perseverance to really make the change. So go in it for the right reasons.

So, how do you actually start to change your data governance practices and overall approach? Per Laura, start by evaluating the ways you currently do data governance and start to look for ways to break your approaches down into smaller pieces so you can do small-scale disruption and deliver incremental value. Delivering in small increments will make it quicker and easier to deliver value while lowering you chance of failure. As you start to show you are adding value, you will gain momentum as most organizations do not really drive value from their data governance. People are typically pretty happy to enable value creation.

Per Laura, you should really rethink the way you do all aspects of data governance. Nothing is sacred. Spend the time to really consider all aspects of your data governance and think if you should change it. And Laura even recommends to look to commit to tossing aside practices before you evaluate if they work. That way, you are having to pick things from the garbage pile rather than sticking with the status quo. It's a bit of a psychology but could be useful.

Scott asked what typically causes organizations to really rethink their data governance. At least for Laura, she typically gets a call when the data governance leader leaves and the CDO or CIO need some help cleaning up the issues. And a data governance leader is typically a short-term role, per Laura. That leader typically drove so much through their own knowledge of "where the data bodies are buried" rather than through really scalable process so it can cause a major disruption when they leave.

When the opportunity and drive to change your organization's data governance does arrive, Laura recommends rethinking data governance at the highest level. What are you really trying to accomplish? How do you get to "good enough"? How do you get comfortable with "good enough"? And it's crucial to understand that good enough for now may not be good enough for the future and build in a plan to reevaluate processes. But that bridge solution is still viable and valuable. It can be quite challenging to change the way people have approached data governance for the last 30 years. Break it up into small changes and get moving, build the muscle memory of change.

For Laura, data governance is often a proxy or a reflection of your broader data culture. It's important to seek balance in your data governance approach between flexibility and rigidity, much like your data culture. Too much flexibility will create too much chaos to move things forward - there is too little communication and/or coordination. Too much rigidity is essentially the world we are in today for most organizations and we can probably agree that's not great.

Laura wrapped up on a few points. 1) Data governance is crucial to driving trust in data. Work with people to really communicate what is happening in your governance approach to increase that trust. 2) It's very easy to try to tackle everything in data governance but focus on what matters. It's okay to have some sharp edges, what will drive more value? And 3) Data governance work is incredibly hard. Laura wants you to know she gets it and you data governance folks are seen. You do incredibly difficult and valuable work.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes – Week of August 7, 2022 Aug 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#109 Tying Data Strategy and Architecture to Business Strategy - Interview w/ Anitha Jagadeesh Aug 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Anitha's LinkedIn: https://www.linkedin.com/in/anithajagadeesh/

In this episode, Scott interviewed Anitha Jagadeesh, Principal Enterprise Architect at ServiceNow. To be clear, she was only representing her own views on the episode.

Some key takeaways/thoughts from Anitha's point of view, some of which she specifically wrote:

It is absolutely crucial to tie the data strategy to the business strategy. The business strategy must drive the data strategy which drives your data architecture.
Architects need to lead the way in digging into use cases to get the specifics on what data producers are trying to solve for data consumers. Then, those architects can find the common patterns across use cases to tie to your organizational data strategy and also tie to your data architecture guidelines and principles. That way, instead of addressing challenges via point solutions, you can drive organization-wide choices that support many use cases via your data architecture.
Architects also need to ask the probing questions to continuously tie work back to the business strategy and value or expected outcome for customers. If you aren't driving the business strategy forward, if you aren't helping the big picture, is the work worth doing?
When it comes to data, companies shouldn't be entirely offensive - trying to leverage data for as much value as possible - or defensive - trying to minimize risk as much as possible. So organizations that have been very conservative need to push to be creative/offensive and high-risk organizations will get themselves into trouble if they don't start going defensive too.
As we build the data strategy we have to catalog our data assets/products and contracts to access these data assets/products – internal, external, and third party. Next steps we have to enable active metadata to ensure the catalog is always current.
Data contracts - especially SLAs and SLOs - are really crucial to driving reliable and scalable data practices forward. How can people trust what they are consuming without having to check it themselves unless there are very specific parameters and documentation of what they're getting? The data space needs to rework the way we approach data contracts.
We need to be careful to not head down the same paths/ways of working - just with different names - that we've tried and didn't work. But we also need to focus on what we've learned from different approaches instead of reinventing the wheel where appropriate. Hopefully data mesh can thread that needle.
When thinking about how you should split into domains, look at the business strategy. How does your organization tackle business challenges? That should inform how you create domain boundaries.
One of the biggest challenges in data at the moment is centralization versus decentralization and/or federated. How far to go towards one or the other side across many many decisions is really crucial to your data strategy. Look for places to centralize support of multiple use cases but not take the decisions out of the hands of people who know best where possible.
API-first is an important strategy for modernization of use cases. But it can easily lead to massive inefficiencies on the analytical side with large-scale queries. So we need to think about how we can do APIs in an analytical world and consider patterns and guidelines to support bulk data consumption with volume, performance, and limits. .
It will be difficult but worthwhile for organizations to migrate existing data assets to decoupled data products. There are many ways to approach that challenge, such as the Strangler Fig Pattern, but you need to take lots of care to do it right rather than disrupt the ongoing business.
Trying to serve real-time use cases - measured in millisecond latency - and certain types of other analytical queries from the same data product is likely to cause big issues. If you have a very large data pull from a service, that can greatly impact performance. Let's not go back to the days of trying to run large-scale queries against production and causing outages or look for other architectural patterns to enable performance like replicating data to other data products. Ideally large consumption analytical use cases should be managed with some limits for real time use cases or run them on analytical data products , which will have different infrastructure that is fit for analytical use cases.
It is crucial to have your governance team switch from defensive-only and a bottleneck to an enabling team - allowing domains to make smart decisions and providing the center of excellence and standards to let the domains focus on making the value-add and domain context specific decisions where possible.
It's crucial for both sides in a potential data initiative/project to share as much context as possible about what are the potential outcomes weighed against the potential costs. How can both sides collaborate to maximize the return on investment? Just seeking the highest return possible is what has doomed many data initiatives - let's move past that way of working.
We are heading towards Hybrid cloud, multi-vendor, multi-region, real-time needs of data. That will require us to rethink architecture that can scale and support agility.

Anitha has seen a lot of data and engineering practices and patterns over her long career. In some ways right now, she is seeing many people heading down close to the same paths - just with new labels - that haven't worked. Data mesh tries to address a number of these historical challenges but we should make sure to deeply understand what history has taught us so we don't need to reinvent everything or make the same mistakes. History may not specifically repeat but it's easy for it to rhyme.

For Anitha, a lot of the approaches people are trying in data miss the mark by not focusing on the big picture first - what is your business strategy? Your business strategy should drive your data strategy, not even just inform it, and then your data strategy should drive your data architecture. Far too many people start at the data strategy or even data architecture level.

Anitha - like many of us - is seeing major changes in the industry with most organizations transitioning to cloud, product-centric, and/or API-first approaches. Industries and organizations that have traditionally focused on defensive data strategies - those that protect the data to minimize risk such as compliance - will need to get offensive to compete and drive value. Luca Paganelli's episode covered how HERA is transitioning from defensive to a balanced approach. But on the flip side, the companies that have focused much more on offensive data strategies - trying to derive as much value as possible from data with little controls in place - really need to step up their defensive game. Companies with a balanced approach to offensive and defensive strategy are generally the most likely to win.

Data contracts is one of the biggest unresolved or not well solved issues in data for Anitha. The way most organizations are still doing data contracts - which is often not at all... - just isn't working. SLAs (service level agreements) and SLOs (service level objectives) are crucial to driving data trust when it comes to contracts. Emily Gorcenski's episode covered data SLAs and SLOs in-depth. There are some approaches emerging but as there have been many episodes of this podcast covering data contracts, it's still a quite immature data practice that needs further work.

When asked about how to drive good, broadly applicable choices rather than just solving for the specific use case, Anitha talked about again circling back to the business strategy and the business use cases. You need to not be simply reactive to requests but look at how those requests play into the bigger picture. Architects should play a role in digging deep into use cases and requests and then finding the common patterns that support your architecture runway for teams to develop products. They need to dig in with many people across the organization and find what you really need to solve for in general across your many use cases. That way, you can address a broader scope of challenge more easily rather than building to each use case. Much easier said than done or course.

Anitha made an interesting point about how your business strategy and data strategy should drive your domains. Yes, every organization or industry has a different domain map but even when thinking about what you are trying to accomplish and how your organization tackles challenges, you should look to use that as your general approach to mapping out your domains. Is that business need/capability? Is that application-first? Is this a domain specific compliance need? Etc. Piethein Strengholt talked about multiple different ways to map domains in his episode.

When asked about how organizations can think about centralization versus decentralization in data, especially regarding governance, Anitha admitted it's very hard to create rigid rules that are actually good despite how easy that might make things. You need to ask how you can centralize the standards and the tooling so you can support multiple use cases but not have centralized decisioning when the domains know best. Having a hybrid governance - grassroots and centralized would drive better data management practices. Per Scott, it's definitely not a black-and-white decision and this will be one of the hardest challenges for many organizations in the next few years.

A big trend for Anitha in digital modernization is more and more vendors heading towards API-first. But trying to use APIs like we have in the past will not be efficient at all in many cases in the analytics space. A very large query via API could be extremely inefficient. We need to think about how we can serve analytical needs better in an API-first world. What actually is an analytical API? How can we grab 100,000 records in a single query that isn't a 1 by 1 pagination? Still remains to be seen.

While many organizations would love to have a greenfield to deploy their data strategy and data initiatives, it's just not a reality for most according to Anitha. There are existing data assets in place. Moving them to being data products is essential but business also must go on with their current day-to-day. It's a difficult challenge to migrate people over to new data products. And data monoliths have very unclear sets of data products all intertwined. As an example, in the past, Anitha and her team were the victims of their own success as they built a very successful data warehouse that more and more teams moved to use. As the number of use cases and load increased, the performance decreased. We need to move to decoupled and more scalable ways of working in data to prevent success from being the path to failure and/or pain.

Anitha and Scott discussed how important it is to build specific solutions to be fit-for-purpose, especially around SLAs and SLOs. On the data side, if something really needs to serve something "in real time", meaning measured in milliseconds, you wouldn't also want to allow heavy analytical queries that could slow down what it is serving. Which circles back to why API-first is currently challenging. But you should also dig into what people mean when they say "in real time" because it is often "not on a 24 hour delay but 2 hours is fine". Get specific, dig into details and the why.

Anitha has some specific recommendations regarding data governance as she views it as crucial to really getting data products rights. She recommends creating a general center of excellence and central tooling support but with grassroots decisioning when it makes sense. So create the standards centrally and look to empower teams but also have that center of excellence to serve as a backdrop to be the experts on how to meet general governance needs like comply with GDPR, CCPA, etc. You want your domains focusing on the value-add decisions, have the central governance team look to be an enabler.

A key responsibility for all architects in Anitha's view, is digging into how and where are you planning to use this data? What is the target use and more importantly, what is the target outcome? When digging in, you can really assess not just what might be the return, but what is the cost - both upfront and ongoing. It's easy to think about how great it could be to have this massive set of data but there needs to be a balanced return on investment. And time-to-market is also crucial. So we need to encourage collaborative negotiation around data requests instead of simply handing over requirements.

In wrapping up, Anitha circled back on the concept of architects - and other roles too - really asking probing questions on specific use cases while keeping the big picture in mind at all times. Ask if this work supports the big picture, the business strategy. If it doesn't or if there isn't a clear tie to the business strategy, is the work worth doing?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#108 The Slippery Slope of "Real-Time" and Data Mesh - Mesh Musings 24 Aug 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#107 Focusing on Outcomes and Building Brave Teams in Data - Interview w/ Gretchen Moran Jul 31, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Gretchen's LinkedIn: https://www.linkedin.com/in/gretchenmoran/

NGS' current openings: https://ngs.wd1.myworkdayjobs.com/ngs_external_career_site

In this episode, Scott interviewed Gretchen Moran, the Senior Director, Data Products at the National Geographic Society (NGS; the non-profit arm of National Geographic).

Some key takeaways/thoughts from Gretchen's point of view:

NGS is a bit unique in that they don't have a widely deployed data architecture so they do not have a lot of habits to unlearn. Starting with a greenfield means likely more training and learning/experimenting will be required but at least no institutional unlearning.
To move forward with data mesh, organizations must be able to embrace change - and the pain that it will inevitably bring - and embrace ambiguity. You need to move forward and figure it out together but also be okay with failure as a learning experience as you test what works for your organization.
To win the hearts and minds of data producers, show them what high-quality data can mean for the organization and their domain/role. Work closely with them, understand their context, hold their hand to bring them along and align them to the vision of data mesh.
It's easier to drive buy-in widely if you find the organizational influencers and win them over. It is the domino effect in practice. Partner closely with the influencers early on to drive your initiative forward.
For NGS, they are working with a single initial data producing team for their proof of value. The data mesh world seems to be split a bit between working with one or two to three teams in the initial proof of value stage.
"Any technology effort is still a people effort."
We have yet to learn how to leverage the knowledge and context of people without data knowledge in general in the data and analytics space. This is what data mesh tries to unlock but we are still figuring out how to do it well.
It's very easy to intimidate people with data. We need to make tech and especially data much less intimidating to push broader adoption. The business context of those who aren't yet data literate can be extremely valuable. We need to lower the actual bar to leveraging data but also lower the perceived bar to leveraging data.
"Metrics + outcomes = value" - without outcomes attached, metrics have no value.
Automation is going to be key to many aspects of data mesh. Upskilling people to leverage data will only really pay off if it doesn't mean a large increase in the amount of work to leverage data.
User experience is crucial to getting the most value out of your data. Think about your data user experience (DUX) and bring in designers to help optimize the experience and really focus on data as a product thinking.
NGS is still trying to find who should own generating and sharing insights on data combined from multiple domains. Is that a centralized insights team? Does that push us too far back towards centralization? It's still early days but those insights are crucial to driving value from data.
We will see where new insights come from in data mesh. Will it be more insights from data consumers as they can spend time on data analysis instead of data cleaning? Or will it be the data producers as they learn how to really leverage their own data? Both?
Many are waiting for vendors to validate data mesh but they really haven't done that very well yet. It remains to be seen if they even could validate something so complex and large in scope.
Building brave teams - teams that aren't afraid of new challenges or of failure or especially of ambiguity - will be crucial to getting data mesh right. Teams might be afraid of doing things "wrong" but as long as they work to get to a good state and incorporate feedback and learnings, that's what will drive much more value in the long-run.
There is really shared ownership of data even inside the same domain. The subject matter experts and the people shaping the data products must build a strong relationship with good communication.
Data mesh aims to - and needs to - solve for the cost of change in data. Traditional data warehouses have had an extremely high cost of change. The huge cascading pipeline setups most have with data lake too. Data mesh needs to make evolution in data much easier, quicker, safer, less costly, etc. It's still early days there.
There is a delicate balance between over architecting and underinvesting in your platform. Look to build for reuse and don't lock yourself into decisions where possible. Far easier said than done.
Many teams are worrying if they are doing data sharing wrong. But can they actually really do it wrong? Yes, probably, but if they are open to feedback and paying attention, they don't have to get it "right" the first time. You can evolve to get to a very good place but prior data setups have been so rigid, that evolution has been tough and costly.
You probably don't need to build out as large of a team as you might think to start on your data mesh journey - depending on your timeline. It will take years to get to delivering fully on your vision but you can add a lot of value as you progress and learn - you don't have to get it perfect at the start! And consider what skillsets are really crucial.

Gretchen started off by giving her background and some of the ways her history has played into her perspective and current role. A big factor in her interest in data mesh was helping a number of large organizations evolve their data platforms and how that helped those organizations deliver better results - but they still often struggled somewhat to derive full value from their data and data mesh can hopefully unlock that value.

For the National Geographic Society relative to data mesh, preparation has been Gretchen and team's keyword. Rather than trying to move forward with their data mesh implementation as fast as possible, they've spent the last year testing and preparing for implementing their data strategy. And they are in a bit of a unique situation because even though the organization is over 100 years old, really their tech stack is about seven years old and they don't have a cohesive data architecture deployed. This means they also don't have a lot to unlearn but have a ton to learn and experiment on.

NGS is already organized in a product-centric approach and the technologists really understand their domains. Now, Gretchen and team just need to get them bought in that they should treat their data like they do their applications - like a product - as they move forward with their data mesh implementation. Easier said than done but the organization in general hasn't been pushing back on these ideas, which has meant good initial collaboration.

While embracing sharing data is crucial to NGS' overall organization-wide strategy, it's not all sunshine and rainbows as Gretchen knows there will be a lot of heavy lifting. Heavy lifting around change, heavy lifting in going against the status quo. To be successful with a data mesh implementation, the organization is going to have to embrace change and ambiguity. And both are typically painful.

Gretchen and team knew to lay the groundwork for something like data mesh, their organization needed a base layer of data literacy. Without an understanding of data, would they even have people to consume data, much less people capable and willing to produce their data like a product? So they started by bringing in consultants to help people start learning and to build a general business glossary.

But to really reach the "hearts and minds" of the general organization, Gretchen knew they needed to show people what value data can bring them. What are their goals and needs and how can data support that? How is something like having high-quality data available valuable to data consumers but more tricky, how is it also valuable to data producers? And part of their data literacy/upskilling process was showing people what using data could mean for them, not just a training course in SQL or Tableau. Just training people how to use a piece of technology in a vacuum hasn't worked well. A success vector for Gretchen has been finding the organizational "influencers" that provide the leverage to drive buy-in across the org.

So, how are Gretchen and team getting going after their preparation period? They are partnering very closely with an initial pilot team and are going to prove out the value to share more broadly across the organization. This has been an interesting question in data mesh - how many domains and/or data products should be involved in your proof of value?

Broadly speaking, Gretchen believes - and Scott agreed - any technology effort is still very much a people effort. It is very hard to do something like make data self-evident so the people need to steer and steward any technology effort to do something like that forward. Then add in the fact that data mesh is much more organizational/process focused and the people side becomes even more crucial.

Gretchen talked about metrics in general and her theory of the "bad metrics sin" - that it is worse to have bad metrics than no metrics at all. And to identify early and then stay away from vanity metrics. She strongly believes that metrics + outcomes = value so without the outcomes, metrics don't have value. As Sadie Martin and Katie Bauer mentioned in their episodes, measure what matters and measure what you will act on. And measuring impact in the NGO (non-governmental organization; a term for many non-profits) space is particularly difficult - Gretchen used the word persnickety - so really finding your useful metrics and backing them up can be a challenge but is crucial.

One behavioral change Gretchen is pushing heavily as people are learning more and more about data - asking about people's rationales when making a choice working with data. The outcome is more context for all involved because people use their context to make choices so learning why they made choices can highlight very interesting points. Why did they go with X versus Y? It's crucial to do this to enhance curiosity and learning rather than asking people to prove their reasoning/understanding. So ask with the tone and goal of "tell me more".

And it's easy and quite common to intimidate people with data, per Gretchen. We need to lower the actual bar to leveraging data but even moreso, we need to lower the perceived bar of how challenging it is to leverage data. Part of doing that is meeting people where they are, showing them how they can leverage their current knowledge and skills while upskilling them to be even more effective.

Gretchen is seeing people in NGS going through so many hoops to produce reports and data in very manual ways; so enabling them to do produce and consume data automatically and more reliably is something she's excited to take on. That way, NGS can leverage their knowledge and skills without the manual effort - allowing them to focus on the value-add aspect of working with data, the insights and how to act on them.

User experience (UX) is really crucial to everything NGS does, per Gretchen. Their product managers spend a lot of time to really understand the business aspect of things, not just the software pieces. So they now need to learn how to do the same with data. Product thinking is crucial to getting data mesh right, not just creating data products. How can we move to sharing actual insights instead of just data? And especially, who owns creating and sharing insights on data combined from multiple domains?

For Gretchen, it will also be interesting to see what additional insights can be generated when we focus on keeping data clean from the start, not just cleaning it up after the fact by data consumers. What additional insights might come from people actively monitoring the collection and processing of information? Who will generate the new insights? Will it be the traditional data consumers who can now spend the time to work with the data instead of clean it? Or will more insights flow from the data producers as they really get their arms around their own data? The answer is probably both.

Building brave teams - teams that aren't afraid of new challenges or of failure or especially of ambiguity - will be crucial to getting data mesh right in Gretchen's view. People have to welcome change and understand that while change is painful, there is a point and purpose for it. Give them the understanding of what the change is for, what is the reasoning.

Gretchen and team are trying to ensure they aren't over architecting the data platform, putting in too much work too early and locking themselves into choices if there isn't a need. But then it is quite easy to underinvest and not provide what people actually need. So she's really focused on making the platform robust enough but not too rigid or expensive. It's a hard needle to thread.

Many teams are worrying if they are doing data sharing wrong in NGS, per Gretchen. But can they actually really do it wrong? Yeah probably, but if they are open to feedback and paying attention, they don't have to get it "right" the first time to get it right eventually. You can evolve to get to a very good place - prior data setups have been so rigid where change has been extremely painful so that evolution has been tough. Data mesh needs to solve for lowering the cost and fear of change in data but it's still early days.

Gretchen doesn't think you need to build out a huge team to do data mesh, or at least to get moving. Her team's approach is to build a reusable base for generating and managing mesh data products and have a few data architects to keep moving things in the right direction. Then, they have the team and the drive to teach developers how to manage data as a product and get them bought in that it's necessary to do so.

Some rapid-fire insights from Gretchen to wrap up:

We have yet to learn how to leverage the knowledge and context of people without data knowledge in general in the data and analytics space. This is what data mesh tries to unlock but we are still figuring it out.

There are good incentives for teams to produce high quality and reliable data but you have to work with them closely to explain it.

The concept of data lake was to only invest in cleaning and maintaining the data when there was a clear use-case, a clear reason to invest that time. But it was clean-up, not proactive cleaning, and typically had opaque and/or mediocre ownership - that made it much harder to derive the value.

Vendors have yet to really validate data mesh and that means many folks are still sitting on the sidelines. It will be interesting to see if vendors really can ever validate it given how complex and large in scope data mesh really is.

To do data mesh right, many stakeholders need to parse the principles - or at least what the principles are trying to achieve - and then, crucially, adapt them to your culture. Data mesh can't be about cutting and pasting from someone else's implementation.

Shared ownership of data is very hard. That seems obvious but even within the domain, there is shared ownership between the subject matter experts and those shaping the data to share externally. There needs to be strong communication and a good relationship between those parties.

Really spend the time to consider what skillsets you actually need. And when you will need them. It's okay to have more basic data products in the early days of a mesh implementation as developers learn how to work with data properly.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of July 31, 2022 - Data Mesh Radio Jul 31, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#106 Building an Effective Data Strategy: Why oh Why Don't You Start with the Why - Interview w/ Liz Henderson Jul 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Liz's LinkedIn: https://www.linkedin.com/in/lizhendersondata/

Liz's website: https://lizhendersondata.wordpress.com/

In this episode, Scott interviewed Liz Henderson AKA The Data Queen, Executive Advisor at Capgemini. To be clear, Liz was only representing her own views on the podcast.

Some high-level takeaways/thoughts from Liz's view:

To drive buy-in and engagement in a data strategy - especially with people outside the IT/data team - focus on the "why". Why are you doing this initiative or approach? What business goals is it supporting?
Also on driving buy-in for a data initiative, start by listening instead of selling/pitching. Focus on the business needs and work backwards to show how data can help address those needs.
To be successful with a large change-management data initiative, you need the patience, leadership, courage, and will to push forward. And the budget - don't forget the budget :D
You can't have an effective data strategy if it isn't directly tied into the business strategy. Really consider how data can help to support and execute on the business strategy. Data strategy in a vacuum away from the business is a recipe for trouble.
It's very easy to get overly focused in data on what you are delivering instead of why you are delivering it and who it is supposed to serve. If you want to be successful, you need to focus on the latter two. And look to deliver continuous incremental value rather than a back-end loaded value delivery.
Change management is very easy to get wrong in data. Really consider if you can not only get the ball rolling, but keep it rolling and in the right direction to implement a large change. Loss of momentum can mean loss of funding.
If data mesh follows a similar pattern to data literacy, it's likely to be 3-4 years from initial large swell in hype around data mesh - whether that was late 2021 or more now - until we really see a clear picture of how more organizations have implemented. There needs to be a time for trial and error.
Data literacy can only get you so far - you need data storytelling and visualization as the business people need to be able to understand what the data is saying to drive decisions.
The 3 most common ways organizations go wrong with data are 1) technology-first / expecting to buy your way to a solution to challenges; 2) not asking the "why" questions; and 3) not having a data strategy.

Liz started the conversation talking about how data is an asset to the company - not treating data as an asset, an important differentiation - and started with a common theme for the conversation: "why?". Why is data important to the business? Why are we doing the data work we are doing? What is the purpose?

When speaking with company executives, especially those outside the IT/data team, Liz starts every conversation by digging into what is the business trying to do. What is the business strategy? How does data currently play into that strategy? What role do they want data to play in the business strategy? What role might data play if everything were perfect? You can't have an effective data strategy without it tying directly to the business strategy.

When considering any aspect of a data strategy, whether that is data mesh or something else, Liz again recommends starting from the why. In some conversations, IT leaders have said "we want to do data mesh" but when Liz dug deeper, they couldn't answer why. Huge red flag. So when considering a data initiative, think about what is the target impact of the work. How would executing on the data strategy impact the business, not the IT environment? Always ask "so what?". Why is this the right path forward for the business and how is data pushing the business results forward?

For Liz, if you are speaking with executives outside of the IT/data team about a data initiative like data mesh, to drive buy-in, don't start by selling, start by listening. What are the challenges they are having and work backwards to address those challenges through data. What are the business needs and wants, and how can data help them to address them? That should heavily inform your data strategy.

From experience, Liz knows it is quite easy to deliver a very large, costly data initiative that no one really uses or benefits from, AKA a "white elephant". So people play probably an even bigger role in data initiative and data strategy success than most would assume. The cultural aspect is crucial to doing something like data mesh well - if the data consumers still only want to use spreadsheets, it doesn't matter if you are delivering the best data products in the world, there won't be the demand to make the work worth it.

Liz gave a specific example of a recent conversation with a company wanting to do data mesh when talking with IT leaders. When she started to dig in, the business wasn't involved at all with the decision to do data mesh. And a big part of doing data mesh is, you know, the business teams owning the data. It's the first data mesh principle! So if you run across a situation like this, you should ask why they want to implement without the business' involvement. Will that be change for the sake of change instead of driving business results?

Patience, leadership, courage, and will are all necessary to effectively execute a data change management initiative or overarching data strategy in Liz's view. And don't forget the budget - both for getting going and for the continued improvement and maintenance. It's often easy to get data change started but maintaining the momentum, especially in the right direction, can be quite difficult. And if momentum starts to falter, budget can go away quickly in many orgs. Really consider if you are ready for a large-scale change before moving forward. As Zhamak has said many times, "Think big. Start small. Move fast."

Liz shared her insights into how it often takes 3-4 years for a new, large-scale approach to how people work in data to go from everyone talking about it until we see how people are actually implementing it. Data culture and data literacy were all the rage 3-4 years ago but there wasn't much info about how to actually implement a data literacy strategy - we are just starting to see adoption stories being shared. It might take that long for data mesh.

So what happened with data literacy where it is now relatively widespread? For Liz, a lot of it was the general data and analytics industry maturing with a strong general awareness of the concept and need for data literacy rather than any one point or push. With data literacy, employees can understand how data impacts their role and thus how it impacts the business. So that is possibly how data mesh might evolve - broad awareness and the brave bleeding edge folks helping to mature the concept and find the useful patterns and anti-patterns but it still takes quite a while.

Circling back on how crucial the people aspect is for data mesh - or any data initiative - Liz is aligned with the research that people are the silent success or failure point for most data initiatives rather than technology or architecture. Technology-led initiatives in data are quite likely to fail.

Liz recommends that companies looking to be more "data driven" or "data informed" really show to employees how data impacts the overall business and encourage them to consider "the art of the possible" relative to data. And consider ways to take feedback and data requests to enhance the business but not in a single data run for one person type of way, but new data products to share that information with many more people. That way, people know that their insights might actually mean something and might result in a new service or opportunity. And it encourages people to speak up more about what might be interesting additional information to have, leading to better data products.

Data storytelling and strong visualization are an often overlooked part of doing data right per Liz. Having that data translator is crucial to take the information and make it so 1) people can understand what the data is telling us - the insight - and 2) what that might mean - the potential action. Just sharing information without the understanding is overwhelming and confusing.

Liz wrapped up her thoughts with 3 key points on where people go wrong in data: 1) using a technology-first approach - stop picking solutions to try to be a silver bullet, really consider what you are trying to do first; 2) not asking the why, especially about "why do we want to be data driven"; and 3) not having a data strategy at all - you need a compass to move forward.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#105 Data Modeling in Data Mesh Part 1 - Mesh Musings 23 Jul 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#104 How Does Data Mesh Impact the Business: Learnings from T-Mobile Polska's Early Journey - Interview w/ Karolina Henzel Jul 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Karolina's LinkedIn: https://www.linkedin.com/in/karolina-henzel/

In this episode, Scott interviewed Karolina Henzel, Data Enablement Tribe Lead at T-Mobile Polska. FYI, businesses and domains are fundamentally similar in this conversation and are used essentially interchangeably.

Some high-level takeaways/thoughts/summarizations from Karolina's view:

Business transformation and impact is what really matters. Digital transformation is just a mechanism to transform your business into being more digital native and focused. Data transformation is just a part of digital transformation. Transformation should all come down to driving positive business impact.
To drive something like data mesh forward, you really need top management support, likely a C-level executive sponsor. Otherwise, it is very easy for work to get deprioritized and pushed out.
Don't take on a large-scale data initiative unless there are specific business challenges to address. Don't do data mesh for the sake of "being data driven"; what are the issues and why will addressing them help your business? Explicitly define the problems and the pain points.
To drive change, look for "change agents" in the domains. They are people with the will and capabilities to drive large-scale change. They aren't always easy to find but once you start to identify them, patterns will emerge.
The big pain points T-Mobile Polska was facing were: 1) poor/inconsistent data quality; 2) data discovery difficulties; and 3) slow time-to-market for new data and insights.
T-Mobile Polska was able to move forward with data mesh because business representatives in the domains were bought in that addressing the data pain points would drive incremental business value - there would be a return on the data work investment.
Look for quick wins and how to deliver continuous incremental value. If it is all about producing a big bang, you will very likely lose momentum, prioritization, and funding. Continuous value delivery is crucial to keeping people excited about data work.
T-Mobile Polska's data quality issues were caused mostly by a lack of accountability/ownership and not adhering to standard definitions across domains and reports.
Lack of standard definitions - or at least very clearly differentiated definitions - can cause numbers to not match across reports. And that makes people not trust the data. So drive domains to clearly define terms and, if possible, look to create standard definitions across the organization.
Find KPIs that are focused on what actually impacts the business. Dave Colls mentioned fitness functions to break down measuring progress against big challenges into smaller measurements. Karolina and team are making sure the end result is business impact, not technical-only change.
You can drive buy-in that data producers should provide high quality data products by showing producers the impact their efforts are having. Can be a chicken and egg issue at first but you can take the results from one domain or business and show them to another to drive buy-in.
T-Mobile Polska has a data owner in each domain that has high-level data ownership accountability. There is a data steward in each domain as well that is the subject matter expert. When needed, there is also a technical data steward (embedded data engineer) but it's not necessary in many domains.

As a leader in the data governance team at T-Mobile Polska, Karolina has a considerable number of different aspects under her including the data platform, the data warehouse, data quality, etc. Much like data governance as a term has many aspects.

Karolina started off the conversation with an important and useful point she emphasized a few times: you need to have specific challenges you want to solve before you embark on something like a data mesh journey. Not just "let's be data driven" - what business challenges are you trying to solve? And look at them from a "why" perspective - why does tackling this challenge matter? Those challenges could be poor data quality, time spent on non-value-added tasks, data discovery issues, etc. Define the problems and the pain points.

It's important to understand that digital transformation is about business transformation first and foremost. What is your business trying to achieve? And what are the target/expected business outcomes? More revenue? Cost savings? Etc. You need to define the pain points and what doing something like data mesh will do for the organization to secure cooperation from business leaders. And don't count on patience as you work towards a big value delivery in the future - you need to continuously create "incremental value" along the way.

For Karolina, there were three main challenges they needed to address relative to data with their data mesh implementation: 1) data quality was a constant issue - typically stemming from lack of real ownership/accountability and no standard term definitions; 2) data discovery - it was very difficult for data consumers to find data; and 3) time-to-market for new data and insights - the data function was becoming a major bottleneck to the business side.

Data governance can't only be an IT problem/challenge, per Karolina. In their data mesh implementation, they are focusing the central governance team on creating the tools and frameworks for the distributed teams to leverage. For instance, the business and technical metadata comes from the domains but the data catalog is offered as part of the platform by the governance team. This separation of duties has allowed quick time to business benefits when bringing on new teams to their data mesh implementation.

Karolina and team knew they were facing issues with data so they started interviewing business representatives to ask what were their biggest challenges. The governance team heard repeatedly data quality was an issue but didn't know exactly why they were having data quality issues. So they moved to increase accountability, assigning data owners and data stewards. Collaborating with the owners and stewards, they were able to figure out a few major causes were: a lack of real ownership, no common definitions, no real standard measurement of quality, etc. And addressing those challenges resulted in some quick wins to get positive momentum towards delivering continuous incremental value.

At T-Mobile Polska, Karolina has seen how crucial having a C-level sponsor is to succeeding with something like a data mesh implementation. It is very easy to lose prioritization - there is always a more pressing short-term business need than producing high quality data so you need someone that can make sure that data work isn't unreasonably pushed out. Specifically, they created a data governance committee to have strategic supervision of the data governance and data quality efforts and identify the strategic initiatives to continuously deliver incremental value and put things on businesses roadmaps.

Scott asked a question he asks many people: what is the reason for creating new mesh data products at T-Mobile Polska? Karolina shared that data products are initially created to serve reporting specifically in most cases. They can expand to serve additional use cases but there is a specific use case in mind for each new mesh data product.

Karolina discussed some of the new ways of working and the challenges around the necessary mindset shifts to implement something like data mesh. People were just used to data engineering delivering the data. So producers were used to throwing things over the wall and data consumers were used to making asks to a highly data literate group of people. So, they are inventing new ways of working and processes to not have data engineering handling the communication between teams. Business owners are in charge of explaining why owning and serving their data as a product can add value to their org, what is in it for each person in their own org. One explanation that has resonated well - and been proved out repeatedly - is that by moving to a data mesh way of working, there is a significant reduction in time-to-market for new data and insights including for the producing domain.

As part of their data mesh implementation, Karolina and team have been restructuring KPIs to make it possible to measure the impact of the data work they are doing. Their focus is on the impact to the business, not technical focused KPIs. One big goal - with a few proof points thus far - has been a reduction in data work that doesn't add value - reducing the time your data science team spends on things that aren't valuable means they can put more value-add models into production. Another big goal, as previously mentioned, is reducing the time-to-market for new data and insights as many other data mesh implementers are seeing. And Karolina's team is driving buy-in through results by showing data producers how much impact they are having or could have by providing quality data.

As for how T-Mobile Polska started their journey, Karolina and team started with laying the foundation for good data governance. They first found the data owners and the data stewards in each business. Then they explained the new responsibilities for those roles and why they were necessary. The data owner is at the Director level, essentially the business or domain owner, and the data steward is more of a subject matter expert. And if there are complicated data needs, that domain needs a technical data steward - an embedded data engineer - as well; but not many domains need a technical data steward. Another thing specifically mentioned was leveraging "change agents", the people with the will and the capabilities to drive large-scale change.

Karolina then shared some of the issues they've had with data democratization. Similar to what Ust Oldfield mentioned in his episode, just giving access to data when people don't really understand how to leverage data can do more harm than good. So T-Mobile Polska is pushing the not as data literate people to the data catalog as their only point of interface with data on the mesh; the governance team is focused on enabling producers to create standardized reports and datasets to serve those people. The more technical folks have more options to interface with data with fewer technical guardrails.

In wrapping up, Karolina reiterated a few of her main points. 1) Focus everyone on what you are trying to accomplish - what are the priorities? What is the impact to the business? 2) Look to deliver incremental value continuously to build and maintain momentum in your data mesh implementation - without that incremental value, support for your implementation is likely to falter. And 3) C-Level management support is crucial to really drive an initiative like data mesh - without it, your work is likely to get deprioritized and will be continuously pushed out.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of July 24, 2022 - Data Mesh Radio Jul 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#103 4 Years of Learnings on Decentralized Data: ABN AMRO's Data Mesh Journey - Interview w/ Mahmoud Yassin Jul 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Mahmoud's LinkedIn: https://www.linkedin.com/in/mahmoudyassin007/

In this episode, Scott interviewed Mahmoud Yassin, Lead Data Architect at ABN AMRO, a large bank headquartered in the Netherlands.

Some high-level takeaways/thoughts from Mahmoud's view:

It's very difficult to do fully decentralized MDM, which led to some duplication of effort - that can mean increased cost and people not using the best data. ABN tackled this through their Data Integration Access Layer - similar to a service bus.
They are using that centralized layer - called DIAL - to help teams manage integrations that are both consistently running and on-the-fly. It helps monitor for duplication of work instead of reuse.
If Mahmoud could do it again, he'd focus on enabling easy data integration earlier in their journey to encourage more data consumption. Cross domain and cross data product consumption is highly valuable.
The industry needs to develop more and better standards to enable easy data integration.
Data mesh and similar decentralized data approaches cannot fully decentralize everything. Look for places to centralize offerings in a platform or platform-like approach that can be leveraged by decentralized teams.
Most current data technology licensing models aren't well designed for or suited to doing decentralized data - it's easy to pay a lot if you aren't careful - or even if you are careful!
A tough but necessary mentality shift is not thinking about being "done" once data is delivered. That's data projects, not data as a product.
Try to keep as much work as possible within the domain boundary when doing data work. Of course, cross-domain communication is key but try to limit the actual work dependencies on other domains if possible.
A data marketplace enables organizations to more easily create a standardized experience across data products and make data discovery much easier. You don't necessarily have to tie your cost allocation models to the marketplace concept.
Sharing what analytical queries/data integration "recipes" people are using has been important for ABN. It drives insights across boundaries and also creates a lower bar to interesting tangential insight creation/development.
You should consider not allowing integrations across multiple data products by default. Producers should be able to stop integrations - for compliance purposes or because the integration doesn't actually provide good/valuable/correct insights.
Traditional ETL development is about translating the business needs to code. But centralized IT usually can't deeply understand the business context and needs so they deliver substandard solutions. If you consider that business needs evolve, it gets even worse.

Mahmoud started his career in data as an ETL developer so he saw the ever increasing issues with the traditional enterprise data warehouse approach in large organizations. Then he moved on to working with the common way people have approached data lakes - managed by a centralized team - and the issues seemed pretty similar to with data warehouses to him. So he was glad to start working with ABN AMRO on decentralizing their approach to data starting about 4 years ago.

ETL development is about translating the business needs to code. But Mahmoud saw the same problem many organizations are having - it is very hard for IT to really understand the real business context and needs relative to data. They try, but it often is only on the second or third attempt - if at all - that IT would really understand and get it right. They simply cannot get enough context to serve needs well.

So to kick off the discussion, Mahmoud made it clear: there is no perfect data architecture. Not one that fits all organizations and not even one that will fit your organization throughout time as it evolves or potentially across all needs in your organization, so look to what fits your organization at the moment. And it's okay to take pieces from multiple approaches and try them to see if they fit as a cohesive strategy. But do make sure to not just pick the easiest or most fun parts from multiple strategies - cohesion is crucial.

For Mahmoud, a key mentality shift in doing decentralized data, especially data mesh, has been around what "done" looks like. When you think about a physical goods-type product, it's not done once it goes to production. With IT-run data, it was typically that project mentality and "done" was when you delivered the data and moved on. It is crucial to learn how to do actual product management, not just software product management, to understand how to do data as a product right.

A key learning from Mahmoud and team, which echoes something Jesse Anderson mentioned, is trying to keep as much of the work done around data inside the domain. There is obviously need for cross domain boundary communication and collaboration but there is a big cost to crossing domain boundaries when doing data work.

While some people think we should decentralize everything we can in data - Scott calls those people simply "wrong" -, Mahmoud and team found there to be a significant cost to decentralizing the wrong things. They have a centralized governance layer to make things easier on data product producers and consumers. And trying to do fully decentralized MDM (master data management) can quickly lead to duplication of effort and data. Omar Khawaja mentioned similar issues early in Roche's journey.

So, how did ABN tackle these data duplication challenges? Per Mahmoud, they created DIAL or their Data Integrational Access Layer. This is similar to a service bus on the operational plane with the DIAL layer handling data quality checking, business metadata, technical metadata, checking against an interoperable data format, etc. Another instance of a centrally managed service that is leveraged by decentralized teams.

Similar to a number of other organizations, Mahmoud discussed how ABN AMRO is creating an internal data marketplace as the mechanism for centralized data discovery and consumption. This way, there is a standard user experience when looking for and trying to understand what data is available. A standardized experience is crucial to really drive data consumption. The marketplace requirements also lead to a very transparent way to share data.

Per Mahmoud, ABN is also working on making the data integration experience standardized in a few different ways. The previously mentioned DIAL layer is a centralized way to do integrations, whether that is creating new data sets that are reused across multiple downstream data products or integrating in more of a virtualized, on-the-fly way. If you aren't careful, it is pretty easy - especially if there are domains that might naturally touch on similar concepts - to duplicate work, which can cost A LOT. Especially because most data tool licensing isn't designed for doing decentralized data.

As part of or similar to the marketplace concept, Mahmoud talked about how ABN is creating integration recipes. So while recipes may not be a data product, these repeated integrations may be similar to a downstream data product in how they present to data consumers. And other consumers can leverage the same recipe or clone it and adapt it to their needs. It has been very important to share what recipes others are using to drive insight sharing across domains.

To help manage compliance/governance and also to make sure data consumers understand what they are actually consuming, the DIAL layer prevents people from doing data integration without consent from data producers. Ust Oldfield mentioned something similar regarding how self-serve without understanding by data consumers can cause major issues.

Mahmoud and Scott discussed how different just creating data products and data as a product thinking are. If you are really thinking of your data as a product, versioning and the actual data product interface are crucial. And with versioning, it's important to know who will be impacted by a change when assessing if and how a change should happen.

One thing Mahmoud would do differently is focusing more on encouraging/enabling data consumption earlier in the journey. While consumption is picking up, it still is below desired levels and is behind how mature they are with getting data on to the platform to share. Part of the reason for lower than desired consumption came from leaving the focus on data integration until later in their journey. They are trying to find - or if not find, then develop - better standards to make data integration easier. While there are some standards for metadata like OpenMetadata, it's still early days.

Lastly, Mahmoud mentioned how their metadata was just getting to be in too many places so they are building out a metadata lake - a tool-agnostic lake for their metadata. It remains to be seen if this is a common pattern in data mesh but it may address one of Scott's big concerns - the "trapped metadata" problem.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#102 Share Data by Default and Other Stories/Advice from Leboncoin's Data Mesh Journey So Far - Interview w/ Stéphanie Bergamo and Simon Maurin Jul 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Stéphanie Bergamo

LinkedIn: https://www.linkedin.com/in/st%C3%A9phanie-baltus/

Twitter: @steph_baltus / https://twitter.com/steph_baltus

Simon Maurin

LinkedIn: https://www.linkedin.com/in/simon-maurin-369471b8/

Twitter: @MaurinSimon / https://twitter.com/MaurinSimon

In this episode, Scott interviewed Stéphanie Bergamo and Simon Maurin of Leboncoin. Stéphanie is a Lead Data Engineer and Simon is a Lead Architect at Leboncoin. From here on, S&S will refer to Stéphanie and Simon.

Some key takeaways/thoughts from Stéphanie and Simon's point of view:

"Bet on curious people", "just have people talk to each other", and "lower the cognitive costs of using the tooling" - if you can do that, you'll raise your chance of success with your data mesh implementation.
Leboncoin requires teams to share information on the enterprise service bus that might not be directly useful to the originating domain on the operational plane. They are using a similar approach with data for data mesh - sharing information that might not be useful directly to the originating domain by default.
Leboncoin presses teams to get data requests to other teams early so they can prioritize it. There isn't an expectation of producing new data very quickly after a new request, which is probably a healthy approach to data work/collaboration.
Embedding a data engineer into a domain doesn't make everything easy, it's not magic. Software engineers will still need a lot of training and help to really understand data engineering practices. Tooling and frameworks can only go so far. Be prepared for friction.
Similarly, getting data engineers to realize that data engineering is just software engineering but for data - and to actually treat it as such - might be even harder.
Software engineers generally don't know how to write good tests relative to data. Neither do data engineers. But testing is possibly more important in data than in software. We all need to get better at data testing.
Start with building the self-service platform to solve the challenges of the data producers first. You may make it very easy to discover and consume data but if the producers aren't producing any data...
If your software engineers are doing data pipelines at all before starting to work with them in a data mesh implementation, you can probably expect they aren't using best practices.
It's pretty common for good/best practices to be known by only a few people inside an organization, such as with a specialty-focused guild. Look for ways to cross-pollinate information so more people are at least aware of best practices if not able to fully implement them yet.
Trying to force people to share data in a data mesh fashion didn't work for Leboncoin and probably won't in most organizations. Find curious developers and help them accomplish something with data, that will drive buy-in.
As part of #10, data products often start as something serving the producing domain and then evolve to serve additional use cases. They start by serving a specific business need and evolve from there.
Look to build your tooling to enforce your data governance requirements/needs. Trying to put too much on the plate of software engineers probably won't go well.

Around the time Zhamak's first post on data mesh came out in mid 2019, Leboncoin was experiencing many of the pain points Zhamak laid out quite clearly in her article. Their teams were already organized in the "Spotify model" so data ownership was already distributed to many of the domains. But, they were seeing increasing time-to-market - often hitting what Simon called "very long" - for new data initiatives. They already had an organizational model and some ways of working that fit well with data mesh so they decided to give it a try.

So, per S&S, they tried using the data mesh principles for a first use case - building out their recommendation engine. It was a greenfield initiative so it was a good one to test out how well data mesh could work for incremental data needs.

In order to proceed with the pilot, S&S and the rest of the data team had to negotiate with the CTO. Once the pilot was successful, they started embedding data engineers into the teams with the most obvious needs while starting to build out the self-service platform. They already had their CI/CD platform for the operational side so they adapted it to also work with data products. And then they added additional data processing requirements, the governance, etc. to make it as self-service as possible for data producing teams.

The good news, per S&S, was immediate traction with the self-serve platform with the back-end engineers. But they were still suffering from the distance between the data and the software engineering people/capabilities. It was difficult to get the software engineers to see data engineering as a type of software engineering, and many of the data engineers also had a hard time seeing data engineering as a subset of software engineering.

This is a common complaint from many organizations - just because you embed data engineers into domains, that doesn't mean everything becomes easy, you still need to get the software engineers/developers to understand and care about data and data engineering practices; and the data engineers need to learn more about software engineering to best collaborate with the software engineers.

Data pipelines were a major blind spot for a number of the software engineers according to S&S. If the software engineers were doing pipelines, most were not doing them that well with a number of not-so-great practices to put it nicely. So there was a focus on communicating why data pipelines are so crucial to the overall company and how software engineers can learn to do them better. Data mesh can help to facilitate sharing that vision and giving the software engineers ownership over data got them excited in many cases.

S&S are reevaluating if their current internal guild setup is really working with a data mesh approach. It is currently organized only by specialty and that means there isn't a lot of cross-pollination of information - people outside the specific guild don't have easy access to learning new best practices shared with members of that guild. Tim Tischler mentioned the idea of broad group show-and-tells / info sessions around data products that may help with these challenges if done around data practices.

This lack of broader informational best practice sharing is biting S&S and Leboncoin in the behind, especially around testing. While software engineers know how to write really good software tests, most data engineers aren't as good at writing tests and software engineers aren't good in general at writing data-specific tests. But testing is really crucial to be confident in future changes - if you don't know what will happen with a change, that's a bad spot to be.

On driving buy-in, S&S shared that trying to force people along the path to sharing their data just didn't work well. What they found that worked was finding the curious developers and helping them accomplish what they wanted with data. And finding actual projects that can add value, ones with specific use cases - often ones that are directly useful to that domain itself first.

At Leboncoin, many of their data products start off serving the producing domain and then the domain lets others know they've created potentially useful data. This is similar to what Leboncoin does on the microservices side as well with teams often consuming their own events from the enterprise service bus. So the first step for a data product is to build to explicit business needs and then see if additional business value comes from the data product.

Per S&S, another thing that's been helpful is their roadmap process - teams should tell other teams what they will need from them early. If you have need for data, you need to communicate it early so other teams can prioritize it. There isn't an expectation of immediately producing data, which is a healthy way to collaborate.

Leboncoin has an interesting approach to sharing information. On the operational plane, as mentioned earlier, they have a enterprise service bus and teams are supposed to share information that might not be explicitly useful to them - they are asked to consider what might be useful for other teams and to share that at the start of the development process - so there isn't a request to add it in later, it was added from the start! They are doing the same approach on the data side with data mesh. It might not be in data products with strong SLAs but other domains can at least understand what data could be formed into data products.

S&S recommend that when you start building out your federated governance, really start with following the pain. Put data engineers and back-end engineers in the same room to find out what's actually necessary to do and what should be built into the platform. If you can make the tooling enforce governance requirements/needs, that's easier for pretty much all parties.

S&S finished the conversation with a few quick quotes: "bet on curious people", "just have people talk to each other", and "lower the cognitive costs of using the tooling" - if you can do that, you'll raise your chance of success with your data mesh implementation.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of July 17, 2022 - Data Mesh Radio Jul 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#101 H&M's Data Mesh Journey So Far Including Finding Reusability in Interesting Places - Interview w/ Erik Herou Jul 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Erik's LinkedIn: https://www.linkedin.com/in/erikherou/

H&M Career page: https://career.hm.com/

In this episode, Scott interviewed Erik Herou, Lead Engineer of the Data Platform at H&M. To be clear, Erik was only representing his own views and perspectives.

A few key thoughts/takeaways from Eric's point of view:

Data mesh can work well with a product-centric organization strategy as both look to put ownership and product thinking in the hands of the domains.
To develop a good data/enablement platform for data mesh, look to work with a number of different types of teams. That way, you can see the persistent/reusable patterns and capabilities to find ways to reduce friction for future data product development/deployment.
H&M had an existing cloud data lake that was/is working relatively well for existing use cases. But the team knew it likely wouldn't be able to handle where they wanted to go with many more teams producing data products of much higher quality and potentially sophistication.
When implementing data mesh - or any data initiative really - it is easy to fall into the trap of doing things the same way you did before. The "old way" feels safe and it was/is still working relatively well for H&M. So they treated their data mesh implementation as almost a greenfield deploy.
Because of the long-term focus on making it low friction and scalable to share data - the consumers will come as you make them more data literate - most of the early data/enablement platform work has been focused on helping data producers. A common pattern in data mesh but your constraints and needs may not match.
Erik's team is focused on enabling data producers first specifically so his team doesn't become a bottleneck. It is easy for a platform team doing any part of the individual work to become that bottleneck.
Consider how much organizational change you require before starting to create mesh data products. H&M did a large amount of that organizational change, other companies start in their current structure and evolve as they learn more. Both are valid and can work well.
Specific to H&M, a strong track record of good return on investment in AI meant there was less pushback than in many organizations when they started driving buy-in for implementing data mesh.
In the historical data warehouse world, there was less need for data literacy because most people were pushed reports but also couldn't do much, thus not "getting themselves in trouble". If we move to a more self-serve approach, that means we need much better data literacy - it can be a big risk to allow access without understanding. Otherwise, it could be like turning a six year old loose in a fully stocked kitchen where they intend to "make dinner".
Data catalogs could really help push forward general data practices but we still need to have actual conversations too. Being able to ask someone about what data means and similar high context exchanges are crucial.
"If you have a complicated business, you have complicated data."
If your mesh data products don't maintain loose coupling, your data mesh implementation is probably headed for troubled territory. It's one of the key tenets of Zhamak's concept of a data product/quantum, to be architecturally independent.
Input ports are an easily overlooked place to find reuse. Many teams need the same type or style of processing from similar source systems. Having standard input ports can significantly help reduce the complications around building data product ingest mechanisms.

About 3 years ago, when Zhamak's first data mesh article was published (May 20, 2019), H&M was reorganizing to be a product-centric organization; data mesh dovetailed nicely with that strategy - they were moving away from IT as a service-oriented organization.

Erik and team knew that with a move to a product-centric approach, teams would need to be and would become data savvy and "data intense". With their existing setup and knowledge, many teams would not be able to meet the new requirements because while H&M's early AI investments were paying off, many teams just weren't ready to do that complicated of data work. To scale their org-wide data capabilities, they would need something like data mesh as the teams doing AI were the very mature teams - maturing ~200 teams to that level would be essentially impossible. Especially when you think about getting to self-serve data producing and consuming as necessary to scaling ways of working.

The management team at H&M were bought in to the product-centric reorganization so overall, it was not too difficult to drive buy-in for implementing data mesh at the same time, per Erik. There was buy-in and interest in participating from all types of teams from the pure data producer to pure consumer and everywhere in-between. There were a number of teams with the capabilities and resources to participate.

As part of the platform/core enablement team for data mesh, Erik saw how helpful it was to work with multiple types of teams serving different needs. Because they worked on multiple pilots across a range of teams with differing capabilities as well as needs, they were better able to identify reusable parts of the data product development/deployment/management process to add to what the platform team offered.

Erik and team had a leg up on many other organizations considering data mesh: a data platform that was working well and serving current customer demands. Erik called the data consumers "happy enough" with their existing cloud data lake as they could mostly do what they needed to do. But the data team also knew that their existing cloud data lake would not scale to what they needed in the mid- to long-term as it would not likely be able to handle ~200 teams all producing data products. A key benefit of this existing well-functioning solution was there wasn't a rush to get a replacement in place.

H&M's approach to building out their enablement/data platform was almost a greenfield approach, per Erik. He said it is easy to fall into similar patterns of what you've done in the past, especially since their existing solution was already working. But they knew they had to stay away from the gravity of what they'd done before and look to new ways of working. But again, they had the time to do it right and think about the initial stages as a bridging solution, not a rip and replace. And thus far, it is working well.

To date, the main focus of H&M's data/enablement platform team has been building the self-service capabilities for the data producers. There is a large pool of highly data literate data consumers already, especially the teams mentioned above that are advanced in applying AI. So these initial stages are about testing so they can discover the ways to make it easy on data producers to create and manage data products. Most of the initial data products are source-aligned, generic data products not tailored to any specific use case.

The mid-term data/enablement platform strategy focuses on iteration and learning patterns. Erik and team know they won't get it all right upfront, so making sure people understand there will be iteration and evolution is key to keeping people bought in to the long-term, big picture vision. That's where they plan to really focus on making the platform as easy as possible for consumers as well.

Erik shared the big reason for focusing on building the enabling capabilities into the platform rather than the data processing or other capabilities. First, they already have a good platform that can do the data processing :D But also by not taking on any of the work themselves and finding ways to reduce friction, they can stay away from becoming a bottleneck and make it easier for more teams to participate. It is easy to get dragged into specific work.

Per Erik, as many guests have said, data mesh is very much of an organizational-focused effort. The technology and architecture side aren't easy but to have a successful implementation, more effort will need to be spent on the organizational aspects. H&M was inspired by what Spotify has done with their organizational approach, leading to their return to the previously mentioned product-centric thinking/approach. One interesting point is that Erik believes you need to implement at least a decent amount of your organizational change at the start of your journey or teams will struggle to deliver mesh data products.

So why did H&M not have much pushback, why were so many teams including data producers bought in to participating in the data mesh implementation? Per Erik, H&M has had a good track record of driving strong returns on their early investments in AI, especially around driving business optimizations. But overall, people understood that the current AI setup would not scale to a wider audience. So they've seen strong returns from doing data well and trust the data leadership to deliver further.

Erik made the interesting point that in the data warehouse world, most data consumers were plenty data literate relative to needs - but that was because they were fed the reports directly with no real push to be inquisitive. Everything was also controlled so there was a good data quality filter. Once you open up to self-serve consumption, that can cause issues.

The big issues Erik has seen with allowing self-serve access without proper training / data literacy efforts are mostly around data misuse. Not unethical or inappropriate use but simply misunderstanding what the data means and which data to use to answer important questions. But he hopes that their data mesh implementation will guide people to the right information, especially by providing the right contacts to get more information.

Per Erik, many many people in data are putting a lot of hope in where data catalogs are headed. But the data catalogs should not be the only way people learn about what data is available or what said available data actually means. Conversations about data are valuable - and they can be fun! A good example Erik gave was if people are asking a lot of unexpected or possibly strange questions about your data product, it might be a signal you should re-engineer it.

Erik and Scott agreed that part of where data mesh approaches things so differently is the emphasis on loose coupling between data products. Coupling in data has made it extremely difficult to make changes historically so we need to prevent that BUT still make data interoperable. Otherwise it's just high quality data silos. But not every data product needs to interoperate with every other data product. And there also needs to be different types of data serving based on consumer needs so data products will need multiple output ports.

In wrapping up, Erik shared the specific types of patterns and practices the data/enablement platform team is working on. Schemas and generally schema handling, sensible defaults, input ports, etc. The input port example was really interesting and enlightening - Scott hadn't heard that example in 80+ interviews.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#100 A Lookback at What We've Learned So Far - Mesh Musings 22 Jul 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#99 Getting Philosophical About Knowledge and Sharing Experiences via Data - Interview w/ Andrew Padilla Jul 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Andrew's LinkedIn: https://www.linkedin.com/in/andrew-padilla-8988094a/

Datacequia website: https://www.datacequia.com/

Andrew's personal Substack: https://datacequia.substack.com/

Data Mesh Community newsletter Substack: https://datameshlearning.substack.com/

In this episode, Scott interviewed Andrew Padilla, who runs a data and software consulting company - Datacequia - and serves as editor of the Data Mesh Learning community newsletter.

This one is a bit more philosophical about sharing information/knowledge so it's one to sit and think over. Things in quotes are direct from Andrew.

Some key takeaways/thoughts that come from Andrew's view of data mesh and the data space in general:

To move from sharing the 1s and 0s of data to actually sharing knowledge, we need to harmonize data, metadata, and code - "the digital embodiment of knowledge". That's where Andrew hopes the mesh data products can head.
Software development isn't cutting it for sharing knowledge. Will data product development? Do we need to move to knowledge-centered development instead? Remains to be seen.
We still don't know how to model well - in data - what is going on in the real world. What are the experiences of the organization? Can we really define an "organizational experience"? Event storming tries but seems to fall short often.
We must learn to treat organizations like living entities. Organizational experiences cross multiple domains and the types of experiences will change, will evolve - possibly quite quickly. We again have to get better at modeling those and evolving how we share knowledge about the experiences.
Knowledge graphs are the best way we have currently for combining information across domains. We still haven't fully figured out how to leverage our cross domain knowledge though.
Historically, we've bent our ways of working to the limitations of the machines. We need to spend more time on bending the machines to better match the way humans store, process, and share knowledge.
Data centricity is an interesting concept but might take our current imbalance of data versus operational focus too far towards data. But that might be what is necessary to really get to balance. It remains to be seen. But it's crucial to understand a data-first focus isn't necessarily a knowledge-first or knowledge as a first class citizen approach.
It's important to understand that mesh data products are a means to an end in data mesh. Yes, they are crucial to sharing information but they are there to serve a purpose, not that they are the purpose.
In data mesh, it can be easy to focus too much on creating data products of immediate utility or that are high value in and of themselves. But it's important to think about how data products together create value - and maybe not immediate value - to really drive forward our understanding of the organization's knowledge and experiences.

Andrew started the conversation with his hope and vision for data products - or the data quantum - in data mesh. Historically, data, metadata, and code are not often grouped together and even less frequently are they in harmony. They belong together as that harmony creates a higher level abstraction to share knowledge, not just the 1s and 0s of data. To get data mesh right, Andrew believes you have to really figure out how to build mesh data products with that harmonization in mind. And you probably won't get it right at the start of your journey and that's okay - we are all still figuring it out.

We possibly - or even probably - need to move from software development and even data product development to knowledge development in Andrew's view. Knowledge development meaning centering the development process on sharing knowledge. So much of what data we share lacks the actual context of what happened in the real world, the experiences of the organization. But we still don't have a great way of sharing those organizational experiences - event storming in DDD (Domain Driven Design) tries to address this but it often falls short. How can we progress towards modeling organizational experiences?

For Andrew, as many have said, if we can figure out how to do data mesh well and create some good standards, it is very well-suited for cross organizational knowledge sharing and collaboration. Not data selling but collaboration - similar to what Jarkko Moilanen mentioned in his episode's discussion about the data economy. But it's still early and data mesh will not be the silver bullet to figuring out how to do that cross organizational collaboration well - and crucially safely and compliantly.

We are just starting to understand, to develop a point of view on what Andrew called the "knowledge of experience". Per Andrew, "knowledge, by definition, is just the acquisition and use of experience and/or education". We must learn to treat organizations like living entities -> organizations have new experiences and are changed by them. And the types of experiences are also changing as the real world changes. In the 80s and 90s, much of communication between entities was done by fax. Faxes aren't nearly as common for most organizations nowadays. But the evolution of experiences seems to be accelerating and we are still struggling to capture those in data.

For Andrew, software and data must reflect or "be the embodiment of" those organizational experiences. Do we even know what it really means for an organization to have an experience? Much less model it? And those experiences don't take place in isolation to a specific domain. So once we figure out the experience modeling for the domain, we then get to try to figure out how to scale that for experiences across domains. Yes, we have some work ahead of us!

Knowledge Graphs probably hold the key to the cross domain information sharing, per Andrew. He called them the "glue". They are as good as we have technologically currently for developing and leveraging the knowledge tie-ins across domains. They allow people to bring more of a "history of experiences" to the conversation.

In Andrew's view, we've historically had to deal with the limitations of what technology could do when thinking about knowledge sharing. Our ways of working have bent to those limitations. But we should start to try to bend the machines more to the way humans store, process, and share knowledge. Almost a higher-order Law similar to Conway's Law that we must push the machines to communicate and work in the way humans think.

Scott asked about Dave McComb's data centricity concept. Andrew thinks that data centricity might swing too far towards data only instead of maybe knowledge first or knowledge as an equal footing to the 1s and 0s of software. It might shake things up to move towards data centricity but does it get us closer to knowledge as a first-class citizen? Per a follow-up, Andrew sees the data, metadata, and logic as components that must work together in their specified functions in harmony and balance much like your vital organs. So just swinging to data doesn't necessarily swing us towards knowledge.

Andrew brought up the concept of the 2D person in physics. What might be just a simple line in three dimensions - something you can easily avoid - is a full stop blocker to the 2D person. 2D is choosing between is this data or is this code. We are living in a 3D world - even 4D when you think of time - so we need to move past our current ways of working and think on a higher plane about how to approach our work.

Data products in data mesh are simply a means to an end for Andrew. They are the building blocks to build out your knowledge repository and sharing. But it's crucial to understand that they are for a purpose in the greater organizational sense, not the end accomplishment.

Andrew finished up by sharing his ideas around data monetization. If you are specifically thinking of data, even for internal sharing, as a monetary asset, does that put experimentation on the back burner? Do you only go for the sure bets? Or only focus on things you know have a specific use instead of simply potentially valuable? Andrew thinks that there is significant value to R&D incubation in general and specifically to data. We aren't in a world yet where producing those speculative or interesting with no specific utility data products is cheap so maybe those come later. But it's important to not become overly focused on the immediate utility of each data product itself instead of how it fits into the greater picture of your organization's knowledge.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of July 10, 2022 - Data Mesh Radio Jul 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#98 How to Nail Your Data Mesh Vendor Assessment: A Journey Story - Interview w/ Jen Tedrow Jul 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Jen's LinkedIn: https://www.linkedin.com/in/jentedrow/

In this episode, Scott interviewed Jen Tedrow, a Product Management Consultant at Pathfinder Product Labs who is currently working with a large client on a data mesh implementation. She was only representing her own perspective in this episode.

Some key takeaways/thoughts from Jen's point of view from the conversation:

A data mesh vendor assessment is likely to be different than almost any other vendor assessment you've done before, especially if you aren't evolving something existing. There is so much more to cover and the overall platform needs to meet your needs, integrate with how you handle data on the application side including integrations, comply with your governance standards, fit within budget, etc. That's a lot of needles to thread.
Spend considerably more time doing the discovery process in your data mesh vendor assessment than you would for a normal vendor assessment. There are a lot of potentially hidden needs / wants and it is far better to surface them early.
By digging deep into stakeholders' desired outcomes, you can understand what you need to deliver but also, you can get insight into driving buy-in. Address the challenges preventing them from the desired outcomes and they will feel seen and heard.
As many guests have said, lead with empathy. Change is painful. But if you are realistic with people and make them feel seen and heard, it will be much less painful.
When speaking with potential users, again really spend the time to make them feel seen and heard - reflect back to them what you heard. And have them share what is their ideal state. You may not be able to fully deliver on it but it's important to understand where they want to go.
As you are learning new information, share that in a continuous stream with stakeholders so they understand the recommendations you are making along the way and at the "end" of the assessment - it doesn't really end when you finish the assessment, so "end".
Be prepared for there to be capability gaps - possibly significant - between what you want now and what is available in the market or that you are able to build in your budget. There are just a number of capabilities that aren't really part of any vendor offering at the moment. Waiting until everything is perfect will mean you are waiting for a while!
It's crucial to focus on what do you value most right now and also in the future when making vendor assessments. There are so many nice-to-haves with a data mesh implementation but you'll need to compromise. Jen and team developed a good framework for evaluating offerings as to whether they fit actual needs or just wants.
The three most important aspects to meet right now for Jen and team through the self-serve platform were user experience/low barrier to usage, automation, and ability to easily integrate the tools together and with the existing stack.
When figuring out what capabilities you need from your platform, create high-level task-based use cases, not systems requirements. This will prevent you from steering too much towards trying to serve any one specific use or getting bogged down in tech compared to capabilities.
Look to past instances of failed or underwhelming implementations of tools and processes in your organization to find the common ways implementations fail in said organization. And then work to avoid those :D
It is crucial to make sure everyone understands a data mesh implementation is an iterative process. You will continue to listen to feedback and evaluate how things are working and then make further improvements - it's a journey!
It will likely be quite tough to move forward in a data mesh implementation if you don't align your data strategy and work with your business partners to create a mutually beneficial target outcome. And carve out time for teams to actually be able to deliver data products and participate in your data mesh implementation - they need time to do the actual work. Make it a priority.
It's very important to provide an easy way for teams to start participating in your data mesh implementation/journey. Just asking a team to participate won't cut it. Make it low friction, make it beneficial to them. Easier said than done but still very important.

Jen has done a number of vendor assessments in the past. But this one was a doozy. There isn't a ton of information yet about how to do data mesh really well - especially the platform side - so it is difficult to assess exactly what capabilities you need. There are still a number of gaps in vendor offerings when you do know what capabilities you require to meet so that is more difficult. Then add in that you are likely bringing on multiple new vendors at once and making sure they play nicely together and with your existing technology stack. And then there is budget... So, as mentioned, it was a doozy.

Overall, Jen's role was trying to account for specifically 4 different capabilities: data discoverability, provisioning, observability and quality, and access control.

For Jen, in most vendor assessments, there is a much tighter scope around what you are trying to assess. E.g. looking for a streaming technology or looking for an integration provider. But with a data mesh platform, there are so so many moving pieces, it was a very unique and difficult challenge to find a good harmonious match to cover as much of the needs for now and to work with the vendors to grow together to cover future needs.

Jen used a framework she's used for her vendor assessments historically of Discover -> Align -> Assess. The goal is to answer what is important now and where do we need to go - and then does a certain offering help us address both of those points. But the big difference with this assessment was how much more time was spent on discovery.

Discovery in general can be challenging for a few reasons, per Jen. One is that people want to move quickly - make the decision and plow forward. But that can often lead to not picking a good direction and creating hard to pay down tech debt. So investing the time to understand all your needs/wants is crucial. Digging deep into stakeholders' desired outcomes has a few benefits: 1) you know what you need to deliver and 2) you know how to drive buy-in - by addressing the challenges preventing them from their desired outcomes.

Jen stressed, as many many past guests have, the importance of leading with empathy when working on anything data mesh related. Change is on the horizon for everyone and change is painful. When leading with empathy, stakeholders and users alike have been very willing to share with Jen their challenges and where they want to go in the future with data.

"Why are we spending this time together?" is an important question to answer, per Jen. The people you interview will be more willing to openly share if you spend your time listening rather than selling. Make them feel seen and heard, spend the time to reflect back what someone said - let them know you weren't just hearing but listening. And talk with them about what is the current state and what is their desired state. And of course let them know that implementing data mesh isn't a threat to their jobs.

Per Jen, you may not be able to give everyone their desired state, especially in the initial implementation phase, but understanding why they want that certain capabilities might make it easier to deliver something of value to them if not the ideal. And then spend time with the stakeholders to constantly share what you are learning so it isn't a sudden recommendation at the end of an evaluation phase - that constant feedback lets people know why you are making the decisions you are.

Right now, pretty much no matter what capabilities you are looking for in your data mesh platform as part of a vendor assessment, expect gaps per Jen. Zhamak has mentioned this frequently too. It is important to evaluate what is necessary and what is nice to have now and then what will be necessary down the road. Will the vendors or offerings you are looking at be able to grow into those gaps over time to meet future needs too?

One aspect that made this assessment so different for Jen was that vendor assessments are typically looking for a single or logically bundled capability and doing a vendor bakeoff. It is based on the "known knowns". But for data mesh right now, there had to be so much more discovery work. It was crucial to focus on what stakeholders really value. For Jen and team, right now, that was on lowering the barriers and friction to usage so the UX (user experience) was pretty crucial. As was being able to stitch solutions together without a ton of custom work.

For Jen, a few requirements really came to the surface as crucial. Again, that user experience was one. But automation was another. How could they make creating and managing a data product an easy transition - or at least as easy as possible?

What worked well for Jen and team to really understand what capabilities were actually crucial was focusing very much on task-based high level use cases without involving any necessary systems requirements. It meant they could focus much more on what needed to get accomplished instead of specific examples that had more custom needs. Doing this created a very clear picture of what they actually needed.

When thinking about how to adapt data mesh to your organization, Jen recommends looking at what has worked and even more closely at what hasn't worked in past tool and process implementations specifically in your organization. What caused failures so you can look to avoid going down the same path. There are so many potential areas of friction in a data mesh implementation, do your diligence to find common failure patterns to your organization to avoid them.

Jen talked about how a successful data mesh implementation will really be about the intersection of people, process, and technology. You need to be including good change management principles into everything you touch. And make sure people understand that this will be iterative. It won't be perfect from day one but you are going to be listening and improving along the way.

To drive momentum, Jen recommends highlighting - loudly and often - early adopter successes. It shows that you are adding value but also rewards your early adopters - that hopefully spurs others forward to move forward in their data mesh participation too. And be honest with everyone that doing something like data mesh will involve change - and change is painful.

Jen and Scott discussed what use cases to look for in your early journey. Jen recommends balancing three factors: 1) what will be impactful, 2) what is possible, and 3) who is willing to partner with you. It's important to show those early successes to keep the funding coming as data mesh is not a single upfront cost - it requires continuous investment.

Jen recognizes how lucky she is to have a leader that is sharing their vision widely, driving buy-in and aligning strategy on data mesh with business partners to make it successful for all parties. A big crucial aspect is that teams have enough time carved out to actually create data products - without this, incentivization is tough. And constantly look to raise the visibility and amplify the wins.

For Jen, it's been important to repeatedly paint a compelling vision in many conversations. It's fine to be a bit repetitive. Share the current picture and then talk about what it could become. This is important to making participants "willing to accept the pain of change". You want to develop a symbiotic, mutually beneficial relationship with those early adopters. Participating in data mesh has to be a win for them too. And teams aren't ready to just adopt data mesh, you need to create the processes to support and enable them.

Jen wrapped up the conversation reiterating a few points about your vendor assessment process: 1) be prepared to spend more time on discovery than you probably think is necessary going in because it will highlight the pain points and capabilities that are most crucial; 2) focus on task-based use cases when considering necessary capabilities - keep the systems out of it; 3) really spend the time to understand your sourcing process internally; 4) it's okay to have very frank discussions with vendors - look to spend your AND their time wisely and share your hard constraints and requirements; and 5) constantly reflect back your progress and learnings in your assessment along the way and especially share the results of the assessment broadly to continue to share information and drive buy-in.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#97 What is a Mesh Data Product to the Business Owners and Users? - Mesh Musings 21 Jul 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Community open discussion meetup hosted by Eric Broda: https://www.youtube.com/watch?v=OwtQ37WYK1g

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#96 The Power of Empowerment and Driving Business Value: Data Mesh at Roche - Interview w/ Omar Khawaja Jul 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Omar's LinkedIn: https://www.linkedin.com/in/kmaomar/

Omar's State of Data Mesh presentation: https://www.youtube.com/watch?v=S5ABE4bevn4

Adam Grant book Think Again: The Power of Knowing What You Don't Know: https://www.amazon.com/Think-Again-Power-Knowing-What/dp/B08HJQHNH9

Lean Value Tree definition: https://openpracticelibrary.com/practice/lean-value-tree/

In this episode, Scott interviewed Omar Khawaja, Head of Business Intelligence at Roche Diagnostics. To be clear, Omar was only representing his own viewpoints and learnings, not necessarily those of Roche.

Some interesting thoughts/takeaways from Omar's point of view and learnings:

If you are going to make progress in a data mesh journey, you must be okay with "good enough". Perfect is the enemy of good and done. Measure, learn, and adjust along the way but get moving and keep moving. It's okay to make mistakes - recognize and correct them.
Echoing a number of past guests, change management and organizational challenges will take a large portion of a data mesh implementation leader's time and effort - likely far more than most would expect. Focus on empowering people and showing them why this can work for them. And what it means for them.
Data mesh cannot be your entire data strategy. If you are implementing data mesh, it must only be part of your data strategy. Start from the why. Why undertake something as transformational and difficult as implementing data mesh? What business value will it deliver?
Data-as-a-product thinking is the true heart of a data mesh implementation. It's far more than just creating data products. Data product discovery is crucial, much like discovery in regular product management. Take considerable learnings from product management in other disciplines.
Focus on outcomes in day-to-day data work. What are you trying to deliver? What is the value in it? For whom? How will we measure if we are successful? And were we actually successful?
We need to get data people to rethink creating point solutions - sometimes called project management thinking - where they deliver a dashboard and the dashboard itself is the focus. This leads to fragility that could be prevented by focusing on the entire data lifecycle to create the dashboard with the dashboard - and many other chances for data reuse - as an output.
Roche is being quite flexible around who develops data products - it is all about the capabilities and needs. Often, it is data engineers in the domains, enabled by the central platform team. But it can be data/business analysts or software engineers too. If the data product isn't overly complex or if a business analyst really understands data, why can't they be the data product developer?
It would have been the definition of insanity - trying the same thing over and over and expecting different results - for Roche to just move from an on-prem data lake that was having scaling and quality issues to a cloud data lake. Many other aspects needed to change. The organization needed to unlearn and relearn a number of things and data mesh was a great vision for where they could go.
Roche saw some duplication of work across data products so they adjusted and made their data product discovery and design phases very public. Making it public can increase collaboration early in a data product's life as well so you might find additional data consumers in the development phase.

Omar started the conversation with a definition of what Business Intelligence means to him and how it has evolved from a mostly reports-based function - or descriptive analytics - to include predictive analytics and then prescriptive analytics. But monolithic approaches - enterprise data warehouse, data lake, etc. - just haven't led to great outcomes for many (most?) large organizations. Omar quoted Albert Einstein, "Insanity is doing the same thing over and over and expecting different results." So why keep trying to throw technology and a monolithic architecture at our growing data and analytics challenges. So in 2020, Omar was happy to help evaluate if decentralization could work in data for Roche.

They moved to a domain-aligned model with the business intelligence, analytics, and data engineers - at least those not building the platform - moving into the domains. That way, they can become the people who know the data best including the business context. They can help to shape and develop data products.

For Omar, the way most analytics work has been done historically - and how people have thought about analytics work - the outcome of analysis itself is the focus. So if you build a dashboard, an analyst creating a dashboard might spend all their time focusing on the dashboard with little to no thought about the fragility of the inputs to the dashboard. What happens if upstream data changes? In data mesh, the dashboard is an output of data work that is more easily created and managed because the analyst knows the upstream will be maintained as a product and even if there are changes, there is communication about impending changes. And they might be able to reuse the data in another analysis.

Regarding roles needed in domains for data mesh, Omar gave the incredibly common data mesh answer of "it depends". There needs to be an owner who understands the data-as-a-product concept - not just creating data products - as the lifecycle of data is crucial to doing data mesh well. But each domain and each data product has different needs; so focus on building the cross-disciplinary team needed to take care of the job now - and in the future - and not on the exact composition of each team. The use cases aren't cookie cutter, why make the teams cookie cutter.

At Roche, for many domains, the initial data product design is done by a senior architect but the data product development is done by a data engineer - possibly called an analytics engineer - but if there isn't a need for extensive data engineering for the data product, it can be developed by a data/business analyst or software engineer. The role title doesn't end up mattering, the team capabilities and the needs do. Design with data consumers in mind.

Per Omar, for all organizations, there is some kind of existing analytics practice - or brownfield. It is important to leverage what you've built historically - both the types of analytics and the team. You might have great insight into what information consumers want. And many of the people involved in the data warehouse or lake can evolve into a role that is highly valuable in data mesh - they probably know your data quite well. But, it is crucial that they understand why they need to evolve and are given the resources to do so.

Echoing many previous guests, Omar said to focus on the outcomes in data mesh, not the exact structure. Empower your teams to figure it out and enable them to do the work, especially via the data platform. The role of the "citizen data scientist" never really worked; now we can focus on giving many more access to information and insights, not just access to data with little to no context. And sharing across the company is crucial to finding scalable, repeatable practices and patterns.

At Thoughtworks' State of Data mesh conference, Omar presented how each of the four pillars or principles of data mesh represents Mind, Body, Heart, and Soul. Digging into data-as-a-product, or the heart of data mesh in Omar's analogy, there is far more than just creating data products. He shared their learnings in data product discovery - how do you figure out what products you need? What are the expected outcomes from creating this data product? Who is going to use it?

Omar discussed the need for a real mindset shift to understand data-as-a-product especially. It doesn't come naturally for most people so you need very conscious change management - change management and organizational challenges will almost certainly take considerably more of your time in a data mesh implementation than you expect. And it's not shoving people forward or dragging them - it's taking them by the hand and working with them to find a good way forward. Once they feel the empowerment, Omar is seeing most people really like this new way of working. And they are focused on continual value delivery instead of the project, one-off type of value creation.

When Omar joined Roche a couple of years ago, his first big task was forming the data strategy. Should they just do a data lake in the cloud? Was that transition working well for most organizations? The answer for him was no and he repeated about the definition of insanity - trying the same thing over and over and expecting different results. He took a lot of inspiration from the book Think Again by Adam Grant. There was a lot the organization needed to unlearn or relearn. And when COVID-19 hit, it forced organizations to get much better at communication and collaboration so data mesh was a great fit - they were already embracing decentralization.

Omar talked about how data mesh was one of the aspects of their data strategy but only one aspect. The overall goal was and is "how do we become data-driven as a company?" So you should start from the why. Why consider data mesh? What are you trying to achieve? Is it worth it?

This set of questions should also apply to everyone's day-to-day work in a data mesh implementation, per Omar. Again, focus on outcomes, what are you actually trying to deliver? Is it just a dashboard or are you trying to deliver the insights that dashboard will create? Omar recommended the Lean Value Tree approach as one way to focus your time.

Omar returned to the concept of product mindset and product thinking in data. What value are you trying to deliver? Then how much value do you expect it will deliver? Then how will we measure if we are successful? Then were we successful in delivering expected value? A big part of this is the discovery process - drive towards a business focused discussion about outcomes.

In Omar's view, no one will get really everything correct, much less in something as large, complex, and new as data mesh. It's okay to make mistakes. That's part of learning. But you need to get to a "good enough" place and move forward. Measure along the way and adjust. It's a journey, there will be trials and tribulations along the way. Learn from it and adjust. Collaborate and move forward together.

In Roche's early journey, they found some teams were duplicating work so they moved to fix that. What they learned was they should provide very early visibility in to plans to prevent teams from spending time on the same things. The data product discovery and design phases are now quite public and that's worked well. Instead of teams duplicating work, they are often early consumers of data work other teams have done.

In wrapping up, Omar again reiterated to focus on what you are trying to deliver, what is the value and that it's okay to move forward with an incomplete picture. You'll make some mistakes but prepare to learn and adjust and just get to making progress. Pretty sound advice.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of July 3, 2022 - Data Mesh Radio Jul 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#95 Measuring Your Data Mesh Journey Progress with Fitness Functions - Interview w/ Dave Colls Jun 30, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Dave's LinkedIn: https://www.linkedin.com/in/davidcolls/

Zhamak's data mesh book: https://www.oreilly.com/library/view/data-mesh/9781492092384/

Building Evolutionary Architecture book: https://www.oreilly.com/library/view/building-evolutionary-architectures/9781491986356/

Team Topologies book: https://teamtopologies.com/book

The Agile Triangle regarding helping you decide how thin to slice: https://www.projectmanagement.com/blog/blogPostingView.cfm?blogPostingID=5325&thisPageURL=/blog-post/5325/The-Agile-Triangle#_=_

In this episode, Scott interviewed Dave Colls, Director of Data and AI at Thoughtworks Australia. Scott invited Dave on due to a few pieces of content including a webinar on fitness functions with Zhamak in 2021. There aren't any actual bears, as guests or referenced, in the episode :)

To start, some key takeaways/thoughts and remaining questions:

Fitness functions are a very useful tool to assess questions of progress/success at a granular and easy-to-answer level. Those answers can then be summed up into a greater big picture. You should start with fitness functions early in your data mesh journey so you can also measure your progress along the way. To develop your fitness functions, ask "what does good look like?"
Focus your fitness functions on measuring things that you will act on or are important to measuring success. Something like amount of data processed is probably a vanity metric - drive towards value-based measurements instead.
Your fitness functions may lose relevance and that is okay. You should be measuring how well you are doing overall, not locking on to measuring the same thing every X time period. What helps you assess your success? Again, measure things you will act on, otherwise it's just a metric.
Dave believes the reason to create - or genesis of - a mesh data product should be a specific use case. The data product can evolve to serve multiple consumers but to start, you should not create data products unless you know how it will (likely?) be consumed and have at least one consumer.
Team Topologies can be an effective approach to implementing data mesh. Using the TT approach, the enablement team should focus on simultaneously 1) speeding the time to value of the specific stream-aligned teams they are collaborating with and 2) look for reusable patterns and implementation details to add to the platform to make future data product creation and management easier.
We still don't have a great approach to evolving our data products to keep our analytical plane in sync with "the changing reality" of the actual domain on the operating plane. On the one hand, we want to maintain a picture of reality. On the other, data product evolution can cause issues for data consumers. So we must balance reflecting a fast-changing reality with data consumer disruption, including downstream and cross data product interoperability. There aren't great patterns for how to do that yet.
There is a tradeoff to consider regarding mesh data product size. Dave recommends you resist the pull of historical data ways - and woes - of trying to tackle too much at once. The smaller the data product, the less scope it has, which makes it easier to maintain and the quicker to deploy and feedback cycle. But smaller-scope data products will increase the number of total data products, likely leading to harder data discovery. And do we have data product owners with many data products in their portfolios? Dave recommends using the Agile Triangle, framework to figure out a good data product scope (link at the end).

Dave mentioned he first started discussing fitness functions regarding data mesh to shift the conversation from people asking "what do we build?" to "what does good look like?" Fitness functions, when done right, can give a good view of how well an organization is doing relative to data mesh implementation goals by providing objective measures of success at a granular level that can be summed up to a bigger picture.

So what is a fitness function? As defined by Thoughtworks in a May 2018 technology radar, "Borrowed from evolutionary computing, a fitness function is used to summarize how close a given design solution is to achieving the set aims. ... An architectural fitness function , as defined in Building Evolutionary Architectures, provides an objective integrity assessment of some architectural characteristics, which may encompass existing verification criteria, such as unit testing, metrics, monitors, and so on." Source: https://www.thoughtworks.com/radar/techniques/architectural-fitness-function

Fitness functions can take us from measuring success on vanity metrics - like amount of data processed or stored - to value-based metrics, per Dave. It is important to think about what good looks like for the now and the future. So starting to put your fitness functions in place early in your data mesh journey can give you a good sense of where you've been when designing where you want to go. Fitness functions give you an ability to stay focused on "why are we doing this?" - intentionality is crucial.

When thinking concretely about some fitness functions for a data product, Dave gave a few examples. E.g. does this meet health checks for testing, is it satisfying SLOs, etc. It can be a good idea to have a target metric with yes/no type of answer as you start to use fitness functions. Metric measurements without context are typically not valuable. Latency of 5min might be great for one data product and not another. Accuracy of 90% might be atrocious for one and great for another.

You can implement fitness functions for all aspects of a data mesh implementation in Dave's view. Look at the four key principles of data mesh from Zhamak's work and you can start to break down your goals for each one into fitness functions. A good overall question to try to answer is are we reducing the interdependence of domains. So for the domains, are they providing value via their data products to consumers? For the platform team, have they made it easier to create and manage data products? For governance, overall is the value of the whole implementation greater than the sum of the parts? You can answer fitness functions at a micro level and then take your overall measurements to get a more complete picture to assess how your implementation is going.

For Dave, when assessing that bigger picture, as previously mentioned, it is good to think about your bigger picture and measurement of success over time. What you measure with fitness functions can - and should - evolve but having your rates and ratios spanning your implementation timeline can give you a good indication of where you've been improving and where you need to work more.

Similar to Shane Gibson's episode on using patterns in data, fitness functions may be valuable to other organizations and not yours and they may lose relevance over time per Dave. Measure things that will cause you to act based on the outcomes. Nothing in your data mesh journey should really be seen as done and fixed. Things should evolve or it means your organization is stagnant. Change for the sake of change is obviously bad but you should evaluate if your fitness functions are still helping you measure against your idea of what good looks like. Do we need fitness functions against our fitness functions?

Dave talked about how well Team Topologies aligns with implementing data mesh as organizational changes are such a crucial part to success. The Team Topologies approach focuses on enabling the domain team - called a stream-aligned team - to be "the primary unit of value delivery in IT". A platform team enables but loosely coordinates at most when possible to prevent blocking the work of the stream-aligned teams as much as possible. Per some past research conducted by Dave, it took 12x longer for a team to do work if they had to go outside their team - so prevent that if possible! But, Dave warned that right now, especially in data mesh, it is important to not just add more and more work to the stream-aligned team.

Team Topologies can help us answer how do you build capabilities in a decentralized world, especially to implement something like data mesh. Per Dave, it is helpful when there is multi-disciplinary collaboration by the stream-aligned teams with the enablement teams to develop their first data products. The enablement team is also tasked with bringing back incremental learnings to add to the platform to make the next team's work creating and maintaining data products better - continuous improvement via learning from each implementation.

On this topic, Dave talked about how useful it is to optimize for learning - rather than optimize purely for initial value-creation of each data product - in a data mesh implementation. With the enablement teams bringing learning back to improve the core platform, we can implement friction-reducing enhancements like sensible defaults and starter kits/templates. Focus on efficiently learning.

When designing/creating a mesh data product, Dave recommends a customer-led approach. There should be a consumer with a specific use-case as the reason to create a data product. A mesh data product should be a valuable representation of the domain through data. But the operational and analytical planes naturally diverge unless we evolve the analytical to match the new reality in the operational. But we don't have a great way of evolving that analytical without data consumer disruption.

Per Dave, creating a "thin slice" is one way to help maintain representing your domains via mesh data products. Dave recommends you resist the pull - and resulting woes - of past data ways of trying to tackle too much at once - so thin slicing. Get to value delivery and feedback quickly. A thinner slice has a reduced scope so you will likely not have as much difficulty maintaining that singular data product. But Scott wanted to make sure people understand that the micromicroservices model - as in small even for microservices - was kind of a disaster so be careful to not slice too thinly - you don't want to have way too many data products - it can also make data discovery more difficult. It's all tradeoffs in the end :) And you can use fitness functions to measure if you are making the right tradeoffs too!

Dave's LinkedIn: https://www.linkedin.com/in/davidcolls/

Zhamak's data mesh book: https://www.oreilly.com/library/view/data-mesh/9781492092384/

Building Evolutionary Architecture book: https://www.oreilly.com/library/view/building-evolutionary-architectures/9781491986356/

Team Topologies book: https://teamtopologies.com/book

The Agile Triangle regarding helping you decide how thin to slice: https://www.projectmanagement.com/blog/blogPostingView.cfm?blogPostingID=5325&thisPageURL=/blog-post/5325/The-Agile-Triangle#_=_

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#94 Data Mesh Therapy - Come Vent and Chat - Mesh Musings 20 Jun 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#93 Empower to the People: Data Collaboration and Observability at Enterprise Scale - Interview w/ Jay Sen Jun 26, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Jay's LinkedIn: https://www.linkedin.com/in/jaysen2/

Posts by Jay:

Next-Gen Data Movement Platform at PayPal: https://medium.com/paypal-tech/next-gen-data-movement-platform-at-paypal-100f70a7a6b

How PayPal Moves Secure and Encrypted Data Across Security Zones: https://medium.com/paypal-tech/how-paypal-moves-secure-and-encrypted-data-across-security-zones-10010c1788ce

The Evolution of Data-Movement Systems: https://jaysen99.medium.com/evolution-of-data-movement-f12614d6e9de

In this episode, Scott interviewed Jay Sen, Data Platforms & Domain Expert/Builder and OSS Committer. While Jay currently works at PayPal, he was only representing his own view points.

Some key takeaways/thoughts from Jay's view:

When you get to a certain scale, any central team should focus on, as Jay said, "Empower people, don't try do their jobs." That's how you build towards scale and maintain flexibility - your centralized team likely won't become a bottleneck if they aren't making decisions on behalf of other teams.
To actually empower other teams, dig into the actual business need and work backwards to a solution that can solve that. If there is a solution already in place that isn't working any more, look to find ways to augment that rather than trying to replace or reinvent the wheel.
Self-service is a slippery slope - it often solves the immediate problem of time to market but also creates next level challenges. A big issue is that when you remove the friction to data access, you are throwing challenge of finding right data on consumers plate.
Data contracts are great when everybody aligns on a single contract and there are enough tools to support the contracts. But they also create a proliferation of data to enforce the contracts required by multiple consumers - thus, they often don't survive the real world.
The data catalog space is finally getting some needed attention. But there are still a myriad of issues that need solving. Will those be solved by technology or by leveraging a "data concierge" remains to be seen.
It's insanely easy to overspend in the cloud. Everyone is vaguely aware but cost should be part of every important architectural discussion. You can drive business value but it absolutely must also be focused on the cost as return on investment is far more important than simply return.

Jay took a few lessons from working on a central services team in a company of ~200 people. Having a centralized team was doable at first but as the org scaled, it quickly got complicated. As a centralized team, it's very easy to become a bottleneck but Jay learned a lesson that has continued to help in his subsequent roles: "empower people, don't do their jobs." Focus on reducing the friction to others doing their work instead of doing it for them.

Easier said than done so how do you empower people? Per Jay, you must understand the business aspect and what the requestor actually needs. That isn't really going to get communicated well in a ticket most times so you should have a high context information exchange to take what they need and convert it into a workable solution. And often, there is already a solution in place but it's just not handling the job anymore. So you want to consider if you should solve the same issues in a better way. It's much easier to do a greenfield deploy but brownfield is an inevitable facet of enterprise data work.

Per Jay, a few good things to remember: 1) frameworks and technology come and go but the concepts are the things that stick around. Focus on solving issues by leveraging technology and frameworks, not relying on them. 2) When working in data, you can't favor data producers or consumers over the other. It is easy for many to align with consumers but all stakeholders need assistance. 3) Beware the cool and trendy tech or approaches. Yes, it's funny to say that in a data mesh podcast but often, engineers just want to play with cool things and take on gnarly challenges. Stay focused on the business issues and value. 4) Trust in data takes a long time to build and seconds to break in Jay's experience.

For Jay, self-service can be a very slippery slope. It often creates more issues than it solves. But it feels like it solves something and that is the allure. Part of the issue of self-serve is that it removes the necessary governance in order to reduce friction to accessing data. So you can get to data but don't have the understanding necessary to actually use it properly. It's also crucial to embed governance into data access if you are going for something like self-serve.

There is often a high cost to adopting any technology in Jay's view. Not just initial adoption but tool stewardship. But it's typically much higher for the latest technology. Make sure if you are looking at an immature solution that you're focused on solving the right problems.

Jay shared that data contracts are evolving in a good direction. But, they are still not addressing all the challenges people want data contracts to handle. They can be really good at making sure people understand their responsibilities. As Emily Gorcenski noted in her episode as well, you must drive to meaningful conversations between producers and consumers to define quality very specifically as well as other SLOs. In Jay's view, contracts work in an ideal world of one-to-one communication between domains but often, there are multiple parties from each side that view things slightly differently. So the contracts rarely fully cover all use cases and can be, at best, a good conversation point for negotiations.

Jay is excited that the data catalog space is getting some very necessary attention. Every company of any size is now dealing with petabyte level of data so organizing it is becoming a major necessity. There are quite a few challenges left to tackle: 1) automated dataset discovery and definitions; 2) naming conventions still aren't standardized; 3) over-reliance on auto-documentation when human input is required; 4) how do we build trust? 5) how can we empower the data applications? 6) how do we deal with the trapped metadata problem? etc. Jay believes that the catalog must have the understanding not just of what the dataset is but also why it exists.

When asked about whether he thought systems or people should be the focus in enabling data discovery, Jay said to focus on systems to make the onboarding experience the best it can be. That will make it easiest for people as you scale. Scott disagrees and believes a "Data Concierge" role will serve organizations well - but that exceedingly few organizations will actually create and leverage such a role.

Jay then shared his thoughts on understanding, containing, and preventing unnecessary costs in data management in the cloud. It's quite important as it is VERY easy to spend a lot when you move to the cloud. Scott agreed as he previously managed AWS costs for a public company and saw that first hand. Jay pointed out that it's often a bad idea to do a one-to-one mapping of what you were doing on-prem to the cloud. The cost structure is often very different and it can cost you a fortune. But rearchitecting also has a cost. Evaluating cost should play a role in every part of data work if you really want to drive good business outcomes.

In wrapping up, Jay reiterated that technology needs to solve real business problems, not just be cool. Really consider the long term costs of adopting a new solution.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of June 26, 2022 - Data Mesh Radio Jun 26, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#92 Good Data Mesh Governance Through Empathy and Partnership - Interview w/ Jay Como and Elizabeth Calloway Jun 23, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Liz's LinkedIn: https://www.linkedin.com/in/elizabeth-negrotti-calloway/

Jay's LinkedIn: https://www.linkedin.com/in/jaycomoiii/

In this episode, Scott interviewed Jay Como, Head of Finance Data, and Elizabeth (Liz) Calloway, Director of Finance Data Products at Silicon Valley Bank. To be clear, they were only representing their own views and experiences.

Some key takeaways/thoughts from the conversation:

The governance team should "wear them down with empathy." Take the time to share your context, learn their context, make them feel seen and heard. That will get them to see you as a partner and good governance is truly about partnering, not mandating or being a gate/hurdle to get past.
Great governance is the pathway to great data. Great data leads to great decisions which lead to great outcomes. Share that path to great outcomes so people can see a clear answer to "why are we doing this?" Governance isn't just risk mitigation, it can be a significant - if almost always hidden/secret - value driver.
To drive governance buy-in from data producers, again, lead with empathy. Let them in on the "why" - why does this matter? What is the business value? How can this benefit them?
"Help me help you" is a good approach to talking to internal teams about data governance. You are there to drive value for them, take work off their plates when appropriate.
You can further drive buy-in through helping teams get to quick wins. While the long-term is obviously important, incremental value-add is better than a big bang approach.
Provide a constant stream of value, including executing on where you are helping, will drive teams to want to work with the central governance function. Drive value and the buy-in naturally comes with it.
Good data governance is necessary to avoid fees, fines, and the huge revenue/business impact bad data can have. But don't use those as a boogeyman, don't use fear to sell good data governance.
It's easy for a centralized governance team to become a bottleneck. Focus on not solving all the problems for teams but being there to help when they need it. You are the backstop, not the stop sign. Be the support, not the roadblock.
To prevent governance in general from being a bottleneck, you must have flexibility and pragmatism. If exceptions to requirements are necessary, then those exceptions are often a valid response to other constraints/pressures. But be very explicit about the reason and type of exceptions and also very explicit about the expectations for if/when those exceptions will be remediated.
Good governance is about incremental improvements, not trying to do everything as a big bang. Set expectations, move forward. Fix for today and prevent the same issue in the future.

Jay started the discussion talking about the concept of governance-as-a-service - in other words, providing a service to internal stakeholders instead of a mandate of comply or else. And while governance may not be the most "sexy" aspect of data, it certainly is one of the most crucial.

For Jay, without good governance, data is often not nearly as correct and clean as it could be so internal stakeholders aren't making as good of decisions as they could. He laid out a simple framework of great data leads to great decisions which lead to great outcomes. There obviously is a law of diminishing returns but find the line where the juice isn't worth the squeeze.

As for driving buy-in internally, Jay recommends to "wear people down with empathy". And be ready to repeat yourself, maybe with small variations until you find what resonates, to be able to make change. You don't often win them over with a single conversation or presentation.

Liz started off by pointing to the fact that it's quite easy to drive buy-in that good data governance provides value for data consumers. It's driving that buy-in upstream that becomes a lot more difficult...

To get moving on that buy-in, Liz recommends flipping the script and changing the narrative and dialogue. It isn't driving a mandate, it's talking about what benefit this has to them and the organization. It's not governance for the sake of it! That way, they are more open to guiding or shepherding them towards providing good data. So in all your communications, lead with empathy.

Jay discussed how they are taking on the really hard parts of governance from their business partners. That way, those business counterparts will see the governance team as a legitimate partner, not a dreaded gatekeeper. Provide them with a constant stream of valuable insight and work - decks, plans/roadmaps, etc. - and then execute on what you say you will. That will drive buy-in.

Jay also mentioned that acting like you are a highly regulated entity even if you aren't will set you down a good path to understanding and protecting your data. It's an investment but a big risk mitigation and potential value driver.

Liz talked about while governance is not a direct revenue stream - even if they add value to data, driving better outcomes - it is often also a major cost avoidance. Obviously for a highly regulated industry like banking, that can mean fees and fines. But many people are also pointing things like Unity Software's $110 million negative revenue impact from ingesting bad data, announced in mid May 2022.

It's not about driving fear for Liz. And talking about how bad data has a negative impact on the business feels basic or a bit obvious for her. But having the conversations, showing how better governance drives better outcomes, executing on the tough parts the governance team takes on, all that will drive buy-in.

Scott asked about how, if the governance team is taking work on from other teams, do they avoid becoming a bottleneck. Jay first off acknowledged it's a real possibility and one to specifically try to avoid. One thing that has worked for him historically is to play a neutral party and extract the context so both sides understand the other instead of battle each other. Make everyone feel seen and heard rather than solving all the problems for them. But fixing issues where possible, not so teams are reliant on you but see you as the partner, has worked well for him to date.

Jay discussed a few ways to prevent general bottlenecks in the governance process. In data, we often take on tech debt unintentionally or sweep it under the rug. But Jay recommends being flexible with requirements BUT calling out very specifically when, why, and how you are waiving requirements in that specific situation. It's not being lax, it's being pragmatic that not everything has to be perfect as long as all parties are aware and there is a mitigation plan in place. Being explicit about exceptions and expectations means you can move much quicker. Be flexible but realistic.

Speaking of being realistic, Liz talked about how there is no magic wand to suddenly fix governance challenges. And things are never going to be in a perfect state. Good governance is about incremental improvements, providing roadmaps and executing on the roadmaps as a partner - and make your business partners feel heard along the way. Things change and being flexible is crucial. How can you fix the problem of today and then put processes in place to prevent that problem in the future? Nothing is black and white.

Jay quoted Jerry McGuire, "Help me help you!" The governance team is all about enabling other teams to drive value and prevent issues. So drive a perception of being the helpful team. It's also quite difficult to get "your" plan implemented - it's much easier to get "our" plan implemented.

In the new school data governance, as opposed to the old school, Jay believes we need to drive people towards caring about data governance at all. It can be a value-add, not just a hassle. Both Liz and Jay emphasized you can prove data governance as a value-add by helping teams get to quick wins - show the value. You want to balance that out with some long-term wins, not only the short, but it's still another endorsement for driving to quick wins as Joe Reis emphasized in his episode.

Jay and Liz wrapped up by talking about the need for a great team and to drive forward with empathy. You need to work as partners with your internal constituents. Take some work off their plate to make things easier for them so you can drive more value together.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#91 The Case of the Missing Data Mesh Zealots - A Mesh Mystery! - Mesh Musings 19 Jun 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#90 Sharing Data Reliably in Hyperscale Mode - Interview w/ Björn Smedman Jun 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Björn's LinkedIn: https://www.linkedin.com/in/bjornsmedman/

In this episode, Scott interviewed Björn Smedman, Engineering Manager at Communication Platform-as-a-Service (CPaaS) company Sinch.

Some interesting thoughts or takeaways:

A good indicator for when decentralizing your data team might make sense is the cognitive load of a centralized data team. How many systems - including a measure of how complex - are they managing? How much of their time is spent in meetings, especially trying to understand context/requests? Is there starting to be combative prioritization from multiple domains?
It can be very beneficial and scalable to apply data mesh principles to non analytical use cases, especially sharing data for application purposes.
It is still often difficult to prioritize creating a data product for machine learning without knowing the business value of the ML model. But the ML team needs the data first before they can figure out the business value of the ML model. You have to make speculative bets.
If you see the data platform team start to dig into the semantics of a use case, that's a red flag that people are trying to leverage them as a data team. And while you want a centralized data platform team, you probably don't want them to become a centralized data team.

Since December 2020, Sinch raised nearly $2 billion USD. With this funding, they have made a number of sizeable acquisitions, with the company growing from 500 employees to over 3,000 in about a year. This has led to some interesting challenges in sharing data in a hyper-scaling environment.

Per Björn, data is a very key part of Sinch's plans for growth. Sinch's operational systems are often very transactional, as some product lines can process tens of thousands of monetary transactions a second, so data that might be typically shared on the operational plane in other companies is shared on the data plane lest the operational data stores deal with billions of events, making the data challenges even more complex than for most organizations. Then add in the regulatory requirements of telecom.

Björn helped lead the move to decentralizing the data team. When Björn joined, the central data team organization was 4 teams and 25 people. The data function was previously centralized and that was becoming a bottleneck, even for the legacy business. Now that the company had acquired a number of other sizeable companies, that central data team setup clearly wasn't going to scale. The company reorganized around business units and started to build data and analytics teams inside each BU.

For Björn, who started in December 2021 just as Sinch started acquiring new businesses, the central data team was clearly not going to be able to meet the needs of this new organization that was about 8x larger than a year earlier. There was too much cognitive load on the team, especially trying to understand the product lines of five distinct business units, many of which were entirely new to the company.

Björn gave a few good indicators of what to look for when considering if you should decentralize your data team. A big one is team cognitive load. Cognitive overload can take many forms: how many systems - especially complex ones - are your teams managing? Do they really deeply understand the systems? Do they need to deeply understand them to work with them? How many competencies does the team and each individual on the team need to have to do their day-to-day work? What percent of time is spent in meetings, especially follow-up meetings? One that wasn't mentioned is request turnaround time lengthening.

Sinch had a strong signal that they should move towards a decentralized data team approach, per Björn. The business announced in early February 2022 it was organizing itself into five distinct business units. The business units started to build data and analytics capabilities internally but there would be a very distinct need for teams to share data with each other so a common self-serve data platform was necessary. If they didn't have a common platform, each business unit would need to do custom integrations with the other BUs, and that could be four custom integration points per BU. And each one probably would not be bi-directional so they might need eight per BU, four for sharing data out and four for ingesting data. Obviously, that would be a pretty bad situation.

Regarding machine learning, Björn mentioned how difficult it can be to prioritize - there is a chicken and egg issue here: before data producing teams are willing to do the work to create a data product that will feed an ML model, they need to know how valuable or what will be the result of the ML model. But the ML teams need to get access to the data first to determine how valuable the data is before assigning a value. Thus there is a need for speculative bets, and those are hard to prioritize.

Björn worked with the central data platform team to build out a common data platform with a data lake, data warehouse, and streaming capabilities. They are using as many open standards as possible as it prevents lock-in and also often means more integrations are available. The goal of the platform is to make it easy to do the necessary data engineering work for every business unit. But Björn mentioned it is important to prevent the data platform team from becoming another data team - if you are seeing your data platform team start to dig into the semantics of a use case, that's a red flag.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of June 19, 2022 - Data Mesh Radio Jun 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#89 Flexibility is Your Friend in Delivering Buy-In; But Be as Rigid as You Can - Interview w/ Luca Paganelli Jun 16, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Luca's LinkedIn: https://www.linkedin.com/in/paganelliluca/

In this episode, Scott interviewed Luca Paganelli, Data Architect at the Italian utility Gruppo HERA.

To start, some interesting points and/or key takeaways and questions:

Introducing new concepts and ways of working around data slowly - not looking to make a hard shift - has worked well. When Gruppo HERA debuted their new data strategy manifesto, none of it was a surprise and it was already relatively in-line with the way many were talking about data and moving forward on their data journeys.
HERA's Data, Analytics, and Intelligence Automation (DAIA) team is not forcing domains to comply with HERA's data mesh-inspired guidelines but instead working with them closely to help the domains achieve their data related goals - delivering the "right thing". That gives the DAIA team strong influence to direct the domains' approach to data work without pushback and gives domains better confidence in the guidelines and can mitigate analysis-paralysis risk. This lack of rigidity and strong rules created a better sociotechnical environment to innovate but it can mean nothing really feels standardized because the domains can still choose to go a different direction.
The paradigm-shift was initially "steep" for both IT and domain owners. But domain owners realized how much better they could serve themselves and external data consumers if they took over more data ownership. IT was afraid to give up control but started to buy in to the leverage and expertise they can provide by empowering the business domains to do great things with their data.
A concern with not having broad standardization is bespoke solutions so it is hard to create broad reuse. There is also a challenge of people not being sure how much they can trust the data products. The DAIA team believes the tradeoff is worth it to drive initial buy-in with domain owners.
Defining data products has been a struggle. There is a chicken and egg issue of 1) needing to understand who from the business should be involved in designing a data product but 2) data domains must be discovered to know who are the subject matter experts from the business to involve.
For HERA, they are looking for data products to first serve their domain owners. This can be a slippery slope as domains may have valuable information but that isn't useful for them to analyze for their own purposes. So then other domains can't get to that data. But getting domains to freely share their data is a common incentivization problem.
Purely technical focused data products will probably not serve demand. We need to focus on sharing information - what is the data saying, what is it about? Information is more than just the 1s and 0s of data.

Gruppo HERA had to develop a sophisticated and reliable way to do their reporting to regulators but had not focused nearly as much on their internal data and analytics. But a progressively larger number of experimentations (spanning BI to AI) emerged where data started being used to drive the company. About 2.5 years ago, they developed a new team around data, analytics, and intelligence automation (DAIA) to start to rectify that and bring their data and analytics up to the same level as their regulatory reporting. As Luca said, they "were ready to scale data governance".

One key change, per Luca, was the business had embraced a digital workplace program so the teams were able to create small-scale applications to fill gaps where business processes were not yet digitized. While small scale apps could be not scalable in the long-run, it still gave people a good idea of the benefits of moving to a more digital or data native approach.

Thus far, the most important aspect of change management around data for HERA - at least from Luca's view - has been not making the DAIA guidelines mandatory for domains while helping domains understand what good looks like. This has meant better conversations where the DAIA team can focus on listening and responding to issues domains have instead of enforcing a rigid set of rules.

Luca's team has seen a lot of success working with teams to deliver the "right thing" but with a lot of flexibility in how that is achieved. This has meant the DAIA team can make people feel seen and heard, which gives the team a good way to influence direction. The current risk or challenge from that is the large variety of quality of data products; but overall, the DAIA team views it as a success, mostly owed to not being overly rigid. They have created a better sociotechnical environment to innovate.

Patience is a crucial part of the DAIA team's long term strategy, per Luca. They are helping domains address their current needs all while constantly rearticulating the overall vision. They are also helping teams by continuing to support active projects so there can be a good transition to new ways of working. The new data strategy manifesto wasn't like a bolt of lightning, the DAIA team had been working with domains to move them more towards the manifesto's approach so it already aligned with the domains' ways of thinking.

Speaking of technical and organizational challenges, Luca mentioned that historically, data ownership was mostly a technical thing and was owned by a centralized IT team. Domains were at most owning very high level concepts and IT owned the rest. So, when they moved to their new approach, domain owners often reacted with fear at first. But once they got over the initial fear, they saw the power that owning their data has to make their data a great resource to the company. IT also initially reacted with fear but started to buy in when they saw how they could empower the business users. This process is also progressively moving things out of the siloed ways of working to cross-functional, Agile teams.

Scott expressed his concerns about what a too flexible approach means for reuse, first on the data side if there are a wide variety of data products, and second about how to find reusable patterns which are necessary to scale a data mesh implementation.

Luca reiterated that the DAIA team is neither IT nor business so they have to partner with both sides to get things done. Their strategy is to have the guidelines to make the standardized way the easy pathway but to not force the domain owners to comply. They partner with them and give them guidance but only try to influence them to do things the right way. Per Luca, by laying out the tradeoffs in an honest way, you can help the domain owners understand why you recommend a certain way. By letting people decide, they will commit more to making it work.

As for defining data products, Luca mentioned how it has been a struggle internally. Many organizations implementing data mesh are struggling with this, including the very high level questions of how big should data products be and how many should we have. Should they mirror the source system? - In general, this is an anti-pattern - How do they encapsulate the subject matter expertise in the domain into their data products? There is a chicken and egg issue of needing to understand who from the business should be involved in designing a data product but they need to design the data product first before knowing who are the subject matter experts in that domain.

HERA is using two different types of data products, source aligned and consumer aligned. Consumer aligned - which they call Consumption Data Products - are designed to serve a specific use case. These are fit-for-purpose data products and there is often a working backwards process to figure out which domains need to deliver what once the use case is established. There is also a focus on making sure to limit the scope of a data product so it doesn't get too complex or complicated to create or maintain. The source aligned data products - which they call Domain Data Products - are then built initially to "power" CDPs (use-cases) but are also designed to be more general purpose.

For Luca, data products must first serve the domain owner as it is hard to find domains that are willing to be so altruistic, they will create data products simply to share with other teams. This can be a slippery slope as there are likely many use cases where a data product or even a small part of a source-aligned data product is not useful to the data-owning domain but is extremely useful to other domains. Incentivization can be very difficult though.

When it comes to Domain Driven Design (DDD), when you first start to share the definitions of domains, many people create an extremely complex pictures of what a domain is in Luca's experience. He recommends trying your best guess at domains and moving forward, not getting overly exact. It's okay to make some initial guesses and work with the domain to define the boundaries. He also mentioned that data products that are purely technical solutions won't satisfy the demand for information by the data product owner so focus on delivering a complete product, not just data, but the information of what it is you're sharing.

Luca wrapped up with some thoughts about how crucial it is work on the organizational operating model, to try to embrace domain driven design for data, and to be as rigid as the organization can handle in your guidelines - too rigid of guidelines can be seen as regulatory without any value so start less rigid than you'd like. Domains can see the rigid way as creating no value so they will pushback and often deliver nothing. "Perfect nothing is still nothing" as Luca said.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#88 Data Engineering and Data Engineers' Future in Data Mesh - Interview w/ Joe Reis Jun 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Links:

Joe's LinkedIn: https://www.linkedin.com/in/josephreis/

Ternary Data Website: https://www.ternarydata.com/

Monday Morning Data Chat: https://anchor.fm/ternary-data

Joe and Matthew Housley's interview with Zhamak Dehghani: https://www.linkedin.com/video/event/urn:li:ugcPost:6915063013410582528/

Joe's upcoming book, "Fundamentals of Data Engineering": https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/

In this episode, Scott interviewed Joe Reis, CEO/Co-Founder of data consultancy Ternary Data, Co-Host of the Monday Morning Data Chat, and author of the upcoming book Fundamentals of Data Engineering.

Some key points or takeaways specifically from Joe's point of view (not necessarily those of the podcast):

Find quick, high-value wins. Too often people focus on the big wins and those become overly complicated and end up in failure.
Most software engineers don't understand data well enough to be data product developers in data mesh, at least yet.
Data mesh is a polarizing topic. And that makes sense as it is pushing boundaries. Many hope it can come to fruition but it is a bit of a utopian view.
The future of data engineering is to move past managing pipelines to much higher-value work.
Speed to achieving wins with data - with a clear return on investment and trust - is the first thing you should focus on. Get this right and you can have the "luxury" of building great data products.

Joe started by discussing the kind of nebulous area within software engineering and data that data engineering has always played - sit between the source systems and the data output, converting the data in the source systems into something consumable for data users. Previously, that was mostly about making sure reports got pushed through and you hoped people derived insights. Now it's more about pipelines. But the way we store information in source systems, it is not in the format or shape we need for analytical purposes. So there needs to be a go-between.

A big trend in data engineering currently for Joe is the abstraction of tooling. Some of that can be good - makes people more productive - or bad - means it's harder to understand what is actually happening under the covers. But for Joe, it's probably worth it to use the abstractions as they are able to do the heavy lifting and data engineers can focus on the higher value work. We might be coming to the end of the "pipeline monkey" era of data engineering so we can shift more focus to the data output, DataOps, orchestration, security, etc.

For Joe, the biggest value-add the data engineering team can have is getting wins quickly. When asked about speed to returns versus repeatability, Joe said that the speed is more important, especially when you are trying to prove out the value of your data team. Trust is crucial, so you have to be careful to not move too fast, but trying to do big-bang projects is often a recipe for failure in his view.

When asked what could be the signs an organization is ready to implement data mesh, Joe mentioned that if an organization is already seeing "wins" with data across a number of teams/domains, that's a very good sign. But you can't only have a few teams getting those wins as that means the overall organization data maturity is still probably low.

Joe made a good point about how polarizing data mesh can be. When he speaks with some organizations, there are a few leaders who simply reject the idea outright. But many also simply don't see data mesh as ever being possible specifically in their organization. And that is probably true - low sharing / low empathy organizations need cultural change BEFORE trying to implement data mesh or the implementation will likely fail. Others, including Joe, see data mesh as a bit of a utopia vision - "imagine a world where..." - and that's pretty common. But Joe made two good points there: 1) if it were a safe concept, it would already be obvious - and so these large change concepts cause concern; and 2) just because we can't necessarily achieve the ideal, we can strive for a goal e.g. "being a good person" - being a better person is still a win even if you don't become the best person possible, right?

A recurring theme throughout the conversation was the need for speed relative to data. Identifying and then executing on quick wins is crucial for data teams in Joe's view. So first, the data teams need to learn how to identify those opportunities to build momentum around the data organization as a profit and innovation center instead of a cost center. As Joe said, "you have to see problems before you can fix them".

When working with software engineers to teach them data engineering skillsets, Joe has seen the software engineers are often easily able to pick up a lot of the mechanisms used in data engineering, e.g. managing the pipelines themselves. But they have little understanding of what the data consumers want. So it might be too early for most organizations to have their software engineers as the main data product developers for data mesh just yet. And Joe regularly sees that software engineers both don't understand data, as stated above, but also often don't care to either. It can be easier to teach data analysts and data scientists data engineering because they understand what data consumers really need.

Wrapping up, Joe again circled back on the need to find high value wins quickly in data. He recommends to not get too complicated, look for the small wins. And to look at what you want to produce and somewhat work backward from there.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of June 12, 2022 - Data Mesh Radio Jun 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#87 Choosing Tech for the Now and Future and Potential Woes of Decentralizing Data Teams - Interview w/ Jesse Anderson Jun 09, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Relevant Links:

Jesse's Data Teams Book: https://www.amazon.com/Data-Teams-Management-Successful-Data-Focused-ebook/dp/B08JLFTPBV

Big Data Institute website: https://www.bigdatainstitute.io/

Data Dream Team podcast: https://sodapodcast.libsyn.com/site

Jesse's LinkedIn: https://www.linkedin.com/in/jessetanderson/

In this episode, Scott interviewed Jesse Anderson, Managing Director at consulting company Big Data Institute, host of the Data Dream Team podcast, and author of 3 books, most recently Data Teams.

To start, a few takeaways from Jesse's perspective on the choosing technology side:

You should make sure you have the right team in place to make good technology decisions - the team needs to be in place first
Before selecting any technology, it's crucial to understand what you are trying to accomplish. And to understand that the technology will provide help in addressing the challenge but won't solve anything itself
Focus on: is this the right tool or solution for us now and in the future? What is the roadmap and vibrancy of the solution?
"Technology must earn its keep", meaning you should understand the total cost of ownership and what is your expected return on investment
Data tooling cycles are probably going to be 10 years at the most - prepare for obsolescence so you aren't overly reliant on any one technology

And some takeaways from Jesse's point of view on decentralizing data teams:

Currently, software engineers aren't ready to be data product developers so you'd need embedded data engineers to handle creating and maintaining data products in data mesh
But many data engineers are not willing to be embedded into domains
Managing the dotted line versus solid line of reporting between a functional team and the domain is very difficult
There are a number of cracks where crucial data can fall into and fail to find a good owner in a decentralized structure, especially aggregate data products

Jesse started the conversation on how important people are to getting things right with data, especially making technology decisions. The chicken and egg question is do you need to have the right people in place first or do you want to make technology decisions that will attract people. In Jesse's view, you need the right people in place first as they will be the ones to make the right decisions on technology selection.

The most important question for Jesse when selecting technology is what are you trying to accomplish with technology. If you don't focus on the target outcome, that is not going to work out well. And you should know, in general, what most of your use cases will be for the technology - use that to assess what is the right technology to choose.

Also, for Jesse, "technology must earn its keep". Just because you made a decision on using that technology at one point, it must continue to be of more value than its cost. And you want to strongly factor in your long-term total costs, as best as you can estimate then, when looking at adding a technology. This is important for build versus buy, can you continue to keep something running, is the long-term roadmap a match to your goals and vision, etc.

Jesse also pointed to how different data is to the operational side relative to technology cycles. Considering Hadoop, where Jesse focused in his time at Cloudera, 10 years - or even less - is realistic for how long data technologies might be around. Thinking in those cycles, you should think about where a technology is and where it is headed when choosing: what is the chance of obsolescence? How healthy is the project? You must have a longer-term vision, more than just does it solve our today problems.

You should consider how aggressive you will be in tech adoption, per Jesse. Will you be comfortable with making early bets? How can you set yourself up to be able to migrate away once technology is no longer a great fit for you? Data mesh can make it easier to wean off a technology as what you expose to data producers and data consumers is rarely the underlying tech instead of an interface.

Jesse talked about how right now, general software engineers / application developers are not ready or able to create good data products. One big issue is a lack of understanding about schema changes - on the one hand, you can't tell software engineers they can't make schema changes because that blocks application development but on the other, most software engineers do not understand the downstream impact of those schema changes. They are also, per Jesse, not well versed enough in how to store and share data about the domain to 1) maximize reuse and 2) create datasets that will be useful for analytics.

Aggregated domain ownership is one issue Jesse pointed to regarding decentralization of data teams - who owns these products? Do they need to be products? Another aspect is something that's run through many conversations on the podcast - if we give domains the authority to do whatever they want, won't that cause chaos? Probably. So establishing best practices and giving people a common platform to use and reusable frameworks is necessary to make something like data mesh work.

Another issue with team decentralization Jesse has is how to manage the career growth and happiness of data engineers. Many data engineers may not want to be embedded in domains. And do they follow best practices of the organization or, if the domain owner says do something quickly and not adhering to best practices, who do they listen to?

Jesse finished by saying all your data work should have a purpose. Every organization should ask if the data mesh is truly worth it for them, both now and in the future. It's okay to say not now. It's okay to say not ever.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#86 Data Product Documentation - A Primer? - Mesh Musings 18 Jun 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#85 The Move from Legacy to Leader in Data and Analytics - Interview w/ Immanuel Schweizer Jun 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Immanuel's LinkedIn: https://www.linkedin.com/in/immanuel-schweizer-17839242/

In this episode, Scott interviewed Immanuel Schweizer, the Data Officer for EMD Electronics.

Some interesting thoughts and questions from the conversation:

Good governance starts at data collection - what are ethical and compliant ways to collect data from the beginning? This points to intentionality around data use stretching into the application - what should you collect that might not be part of the day-to-day application function but that might might lead to generating insights that will be used to generate a better user experience? And what are the ethical concerns?
Should we initially create data products to serve specific use cases or should we focus on sharing data first and then shaping what people consume most into data products? EMD is approaching data products from a different angle than most, using the second approach.
When looking at data mesh, should you start with the high data maturity teams or work to pull everyone up to at least a decent baseline maturity level? If you work with the most mature teams, will their challenges really be applicable to the not-so-mature domains? Can you find good reuse patterns to scale your mesh implementation?
Domain owners are much more willing to share data if they understand use cases for how their data will be used and maintain control to prevent misuse. Reluctance comes from an incomplete picture causing concerns - the more visibility into how data can be and is being used, the more willing domain owners are to share. But understanding your end-to-end data supply chain is tough, especially to start.
How do you evaluate when to spend the time with a domain to get them data mesh ready? If you need a high value use case to justify spending time with that domain, are you leaving many domains behind? This ties to #2 and #3.
Set your target picture but be ready to adjust your target picture along the way. The world is ever changing, don't lock in to an expected target outcome.
Good data governance is about speeding up 1) access to and 2) usage of data.
EMD launched a data literacy program where the employees spend the majority of a 10 week timeframe learning about data and how to make use of it. For Immanuel, making things tangible relative to data makes people much less hesitant to explore and use data.
You should make using data a "part of the job" so it is tracked and part of the review process. Otherwise, you are missing out on a key incentive to leverage data.
How many people in your organization wish they could be leveraging data more often to make decisions? What's holding them back? Is it tooling, knowledge, incentivization, access, etc.?
How can we democratize insights? So much of insight generation is one-off, how do we make that scalable, shareable, and repeatable?

Per Immanuel, EMD's data mesh journey is not that typical in that they are still getting their arms around centralizing data in a constructive way. It was previously locked away in the domains. So, they are starting their data mesh - or decentralization - journey by centralizing data in a certain sense. Wannes Rosiers mentioned this at DPG Media as well. This enables breaking down silos and starting from common ways of working so there is more cohesiveness around centralized data sharing. There are some concerns from the domains about how do they maintain control to ensure compliant usage.

As the team learn how to put their data onto centralized infrastructure, Immanuel shared that they are simultaneously working on how they will hand more control back to domains. EMD is in the initial stages of their data maturity journey but they are mapping out how they plan to move forward with next steps. They are focused on giving data producers the visibility to how their data is used so the data domain owners can feel comfortable.

And Immanuel gave a good insight to those starting their data mesh journey: understanding the end-to-end data supply chain is really hard when you are just establishing that supply chain. And understanding that supply chain, how data is consumed downstream, is very important to giving the domains the visibility they need to feel comfortable sharing more and more of their data. Some are calling this data on the threshold - data might be data on the outside but isn't yet.

Immanuel and team are starting their data mesh journey by bringing a significant amount of data into a central data lake and then watching for data consumption patterns emerge. Then, when there is a use case that is worthwhile, that will get promoted to being data product worthy. Then they work backward to find the owners so they can ensure the upstream data production is actually managed as a product. Now, since data challenges aren't flagged to a centralized team, the data can be fixed where it should be fixed - upstream, at the source systems.

Based on that approach of getting access to data outside of data products, Immanuel mentioned how this could bring domains to the table sooner in a data mesh implementation. In his view, it is often quite expensive to get a domain on board and capable of sharing data like a product. So, if you require a specific, data consumer-driven use case or use cases before investing in that domain enough that they can share data, it can mean you only look for very large return use cases. Or that the initial cost of bringing a domain on to sharing on the mesh falls disproportionately on your early data products. In Immanuel's approach, the initial cost of sharing is much lower, and then use cases emerge to further justify work. So more domains share sooner. But it has the drawback of the domain not sharing their data intentionally upfront so it might not have as much of the domain context.

Immanuel mentioned that you can't centralize ownership of data quality and access and expect to scale. You really need to figure out how you distribute your data ownership appropriately. And that you can't rely on a data engineer in every domain, so how do you lower the bar to sharing?

Immanuel's approach to the big picture is to set a target picture and a north star but adjusting the expected target picture along the way. If you aren't flexible and aren't taking in new information and adjusting accordingly, are you really ready for the flexibility required to do data mesh?

So how is EMD approaching their data strategy? Per Immanuel, they started with a company-wide data strategy - again, setting that high-level target picture. Then, they started to measure data maturity across each domain. They had to answer should they move ahead with domains that are already high data maturity or try to level up the capabilities of less mature domains so they can participate in a data sharing economy.

They decided to focus on bringing all domains up to a certain maturity level - if you don't, it can mean issues scaling your approaches, per Immanuel. An example is that if one domain is doing MLOps with 10s of ML models in production and another domain is running everything off Excel and Access - or worse, PowerPoint - their needs may be completely different. The things you learn and ways of working you get from the very data mature team just won't translate well to immature domains so every domain will need specialized help. That just isn't scalable.

Immanuel mentioned how data governance can often create mixed feelings. He said that good data governance is like brakes in your car - brakes are there to allow you to go fast safely. Governance is not about overseeing every bit of data usage, it's about speeding up access and usage of data. Governance should be an enabling factor in a best-case scenario. And that is why federated governance can be so powerful - we give the control to the people most worried about data usage and give them the tooling/knowledge to own most of that governance.

EMD created a general data literacy campaign, making everyone aware of and on the same page about definitions for a lot of core concepts around data, analytics, use cases, etc. - making them all aware of the vocabulary. This was just the first step though.

For Immanuel, he has seen data literacy success come from programs that take people by the hand regarding data and make them use the systems - the workflows and mechanisms - especially around governance. Hence, they created a 10 week program for focusing on using data. He said it is looking successful because it makes things so much more tangible. People can understand the entire flow of data through the system and then they have a better idea of what they could do. It's only just wrapping up its first cohort of people but the attendees seem very excited about it.

In Immanuel's view, people are naturally curious around data. The issue with using data has been lack of tools and access to data in domains - sharing and leveraging data wasn't viewed as part of the typical job in most domains and it wasn't part of most people's KPIs or reviews. It is important to give them the right tools and the right incentives so people can and want to explore data and be curious. Pushing reports at people doesn't engage curiosity.

Who is this new data and analytics world for? For Immanuel, it's for the people who are already data curious but aren't leveraging data nearly as much and/or as scalably as they could be - we need to give them the tools to make working with data more scalable and sharable. Then, data becomes a topic that is tangible for everyone in the business - can we start replacing Excel and PowerPoint but still make it simple for people to explore the data?

Per Immanuel, part of moving towards scalable analytics requires us to unlock the one-off insight generation, make it so we can democratize the generated insights. There are too many instances of people generating good and useful insights that are just lost, only seen by that single person. And/or only seen that one time. How do we make it so there is an easy, happy path to sharing insights that are long-lived?

When asked about the initial ROI on data mesh and the big data literacy campaign to-date, Immanuel pointed to the main value thus far has been giving people the capability and encouragement to explore data. Exploration has allowed people to understand the organization and they've found the most reused data - and then focused their data work efforts on the places people are using the most. That prevents time spent on data products that aren't valued by consumers.

In wrapping up, a question Immanuel thinks it's important to ask: how do you define a happy place for your org and for each domain? No journey to a happy place will look alike. And no happy place will look alike either. What does a good state along the way look like? We don't need to be in a rush to get to the finish line.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of June 5, 2022 - Data Mesh Radio Jun 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#84 Driving to Better Data Outcomes with Collaborative Questions - Interview w/ Jean-Michel Coeur Jun 02, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Jean-Michel's LinkedIn: https://www.linkedin.com/in/jmcoeur/

In this episode, Scott interviewed Jean-Michel Coeur, who is the Head of the Data Practice at the consulting company Sourced Group.

Jean-Michel has developed a simple three question framework that works well with people asking for data, especially business counterparts. The questions typically lead to collaboration instead of confrontation and gets data consumers to share what they want to accomplish with the data instead of what is their request. It feels more like a friendly chat than an interrogation or "prove to me why this is worth my time". He also recommends following up each question with "the reason I am asking is..." to explain specifically you aren't pushing back, merely information gathering.

The three questions:

Do you know what this is for?
Do you know who is going to use it?
Do you know how they are going to use it?

Jean-Michel developed his three question framework after watching people struggle for years to properly request data and/or properly understand the use case of data consumers, often delivering solutions that did not meet business needs, wasting everyone's time. Oftentimes, the technical person wouldn't ask the right questions or they couldn't even get to the end data consumer so they didn't really understand the reasons for the data ask.

For Jean-Michel, the first question - Do you know what this is for? - helps to set the tone. It is not "why do you want this?", which often makes people defensive. He tells the person making the ask that with more context, his team can better understand how to make what they deliver better. And sometimes, the person making the request will realize they aren't really sure what it will be used for and can go back to the end user. A key is to not be a gatekeeper to the data, both in reality and perception.

The second question - Do you know who is going to use it? - starts to drive towards who will consume the data output and how - the use case is pretty important for delivering valuable data after all. For Jean-Michel, asking it in this way can often empower the person making the data request to lead the journey rather than undercutting them to get to the end user. It makes them part of the team. This is far better than the reaction he got previously when asking "who is the end user and what do they want?"

The final question - Do you know how they are going to use it? - means you can own the data user experience all the way through to the end consumption. Oftentimes, people deliver just the data into a warehouse or similar but if that is used in a dashboard, the dashboard is the user experience. This final question also helps data producers to suggest additional value-add features or to push back on some ill-defined requirements. E.g. "real-time" rarely actually means real-time. And Jean-Michel recommends understanding what you are delivering well enough to be able to answer the question of "okay, what does this mean?" After all, data projects/products are meant to deliver value and the value is informing business decisions with data.

And again, a crucial part of the conversation to add after each question is an explanation of why you are asking. This isn't pushback, it's information gathering. It's how product marketing and product management are done! If you come at these questions from an empathetic angle, you all have a better chance of a positive outcome.

The output from the three questions is also immensely helpful to the data producing team. It ensures there is a real use case with real business value. It validates there is really an intention to use the data - otherwise Jean-Michel will push back. And it sets up a collaborative approach where the data producing team can make good suggestions for additional value.

Jean-Michel talked about the fallacy of the data 'Field of Dreams' of "if you build it, they will come." The users will come. The value will come. Instead, work with intentionality. Any data you produce to share should be useful to someone. If you aren't sure, have the conversation with potential users. But just storing data has a cost and are you really going to be able to enrich it enough to make it useful in the future?

One thing Jean-Michel thinks many companies get wrong is that when building a data platform, there is a window of opportunity and many data infrastructure teams want to build everything themselves, pushing out your time to returns. The tendency to be tech-oriented often comes at the expense of user experience as well. Data infrastructure teams need to think about the primary reason for a data platform: collecting and producing data.

For Jean-Michel, the concept of a data product owner in data mesh might not extend far enough. If you are producing data for consumers, you should own the full user experience. If people are consuming from reports, you should own that report. Same for a dashboard. We have to be careful to not leave those less data literate behind while helping them up their data skills.

Jean-Michel wrapped the conversation by saying every time you are asked to deliver data, be prepared to answer the "so what" question from a data consumer/stakeholder. Be prepared with something to answer. You are a partner to the business, not just a "data report monkey" so add that value and build the relationship with your business counterpart.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#83 What is Your Actual Data TCO - Mesh Musings 17 May 30, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#82 A Better Way to Map Domains? & Searching "For" Data, Not Just "In" Data - Interview w/ Ole Olesen-Bagneux May 29, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Links:

Ole's Book (O'Reilly Early Release): https://www.oreilly.com/library/view/the-enterprise-data/9781492098706/

Ole's LinkedIn: https://www.linkedin.com/in/ole-olesen-bagneux-2b73449a/

Ole's Other Recommended Reading:

Zhamak Dehghani's Data Mesh book: https://www.oreilly.com/library/view/data-mesh/9781492092384/

Piethein Strengholt's Data Management at Scale book: https://www.oreilly.com/library/view/data-management-at/9781492054771/

The Elements of Knowledge Organization: https://link.springer.com/book/10.1007/978-3-319-09357-4

In this episode, Scott interviewed Ole Olesen-Bagneux, an Enterprise Architect who focuses on data at GN and the author of an upcoming book on data catalogs with O'Reilly. To be clear, Ole was only representing himself and not GN.

The two main topics, which are somewhat intertwined, were: 1) how can we better understand and handle the concept of a domain when discussing data; and 2) how can we build systems that better enable us to search "for" data, not just search "in" data that we know exists?

Some practical advice and general conclusions from Ole:

Leverage what the Library and Information Sciences discipline - which is centuries (millennia?) old - has formed around the domain concept. It will help you better dig into the actual business dealings of the domain first before trying to focus too much on the technical/software aspects. The software aspects hinder your initial domain mapping - especially in depth - and business context understanding when you start from a DDD perspective in data.
Spend a lot more time on enabling people to understand what data is available. We focus a lot on optimizing for searching "in" data, but we don't spend near enough time setting up our systems to allow people to search "for" data. To do that, work seriously on your metadata tools system and look for ways to harmonize data across those tools.

Ole started the conversation sharing his view that Domain Driven Design (DDD) has some shortcomings when used especially for data domain mapping and in general in data. In his view, DDD is overly tied to software engineering so there is too much of a technical bent to understanding and even mapping out domains. He recommends taking domain analysis and domain theory learnings from the Library and Information Sciences discipline and using that to start your domain mapping and then look to bring in DDD after you get a good initial understanding of your domains. DDD and domain analysis can work together harmoniously, they don't really contradict, but domain analysis focuses on the knowledge first instead of the technical first.

While Ole was inspired by Zhamak's book as well as the book by Piethein Strengholt, his believes domain analysis lowers the significant friction and often frustration organizations feel when trying to start doing DDD for data. Domain analysis digs much more into what the domain does and why instead of how the domain communicates via software. He believes that data mesh should focus more on the information sharing and less on the software and that DDD will overcomplicate your domain mapping.

For Ole, DDD is overly concerned with modeling domains into software but you need to get to a deeper understanding of your domains and organization first before focusing in on your model. It may be that you truly can't fully communicate your domain's context in a data model either and it's good to know that upfront and take steps to communicate in other ways, such as enhanced documentation.

Ole believes we focus too much on the data model and that often sends people down the path of overly technically-focused solutions. Other guests have mentioned that documentation around data sets and data products is often much more focused on the technical aspects, not actually describing the information represented by the data. How do we store our data so we can make it usable for humans, not just software? Make it searchable, findable, understandable, etc. At both the micro level - a dataset or data product - and the overall macro data mesh level.

Building a semantic model is at least as important as a data model for Ole. We need to again focus on that searchability but what different search capabilities do we need? A simple search experience on a keyword or two, browsing what data is available, complicated queries with filters, etc. Can we enable querying by data lineage, by relationships with a knowledge graph, across a domain, etc.? It's a different way of approaching data that is not similar to a data model.

For a functional data mesh, Ole believes there needs to be a big focus on the metadata layer. You need to enable data consumers to find what data exists in their "knowledge universe"; focus on also serving the use case of searching "for" data, not just the typical searching "in" data for a specific answer.

Possible ways forward on semantic knowledge sharing should come from the Library and Information Sciences space in Ole's view. They've been doing this for centuries in one way or another. We need to start thinking in a metadata way to move forward. And we need the industry to help develop better metadata tooling and for data practitioners to focus on what metadata tooling they have, working to harmonize data across those tools. It doesn't have to all happen at once either, we can work from domain to domain to focus on that metadata harmonization.

Ole finished the conversation talking about the fine balance between leveraging tooling and trying to do everything with tooling. There will be important roles for humans in the middle of knowledge sharing - whether they will be more consumer facing like a data concierge or more behind the scenes, we shall see, but Ole bets it will be the latter.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of May 29, 2022 - Data Mesh Radio May 29, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#81 Finding Useful and Repeatable Patterns for Data - Interview w/ Shane Gibson May 26, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Shane's LinkedIn: https://www.linkedin.com/in/shagility/

Shane's Twitter: @shagility / https://twitter.com/shagility

AgileData.io website: https://agiledata.io/

AgileData Way of Working: https://wow.agiledata.io/

Shane's Podcasts: https://agiledata.io/podcasts/

In this episode, Scott interviewed Shane Gibson, CPO/Co-Founder of AgileData.io and Agile Data Coach.

A few takeaways from Shane to start:

- Agile methodology is about finding patterns that might work, trying them out and deciding to iterate or toss out the pattern. It's going to be hard to directly apply software engineering patterns to data but we should look for inspiration there and then tweak them.

- Any time you look at a pattern you might want to adopt or evaluate if a pattern is working for you, ask yourself: will this/does this empower the team to work more effectively?

- Applying patterns is a bit of a squishy business. Get comfortable that you won't be able to exactly measure if something is working. But also have an end goal in mind for adopting a pattern - what are you trying to achieve and is this pattern likely to help you achieve that?

- Share your patterns to not only help others but to get feedback and maybe ideas to iterate your pattern further.

Shane's last 8 years have been about taking Agile practices and patterns and applying them to data as an Agile Data Coach. And those patterns required a lot of tweaks to make them work for data. A big learning from that work is that when applying patterns in Agile in general, and specifically in data, each organization - even each team - needs to test and tweak/iterate on patterns. And that patterns can start valuable, lose value, and then become valuable again. Shane gave the example of daily standups drive collaboration as a forcing function but then lose value when that collaboration becomes a standard team practice. If there is a disruption to the team where collaboration is no longer standard practice, daily standups could get value again. So how do we apply these Agile concepts to data?

Currently, Shane sees no real patterns emerging in the data mesh space - it is quite early as patterns often take 5-8 years to develop and data mesh is maybe 12 months in to even moderately broad adoption and is such a wide practice area, there are many practice areas that patterns will need to cover. But, that lack of patterns makes it quite hard for even those who want to be on the leading edge of implementing data mesh instead of the true bleeding edge - having to invent everything yourself is taxing work! So we need companies to really take existing patterns, iterate on them, and then tell the world what worked and what didn't. If people aren't sharing patterns, that's going to make it hard to adopt data mesh for many organizations.

Shane believes that it will likely be pretty hard for many organizations - or at least many parts of large organizations - to give application developers in the domain the responsibility of creating data products. If your domains aren't already quite technically capable in building software products, it's going to be very hard for them to really handle data needs. So looking at domains that are using large, out-of-the-box enterprise platforms or SaaS solutions instead of rolling their own software, will they really have the capability to manage data as a product? If their domains don't have the most complex of data, maybe? But if they do, are they really mature enough to handle it? A very valid question.

To really be agile, using Agile methodologies, you need to first adopt the Agile mindset and not just patterns and practices, per Shane. Agile is really about experimenting with a pattern and either iterating to make it better or throwing it out. It's not about being precious. As mentioned earlier, you should also throw out patterns that were effective and aren't helping you any more. You need to do the same at the team and organizational level if you are going to successfully implement something like data mesh. Your teams and your organization are like a living, changing, evolving organism - treat them as such.

A very important point Shane made is data mesh isn't solution - it needs to at most be a way of approaching your data and analytical challenges of organization but with a true purpose in mind. The purpose isn't implementing data mesh. The purpose is a business objective or challenge and data mesh is helping you tackle that. Also, data mesh is not the right solution for many organizations, especially smaller ones or ones that don't have highly complex data needs - those organizations should review data mesh and understand the principles and work towards them but your real challenge isn't the centralized team being a bottleneck so don't take on the pain of decentralizing to be hip and trendy.

For those who haven't really dealt with Agile, a "fun" potential learning, per Shane, is that there isn't really a great pattern for measuring if a certain pattern is working. Proving how well something is working is pretty impossible so a large part of it is feel - we chose this pattern to improve collaboration. Do we believe our collaboration has improved? If yes, great, let's try to iterate and improve a bit more. If no or our collaboration has even gone down, get rid of it!

Per Shane, when evaluating if you are effective in your Agile methodology, ask: does the organization empower this team to work effectively? You will probably need to look at this on a team by team basis and repeatedly ask this question over time. Trying to scale Agile to fit all teams in an organization is often an anti-pattern. And if you are in a hierarchical company, adopting Agile patterns alone is probably not going to change the way you work in the long-run, you need to break the hierarchies in some way.

For Shane, there is a big question that data mesh has yet to answer: can we really move the data production and ownership to the application developers? He thinks if we look at DevOps and how developers took on the necessary work for testing and CI/CD, we can. But then the even bigger question is how. How can we map the language of what needs to get done to the software engineering semantics?

For Shane, the idea of a proof of concept - PoC - is just broken. We need to rethink it entirely, especially for data mesh. What are you really trying to prove out? He believes there are typically two types of PoCs and most default to Type 1 when potential beneficiaries expect the output of Type 2. In Type 1 PoCs, you are out to prove a high-level hypothesis that has lots of uncertainty. It's about experimentation and doing it in a "quick and dirty" way that is not ready for production. But the output of Type 1 is all about proving out the hypothesis - not a production-ready result.

Type 2 is a minimum viable product or minimum valuable product - what can we strip away from our end goal to get to something that can be used and is - mostly - productionalizeable. Literally what is the minimum that is viable? It is about proving the capability to deliver and delivering something of value sooner. So ask yourself, what are you really trying to prove in your PoC?

Shane finished on three points:

- Empower your teams to change the way they work

- Stop vendor and methodology washing data mesh

- Regarding data mesh, share what patterns you are trying to adopt, why you chose them, and what is working/not working. Data mesh can only evolve to something really great if we work together and share more information.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#80 A Data Mesh Prisoner's Dilemma? - Sharing with the Community - Mesh Musings 16 May 23, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#79 A Data Success Secret Recipe: Comfort with Ambiguity and Change Management - Interview w/ Vincent Koc May 22, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Vincent's LinkedIn: https://www.linkedin.com/in/koconder/

In this episode, Scott interviewed Vincent Koc, Head of Data at the merchant platform company hipages.

To start with some big takeaways from Vincent:

If you aren't comfortable with an agile mindset and ambiguity, bleeding edge probably isn't for you - and that's okay!
You and your organization need to be comfortable with failing, learning, and then iterating
To get data mesh - or really any big change initiative in data - right you should focus on change management much more than you probably think
It ends up being the secret sauce or crucial lacking factor much more often than the tech
Think problem-specific, not technology specific
It's easy to over-engineer the problem - technologists want to technology
In general, consistency is key to achieving widespread success in data
One domain having a major success won't lead to broader org-wide success if you don't look to leverage reusability factors to make consistency across other domains easy - a bunch of great but non-consistent solutions doesn't add up to a valuable whole picture

For Vincent, every organization considering data mesh should ask if it is really the correct approach for them. Data mesh really isn't for a large subset of organizations, whether that is right now or even ever. If your organization doesn't have an appetite for change, it's going to be very tough to move towards data mesh. If you want to implement data mesh, he recommends embracing an agile methodology e.g. fast feedback and trial and error.

When thinking about splitting your data monolith into domains, Vincent recommends taking a lot of learnings from what works well in the microservices realm. You shouldn't decompose everything all at once - that just creates chaos. You can split out larger domains one by one and then figure out if you need to split them further when there is more value in doing so. Peel them off instead of a big bang approach.

Vincent believes that, in general, ~20% of your teams will consume ~80% of your data team's time and energy. There are a few ways to work with those teams to reduce that but it is also somewhat a fact of reality. Whether that is because those domains are more prominent, noisy, well loved, or for many other reasons. That data work disparity often leads to those areas being more data mature.

When discussing disparate data maturity, Vincent talked about the need to drive all domains that will participate in something like data mesh to at least a common base level of maturity. You have to have a relatively mature domain to be at what he referred to as "mesh-level capability". Domains that aren't at that capability will still need to rely more heavily on centralized data teams and capabilities as they improve their data maturity. And it's okay to work with them closely to up their maturity level - just telling them to catch up is probably not going to work, there will need to be a bit of hand-holding.

Vincent believes embedding data analysts - whether you call them data analysts, analytics engineers, or something else - into domains is crucial, especially if you are going to attempt to implement data mesh. They serve as the custodians of the data for the domain, whether that is data shared with others via a data product like in data mesh or for data the domain regularly consumes and uses from the domain and external sources. One point Vincent thinks is crucial to moving forward in data mesh is that those data custodians need to help disseminate data knowledge to software engineers and the organization needs to build tools and frameworks to make it easier for software engineers to own and manage data.

When asked about the balance between long and short-term planning, Vincent talked about with an agile methodology, you need to "cater to today" and not get overly focused on the long-term exact plan/roadmap. Things will change. Set yourself on a good path, set that North Star, and keep your ears and eyes open for the signals you need to change your plan. E.g. look at what a domain could become with the right direction and set it on that path to positive evolution as best as possible. But don't overly define the path.

Vincent talked about how crucial - and often overlooked or ignored - good change management is for data organizations. Now more than ever due to how fast the world and the data landscape is changing. It's crucial to break down changes into terms and/or actions that all constituents can understand. The organizational is much more important than the technology in most respects.

To drive buy-in , Vincent has seen giving people agency over their data work well. Giving domains the trust AND the tools/frameworks/resources to manage their data gets those business leaders/domain owners to come to the table quite often. But you can't just give them the responsibility without the additional help.

Vincent discussed some past approaches that he would do differently now. He focused too much on the technology and telling others exactly how something should work or look instead of working with them to drive to the outcome and let them figure out the right path - while helping them along the way. Give them the big picture and talk to the outcome rather than the technology.

To do that, Vincent talked about building out a pattern but not the whole picture; give them a defined enough idea of a good outcome but not overly defined. And build in the easy path or golden path boiler plates where it makes sense through templates but with extensibility - if things are too rigid, people will thrash against that. It is quite easy to over-engineer solutions - think problem-specific, not technology-specific.

Specific to data mesh, Vincent sees one of the big remaining questions out there is how to actually automate and decentralize governance, especial things like access control. There isn't much good specifics on how to set up security as code in a scalable way. And we need to think about security as a sliding scale of risk.

Vincent shared a few words of wisdom near the conclusion for data leaders. The first was how easy it is to see something like data mesh, get very excited, and try to do a whole lot of changes at once. Making mass changes causes instability - instead, think about where you can be more targeted in support of the long-term big picture. Don't boil the ocean. The second was get comfortable with ambiguity and think if your organization is aligned to accepting ambiguity - if you aren't comfortable with ambiguity, stay away from bleeding edge. It's called bleeding for a reason. You need the ability to try/test, fail, learn, and then iterate towards a better solution. If failure isn't allowed and maybe even celebrated, the bleeding edge is probably for braver souls.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of May 22, 2022 - Data Mesh Radio May 22, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#78 Minimum Viable Data Mesh? - Interview w/ Paul Andrew May 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Paul's data mesh blog series: https://mrpaulandrew.com/tag/data-mesh-vs-azure/

Paul's LinkedIn: https://www.linkedin.com/in/mrpaulandrew/

Paul's Twitter: @mrpaulandrew / https://twitter.com/mrpaulandrew

In this episode, Scott interviewed Paul Andrew, Technical Architect at Avanade and Microsoft Data Platform MVP.

Paul started by sharing his views on the chicken and egg problem of how much do you build out your data platform and when to support your data product creation and on-going operations. Is it after you've built a few data products? Entirely before? And how that discussion becomes even more in a brownfield deployment that already has existing requirements, expectations, and templates.

For Paul, delivering a single data mesh data product on its own is not all that valuable - if you are going to go to the expense of implementing data mesh, you need to be able to satisfy use cases that cross domains. And the greater value is in cross-domain interoperability, getting to a data product that wasn't possible before. And, you need to deliver the data platform alongside those first 2-3 data products, otherwise you create a very hard to support data asset, not really a data product.

When thinking about minimum viable data mesh, Paul views an approach leveraging DevOps and generally CI/CD - or Continuous Integration/Continuous Deliver - as very crucial. You need repeatability/reproducibility to really call something a data product.

In a brownfield deployment, Paul sees leveraging existing templates for security and infrastructure as code as the best path forward - supplement what you've already built to make it usable for your new approach. You've already built out your security and compliance model, make it into infrastructure as code to really reduce friction for new data products.

For Paul, being disciplined early in your data mesh journey is key. A proof of concept for data mesh is often only focused on the data set or table itself, not actually generating a data product and much less a minimum viable data mesh. It's pretty easy to put yourself in a very bad spot because taking that from proof of concept to actual production is going to be a very hard transition and telling users it will take weeks to months to productionalize is probably not going to go well. Be disciplined to go far enough to test out a minimum viable data mesh.

Paul emphasized the need for pragmatism in most aspects when implementing a data mesh. Really think about when to take on tech debt and do so with intention. When shouldn't we take on tech debt? And how do we pay down tech debt and when? There is a balance between getting it done and technical purity. How do we choose what features to sacrifice? What is the time-value to money aspect, or how much importance do we have on getting it done sooner rather than more completely? These are questions you'll need to ask repeatedly.

Similar to what previous guests mentioned, Paul is working to encourage and facilitate the data product marketing and discovery process - discussing with data consumers what they want, pie in the sky thinking. Then taking that and speaking with data producers and figuring out pragmatic approaches and what is simple to deliver. Is one aspect going to be very difficult? Go to the consumers and let them know it will delay delivery and they need to fund that aspect. Do they still want it? Use that back-and-forth discussion to drive negotiations to a valuable solution with less effort. Look for that return on investment. Be pragmatic!

Paul recommends making business value your general data mesh 'North Star'. Ask the pragmatic questions - so shift the data function from taking requests/requirements to leading those negotiations. Have the conversation of "Is this worth it? Who is going to pay for it? What is it worth to them?" As of now, Paul and team are still often functioning as the translator between data producers and data consumers.

But, when discussing the goal of getting out of the middleman/translator role, Paul pointed to a few signs that an organization is ready for producers and consumers to directly work with each other. Some aspects are general company culture, how data literate/capable are the execs, data platform maturity, etc. If you can mature your organization's approach and skill, you can move towards not needing a data translator.

Paul talked about how to think about your data mesh journey and different elements of it, even a data product, in a crawl, walk, run fashion. Think about your data products first and foremost as serving at least one specific purpose. Still create with reuse in mind but they should have a use-case to serve and can expand from there. At a mesh level, part of crawling is getting to a few standard interfaces for data products to use to communicate. At the data platform level, part of crawling is getting to a place where it is possible to publish new data sets but walking might be a significant reduction in friction to data product production. While this means that a minimum viable mesh is still a pretty high bar, you can get to a place that is comfortable with being at a crawling stage with a good roadmap towards walking and running. Done and good is better than perfect and the forever "just 3 more weeks".

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#77 Why DO Data Warehouse Fans Fear Data Mesh So Much? - Mesh Musings 15 May 16, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#76 A Skeptic's View of Data Mesh and Learning Your Data Product ABCs - Interview w/ Tim Gasper May 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Tim's Twitter: @TimGasper / https://twitter.com/TimGasper

Catalog & Cocktails page: https://data.world/podcasts/

Data.world blog content:

Do You Know Your Data Product ABCs? https://data.world/blog/data-product-abcs/

The Role of a Data Catalog in Data Mesh https://data.world/blog/data-catalog-data-mesh/

In this episode, Scott interviewed Tim Gasper, VP of Product at data.world and the co-host of the Catalog & Cocktails podcast.

They covered two main topics - 1) the skeptic's view of data mesh and 2) Tim's/the data.world team's "ABCs of Data Products" framework.

Skeptics have a few main pushbacks on data mesh in Tim's view. Tim listed the top 6 that he sees and then discussed them with Scott.

#1: Data mesh isn't for every organization depending on size, number of domains, data/problem space complexity, etc. Tim said this. Zhamak has said this. Most data mesh advocates/fans say this regularly. This is one of the myths of data mesh - that it's designed for everyone. Don't go to a decentralized data setup if you don't need to. Tim made the very good point that we need more conversations and better guidance on what to measure if centralization of your data team and processes is your actual challenge.

#2: Tooling doesn't exist - yet? - to make it easy for domains to easily take over data ownership. A big conceptual myth of data mesh is that it has to solve every data problem, even the most difficult, right out of the gate. Tim mentioned that your team needs to really think about self-service being about empowerment, not necessarily a single big red easy button. And your implementation will evolve - it MUST evolve. It's not easy yet and if your team isn't prepared to roll up their sleeves, it's okay to wait to implement.

#3: There shouldn't be anyone who "owns" the data. Tim made a really good point here on accountability to sharing your data versus the "fiefdom" model - where someone has complete control over how the data is used. Yes, someone shouldn't be able to prevent other domains from using data. But that's not at all in the spirit of data mesh anyway. Why would you make data reusable and discoverable if people can't use it?

#4: There aren't enough case studies yet. Tim mentioned this briefly. It is a bit of a chicken and egg issue: if we wait for people to be "done" with their journeys, it will be another 5 years before good case studies emerge. It's okay to need more proof before wanting to go forward but it might mean lost opportunity. And there are good examples out there, including guests from this podcast (20+ so far).

#5: Lacking guidance on exactly how to handle cross domain data combinations. Tim mentioned that there is the question of how do those combinations get managed as right now, in a data warehouse or data lake world, there are clear owners - the data team. Unfortunately for those who want a direct data mesh playbook, this is situational and you have to figure it out yourself for each situation and be ready to evolve.

#6: Data mesh will create data silos. Sure, if you have the data mart model of old where data is created only for the domains to use internally. But that's not data mesh. Tim talked about how important iteration and collaboration is to prevent data silos. So much is about the intent to not let data silos become a problem and iterate towards interoperability.

Overall, Tim and Scott agreed that a lot of the pushbacks are probably coming from orgs where data mesh would create a lot of friction in their existing cultures and as Tim said, changing culture is very hard and "fixing" culture is even harder.

Tim talked about how we too often think about data implementations, whether macro or micro, as a singular event, something that doesn't evolve - data implementations aren't a house, more like a garden. Seasons change, you might have to weed a bit - or a LOT -, you might change what the focus of your garden is - are you sick of zucchini? Is this data product or report/dashboard no longer relevant?

The Data Product ABCs framework:

Tim and the team at data.world put together a framework for thinking about data products. An important aspect is that this, like much of data mesh, isn't about providing specific answers but more the questions you must answer to get to a good outcome. A key point Tim made at the end was just how many data challenges come from implicit expectations and knowledge versus getting very explicit to make sure everyone is on the same page and that knowledge is shared and documented. Tim basically said get in the room -> negotiate -> come to a conclusion and shake hands -> document.

A is for Accountability - Who owns the data product? And what does ownership specifically mean?
B is for Boundaries - What is a data product? What interfaces does it use? And crucially what isn't a data product? And also what isn't part of a specific data product?
C is for Contracts - What are the explicit expectations of this data product? Who can use it? What are the SLAs? Abe Gong mentioned in his episode, #65, how often these contracts at least start as implicit - let's get communicating and negotiating folks!
D is for Downstream - Who uses the data product, who might want to use it, and why? What is the roadmap? Etc.
E is for Explicit Knowledge - Because ABCDK doesn't sound as good? Don't believe your data products are self-describing. Document things, explain in detail. What are the relationships to other data products or concepts outside the data product?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of May 15, 2022 - Data Mesh Radio May 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#75 Let's Get Intentional With Data: DDD for Data, Hyper Objects and More - Interview w/ João Rosa May 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

João's Twitter: @joaoasrosa / https://twitter.com/joaoasrosa

João's personal space: https://www.joarosa.io

Kent Beck talk at DDD Europe 2020: https://www.youtube.com/watch?v=3gib0hKYjB0

Timothy Burton book on Hyper Objects: https://www.upress.umn.edu/book-division/books/hyperobjects

In this episode, Scott interviewed João Rosa, Principal Consultant at Xebia. They discussed domain driven design for data, the importance of intentionality in preventing chaos, being effective instead of efficient, and the concept of a hyper object.

To start at the end, João talked about the need to embrace complexity when dealing with software - and we need to treat data and analytics as a software process. If we try to abstract away the complexity, we lose the nuance and that nuance is what can make all the difference in terms of the value of your data. Software is not like manufacturing where complexity is very costly.

This was a pretty broad-ranging conversation starting with Domain Driven Design - or DDD - for data. João believes we should apply the principles of DDD to everything controlled by software - and when thinking of data as a product, data is definitely controlled by software.

One of the big challenges with bringing something like DDD to data is that there aren't tools - and most challenges in the data space have historically been addressed with a tool-first approach. There is a desire to move quickly and just solve challenges but it's not possible to do that with DDD in João's view. A very interesting point of view João has is developing software is a learning process and working software is a consequence of that learning.

With the move to cloud and the easy consumption of new tools, creating data is very easy. But João believes that in an enterprise, there needs to be very clear boundaries and contracts between domains to prevent overlap and confusion. The conversations between teams are hard because all of them are context-dependent. Even at the software level, your interface to your data products is a form of communication.

João brought up the manufacturing-oriented philosophy of software development and why it causes so many challenges. It is very much about efficiency and lean development. That works well when you are producing physical goods but he doesn't think it does for software. Small incremental changes to software are not costly in a CI/CD world but the creation of software is expensive. So we need to move away from the manufacturing approach. But that would mean management releasing more control, which many are not willing to do.

For João, there is also a major value to discovery about what you've already deployed. How are people using it, what is the market / consumer-base telling us? But in general, we spend far too much time focused on new features and not discovering new things about what is already in production. And those small incremental improvements are often the things that generate real value - and if the investment is small to generate good returns, those small changes are a significant point of potential value leverage.

João brought up Kent Beck who said "once software arrives to production, it changes itself". Measuring that feedback is crucial. Data mesh, if done well, can really set up organizations to succeed because it can make people effective rather than efficient - we create data products that are easy to use but have unexpected consumption. People can discover new things. We lower the friction to new, useful insights. Efficiency is doing the task at hand with little waste. But is that effective in creating business value?

Intentionality is a key theme for João - if you have autonomy without direction, it can create chaos. In her episode, Jessitron (Jessica Kerr) mentioned the need for agency instead of autonomy. Autonomy is "you figure it out" - João quoted Jessitron as saying "you provide me the direction but not the path". We should also be constantly assessing what are we trying to accomplish and are we actually headed in that direction. What is the business problem you are trying to solve? Apply intentionality to your work to stay focused on the real goals.

When ~80% of our time is spent trying to code and only ~20% is spent on setting our intentions, what is the outcome? João believes if we flipped that and focused much more on what we are trying to achieve, solidifying the communication before going and coding it, that would have a far better outcome.

Right now, João believes that data is where DevOps was about five years ago - we still, as an industry, need to build the body of knowledge on how to do this right. The DevOps engineer title is starting to fade and we are calling them what they are - platform engineers. But as with DevOps, we need to look at the long-term payoff of building a platform - not all organizations should build a platform!

João brought up the need to think about the long-term viability of all data initiatives, not just the platform. Data products must be sustainable - which is why so many guests have recommended starting with source-aligned data products. Lorenzo Nicora does a great job explaining why in his episode. One of João's clients is leasing large industrial equipment and has switched to proactive maintenance instead of waiting for things to break and fixing them then. This has created a more reliable service for customers and lowered maintenance-related downtime costs. How can we apply that to data?

A hyper object is an object that spans time and space. João sees data as a hyper object but we typically think of data as a snapshot in time. How do we store data today to answer the questions of tomorrow? And how do we apply intentionality to data so we stop storing data for the sake of storing it. This philosophy better enables us to think of data as a product and reason about the evolution of a data product.

To attempt to sum up João's thoughts: focus more on intentionality - why are we doing something and is it working -, embrace complexity, and look to solve more through conversation instead of tooling.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#74 What is Data Mesh Trying to Achieve? - Mesh Musings 14 May 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Scott shares some thoughts on some recent data mesh FUD and market confusion.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#73 Ship-Posting and Cake Recipes: Measuring the Return of Your Data Initiatives - Interview w/ Katie Bauer May 09, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Katie's LinkedIn: https://www.linkedin.com/in/mkatiebauer/

Katie's Twitter: https://twitter.com/imightbemary

In this episode, Scott interviewed Katie Bauer, a Data Science Manager at Twitter in their Core-Tech group. To be clear she was not on representing Twitter, only her own opinions. The main topic of discussion was how to measure the value and success of your data projects/implementations.

Some very useful advice from Katie that can feel a bit obvious when said but is VERY often and easily overlooked: measure for what would make you drive actions. If getting a 10x higher than expected or 90% below expected result isn't going to change your decision, while it may be interesting information, is it really important? If not, don't waste the time to measure it. Especially early on in your data measurement maturity. The point is also to get to an objective evaluation, not overly precise measurements. Set yourself up to improve and iterate. Don't make this hard on yourself.

She also gave the pithy statement: what is valuable is not necessarily valued.

Katie has a cake analogy that plays into data maturity well. Think about your need and the other person's capability regarding making a cake. Do you need a fancy cake for wedding or is this for a 3 year old's birthday party? One, you probably want to be special. One, if it vaguely resembles something from TV and tastes decent, the consumer will probably be happy. Is the other person capable of making a super fancy layered red velvet cheesecake or is a cake mix in a box probably more up their alley. How mature are the parties on creating measurement data and how mature or advanced do you need the output to be?

Katie started the conversation talking about some survivorship bias / other biased ways of measuring. Often, she has seen throughout her career that people having success seek to prove their success via metrics instead of find the metrics that matter the most. That has some pretty obvious flaws so we need to move forward towards better measurement practices. For Katie, measuring the value of data science is pretty meta.

Katie recommends starting out with some really easy measurements around engagement and usage. If it's a platform, what are your daily active users, weekly active users, and/or monthly active users - and what is the actual most useful metric? Should people actually be leveraging your project daily? Think about what is your addressable market and what percent of that market you have. And NPS (net promoter score) is a very lagging indicator.

When thinking about metrics, there are two things that really stand out to Katie: first, what is your useful granularity? Don't get overly precise if you don't need to. You want an objective evaluation and anything past that can become overkill, which has an inherent cost. And second, what is your useful time-scale? Is this on a micro-scale, where the task should take 5min to complete so a difference of 5min is a big deal? Or is it a much longer time scale?

When thinking about what to measure, ask yourself what does your company value. Is it shipping, usage, cleaning up tech debt/deprecation, etc.? Katie threw out a bit of a mind bender: what is valuable is not necessarily valued. So think about what people care about regarding information flow. It might not be the most valuable information, but it might be highly valued. Or vice versa. At the end of the day, are you there to be right or to serve your constituents?

Katie's getting started on measurement advice includes starting with something concrete - use that initial measurement as a learning stepping stone. She mentioned that it can be hard to recover from measuring the wrong thing or getting your measurement wrong - people can jump to measuring is bad so set yourself up via expectation setting that you will iterate on your metrics.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#72 Reliability in Data Mesh: Why SLAs and SLOs are Crucial - Interview w/ Emily Gorcenski May 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Emily's LinkedIn: https://www.linkedin.com/in/emily-gorcenski-0a3830200/

Emily's Twitter: @EmilyGorcenski / https://twitter.com/EmilyGorcenski

Emily's Polywork profile: https://www.polywork.com/emilygorcenski

Emily's website: https://www.emilygorcenski.com/

Alex Hidalgo's Implementing Service Level Objectives book as mentioned: https://www.alex-hidalgo.com/the-slo-book

In this episode, Scott interviewed Emily Gorcenski, Head of Data and AI at Thoughtworks Germany. Emily has put out some great content relative to data mesh.

As a data scientist by training, Emily has a data consumer bent in her views on data mesh. She is therefore often focused on how can data mesh help "me" (her) as a data consumer.

SLAs and SLOs come right out of the site reliability engineering playbook from Google. Overall, systems reliability engineering practices are crucial - Emily asked why don't we bring the rigor of other engineering disciplines to software engineering?

So, what is an SLA and an SLO? Per Emily, an SLA is a contract between two parties - hence why agreement is in the name. This agreement should be written around an SLO with the SLO serving as a specific target. That can be uptime or latency in the microservices realm but with data, SLOs can get a little - or a lot - more tricky.

The theory around developing an SLO is for it to directly connect to business value. Emily believes that when we think about SLOs and data, we shouldn't apply SLOs directly to the data but should shift those SLOs to the left and have SLOs in the software engineering practice that apply to data.

Emily mentioned another antipattern for SLAs in general, which is not connecting them to SLOs. But when it comes to data, most teams don't even have any SLAs, connected to an SLO or not. As an industry, software engineering has figured out how to offer great SLAs to external parties but many organizations still struggle to offer good SLAs internally.

For Emily, software-focused SLAs can even result in worse outcomes for data. If an SLA is about uptime, it might result in pushing bad data into a system so a service can maintain its SLA.

When developing SLAs, Emily recommends starting with conversations and negotiations between both parties. If 5 9s of uptime is not valuable to your consumers, why build to ensure 5 9s? Dig into actual user needs and what will actually drive user value. And start to differentiate between infrastructure focused SLAs - like is the data product available - and data SLAs - like is the data updated and does it meet quality thresholds.

Emily then started to talk about some of the fun very specific SLAs around data and what does data availability mean. These SLAs can get complicated but they can start to really drive towards what is actually valued by the consumers, what is the actual value of the data so you can then start to negotiate to drive a high return on investment. Again, we can avoid pre-optimizing for facets that consumers don't care about.

Per Emily, good SLOs will tell you what you should improve. We should make sure our SLOs are decomposable to again, get quite specific when useful and/or necessary. It is much more difficult to do in data than in general software engineering - we can't think about data in a binary way such as accurate or not it is much more of a continuous spectrum. Emily recommends to look at the error budget concept and think about how we can apply that to data.

Emily believes SLOs can help you to avoid building unnecessary complexity - if your users don't need real time results, don't build a real-time system. It's the conversations and negotiations that take you from the state of what's possible to what's valuable. We should use SLOs to align closely to the use case - there is definitely such a thing as good enough. And don't create Franken-Data-Products - monstrosities that try to solve every single need. It's fine to have two similar data products to serve two distinct needs.

For Emily, data consumers keep complaining to a centralized data engineering team. People on that centralized team are the unfortunate middle-people with little power to change what consumers are getting. We should use SLOs and move them to the responsibility on the software development teams - the domains - much like we do data ownership in data mesh.

Once an organization learns to do SLOs well, Emily recommends extending that to use SLOs around the data platform. But to not mistake the SLOs and SLAs around infrastructure and data products as mentioned earlier.

Emily believes the governance team also has a responsibility to drive standardization around SLOs. This includes sensible defaults.

What should we learn in the data space from DevOps? For Emily, the philosophy of resilience is crucial. Repeatability and safety through continuous integration / continuous delivery - or CI/CD - is a major driver of value in software engineering. How can we apply it to data?

In data, we all too often use a systems oriented approach so we don't properly attribute value well per Emily. How can we measure the value of being able to do ad hoc analysis? Not the value of the analysis itself but almost the inverse of opportunity cost - what is the opportunity value? If we remove the abstractions, can we get to a specific value measurement?

Emily believes we need to get much more serious about creating good data about our data practices. It takes a fair bit of effort to get to a place where we can repeatedly get good, usable data on our data initiatives at scale. We also need to give people more slack in their work time to chase down additional information. Serendipity can only strike if people have the room to create it and then react to it.

Emily wrapped up her thoughts on a few points - first, the pace of change of business has accelerated significantly and it requires us to philosophically reorient how we think about data. There needs to be more space for people to doing the new necessary work to drive the high incremental value. But because everyone is so overloaded already, that isn't happening in most organizations. And second, start from the consumers and their needs and work backwards. It's okay to not create every piece of potentially useful data in a usable fashion upfront. Figure out what are the needs you know about and build towards those - additional use cases will emerge.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of May 8, 2022 - Data Mesh Radio May 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#71 Adventures in Data Maturity - Creating Reliable, Scalable Data Processes - Interview w/ Ramdas Narayanan May 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Ramdas' LinkedIn: https://www.linkedin.com/in/ramdasnarayanan/

In this episode, Scott interviewed Ramdas Narayanan, Vice President Product Manager of Data Analytics and Insights at Bank of America. To be clear, he was not representing the company and was sharing his own views.

Ramdas came on to discuss lessons learned from building effective data sharing at scale on the operational plane over the last 5-10 years so we can apply those to our data mesh implementations.

A key output of the conversation is a guiding principle for getting data mesh right - your goal is to convert data into effective business outcomes. It doesn't matter how cool or not cool your platform is or anything else - drive business outcomes! It's easy to let that get lost in the tool talk and everything around data mesh.

Per Ramdas, when looking at creating a data product, or really any data initiative, you need to align first on business objectives and that will drive funding. In the financial space, that is direct literal funding but even outside, you should have the same mindset. Make sure you get engagement and alignment across business partners, technologists, and subject matter experts. How are you using technology to address or solve the business problem?

Ramdas has seen that if you don't focus on creating reusable data, you can create silos - you need cohesive data sets, not bespoke data sets for every challenge as that just doesn't scale. You should also study the data sources you are using - is there additional useful data you could add to your dataset or could you use that data for other purposes - keeping an eye out for additional data to drive business value will really add a lot to your organization.

When working with developers, Ramdas recommends helping them understand how the business is going to consume and use the data and then figure out if they should deliver data as something like an API or web service or more of a custom batch delivery. It is important to also work with data consumption teams to be reasonable in their consumption demands - getting them to modernize can be a challenge and that can put an unreasonable burden on producing teams.

Ramdas talked about how crucial conversations and culture are to getting data projects/products right. Sometimes the conversations can be tough but often they really aren't and there just needs to be open exchange of context and information, especially aligning on business objectives. Projects that fail typically have poorly defined business objectives or lack alignment.

Per Ramdas, it is important to educate the business people on what data exists and even what data doesn't. That clouded vision of what data is available creates a lot of frustration - we need to get better in general at data discoverability so the business folks can know what is available and get access easily. Ramdas has seen repeatedly that good context via rich metadata also leads to better context sharing at the person-to-person level as it generates additional conversations.

To emphasize that point a bit more, Ramdas believes that data discovery is the main spark for sharing context. Otherwise, we are at best exchanging data as 1s and 0s instead of the actual information.

Ramdas believes everyone needs to understand how information flows through your systems - it can help you better understand the art of the possible and also identify gaps in how you will approach your challenges. Start your projects, whether that is a new data product, a new platform feature, or anything else, by having a lot of information architecture meetings. After that, start to focus on data discoverability. "Show and tell" sessions have worked well for him as they spark new thoughts and can help surface issues.

Ramdas wrapped on a really crucial part of data maturity which is the curiosity factor. Always be asking why you are doing something, what problem are we actually trying to solve? Do we have the capabilities to solve them? How does the data flow through our systems? Can we push data quality upstream to prevent quality issues instead of remediate them? What guardrails can we put in place to prevent issues? How can we enrich our metadata to make this data even more valuable? Etc.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#70 For Your Sanity, Stop Trying to Solve it with Technology - Mesh Musings 13 May 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#69 Getting Data Sharing Right at Netflix Scale - Interview w/ Justin Cunningham May 02, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Justin's LinkedIn: https://www.linkedin.com/in/justincinmd/

In this episode, Scott interviewed Justin Cunningham, who worked as a tech lead and data architect on data platforms at Netflix, Yelp, and Atlassian over the last 8.5 years. In that time, Justin was involved in initiatives to push data ownership to developers / domains.

To sum up one of Justin's points he touched on repeatedly - he recommends to create a pool of low effort data which will inherently have low quality. Use that for initial research into what might be useful. Focus on maximizing accessibility - you can have governance and use things like join restrictions or give consumers an ability to self-certify that they are using the data responsibly. Once you get the use cases, then you go for the data mesh quality data products.

Justin saw a lot of success at Yelp focusing on data availability - getting data to a place it could be found and played with - was a bigger driver for success than focusing initially on data quality. Once people discovered what data was available and how they might use it, the organization was able to work towards getting that data to an acceptable quality level.

Another point Justin made was figure out which you want to optimize for in general: getting things right upfront or testing and changing. He believes in optimizing for change. Create an adaptive process and optimize for learning. Keep it simple and focus on value delivery - it will set up more tractable bets.

At Yelp, they were trying to ETL a huge amount of data in their data warehouse to build reports for the C-Suite. But they were never really going to get enough data ingested to really meet their goals. It was taking them 2 weeks to create each new set of ETLs and that was just creation, not maintenance - it was looking like they'd need 5x the number of people.

What Justin found the most useful at Yelp was to focus on getting as much "usable" data in an automated way. They achieved this initially through the data mesh anti-pattern of copying direct from the underlying operational data stores and building business logic on top of it. But, that data getting into the hands of the data team meant there could be an initial value assessment - once they proved out there could be value in the data, the conversation with developers was much easier to get them to care about providing clean and reliable data.

Justin mentioned the same thing Wannes Rosiers mentioned in his episode: there are operational and analytical workloads but there should absolutely not be that separation when it comes to data. Data from operational systems is useful for analytics and vice versa. One thing that really helped developers understand how to share data was thinking of data sets as being similar to public APIs.

At Netflix, there were just too many bespoke data sets - made it very hard to manage quality. What they found that worked was a data certification program for data sets, creating tooling to prove a data set was complete and accurate. That and upping the amount of focus on data set reuse significantly helped them to combat the data sprawl.

Back to data accessibility and availability versus quality, for Justin, he believes data analysts and data scientists initially care far more about getting data access as you can work to improve the data quality later - especially if there is a clear owner. I discussed this in a mesh musings about speculative data products but a key hack for them was being able to mark data as low quality.

On driving buy-in for data producing teams, Justin again talked to proving there was value in data before the producers were bought in. Asking them to serve their data upfront without a clear, specific use case was very tough. The return on investment or ROI was very squishy. So they got out low quality data initially and then came back to producing teams to up quality and reliability once they proved certain data was valuable. This is someone similar to the emerging data mesh pattern of creating your data products for a consumer-focused use case. It might be a source-aligned data product but it should still be initially serving a specific purpose with a targeted outcome. It can grow from there.

Justin also shared his thoughts on how the way we do data lineage is broken - we should look to do lineage declaratively instead of just as a reference. This should flow through both the schema registry and the data catalog. What is the data movement supposed to be? This would enable us to much more easily test data flows and alert downstream users of upcoming changes.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#68 The Build vs. Buy Dilemma of Data Platforms - Interview w/ Doron Porat May 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Doron's LinkedIn: https://www.linkedin.com/in/porat-doron/

Our journey towards an open data platform: https://medium.com/yotpoengineering/our-journey-towards-an-open-data-platform-8cfac98ef9f5

A simplified, lightweight ETL Framework based on Apache Spark by Yotpo: https://github.com/YotpoLtd/metorikku

The Data Swamp (in Hebrew): https://open.spotify.com/show/5YDdtRhp1RVw7r5fbYFtPQ?si=5x4HzOyhTX6n46HqY5kV6w&nd=1

In this episode, Scott interviewed Doron Porat, a Data Infrastructure Leader at the SaaS company Yotpo.

Some crucial points Doron made:

1) Be kind to yourself when you make mistakes - it's worse to stagnate so don't be afraid of change and making choices

2) Build versus buy is always tough but don't let your ego get in the way and push you towards building everything

3) If you do buy, build a close relationship with your vendors to help influence the roadmap and have an outlet if you are having issues

4) A data platform team's job is to drive usage as usage means creating value - drive towards that and set your KPIs around platform usage

5) There will likely be many different types of consumers of your data platform - work to improve / optimize the user experience for most folks

Doron is a technologist at heart so for each decision she instinctually wants to build instead of buy. And at the start of building out the data platform for Yotpo, that was typically her decision. But as the demands for more and more capabilities from the platform, the increasing ubiquity and quality/scalability of as-a-service offerings, and the growing need to drive usage and developer happiness instead of manage cool tech, she started to consume more and more managed services.

When you are building out the platform, vendors, in the long run, can often better serve your needs because they have a whole lot of people focused specifically on making what you use better. You need to make bets on the vendors getting to where you need them to be and sometimes they don't pay off either. To up the chance of making those bets pay off, Doron recommends building relationships with your vendors to influence their roadmaps and get help when necessary.

Doron strongly recommends putting together a framework for evaluating build/buy decisions. Some of the factors she considers are how extensible is the offering, is this taking on too many challenges in one solution, cost, the cost of later migration to or from a managed service, open source compatibility, etc. One thing Doron talked about that many teams seem to struggle with is the ego hit of saying someone else managing a service we use will drive more value. That's always tough but needs to be addressed.

Doron talked about the strong need to drive your platform forward, not just be responsive. Provide a roadmap, set time aside for innovation, etc.

Doron made the point there is a difference between learning to leverage a tool and operate it. You want to make sure you build out the knowledge around leveraging a tool whether you are operating it yourself or not. It's an interesting balance when you build that you don't focus too much on one or the other.

Doron talked about the general job of a data platform team is to drive usage because that usage means creating value. Serving critical needs for users is crucial to driving adoption and focus the user experience on the business logic - no matter how cool the tech is, a data platform team's job isn't to expose that to users. And your team's KPIs should reflect usage. However, there are all kinds of users - on both the consumer and producer side - that you need to serve so don't create a one-size-fits-all experience.

If your data platform isn't easy to adopt, no one will want to use it. Your data platform should make things fun and casual to do. There needs to be a big focus on enabling data testing including staging environments for data - that will drive usage.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of May 1, 2022 - Data Mesh Radio May 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#67 All About Interoperability and Standards in Data Mesh - Interview w/ Samia Rahman Apr 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Samia's LinkedIn: https://www.linkedin.com/in/samia-rahman-b7b65216/

FHIR standard cheat sheet: https://www.healthit.gov/topic/standards-technology/standards/fhir-fact-sheets

In this episode, Scott interviewed Samia Rahman, Director of Data and AI Strategy and Architecture at life sciences company Seagen. Samia is helping to lead Seagen's early data mesh implementation after helping with two implementations at Thoughtworks since the start of 2019.

For Samia, interoperability is about taking information from two systems and combining them to get a higher value. A simple definition but a good one.

Two potential key takeaways:

1) don't try to plan too much ahead for developing interoperability standards but definitely keep an eye out for places where you could start to develop those standards. And your standards really, really should evolve - you don't have to nail them right out of the gate.

2) your interoperability will also evolve - you don't need to make every data product interoperable with every other data product and you can start with basic interoperability first. The more you can standardize around unique identifiers, the better, but it's okay to not get it right first thing out of the gate.

Samia started her career - and even before in school - focusing on software, especially end-to-end development. A repeating pattern for her has been how crucial contract testing is to getting things into a trustable and scalable state. We've had them in hardware and software for a long time and if you don't have easy testing, those systems often get replaced pretty quickly. Those tests are the safety net to allow for fast and reliable evolution. And that evolution is a key theme for this conversation - set yourself up to iterate and evolve as you learn. Work to not paint yourself in a corner

Data standards, including specifically for interoperability, are everywhere in the life sciences space - FHIR, FDA has lots, etc. but it's still not great for truly sharing the meaning of the data. FAIR is trying to get there but the interoperability and domain knowledge isn't really standardized yet.

Samia strongly recommends not getting ahead of yourself on interoperability and standards. It's perfectly okay to start small - iterate and build on your standards for interoperability, To start have some key identifying "linkers" done. Get things out in front of consumers so they can explore and give feedback and use that to power your iterations. Incrementally building towards a standard is crucial.

If you are going to build a standard, reusability should be your first goal. If it is only for a single use case, that isn't a standard, it's just an implementation detail. Samia again recommends contract testing / a schema checker. And definitely leverage existing standards. It's also not a huge deal if you have more than one standard internally. You don't need one standard to rule them all.

Per Samia, if you implement versioning, data consumers are usually very willing to work with data producers as they evolve data products. But without versioning, you are just pulling the rug out from underneath them. And right now, there isn't a lot of good info on versioning data out there, nor tooling. The need to evolve data products is why absolute self-service is probably never possible. The human-in-the-middle is important to help consumers evolve their thinking as the business model evolves.

Samia mentioned the data consumer responsibility to inform data producers - inform them about need changes, issues with their data products, etc. We can't have data consumers going off and all creating their own fixes to data quality issues, the data producers need to know so they can fix them at the source.

You need to be on the lookout for interoperability opportunities and then validate that there is a need for for interoperability. An important point is that not all data needs to be interoperable.

Samia finished with her interoperability vendor wish list - some kind of tooling that can more easily detect when someone should use an existing standard and that can put those standards in front of data product producers much more easily. How can we make it very easy for data product producers to build in interoperability and leverage existing standards from the start?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#66 Negotiation as Your Avenue to Success in Data Mesh - Mesh Musings 12 Apr 26, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Scott shares his views on the importance of collaboration via negotiation, not requests, to make your data mesh implementation a success.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#65 What's a Data Contract Between Friends - Setting Expectations with Data Contracts - Interview w/ Abe Gong Apr 25, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Abe's Twitter: @AbeGong / https://twitter.com/AbeGong

Abe's LinkedIn: https://www.linkedin.com/in/abe-gong-8a77034/

Great Expectations Community Page: https://greatexpectations.io/community

In this episode, Scott interviewed Abe Gong, the co-creator Great Expectations (an open source data quality / monitoring / observability tool) and co-founder/CEO of Superconductive.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#64 The Crucial Value of Data About Your Data: Approaching Data with a Product Mindset - Interview w/ Sadie Martin Apr 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Sadie's LinkedIn: https://www.linkedin.com/in/sadie-martin-06404125/

In this episode, Scott interviewed Sadie Martin, Senior Product Manager, Data Platform at Q4 Inc about applying a product mindset to data in general. This is really crucial to getting data as a product right but also in building out your data platforms and even some processes for data mesh.

Scott's summation of some key points:

Anyone can apply a product mindset, not just the product manager
Giving yourself the time before starting work to investigate and create you measurement framework, including your baselines, is crucial to measuring data work progress and choosing where to focus
Approach your data work with intentionality
Really understand what you are trying to accomplish and what your immediate customers/consumers are trying to use the data for to accomplish.

Sadie started as a data analyst where the team didn't have a product manager - they were doing a lot of work and weren't sure if things were likely to work or even if what they did had a positive impact after it was done. So she started to take on some of the task of answering those questions and transitioned into being a product manager for data.

So, what is a product mindset? For Sadie, the easy definition but with lots of hidden depth, is "it's all about really understanding the problem". For most organizations, really thinking about the problem you are trying to solve is new relative to data. There may be a data request but what product or process is that data contributing to and what is that product or process trying to solve?

Sadie believes measuring the problem is really crucial. Once you figure out what you are trying to solve, what is the scope of the problem? How are you going to measure if you are actually solving the problem? Especially is it better than what you were previously doing? She also talked about the importance of customer-centricity - really why are they making a data ask? Should this really be a one-off or a repeatable process? Did they ask for the complete set of what they need? Etc.

One crucial insight Sadie has brought from product management to data is to be willing and ready to throw things away. If it ain't working, don't be too precious. That's a very different mindset than we've historically had relative to data. There's also the idea that processes can devolve quickly so ensuring when you start a repeatable data process, understand the effort to keep it going.

While it feels counter-intuitive, Sadie laments that for most, it's often quite difficult to get the buy-in that you need data to measure if your data work is actually providing value. It's still worthwhile to do however. You need to take the time to do spikes and investigate ahead of time and slow down enough to set yourself up to measure results. Just continuing to go off assumptions and gut feelings is going to put you in a vulnerable spot to a competitor doing the work.

Sadie looks at measuring the success of data work in two ways. The first feels obvious once said but really isn't: start by measuring the baseline. Without that baseline, you can't measure if you're having an impact. And lots of data work proves to be low value or negative value - you tried a hypothesis and it isn't working so stop and move on. How do you get to that answer fast? You measure the incremental change for the effectiveness.

So what happens when you do look at your work and find out it's not been valuable? Per Sadie, you have to get away from the sunk cost fallacy. It's absolutely okay to make bets and they don't pay off and you move on. You need to really investigate if you are solving the problems you set out to solve. And by proving out the value of the product mindset so you can make better bets in the future.

A lot of the product mindset is also thinking about the return on investment, not just maximizing the return or value of data work. Can the simple get you where you want to go without doing the extra cool but complicated and/or risky parts?

Sadie mentioned a few things getting in the way of applying the product mindset to data. One is that there are often teams making promises on behalf of the data team without checking with them first. The other is many data consuming teams view the data platform teams as simply service teams, not partners.

While there has been a lot of hiring for data product managers in the last year or so, Sadie sees that often the companies aren't making the product mindset an actual priority and that feels like a waste of a good product manager.

There is a misconception that data work is all about facts. A large part of it is discovery work, much more than in most disciplines. Per Sadie, measuring a team's effectiveness should focus more on getting to an answer than getting to preferred answers. Evaluating a lot of hypotheses and proving them invalid isn't a bad thing - you prevented a lot of toil work that wouldn't have added value. Make sure to measure teams based on that.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Apr 24, 2022 - Data Mesh Radio Apr 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#63 Driving Domain Maturity Through Empathy, Respect, and Understanding - Data Innovation Summit Takeover Interview w/ Henrik Göthberg Apr 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

This episode is part of the Data Innovation Summit Takeover week of Data Mesh Radio.

Data Innovation Summit website: https://datainnovationsummit.com/; use code DATAMESHR20G for 20% off tickets

Free Ticket Raffle for Data Innovation Summit (submissions must be by April 25 at 11:59pm PST): Google Form

Henrik's LinkedIn: https://www.linkedin.com/in/henrikgothberg/

Dairdux website: https://dairdux.com/

Airplane Alliance website: https://airplanealliance.com/

In the last of the interviews for the Data Innovation Summit Takeover week, Scott interviewed Henrik Göthberg, the Founder and CEO of consulting company Dairdux, the Co-Founder of the Airplane Alliance, and the Chairman of the Data Innovation Summit.

Let's start with some conclusions/advice from Henrik:

When working with other departments, in data mesh or not, you need to start from respect, empathy, and understanding for people in different roles.
When you think about maturing a domain or process, a big bang approach very rarely works. You need to think about evolution, not revolution.
To find a good pathway to maturity, start with the domains already on the leading edge, the innovators; trying to get the laggards to catch up instead of focusing on those who see value in maturity is going to lead to pain and likely not much progress.
Start with less complicated and high risk challenges so you can learn and develop the right muscles to do things easier in the future.
Focus heavily on reuse - reusable data, yes; but also templates and other "easy path" enabling things. To succeed in data mesh, you need to get to a place where you can have broad reusability. Reusable data, reusable processes, reusable templates, reusable tooling, etc.
In a data mesh implementation, start with an initial domain but move on to adding a second domain quickly if possible. Templates will get you to value quickly.
It's okay to skip automating or building out a great solution for certain pieces of your data mesh implementation. What will get you in trouble is building half-solutions that end up as major pain points. This is the biggest source of unintended tech debt.
If you business people don't understand they own the processes and the data, your data mesh implementation is much more likely to fail.

Background and other color:

Henrik covered his journey from 2012 to present in most of the first 30 minutes - from joining a domain to add analytics capabilities to that domain to building out a large data and analytics central team at the same company to joining a new company in 2019 to help them implement a new data strategy which has evolved into implementing data mesh.

Henrik joined Vattenfall to build out the data and analytics team inside the sales org. They had a multi-country domain with different maturity levels across each country. They needed to improve the data and analytics capabilities and operations in all three countries so they could have strong data and analytics capabilities at the country and European level. The team had some technical savvy but they were struggling with actually getting the data - the data was locked into the source systems. It was difficult to even do basic customer analysis and data science, not to mention anything fancy. So they needed a lot of help in maturity.

In 2015, Henrik became the Business Intelligence Officer at Vattenfall. That meant taking ownership of the centralized team with lots of core data and analysis. A big part of the role was owning providing costs in very granular ways so needed to try to move to a very standardized reporting model for P&L.

A big change was in consumer maturity. When Henrik first started the role, people were mostly consuming reports. They moved to consuming data sets and even raw data. As part of that, they often moved from ETL to ELT, which caused some major headaches as many have seen with the data lake.

All of that background maturing the data and analytics capabilities helped Henrik when he joined Scania, a truck manufacturer, in their financial services division. The culture of the company was already very decentralized and modular, which can set up well for data mesh but that also meant domains were very independent with limited standards or standardization around data enterprise-wide. They had a big data lake implementation with a good raw data layer and a semantic layer but the analytics layer on that was lacking. The centralized data team was struggling to even manage the raw data layer from a governance perspective and they were feeling increasing strain from issues trying to manage data pipelines.

Henrik mentioned the necessary evolution process for domains - a "big bang" approach very rarely works. And Henrik started with the domains in the innovator category as they were the most bought in on domain maturity. As part of this process, they were able to decommission many large data warehouses.

To start, Henrik focused on what was valuable to build for the domain - the micro level - instead of valuable to the greater organization. That way, he could mature that domain much faster and if there are multiple mature domains, those mature domains are better prepared and capable to work with each other. There was a focus on building reuse wherever possible - not just reusable data but what templates and other easy path things could the team create.

After year 1 of focusing on creating value from the data products individually, Henrik and Scania started to focus more on creating value at the overall mesh level - this is where data product interoperability really can come into play.

Before you get going on a data mesh journey, Henrik recommends spending the time to really plan out how you think your implementation will work and how it will create value for the organization. And what will be the near-term value adders and what will be the longer-term value adders.

Henrik strongly believes in either taking challenges on with the intention to get to a good solution now or not tackling the challenge at all. The half-assed solutions just lead to far more pain so either commit to take it on or leave it entirely for later.

Another piece of advice is to not have the domain teams just hire without consulting the central team, especially if there is a central team around that competency. Look instead to embed people from the central team into your domains so they can understand the friction points to build out templates to address that friction.

For Henrik, it's key to find the right people in each domain who can be a sensible buyer. There needs to be a high level of trust between the business and IT and so you need someone who can develop a strong relationship with IT.

For Henrik, you need to start from respect, empathy, and understanding for people in different roles in order to actually form a strong relationship. Business people often think it's not that hard to set up your data and analytics processes well. You should focus on investing time and energy with the key players to develop a good relationship. That way, it is much easier to get to each other's context.

Henrik wrapped up talking about to succeed in data mesh, you need to get to a place where you can have broad reusability. Reusable data, reusable processes, reusable templates, reusable tooling, etc. He also believes that domains, especially the business people inside the domains, need to understand they own the business processes AND the the data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#62 Can We Make Data Mesh 'The Good Place': What We Owe Each Other - Mesh Musings 11 Apr 19, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#61 Driving Value Through Participating in the Data Economy - Data Innovation Summit Takeover Interview w/ Jarkko Moilanen Apr 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Jarkko's LinkedIn: https://www.linkedin.com/in/jarkkomoilanen/

Open Data Product Spec: http://opendataproducts.org/#open-data-product-specification

Jarkko's Data Product Business website: https://www.dataproductbusiness.com/

This episode is part of the Data Innovation Summit Takeover week of Data Mesh Radio.

Data Innovation Summit website: https://datainnovationsummit.com/; use code DATAMESHR20G for 20% off tickets

Free Ticket Raffle for Data Innovation Summit (submissions must be by April 25 at 11:59pm PST): Google Form

Scott interviewed Jarkko Moilanen, a Data Economist and Country CDO ambassador for Finland. Jarkko will be presenting on "Data Monetization and Related Data Value Chain Requires Both Data Products and Services" on May 6th in track M6.

Jarkko is approaching measuring the value of data and then trying to extract that value from data from many different angles. He is thinking about data products, data as a service, data as a product, etc.

Per Jarkko, treating data like a product can apply a LOT of the learnings from the API revolution - this time around we can skip a lot of the sharp edges. APIs are about an interface to value creation - how can we treat data products the same way?

We discussed the difference between return and return on investment. A data initiative may have a very high return but if the investment to get that return is too high, it's a bad initiative. How do we get to figuring out what quality level we need to solve our challenges - there is no reason to go for 5 9s quality if that doesn't move the needle.

Jarkko coined a new concept on the call - the half life of data value. For a large percent of data, Jarkko believes the value of the data starts to fall considerably over a relatively short period of time. How can we extract the value when it is most valuable? If the half-life is weeks, days, hours, or even less? And how do we set ourselves up to get the most "bang for the buck"?

Jarkko is firmly in the camp of intentionality when it comes to data. We can't keep betting on "this data might have value" or collecting data for the sake of collecting it. The data cleansing after the fact is difficult - what was the context at the time? Can you enrich the data further? Etc. - and the cost to do so is typically quite high compared to the value. And you keep incurring costs just to keep data around. If you ascribe to his data half-life theory, the value diminishes quickly so stop keeping around so much data you aren't using!

Jarkko's data economy model has three layers that he adapted from the API economy:

The bottom layer is private/internal to the organization only use. In Jarkko's view this is typically for organizations that don't have the capabilities to productize their data, that have low data maturity. If they can move toward productizing, it will enable reuse - not just for themselves but potentially third parties.
The middle layer is to have closed sharing agreements with other organizations, creating data sharing ecosystems. These ecosystems are typically very limited in the number of other organizations involved. These are often also about creating value for a joint purpose, e.g. two suppliers sharing data specifically to meet a customer need. To do this, you need to productize your data enough to make it generally understandable and relatively easy to use.
The top layer is the completely public data marketplace layer. Organizations participating at this layer are packaging data or even algorithms for sale.

Jarkko also has four key elements that define a data product - whether a data mesh type data product or not - in his mind:

The technical data flow layer - how the data is processed/created/handled by the underlying infrastructure
The business plan layer - descriptions, plans for the data; really, what is the business objective of the data product
The legal layer - what are the conditions for using the data
The ethical layer - while this is becoming more important in the AI space, we should think about ethical use in all data products

Jarkko talked about trust as a measurement for value - if you can't trust data, it's value is significantly less. Lack of trust can significantly raise the total cost as consumers have to work to enrich the data and thus lowers the value of data. Jarkko has been working on a trust index for data products, which is especially applicable in a data exchange scenario.

For Jarkko, there are 3 key things to managing data: First, treat every bit or set of data as if you'd share it externally. That means enrich it, make it trustable, usable, secure, etc. Data has a habit of going external in some way. Second, make your data actually usable in your scenario - what level of data literacy do you have so you know what bar you have to meet? How can you find that core 80% in a 10/80/10 split that will drive insights with data? Third, have a ready-made toolkit to mock data products at the business layer with consumers. This is more about the process than tooling, but have a set of canvases so you can share ideas about new data products and get good feedback. That good feedback from users before creating a data product is very useful.

Jarkko summarized his thoughts with let the business people lead the way; if they aren't enabled to lead, we need to educate them so they can leverage the data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#60 Managing Organizational Structure in a Traditional Company: Can You Have Two Solid Lines - Data Innovation Summit Takeover Interview w/ Daniel Engberg Apr 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Daniel's LinkedIn: https://www.linkedin.com/in/danielengberg/

This episode is part of the Data Innovation Summit Takeover week of Data Mesh Radio.

Data Innovation Summit website: https://datainnovationsummit.com/; use code DATAMESHR20G for 20% off tickets

Free Ticket Raffle for Data Innovation Summit (submissions must be by April 25 at 11:59pm PST): Google Form

Scott interviewed Daniel Engberg, Head of AI, Data, and Platforms at Scandinavian Airlines. Daniel will be presenting on "Structuring an Enterprise-Wide Data Organization" on May 6th in track M4.

A key point Daniel made right away was that organizational structure should be tailored to accomplishing your goals - so we have to know what those goals are first. What are the capabilities we need to meet those goals? "Traditional" companies are often locked into their structure - silos by competence; so data engineering in one silo, marketing in another, sales in another, and so on. Daniel is interested in figuring out how we can split up the competencies to create cross-functional, cross competency teams but not cause chaos to the organization as a whole.

Daniel gave an example of creating a cross functional team early in the pandemic as there were some very big threats to the business - being an airline when no flights are happening is a scary place. The cross-functional team was able to move so much more quickly than the way the company tackles challenges when it is business-as-usual, achieving their goals in a few days instead of what typically would have taken months. This cross-functional work also created new information sharing connections across the entire company which continues to create additional value.

What Daniel learned from that experience, he is trying to replicate as best he can to make it the new business-as-usual instead of a one-off. As the head of AI, Data, and Platforms, he is working to infuse members from his team directly into more projects so they can be part of the teams and decisions instead of handling requests after decisions are made. It also gives his team members the ability to rationalize goals so there is a better ability to do maybe 80% of what would be requested with only 20% of the work in a month instead of the whole 100% in 6+ months - where is the value cut-off? Negotiate instead of take requests.

For Daniel, product owners must start working to gather the competencies they need on their own cross-functional teams. But that can cause issues when domains start to hire when they lack strong knowledge in that competency. E.g. domains hiring data scientists when they have no idea how to find a good data scientist whose capabilities match their needs and goals. Or do they even want a data scientist instead of a data analyst? Then the career growth aspect gets challenging too - does a product owner need to know how to grow the career of 10 different types of widely varying roles.

We talked about the challenges of dotted lines versus solid lines between a functional manager and a competency manager - who do you listen to? Can we have two solid lines for reporting structure? Daniel believes people want managers who understand their day-to-day work.

As stated, hiring into domain teams directly is very tough. Competency Leads need to ensure the company has the right talent and the right amount of talent in their competency and then work with the domains to place resources into those domains.

Per Daniel, there is also the natural resistance to change with changing responsibilities. Employees end up worried - a new way of working always causes some amount of fear. Daniel and Scandinavian Airlines are currently trying to do this type of change with some of their competencies.

One tip Daniel gave for making change a little easier to accept is to very explicitly lay out expectations - heavily communicate and clarify so we can take the fear of change head on. What will it mean for job security? How people work together? Be clear how you want it to work so people can adjust to meet the goals, not the implied structure.

Daniel finished up by talking about how data governance is still a major challenge in traditional organizations. They are looking to make the data governance as close to the product teams as possible - the decisions should be owned by the product teams wherever possible. When turning over those responsibilities to the domains, it is important to give them good decision frameworks and also let them know they have a backstop if they need help.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

DIS Takeover - Weekly Episode Summaries and Programming Notes - Week of Apr 17, 2022 - Data Mesh Radio Apr 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#59 Knowledge Graphs as the Engine for Collaboration Across Data - KGC Takeover Interview w/ Philippe Höij and Guest Host Ellie Young Apr 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Ellie's LinkedIn: https://www.linkedin.com/in/sellieyoung/

Philippe's LinkedIn: https://www.linkedin.com/in/hoijnet/

Philippe's Twitter: @hoijnet / https://twitter.com/hoijnet

DFRNT website: https://dfrnt.com/

Knowledge Graph Conference website: https://www.knowledgegraph.tech/

Free Ticket Raffle for Knowledge Graph Conference (submissions must be by April 18 at 11:59pm PST): Google Form

In this episode of the Knowledge Graph Conference takeover week, special guest host Ellie Young (Link) interviewed Philippe Höij, Founder at DFRNT.

At the wrap-up, Philippe mentioned that data architects should be able to communicate in ways other than PowerPoint. We need new and better ways to express ourselves, and the way things are connected. We will always need metadata around our data, we need text to express our ambiguity; but we don't have great ways to express things that are slightly ambiguous - not fully formed but also mostly known. A good tool allows to more easily query your model of the world to iterate and increment on it. That is where knowledge graphs can be the most helpful. Ellie responded, "It's not difficult, it's just complicated."

Philippe shared his journey towards knowledge graphs, especially thinking about the AIDITTO project he and a team built out of the "Hack the Crisis Sweden" event in 2020 around COVID-19. He needed a way to prototype, visualize and collaborate on data and the connections between data at scale. A regular data model does not convey enough information about what the data is and how it relates.

Ellie then shared some insight into the difficulties around collaborating on data across organizations and people in her climate change work at Common Action. Collaborating across organizations, all with different ways of working, you need a common "language" or way of communicating relative to data but can't easily develop a shared schema. Knowledge graphs provide incremental capabilities for collaboration.

Philippe talked about when collaborating across organizations, you still have the needs master data management (MDM) tries - and often fails - to address but there is zero capability to manage the other organizations' data flow into the shared data "pool". Philippe was having issues with open-ended knowledge graphs like SparQL or OWL - they needed composable data structures to be able to be flexible in the case they can't fully decompose a concept, especially as the concept or their understanding of the concept evolves.

For Philippe, TerminusDB was a big win because it allowed for composable data structures and much easier querying across the graph. Ellie discussed the origins of TerminusDB being about collaboration across many entities/organizations so it has a much different approach to accepting data that doesn't necessarily conform to a schema or data model. The "git for data" concept in TerminusDB also really was a big win for Philippe as it made experimentation much easier.

Ellie shared some of the challenges in her work at Common Action around working with many different entities, many of which are small and not that data literate - or "data native" - and how they need to enable collaboration without rigidity as things are so dynamic. Philippe discussed the need for enabling people to collaborate in a "messy" environment - the world is changing and trying to spend all your time and effort categorizing it into a single schema isn't realistic. He believes you need to enable collaboration in a truly distributed environment; the value is driven by micro level action via autonomy - people making progress in their own domain - which creates global value. Too often, we've tried centralized collaboration and it doesn't work. The collaboration shouldn't be a heavy overhead to driving that value - how can we flip the script to make the collaboration the enabler?

Philippe shared about how knowledge graphs can be used to manage compliance with security standards. You can map out much more easily who has responsibility for what and even identify gaps in your compliance adherence process. Being able to query that information easily makes it far easier to make sure you are identifying and mitigating risk. Ellie talked about breaking things down into paths for what is happening, what is not happening, and then what needs to happen to actually hit future goals. She mentioned it's a new way to interact with change and the unknown.

For Philippe, we need to start somewhere in breaking down the complexity and visualize what's going on; what are the patterns that we can see? Let's model and share them in a low complexity enough way. You can start to see the concepts and connect them in a way that we as humans can understand. It's almost like building a hivemind concept - each brain has it's own context and then when you share that context into the greater whole, it's impossible to know what incremental information or knowledge will be generated but it's almost an inevitability that it will happen. Many more patterns will emerge - patterns of patterns. But we need to be able to share that context in some way to have those patterns emerge.

Ellie shared thoughts about what is complexity - just something that is so big and/or gnarly that it is - at best - difficult for a human to understand it all. Things are so interconnected, you can't just adjust one piece or aspect. BUT it's okay to not understand the absolute complete picture, we can move forward with confidence and/or identify the most likely best challenges to address. Philippe believes it is often sufficient to understand directionality to move forward and make progress. Knowledge graphs help us deal with complexity and capture aspects of complexity in a way that is more understandable. How can we unravel the giant knotted ball of yarn one bit at a time instead of mapping out the entire unraveling process ahead of time? Ellie talked about knowledge graphs creating better information flow about what impact changes or prospective changes to data we are sharing will have on downstream consumers.

Philippe mentioned that knowledge graphs aren't great for every use case - look for the places where it really makes sense. Look at the specifics. While you can use graphs to manage the interconnectivity between the data, not all relational structures benefit.

Then Philippe discussed what he is working on and when to release it - he's focused on making data modeling much easier but doesn't want it to be overly technical. He wants to guide and enable the changemakers. It will be about enabling the collaborative aspects of knowledge graphs so people can have much better conversations about the data. And then people can express changes in data instead of in a PowerPoint.

Ellie mentioned what Veronika discussed in her episode, that you can model your data in the same way as you are used to in a knowledge graph but it's as a form of data immediately rather than taking it and transforming it into data. We have to talk to each other to discuss our data conventions and develop a new relationship for business users to data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#58 No, You Don't Sell a Data Mesh: Vendor BS in Data Mesh - Mesh Musings 10 Apr 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Scott gets grumpy and calls out some misbehaving vendors. And adds some fun new music to the episodes.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#57 Using a Knowledge Graph for a Data Marketplace and Data Mesh for Retail - KGC Takeover Interview w/ Olivier Wulveryck and Guest Host Ellie Young Apr 12, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Ellie's LinkedIn: https://www.linkedin.com/in/sellieyoung/

Olivier's LinkedIn: https://www.linkedin.com/in/olivierwulveryck/

Knowledge Graph Conference website: https://www.knowledgegraph.tech/

Free Ticket Raffle for Knowledge Graph Conference (submissions must be by April 18 at 11:59pm PST): Google Form

In this episode of the Knowledge Graph Conference takeover week, special guest host Ellie Young (Link), founder of Common Action, interviewed Olivier Wulveryck, Senior Consultant and Manager at OCTO Technology.

In the first two thirds of the interview, Olivier and Ellie chatted about a lot of concepts specifically around data mesh and then they linked in the concepts around knowledge graph in the last third.

For Olivier, a knowledge graph is the map for the data that is available - each data product or node in a data mesh is the representation of the knowledge within the organization. The knowledge graph is the abstraction of that knowledge across the data mesh - a logical representation on top of the data mesh nodes to help people make sense of the data mesh.

Currently, Olivier is working with a client sharing their data in a data marketplace. They are working on implementing a knowledge graph on that but not on their internal data. If they are seeing value from applying a knowledge graph externally, they may apply to their internal usage.

Olivier shared his view that it's easier to start with a data mesh than a knowledge graph - any first steps with a data mesh will bring you value. It is not the same with knowledge graphs - you have to do more work to get to value with knowledge graphs.

Olivier previously worked on the operational side of software engineering. He realized they had lots of data sitting in databases but the data was just a consequence - it was state data, there was no temporal dimension. He wanted to apply machine learning to the data but he was suffering from low quality data, just like the data team. The producing team never really thought about the data to be shared with others. For Olivier, the way to fix this is to put data back at the center of the domain and flip the script - make the operational datastore the consequence of changes in the data.

Olivier believes there is a need to think about the semantics of the data as it will be used by the rest of the organization - data can no longer be just an internal asset of the domain. If we believe we need the data to actually be used, we need that data to actually provide value. Make the data usable and useful.

To get specific, Olivier shared a use case of a clothing retailer. They might be getting people's body type and measurements to help them choose clothes that fit better. But the company could also use the data to make better fitted clothes for a broader range of their customers. They might have more insights to change the way they tailor or design their clothing.

Ellie asked about can we standardize how we capture and share data. Olivier is not sure if we can harmonize how we capture data. So instead, we need to harmonize on aggregation and integration. Olivier also talked about the challenges around finding the equilibrium between data consumer and data producer needs/wants.

Per Olivier, adapt is better than adopt. There is no by-the-book way to implement data mesh because that would never be applicable to any real organization. Olivier also mentioned the need for a framework for how teams will communicate and work together, e.g. Team Topologies.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#56 Insights from Deploying Data Mesh and Knowledge Graphs at Scale - KGC Takeover Interview w/ Veronika Haderlein-Høgberg and Guest Host Ellie Young Apr 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Ellie's LinkedIn: https://www.linkedin.com/in/sellieyoung/

Veronika's LinkedIn: https://www.linkedin.com/in/veronikahaderlein/

Knowledge Graph Conference website: https://www.knowledgegraph.tech/

Free Ticket Raffle for Knowledge Graph Conference (submissions must be by April 18 at 11:59pm PST): Google Form

In this episode of the Knowledge Graph Conference takeover week, special guest host Ellie Young (Link) interviewed Veronika Haderlein-Høgberg, PhD. Veronika was employed at Fraunhofer-Gesellschaft at the time of recording but was representing only her own view and experiences. She was invited for her special mix of both data mesh and knowledge graph know-how.

At Fraunhofer-Gesellschaft, Veronika's employer up until recently, she and team were currently implementing a knowledge graph to help with decision support for the organization. And previously, Veronika worked on a data mesh-like implementation as part of the Norwegian public sector at the Norwegian tax authority before the data mesh concept was really congealed into a singular form by Zhamak.

Veronika and Ellie wrapped the conversation with a few key insights: to share data, groups need to agree on common standards to represent it, and they also need to be able to share information with each other about that data into the future. To develop these initial data standards, and to build the relationships to coordinate around that data long term, different departments in the enterprise have to converse with each other. Building conversations across departments requires also building trust, and for this curiosity is a crucial ingredient, both on the individual level, but also at the domain and organizational levels. If people don't feel comfortable asking questions, they can’t understand each other’s perspectives well enough to contribute to that shared context.

What does this look like in practice? Different departments coming to discuss the difference definitions they have of different terms, and finding out what data they need from each other, therefore, what data they must collect and protocols they must develop. And, computer scientists discussing data with business people—understanding both what business requirements are, and conveying the needs of data systems in order to provide organized, quality data.

Veronika's recent organization, Fraunhofer, is using a knowledge graph as they need to make their investment decisions much more data driven. They need to do analysis across many different sources - they have some slight control over internal data sources but essentially none over external sources. They are repeatedly doing harmonization across these sources, often the same harmonizations. Veronika believes they shouldn't have to do the harmonization manually, so they needed a translation layer - the knowledge graph.

To build out their knowledge graph, they need business experts to work with the ontology experts - however, it is a struggle for time and attention from the business experts and they need to learn the importance and how to do ontologies. This is when Ellie mentioned that by centralizing the integration, it might cost a lot of effort up front, but it's necessary if you only want to do the harmonization work once.

For Veronika, thinking in the data as a product mindset and having data owners is crucial to getting a knowledge graph implementation right. She said a knowledge graph is a different way of expressing and sharing your knowledge, it's just in a way that computers can also understand it - not just humans. That framing helps people to understand why knowledge graphs are useful and important.

Data mesh implementation background and insights:

At the tax authority, Veronika's team of information architects were working to translate tax law into data models. They discovered the need of a common methodology to create the models - they chose UML - and then other authorities needed to use the data so the team began working to create data standards for efficient data sharing. The Norwegian data mesh implementation has even extended into a public/private partnership. Veronika also mentioned that Denmark now has standards for how bills are written so the legal aspects can be translated into data.

Per Veronika, their data mesh journey started from pains - they really struggled to consume data from other entities as well as internally. They started by making it easy to consume data from those other entities without creating a large burden on those producers. They learned about good - and bad - practices on sharing data and which tools were best to use for data modeling. All the decisions were in small partnerships.

For Veronika, a big key to success was taking on small partnerships and making an environment where asking questions was highly encouraged and even making mistakes was okay - this has been a recurring theme in many DMR episodes. Ellie made points that the community and communication are key to being able to make something like data mesh work. Veronika followed up that culture and "fuzzy factors" are more important than tooling and even methodologies.

Veronika discussed the silent fear that change brings and how that is such an impediment to getting things done. There is also a fear of looking silly when asking questions. So we need to work with people to get to a place far more comfortable with change and where curiosity is rewarded instead of shamed or looked down upon. Asking questions gives people the ability to grow as they learn new things and gives a conduit for far more information sharing - documentation can't be the end-all, be-all.

Ellie mentioned the need to stop focusing so much on the specific data in data processes. There needs to be a much bigger focus on people and the data creation process. Think about creating data for others to ingest with intentionality.

For Veronika, she saw the organizational impact of implementing a data mesh very strongly - it led to a greater sense of doing meaningful work, which led to far lower churn across the organization. Those cross functional implementations had a strong impact and the people working on those implementations were far more engaged. Veronika believes the fear of data mesh silos in data mesh makes sense but people naturally want to prevent those silos so allow people to focus on consumer needs.

Veronika believes knowledge graphs are key to preventing silos in data mesh and somewhat vice versa - it is very difficult to maintain a knowledge graph across an organization that doesn't think of data as a product.

The two discussed the importance of data reuse, even in the same use case, to prevent manual work. It's like a golden source-like concept - you don't have to figure out which data you can trust. That manual repetition of harmonization is what kills productivity and people's desire to work with outside data.

Veronika made an interesting point that any time you model data, it is almost like a mini knowledge graph. Each data model has a tiny ontology of the domain.

At the Norwegian Tax Authority, the working groups around the initial data mesh implementation started very informally - people who knew each other beforehand. In a larger org, Veronika believes the key to success is talking, sharing what you are working on and talking about how it might work. Specific goals are very important, you need concrete deliverables which make it easier to get funding.

In the Norwegian government, they worked closely with lawmakers to define business concepts and initially thought that everything had to harmonize but found that it was more important for each domain to define their business concepts the way they understand it and make it transparent - it isn't possible to make all data harmonizable, the world is full of variation. You need to think about your business concepts and then not force them on others - focus on the translation. Otherwise, you will never get to publishing anything. They didn't go quite as far as the Danish government though, which has created a role in the law making process, where there is someone who understands data enough that they can word the law to be able to translate it into data or a data model.

Veronika believes you shouldn't use terms to identify concepts, use URIs. Terms are just a label, not the concept. She also believes it is often not necessary to make things computer readable and that you need to focus on creating a living organization.

To finish up, Veronika reiterated: don't be afraid, be curious - curiosity is a prerequisite for success with data mesh or knowledge graphs.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#55 Just What is a Knowledge Graph - Short Answers to a Tough Question Apr 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Knowledge Graph Conference website: https://www.knowledgegraph.tech/

Free Ticket Raffle for Knowledge Graph Conference (submissions must be by April 18 at 11:59pm PST): Google Form

Thank you to our contributors! You can find additional introductory resources below.

Contributors and their contact info:

Karen Passmore, CEO and Founder of Predictive UX

Karen LinkedIn: https://www.linkedin.com/in/karenpassmore/

DMR Episode 30

Xhensila Poda, Machine Learning Engineer at CARDO AI

Xhensila's LinkedIn: https://www.linkedin.com/in/xhensilapoda/

Jens Scheidtmann, Lead Architect at Bayer

Jens' LinkedIn: https://www.linkedin.com/in/jens-scheidtmann/

Juan Sequeda, Principal Scientist at Data.world

Juan's LinkedIn: https://www.linkedin.com/in/juansequeda/

DMR Episode 14

Steve Stesney, Senior Product Lead and Data Practice Lead at Predictive UX

Steve's LinkedIn: https://www.linkedin.com/in/stephenstesney/

DMR Episode 30

Tim Tischler, Principal Engineer at Wayfair

Tim's LinkedIn: https://www.linkedin.com/in/timtischler/

DMR Episode 43

Further Introductory Resources (provided by Ellie Young and Juan Sequeda):

What is a Knowledge Graph video by Martin Keen: https://www.youtube.com/watch?v=y7sXDpffzQQ

Knowledge graphs: Introduction, history, and perspectives (paper): https://onlinelibrary.wiley.com/doi/10.1002/aaai.12033

Knowledge Graphs book (free): https://kgbook.org/

Knowledge Graphs paper by Juan Sequeda and Claudio Gutierrez: https://cacm.acm.org/magazines/2021/3/250711-knowledge-graphs/fulltext

Juan's book "Designing and Building Enterprise Knowledge Graphs", 25% discount using code DATAWORLD

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

KGC Takeover - Weekly Episode Summaries and Programming Notes - Week of Apr 10, 2022 - Data Mesh Radio Apr 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#54 Data Mesh Evaluation and Implementation Insights - Interview w/ Steven Nooijen and Guillermo Sánchez Apr 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

GoDataDriven Data Mesh Webinar: https://godatadriven.com/topic/webinar-data-mesh-9-feb-2022-thanks/

GoDataDriven Self-Service Whitepaper (info-gated): https://godatadriven.com/topic/data-democratization-whitepaper/

Steven's LinkedIn: https://www.linkedin.com/in/stevennooijen/

Guillermo's LinkedIn: https://www.linkedin.com/in/guillermo-s%C3%A1nchez-dionis/

In this episode, Scott interviewed two people from the European data consultancy GoDataDriven - Steven Nooijen, Head of Strategy, and Guillermo Sánchez, Analytics Engineering Tech Lead.

Guillermo started off by talking about how for the last ~3 years, he was seeing the data engineering team as the bottleneck before data mesh came onto the scene. For Steven, they were seeing lots of companies that were building out data platforms, especially data lakes, and then not really getting the promised benefits so data mesh made sense.

All agreed data mesh is not right for every company and then mentioned some good signs that an organization should consider data mesh. Guillermo pointed to a lot of the usual suspects: size of company, size of data team, how many data consuming teams do you have, how many data sources do you have, etc. He then gave a specific example: if you have a data analyst in a consuming domain that has to wait more than 1 week for data, there is a bottleneck somewhere. Is it centralization? Not sure but time to investigate and that might be where you start to consider data mesh.

Steven gave the example of an even earlier indicator that bottlenecks are occurring: teams start to hire their own data people rather than leverage the central team. Guillermo also pointed to the rise of consuming teams getting direct data access from producing teams instead of going through the data team.

Guillermo made a very crucial point: data mesh is really about interfaces. People talk about data products being the technical communication interfaces between teams. So as long as people trust the data product and teams adhere to interface standards, data mesh can work. But that trust is earned, not given.

Both Guillermo and Steven talked about the need for an easy path, a golden path. With that golden path, it can be relatively simple and low friction for domain teams to share their data for the vast majority of use cases. There is also a need for some quality control to make sure data products are trustworthy, have reasonable SLAs, etc.

Per Steven, what they've seen to date at customers is most domains are happy to follow that golden path. But, you certainly need that path to start out where they are - trying to make them use foreign tooling/workflows is going to be a tough sell. They are also leveraging the concept of a "certified" data product label. The core data mesh implementation team awards this to high-quality data products and it's easy for teams using the happy path get - it is harder to get certified if you use your own tooling. This acts as a quality control measure.

Steven talked about when evaluating where on the centralization/decentralization slider decisions should fall, you should really look at domain maturity and organization maturity. Centralization provides more support, if being more of a bottleneck, in many cases so there is a trade-off. Are domains really ready to take on the full responsibility of whatever task you are considering? It's okay to not have a one-size-fits-all approach.

Somewhat atypically, Guillermo and Steven recommended starting your proof of concept, if you need to do one, with a single domain. They recommended starting with an eager domain and to make sure they have support from the platform team. Pick a consumer-aligned domain with a high demand for data. Again, this advice runs somewhat counter to what many have said and that is absolutely valid.

Instead of the proof of concept, Steven recommends driving high level buy-in via a C-level sponsor. That way, you have time and funding to get things moving. Obviously easier said than done but if you can go this route, your journey is likely to be smoother as you won't have to constantly be looking for funding. When driving buy-in via the C-level sponsor, Steven mentioned never once using the phrase "data mesh". The execs really responded to using "self-service" and the 4th industrial revolution concept.

When driving buy-in, Guillermo talked about directly telling the teams what does this mean explicitly for them. You have to make it mutually beneficial!

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#53 To Analytics Engineer or Not to Analytics Engineer - That's a Question - Mesh Musings 9 Apr 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

A mesh musings asking if we should embed data/analytics engineers into the domains to serve as the data product developers or not.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#52 Data Mesh Data Governance: Getting Out of Your Own Way - Interview w/ Sarita Bakst Apr 04, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Data Mesh Learning Meetup Presentation: https://www.youtube.com/watch?v=7iazNKG8XQo

Sarita's LinkedIn: https://www.linkedin.com/in/saritabakst/

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#51 A DevOps Angle to Data Mesh and WePay's Journey - Interview w/ Chris Riccomini Apr 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

What the Heck is a Data Mesh?! Post: https://cnr.sh/essays/what-the-heck-data-mesh

Chris' Twitter: https://twitter.com/criccomini

Chris' LinkedIn: https://www.linkedin.com/in/riccomini/

Chris' website: https://cnr.sh/

The Missing README book: https://themissingreadme.com/

In this episode, Scott interviewed Chris Riccomini, a Software Engineer, Author, and Investor. Chris led the infrastructure team at WePay when they embarked on a data mesh journey and made a well-written post on thinking about data mesh in DevOps terms.

Like a number of people/organizations that have come on the podcast, at WePay, Chris was pursuing the general goals of data mesh and was applying some of the approaches as well - but it was not nearly as cohesive as Zhamak laid things out.

Their initial setup had two teams managing the pipeline/transformation infrastructure. Chris's team was mostly handing the extracting and loading and then there was a team of analytics engineers handling the transformations. The Transformation team saw a major increase in demand and quickly became overloaded -> a bottleneck. Chris' team also started to get overloaded so they knew they had to evolve.

One way the team started to address the bottlenecks was by decentralizing the pipelines. Teams could make a request and a scalable and reliable pipeline would essentially get automatically set up for them. WePay is in the financial services space so as part of those pipelines, to prevent risk, teams could mark their sensitive/PII columns and the infra team also put in some autodetection capabilities to make sure they didn't miss any.

WePay created a "canonical data representation" or CDR, which is pretty analogous to a data product in data mesh. Chris really liked WePay's use of the embedded analytics engineer to serve as a data product developer.

One key innovation for WePay was tooling to enable safe application schema evolution. They looked for things like dropped columns and had more comprehensive data contract checking mechanisms. It allowed developers to test changes pre-commit. 80-90% of data breakages were things the developers had no idea would cause an issue and they reverted those changes. 10-20% of the time, the developers still wanted to go through with the changes and that kicked off negotiations with data consumers. That forced conversation was very helpful for a few reasons.

Chris talked about standardizing around technologies for the platform but allowing teams to roll their own if they wanted. But they were super clear with those teams that the infra team wouldn't support those other technologies, even if it was okay to use it. He also sees a major need for an API gateway concept for data. Currently, everything around versioning, auto-documentation, etc. is way too manual and high friction.

Chris talked about taking the learning from DevOps and applying them to data mesh. One good one to look at, per Chris, is the embedded SRE concept - should you do the same with a data or analytics engineer? There is also a need for many standards and replicable patterns. They launched a data review committee, similar to a design review committee, that helped come up with standard data models and other standards.

Sherpas not gatekeepers - build out your review functions as councils to guide and disseminate knowledge. The team's role should be about assisting where they can, being a trusted partner. And what WePay saw was as people went through more reviews and similar, they saw there was less of a need for them as people learned what good/best practices were.

Lastly, Chris wrapped on a point others have made about the cost of making a mistake. You need to make it as low as possible. Back processing data can be very costly in time and compute.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Apr 4, 2022 - Data Mesh Radio Apr 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#50 Evaluating if Data Mesh is a Fit; Team Structures; and WTF is Federated Computational Governance - Interview w/ Marius Ingjer Mar 31, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Marius' LinkedIn: https://www.linkedin.com/in/marius-ingjer-a155313/

Marius' Medium: https://medium.com/@marius.ingjer

In this episode, Scott interviewed Marius Ingjer, Co-founder and Senior Consultant at Knirkefritt AS. Marius has been working with a few clients on implementing data mesh.

They covered 3 distinct topics: evaluating if data mesh is a fit for your organization, team structure challenges in data mesh or data mesh-like implementations, and a simplified definition of federated computational governance.

Evaluating if data mesh is right for you:

To start, Marius provided a list of evaluation questions to help you determine if data mesh might be right for you:

How many data sets are you producing?
What is the lead time to creating a new dataset?
How well are your datasets serving your data needs?
How many domains do you have?
How complex are your domains?
How does the team respond to new data requirements?
How usable in general is your data?

Every company wants to share data well but the centralized data team isn't the bottleneck yet for many. Centralization can add a lot of value until it starts to become more hurtful than helpful and yes, figuring out that point is easier said than done. Centralization of data fights Conway's Law and can become way too much cognitive load so it will eventually become an issue for many organizations.

A key question in evaluating if data mesh is a fit: what is the cost of allowing your data processes to fail? Per Marius, the business consequence of failed reports has historically not been that high. But if you are driving business decisions, whether that is ML or just crucial day-to-day decisions on your data, data mesh might become more attractive.

Team structures and challenges in data mesh:

In general, it's important to understand that implementing data mesh will cause cultural challenges - Marius believes developers generally don't want to ALSO share their data. It's additional work so you have to align incentives, which is far easier said than done.

That additional cognitive load on developer plates is very crucial. We need to make we address that load to not burn them out. That means realigning incentives but also having extra help with things like grooming the work backlog. Providing extra resources helps but that is more about tackling the work, not handling the increased cognitive load. And learning about how to do data well is a pretty big learning task.

Marius recommends giving teams the extra resources but also reshaping the team and business structure, such as the KPIs, to effectively prioritize and shape the requirements. He also recommends having a stick, not just carrots, or teams will try to just opt out.

Simplified definition of federated computational governance:

When talking about this topic with developers right now, it feels far too complex. To make it less complex, reduce the friction to developer decisions but not add much to cognitive load. An example might be providing easy data masking tooling for PII or extensible data APIs so developers can focus on the value-add.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#49 Differentiating the Baby from the Bathwater in Data Mesh - Mesh Musings 8 Mar 29, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

An episode about how to figure out what is good to reuse from your existing data approach and what to toss out when it comes to data mesh.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#48 Overcoming Obstinate Organizational Obstacles in Data Mesh - Interview w/ Scott Hawkins Mar 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Scott Hawkins' LinkedIn: https://www.linkedin.com/in/scott-hawkins-8934393/

In this episode, Scott interviewed Scott Hawkins, Principal Data Architect at ITV.

Per Scott, there are 3 good ways to drive buy-in for the domain teams:

At the senior level - so it trickles down as the management for the team is bought in.
Via a strong carrot - solve a problem for them as a kind of quid quo pro / mutually beneficial solution. Trying to solve an unrelated problem will drive lower buy-in.
Work on realigning the team KPIs/OKRs with the senior leaders to actually realign incentives.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#47 Skipping the Fluff of Domain Driven Design for Data Mesh - Interview w/ Lorenzo Nicora Mar 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

Lorenzo's LinkedIn: https://www.linkedin.com/in/nicus/

Pat Helland Data on the Inside/Outside: http://cidrdb.org/cidr2005/papers/P12.pdf

Mesh-AI careers page: https://www.mesh-ai.com/join-us

In this episode, Scott interviewed Lorenzo Nicora, Principal Data Consultant at data mesh and AI focused consultancy Mesh-AI.

Scott asked Lorenzo to be on to continue the series of interviews on domain driven design (or DDD) for data. It is a topic that many are struggling with so having lots of perspectives on it is crucial. On the episode title, a key output was explicit permission to skip a lot of the tactical patterns of DDD. Others have also said similar things but I wanted to make sure it was explicit.

Before we jump into the DDD parts, Lorenzo made a good point on your data mesh Proof of Concept / starting your journey. You need to start with manageable problems. Start with a consumer-driven problem but a source/producer-aligned data product. There is a lot of nuance in the interview on why this matters.

Per Lorenzo, identifying the domains is crucial but it is the hardest part of DDD. That shouldn't scare you because you can start with things being a bit blurry. It's important to understand your high-level domains but you can get moving without mapping out all of your domains.

A key theme from Lorenzo: the language is at the center of everything in DDD. It is part of the data modeling and it goes all the way down to the code.

Per Lorenzo, DDD is all about communication, knowledge capture, and knowledge sharing. Knowledge capture is about extracting knowledge and then writing it down. Knowledge sharing is about finding scalable ways to share context.

Some advice/pointers from Lorenzo:

Teams have to truly understand the language of their own domain - remove the ambiguities, even if that feels like it's putting in too much work.
Event storming is a great way to approach tackling DDD for Data.
Event sourcing is crucial for modeling the problem of the domain.
Terminology is very key - identify the domain experts who can find/choose the right name for each concept.
Keep a live document of terms and meanings - keep it updated!
Encourage everyone to use the identified terminology when naming and in the code directly.
Find your high-level domain first instead of your granular sub-domains.
Ask your consumers for their specific data asks and then back into what would be a good data product or set of data products to start with. Again, look for high return, low effort/investment to get some wins under your belt and build your muscle memory.

Some key things to understand:

Language changes - it changes across time and across the organization. The same words mean different things or different words mean the same thing. A major weakness of the central/enterprise data warehouse is the inability to easily deal with changes through time or nuance across the organization.
When you first identify your domains, the boundaries might be blurry and that's okay!
Data contracts are really crucial and the semantic issues, not the schema, are the most important - and hardest - part. And you can't just break contracts, there has to be a reason or no one will trust it is an actual contract instead of just a pub/sub model.
Study up on and really think about your data on the inside versus data on the outside. If you aren't familiar, there is a link in the show notes to Pat Helland's work on the concept.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Mar 27, 2022 - Data Mesh Radio Mar 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#46 Designing a Data Literacy Approach for Data Engineers - Interview w/ Dan Sullivan Mar 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here.

Dan's LinkedIn: https://www.linkedin.com/in/dansullivanpdx/

Dan's Email: dan.sullivan at 4mile.io

In this episode, Scott interviewed Dan Sullivan, Principal Data Architect at 4 Mile Analytics.

A key point Dan brought up is tech debt around data. Taking on tech debt should ALWAYS be a very conscious choice. But the way most organizations work with data, it is much more of an unconscious choice, especially by data producers, who are taking on debt that the data engineering teams will have to pay down. We need to find ways to deliver value quickly but with discipline.

Zhamak has mentioned in a few talks that data engineers soon may not exist in orgs deploying data mesh. Dan actually somewhat agrees that data engineering will change a lot as right now, there is a big rush to build out the initial iterations of data products (the industry definition). Going forward, Dan thinks there will be a need for data engineers that can really understand consumer needs and build the interactions, e.g. the SDKs, to leverage data.

Dan has 3 key pillars for driving data literacy for data engineers are domain knowledge, learning, and collaboration. Data engineers should pair with business people to acquire domain knowledge, they should be given the opportunity to spend time doing things like online training to learn, and they should collaborate across the organization instead of just being ticket tacklers.

Per Dan, not all data engineers are the same depending on background - some come from a data analyst/data science background but many come from a software engineering background. So we can't treat training all data engineers as if it's the same. But we do need them to have a well-rounded background. A big need is for them to understand more about the data consumers and/or the producers so embedding them in the domains can really help.

For driving buy-in with data engineers, Dan points to the problems typically being around incentives. Data engineering is often hampered by organizational issues and a lack of clear direction. So if you can tackle those, you can often win over DEs.

In any organization but especially in one implementing data mesh, standards, protocols, and contracts are all very important. However, most data engineering teams are not given the time to create them. They take a lot of effort and are hard to get right!

Dan talked about how data can take a lot of useful practices from Agile, especially the fast-cycle feedback loop. And that data people really need to think more about the user experience (UX) for data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#45 Data Governance in Data Mesh: Address the Micro and the Micro - Interview w/ Mohammad Syed Mar 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Mohammad's Latest Data Governance article: https://www.linkedin.com/pulse/some-thoughts-data-governance-mohammad-syed/

Book mentioned: Disrupting Data Governance: A Call to Action by Laura Madsen

In this episode, Scott interviewed Mohammad Syed, Lead Strategist - Data at Caruthers and Jackson about how data mesh governance has to be different from what we've done historically.

Per Mohammad, data governance in data mesh is very different to doing governance for either a data lake or a data warehouse. The warehouse has a focus on high-level quality and usability but at the expense of context and agility. Data lake is about metadata and lineage but at the severe expense of usability - schema on query is not fun for consumers - and often quality.

For most data organizations, governance has been very macro focused - governing the data warehouse or lake as a whole. That is part of why data governance has become a major bottleneck - the focus is on the macro but the individual requests are the micro.

In data mesh, governance can shift to being about maximizing the value of the data instead of mostly preventing risk. Of course, there is a balance between local maximization - the value of each data product - and global maximization - the value at the overall data mesh level.

A key focus to data mesh data governance is enabling - especially enabling the domains to govern their data products. Mohammad made the point that you need to enable your domains by creating the technical and business definitions of a "good" data product. Then the governance team needs to teach teams about the quality definitions, e.g. data product consumability. There is a need for policies of course but mostly focus on frameworks to enable policy creation and enforcement - decentralize!

A key point Mohammad made was: governance only works with informed governors - you must teach domains to govern properly. Transparency is key to make data governance work.

Mohammad emphasized the "good" data product definition leads to the separation of data quality and data product quality. A data product might be more valuable for other reasons - or less costly - by having relaxed data quality standards. In a data warehouse implementation, there is really only a single definition of "good" quality, but that just won't work in data mesh. We really need to develop better frameworks for what data quality means at the micro level.

To get data governance right, strategy and maturity are crucial - what are you actually trying to accomplish? Data mesh for the sake of data mesh is worthless, just like any other paradigm.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#44 A Pragmatic Approach to Getting Started with Data Mesh at Northern Trust - Interview w/ Khanh Chau Mar 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/

In this episode, Scott interviewed Khanh Chau, Lead Architect for the Data Mesh Initiative at Northern Trust.

A big focus and a key driver for Northern Trust has been building muscle and learning/evolving along the way. It's important to evolve quickly and not build muscle in the wrong way.

There are a lot of great learnings to takeaway from the Northern Trust journey.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Mar 20, 2022 - Data Mesh Radio Mar 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#43 Applying Resilience Engineering Practices to Scale Data Sharing - Interview w/ Tim Tischler Mar 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Books/posts/papers mentioned:

Blameless PostMortems and a Just Culture by John Allspaw - Link

The Theory of Graceful Extensibility: Basic rules that govern adaptive systems by David D Woods - Link

The Field Guide to Understanding 'Human Error' by Sidney Dekker - Link

In this episode, Scott interviewed Tim Tischler, Principal Engineer at Wayfair. Prior to Wayfair, Tim worked as a Site Reliability Champion at New Relic and is well known in the "human factors" and resilience engineering space.

Per Tim, our current work culture is overly action-item driven - every meeting must have a set of agenda items generated from it. This prevents people from having learning-focused meetings exclusively designed for context sharing. Humans' brains work differently between learning and fixing mode and we ask totally different questions. To be able to scale our knowledge sharing, we need to have the space to have learning-focused meetings.

A good way to center learning-focused meetings, be they "show and tell" or event storming sessions, is via sharing stories - human communication is founded on story sharing through the millennia. Tim's "show and tell" and event storming sessions at Wayfair have had extremely positive reviews so far.

Tim sees ticket-based interactions - just throwing requirements on someone's JIRA backlog or similar - as fundamentally flawed. If Team A gives Team B requirements, Team B just looks to close the ticket versus getting both sides in the room to exchange context and have a negotiation. Tim prefers two modes of interactions over ticket systems: #1 - no human-touch, automated interactions, e.g. an API; and #2 - high touch, high context sharing interactions.

For resilience engineering specifically, you should apply learnings to each data product AND the mesh as a whole. Part of that is a broad acceptance that you are in a highly dynamic and highly changing org - there will be changes! A few anti-patterns to resilience engineering that apply to data mesh are: 1) a hub and spoke relationship model where one person is the key glue - this is bad at a human level and even worse at a technical level :); 2) business leaders pushing for metrics without sharing the specific context as the results end up as completely empty and useless things you are tracking; and 3) not embedding people building platforms into the teams they are building the platform for - they must really understand the workflows.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#42 Self-Serve Consumption Means Empowerment, Not Chaos - Interview w/ Ust Oldfield Mar 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Ust's LinkedIn: https://www.linkedin.com/in/ust-oldfield/

Ust's Twitter: @UstDoesTech / https://twitter.com/UstDoesTech

In this episode, Scott interviewed Ust Oldfield, Principal Consultant at Advancing Analytics. They covered the concept of self-serve from a consumer standpoint in data mesh, and some ideas around how to get it right.

According to Ust, the overall data and analytics industry is just starting to move from data consumers only consuming what others have prepared towards self-serve data consumption. But, it is important to still provide those prepared reports so 1) people are working from the same info / on the same page and 2) you give people an easy - and maintained - path to important business information.

Ust also mentioned one key to getting self-serve right is to not just enable consumers to get to the data they want, they really need to be able to understand what they are seeing so documentation, sample queries, and other similar tactics are very crucial. Consumers also need training on how to use the platform - in general, training for self-serve data consumption is very lacking across the industry right now.

How do we share information at scale? Forums? "Show and Tell"? Office hours? Neither Ust nor Scott had great answers just yet. Time will tell.

Ust finished with a recommendation for those building out their self-serve platforms for data consumption: spend a lot of time interviewing your data consumers to figure out what will empower them rather than just trying to deliver what you would want. Also, make sure to enable those who just want to consume data as prepared - those who want to be spoon-fed the info, that's fine, allow them to self-select as that is a valid approach to leveraging data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#41 Winning Over Application Developers: Agency not Autonomy - Interview w/ Jessitron (AKA Jessica Kerr) Mar 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Jessitron pithy tweet: https://twitter.com/jessitron/status/1471352190149087233

Jessitron Twitter: @jessitron / https://twitter.com/jessitron

Jessitron emails: jessitron @ gmail.com OR jessitron @ honeycomb.io

Jessitron Presentation: Principles of Collaborative Automation: https://jessitron.com/2020/08/03/talk-principles-of-collaborative-automation/

Jessitron blog post: To share the work, share the decisions: https://jessitron.com/2022/02/01/to-share-the-work-share/

Book mentioned: Why Information Grows: The Evolution of Order, from Atoms to Economies by César A. Hidalgo

In this episode, Scott interviewed Jessica Kerr more widely known as Jessitron who is a Principal Developer Advocate at Honeycomb.io. Or as many affectionately refer to her, the Empress of Software. You're probably gonna enjoy this one :)

Scott asked Jessitron on because she had a very pithy tweet about data mesh being "conscious design for unexpected use" and because she knows the developer mindset extremely well and many folks are having trouble working with the developers to get them bought in to sharing their data well.

Jessitron started off by discussing one of the big issues with application development: despite the tooling and process advancements of the last 20 years, it's all somehow only made application development harder. So the starting advice is don't just add more to their plate. It probably won't go well!

Her biggest point was giving application developers agency in how they share their data is key. Autonomy is just passing the responsibility over without the help. Application developers want guidance/direction to the target outcome, but they want to make the choices on how to achieve said target outcome while being given the resources to do so. The information and capability to do their job is key.

For driving buy-in, start with the why, not the ask. Let them know why their data is valuable. And BE SPECIFIC! The conversation should be about their potential impact, not just the negative of "you changed this and it broke" but the aspirational. You want them to start thinking about how can you work together to enable them to share their data in a high context / highly meaningful way.

Data mesh is going to be a big culture shift for application developers - you need to not just put something high priority on the backlog. You need to give them the space - meaning that they have enough points or whatever on their backlog - to really understand and learn how to share their data well. You have to focus on teaching them how - possibly via an internal hackathon to start building that muscle - or possibly even a cross functional pair programming-like initiative to show each other your ways of working and share knowledge. Also, show them the impact they are having along the way as they get going, that will motivate them to do more.

Be very conscious of language. The interview with the NAV team building the application platform talked about this a lot. Application developers and data people don't speak the same language. Work with them to put it into their language.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Mar 13, 2022 - Data Mesh Radio Mar 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#40 Getting Data-as-a-Product Right and Other Learnings From Adevinta's Data Mesh Journey - Interview w/ Xavier Gumara Rigol Mar 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Xavier's Twitter: @xgumara / https://twitter.com/xgumara

Xavier's LinkedIn: https://www.linkedin.com/in/xgumara/

Adevinta meetup presentation: https://www.youtube.com/watch?v=av6cT_r4orQ

Xavier's Medium Articles:

https://medium.com/adevinta-tech-blog/building-a-data-mesh-to-support-an-ecosystem-of-data-products-at-adevinta-4c057d06824d

https://medium.com/adevinta-tech-blog/treating-data-as-a-product-at-adevinta-c1dce5d394c5

https://towardsdatascience.com/data-as-a-product-vs-data-products-what-are-the-differences-b43ddbb0f123

Scott interviewed Xavier Gumara Rigol who has been helping lead Adevinta's data mesh implementation as Area Manager for Experimentation and Analytics Enablement. The discussed the data as a product concept and learnings from Adevinta's journey thus far. Xavi has put out some great articles and did a Data Mesh Learning meetup that are linked below.

One key aspect to data as a product is to understand the need for data product evolution, both relative to maturity and to what is consumed. This is a common theme in many data mesh conversations as historically, data consumption has resisted evolution and change. Consumers need to really understand that the business is evolving so what they consume will too. If you manage data products well, it won't be a sudden change but if we are trying to share insights into a domain, those insights will change. When thinking about data product maturity, it's totally okay to start by thinking of a data product as a single table or view.

Xavi also mentioned some pitfalls to forced data product evolution - e.g. getting it wrong as changes can be quite costly to backfill. Adding new attributes is easy but computing something for 3 to 6 months in hindsight can cost a lot of compute charges. To do do evolution right, versioning and deprecation plans are key.

To get data as a product right, Xavi recommends start by prioritizing which data you want to make available; this is a process, not a switch to flip. You should figure out which data is important for each domain and at the broader organization level.

Applying data as a product thinking to your data sets is easier said than done. While data mesh is a leading proponent, companies not doing data mesh can also use data as a product thinking - Adevinta started down this path before embarking on their data mesh journey. Of course, data as a product is far easier said than done.

For Adevinta's data mesh journey, they started with every data product being a single table. Data was originally centrally managed so interoperability was already established. However, the documentation was lacking and the general usability wasn't great. They spent their first few quarters just focusing on splitting their monolithic data production into separate pipelines for each domain instead of one giant cluster.

The giant cluster was becoming a major bottleneck as changes were hard and maintainability was getting harder every day. Now, each domain essentially has one data product but with multiple dimensions/tables. Each product is layered and each layer has different granularity and SLAs.

A few other notable points:

Xavi believes all data products should be accessible via SQL but definitely not only SQL.
Template/blueprints for data products are incredibly useful and important.
The tooling/practices to prevent application changes from breaking the data are just very lacking - Adevinta uses data model reviews but it's still not perfect.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#39 Driving Buy-In for Data Mesh - Mesh Musings 7 Mar 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Transcript provided by Scott Hirleman here

An episode all about driving buy-in by different personas for data mesh.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#38 Pursuing "Platform Thinking" Through Data Mesh - Interview w/ Eric Broda Mar 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Eric's LinkedIn: https://www.linkedin.com/in/ericbroda/

Eric's Twitter: @ericmbroda / https://twitter.com/ericmbroda

Eric's Medium (multiple posts on data mesh): https://medium.com/@ericbroda

In this episode, Scott interviews Eric Broda, an Executive Consultant in the financial services space. Eric shared his learnings from aiding a large financial services firm to implement data mesh from the infancy of the project.

Eric's big thesis for companies looking to be data-driven is to think of themselves as a platform for connecting supply and demand. The internal company may be the supplier, e.g. if a bank is lending money directly, but more often it is about being the platform - so here to match investors to consumers looking to take out a loan. Lowering the friction between both sides of your constituency is crucial and getting really good at data can help there.

To Eric, the technology is like plumbing - you expect it to work but most businesses at the high level don't care about it as long as it works. You don't buy a house for the plumbing.

Eric's big point of advice is that you shouldn't underestimate the organizational change required to do something like data mesh right. Plan for the change and don't try to skip the necessary change, that will lead to disaster. Speaking of organizational structure, Eric firmly believes that centralization of data ownership fails Conway's Law. While companies can overcome that with a LOT of effort, most don't get there due to fatigue.

When developing a new data product, Eric recommends to first start with expected usage patterns pretty explicitly via a 1:1 relationship model; at least early in a data product's life, the data produced needs to explicitly match the needs of the first target data consumer. This is a departure from data mesh recommended practice but seems to be somewhat of a common emerging pattern, at least in financial services.

Eric also stated his belief that master data management - or MDM - "is dead", especially in data mesh. It hasn't ever really worked and it's not worth trying to do it with data mesh. Time will tell on that one.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#37 Building a Self-Serve Platform Developers Will Actually Use - Interview w/ Audun Fauchald Strand and Gøran Berntsen Mar 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Gøran's Twitter: @gorzan / https://twitter.com/gorzan

Audun's Twitter: @audunstrand / https://twitter.com/audunstrand

Gøran's LinkedIn: https://www.linkedin.com/in/g%C3%B8ran-berntsen-66066517/

Audun's LinkedIn: https://www.linkedin.com/in/audunstrand/

NAIS Platform Website: https://nais.io/

In this episode, Scott interviews Audun Fauchald Strand and Gøran Berntsen of NAV. Audun is the Principal Engineer and Gøran is the Product Manager for NAV's NAIS application platform as well as their emerging self-serve data platform for data mesh called NADA.

They covered a lot of different topics including: 1) building out the platform; 2) working with consumers to set expectations for common data products; 3) definition of a data product - and how it will evolve; 4) setting the frameworks for producer teams and allowing them to own the production; 5) communicating across teams; and 6) Cake! No really, a secret to success is cake.

While NAV is early days in building out their data platform for data mesh, they are taking an interesting approach: work with the developers to set data product expectations and then see how the developers would go about creating those data products. Then, the data platform team will build the platform out to make developer workflows much easier. While Gøran, with a background as a data person, feels the pull to make the self-serve platform as data-centric as possible, he understands the need to make it developer friendly from his time building the application platform with those from a developer background like Audun.

They both talked about reducing friction, including via sensible defaults, as a big part of their path forward. Stop trying to make developers come up with everything themselves. While they are still early days on developing those defaults, they are comfortable in their process to get there. And working with developers along the way is key.

To start, NAV's definition of a data product is a single table or view. It will probably evolve to be more of a data set focus but they don't see a need to prematurely optimize or overcomplicate.

Gøran emphasized the need to have empathy for data producers, to build that into the platform. Teams, whatever the strategic direction, can choose where they focus their time. Don't try to force them to spend it on data, spend the time to really work with them. As Brian McMillan said: find the opportunistic data folks.

NAV tried to put analytics or data engineers into the domains but saw them sitting next to the team, not as part of the team. So they decided to rethink. Those data product developers were likely to become overly crucial to serving the data and thus were a likely single point of failure if they moved on.

Okay, the most important aspect: Cake! For each team that puts a data product onto the mesh, they give that team cake. As in, an actual cake. It might seem silly but it really does work. It makes it feel less daunting to publish a data product and a bit like you are just having fun. It also means the team can show off a bit when they get their picture out there with their cake in the company Slack. And then people can use that cake picture as a jumping off point for learning more about the data product they just shared. It really is a fun community-building hack.

This is a must-listen for anyone involved in building a self-serve platform for the application developers/data product developers.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Weekly Episode Summaries and Programming Notes - Week of Mar 5, 2022 - Data Mesh Radio Mar 06, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#36 Is "Data at the Core" the Way To Develop Applications - More on Data-Centric Application Development - Interview w/ Dave McComb Mar 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Semantic Arts website: https://www.semanticarts.com/

Dave's book: https://www.semanticarts.com/software-wasteland/

Contact Dave: https://www.semanticarts.com/contact-us/

Dave's LinkedIn: https://www.linkedin.com/in/davemccomb/

In this episode, Scott interviewed Dave McComb, the President and Co-Founder of Semantic Arts. Scott asked Dave on as part of the continuing deep dive into Domain Driven Design for Data and Data-Centric Application Development as Dave wrote the book on Data-Centric Application Development - literally.

Dave's overall argument is that most businesses really have very few "business events", ~500-2000 for even the largest companies. Those large enterprises may have 10K+ applications, each with their own data model and application model, leading to possibly 100M+ data attributes. All that leads to far more complexity than is necessary if companies just focused on building applications from the business events side.

They discussed the amount of work an application developer would need to learn to be able to do data-centric application development; while it is mostly about learning data modeling, especially for graph databases, Dave has seen the application developers really not want to move to this model. This has meant a slower roll-out at a number of clients than if they were embracing it.

Scott asked about the user experience (UX) in data-centric application development, both for the data producer and data consumer. Per Dave, the UX is pretty lacking, especially on the data producer side so there seems to be a need for better developer tooling for graph databases. Despite the "crude" UX, Dave says he sees data consumers really loving consuming data from a graph.

The overall goal of data-centric application development is to provide simplicity and flexibility to organizations as most applications are too rigid for Dave and the system integration is even worse.

As mentioned, the first 3 people who fill out a Contact Us on the Semantic Arts website and mention Data Mesh Radio will get a free copy of Dave's book.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#35 Data Mesh Anti-Patterns - Mesh Musings 6 Mar 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Scott shares his emerging data mesh anti-patterns.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#34 Tackling Challenges Together at Talkdesk: An Early Journey Story – Interview w/ José Cabeda Feb 28, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

José's LinkedIn: https://www.linkedin.com/in/jecabeda/

José's Twitter: @jecabeda / https://twitter.com/jecabeda

In this episode, Scott interviewed José Cabeda, Data Engineer at Call-center-as-a-service provider Talkdesk. They talked about Talkdesk's start to their data mesh journey and progress so far.

When José came across Zhamak's original post, it spoke to a number of the challenges Talkdesk was facing, checking many of the boxes to where they wanted to head. The team started from a single data product and iterated from there. While they are still relatively early in their journey, like every company, they have advanced far past their initial use case.

At Talkdesk, a data product is typically a single table or view in Snowflake but the company's North Star is event streaming as their key information storage and sharing mechanism. However, it was sometimes difficult to train people to understand the difference between a business event - something that occurred in the real world - and an event streaming event.

José had a few key takeaways and recommendations for those implementing data mesh:

1. Change will be constant in a data mesh implementation so it is best to standardize the way people and systems will interact as much as possible. Define expectations!

2. Be open to new ideas, there are many challenges ahead so it's best to face them together.

3. Use a single universal ID for major concepts like account or business events to make interoperability easier / possible.

4. Don't be afraid to slice your data in different ways to serve different use cases.

5. To drive buy-in, start with a single use case, whether that is a data product or multiple data products - most people recommend 2-3 data products in your PoC - so you can show why data mesh is a good idea.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#33 10 Reasons Why You Aren’t Ready for Data Mesh Article – Interview w/ Thinh Ha Feb 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Twitter: @thinh_ha / https://twitter.com/thinh_ha

LinkedIn: https://www.linkedin.com/in/%E2%98%81%EF%B8%8F-thinh-ha-58945969/

Medium post: https://medium.com/google-cloud/10-reasons-why-you-should-not-adopt-data-mesh-7a0b045ea40f

In this episode, Scott interviewed Thinh Ha, Strategic Cloud Engineer at Google Cloud Professional Services. To be clear, Thinh was only representing his own views and was not representing Google/GCP in any way.

Scott had asked Thinh to be on after Thinh wrote a post on Medium called 10 Reasons Why You Should Not Adopt Data Mesh (later changed to 10 Reasons Why You Are Not Ready to Adopt Data Mesh). While Thinh is a self-professed "believer" in data mesh, he brings up a number of very reasonable checklist/self-check reasons you wouldn't be ready to move towards data mesh yet.

Scott and Thinh go down each of the 10 objections/reasons through the episode and it is advisable to read the article before proceeding. You can see the high level reasons below. There are a lot of very valuable insights into each of the reasons that could make this a 5 page summary so just listen to the episode instead ;)

1. You are not operating at a scale where decentralization makes sense

2. You do not have a strong business-case for how adopting Data Mesh will deliver business value for individual business units

3. You treat Data Mesh as a technical solution with a fixed target rather than an operating model that continuously evolves over time

4. Your organizational culture does not empower bottom-up decision-making

5. You do not have clearly established roles & responsibilities and incentive structure for distributed data teams

6. You do not have a critical mass of data talent

7. Your data teams have low engineering maturity

8. You expect to find off-the-shelf software to help you adopt Data Mesh

9. You do not have buy-in to “shift-left” security, privacy, and compliance

10. You do not consider Data Governance to be a core activity to be prioritized against other activities in every data team’s backlog

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Programming Notes - Week of Feb 27, 22 - Data Mesh Radio Feb 27, 2022

4 quick things:

4 episodes this week; interviews with Thinh Ha (re 10 Reasons You Aren't Ready for Data Mesh article), José Cabeda (Talkdesk's User Journey), and Dave McComb (more on Data-centric Application Design) and one mesh musing on anti-patterns.
Still having issues getting transcripts done for episodes. If you are at a vendor that wants to sponsor transcripts, please let me know.
Patreon to launch soon, probably Friday.
Starting work on the getting started / proof of concept guide very soon.

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#32 Applying a Historical Lens to Data Mesh - Interview w/ Azmath Pasha Feb 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here (info gated)

Azmath's LinkedIn: https://www.linkedin.com/in/azmathpasha/

In this episode, Scott interviews Azmath Pasha, member of the Forbes Technology Council, who has 25+ years in implementing large-scale IT projects including at CapGemini and Paradigm Technology.

Azmath gave his 3 key measures for data value: cost savings, business value (e.g. driving new initiatives), and data reuse. For data mesh, the long-term value is in the second two but for Azmath, a PoC could be better served focusing on cost savings as it is easier to track and faster to realize.

They dove into the concept of data discovery with human interaction, not purely an online experience. Similar to event storming for discovering your domain events (see DDD for Data episodes), discovery as a purely tool-based experience is always likely to be somewhat lacking. Scott was intrigued about this as that aspect of data discovery hasn't been widely discussed.

To Azmath, the data product experience, part of what Zhamak calls 'the experience plane', is crucial. It is much harder to drive buy-in if your product is hard to use / has a bad user experience.

Azmath's other crucial aspects to getting a data mesh (or any large scale data project) implementation right included: staying tool agnostic so you can remain "future proof"; supporting data producers to reduce time to delivery, especially initial delivery; and looking at your architecture and tool investments over a 5 year time horizon, not just for the short to medium-term.

Azmath wrapped up by saying we are entering a new era of using data, we must democratize the data and also look to new metrics for evaluating the business value of data.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#31 Cliché Quips and Useful Advice - nib Group's Data Mesh Journey So Far - Interview w/ Kurt Gardiner Feb 21, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Application Strangler pattern (recently renamed Strangler Fig Application pattern): https://martinfowler.com/bliki/StranglerFigApplication.html

CQRS: https://www.martinfowler.com/bliki/CQRS.html

Kurt's LinkedIn: https://www.linkedin.com/in/kugardiner/

nib Group careers page: https://nib.wd3.myworkdayjobs.com/careers

In this episode, Scott interviews Kurt Gardiner, Engineering Manager of Data Engineering at Australian Insurance company nib Group.

Kurt shared some insights into nib's journey so far, including the search for something like data mesh before Zhamak published, tool choices (Snowflake, dbt, Fivetran, EventBridge, Kinesis), the slow-role approach to replacing legacy implementation (the "application strangler" pattern mentioned), how they got started, and much more.

Much of nib's approach is the small-scale tactical while building incrementally for the bigger strategic focus. E.g. helping teams to design their data products somewhat manually while building the reusable tooling to be far less manual going forward.

Along their journey, there was some internal pushback from data consumers, especially those used to consuming from the data warehouse. To do data mesh right, Kurt and Scott both emphasized the need to set things up so they can evolve. That will frustrate or scare some people and it's important to work with them to see why that matters. There also needs to be a high tolerance for failure - you will NOT get everything right on your first go.

Kurt also waxed poetic (said nice things) about event streaming patterns, especially CQRS - see link below for more info -, for a useful and scalable pattern that is good for both application development and creating a scalable and useful domain data model. But it requires a complete redesign so it is probably something to slowly introduce where it makes sense, if at all.

Some pithy nuggets of wisdom from Kurt that are highly applicable to data mesh:

"The single biggest problem in communication is the illusion that it has taken place"

"Nobody cares what you know until they know that you care"

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#30 Don't Sleep on the User Experience in Data Mesh - Interview w/ Karen Passmore and Steve Stesney Feb 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

PredictiveUX website: https://www.predictiveux.com/

PredictiveUX partnered meetup: https://www.meetup.com/hexagon-ux-dc-chapter/

Karen LinkedIn: https://www.linkedin.com/in/karenpassmore/

Karen Email: karen at predictiveux.com

Steve LinkedIn: https://www.linkedin.com/in/stephenstesney/

Steve Email: sstesney at predictiveux.com

In this episode, Scott interviewed Karen Passmore (CEO) and Steve Stesney (Data Product Lead) at consulting firm PredictiveUX. They touched on a lot of different topics - a key theme throughout is the importance of the user experience in data mesh, for both data product producers and consumers.

Karen highlighted some parallels between data mesh and content management projects and how to take some key learnings from the past and apply them to data mesh implementations. They discussed the importance of providing your internal people with the right content at their current point in their learning journey - a successful implementation of data mesh requires making it far easier and more scalable to share incremental work artifacts and knowledge - it also means your crucial company knowledge actually gets documented properly.

Steve talked about some historic challenges he had personally with decentralized teams - if you don't manage the cross domain collaboration, both at the business and the technical implementation levels, it is a major pain to stich your data together from all those sources. So there needs to be good alignment on interoperability. Basically, data mesh without a good interoperability strategy is just high quality data silos.

Karen and Steve both emphasized the importance of UX (user experience) for driving adoption. You can have the best solution in the world but if the users don't want it, it's not going to be successful. So working with them throughout the process is crucial to get to a successful implementation, whether that is data mesh or not.

Karen wrapped up by emphasizing the need to be patient and to not expect the same results or try to copy the exact path of other organizations implementing a data mesh. Every organization is very unique and you need to figure out what might work for your organization. Take learnings, not exact blueprints.

The last key point to extract is the need for multiple communication methods, especially for data requests. There may be some overlap but it's a great way to ensure reliability and scalability of your business processes.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#29 Early Learnings from and Replacing CDC with Data Contracts - Interview w/ Andrew Jones Feb 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Andrew's Medium post called Improving Data Quality with Data Contracts: https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698

LinkedIn: https://www.linkedin.com/in/andrewrhysjones/

Twitter: @andrewrjones / https://twitter.com/andrewrjones

In this episode, Scott interviews Andrew Jones, Tech Lead of the Data Infrastructure Team at GoCardless.

Andrew shares the story of how operational system changes kept breaking downstream data consumption (sound familiar?), especially using CDC. The software engineers couldn't easily use the CDC tooling and the data engineers could easily use the data from it either as CDC didn't structure the data for easy consumption.

Andrew wasn't really sure how other people were handling taking the API contract concept and leveraging it for data but started building out some generic simple tooling to let consumers and producers feel somewhat comfortable with their data contracts.

A big revelation was in helping data consumers make better asks for data. The data consumers weren't used to asking the producers for data, especially in a reliable and scalable way (sound familiar?). GoCardless now has an actual standard form for data consumers to use to request data and that is working quite well.

GoCardless plans to completely remove their CDC architecture by 3Q of this year to replace with data contracts. They are focusing on providing tooling to give domains the autonomy to serve data to consumers in their own way. While it isn't data mesh, especially with the lack of interoperability between data products and lack of source/producer-aligned data products, it seems to be working for GoCardless thus far.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#28 Domain Driven Design for Data, a Primer - AKA Just Get People to Talk to Each Other - Interview w/ Danilo Sato and Andrew Harmel-Law Feb 14, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Danilo's Webinar with Zhamak called "Data mesh and domain ownership": https://www.thoughtworks.com/en-us/about-us/events/webinars/core-principles-of-data-mesh/data-mesh-and-domain-ownership

Vladik Khononov, 7 Years of DDD: https://www.youtube.com/watch?v=h_HjtYAH0AI

Danilo LinkedIn: https://www.linkedin.com/in/danilosato/

Danilo Twitter: @dtsato / https://twitter.com/dtsato

Andrew LinkedIn: https://www.linkedin.com/in/andrewharmellaw/

Andrew Twitter: @al94781 / https://twitter.com/al94781

In this episode, Scott interviews two Domain Driven Design (DDD) experts from Thoughtworks - Danilo Sato Director and the Head of Data & AI (part of Office of the CTO) and Andrew Harmel-Law, Technical Principal. This was a further delve into Domain Driven Design for Data after the conversations with Paolo Platter and Piethein Strengholt.

Danilo and Andrew gave a lot of great information about Event Storming, domain definitions and boundaries, ubiquitous language, and so much more but the main theme was "just get people to talk to each other".

DDD is about bridging the gap between how the tech people talk and how the business/business people talk; if you are doing it right, both sides can understand each other and then the engineers can implement those business process learnings as part of the code.

For an initial PoC, Danilo recommends starting with 2-3 data products. It is better if you can do the PoC across multiple domains but it isn't necessary. Validate value and do it quickly. As Andrew mentions, the earlier you can show value, the less pressure there is overall. Look for the initial quick wins while also building for the long-term.

One key thing to remember, per Danilo, when doing DDD for data and data mesh in general: it is always an iterative process. Andrew briefly discussed a way to do DDD in more of a guerilla style than the blue/red books (well known DDD guides). Don't get ahead of yourself as Max Schultze mentioned in his episode. Do not let the size of the eventual task throw you into analysis paralysis.

Andrew talked a lot about how normalization and strong abstractions on the application side make it very difficult to re-add the context lost when you normalize. Both Andrew and Danilo talked about the need to embrace complexity. If you want context, you have to accept there will be complexity. In the pursuit of simplification, you lose the richness, and that is VERY hard to reconstruct afterwards.

Some practical advice for boundary definition is that the boundaries need to be very clear but malleable. Build everything with an eye that it will evolve. Before you start splitting into many 2 pizza teams, look at the big picture and select some coarse-grained boundaries. It is MUCH easier to split later than it is to glue things back together.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#27 Four Key Pillars to Driving Data Mesh Buy-in and Other Insights - Interview w/ Angelo Martelli Feb 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Angelo's LinkedIn: https://www.linkedin.com/in/angelomartelli/

In this episode, Scott interviews Angelo Martelli, Group Leader of Data Services at logistics company Vanderlande.

Angelo laid out his framework for driving data mesh buy-in internally at Vanderlande which helped them take the idea from a small group to a company-wide initiative:

Start with proving there is a problem that you are trying to solve - if everything is functioning well, why focus your efforts on that area instead of trying to fix another? Your proof should be as fact-based as possible, e.g. how long does it take to make a change to your data warehouse. Focus on proving that your incremental investments are driving sub-linear returns. Other areas to look to prove problems: how many people are involved in a change to your data warehouse, percent of time spent on regression testing versus development, mean time to resolution of challenges, etc.

Once you have some proof, you need to work towards understanding the problem you are trying to solve. It's not "deploying a data mesh", it's scaling the organization to be agile relative to data and be able to make more (and better) data-informed decisions.

Next, you need to understand your organization. Who are the right people that can help you? How does your organization work relative to culture and process? Which domains are struggling and how? Tie the implementation goals to the actual business challenges.

Then, you need to demystify data mesh, make it easy to understand for people not well versed in data - what are we actually trying to accomplish and why?

Last, make it concrete / prove it out. Make a few data products, make a simple platform for folks to use.

Angelo then recommends that once you have momentum, sharing a very clear vision is crucial. Not just sharing in a document but actually having conversations to really make sure the context and vision is understood. Data mesh is about collaboration, you must work together so it is imperative to make expectations very clear.

Similar to Abhi Sivasailam, Angelo also stressed the importance of the domain data model and abstracting that away from the application model(s). The business model is what matters for data.

All of that and so much more. Also, Angelo gives a shout out to the usefulness of the Data Mesh Learning community. 😎

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Programming Notes - Week of Feb 13, 22 - Data Mesh Radio Feb 13, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#26 Leveling Up Your Domain Teams w/ Introductory Data and Analytics Engineering – Interview w/ Brian McMillan Feb 10, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Brian is giving away 10 copies of his book for free to those who sign up for a chat to share more about your current challenges related to the book topic or shadow/domain-based IT. To claim your free book, fill out a contact form here and mention "Data Mesh Radio" in the comments.

Brian's contact info:

Email: brian at minimumviablearchitecture.com

LinkedIn: https://www.linkedin.com/in/brianmcmillan01/

Website: https://www.minimumviablearchitecture.com/

Scott interviews Brian McMillan a former Enterprise Architect who took time off to write a book called 'Building Data Products: Introduction to Data and Analytics Engineering for Non-Programmers'. You can learn more about the book - and get a free copy, see below - here: https://www.minimumviablearchitecture.com/

Brian's book lays out a path for the people who are doing the most with data in domains to elevate their skill sets and produce small-scale data products. They do this through a slow ramp from understanding SQL queries to learning data modeling to learning how to publish their data and use simple orchestration tooling. It isn't magic, it will take time and training, but it means you have more people with strong domain knowledge becoming part of the data and analytics engineering process, sharing their business context in scalable and repeatable ways.

Brian's approach can also be used for a pretty easy path to an exploratory platform. There isn't a lot of pre-build to get going so teams can much more easily test out a hypothesis or two rather than it being a lengthy and costly approval and build cycle. There is also an easy path once someone finds a "there" there, to move it to something far more scalable and reliable in the cloud.

There is a lot from the book and interview that can be adapted to help level up your teams' data literacy.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#25 The Most Key Data Mesh Takeaway: It's Not Just You - Mesh Musings 5 Feb 08, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Somewhat short episode. Scott emphasizes that you aren't the only one facing challenges with data mesh. You aren't behind the curve, you are ahead of it - data mesh is bleeding edge. But with a bleeding edge, there is some blood...

Also, you need to think about how others' implementations works for them. Trying to copy-paste another organization's implementation is going to lead to a failed implementation in yours. Take the learnings away and think about how to apply them to your organization.

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#24 Getting Started (and Keeping Going) with Your Data Mesh Journey - Interview w/ Sheetal Pratik Feb 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Sheetal's (and Divya and Madhu from Thoughtworks) Data Mesh Learning Meetup presentation: https://www.youtube.com/watch?v=5btUaLPdaNk

Sheetal's post re data mesh and using DataHub: https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f

Sheetal's LinkedIn: https://www.linkedin.com/in/sheetalpratik/

Data Governance using Data Mesh paper: https://easychair.org/publications/preprint/qZ3m

In this episode, Scott interviews Sheetal Pratik, Director of Engineering, Data Integration at Adidas. If Sheetal's name sounds familiar to data mesh officianados, she presented at the Data Mesh Learning meetup in August 2021.

Sheetal is passionate about giving companies the permission AND a workable plan for getting started with data mesh. She covered a wide range of things regarding getting starting but a few really stood out:

Don't try to tackle tomorrow's challenges today
Break your implementation into phases: development, adoption, and scaling
Start your initial data mesh MVP with a simple data product with a simple schema - your goal is to develop the "muscles" around creating and deploying data products rather than shooting for a high-value product first
Keep to a reasonable budget and prove viability and value

Sheetal also covered how much you really have to have in place to create and evaluate your MVP. There will always be evolution and change and your organization has to be ready for that. That can be frightening or inspirational. Sheetal chooses it as inspirational - it gives you the freedom to move quickly as long as the organization understands that things will change in the future.

She wrapped up with saying that data mesh shouldn't be scary, you should be excited about this journey and what it can mean for your organization. Get an MVP out the door, it will take time, don't get ahead of yourself. Data mesh success can happen if you let it.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#23 Where and How Can Data Virtualization Work in Data Mesh - Interview w/ Dr. Daniel Abadi Feb 03, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Dr. Abadi's blog post on data federalization and data virtualization: https://blog.starburst.io/data-federation-and-data-virtualization-never-worked-in-the-past-but-now-its-different

Dr. Abadi's contact info:

LinkedIn:

Twitter: @daniel_abadi / https://twitter.com/daniel_abadi

Starburst blog posts: https://blog.starburst.io/author/daniel-abadi

Don't forget to catch Dr. Abadi at Datanova - the Data Mesh Summit on Feb 9-10th. Thanks to Starburst for sponsoring the transcripts for Data Mesh Radio, check out the transcript here. And check out Starburst's other free data mesh resources here.

In this episode, Scott interviewed Dr. Daniel Abadi, the Darnell-Kanal Computer Science Professor at the University of Maryland with a focus on scalable data management research. Dr. Abadi will be presenting next week at the Data Mesh Summit on Data Fabric and Data Mesh alongside Zhamak and Sanjeev Mohan.

This was a pretty wide ranging and free wheeling conversation about data virtualization in general and how it can be used in data mesh. Both agreed that there are many places where data virtualization can play in data mesh, whether in extracting information from operational systems, stitching together a data product once data processing has been done, or at the mesh experience plane re combining data across multiple data products. Dr. Abadi specifically mentions something like a query fabric that makes use of a data virtualization approach, not just tools that only do data virtualization.

There is a natural side effect of having multiple different technologies in use - when you give the domains the ability to use what they choose, the difficulty of combining data from multiple sources needs to be solved. There is always a balance between how much you just copy data and how much you can access in the source system and data virtualization can give a few more options rather than all or nothing.

As data virtualization has been around as a concept for 30+ years, there is a lot of baggage with the term but Dr. Abadi sees there being recent advancements that mean more people should take a second look at where they can be useful. But warns to do your homework and really think through whether they fit your use case. A query fabric can make your user experience much more pleasant. Trying to create data products entirely within a data virtualization platform probably won't be, at least according to Scott.

Additional topics included retransmitting or reprocessing data, versioning, the importance of denormalizing data for analytics and how that plays with data virtualization, and much more. It is a really fascinating deep dive into the history of computing and how it impacts what we are trying to do today.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#22 Food Fights and the Modern Data Stack re Data Mesh - Interview w/ Dr. Colleen Tartow Feb 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Colleen's contact info:

Email: colleen at starburst.io

LinkedIn: https://www.linkedin.com/in/colleen-tartow-phd/

Twitter: @CTartow / https://twitter.com/CTartow

Colleen's blog: https://thesequel.substack.com/

The Data Buffet blog post: https://thesequel.substack.com/p/the-endless-data-buffet

Don't forget to catch Colleen at Datanova - the Data Mesh Summit on Feb 9-10th. Thanks to Starburst for sponsoring the transcripts for Data Mesh Radio, check out the transcript here. And check out Starburst's other free data mesh resources here.

In this episode, Scott interviews Dr. Colleen Tartow, Director of Engineering at Starburst. They chatted about the fun and usefulness of food-related analogies to data mesh - you want it to work like a brunch buffet as that is basically a perfect Saturday in Colleen's eyes. And Scott shared the concept of a grocery store - intentional food preparation with different degrees of ingredient and meal preparedness.

Colleen and Scott then covered the "Modern Data Stack" and its relevance to data mesh - in Colleen's eyes, the Modern Data Stack isn't all that modern, it is just the same old paradigm of data teams trying to do their best to work with the output of application/operational data stores and systems that aren't designed with the data in mind. Scott somewhat agreed with his own spin.

Colleen shared her 4 S paradigm re doing data well: speed, simplicity, scalability, and SQL. Yes, slightly Starburst self-serving (4 more S words!) but still relevant and interesting.

They wrapped up with trying to figure out how much of data mesh can companies get away with not doing, which seems to be the topic on everyone's mind. More questions than answers there...

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#21 Data Mesh in Practice: Insights from Zalando's 2 Year Journey w/ Data Mesh - Interview w/ Max Schultze Jan 31, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Max's contact info and related links:

LinkedIn: https://www.linkedin.com/in/max-schultze-b11996110/

Twitter: @mcs1408 / https://twitter.com/mcs1408

Presentations:

Similar to content of the book (Data Innovation Summit): https://www.youtube.com/watch?v=rqYFqtztWi4

Zalando's Story (Spark AI Summit): https://www.youtube.com/watch?v=UrM8yCjmzzw

Max and Arif's 'Data Mesh in Practice' book links:

O'Reilly: https://www.oreilly.com/library/view/data-mesh-in/9781098108502/

Starburst (free gated download): https://www.starburst.io/info/data-mesh-in-practice-ebook/

Max and Arif's upcoming training (Feb 14th): https://www.oreilly.com/live-events/data-mesh-in-practice/0636920508816/0636920068685/

Transcript (link) courtesy of Starburst; check out their other data mesh resources here

Part of Starburst's Datanova Data Mesh Summit takeover week.

Max and Arif's upcoming (Feb 9th) Datanova/Starburst Data Mesh Summit Presentation: https://www.starburst.io/info/datanova2022/

In this episode, Scott interviews one of the most prolific content producers in data mesh, Max Schultze, Data Engineering Manager at Fashion E-Tailer Zalando.

Max shares a LOT of very valuable advice while reflecting on Zalando's data mesh journey so far two years in. Max recommends starting with empathy building and knowledge sharing but looking for ways to scale versus only 1:1 or small group conversations. He had a clever way to force the hands of data consumers to speak to data producers via an Inverse Conway Maneuver that might work for your org too.

Overall, 5 main points/themes emerged:

Data mesh is a journey
Empathy is crucial - have it for those you are working with and work towards building it between teams
Technology is not the most important aspect of an implementation - no matter how cool it might sound or be to focus on it
Start from knowledge sharing, getting people to understand each others' roles and contexts (see empathy!)
Try not to get ahead of yourself

Highly recommend giving this one a listen, maybe twice to pick up the nuggets in there.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

Data Mesh Summit Takeover - Programming Notes - Week of Jan 31, 22 - Data Mesh Radio Jan 30, 2022

Very quick one! The Data Mesh Summit (aka Datanova) is taking over Data Mesh Radio for the week! Sign up for an awesome two-day event (Feb 9th and 10th) here.

We will have 4 (scheduling willing) amazing guests from the Data Mesh Summit on over the next week: Max Schultze (Zalando), Dr. Colleen Tartow (Starburst), Dr. Daniel Abadi (University of Maryland), and Dr. Teresa Tung (Accenture).

In exchange, Starburst is doing a beta, sponsoring transcripts. So please let them know you want more transcripts and again, use the link to sign up to show your support!

Again, sign up here.

To get Max Schultze + Dr. Arif Wider's 'Data Mesh In Practice' book, click through here

#20 Domain Driven Design for Data - Where to Start - Interview w/ Piethein Strengholt Jan 27, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Mentioned articles and books:

Data Management at Scale book: https://www.oreilly.com/library/view/data-management-at/9781492054771/

Data Domains — Where do I start? https://towardsdatascience.com/data-domains-where-do-i-start-a6d52fef95d1

Implementing Data Mesh on Azure: https://towardsdatascience.com/implementing-data-mesh-on-azure-c01ee94306cd

Data Domains and Data Products: https://towardsdatascience.com/data-domains-and-data-products-64cc9d28283e

"The Blue Book" AKA Domain-Driven Design: Tackling Complexity in the Heart of Software: https://www.oreilly.com/library/view/domain-driven-design-tackling/0321125215/

Find Piethein online:

LinkedIn: https://www.linkedin.com/in/pietheinstrengholt/

Twitter: @phstrengholt / https://twitter.com/phstrengholt

Scott interviews Piethein Strengholt, Senior Cloud Solution Architect at Microsoft and author of the O'Reilly book Data Management at Scale.

Piethein shares his tips and tricks for how to approach Domain Driven Design (DDD) for data. There are puts and takes to each approach so unfortunately for those looking for an easy button, there is a lot to consider. When starting, Piethein recommends looking at your applications and deciding if there is a logical mapping to a single domain or if the application is shared across domains. As you learn more about DDD you can start to approach your domains from multiple other angles to find the best solution for your org.

There is a ton of really great advice, too much to sum up well here. Scott highly recommends reading this article on data domains by Piethein before jumping in to the podcast episode.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#19 Data Mesh in Government: Early Lessons Learned - Interview w/ Bente Busch Jan 24, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Bente's LinkedIn: https://www.linkedin.com/in/bentebusch/

Scott interviews Bente Busch, Director of Teams Service Platform - essentially the applications platformm design system, and data platform - at Norwegian government entity NAV (Norwegian Labor and Welfare Department).

Bente talked about some really interesting aspects of NAV's data mesh implementation including:

The unique challenges of having producers more excited to share the data than consumers demanding more data
Changing the culture from doing "projects" to building products
How the on-prem enterprise data warehouse just didn't offer them the same agility they need
Where they are in their data mesh journey so far
And much more

It's a very interesting look into how data mesh could be applied in government with eventual plans to share information outside of the organization.

One very interesting insight is that including those building the application platform in the data mesh self-serve platform build out has been a big win - they are already familiar with application developer workflows and how they think so they are better able to anticipate needs when it comes to building the data platform.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#18 Finding (and Sharing) Your Internal Data Pain Points - Interview w/ Molly Vorwerck Jan 20, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Molly's LinkedIn: https://www.linkedin.com/in/vorwerck/

Monte Carlo's blog: https://www.montecarlodata.com/blog/

In this episode, Scott interviews Molly Vorwerck, head of content and communications at data observability vendor Monte Carlo. Scott asked Molly to be on as she is a great community member in general and as Monte Carlo is well known in the data space for putting out high quality content on hot-button issues. The idea is to take how Molly and team get their ideas and apply that process to figuring out major pain points internally and creating a cohesive strategy to at least start discussing them.

Molly recommends to constantly be interviewing stakeholders. She talks about interviewing people from multiple sides of a challenge, e.g. not just the data consumers but the data producers and the data engineering teams re data challenges. She tries to give people the space to tell their story and asks open-ended questions to truly get their perspective, not arrive at a pre-specified answer.

Molly talks about ways to make the other person feel valued by active listening and making the conversation mutually beneficial. It may be a person you want to interview again so building the relationship is crucial, not just extracting info in a one-time manner.

Scott and Molly dig a bit into the idea of blameless post mortems and how valuable they can be for doing data mesh, especially to figure out what happened to cause data downtime and how to prevent the same issue in the future.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#17 Is Data Mesh Right for Your Org (Part 1) - Mesh Musings 4 Jan 18, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

In this somewhat controversial episode, part 1 of 2, Scott covers topics re is data mesh right for your organization. This should only be used as a jumping off point for discussions.

The 4 segment titles are:

Data quality challenges don't necessarily mean it's time for a data mesh

Data Mesh Lite

Building on a solid foundation

How 'bout now, how 'bout right now? ("Patience you must have, my young Padawan")

Take it with a grain of salt!

Mentioned content links:

Webinar with Zhamak and Sina Jahan - lessons from the trenches with data mesh

Zhamak podcast interview with Barry O'Reilly

Flexport Data Mesh Learning meetup

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#16 Data Quality's 4 Horsemen: Omission, Waste, Divergence, and Downtime - Interview w/ Chad Sanderson Jan 17, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Chad's contact info:

LinkedIn: https://www.linkedin.com/in/chad-sanderson/

csanderson.data at gmail.com

In this episode, Scott interviews Chad Sanderson, Head of Product: Data Platform at Convoy. This episode is part of our continuing series on data contracts and related topics. Chad covers a lot of the challenges relative to data quality, both in maintaining quality and in the challenges poor quality data can cause a company that is heavily reliant on data.

Chad also shares his tale of trying to implement data mesh at Convoy via a large-scale inverse Conway Maneuver.

Chad covered 4 categories of data quality pain, which he calls the "4 Horseman of Data Quality" in this post:

Omission - metadata is missing; no tool out today that solves the omission problem, so users have to bounce between too many tools to try to figure out data specifics like where it came from, the specific meaning, what it's trying to convey, etc.

Waste: growth of unused, unmaintained, or duplicated data; waste happens when the cost of creating new data is less than using something already created

Divergence: the growing divide between what's going on in "the real world" and what's happening in your data warehouse; your business logic, unless it is constantly maintained and updated, starts to diverge from what is happening to your business so what you show on dashboards and reports no longer matches business reality

Downtime: periods of time where the data is missing, wrong, late, etc.; traditionally what most people think of regarding data quality issues

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#15 Data Definitions and "Durable in the Middle" Model - Interview w/ Benn Stancil Jan 15, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Links from show:

Benn's Twitter: @bennstancil / https://twitter.com/bennstancil

Benn's Substack: https://benn.substack.com/

Benn's post on entities: https://benn.substack.com/p/metadata-money-corporation

Starship Technologies Data Mesh Post: https://medium.com/starshiptechnologies/dodging-the-data-bottleneck-data-mesh-at-starship-5925a2de45e6

In this episode Scott interviewed Benn Stancil, co-founder and Chief Analytics Officer at Mode about his emerging concept of global definitions that he is calling an "entity" - essentially, how are people defining terms like customer and what does that mean in each instance so people are on the same page. Data mesh also requires some clear collaboration on definitions, whether centralized or decentralized, in the federated governance pillar. This interview was generated from a Twitter conversation here.

Benn then went into some details about how he views the importance of having visibility into how data "breaks" so it is much easier to identify and fix and that limiting custom integration is crucial so when you fix at the source, it properly propagates downstream. They wrapped up discussing the need to make data producers' lives easier while simultaneously doing the same with data consumers.

Overall, there are some agreements and disagreements and Scott came out thinking more about what are the real causes of pain that would make a full journey to data mesh make sense. It's a good episode to see some of the challenges people are trying to tackle outside of the data mesh community.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#14 Knowledge First Approach and Reusing Existing Standards for Data Mesh - Interview w/ Juan Sequeda Jan 11, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts (most interviews from #32 on) here

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Juan's contact info and related links:

Email: juan at data.world

Twitter: @juansequeda / https://twitter.com/juansequeda

LinkedIn: https://www.linkedin.com/in/juansequeda/

Catalog & Cocktails Podcast: https://data.world/podcasts/

Juan's post about Zhamak's appearance on the Data Engineering Podcast: https://www.linkedin.com/pulse/my-takeaways-data-engineering-podcast-episode-mesh-zhamak-sequeda/

Juan's post about knowledge first: https://www.linkedin.com/feed/update/urn:li:activity:6884179569277059072/

Standards related links:

Dublin Core Metadata Initiative: https://dublincore.org/

RDF (Resoruce Description Framework): https://www.w3.org/2001/sw/wiki/RDF

OWL (Web Ontology Language): https://www.w3.org/OWL/

PROV-O: The PROV Ontology: https://www.w3.org/TR/prov-o/

In this episode, Scott interviews Juan Sequeda, Principal Scientist at data.world and co-host of the Catalog and Cocktails podcast. They discussed Juan's knowledge first approach: putting the meaning and value of the data first instead of focusing on the amount of data we are handling/producing. Knowledge first has 3 components, 1) context, 2) people, and 3) relationships. Juan is a big proponent of knowledge graphs and the relationships side is one many people miss.

Juan also gave some thoughts on what his approach to data mesh hinges on: treating data as a product and finding a balance between centralization and decentralization for all the aspects of building out an implementation. Juan mentioned Intuit's approach of fixed, flexible/extensible, or customizable as a good general tool and to look for (and embrace) what he calls intellectual friction.

Lastly, Juan and Scott talked about the general drive to reduce toil, of reinventing the wheel re data interoperability and standard schemas in data mesh. Juan points to a lot of existing research and standards - e.g. RDF, OWL, and many more (see below) - as a starting point.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn:

https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#13 Is There a Data Mesh Option for Startups and SMEs? - Interview w/ Peter Hanssens Jan 07, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Peter's contact info and relevant links:

Email: peter at cloudshuttle.com.au

LinkedIn: https://www.linkedin.com/in/peterhanssens/

Twitter: @petehanssens / https://twitter.com/petehanssens

Cloud Shuttle website: https://www.cloudshuttle.com.au/

Sydney Data Engineering Community: https://sydneydataengineers.github.io/

DataEngBytes Conference: https://dataengconf.com.au/

Article mentioned by Francois Nguyen (CIO of L'Oreal): https://francois-nguyen.blog/2021/03/07/towards-a-data-mesh-part-1-data-domains-and-teams-topologies/

In this episode, Scott interviews Peter Hanssens, Founder and Solutions Architect at cloud/serverless consulting company Cloud Shuttle. Peter also runs a few large data engineering focused communities (meetups, Slack, conferences) in Australia. Peter had reached out about how can startups and SMEs (small to medium enterprises) get the benefits of data mesh without the costs of building a solution fit for a 10,000+ employee company. Peter does a great job seeking information on behalf of his constituents :)

We covered a number of topics including:

The needs for cultural change as technology will only get you so far
The beauty of pay-per-use solutions for startups/SMEs - and where there are still gaps in the market
The challenges of not having a large pool of data engineers to build and manage a platform - or to help domains model their data
The benefits of centralization until it starts to cause bottlenecks
The importance of tracking lineage for upstream producers/domains - to see who is using your data and why/how
And much, much more

I think you will really enjoy Peter's perspectives and there are some useful conclusions if not a perfect blueprint for startups.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#12 Data-Centric Application Development and Data Mesh - Interview w/ Dan DeMers Jan 05, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Dan's LinkedIn: https://www.linkedin.com/in/demersdan/

Cloud Information Model Standard: https://cloudinformationmodel.org/

Cinchy website: https://cinchy.com/

In this episode, Scott interviews Dan DeMers, Co-Founder and CEO of Cinchy, a dataware platform / data fabric provider. Dan shares his thoughts on why data-centric application design is the best way to deal with the challenges of applications and analytics needing the same data for different purposes - the current approach is to let the application schema evolve whenever and however necessary and the underlying data applications suffer. Dan's view is "share access to data, not copies."

The interview ties loosely with previous interviews re schema/data contracts and data testing as Dan argues using a dataware approach will prevent the issues of an evolving application schema breaking data consumption downstream.

It isn't all rosy as this will take a fair bit of work for an organization to move to this approach. Food for thought and the first of a series of interviews re DDD (domain-driven design) for data and data-centric application development.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#11 The Grinch Who Spoiled Christ-Mesh – 1) Why I Don’t Like Reverse ETL; 2) Dog Fooding; and 3) The Generalist Data Modeler – Mesh Musings 3 Jan 02, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

In this episode, Scott discusses three concepts that are at best a concern. Consider it a late Grinch-inspired present for Xmas :)

Reverse ETL meets a real need for analytical data being pushed into CRM, marketing, and other similar systems. But treating another pipeline as a first order concern is fraught with the same issues of most similar data pipeline treatment: who owns it, how does it evolve, who is observing/monitoring it for uptime and semantic drift, etc.? Should we look to create data products on the mesh to serve those needs instead of another ETL tool?

Some organizations implementing data mesh are forcing their domains to consume any analytics from their own data products on the mesh. The good of this is that it aligns the domain with creating a high-quality data products. But will those data products be designed to fit the general organizational needs or specifically the domain's needs?

There is an emerging push for software engineers to also own the data modeling. To get to a place where this is even feasible, don't we need far better abstractions for domains to _do_ the data modeling? And will this overload software engineers that are already dealing with a metric buttload of technologies and requirements already? Where would a junior engineer fit in that kind of organization? Does this mean more software engineers on the team -> 2 pizza teams now 3? 4? 5? 10? Maybe we pump the brakes on this for now?

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#10 Ensuring Data Quality via Data Testing and Versioning – Interview w/ Jesse Paquette Jan 01, 2022

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Jesse's contact info:

Email: jesse at tag.bio

LinkedIn: https://www.linkedin.com/in/jessepaquette/

Twitter: @bzdyelnik / https://twitter.com/bzdyelnik

Website: https://tag.bio/

Tag.bio vendor interview for Data Mesh Learning: https://www.youtube.com/watch?v=acQADu7ttqQ

In this episode, Jesse Paquette, Chief Science Officer and Co-founder at Tag.bio - a data platform vendor in the life sciences space, and Scott dive a bit deeper into data quality in general, especially data testing and versioning.

You can see the LinkedIn post that sparked this discussion here

Jesse recommends a number of things to ensure data quality, especially data testing and versioning. This includes versioning of 1) the code used to create the data (generally the ETL code), 2 the schema, 3) the business logic layer, and 4) timestamping / temporality based versioning.

Jesse's general calls to action are 1) make data testing frameworks so testing is much less tedious and time consuming; 2) work with stakeholders to gain trust in the data and then continue the dialogue to keep said trust; and 3) create schema/domain model blueprints so that domains have a starting point - whether they use it is irrelevant but shortening the path to a working domain model is crucial.

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#9 Data Contracts Deep Dive – The Pains and the Solution(?) – Interview with Abhi Sivasailam Dec 27, 2021

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Abhi's contact info:

Twitter: https://twitter.com/_abhisivasailam

LinkedIn: https://www.linkedin.com/in/abhi-sivasailam/

Abhi's Data Mesh Learning Meetup: https://www.youtube.com/watch?v=-POiudR2_R0

Debezium blog post mentioned by Abhi: https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

In this episode, Abhi Sivasailam, Head of Growth and Analytics at unicorn startup Flexport, and Scott deep dive into data contracts. They covered a LOT of ground including:

What is a data contract and how does it relate to API contracts and schema contracts
Why data contracts are so crucial to treating data like a product and keeping data usable by consumers
Abhi's rules for a viable data contract
The importance of the analytics engineer to overall data usefulness, especially re domain data ownership in data mesh
Why of the socio of the socio-technical approach to data ownership is the most crucial aspect
The lack of proper tooling to monitor and execute data contracts
How to minimize disruptive changes to downstream data consumers
The issues with domains sharing their data as it is persisted/stored in the database instead of sharing the context via the domain model
Much, much more

Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/

If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf

#8 Platform Re-use, PoC Advice, and More Data Mesh Nuggets – Interview with Matthew Darwin Dec 25, 2021

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/

Please Rate and Review us on your podcast app of choice!

If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here

Episode list and links to all available episode transcripts here.

Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.

Matt's contact info:

LinkedIn: https://www.linkedin.com/in/matthewdarwindba/

Twitter: @EvoDBA / https://twitter.com/EvoDBA

Matt's post on platform reuse: https://medium.com/slalom-data-analytics/data-mesh-is-the-argument-a-strawman-3cffaf55ce5e

Matt's LinkedIn poll on testing data pipelines: https://www.linkedin.com/feed/update/urn%3Ali%3Aactivity%3A6877216459458719744/

In this episode, Matthew Darwin, Principal Data Engineer at Slalom Consulting, and Scott cover a wide range of topics including:

Re an article Matt had posted on data platform re-use and shifting data ownership left:

You can re-use the technologies you already know and love when building a data platform - there is literally no good reason to toss those out the window
If your existing setup enables domains to easily transform, serve, and store their data, sure use your existing configuration; if not, there will need to be changes
Your data platform will need to evolve and it is okay to start with a bit of an underwhelming data platform

Our TOPPODCAST Picks

Follow Us

Stay Connected

Related Podcasts

Links

Stay Connected