Skip to content
DataOps.live Professional EditionNEW
Purpose-built environment for small data teams and dbt Core developers.
DataOps.live Enterprise Edition
DataOps.live is the leading provider of Snowflake environment management, end-to-end orchestration, CI/CD, automated testing & observability, and code management, wrapped in an elegant developer interface.
Spendview for Snowflake FREE

An inexpensive, quick and easy way to build beautiful responsive website pages without coding knowledge.


Pricing and Edition

See whats included in our Professional and Enterprise Editions.

Getting Started
Docs- New to DataOps.liveStart learning by doing. Create your first project and set up your DataOps execution environment.
Join the Community
Join the CommunityFind answers to your DataOps questions, collaborate with your peers, share your knowledge!
#TrueDataOps Podcast
#TrueDataOps PodcastWelcome to the #TrueDataOps podcast with your host Kent Graziano, The Data Warrior!
Resource Hub
On-demand resources: eBooks, white papers, videos, webinars.

Customer Stories
Academy

Enroll in the DataOps.live Academy to take advantage of training courses. These courses will help you make the most out of DataOps.live.


Learning Resources
A collection of resources to support your learning journey.
Events
Connect with fellow professionals, expand your network, and gain knowledge from our esteemed product and industry experts.
Blogs

Stay updated with the latest insights and news from our DataOps team and community.


#TrueDataOps.org
#TrueDataOps is defined by seven key characteristics or pillars:
In The News

In The News

Stay up-to-date with the latest developments, press releases, and news.
About Us
About UsFounded in 2020 with a vision to enhance customer insights and value, our company has since developed technologies focused on DataOps.
Careers

Careers

Join the DataOps.live team today! We're looking for colleagues on our Sales, Marketing, Engineering, Product, and Support teams.
Kent Graziano - The Data WarriorApr 10, 2023 11:46:37 AM6 min read

Federated Governance: Why it Matters and Why it’s Essential for Data Mesh

How do you balance agility and governance, particularly when you now have the capability to ingest, deploy and transform massive amounts of data at a scale that no-one had previously dreamed of? And often this is sensitive data. Perhaps you’re working a highly regulated area like healthcare or financial services.  

How can you ensure domain teams adhere to the rules and standards while also giving them freedom and autonomythe responsibilityto manage and develop data products appropriate to their domains, without it turning into a free-for-all? 

The concept of federated governance isn’t new but the popularity of data mesh means it’s become a much hotter topic, and a challenge for teams to solve. One of the four pillars of data mesh, a federated data governance framework provides for some centralized management, for enterprise level rules and standards,  but the ultimate responsibility to apply and execute the rules is decentralized to the domains.  

In short, it’s a way to ensure good governance with the right rules being applied in the right ways to the right data but through a more flexible approach, one that empowers domain team developers and supports collaborative working.   

For example: you have a governance policy that all PII data must be masked. In federated governance, it’s up to teams in finance, customer service, product management, etc. to apply this policy appropriately to the data in their domain. They are responsible for looking at what PII data they have, and defining the best ways to apply the rules in their area: proper tagging, making sure masking is being applied, that clear rules exist around who has access to the data, and so on.  

 
Control + Governance + Collaboration + Managed Self-Service 

If you’re in a fully centralized environment with a single IT team responsible for everything, you probably don’t need an approach like federated governance (what would be the point?). But even in that world, IT still needs input from the domains (what we used to call “the business”). Who should be allowed to access this data? What are the rules? Are there regulatory requirements? Data mesh holds that doing things this way means bottlenecks: the IT team has to do all the work find this information around the governance of the data and then apply they rules to that data (assuming they actually understand the data). Add to that the fact that  all the departments and functions that may deal with  a business subject may have very different rules and regulations to apply. It is easy to see how that team may end up with a backlog and become a bottleneck to rolling out the data with proper governance in place.  

The data mesh proposal is to decentralize the application of governance to the people who know their data/sector/rules/regulations best. Federated governance gives them the responsibility to apply and execute proper governance, with a central, enterprise focused team providing guidance, tools and training. In this world, the DataOps.live platform provides you with the management and oversight (control) you need, to put standards in place, with related quality checks and workflows to enforce them.  

Such an approach supports Collaboration, a pillar of #TrueDataOps. Another pillar, of course, is Governance and Security. DataOps.live gives you the tools and functionality to administer all that governance and to distribute it to domain teams, so people can work in faster, easier and more collaborative ways while ensuring the enterprise rules are always followed. In order to move something into production, governance, approvals, and testing are built into the platform. And it’s all traceable and auditable. 

(There are many other interesting aspects, too. For example: a neat feature of Snowflake is that features like dynamic data masking are policy-based. You develop a policy to apply the rule at a various levels and that rule follows the data wherever it goes, even if it is shared out in a data product. That solves another challenge: what happens after the data leaves your domain? It needs to stay masked and secured in appropriate ways after it’s shared.) 

 

Roche: Centralized Policy Definitions and Enforcement 

When Roche Diagnostics wanted to implement a novel data mesh approach in its highly regulated healthcare sector, it chose a forward-looking data stack that included Snowflake and DataOps.live. Omar Khawaja, Global Head of Business Intelligence, said Roche “wanted to address critical issues around our people and our unique decentralized culture, to empower and give ownership to teams, creating and using our tech stack in a very different way while still having the federated governance on top.” With DataOps.live enabling centralized policy definitions and enforcement, application is decentralized so data product teams in different domains could build and adapt products faster and more effectively than previously, using all of the governance best practices such as code check-in and checkouts, and with multiple data engineers working concurrently without creating bottlenecks or interfering with one another. 

According to Paul Rankin, Head of Data Platforms, “Federating this approach across hundreds of developers in 20+ data product teams was the challenge… data is continuously changing and the pipeline is continuously questioned. This could never have been done with a centralized approach.” 

 

Avoiding ‘Data Chaos’ 

Even though you’re giving individuals and teams freedom to be co-operative and creative, you still need governance rules and they need to be enforced. Federated governance is the way to set, share and apply the rules in smarter ways. One challenge (and fear) around data mesh is ending up with more data silos. We already know what a Data Wild West can look like. Decentralizing doesn’t mean giving everybody 100% responsibility for, say, picking their own data stack, and making and enforcing their own rules and standards. A potential pitfall of data mesh is that you don’t have an overarching architecture or framework that allows you to easily and transparently manage all those moving parts, giving you the enterprise-level oversight you need. Unless, as we’ve seen, you have a robust approach to federated governance through a platform like DataOps.live.  

It does sound like a paradox: control plus freedom. But it’s actually about responsibility and where that lies, along with removing bottlenecks. The principles you need to be successful taking this approach are already built-in to the seven pillars of #TrueDataOps. Environment management in tandem with governance and security, alongside automated monitoring and testing to make sure things are working as intended, so enabling collaboration and managed self-service, and so on. DataOps.live ties all this together, making #TrueDataOps a reality and enabling data mesh in such a way that all data is being governed as needed and data chaos is avoided. 

When we talk about federated governance what we’re really talking about is good data governance full stop. But the thing is, the data world and the needs of large distributed organizations have changed. What’s required today is a strong, yet flexible, governance framework that actively supports the speed, autonomy and agility that domain teams are striving for. This is what the DataOps.live platform enables. 

 Want to learn more about DataOps.live?  

 

 

RELATED ARTICLES