Skip to content
DataOps.live Professional EditionNEW
Purpose-built environment for small data teams and dbt Core developers.
DataOps.live Enterprise Edition
DataOps.live is the leading provider of Snowflake environment management, end-to-end orchestration, CI/CD, automated testing & observability, and code management, wrapped in an elegant developer interface.
Spendview for Snowflake FREE

An inexpensive, quick and easy way to build beautiful responsive website pages without coding knowledge.


Pricing and Edition

See whats included in our Professional and Enterprise Editions.

Getting Started
Docs- New to DataOps.liveStart learning by doing. Create your first project and set up your DataOps execution environment.
Join the Community
Join the CommunityFind answers to your DataOps questions, collaborate with your peers, share your knowledge!
#TrueDataOps Podcast
#TrueDataOps PodcastWelcome to the #TrueDataOps podcast with your host Kent Graziano, The Data Warrior!
Resource Hub
On-demand resources: eBooks, white papers, videos, webinars.

Customer Stories
Academy

Enroll in the DataOps.live Academy to take advantage of training courses. These courses will help you make the most out of DataOps.live.


Learning Resources
A collection of resources to support your learning journey.
Events
Connect with fellow professionals, expand your network, and gain knowledge from our esteemed product and industry experts.
Blogs

Stay updated with the latest insights and news from our DataOps team and community.


#TrueDataOps.org
#TrueDataOps is defined by seven key characteristics or pillars:
In The News

In The News

Stay up-to-date with the latest developments, press releases, and news.
About Us
About UsFounded in 2020 with a vision to enhance customer insights and value, our company has since developed technologies focused on DataOps.
Careers

Careers

Join the DataOps.live team today! We're looking for colleagues on our Sales, Marketing, Engineering, Product, and Support teams.
Young engineer building with Legos
Luke CrumleyFeb 22, 2023 1:30:51 PM4 min read

Empowering data engineers to develop more, faster, better

CloudOps & DataOps Engineer Luke Crumley describes the lightbulb moment when he fully grasped the potential of DataOps to deliver new levels of speed, innovation and quality for business teams.   Data engineering is about solving problems: helping organizations to make decisions based on facts rather than guesswork. As a result, businesses need increasingly powerful tools to help them to gather, sort, manage, manipulate and present insights to support those decisionssolutions like DataOps.live.  

In short, DataOps makes it possible for developers and engineers to deliver more, do it faster, and more consistently. 

Screenshot 2023-02-01 at 3.24.59 PM


I wasn’t originally a data engineer. Describing myself today as a ‘broad engineer’, I’ve been programming since childhood, LEGO® Mindstorms and websites, dabbling in amateur projects for friends and family. I developed a strong working knowledge of a variety of technologies and techniques. Around 2013, a close friend was working as a software engineer, and said I’d love it.
 

Looking back at earlier roles, people would use Excel and other rudimentary tools, and write their own scripts. They might have heard about source control and versioning but were not shown how to apply the concepts to their work. The focus was on getting data transformed and producing some business benefit. It was a ‘cottage industry’, and used other tools to try and stitch the separate parts together. People worked in silos. They simply weren’t benefiting from the knowledge and experience accumulated during decades of software engineering, in particular automation and what that enables.  

In my previous role, I worked in a cloud engineering team supporting data engineers. We understood the tools being used in the data pipelines, but we didn’t understand the data in the same ways the data engineers did. I hadn’t really thought through the DataOps concept, and the benefits it could deliver: how it could help to solve our users’ problems. Until—true story—I was interviewed by Guy Adams, DataOps.live co-founder and CTO, and his team for this job. It all clicked into place. I realized the teams I’d been supporting would benefit massively from a DataOps approach, one that made the entire process more automated, collaborative, efficient and scalable. 

Untitled design


A central problem with the homegrown cottage-industry approach is that when people have a great idea and want to innovate with their data, or scale up their activities, they can struggle, because that would impact on the value they are currently delivering in a production environment. This risk-averse way of working stifles creativity and innovation. By contrast, DataOps.live opens up a whole new world of opportunity for developers: you can create all these different branches, create and test new stuff, knowing you’re in control of the environment. You can feel safe innovating on a platform like this.
 

While there’s obviously a great deal of crossover, data engineers and software engineers do have different mindsets and approaches. For example, software engineers benefit from a fundamental understanding of how computers and operating systems work, and how you interact with them. Data engineers may not necessarily come with the same experiences and tend to have a more analytical approach, focusing on the data and getting to the value fast. They haven’t previously owned the data-to-operations process themselves. So this is where the automation, testing and orchestration in DataOps.live come in. You can develop things, test them, fix them, iterate faster, and deliver to a consistently high quality with a tool focused on data engineers. 

What’s so special about DataOps.live is that it enables exponential improvements. I can’t think of any scenario where you wouldn’t want to use it, even smaller businesses if they have ambitions to grow. To put it another way, if you’re using Snowflake and have enough data to warrant a data engineer, you’ll want to look at DataOps.live. 

hunter-harritt-Ype9sdOPdYc-unsplash


It enables you to
focus on the data. You’re not worrying if this change or that one affects production, or what else might happen. The risk is removed. It’s an end-to-end platform, so you can see what comes in one end, and what goes out the other—all fully observable, all fully tested. There are so many utilities and helpers baked-in, removing manual steps, that data engineers save significant time and effort. You gain the visibility and control you simply wouldn’t have with a ‘build your own’ mix of separate tools. Yet you still have the freedom and flexibility to incorporate specialized tools should you choose. 

In my view, there are four imperatives when it comes to data engineering: to deliver what the customer wants; to automate as much as possible; to test (and test); and have the ability to store configuration as code in source control. DataOps.live enables you to do all this, and far more. 

Based in the United Kingdom, Luke has been with DataOps.live since May 2022. Previously a Principal Cloud Engineer with Songtradr, Inc. and Principal Software Engineer at the UK Civil Service, he has a BSc from the University of West England Bristol. Connect with Luke here: https://www.linkedin.com/in/luke-crumley 

avatar

Luke Crumley

Based in the United Kingdom, Luke has been with DataOps.live since May 2022. Previously a Principal Cloud Engineer with Songtradr, Inc. and Principal Software Engineer at the UK Civil Service, he has a BSc from the University of West England Bristol.

RELATED ARTICLES