Blog

Blog - DataOps

#TrueDataOps & CI/CD Pipelines: Delivering Trust to Data Applications on Snowflake

#TrueDataOps & CI/CD Pipelines: Delivering Trust to Data Applications on Snowflake

The rapid and continued expansion of data systems as well as the exponential explosion of data are driving the use cases for advanced data analytics and data science applications. However, without adopting the principles and philosophy of #TrueDataOps, it is always going to be challenging to develop, test, and deploy data pipelines that deliver trusted, reliable data to analytics applications and machine learning models in a short space of time and according to business stakeholder requirements.

DataOps launches support for Soda SQL and Soda Cloud

DataOps launches support for Soda SQL and Soda Cloud

With the exponentially increasing importance of and value attributed to data, it’s never been more critical to test, observe, and monitor the quality of data being used to develop many different data products, driving strategic decision-making at an organizational level. To enhance and facilitate the development of the highest-quality data products, we have recently announced our support for Soda SQL and Soda Cloud. Succinctly stated, Soda SQL has been fully integrated into our DataOps platform, with full support for Soda Cloud.

DataOps launches integration with Matillion

DataOps launches integration with Matillion

The cloud data warehouse or data cloud is increasing in importance exponentially as more organizations understand the value of using data-driven insights as a foundation for and critical part of any decision-making process. As a result, it is essential to move unstructured, structured, and semi-structured raw data from its source to a centralized location (the cloud data warehouse) to be processed, transformed, modeled, and analyzed to derive meaningful insights or information.   

Lifecycling Snowpark and Java UDFs Through DataOps

Lifecycling Snowpark and Java UDFs Through DataOps

We recently (14 July 2021) completed a masterclass with Kent Graziano, Chief Technical Evangelist, Snowflake, discussing Snowpark, the use of Scala and Java UDFs, and how we integrate this new technology into our DataOps platform. In particular, we discussed how we are using our Snowflake Object Lifecycle Engine to recycle these Snowpark objects through our DataOps platform via CI/CD pipelines and automated regression testing. 

DataOps Launches Support for Snowflake Snowpark

DataOps Launches Support for Snowflake Snowpark

Introduction

Recently DataOps.live announced our support for Snowflake Snowpark.

Snowflake is known for its performance, scalability, and concurrency. Before Snowpark, users interacted with Snowflake predominately through SQL. Now, customers will be able to execute more workflows entirely within Snowflake’s Data Cloud, without the need to manage additional processing systems.

FUTURE PERMISSION GRANTs: why they are very useful and why you shouldn't use them in a DataOps world

FUTURE PERMISSION GRANTs: why they are very useful and why you shouldn't use them in a DataOps world

FUTURE GRANTS, ALL TABLES in the Snowflake Data Cloud (and similar constructs in other databases) are necessary and powerful tools for manually administered databases.  However, they have significant downsides in terms of flexibility, auditability, potential information bleeds, Principle of Least Privilege etc. In a DataOps approach all the same convenience is possible, but all of these limitations are addressed.