We recently (14 July 2021) completed a masterclass with Kent Graziano, Chief Technical Evangelist, Snowflake, discussing Snowpark, the use of Scala and Java UDFs, and how we integrate this new technology into our DataOps platform. In particular, we discussed how we are using our Snowflake Object Lifecycle Engine to recycle these Snowpark objects through our DataOps platform via CI/CD pipelines and automated regression testing.
Updates from the DataOps team and the DataOps Community
DataOps launches Support for Snowflake Java UDFs
Recently DataOps.live announced our support for Snowflake Java UDFs. This new Snowflake feature is another important step on the road (especially when combined with the release of Snowpark – see our blog about this here).
DataOps Launches Support for Snowflake Snowpark
Recently DataOps.live announced our support for Snowflake Snowpark.
Snowflake is known for its performance, scalability, and concurrency. Before Snowpark, users interacted with Snowflake predominately through SQL. Now, customers will be able to execute more workflows entirely within Snowflake’s Data Cloud, without the need to manage additional processing systems.
FUTURE PERMISSION GRANTs: why they are very useful and why you shouldn't use them in a DataOps world
FUTURE GRANTS, ALL TABLES in the Snowflake Data Cloud (and similar constructs in other databases) are necessary and powerful tools for manually administered databases. However, they have significant downsides in terms of flexibility, auditability, potential information bleeds, Principle of Least Privilege etc. In a DataOps approach all the same convenience is possible, but all of these limitations are addressed.
PART 6: Imperative Approaches for Database Object Lifecycling
Following on from IMPERATIVE VS DECLARATIVE FOR DATA. In this blog post, we will look at different ways that the Imperative Approach can be implemented and also give an overview on how a basic Declarative Approach could work.
PART 5: Declarative vs Imperative for Data
Let's now consider this in the context of Data and Databases. The most typical example of changing the state of a database is creating a table. We would all initially jump to something like:
Q&A from the Masterclass on CICD and DataOps for Snowflake
Thanks to everyone who attended the Technical Masterclass on CI/CD and DataOps for Snowflake 2 weeks ago. And to everyone who has since watched the recording since.
Part 4: Declarative vs Imperative: Introduction
This series of blog posts builds on our previous set on The Challenges of Repeatable and Idempotent Schema Management:
PART 3: The Challenges of Repeatable and Idempotent Schema Management: Conclusions
Over the previous 2 blog posts, we have seen that managing the lifecycle of database objects in an idempotent manner is impacted by the imperative nature of most SQL statements, which require a known initial state for changes to be applied repeatably.
PART 2: The Challenges of Repeatable and Idempotent Schema Management: Idempotence in Snowflake
It may appear that most of this should be possible with native SQL statements and indeed some DDL operations in Snowflake are naturally idempotent, whereas others have impacts on data and object state, and some are not idempotent at all. Let’s look at some of the ways SQL tries to help us with this and the problems that remain.