Skip to content
DataOps.live Platform

Build, test & deploy  Data Products on  Snowflake

DataOps.live is the leading provider of Snowflake environment management, end-to-end orchestration, CI/CD, automated testing & observability, and code management, wrapped in an elegant developer interface.
Spendview for Snowflake FREE

Spendview  for  Snowflake

An inexpensive, quick and easy way to build beautiful responsive website pages without coding knowledge.
Getting Started
Docs- New to DataOps.liveNew to DataOps.live? Start learning by doing, create your first project, and set up your DataOps execution environment.
Join the Community
Join the CommunityFind answers to your DataOps questions, collaborate with your peers, share your knowledge!
#TrueDataOps Podcast
#TrueDataOps PodcastWelcome to the #TrueDataOps podcast with your host Kent Graziano, The Data Warrior!
Academy
DataOps AcademyEnroll in the DataOps.live Academy to take advantage of training courses. These courses will help you make the most out of DataOps.live.
Resource Hub
An inexpensive, quick and easy way to build beautiful responsive website pages without coding knowledge

Customer stories
Events
Be a part of our community, whether you join us virtually or in person. Connect with fellow professionals, expand your network, and gain knowledge from our esteemed product and industry experts.
#TrueDataOps.org
#TrueDataOps.Org#TrueDataOps is defined by seven key characteristics or pillars:
Featured Blogs
Blogs
Stay informed with the latest insights from the DataOps team and the vibrant DataOps Community through our engaging DataOps blog. Explore updates, news, and valuable content that keep you in the loop about the ever-evolving world of DataOps.
In The News

In The News

Stay up-to-date with the latest developments, press releases, and news.
About Us
About UsFounded in 2020 with a vision to enhance customer insights and value, our company has since developed technologies focused on DataOps.
Careers

Careers

Join the DataOps.live team today! We're looking for colleagues on our Sales, Marketing, Engineering, Product, and Support teams.
DataOpsMar 21, 2019 10:49:00 AM2 min read

Snowflake – the Best Time Series Database in the World?

In the past 2 years, the rise of the time series database (TSDB) has been meteoric – growing faster than any other database model (as defined by db-engines.com):

Trend-of-IoT-data-1024x523

DataOps’ CTO, Guy Adams, has been focused on storing and processing time series data for over 20 years and, of course, followed this trend. In 2016 he started evaluating the best systems on the market and for a while everything looked great – these systems:

  • Have very fast ingest and query of time series data
  • Often have advanced time series functions
  • Are very space efficient at storing time series data
  • Often have nice features such as automatic ageing of old data

As a result, in a simple lab test with a large time series dataset they perform very well. The problem tends to come when a lab test turns into a more operational/production evaluation. These TSDBs typically have a similar set of challenges (not every TSDB has every challenge):

  • A non-SQL and non-standard interface – fine for connecting from custom applications or for data science, but hard to connect to standard systems like BI tools and standard ETL/ELT tools (in fact TSDBs don’t really have much concept of ELT as few transforms are possible once data is loaded)
  • Lack of maturity – while there are many ‘cool technologies’ in the time series database world, the overall space is still relatively niche and low volume and therefore most of these systems don’t have the deep maturity from having a very large customer base
  • Only on-premise or self-hosting options – none of the benefits of a cloud database
  • Relatively immutable – updates are either not supported or very slow – this is because one of the ways that a TSDB can be very fast is to store data in a way optimised for write and read but making updates extremely difficult
  • Narrow focus – while time series databases get great benefits from only supporting one style of data, there is a price to pay for flexibility. In practical terms, how many organisations have JUST time series data? Usually when people say “we have a load of time series data” what they mean is “our data volume challenges are all time series, but we still have a lot of dimensional data we want to join it on to”. The problem with time series databases is that they ONLY support the time series data. When there is other information to store, another database will be needed. Using two different databases doesn’t really help since there is still no ability (without the complexity and performance hit of putting something like Presto over the top of both) to run queries and analytics using time series and non time series data at once.

In the past 18 months, Guy concluded that while the promise of time series databases was very high, the enormous flexibility, scalability and power of a cloud based SQL data warehouse like Snowflake far exceed the performance hit from being a more flexible system. Tuning of clustering using CLUSTER BY can reduce this performance hit to typically <10% which is a very small price to pay as compared to having two separate systems to maintain and a set of additional application layer development.

avatar

DataOps

Born out of nearly a decade of professional services and hundreds of successful data projects, DataOps.live was built to meet the real-life needs of modern, data-driven companies using Snowflake. DataOps.live removes the need for enterprises to balance governance and agility, delivering fundamental improvements in both. The platform brings agile DevOps automation (#TrueDataOps) and IoT data compression to the Snowflake cloud data platform. DataOps.live is a single platform for 100% of an organization's DataOps lifecycle needs around Snowflake. It provides full Snowflake environment management, end-to-end orchestration, CI/CD, automated testing, pipeline observability and release management wrapped in an elegant user interface. Faster development, parallel collaboration, increased efficiencies, reduced costs, data assurance, simplified orchestration and full data product lifecycle management are the result.

RELATED ARTICLES