DataOps Data Glossary

Advanced analytics is data analysis that is more than simple operations like sums, averages, filtering, and sorting. In summary, advanced analytics uses mathematical and statistical algorithms to recognize patterns, predict outcomes and probabilities of outcomes, and generate new information.

Agile is an iterative methodology to project management and software development that helps teams deliver value to the customers faster and without all the headaches that go hand-in-hand with traditional software development and project management. The data world has adopted the Agile principles to deliver data products in atomic increments, saving time, balancing governance versus agility, and increasing product quality and availability.

An application Programming Interface (or API) is an external software interface that allows two unrelated entities to interact and pass data between the objects via a service layer.

The term architectural coupling describes how different pieces or components of the data architecture fit together to form a cohesive, functional unit.

An architectural quantum is a highly independent, deployable component with high functional cohesion. It includes all the structural elements needed to function independently and correctly.

The 1950s definition described Artificial Intelligence as “any task performed by a machine that would previously been considered to require human intelligence.”

Modern definitions are more specific. François Chollet provides a later definition of AI:

“The effort to automate intellectual tasks normally performed by humans. As such, AI is a general field that encompasses machine learning and deep learning, but also includes many more approaches that don’t involve any learning.”

An atomic code unit is the basic unit of executable code. It is reusable, essentially indivisible, and easy to maintain. It is based on a single, specific function, enhancing its reusability. A new feature should be constructed from as many existing atomic code units as possible so that the code base has as little repetition as possible.

Documentation is an integral part of data governance. And because no one likes documentation, Agile development promotes the creation and updating of documentation produced automatically from the code as part of the pipeline and data product build process. Documentation is usually formatted using Markdown language (a markup language) to format the plain text documentation stored in the repository with the code and data.

Automated regression testing is regression testing or the retesting of the entire data pipeline or system via a set of automated pre-writing scripts after every code change with little or no human intervention.

In contrast to manual testing, automated testing is a testing technique that uses automated testing tools and techniques to execute pre-written test sets.

Big Data is a collection of structured, unstructured, and semi-structured raw data that is particularly voluminous in nature. It comprises both external and internal data collected by organizations. It is used for machine learning, predictive analytics, and data analysis purposes. While the Big Data construct does not specify a minimum data volume to be considered Big Data, organizations can generate terabytes, petabytes, and even exabytes of data over time.

A business analyst’s role is to examine and document an organization’s business model, processes, systems, and technology integrations. The analyst aims to advise the business on how to improve its day-to-day operations. The business analyst also acts as a bridge between the business and IT using data analytics to determine new requirements, access processes, and provide data-driven reports and recommendations to executives and stakeholders.

Business Intelligence (BI) is a technology-driven process that analyzes data to deliver actionable information to executives, managers, and stakeholders throughout the organization. BI combines business analytics, data analytics, data visualization, and BI tools and infrastructure to help organizations make data-driven decisions.

Traditional Business Intelligence was first noted in the 1960s. And it was merely a system of sharing information between different organizational role players. In the 1908s, it evolved alongside computer modeling into a methodology for decision-making and turning data into meaningful insights. The modern BI paradigm prioritizes flexible, self-service analytics, trusted and governed data, increasing the speed to insight and powering business users and stakeholders.

CI/CD or continuous integration/continuous delivery is a set of operating principles and practices that provide data teams with the functionality to deliver frequent, incremental code and data updates to the business. CI/CD is an Agile best practice. It enables data teams to focus on meeting business stakeholder requirements and deploy high-quality and highly available data products because the CI/CD pipeline steps are automated.

Collaboration among team members, business stakeholders, and end-users is key to success. It leads to creative solutions and improved results. End-users and stakeholders have a keen understanding of what they want to see in a data product. Therefore, it is imperative to consult with them frequently as they will be the ones to determine whether the data product is a success or not. When stakeholders and end-users feel part of the process, they remain engaged until the data product is completed.

A cron or cron job is a software utility is a time-based scheduler available in Unix-based operating systems (including Linux). A cron is used to schedule time-based jobs to run at pre-determined times automatically.

A cron script is a list of one or more commands issued to a computer server operating system to be executed at a specific time. Each command is executed when its triggering time arrives.

Cross-sectional data is data that observes entities such as customers, companies, and stock items at a single point in time. The opposite of cross-sectional data is time series data.

Dark data is a subset of Big Data that is collected and stored by an organization. But it is not used for data analysis and has no importance to the business. It can hold immense value and can be of enormous relevance to the business.

A data dashboard is a data visualization tool that provides users with a visual representation or insight into their organization or department’s data analytics. Dashboards are particularly useful to non-technical users, allowing them to participate and understand the results of data analytics operations.

Data analytics is the methodical study of examining raw data to derive meaningful insights and reach specific conclusions. Data analytics looks for hidden patterns, unknown correlations, customer preferences, and market trends used to improve strategic decision-making, prevent fraud, and drive innovative product development.

A data artifact is a by-product produced during the data engineering, data analytics, or data science operations. Examples of data artifacts include data pipelines, data products, dashboards, visualizations, and analytics or machine learning reports.

A data architecture is a blueprint of the organization’s data ecosystem. It includes the rules, policies, procedures, and standards used to govern data is collected, stored, and processed. A data architecture also describes how data is extracted from multiple data sources, ingested into data pipelines, stored in the data platform, and transformed into valuable data. Lastly, it defines the processes used to extract, load, and transform the data into meaningful insights.

A data asset is high-value data. It is a data set or data product that has been derived from the raw data. And that the organization uses to derive meaningful insights to generate revenue. Data assets are now some of the most valuable assets that an organization owns.

Data assurance is a promise or guarantee from the data team to the business that the data sets and products delivered to the business are correct and extremely high quality, instilling high levels of confidence in the data by the business.

Data atrophy or data degradation is the gradual corruption or loss of value of the data. If not stored, transformed, and analyzed into meaningful insights, data starts losing value as soon as it has been created.

A data attribute is a data field representing the characteristics or features of a data entity or object. For instance, customer name is an attribute of the customer object.

As described by data.world, a data catalog is a “metadata management tool that companies use to inventory and organize their data.” It provides data teams, business users, and stakeholders with the functionality to find, analyze, and manage the voluminous data that falls within the Big Data paradigm. Advantages of using a data catalog include organization-wide, role-based access to the data, data discovery, and data governance and security.

Data cleansing, data cleaning, or data scrubbing is the process of removing or fixing incorrect, corrupt, duplicate, incorrectly formatted, or incomplete data within a data set. When combining different data types from multiple data sources, there are many opportunities for the resultant data set to be less than perfect. Consequently, it is crucial to cleanse the data before serving it to the business.

A data cloud is a unified data management architecture or platform that breaks down data silos, is highly available, with unlimited scale, and can run multiple workloads in parallel.

Data debt comes from the term “technology debt” and describes the cost of deferring funding for a new data platform feature that will increase the value of the organization’s data. The debt could typically be avoided by funding and implementing data governance and data management functions.

Data discovery is the process through which data teams collect metadata from multiple data sources with the help of advanced analytics tools and visual navigation of the data. It aims to increase the visibility of the entire data environment by consolidating all of the information about the data environment.

A data domain forms part of the data governance model. It is strongly aligned with a business model. For instance, individual data domains can include customer data, product data, and vendor data. Each data domain includes a domain owner responsible for every aspect of the domain, including data governance.

Data drift is the unexplained and undocumented changes to the data structure, semantics, and infrastructure. It is typical of the modern, fast-paced, explosive growth of data and data sources and rapid innovation in technologies such as data science, AI, advanced analytics, and machine learning. Not only does data drift break pipelines and data products and corrupt data, but it also exposes new data usage and analysis opportunities.

A data-driven organization is also known as an insights-driven business. And it is a company that uses data-derived insights to improve its business operations and to drive sustainable growth and innovation over time.

In the traditional data environment, data is siloed and owned by the IT department. Business users do not have access to the data even though they use the data to make business decisions. Data democratization translates into the fact that everyone in the organization has equal access to the data without gatekeepers creating bottlenecks at the gateway to the data.

Data downtime is a data quality issue. And it occurs when your data is inaccurate, partial, or incorrect. It is incredibly costly for the modern data-driven organization. It reduces trust in the data because decisions are made based on this inaccurate data, impacting ROI and day-to-day operations.

A data ecosystem is a data environment designed to grow, evolve, and scale in line with the organization’s unique information needs. Its sole purpose is to collect, store, and analyze the data generated by the organization. A data ecosystem comprises three elements: Infrastructure, including data stores, analytics pipelines, and BI applications.

Data engineering is a sub-discipline of software engineering that focuses on data transportation, data storage, and data transformation. A data engineer aims to provide a consistent flow of organized data for data analysts, data scientists, and self-service business users.

A data entity is an object that replicates a real-world object and forms part of a data model. Examples of data entities include a customer, a supplier, and a stock item record. Data entities are part of the Master Data Management (MDM) model.

A data environment is similar to a data ecosystem in that it provides a platform (or place) for an organization to extract, load, store, and transform data into meaningful information.

Data federation is the creation of a virtual database, or data warehouse that aggregates data from disparate data sources without physically moving the data into the database or data warehouse. In other words, federated data is aggregated virtual data from multiple different data sources. The concept of data federation is similar to data virtualization.

Data fragmentation is a condition where an organization's data is not well-cataloged, redundant, outdated, and difficult to access.

Data Governance is responsible for data security and data quality. It defines the roles and responsibilities that allow access to and ensure accountability for and ownership of data products and assets contained within the data platform. In summary, it covers the people, processes, and technologies needed to manage and protect data assets and products.

Governance is embedded in the data pipelines. Thus, the rules, processes, and procedures that ensure data security, data quality, and data deliverability are encapsulated inside the data pipelines.

Most enterprise data is still collected and stored in standalone data stores or individual data silos. Data integration is the process of amalgamating or bringing the separate data stores together to generate improved data insights and increase the value of the data stored by the enterprise.

A data integration platform is a data platform that allows data from multiple disparate sources to be loaded, stored, and transformed so that it can be analyzed by data analysts and data scientists or served to business stakeholders and end-users in the form of self-service analytics.

A data lake is a centralized repository that stores raw structured, unstructured, and semi-structured data at any scale. The data is stored as-is, in its raw format, without needing to be structured to fit a relational database or data warehouse model. Data teams can also run different types of analytics such as visualizations, dashboards, real-time analytics and train machine learning models directly on top of the data lake, directly accessing the raw data.

A data landscape is the totality of an organization’s physical data assets. It does not include any logical data views. In other words, it is the organization’s overall data storage options, processing and analytics capabilities, and BI applications included in the company’s data environment.

Data latency is the time it takes for data to travel between two points. It is typically measured as the time-lapse between the time of the data-generating event and the time it arrives in the organization’s data platform. It is also measured as the time-lapse between the data arrives in the data platform and its availability to be retrieved for analysis and business use.

The data lifecycle is the sequence of processes (or stages) that successfully manage and preserve the data for use and reuse. The data lifecycle includes the following steps: Plan, collect, data quality and governance, describe, preserve (store), discover, integrate, and analyze. Not all of these stages have to form part of every data lifecycle. For instance, some data activities, such as metadata gathering, might only focus on discovering, integrating, and analyzing steps. Additionally, this lifecycle might not always follow a linear path, and multiple revolutions of the cycle might be necessary to process the data.

The data lifecycle management model helps organizations manage data flow throughout their lifecycle: from initial creation to destruction. A clearly defined data lifecycle management method is integral to ensuring effective data governance. And a robust data governance policy is vital to data quality and data trust.

Data lineage is a map of the data journey from its origins to how it is in its present state. Each movement and transformation step (including its analytics lifecycle) is recorded with an explanation of how and why the data has moved over time. Data lineage provides an in-depth description of where the data comes from and where it ends up.

Data management is a term describing the methodology used to maintain and oversee the processes used to plan, create, acquire, maintain, use, archive, retrieve, control, and delete data. It is also described as enterprise information management (EIM) defined by Gartner as “an integrative discipline for structuring, describing, and governing information assets across organizational and technical boundaries to improve efficiency, promote transparency, and enable business insight.”

Data masking or data obfuscation is the process of hiding the original data by covering it with modified content. This process is most often used to protect classified and commercially sensitive data. Despite being masked, the data must remain usable and valid.

A data mesh is a decentralized, domain-driven data architecture designed to address the architectural failure nodes predominant in the centralized monolithic data architecture. The decentralized domain-driven design pattern is similar to the microservices architecture utilized in software development.

A data model is an abstract representation of the data, organizing the data entities, properties, and how they relate to one another as well as the properties of real-world objects.

Data modeling is the process of providing a structure for raw data to transform and analyze it, creating meaningful information.

Data mining is the process of analyzing large data sets to discover patterns, anomalies, trends, correlations, and relationships that data analysts and scientists might otherwise miss. Data mining uses methods that intersect machine learning, advanced statistical analysis, and database or data warehousing systems with the overall goal of extracting information from the data set. This information assists organizations solve problems, mitigate risks, and determine new business opportunities.

Data observability is similar to data monitoring in that it consistently monitors the health of your data systems by tracking and troubleshooting incidents to reduce and prevent data downtime. Similar core objectives are seen in both software application observability and data observability: The minimal disruption to the service.

DataOps is a set of principles describing how data engineers build, test, and deploy data products and platforms precisely the same way software applications are developed. DataOps (Data + Operations) is rooted in the principles of DevOps, Agile, and Lean. It is a way to approach and deliver enterprise data operations, incorporating agility and governance. DataOps relies on continual and Agile development. And it includes creativity and balances governance and agility, resulting in improved collaboration between data teams, business stakeholders, and end-users.

A DataOps engineer is a technical professional that focuses exclusively on the data development and deployment lifecycle instead of the data product itself. DataOps engineers do not work with the data itself. Instead, they engineer the environment and processes that data engineers use to build the data products. DataOps engineers also improve the data engineers and analysts’ development processes by implementing the DataOps principles as outlined in the DataOps Manifesto and the #TrueDataOps philosophy.

A DataOps platform, like our Dataops.live platform on Snowflake, is a data architecture that facilitates the implementation of the principles of DataOps and #TrueDataOps and the DataOps lifecycle, balancing agility and governance, providing end-to-end orchestration, environment management, automated testing, CI/CD, and ELT.

The root definition of a data pipeline is that it moves data from source to destination. It is essentially is a series of actions or steps that ingests data from its multiple, disparate sources and loads the data into a data platform for storage and analysis purposes. Additionally, data pipelines can be used to transform data to prepare it for analysis and train machine learning data sets.

Data pipeline components are the individual parts that together make up the pipeline that moves the data from its source to its destination.

Data pipeline debt is a form of technical debt that is incurred when data pipeline code is not properly coded and tested, reducing productivity and puts data integrity at risk.

Data pipeline orchestration is the automated coordination of the individual processes or steps that make up the data pipeline. It manages the dependencies between the pipeline tasks, schedules jobs, and monitors workflows, alerting to errors in the pipeline.

A data platform is an integrated data architecture or technology solution that provides end-to-end management of the organization’s data ecosystem. The organization’s data is governed, accessed, and delivered to business stakeholders and end-users via this data platform.

A data point or observation is a discrete unit of information. In statistics, a data point is a set of one or more measurements on a single member of a statistical population.

A data product is not the same as a data project in that a project has a defined scope, start, and end. On the other hand, a data product has a continuous cycle of requirements gathering, development, testing, and delivery to the business. A product has regular releases throughout its lifecycle. It has high agility and an enormous capacity for change. Its workload backlog is continually being reprioritized. And a product has automated regression testing built into the release process. A data product can be as simple as a set of dashboards or a data mart, or more involved such as a contracted data share with a 3rd party. Each have a clear definition and changing requirements over time. And every change to the product (whether a technical change or a simple refresh of the data) should be treated like a new version of product.

Data profiling is the act of extrapolating information or statistics about the data based on its known characteristics, traits, or tendencies.

Data provenance describes the origins of the data, including inputs, entities, systems, and processes related to the data. In other words, data provenance is the data’s historical record keeper. Information derived from a data provenance record helps provide error tracking, source identification, identify data governance audit trails, and helps re-enact the flow of data from its source into the data platform.

Data Quality is the standard for the condition of the data based on factors such as reliability, whether it is up to date, errors, consistency, completeness, and accuracy. Organizations must measure the quality of their data to resolve errors, inaccuracies, incomplete data, and reliability issues; otherwise, the data will not be fit-for-purpose.

The practice aims to maintain a consistently high quality of data. It extends right across the data ecosystem, from data acquisition to data delivery. Its focus is on continually and continuously improving the quality of your data.

Data relationships are the cornerstone of relational databases. A relationship between two data entities describes the commonalities between the two entities or objects, allowing users to explore this relationship by joining data from each object.

Data reliability describes the accuracy and completeness of the data served to the business for self-service analytics purposes.

A data resource is a component of an organization’s data infrastructure and overarching IT infrastructure. It represents all of the data generated by and available to the organization.

Data science is an interdisciplinary discipline that uses scientific methods, analytics processes, and machine learning and Artificial Intelligence to derive new and innovative insights from raw unstructured, semi-structured, and structured data. It uses techniques and theories drawn from the academic fields of mathematics, computer science, statistics, information science, and domain knowledge.

A data set is an ordered collection of related data. Data sets can be massive and span multiple data sources. The data is organized within each data set based on a specific model designed to process and analyze the data, creating meaningful information.

A data silo is an insular data management strategy where stakeholders and business users across the organization do not have access to all of the company’s data. The stakeholders and users will only have access to a specific subset of the organization’s data because it is siloed or partitioned.

Data standardization is converting data to a standardized format to facilitate the processing and analysis of this data. Data usually comes from a wide variety of different sources and in various formats. Therefore, it must be transformed into a standard format before it can be accessed and analyzed.

A data swamp is a data lake that has deteriorated and unmanaged data. Either it is inaccessible to its users, or it is not providing any value to the organization.

The data value is the worth of the data to the organization it belongs to. In other words, it is the value that is derived from transforming, processing, and analyzing the data.

Data vault modeling is a “database modeling method that is designed to provide long-term historical storage of data coming in from multiple operational systems.” It also provides a method of considering historical data that deals with “issues such as auditing, tracing of data, loading speed and resilience to change as well as emphasizing the need to trace where all the data in the database came from.” Data vault modeling does not differentiate between good and bad data. It supplies a single version of the data facts. It means that every row in the data vault must be accompanied by the source of the record and the load data metrics.

Data visualization is the graphical representation of the data and the information derived from this data. It is an effective way of communicating meaningful insights to business users, especially when the data is voluminous.

A data warehouse is a centralized data management system designed to support Business Intelligence and analytics activities. It centralizes and consolidates voluminous data from multiple disparate sources. And it often contains massive volumes of historical data and is solely intended to perform large-scale queries and analytics on this data.

A database grant is a permission or privilege that grants users access to different database functions. These grants include administrator access, where the user has access to the entire database and all of its functionality to read-only access. The user can only view the data in the database without making any changes to the data and its underlying structure.

Database lifecycle management (DLM) combines a business and technical policy-based approach to maintaining databases and database objects. It is not a product. Instead, it is a methodology for managing all the elements in a database, such as a database schema, the data, and the metadata.

A database object is a defined structure in a database used to store or reference data such as a table, view, sequence, synonym, or index.

A database schema is an abstract representation of the design of a database. It describes the organization of data as well as the relationships between the data in the different database tables, columns, and rows.

The SQL Data Definition Language (DDL) consists of the SQL commands used to define a database schema. It deals with descriptions of the database schema. It is used to create, drop, alter, truncate, and rename objects in the schema, such as tables, synonyms, views, indexes, and databases themselves.

A compiler is an engine or software application that converts instructions to be read and executed by a computer. A declarative compiler takes a SQL-based declarative definition describing the state of an object. And it determines how to reach the state without being told how to by the programming code.

A declarative language is a programming paradigm that describes what the state of a data product, database, or data pipeline component should be without describing how to reach a particular state.

Deep learning is a subset of machine learning. It is designed to train a machine or computer to perform human-like tasks, such as recognizing speech, making predictions, and identifying images. It is capable of unsupervised learning from unstructured or unlabeled data. Deep learning’s key focus, the ability to continuously approve and adapt to changes in the underlying data patterns, presents an opportunity to introduce more dynamic behavior into data analytics.

A deployment pipeline deploys or delivers code updates from the Git repository or version control to other environments, making it readily available to end-users in an automated fashion.

A design pattern is a repeatable solution to a commonly occurring problem within a given context. A design pattern is an abstraction, a template that does not translate directly into executable code. In other words, it is a problem-solving template that can be used as a foundation for creating a solution to a problem. Data pipeline design patterns help provide a foundation for building and deploy robust, functional data pipelines.

DevOps (development + operations) is a philosophy and practice that combines software development and operations functions. It aims to shorten the time to value by reducing the software application development and deployment lifecycle and providing continuous delivery and high software application quality.

A directed acyclic graph or DAG is a conceptual representation of a series of activities, such as the steps involved in running a data pipeline from start to finish. The order in which the actions occur is visually represented by a set of circles that form a graph, some connected by lines showing the flow from one activity to another.

A distributed data pipeline is simply a data pipeline that is made up of individual components that run on different computer servers or hardware infrastructure. In order to run a distributed data pipeline, a notification or messaging system is used to orchestrate each step of the pipeline.

Data manipulation language is a SQL-based language that is used to insert, delete, and update or modify data in a database.

A domain-agnostic model is the opposite of a domain model. It is not subject-related. Rather it is an abstract interpretation of what a domain in the organization’s data environment could look like.

The domain-driven design pattern is an approach to software and data architecture development that focuses on the domain model as its core construct.

A domain model is a conceptual model of the domain that incorporates both data and behavior. A domain model is a formal representation of the knowledge domain in a knowledge graph, including roles, data types, concepts, and rules.

Dynamic data masking helps prevent unauthorized access to sensitive data by allowing data teams to decide how much sensitive data to reveal on the client-side without changing the raw data. Dynamic data masking is a policy-based security feature that hides sensitive data and is applied at the application layer.

Extract, Load, and Transform (ELT) is the process of extracting data from multiple, disparate data sources and loading it into a target data platform without transforming it in the process. ELT is the preferred alternative to the traditional ETL process by pushing the transformation component to the target data platform for improved data lineage, performance, data governance, and data provenance.

An ephemeral data set is a logical data set in that it does not contain physical data. Instead, it contains pointers to the physical data. The benefits of an ephemeral data set include only needing to keep one copy of the raw data, the ability to create multiple transformed data sets based on individual stakeholder requirements, and reducing heavy workloads, and improving data platform and analytics performance.

ETL or extract, transform, and load is the traditional data integration process that combines data from multiple sources into a single, targeted data platform. It was introduced in the 1970s to load or integrate data into mainframes or supercomputers for computation and analysis.

Loading the raw data into the data platform before transforming it is a far superior method to transforming the data before loading it into the data platform. However, there are cases where data must be removed, anonymized, masked or obfuscated for privacy reasons or because of regulations such as GDPR.

Fail fast is a philosophy that values the CI/CD principles of incremental development and extensive testing to determine whether an idea has value. It is often associated with the Lean startup methodology and is designed to cut losses when the testing reveals something isn’t working and quickly trying something else. This concept is known as pivoting.

Data engineers can create individual branches that are safe and isolated to develop new data pipeline features and data products simultaneously. These feature branches are then merged back into the master branch via a merge request.

A first-class workflow is similar to a first-class object. It can be dynamically created, passed between different data pipeline components, destroyed, and has the rights of any other programming language variable.