Data Cloning, sometimes called Database Virtualization, is a method of snapshotting real data and creating tiny “fully functional” copies for the purpose of rapid provisioning into your Development & Test Environments.
The Cloning Workflow
There are four primary Steps
- Load / Ingest the Source Data
- Snapshot the Data
- Clone / Replicate the Data
- Provision of the Data to DevTest Environments
Under the Hood
Cloning is typically achieved/built using ZFS or HyperV technologies and allows you to move away from the traditional backup & restore methods, which can take hours.
By using ZFS or HyperV you can provision databases x100 quicker and x10 smaller.
What is ZFS?
- ZFS is a file system that provides for data integrity and Snapshotting. It is available for most if not all major OS platforms.
What is HyperV?
- HyperV is a Microsoft virtualization platform that can be used to create and manage virtual machines. It supports Snapshotting as well.
Problem Statement
Backups are often taken manually and can take hours or days to complete. This means that the data isn’t available for use during this time period, which can be problematic if you need access to your data immediately.
There is also a secondary issue with storage. A backup & restore is, by its nature, a 100% copy of the original source. So if you started with a 5 TB database and wanted x3 restores then you are up for another 15 TB in disk space.
What are the Benefits of Data Cloning?
Data cloning is the process of creating a copy, or snapshot, of data for backup, analysis, or engineering purposes. This can be done in real-time or as part of a scheduled routine. Data clones can be used to provision new databases and test changes to production systems without affecting the live dataset.
Advantages
– Clones can be used for development and testing without affecting production data
– Clones use little storage, on average about 40 MB, even if the source was 1 TB
– The Snapshot & Cloning process takes seconds, not hours
– You can restore a Clone to any point in time by bookmarking
– Simplifies your End to End Data Management
Disadvantages
– The underlying technology to achieve cloning can be complex.
However, there are various cool tools on the market that remove this complexity.
What Tools are available to support Data Cloning?
In addition to building your own from scratch, commercial cloning solutions include:
Each is powerful and has its own set of features and benefits. The key is to understand your data environment and what you’re trying to achieve before making that final decision.
Common Use Cases for Data Cloning
- DevOps: Data cloning is the process of creating an exact copy of a dataset. This can be useful for several reasons, such as creating backups or replicating test data, into Test Environments, for development and testing purposes.
- Cloud Migration: Data cloning provides a secure and efficient way to move TB-size datasets from on-premises to the cloud. This technology can create space-efficient data environments needed for testing and cutover rehearsal.
- Platform Upgrades: A large majority of projects end up going over the set schedule and budget. The primary reason for this is because setting up and refreshing project environments is slow and complicated. Database virtualization can cut down on complexity, lower the total cost of ownership, and accelerate projects by delivering virtual data copies to platform teams more efficiently than legacy processes allow.
- Analytics: Data clones can provide a space for designing queries and reports, as well as on-demand access to data across sources for BI projects that require data integration. This makes it easier to work with large amounts of data without damaging the original dataset.
- Production Support: Data cloning can help teams identify and resolve production issues by providing complete virtual data environments. This allows for root cause analysis and validation of changes to ensure that they do not cause further problems.
To Conclude
Data cloning is the process of creating an exact copy of a dataset (database). This can be useful for many reasons, such as creating backups or replicating data for development and testing purposes. Data clones can be used to quickly provision new databases and test changes to production systems without affecting the live dataset.
This article provides a brief overview of data cloning, including its advantages, disadvantages, common use cases, and available tools. It is intended as a starting point for those who are new to the topic. Further research is recommended to identify the best solution for your specific needs. Thanks for reading!