Deduplication

Data Deduplication and growing Disk/Storage demand.

IDC REPORT – Storage Purchase and expected.

Source: IDC Worldwide File-Based Storage 2011-2015 Forecast: 

Foundation Solutions for Content Delivery, Archiving and Big Data, doc #231910, December 2011 

Using DDPEval.exe we can use to evaluate savings with Deduplication without even installing or enabling deduplication.

Yes I am talking about Windows 2012 deduplication.

How it works?

1. Segment data into small variable sized chunks
2. Identify and remove duplicate chunks
3. Replace duplicate copies of each chunk with a reference to a single copy
4. Compress Chunks.

DFS and Deduplication:

Data Deduplication works fine with Distributed File System (DFS) Replication.

Optimizing or unoptimizing a file will not trigger a replication because the file does not change.

DFS Replication uses Remote Differential Compression (RDC), not the chunks in the chunk store, for over-the-wire savings.

The files on the replica can also be optimized by using deduplication if the replica is using Data Deduplication.

https://technet.microsoft.com/en-in/library/hh831454.aspx --Data Deduplication Interoperability

https://technet.microsoft.com/en-in/library/dn281957.aspx#BKMK_Dedupe --What's New in DFS Replication and DFS Namespaces in Windows Server

Folder redirection and Deduplication

We can use Deduplication on Folder redirection too.  https://technet.microsoft.com/en-in/library/hh831700.aspx 

How Meta data and ACL are managed?

Deduplication is only performed on the primary/unnamed data stream (not on alternate data streams).

The data streams have been moved to a new structure called the chunk store. In place of the data stream, a reparse point is created that points to the chunk store.

In addition, the stream is marked as sparse to indicate that the stream is segmented.

Deduplication is not supported on:

                      i.        FAT volumes

                     ii.        Boot or System Volumes

                   iii.        Remote Drives

                    iv.        Cluster Shared Volumes (but does support cluster configurations)

                     v.        ReFS

                    vi.        3rd party file systems

Deduplication skips the following file types:

                      i.        Encrypted Files

                     ii.        Files with 'extended attributes'

                   iii.        Files less than 64KB in size

                    iv.        Files with reparse points

                     v.        Deduplication volumes can be backed up and restored optimized

Blogs:http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx --Introduction to Data Deduplication in Windows Server 2012

http://blogs.technet.com/b/canitpro/archive/2013/04/29/step-by-step-enabling-data-deduplication-on-windows-server-2012-volumes.aspx --Step-by-Step: Enabling Data Deduplication on Windows Server 2012 Volumes

TechNet Articles:https://technet.microsoft.com/en-in/library/hh831434.aspx --Install and Configure Data Duplication

https://technet.microsoft.com/en-in/library/dn486808.aspx--What's New in Data Duplication in Windows Server