Bulk Data Migration Using Robocopy

Overview

This document explains how to use Robocopy, a Microsoft Windows tool, to migrate data from a Windows file server to the Nasuni Edge Appliance. Robocopy has been bundled with Windows operating systems since the Vista and Server 2008 versions.

Important: Robocopy and the Nasuni Edge Appliance cache

If the amount of data that you are migrating to the Nasuni Edge Appliance is larger than your Nasuni Edge Appliance cache, Robocopy will completely fill your Nasuni Edge Appliance cache during your data migration. When the Nasuni Edge Appliance cache is full, Robocopy writes will fail until the Nasuni Edge Appliance successfully snapshots recently written data.

Tip: The Robocopy in Windows 2019 or Windows 10 version 1809 should be avoided. Alternatives to Robocopy include FastCopy, RichCopy, and SuperCopier.

Caution: With recent versions of Robocopy (such as 2016 and 2019), using the /E switch in cases where Robocopy creates the target directory can lead to incorrect permissions on the target directory. These incorrect permissions appear as permissions from the source directory rather than permissions inherited from the parent directory of the target directory. The inheritance appears as from "Parent Folder". Instead, either create the target directory before running the command, or remove the /E switch from the command.

Caution: With case-insensitive destination volumes, if case is not consistent across the migration source (including during migrations and across incremental migrations), migrations can fail, because files or directories might not have the proper case.

Before using Robocopy to copy data

Before using Robocopy to copy data, be aware of these recommendations:

The user running the Robocopy command should be a CIFS Administrator. This helps ensure that the user has the privileges to write to the folders. By making that user a CIFS Administrator, they can bypass NTFS permissions and run the Robocopy command without issue. It also helps to avoid certain error messages.
The user should first copy only the folder structure, then set permissions on the folders, and then copy the data without permissions. This helps ensure the highest likelihood of success when copying data to Nasuni.

Before using Robocopy to copy data, consider the following configuration options to optimize the copying and snapshot process:

Before performing a migration, set the cache space reserved for new data to up to 90 percent. However, ensure that enough cache space remains available for retaining the data that users are most likely to need. Using the NMC or the Edge Appliance user interface, you can configure the cache settings on the Cache Settings page.
After all data is protected in the cloud, you can then set the cache settings back to the default value.
Turn off snapshots until the cache is near to being full. Since automatic eviction is triggered when the cache usage exceeds the eviction threshold (default: 85 percent), ensure that the unprotected data in the cache is below 85 percent. Using the NMC or the Edge Appliance user interface, you can turn off snapshots by deselecting all Snapshot Scheduling.
When the cache is full, stop the copying process. Then turn snapshots on again and continue snapshots until the cache is empty again. Then repeat the process until all the data has been copied and protected.
Enabling Global File Lock can have an impact on performance, depending on factors that include network congestion, user load, and file sizes. Do not enable Global File Lock until after initial data load and after large data migrations.
If users do not typically collaborate on the same file at the same time, it is unnecessary to enable Global File Lock.
Important: Migrating items from a root drive on a Windows computer changes the attribute of the items to System. This hides the items from normal views. Alternatively, you can use the share path to the items, rather than the drive letter.

Using Robocopy to copy data to the Nasuni Edge Appliance

Robocopy is a Windows command line tool. To migrate data to the Nasuni Edge Appliance using Robocopy, follow the steps below:

Identify the data set that you want to migrate to the Nasuni Edge Appliance.
Create or locate a share on the Nasuni Edge Appliance to write your data set to.
Maximize the amount of local cache space to reserve for new incoming data. On the Nasuni Edge Appliance, click Configuration, then select Cache Settings, and set Select Percentage to 90%.
Run the Command Prompt as CIFS Administrator.
Copy only the folder structure, then set permissions on the folders, and then copy the data without permissions.
Run the command below from the source server’s command line as CIFS Administrator:

robocopy <source> <Filer share> /copy:datso /mir /fft /r:0 /w:0 /secfix /mt:20 /xo /xf thumbs.db *.mp3 /log+:filename.log /nc /ns /np /ndl /nfl /tee

Robocopy switches include the following.

/copy:datso: Specifies the file properties to copy:

d: Data

a: Attributes

t: Time stamps

s: NTFS access control list (ACL)

o: Owner information

/mir: Mirrors a directory tree

/fft: Assumes FAT file times, important when copying data to the Nasuni Edge Appliance

/r:0: Number of retries on failed copies (default is 1,000,000)

/w:0: Wait time between retries (seconds) (default is 30)

/secfix: Fixes file security on all files, even skipped ones

/mt:20: Creates multi-threaded copies with N threads (default is 20).

/xo: Excludes older files

/xf thumbs.db *.mp3: Excludes files that match the specified names or paths

/log+:filename.log: Writes the status output to the specified log file (appends to the existing log file)

/nc: Specifies that file classes are not to be logged

/ns: Specifies that file sizes are not to be logged

/np: Specifies that the progress not be displayed

/ndl: Specifies that directory names are not to be logged

/nfl: Specifies that file names are not to be logged

/tee: Writes the status output to the console window, as well as to the log file

Warning: Do not use the /mov flag, because this moves the data but does not copy the data.

Note: In some situations, you might experience some failed copies with an "error 5: Access is denied" message. In this case, one workaround is to omit the attribute information.

Note: Full details on Robocopy syntax and options are available in Microsoft technical documentation.

After using Robocopy to copy data

After you are done using Robocopy to copy data, change back any configuration options you used:

If you had set the cache space reserved for new data to 90 percent, configure the cache settings back to the default value.
If you turned off snapshots while copying data, enable snapshots once again using Snapshot Scheduling.

Using Robocopy to delete data by copying an empty directory

It can take a long time to delete a lot of data. An alternative to deleting the data is to copy an empty directory on top of an existing directory. Robocopy can perform this kind of operation. This can be much faster than using Windows File Explorer to delete the data.

The goal is to execute a customized Robocopy command that uses an empty source folder and the destination folder that you want to delete. By using the /MIR option, the Robocopy command mirrors the contents of the empty source folder to the destination folder, thus deleting all files and folders within the destination folder.

This is much faster than using Windows File Explorer, because the Robocopy command does not have to perform the "discovery" phase. Also, Robocopy can be set to use multi-threading.

Here is a sample Robocopy command to enable faster deletion of destination data using Robocopy as opposed to deleting via Windows File Explorer. Modify the Robocopy command as necessary.

robocopy.exe
"C:\EmptyTestFolder\<name of destination folder to delete>" "\\<destination path of folder to delete"
/FFT /NDL /NFL /TEE /DCOPY:T /COPY:DATSO /MIR /NP /MT:30 /R:0 /W:0
/LOG+:"C:\temp\<log file name>_cleanup.log"

Here is a sample Robocopy command to delete a folder called “Marketing”:

robocopy.exe "C:\EmptyTestFolder\Marketing" "\\nea101\volume03a\shares\Marketing"
/FFT /NDL /NFL /TEE /DCOPY:T /COPY:DATSO /MIR /NP /MT:30 /R:0 /W:0
/LOG+:"C:\temp\marketing-folder-cleanup.log"

Important: Test the Robocopy command before executing it, to ensure it is doing what you expect and at the proper locations.

Important: The user should be a CIFS Administrator. This ensures that the user has the privileges to write to the folders. If the user is a CIFS Administrator, they can bypass NTFS permissions and run the Robocopy command without issues.

Robocopy switches used include the following.