Below is a collection of perl scripts that make data migration from VNX/Celerra file systems to an Isilon system much easier. I’ve already outlined the process of using isi_vol_copy_vnx in a prior post, however using EMCOPY may be more appropriate in a specific use case, or simply more familiar to administrators for support and management of the tool. Note that while I have tested these scripts in my environment, they may need some modification for your use. I recommend running them in a test environment prior to using them in production.
EMCOPY can be downloaded directly from DellEMC with the link below. You will need to be a registered user in order to download it.
https://download.emc.com/downloads/DL14101_EMCOPY_File_migration_tool_4.17.exe
What is EMCOPY?
For those that haven’t used it before, EMCOPY is an application that allows you to copy a file, directory, and subdirectories between NTFS partitions while maintaining security information, an improvement over the similar robocopy tool that many veteran system administrators are familiar with. It allows you to back up the file and directory security ACLs, owner information, and audit information from a source directory to a destination directory.
Notes about using EMCOPY:
1) In my testing, EMCopy has shown up to a 25% performance improvement when copying CIFS data compared to Robocopy while using the same number of threads. I recommend using EMCopy over Robocopy as it has other feature improvements as well, for instance sidmapfile, which allows migrating local user data to Active Directory users. It’s available in version 4.17 or later. Robocopy is also not an EMC supported tool, while EMCOPY is.
2) Unlike isi_vol_copy_vnx, EMCOPY is a windows application and must be run from a windows host. I highly recommend a dedicated server for any migration tasks. The isi_vol_copy_vnx utility runs directly on the Isilon OneFS CLI which eliminates any intermediary copy hosts, theoretically providing a much faster solution.
3) There are multiple methods to compare data sizes between the source and destination. I would recommend maintaining a log of each EMCopy session as that log indicates how much data was copied and if there were any errors.
4) If you are migrating over a WAN connection, I recommend first restoring from tape and then using an incremental data sync with EMCOPY.
Getting Started
I’ve divided this post up into a four step process. Each step includes the relevant script and a description of the process.
- Export File System information (export_fs.pl Script)
Export file system information from the Celerra & generate the Isilon commands to re-create them.
- Export SMB information (export_smb.pl Script)
Export SMB share information from the Celerra & generate the Isilon commands to re-create them.
- Export NFS information (export_nfs.pl Script)
Export NFS information from the Celerra & generate the Isilon commands to re-create them.
- Create the EMCOPY migration script (EMCOPY_create.pl Script)
Perform the data migration with EMCOPY using the output from this script.
Exporting information from the Celerra to run on the Isilon
These Perl scripts are designed to be run directly on the Control Station and will subsequently create shell scripts that will run on the Isilon to assist with the migration. You will need to manually copy the output files from the VNX/Celerra to the Isilon. The first three steps I’ve outlined do not move the data or permissions, they simply run a nas_fs query on the Celerra to generate the Isilon script files that actually make the directories, create quotas, and create the NFS and SMB shares. They are “scripts that generate scripts”. 
Before you run the scripts, make sure you edit them to correctly specify the appropriate Data Mover. Once complete, You’ll end up with three .sh files created for you to move to your Isilon cluster. They should be run in the same order as they were created.
Note that EMC occasionally changes the syntax of certain commands when they update OneFS. Below is a sample of the isilon specific commands that are generated by the first three scripts. I’d recommend verifying that the syntax is still correct with your version of OneFS, and then modify the scripts if necessary with the new syntax. I just ran a quick test with OneFS 8.0.0.2, and the base commands and switches appear to be compatible.
isi quota create –directory –path=”/ifs/data1″ –enforcement –hard-threshold=”1032575M” –container=1
isi smb share create –name=”Data01″ –path=”/ifs/Data01/data”
isi nfs exports create –path=”/Data01/data” –roclient=”Data” –rwclient=”Data” –rootclient=”Data”
Step 1 – Export File system information
This script will generate a list of the file system names from the Celerra and place the appropriate Isilon commands that create the directories and quotes into a file named “create_filesystems_xx.sh”.
#!/usr/bin/perl
# Export_fs.pl – Export File system information
# Export file system information from the Celerra & generate the Isilon commands to re-create them.
use strict;
my $nas_fs="nas_fs -query:inuse=y:type=uxfs:isroot=false -fields:ServersNumeric,Id,Name,SizeValues -format:'%s,%s,%s,%sQQQQQQ'";
my @data;
open (OUTPUT, ">> create_filesystems_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nas_fs |") || die "cannot open $nas_fs: $!\n\n";
while ()
{
chomp;
@data = split("QQQQQQ", $_);
}
close(CMD);
foreach (@data)
{
my ($dm, $id, $dir,$size,$free,$used_per, $inodes) = split(",", $_);
print OUTPUT "mkdir /ifs/$dir\n";
print OUTPUT "chmod 755 /ifs/$dir\n";
print OUTPUT "isi quota create --directory --path=\"/ifs/$dir\" --enforcement --hard-threshold=\"${size}M\" --container=1\n";
}
The Output of the script looks like this (this is an excerpt from the create_filesystems_xx.sh file):
isi quota create --directory --path="/ifs/data1" --enforcement --hard-threshold="1032575M" --container=1 mkdir /ifs/data1 chmod 755 /ifs/data1 isi quota create --directory --path="/ifs/data2" --enforcement --hard-threshold="20104M" --container=1 mkdir /ifs/data2 chmod 755 /ifs/data2 isi quota create --directory --path="/ifs/data3" --enforcement --hard-threshold="100774M" --container=1 mkdir /ifs/data3 chmod 755 /ifs/data3
The output script can now be copied to and run from the Isilon.
Step 2 – Export SMB Information
This script will generate a list of the smb share names from the Celerra and place the appropriate Isilon commands into a file named “create_smb_exports_xx.sh”.
#!/usr/bin/perl
# Export_smb.pl – Export SMB/CIFS information
# Export SMB share information from the Celerra & generate the Isilon commands to re-create them.
use strict;
my $datamover = "server_8";
my $prot = "cifs";:wq!
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";
open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";
while ()
{
chomp;
my (@vars) = split(" ", $_);
my $path = $vars[2];
my $name = $vars[1];
$path =~ s/^"/\"\/ifs/;
print OUTPUT "isi smb share create --name=$name --path=$path\n";
}
close(CMD);
The Output of the script looks like this (this is an excerpt from the create_smb_exports_xx.sh file):
isi smb share create --name="Data01" --path="/ifs/Data01/data" isi smb share create --name="Data02" --path="/ifs/Data02/data" isi smb share create --name="Data03" --path="/ifs/Data03/data" isi smb share create --name="Data04" --path="/ifs/Data04/data" isi smb share create --name="Data05" --path="/ifs/Data05/data"
The output script can now be copied to and run from the Isilon.
Step 3 – Export NFS Information
This script will generate a list of the NFS export names from the Celerra and place the appropriate Isilon commands into a file named “create_nfs_exports_xx.sh”.
#!/usr/bin/perl
# Export_nfs.pl – Export NFS information
# Export NFS information from the Celerra & generate the Isilon commands to re-create them.
use strict;
my $datamover = "server_8";
my $prot = "nfs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep export";
open (OUTPUT, ">> create_nfs_exports_$$.sh") || die "cannot open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";
while ()
{
chomp;
my (@vars) = split(" ", $_);
my $test = @vars;
my $i=2;
my ($ro, $rw, $root, $access, $name);
my $path=$vars[1];
for ($i; $i < $test; $i++)
{
my ($type, $value) = split("=", $vars[$i]);
if ($type eq "ro") {
my @tmp = split(":", $value);
foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
}
if ($type eq "rw") {
my @tmp = split(":", $value);
foreach(@tmp) { $rw .= " --rwclient=\"$_\""; }
}
if ($type eq "root") {
my @tmp = split(":", $value);
foreach(@tmp) { $root .= " --rootclient=\"$_\""; }
}
if ($type eq "access") {
my @tmp = split(":", $value);
foreach(@tmp) { $ro .= " --roclient=\"$_\""; }
}
if ($type eq "name") { $name=$value; }
}
print OUTPUT "isi nfs exports create --path=$path $ro $rw $root\n";
}
close(CMD);
The Output of the script looks like this (this is an excerpt from the create_nfs_exports_xx.sh file):
isi nfs exports create --path="/Data01/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP" isi nfs exports create --path="/Data02/data" --roclient="Data" --roclient="BACKUP" --rwclient="Data" --rwclient="BACKUP" --rootclient="Data" --rootclient="BACKUP" isi nfs exports create --path="/Data03/data" --roclient="Backup" --roclient="Data" --rwclient="Backup" --rwclient="Data" --rootclient="Backup" --rootclient="Data" isi nfs exports create --path="/Data04/data" --roclient="Backup" --roclient="ProdGroup" --rwclient="Backup" --rwclient="ProdGroup" --rootclient="Backup" --rootclient="ProdGroup" isi nfs exports create --path="/" --roclient="127.0.0.1" --roclient="127.0.0.1" --roclient="127.0.0.1" -rootclient="127.0.0.1"
The output script can now be copied to and run from the Isilon.
Step 4 – Generate the EMCOPY commands
Now that the scripts have been generated and run on the Isilon, the next step is the actual data migration using EMCOPY. This script will generate the commands for a migration script, which should be run from a windows server that has access to both the source and destination locations. It should be run after the previous three scripts have successfully completed.
This script will output the commands directly to the screen, it can then be cut and pasted from the screen directly into a windows batch script on your migration server.
#!/usr/bin/perl
# EMCOPY_create.pl – Create the EMCOPY migration script
# Perform the data migration with EMCOPY using the output from this script.
use strict;
my $datamover = "server_4";
my $source = "\\\\celerra_path\\";
my $dest = "\\\\isilon_path\\";
my $prot = "cifs";
my $nfs_cli = "server_export $datamover -list -P $prot -v |grep share";
open (OUTPUT, ">> create_smb_exports_$$.sh") || die "cant open output: $!\n\n";
open (CMD, "$nfs_cli |") || die "cant open $nfs_cli: $!\n\n";
while ()
{
chomp;
my (@vars) = split(" ", $_);
my $path = $vars[2];
my $name = $vars[1];
$name =~ s/\"//g;
$path =~ s/^/\/ifs/;
my $log = "c:\\" . $name . "";
$log =~ s/ //;
my $src = $source . $name;
my $dst = $dest . $name;
print "emcopy \"$src\" \"$ dst\" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:$log\n";
}
close(CMD);
The Output of the script looks like this (this is an excerpt from the screen output):
emcopy "\\celerra_path\Data01" "\\isilon_path\billing_tmip_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_tmip_01 emcopy "\\celerra_path\Data02" "\\isilon_path\billing_trxs_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_trxs_01 emcopy "\\celerra_path\Data03" "\\isilon_path\billing_vru_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_vru_01 emcopy "\\celerra_path\Data04" "\\isilon_path\billing_rpps_01" /o /s /d /q /secfix /purge /stream /c /r:1 /w:1 /log:c:\billing_rpps_01
That’s it. Good luck with your data migration, and I hope this has been of some assistance. Special thanks to Mark May and his virtualstoragezone blog, he published the original versions of these scripts here.
