I recently had an issue attempting to extend our production NAS file pool on our NS-960.  We had just added Six new 2TB SATA disks to the array, and when I launched the Disk Provisioning Wizard it gave me this error:

“The number of drives available for provisioning additional storage are insufficient”.

That of course wasn’t true, as a 4+2 RAID6 config is indeed supported on this platform and I had just added six drives. I did come up with a workaround to do it manually thanks to some helpful advice from our local EMC technical rep.  I manually created a RAID6 Raid Group in a 4+2 config, and then created a single LUN using all of the available space in the Raid Group (about 7337GB).   Once the LUN is created, you can add it to the Celerrra storage group, in my case it was named “Celerra_hostname”.

When adding the LUN to the storage group, there is a critical step that you must not skip.  The HLU number must be modified!  After you click on a LUN, click add, look for it in the list and notice that the far right column shows the HLU (Host LUN ID).  The LUN you just added will have a blank entry.  It doesn’t look like it’s an editable field, but it is – simply click on the blank area where the number should be and you’ll get a drop down box.  The number you chose must be greater than 15.  Once you’ve modified the HLU for the new LUN, then click on OK to complete the process.

Next, you’ll want to switch back over to the Celerra Management interface, click on the ‘Storage’ tab, then click on the ‘Rescan Storage Systems’ link.  You will get a warning message that states:

“Rescan detects newly available storage and storage systems. Do not rescan unless all primary Data Movers are operating normally. The operation might take a few minutes to complete.”  

Heed the warning and make sure your data movers are up and functional.  You can monitor the progress in the background tasks area.   On my first attempt the Rescan failed.  I got this error message:

“Storage API code=3593: SYMAPI_C_CLARIION_LOAD_ERROR.  An error occurred while data was being loaded from a Clariion.” | “No additional information is available” | “No recommended action is available”.

At the point I got that error I was at the end of my work day and decided to get back to it the next day.  I had planned on opening an SR.  When I re-ran the same scan the next day it worked fine and my production pool auto-extended.  Problem solved.

Server_stats is a CLI based real time performance monitoring tool from EMC for the Celerra and VNX file.  This post is meant to give a quick overview of the server_stats command with some samples on using it in a scheduled cron job. If you’re looking to dive into using the server_stats feature, I’d suggest using the online manual pages (man server_stats) to get a good idea of all the features and reviewing the “Managing Statistics for VNX” Guide from EMC here:  http://corpusweb130.emc.com/upd_prod_VNX/UPDFinalPDF/en/Statistics.pdf.

I don’t personally use it so I can’t explain how to set it up, but there is an opensource tool that you can use to push server_stats data to graphite called vnx2graphite. You can get it here: http://www.findbestopensource.com/product/fatz-vnx2graphite.   You can download Graphite here: http://graphite.wikidot.com/.

Here is the command line syntax:

server_stats <movername>

   -list
| -info [-all|<statpath_name>[,...]]
| -service { -start [-port <port_number>]

| -stop
| -delete
| -status }

 | -monitor -action {status|enable|disable}
|[

[{ -monitor {statpath_name|statgroup_name}[,...]
| -monitor {statpath_name|statgroup_name}
[-sort <field_name>]
[-order {asc|desc}]
[-lines <lines_of_output>]

 }…]
[-count <count>]
[-interval <seconds>]
[-terminationsummary {no|yes|only}]
[-format {text [-titles {never|once|<repeat_frequency>}]|csv}]
[-type {rate|diff|accu}]
[-file <output_filepath> [-overwrite]]
[-noresolve]
]

Here’s an explanation of a few of the useful table options and what to look for:

Syntax:  server_stats server_2 -i <interval in sec> -c <# of counts> -table <stat>

table cifs 

-Look at uSec/call. The output is in microseconds, divide by 1000 to convert to milliseconds. This tells you how long it takes the celerra to perform specific CIFS operations.

table dvol 

-This is for disk stats.  It shows the write distribution across all volumes.  Look for IO balance across resources.

table fsvol 

-Use this to check filesystem IO.  You’ll be able to monitor which file systems are getting all of the IO with this table.

Start with an interval of 1 first to look for spikes or bursts and then increase it incrementally (10 seconds, 30 seconds, 1 minute, 5 minutes, etc). You can also use Celerra monitor to get Clariion stats.  Look at queueing, cache flushes, etc.  Writes should be through to cache on the Clariion, and unless your write cache is filling up they should be faster than reads.

Here are some sample commands and what they do:

server_stats server_2 -table fsvol -interval 1 -count 10

-This correlates the filesystem to the meta-volumes and shows the % contribution of write requests for each meta-volume (FS Write Reqs %).

server_stats server_2 -table net -interval 1 -count 10

-This shows Network in (KiB/s) / Network In (Pkts/s) to figure out the packet size.  Do this for in and for out to verify the standard MTU size.

server_stats server_2 -summary nfs,cifs -interval 1 -count 10

-This will give a summary of performance stats for nfs and cifs.

Here are some additional sample commands, and how you can add to your crontab to automatically collect performance data:

Collect CIFS and NFS data every 5 minutes:

*/5 * * * * /nas/bin/server_stats server_2 -monitor cifs.smb1,cifs.smb2,nfs.v2,nfs.v3,nfs.v4,cifs.global,nfs.basic -format csv -terminationsummary no -i 5 -c 60 -type accu -file “/nas/quota/slot_2/perfstats/data/server_2/server_2_`date ‘+\%FT\%T’|sed s/://g`” > /dev/null

In the command above the -type accu option tells the command to accumulates statistics upon each capture rather than starting back at a baseline of zero. You can also do ‘diff’ to capture the difference from interval to interval.

Collect diskVol performance stats every 5 minutes:

*/5 * * * * /nas/bin/server_stats server_2 -monitor diskVolumes-std -i 5 -c 60 -file “/nas/quota/slot_2/perfstats/data/server_2/server_2_`date ‘+\%FT\%T’|sed s/://g`” > /dev/null

Collect top_talkers data every 5 minutes:

*/5 * * * * /nas/bin/server_stats server_2 -monitor nfs.client -i 5 -c 60 -file “/nas/quota/slot_2/perfstats/data/server_2/server_2_`date ‘+\%FT\%T’|sed s/://g`” > /dev/null

Below are some useful nfs and cifs stats that you can monitor (pulled from DART 8.1.2-51).  For a full list, run the command server_stats server_2 -i.

cifs.global.basic.totalCalls,cifs.global.basic.reads,cifs.global.basic.readBytes,cifs.global.basic.readAvgSize,cifs.global.basic.writes,cifs.global.basic.writeBytes,
cifs.global.basic.writeAvgSize,cifs.global.usage.currentConnections,cifs.global.usage.currentOpenFiles

nfs.basic,nfs.client,nfs.currentThreads,nfs.export,nfs.filesystem,nfs.group,nfs.maxUsedThreads,nfs.totalCalls,nfs.user,nfs.v2,nfs.v3,nfs.v4,nfs.vdm

We were having issues with our backup jobs failing on CIFS share backups using Symantec Netbackup.  The jobs died with a “status 24″, which means it was losing communicaiton with the source.  Our backup administrator provided me with the exact times & dates of the failures and I noticed that immediately preceding his failures this error appeared in the server log on the control station:

A quick google search came up with this description of the error:  “The maximum amount of memory (number of 8K pages) allowed for dynamic memory allocation has almost been reached. This indicates that a possible memory leak is in progress and the Data Mover may soon panic. If Max=0(zero) then the system forced panic option is disabled. If Max is not zero then the system will force a panic if dynamic memory allocation reaches this level.”

Based on the fact that the error shows up right before a backup failure I saw the correlation.  To fix it, you’lll need to modify the Heap Limit from the default of 0x00030000 to a larger size.  Here is the command to do that:

Below is a list of all the kernel parameters:

Name                                                 Location        Current       Default
----                                                 ----------      ----------    ----------
kernel.AutoconfigDriverFirst                         0x0003b52d30    0x00000000    0x00000000
kernel.BufferCacheHitRatio                           0x0002093108    0x00000050    0x00000050
kernel.MSIXdebug                                     0x0002094714    0x00000001    0x00000001
kernel.MSIXenable                                    0x000209471c    0x00000001    0x00000001
kernel.MSI_NoStop                                    0x0002094710    0x00000001    0x00000001
kernel.MSIenable                                     0x0002094718    0x00000001    0x00000001
kernel.MsiRouting                                    0x0002094724    0x00000001    0x00000001
kernel.WatchDog                                      0x0003aeb4e0    0x00000001    0x00000001
kernel.autoreboot                                    0x0003a0aefc    0x00000258    0x00000258
kernel.bcmTimeoutFix                                 0x0002179920    0x00000002    0x00000002
kernel.buffersWatermarkPercentage                    0x0003ae964c    0x00000021    0x00000021
kernel.bufreclaim                                    0x0003ae9640    0x00000001    0x00000001
kernel.canRunRT                                      0x000208f7a0    0xffffffff    0xffffffff
kernel.dumpcompress                                  0x000208f794    0x00000001    0x00000001
kernel.enableFCFastInit                              0x00022c29d4    0x00000001    0x00000001
kernel.enableWarmReboot                              0x000217ee68    0x00000001    0x00000001
kernel.forceWholeTLBflush                            0x00039d0900    0x00000000    0x00000000
kernel.heapHighWater                                 0x00020930c8    0x00004000    0x00004000
kernel.heapLowWater                                  0x00020930c4    0x00000080    0x00000080
kernel.heapReserve                                   0x00020930c0    0x00022e98    0x00022e98
kernel.highwatermakpercentdirty                      0x00020930e0    0x00000064    0x00000064
kernel.lockstats                                     0x0002093128    0x00000001    0x00000001
kernel.longLivedChunkSize                            0x0003a23ed0    0x00002710    0x00002710
kernel.lowwatermakpercentdirty                       0x0003ae9654    0x00000000    0x00000000
kernel.mallocHeapLimit                               0x0003b5558c    0x00040000    0x00030000  (This is the parameter I changed)
kernel.mallocHeapMaxSize                             0x0003b55588    0x00000000    0x00000000
kernel.maskFcProc                                    0x0002094728    0x00000004    0x00000004
kernel.maxSizeToTryEMM                               0x0003a23f50    0x00000008    0x00000008
kernel.maxStrToBeProc                                0x0003b00f14    0x00000080    0x00000080
kernel.memSearchUsecs                                0x000208fa28    0x000186a0    0x000186a0
kernel.memThrottleMonitor                            0x0002091340    0x00000001    0x00000001
kernel.outerLoop                                     0x0003a0b508    0x00000001    0x00000001
kernel.panicOnClockStall                             0x0003a0cf30    0x00000000    0x00000000
kernel.pciePollingDefault                            0x00020948a0    0x00000001    0x00000001
kernel.percentOfFreeBufsToFreePerIter                0x00020930cc    0x0000000a    0x0000000a
kernel.periodicSyncInterval                          0x00020930e4    0x00000005    0x00000005
kernel.phTimeQuantum                                 0x0003b86e18    0x000003e8    0x000003e8
kernel.priBufCache.ReclaimPolicy                     0x00020930f4    0x00000001    0x00000001
kernel.priBufCache.UsageThreshold                    0x00020930f0    0x00000032    0x00000032
kernel.protect_zero                                  0x0003aeb4e8    0x00000001    0x00000001
kernel.remapChunkSize                                0x0003a23fd0    0x00000080    0x00000080
kernel.remapConfig                                   0x000208fe40    0x00000002    0x00000002
kernel.retryTLBflushIPI                              0x00020885b0    0x00000001    0x00000001
kernel.roundRobbin                                   0x0003a0b504    0x00000001    0x00000001
kernel.setMSRs                                       0x0002088610    0x00000001    0x00000001
kernel.shutdownWdInterval                            0x0002093238    0x0000000f    0x0000000f
kernel.startAP                                       0x0003aeb4e4    0x00000001    0x00000001
kernel.startIdleTime                                 0x0003aeb570    0x00000001    0x00000001
kernel.stream.assert                                 0x0003b00060    0x00000000    0x00000000
kernel.switchStackOnPanic                            0x000208f8e0    0x00000001    0x00000001
kernel.threads.alertOptions                          0x0003a22bf4    0x00000000    0x00000000
kernel.threads.maxBlockedTime                        0x000208f948    0x00000168    0x00000168
kernel.threads.minimumAlertBlockedTime               0x000208f94c    0x000000b4    0x000000b4
kernel.threads.panicIfHung                           0x0003a22bf0    0x00000000    0x00000000
kernel.timerCallbackHistory                          0x000208f780    0x00000001    0x00000001
kernel.timerCallbackTimeLimitMSec                    0x000208f784    0x00000003    0x00000003
kernel.trackIntrStats                                0x000209021c    0x00000001    0x00000001
kernel.usePhyDevName                                 0x0002094720    0x00000001    0x00000001

I made a blog post back in January 2014 about creating an NFS export on a virtual data mover but I didn’t give much detail on the commands you need to use to actually do it. As I pointed out back then, you can’t NFS export a VDM file system from within Unisphere however when a file system is mounted on a VDM its path from the root of the physical Data Mover can be exported from the CLI.

The first thing that needs to be done is determining the physical Data Mover where the VDM resides.

Below is the command you’d use to make that determination:

[nasadmin@Celerra_hostname]$ nas_server -i -v name_of_your_vdm | grep server
server = server_4

That will show you just the physical data mover that it’s mounted on. Without the grep statement, you’d get the output below. If you have hundreds of filesystems it will cause the screen to scroll the info you’re looking for off the top of the screen. Using grep is more efficient.

[nasadmin@Celerra_hostname]$ nas_server -i -v name_of_your_vdm
id = 1
name = name_of_your_vdm
acl = 0
type = vdm
server = server_4
rootfs = root_fs_vdm_name_of_your_vdm
I18N mode = UNICODE
mountedfs = fs1,fs2,fs3,fs4,fs5,fs6,fs7,fs8,…
member_of =
status :
defined = enabled
actual = loaded, active
Interfaces to services mapping:
interface=10-3-20-167 :cifs
interface=10-3-20-130 :cifs
interface=10-3-20-131 :cifs

Next you need to determine the file system path from the root of the Data Mover. This can be done with the server_mount command. As in the prior step, it’s more efficient if you grep for the name of the file system. You can run it without the grep command, but it could generate multiple screens of output depending on the number of file systems you have.

[nasadmin@stlpemccs04a /]$ server_mount server_4 | grep Filesystem_03
Filesystem_03 on /root_vdm_3/Filesystem_03 uxfs,perm,rw

The final step is to actually export the file system using this path from the prior step. The file system must be exported from the root of the Data Mover rather than the VDM. Note that once you have exported the VDM file system from the CLI, you can then manage it from within Unisphere if you’d like to set server permissions. The “-option anon=0,access=server_name,root=server_name” portion of the CLI command below can be left off if you’d prefer to use the GUI for that.

[nasadmin@Celerra_hostname]$ server_export server_4 -Protocol nfs -option anon=0,access=server_name,root=server_name /root_vdm_3/Filesystem_03
server_4 : done

At this point the client can mount the path with NFS.

Our naming convention for LUNs includes the pool ID, LUN number, server name, filesystem/drive letter, last four digits of the array’s serial number, and size (in GB). Having all of this information in the LUN name makes for very easy reporting and identification of LUNs on a server.  This is what our LUN names look like: P1_LUN100_SPA_0000_servername_filesystem_150G

Typically, when presenting a new LUN to our AIX administration team for a new server build, they would assign the LUNs to specific volume groups based on the LUN names. The command ‘powermt display dev=hdiskpower#’ always includes the name & intended volume group for the LUN, making it easy for our admins to identify a LUN’s purpose.  Now that we are presenting LUNs through our VPlex, when they run a powermt display on the server the UID for the LUN is shown, not the name.  Below is a sample output of what is displayed.

root@VIOserver1:/ # powermt display dev=all
Pseudo name=hdiskpower0
VPLEX ID=FNM00141800023
Logical device ID=6000144000000010704759ADDF2487A6 (this would usually be displayed as a LUN name)
state=alive; policy=ADaptive; queued-IOs=0
==============================================================================
————— Host ————— – Stor – — I/O Path — — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
1 fscsi1 hdisk8 CL1-0B active alive 0 0
1 fscsi1 hdisk6 CL1-0F active alive 0 0
0 fscsi0 hdisk4 CL1-0D active alive 0 0
0 fscsi0 hdisk2 CL1-07 active alive 0 0

Pseudo name=hdiskpower1
VPLEX ID=FNM00141800023
Logical device ID=6000144000000010704759ADDF2487A1 (this would usually be displayed as a LUN name)
state=alive; policy=ADaptive; queued-IOs=0
==============================================================================
————— Host ————— – Stor – — I/O Path — — Stats —
### HW Path I/O Paths Interf. Mode State Q-IOs Errors
==============================================================================
1 fscsi1 hdisk9 CL1-0B active alive 0 0
1 fscsi1 hdisk7 CL1-0F active alive 0 0
0 fscsi0 hdisk5 CL1-0D active alive 0 0
0 fscsi0 hdisk3 CL1-07 active alive 0 0

In order to easily match up the UIDs with the LUN names on the server, an extra step needs to be taken on the VPlex CLI. Log in to the VPlex using a terminal emulator, and once you’re logged in use the ‘vplexcli’ command. That will take you to a shell that allows for additional commands to be entered.

login as: admin
Using keyboard-interactive authentication.
Password:
Last login: Fri Sep 19 13:35:28 2014 from 10.16.4.128
admin@service:~> vplexcli
Trying ::1…
Connected to localhost.
Escape character is ‘^]’.

Enter User Name: admin

Password:

VPlexcli:/>

Once you’re in, run the ls -t command with the additional options listed below. You will need to substitute the STORAGE_VIEW_NAME with the actual name of the storage view that you want a list of LUNs from.

VPlexcli:/> ls -t /clusters/cluster-1/exports/storage-views/STORAGE_VIEW_NAME::virtual-volumes

The output looks like this:

/clusters/cluster-1/exports/storage-views/st1pvio12a-b:
Name Value
————— ————————————————————————————————–
virtual-volumes [(0,P1_LUN411_7872_SPB_VIOServer1_VIO_10G,VPD83T3:6000144000000010704759addf2487a6,10G),
(1,P0_LUN111_7872_SPA_VIOServer1_VIO_10G,VPD83T3:6000144000000010704759addf2487a1,10G)]

Now you can easily see which disk UID is tied to which LUN name.

If you would like to get a list of every storage view and every LUN:UID mapping, you can substitute the storage view name with an asterisk (*).

VPlexcli:/> ls -t /clusters/cluster-1/exports/storage-views/*::virtual-volumes

The resulting report will show a complete list of LUNs, grouped by storage view:

/clusters/cluster-1/exports/storage-views/VIOServer1:
Name Value
————— ————————————————————————————————–
virtual-volumes [(0,P1_LUN421_9322_SPB_/clusters/cluster-1/exports/storage-views/ VIOServer2:
Name Value
————— ————————————————————————————————–
virtual-volumes [(0,P1_LUN421_9322_SPB_VIOServer2_root_75G,VPD83T3:6000144000000010704759addf248ad9,75G),
(1,R2_LUN125_9322_SPB_VIOServer2_redo2_12G,VPD83T3:6000144000000010704759addf248b09,12G),
(2,R2_LUN124_9322_SPA_VIOServer2_redo1_12G,VPD83T3:6000144000000010704759addf248b04,12G),
(3,P3_LUN906_9322_SPB_VIOServer2_oraarc_250G,VPD83T3:6000144000000010704759addf248aff,250G),
(4,P2_LUN706_9322_SPA_VIOServer2_oraarc_250G,VPD83T3:6000144000000010704759addf248afa,250G)]

/clusters/cluster-1/exports/storage-views/VIOServer2:
Name Value
————— ————————————————————————————————
virtual-volumes [(1,R2_LUN1025_9322_SPB_VIOServer2_redo2_12G,VPD83T3:6000144000000010704759addf248b09,12G),
(2,R2_LUN1024_9322_SPA_VIOServer2_redo1_12G,VPD83T3:6000144000000010704759addf248b04,12G),
(3,P3_LUN906_9322_SPB_VIOServer2_ora1_250G,VPD83T3:6000144000000010704759addf248aff,250G),
(4,P2_LUN706_9322_SPA_VIOServer2_ora2_250G,VPD83T3:6000144000000010704759addf248afa,250G)]

/clusters/cluster-1/exports/storage-views/VIOServer3:
Name Value
————— ————————————————————————————————
virtual-volumes [(0,P0_LUN101_3432_SPA_VIOServer3_root_75G,VPD83T3:6000144000000010704759addf248a0a,75G),
(1,P0_LUN130_3432_SPA_VIOServer3_redo1_25G,VPD83T3:6000144000000010704759addf248a0f,25G),

Our VPlex has only been installed for a few months and our team is still learning.  There may be a better way to do this, but it’s all I’ve been able to figure out so far.

This is a brief post to share the CLI commands and sample output for a quick VPLEX health check.  Our VPLEX had a dial home event and below are the commands that EMC ran to verify that it was healthy.  Here is the dial home event that was generated:

SymptomCode: 0x8a266032
SymptomCode: 0x8a34601a
Category: Status
Severity: Error
Status: Failed
Component: CLUSTER
ComponentID: director-1-1-A
SubComponent: stdf
CallHome: Yes
FirstTime: 2014-11-14T11:20:11.008Z
LastTime: 2014-11-14T11:20:11.008Z
CDATA: Compare and Write cache transaction submit failed, status 1 [Versions:MS{D30.60.0.3.0, D30.0.0.112, D30.60.0.3}, Director{6.1.202.1.0}, ClusterWitnessServer{unknown}] RCA: The attempt to start a cache transaction for a Scsi Compare and Write command failed. Remedy: Contact EMC Customer Support.

Description: The processing of a Scsi Com pare and Write command could not complete.
ClusterID: cluster-1

Based on that error the commands below were run to make sure the cluster was healthy.

This is the general health check command:

VPlexcli:/> health-check
 Product Version: 5.3.0.00.00.10
 Product Type: Local
 Hardware Type: VS2
 Cluster Size: 2 engines
 Cluster TLA:
 cluster-1: FNM00141800023
 
 Clusters:
 ---------
 Cluster Cluster Oper Health Connected Expelled Local-com
 Name ID State State
 --------- ------- ----- ------ --------- -------- ---------
 cluster-1 1 ok ok True False ok
 
 Meta Data:
 ----------
 Cluster Volume Volume Oper Health Active
 Name Name Type State State
 --------- ------------------------------- ----------- ----- ------ ------
 cluster-1 c1_meta_backup_2014Nov21_100107 meta-volume ok ok False
 cluster-1 c1_meta_backup_2014Nov20_100107 meta-volume ok ok False
 cluster-1 c1_meta meta-volume ok ok True
 
 Director Firmware Uptime:
 -------------------------
 Director Firmware Uptime
 -------------- ------------------------------------------
 director-1-1-A 147 days, 16 hours, 15 minutes, 29 seconds
 director-1-1-B 147 days, 15 hours, 58 minutes, 3 seconds
 director-1-2-A 147 days, 15 hours, 52 minutes, 15 seconds
 director-1-2-B 147 days, 15 hours, 53 minutes, 37 seconds
 
 Director OS Uptime:
 -------------------
 Director OS Uptime
 -------------- ---------------------------
 director-1-1-A 12:49pm up 147 days 16:09
 director-1-1-B 12:49pm up 147 days 16:09
 director-1-2-A 12:49pm up 147 days 16:09
 director-1-2-B 12:49pm up 147 days 16:09
 
 Inter-director Management Connectivity:
 ---------------------------------------
 Director Checking Connectivity
 Enabled
 -------------- -------- ------------
 director-1-1-A Yes Healthy
 director-1-1-B Yes Healthy
 director-1-2-A Yes Healthy
 director-1-2-B Yes Healthy
 
 Front End:
 ----------
 Cluster Total Unhealthy Total Total Total Total
 Name Storage Storage Registered Ports Exported ITLs
 Views Views Initiators Volumes
 --------- ------- --------- ---------- ----- -------- -----
 cluster-1 56 0 299 16 353 9802
 
 Storage:
 --------
 Cluster Total Unhealthy Total Unhealthy Total Unhealthy No Not visible With
 Name Storage Storage Virtual Virtual Dist Dist Dual from Unsupported
 Volumes Volumes Volumes Volumes Devs Devs Paths All Dirs # of Paths
 --------- ------- --------- ------- --------- ----- --------- ----- ----------- -----------
 cluster-1 203 0 199 0 0 0 0 0 0
 
 Consistency Groups:
 -------------------
 Cluster Total Unhealthy Total Unhealthy
 Name Synchronous Synchronous Asynchronous Asynchronous
 Groups Groups Groups Groups
 --------- ----------- ----------- ------------ ------------
 cluster-1 0 0 0 0
 
 Cluster Witness:
 ----------------
 Cluster Witness is not configured

This command checks the status of the cluster:

VPlexcli:/> cluster status
Cluster cluster-1
operational-status: ok
transitioning-indications:
transitioning-progress:
health-state: ok
health-indications:
local-com: ok

This command checks the state of the storage volumes:

VPlexcli:/> storage-volume summary
Storage-Volume Summary (no tier)
---------------------- --------------------

Health out-of-date 0
storage-volumes 203
unhealthy 0

Vendor DGC 203

Use meta-data 4
used 199

Capacity total 310T

We recently ran into an SP bug check on one of our VNX arrays and after it came back up several of the initiator paths to the VPLEX did not come back up.  We were also seeing IO timeouts.  This is a known bug that happens when there is an SP reboot and is fixed with Patch 1 for GeoSynchrony 5.3.  EMC has released a script that provides a workaround until the patch can be applied: https://download.emc.com/downloads/DL56253_VPLEX_VNX_SCRIPT.zip.zip

The following pre-conditions need to happen during a VNX NDU to see this issue on VPLEX:
1] During a VNX NDU, SPA goes down.
2] At this point IO time-outs start happening on IT nexus’s pertaining to SPA.
3] The IO time-outs cause the VPLEX SCSI Layer to send LU Reset TMF’s. These LU Reset TMF’s get timed out as well.

You can review ETA 000193541 on EMC’s support site for more information.  It’s a critical bug and I’d suggest patching as soon as possible.

I was attempting to extend our main production NAS file pool on our NS-960 and ran into an issue.  I had recently freed up 8 SATA disks from a block pool and was attempting to re-use them and extend a Celerra file pool.  I created a new RAID Group and LUN that used the maximum capacity of the RAID Group.  I then added the LUN to the celerra storage group, making sure to set the HLU to a number greater than 15.  I then changed the setting on our main production file pool to auto-extend, and clicked on the “Rescan Storage Systems” option.  Unfortunately rescanning produced an error every time it was run.  I have done this exact same procedure in the past and it’s worked fine.  Here is the error:

conflict:storageID-devID: disk=17 old:symm=APM00100600999,dev=001F new:symm=APM00100600999,dev=001F addr=c16t1l11

I checked the disks on the Celerra using the nas_disk –l command, and the new disk shows up as “in use” even though the rescan command didn’t properly complete.

[nasadmin@Celerra tools]$ nas_disk -l
id   inuse  sizeMB    storageID-devID      type   name  servers
17    y     7513381   APM00100600999-001F  CLATA  d17   <BLANK>

Once the dvol is presented to Celerra (assuming the rescan goes fine) it should not be inuse until it is assigned to a storage pool and a file system uses it.  In this case that didn’t happen.  If you run /nas/tools/whereisfs (depending on your DART version, it may be “.whereisfs” with the dot) it shows a listing of every file system and which disk and which LUN they reside on.  I verified that the disk was not in use using that command.

In order to be on the safe side, I opened an SR with EMC rather than simply deleting the disk.  They suggested that the NAS database has a corruption. I’m going to have EMC’s Recovery Team check the usage of the diskvol and then delete it and re-add it.  In order to engage the recovery team you need to sign a “Data Deletion Form” absolving EMC of any liability for data loss, which is standard practice when they delete volumes on a customer array.  If there are any further caveats or important things to note after EMC has taken care of this I’ll update this post.

I’m at EMC World in Las Vegas this week and it’s been fantastic so far.  I’m excited about the new 40TB XtremIO X-bricks and how we might leverage that for our largest and most important 80TB oracle database, also excited about possible use cases for  the Virtual VNX in our small branch locations, and all the other exciting futures that I can’t publicly share because I’m under an NDA with EMC.  Truly exciting and innovative technology is coming from them.  VXblock was also really impressive, although that’s not likely something my company will implement anytime soon.

I found out for the first time today that the excellent VNX Monitoring and Reporting application is now free for the VNX1 platform as well as VNX2.  If you would like to get a license for any of your VNX1 arrays, simply ask your local  sales representative to submit a zero dollar sales order for a license.  We’re currently evaluating ViPR SRM as a replacement for our soon to be “end of life” Control Center install, but until then VNX MR is a fantastic tool that provides nearly the same performance data for no cost at all.  SRM adds much more functionality beyond just VNX monitoring and reporting (i.e., monitoring SAN switches) and I’d highly recommend doing a demo if you’re also still using Control Center.

We also implemented a VPLEX last year and it’s truly been a lifesaver and is an amazing platform.  We currently have a VPLEX local implantation in our primary data center and it’s allowed us to easily migrate workloads from one array to another seamlessly with no disruption to applications.   I’m excited about the possibilities with RecoverPoint as well, I’m still learning about it.

If anyone else who’s at EMC World happens to read this, comment!  I’d love to hear your experiences and what you’re most excited about with EMC’s latest technology.

It was recently asked to come up with a method to alert on the availability of specific CIFS file shares in our environment.  This was due to a recent issue we had on our VNX with our data mover crashing and causing the corruption of a single file system when it came back up.  We were unaware for several hours of the one file system being unavailable on our CIFS server.

This particular script would require maintenance whenever a new file system share is added to a CIFS server.  A unique line must to be added for every file system share that you have configured.  If a file system is not mounted and the share is inaccessible, an email alert will be sent.  If the share is accessible the script does nothing when run from the scheduler.  If it’s run manually from the CLI, it will echo back to the screen that the path is active.

This is a bash shell script, I run it on a windows server with Cygwin installed using the ‘email’ package for SMTP.  It should also run fine from a linux server, and you could substitute the ‘email’ syntax for sendmail or whatever other mail application you use.   I have it scheduled to check the availability of CIFS shares every one hour.

DIR1=file_system_1; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne “: ” && [ -d //$SRV1/$DIR1 ] && echo “Network Path is Active” || email -b -s “Network Path \\\\$SRV1\\$DIR1 is offline” emailaddress@email.com

DIR2=file_system_2; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne “: ” && [ -d //$SRV1/$DIR2 ] && echo “Network Path is Active” || email -b -s “Network Path \\\\$SRV1\\$DIR2 is offline” emailaddress@email.com

DIR3=file_system_3; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne “: ” && [ -d //$SRV1/$DIR3 ] && echo “Network Path is Active” || email -b -s “Network Path \\\\$SRV1\\$DIR3 is offline” emailaddress@email.com

DIR4=file_system_4; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne “: ” && [ -d //$SRV1/$DIR4 ] && echo “Network Path is Active” || email -b -s “Network Path \\\\$SRV1\\$DIR4 is offline” emailaddress@email.com

DIR5=file_system_5; SRV1=cifs_servername;  echo -ne $DIR1 && echo -ne “: ” && [ -d //$SRV1/$DIR5 ] && echo “Network Path is Active” || email -b -s “Network Path \\\\$SRV1\\$DIR5 is offline” emailaddress@email.com

Here is a collection of default passwords for EMC, HP, Cisco, VMware, TrendMicro and IBM hardware & software.

EMC Secure Remote Support (ESRS) Axeda Policy Manager Server:

• Username: admin
• Password: EMCPMAdm7n

EMC VNXe Unisphere (EMC VNXe Series Quick Start Guide, step 4):

• Username: admin
• Password: Password123#

EMC vVNX Unisphere:

• Username: admin
• Password: Password123#
  NB You must change the administrator password during this first login.

EMC CloudArray Appliance:

• Username: admin
• Password: password
  NB Upon first login you are prompted to change the password.

EMC CloudBoost Virtual Appliance:
https://<FQDN&gt;:4444

• Username: local\admin
• Password: password
  NB You must immediately change the admin password.
  $ password <current_password> <new_password>

EMC Ionix Unified Infrastructure Manager/Provisioning (UIM/P):

• Username: sysadmin
• Password: sysadmin

EMC VNX Monitoring and Reporting:

• Username: admin
• Password: changeme

EMC RecoverPoint:

• Username: admin
  Password: admin
• Username: boxmgmt
  Password: boxmgmt
• Username: security-admin
  Password: security-admin

EMC XtremIO:

XtremIO Management Server (XMS)

• Username: xmsadmin
  password: 123456 (prior to v2.4)
  password: Xtrem10 (v2.4+)

XtremIO Management Secure Upload

• Username: xmsupload
  Password: xmsupload

XtremIO Management Command Line Interface (XMCLI)

• Username: tech
  password: 123456 (prior to v2.4)
  password: X10Tech! (v2.4+)

XtremIO Management Command Line Interface (XMCLI)

• Username: admin
  password: 123456 (prior to v2.4)
  password: Xtrem10 (v2.4+)

XtremIO Graphical User Interface (XtremIO GUI)

• Username: tech
  password: 123456 (prior to v2.4)
  password: X10Tech! (v2.4+)

XtremIO Graphical User Interface (XtremIO GUI)

• Username: admin
  password: 123456 (prior to v2.4)
  password: Xtrem10 (v2.4+)

XtremIO Easy Installation Wizard (on storage controllers / nodes)

• Username: xinstall
  Password: xiofast1

XtremIO Easy Installation Wizard (on XMS)

• Username: xinstall
  Password: xiofast1

Basic Input/Output System (BIOS) for storage controllers / nodes

• Password: emcbios

Basic Input/Output System (BIOS) for XMS

• Password: emcbios

EMC ViPR Controller :
http://ViPR_virtual_ip (the ViPR public virtual IP address, also known as the network.vip)

• Username: root
  Password: ChangeMe

EMC ViPR Controller Reporting vApp:
http://<hostname&gt;:58080/APG/

• Username: admin
  Password: changeme

EMC Solutions Integration Service:
https://<Solutions Integration Service IP Address>:5480

• Username: root
  Password: emc

EMC VSI for VMware vSphere Web Client:
https://<Solutions Integration Service IP Address>:8443/vsi_usm/

• Username: admin
• Password: ChangeMe

Note:
After the Solutions Integration Service password is changed, it cannot be modified.
If the password is lost, you must redeploy the Solutions Integration Service and use the default login ID and password to log in.

Cisco Integrated Management Controller (IMC) / CIMC / BMC:

• Username: admin
• Password: password

Cisco UCS Director:

• Username: admin
• Password: admin
• Username: shelladmin
• Username: changeme

Hewlett Packard P2000 StorageWorks MSA Array Systems:

• Username: admin
• Password: !admin (exclamation mark ! before admin)
• Username: manage
• Password: !manage (exclamation mark ! before manage)

VCE Vision:

• Username: admin
• Password: 7j@m4Qd+1L
• Username: root
• Password: V1rtu@1c3!

VMware vSphere Management Assistant (vMA):

• Username: vi-admin
• Password: vmware

VMware Data Recovery (VDR):

• Username: root
• Password: vmw@re (make sure you enter @ as Shift-2 as in US keyboard layout)

VMware vCenter Hyperic Server:
https://Server_Name_or_IP:5480/

• Username: root
• Password: hqadmin

https://Server_Name_or_IP:7080/

• Username: hqadmin
• Password: hqadmin

VMware vCenter Chargeback:
https://Server_Name_or_IP:8080/cbmui

• Username: root
• Password: vmware

VMware vCenter Server Appliance (VCSA) 5.5:
https://Server_Name_or_IP:5480

• Username: root
• Password: vmware

VMware vCenter Operations Manager (vCOPS):

Console access:

• Username: root
• Password: vmware

Manager:
https://Server_Name_or_IP

• Username: admin
• Password: admin

Administrator Panel:
https://Server_Name_or_IP/admin

• Username: admin
• Password: admin

Custom UI User Interface:
https://Server_Name_or_IP/vcops-custom

• Username: admin
• Password: admin

VMware vCenter Support Assistant:
http://Server_Name_or_IP

• Username: root
• Password: vmware

VMware vCenter / vRealize Infrastructure Navigator:
https://Server_Name_or_IP:5480

• Username: root
• Password: specified during OVA deployment

VMware ThinApp Factory:

• Username: admin
• Password: blank (no password)

VMware vSphere vCloud Director Appliance:

• Username: root
• Password: vmware

VMware vCenter Orchestrator :
https://Server_Name_or_IP:8281/vco – VMware vCenter Orchestrator
https://Server_Name_or_IP:8283 – VMware vCenter Orchestrator Configuration

• Username: vmware
• Password: vmware

VMware vCloud Connector Server (VCC) / Node (VCN):
https://Server_Name_or_IP:5480

• Username: admin
• Password: vmware
• Username: root
• Password: vmware

VMware vSphere Data Protection Appliance:

• Username: root
• Password: changeme

VMware HealthAnalyzer:

• Username: root
• Password: vmware

VMware vShield Manager:
https://Server_Name_or_IP

• Username: admin
• Password: default

  type enable to enter Privileged Mode, password is 'default' as well

Teradici PCoIP Management Console:

• The default password is blank

Trend Micro Deep Security Virtual Appliance (DS VA):

• Login: dsva
• password: dsva

Citrix Merchandising Server Administrator Console:

• User name: root
• password: C1trix321

VMTurbo Operations Manager:

• User name: administrator
• password: administrator
  If DHCP is not enabled, configure a static address by logging in with these credentials:
• User name: ipsetup
• password: ipsetup
  Console access:
• User name: root
• password: vmturbo

I recently had an issue where I was unable to log in to the Unisphere GUI on the VPLEX, it would hang with the message “Retrieving Meta-Volume Information” after progressing about 30% on the progress bar.

This was caused by a hung Java process.  In order to resolve it, you must restart the management server. This will not cause any disruption to hosts connected to the VPLEX.

To do this, run the following command:

ManagementServer:/> sudo /etc/init.d/VPlexManagementConsole restart

If this hangs or does not complete, you will need to run the top command to identify the PID for the java service:

admin@service:~>top
Mem:   3920396k total,  2168748k used,  1751648k free,    29412k buffers
Swap:  8388604k total,    54972k used,  8333632k free,   527732k cached

  PID USER      PR  NI  VIRT  RES  SHR S   %CPU %MEM    TIME+  COMMAND
26993 service   20   0 2824m 1.4g  23m S     14 36.3  18:58.31 java
 4948 rabbitmq  20   0  122m  42m 1460 S      1  1.1  13118:32 beam.smp
    1 root      20   0 10540   48   36 S      0  0.0  12:34.13 init

Once you’ve identified the PID for the java service, you can kill the process with the kill command, and then run the command to restart the management console again.

ManagementServer:/> sudo kill -9 8798
ManagementServer:/> sudo /etc/init.d/VPlexManagementConsole start

Once the management server restarts, you should be able to log in to the Unisphere for VPLEX GUI again.

I ran into an issue where one of our brocade switches was inaccessible via the web browser. The error below was displayed when connecting to the IP:

Interface disabled
This Interface (10.2.2.23) has been blocked by the administrator.

In order to resolve this, you’ll need to allow port 80 traffic on the switch.  It was disabled on mine.

First, Log in to the switch and review the existing IP filters (Look for port 80 set to deny):

switcho1:admin> ipfilter –show

Name: default_ipv4, Type: ipv4, State: active
Rule Source IP Protocol Dest Port Action
1 any tcp 22 permit
2 any tcp 23 deny
3 any tcp 897 permit
4 any tcp 898 permit
5 any tcp 111 permit
6 any tcp 80 deny
7 any tcp 443 permit
8 any udp 161 permit
9 any udp 111 permit
10 any udp 123 permit
11 any tcp 600 – 1023 permit
12 any udp 600 – 1023 permit

Next, clone the default policy, as you cannot make changes to the default policy.  Note that you can name the policy anything you like, I chose to name it “Allow80”.

ipfilter –clone Allow80 -from default_ipv4

Delete the rule that denys port 80 (rule 6 in the above example):

ipfilter –delrule Allow80 -rule 6

Add a rule back in to permit it:

ipfilter –addrule Allow80 -rule 12 -sip any -dp 80 -proto tcp -act permit

Save it:

ipfilter –save Allow80

Activate it (this will change default policy to a “defined” state):

ipfilter –activate Allow80

That’s it… you should now be able to access your switch via the web browser.

We recently had a need to zero out and reinstall a NetApp FAS 8080 in order to move it from test to production.  Below are the steps to zero out the disks in the array.

Steps:

1. SSH to each nodes service-processor
2. Halt each node.
   1. system node halt -node Node1 -inhibit-takeover true
   2. system node halt -node Node2
3. At the Loader prompt for each node boot ONTAP (You might want to do these one at a time so you don’t miss the CTRL-C for the Boot Menu)
   1. LOADER A> boot_ontap
   2. LOADERB> boot_ontap
4. Press Ctrl + C when you see the message below to enter the Boot Menu and select option 4 to wipe the configuration and zero disks (Do this on each node)

*******************************
*                             *
* Press Ctrl-C for Boot Menu. *
*                             *
*******************************

(1) Normal Boot.
(2) Boot without /etc/rc.
(3) Change password.
(4) Clean configuration and initialize all disks.
(5) Maintenance mode boot.
(6) Update flash from backup config.
(7) Install new software first.
(8) Reboot node. Selection (1-8)? 4

5. Enter y to the questions that follow:

Zero disks, reset config and install a new file system?: y

This will erase all the data on the disks, are you sure?: y

The node will reboot and start initializing the disks.  Once the disks are zeroed the system should reboot to the cluster setup.

I had a request to publish a daily report that outlines our data reduction numbers for each LUN on our production Pure storage arrays.  I wrote a script that will log in to the Pure CLI and issue the appropriate command, using ‘expect’ to do a screen grab and output the data to a csv file.  The csv file is then converted to an HTML table and published on our internal web site.

The ‘expect’ commands (saved as purevol.exp in the same directory as the bash script)

#!/usr/bin/expect -f
spawn ssh pureuser@10.10.10.10
expect “logon as: ”
send “pureuser\r”
expect “pureuser@10.10.10.10’s password: ”
send “password\r”
expect “pureuser@pure01> ”
send “purevol list –space\r”
expect “pureuser@pure01> ”
send “exit\r”

.

The bash script (saved as purevol.sh):

#!/bin/bash

# Pure Data Reduction Report Script
# 11/28/16

#Define a timestamp function
#The output looks like this: 6-29-2016/8:45:12
timestamp() {
date +”%m-%d-%Y/%H:%M:%S”
}

#Remove existing output file
rm /home/data/pure/purevol_532D.txt

#Run the expect script to create the output file
/usr/bin/expect -f /home/data/pure/purevol.exp > /home/data/pure/purevol_532D.txt

#Remove the first ten lines of the output file
#The first 12 lines contain login and command execution info not needed in the report
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt
sed -i ‘1d’ /home/data/pure/purevol_532D.txt

#Remove the last line of the output file
#This is because the expect script leaves a CLI prompt as the last line of the output
sed -i ‘$ d’ /home/data/pure/purevol_532D.txt

#Add date to output file, remove previous temp files
rm /home/data/pure/purevol_532D-1.csv
rm /home/data/pure/purevol_532D-2.csv
echo -n “Run time: ” > /home/data/pure/purevol_532D-1.csv
echo $(timestamp) >> /home/data/pure/purevol_532D-1.csv

#Add titles to new csv file
echo “Volume”,”Size”,”Thin Provisioning”,”Data Reduction”,” “,” “,”Total Reduction”,” “,” “,”Volume”,”Snapshots”,”Shared Space”,”System”,”Total” >> /home/data/pure/purevol_532D-1.csv

#Convert the space delimited file into a comma delimited file
sed -r ‘s/^\s+//;s/\s+/,/g’ /home/data/pure/purevol_532D.txt > /home/data/pure/purevol_532D-2.csv

#Combine the csv files into one
cat /home/data/pure/purevol_532D-1.csv /home/data/pure/purevol_532D-2.csv > /home/data/pure/purevol_532D.csv

#Use the csv2htm perl script to convert the csv to an html table
#csv2html script available here:  http://web.simmons.edu/~boyd3/imap/csv2html/
./csv2htm.pl -e -T -i /home/data/pure/purevol_532D.csv -o /home/data/pure/purevol_532D.html

#Copy the html file to the www folder to publish it
cp /home/data/pure/purevol_532D.html /cygdrive/C/inetpub/wwwroot

Below is an example of what the output looks like after the script is run and the output is converted to an HTML table.  Note there are columns missing to the right in order to fit the formatting of this post.  Also included are the numbers for Total reduction and snapshots.

Below is the procedure for changing the passwords on a Unified VNX on both the block and file sides.

Please note that Changing the Global VNX administrator password can cause communication failures between the Control Station and the array, the issue is documented in emc261195 and is the reason I’m adding this post, I was researching how to avoid the issue. The article notes that in DART OS versions newer than v7.0.14 the synchronization was automated and the cached credentials are updated automatically, in DART OS v7.0.14 and prior you must do it manually on the active control station.

Whenever a change is made to the active Control Station always verify that the standby control station configuration matches on takeover.  Takeover is initiated by the standby control station, failover is initiated by the active control station. As an example, if the time zone is changed on the active control station it is not part of the synchronization during the failover process. Time zone changes needs to be configured separately on each one and is a setting that requires a reboot to take effect. Unisphere will prompt you to do so as a reminder, however on takeover/failover the newly promoted control station never reboots.

Block Side: Change the sysadmin global domain account

A) Updating global domain account password

1) Log into Unisphere with the sysadmin global account, using the control station IP

2) From the “All Systems” page select “Domains”

3) Select “Manage Global Users”

4) Highlight sysadmin and click on “Modify”

5) Change the password

B) Update Security on Control Station

1) Open a putty session to the primary control station and run the commands below. They should be as is, however a possible exception would be the first 2 might make you add userid/password info before it would be accepted (add -user sysadmin –password “pswd ” –scope 0 to the commands below).

/nas/sbin/naviseccli -h spa -AddUserSecurity -user sysadmin -scope 0

/nas/sbin/naviseccli -h spb -AddUserSecurity -user sysadmin -scope 0

nas_storage –modify id=1 –security

 C) Verify the updated sysadmin password in the following locations:

1) Via Unisphere (Log in with the newly changed password)

2) Verify communication between the control station and storage processors on each array:

* Log in to the active control station via putty using the nasadmin local account

* Run the following commands:

/nas/sbin/navicli -h SPA getagent

nas_storage -check -all

The NAS storage check command should respond with “done”.

File Side: Change the nasadmin and root local accounts

A) Local accounts need to be modified on each array individually

1) Log into Unisphere with the sysadmin global account

2) Select the desired array

3) Click on “Settings” -> “Security” -> “Local Users for File”

4) Highlight nasadmin click on “Properties”

5) Change the password

6) Highlight root, click on Properties

7) Change the password

Note that the password for the local nasadmin and root accounts can also be changed from the CLI:

[root@fakevnxprompt ~]# passwd nasadmin
Changing password for user nasadmin.
New UNIX password: <enter a password>
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@fakevnxprompt ~]#

B) Verify both passwords

1) Log in to the active control station via putty as nasadmin, verify your newly changed password

2) run the su command and verify your newly changed root password

C) Propagate changes to the standby control station

At this point the standby control station local account passwords have not yet been updated. It’s now time to test control station failover.  You can review one of my related prior blog posts on control station failover here.

1) While logged in to the active control station with root privileges, run this command:

/nas/sbin/cs_standby -failover

This will synchronize the control stations, reboot the active control station, and then make the standby control station active.

Caveats: Please note the expected issues listed below as part of out-of-band communication without an online, active control station:

* In-band production data will not be disrupted

* Data mover failover cannot occur

* Auto-extension of filesystems will not occur

* Scheduled checkpoints will not occur

* Replication sessions may be disrupted

2) Log in to the active control station (the previous standby control station)

3) Verify the new nasadmin password

4) su and verify the root password

5) Fail back to the original primary control station:

/nas/sbin/cs_standby -failover

Below is a command reference for almost all of the CLI commands available in the Isilon OneFS CLI.  The basic commands are outlined and in many cases multiple samples of using the commands are provided.  This information was gathered as an easier to use, more condensed reference than the 1000+ page long CLI Administration guide provided by EMC, however I recommend you refer to that guide for more specific information on the commands.  The information gathered here applies to OneFS 8.0.0.0.

For most data migrations from VNX to Isilon EMC recommends that you use the OneFS migration tool isi_vol_copy_vnx. It can often be more efficient than host-based tools (such as EMCopy and Robocopy) because the performance of host-based tools performance is dependent on the network connectivity of the host, while isi_vol_copy_vnx depends only on the network connection between the source device and the Isilon cluster. Below is a basic outline of the syntax, the steps required, and a few troubleshooting tips.

You might consider migrating data with a host based tool if one or more of the following conditions apply to your migration:

• The source device and Isilon cluster are on separate networks.
• Security restrictions prevent the source device and Isilon cluster from communicating directly.

Command Syntax:

isi_vol_copy_vnx --help

The source must contain a source host, a colon, and then the absolute source path name.

 isi_vol_copy_vnx <src_filer>:<src_dir> <dest_dir>
                 [-sa user: | user:password]
                 [-sport ndmp_src_port]
                 [-dport ndmp_data_port]
                 [-full | -incr] [-level_based]
                 [-dhost dest_ip_addr]
                 [-no_acl]

 isi_vol_copy_vnx -list [migration-id] | [[-detail] [-state=<state>] [-destination=<pathname>]]
 isi_vol_copy_vnx -cleanup <migration-id> [-everything [-noprompt]]
 isi_vol_copy_vnx -get_config
 isi_vol_copy_vnx -set_config <name>=<value>
 isi_vol_copy_vnx -h | -help

 Defaults:
   src_auth_user       = root
   src_auth_password   =
   ndmp_src_port       = 0  (0 means NDMP default, usually 10000)
   ndmp_data_port      = any
   dest_ip_addr        = none

Note: This tool uses NDMP to transfer the data from the source VNX to the Isilon.

Migration Steps:

1. Configure NDMP User

Create a new NDMP user on the source VNX. Log in to the control station and run the following command:

/nas/sbin/server_user -add <new_username> -ndmp_md5 -passwd

Select the defaults when prompted and be sure to make note of the password.

2. Determine the absolute path of your filesystems and shares

If you’re using virtual data movers it changes the root path of your filesystem.  Issue the following command to review your file systems and mount paths:

server_mount ALL

Note the the specific path for the file system that is targed for migration. The path when using a VDM will be similar to this:

FILESYSTEM1 on /root_vdm_1/FILESYSTEM1 uxfs,perm,rw

In this case the path will be /root_vdm_1/FILESYSTEM1, which will be used for the source path in the isi_vol_copy_vnx command.

    3. Determine the target Isilon Data Location

Determine the destination location on the Isilon in the /ifs/data folder where the data will migrated.  If the destination folder doesn’t exist on the Isilon, it will create it with the exact same NTFS permissions as the source.  Create the command with the following syntax:

isi_vol_copy_vnx <datamoverIP>:<source_path> <target Isilon path> -sa : <-full or -incr>

isi_vol_copy_vnx 10.10.10.10:/root_vdm_1/FILESYSTEM1 /ifs/data/FILESYSTEM1 -sa ndmpuser1: -full

4. Migrate the Data

The command outlined above will run a full copy using the ndmpuser1 account and will prompt for a password, it does not have to be shown in plain text. The password can be specified in the command by using the appropriate syntax (<ndmpuser:password>), however you will still be required to follow the username with a colon.

If successful, the message “msg SnapSure file system creation succeeds” will appear, which means the NDMP session created a checkpoint successfully and is starting to copy data from that checkpoint.

Note that this does not migrate shares, just data.  Sharedupe can be utilized for that or the CIFS shares/NFS exports can be manually re-created.  It is recommended that any other data migrations on the source VNX be disabled prior to the copy so that you don’t run into performance issues.

Caveats:

• There is no bandwidth throttling option with this command, it will consume all available bandwidth.
• Isilon does not  support Historical SIDs in versions 8.0.0 or earlier, which may result in permission issues due to being unable to resolve historical SIDs post migration from other platforms (see KB468562).  If SIDHistory is in use on the source, then this is not the proper tool.   From the comments section, please note that OneFS does support Security Identifier (SID) history beginning in 8.0.1 and later releases, and 8.0.0.3 and later releases (see latest docu44518_Isilon-Supportability-and-Compatibility-Guide).
• If fs_dedupe is enabled on the Celerra or VNX, you will need to change the backup threshold to zero for each filesystem.  This means that when sending the data over NDMP send the full file, not the compressed or de-duped version.  Note that there is a risk here of inflating existing backup sets if they are being done over NDMP.
• On the source the account performing the copy needs local administrator or backup operators permissions to the source CIFS Server, and full control over the source share.
• Standard NDMP backups and isi_vol_copy_vnx can affect each other and the data backed up by the two NDMP clients.  See KB187008 for a workaround.

Troubleshooting Tips:

1. Checkpoint Creation on the VNX

The most common issue when running the isi_vol_copy_vnx command is with checkpoint creation on the source VNX.

If you are receiving a message that’s similar to msg SnapSure file system creation failed during a copy session, the command is failing to create a snapshot of the source file system. It could be due to many reasons including a lack of available disk space. Try manually creating a snapshot on the source VNX file system to see if it fails, below is the syntax to do so:

#fs_ckpt Test_FS -name Test_FS-ckpt01 –Create

2. Permission or Connection Issues.

In general the error message itself will be self explanatory. Make sure you are using the correct credentials for the NDMP user in the migration command. The user should have sufficient rights on the source system and target systems. The logged in user should be able to create and modify directories and the files contained within.  As an example, in the case below, NDMP Port 53169 is blocked between the VNX and the Isilon.  Opening the port on the firewall resolved the issue.

ISILON568-1# isi_vol_copy_vnx 10.10.10.10:/Volcopytest /ifs/data/Volcopytest -sa Volcopytest:Volcopytest -sport 53169 -full -dhost 10.10.10.11
system call (connect): Connection refused
Could not open NDMP connection to host 10.10.10.10
isi_vol_copy_vnx did not run properly

3.  32bit Unix Application Issues.

If your app is 32bit, 32bit settings on the new NFS export must be enabled.

ISILON# isi nfs exports modify EXPID –zone=NFSZone –return-32bit-file-ids=yes

Replace the EXPID with the ID of the target export, verify the bit settings by viewing the export.

ISILON# isi nfs exports view EXPID –zone=NFSZone | grep -i return

ISILON# _

4. Snapshot creation on the target Isilon array.

Snapshots can fail for many reasons, but most often it’s due to lack of available space. In the example below the snapshot creation failed because there was an existing snapshot with the same name.

ISILON568# isi_vol_copy_vnx VNX-SERVER3:/Test_FS/NFS01 /ifs/data/Test_NFS01 -sa ndmp:NDMPpassword -incr
Snapshot with conflicting name ‘isi_vol_copy.011.1.snap’ found. Remove/Rename the conflicting snapshot to continue with further migration runs.
snapshot already exists
ISILON568-1#

Either delete or rename the existing snap to resolve the issue.  In the example below the snapshot was deleted.

ISILON568-1# isi snapshot snapshots list| grep isi_vol_copy.011
134 isi_vol_copy.011.0.snap /ifs/.ifsvar/modules/isi_vol_copy/011/persistent_store
136 isi_vol_copy.011.1.snap /ifs/.ifsvar/modules/isi_vol_copy/011/persistent_store
ISILON568-1#

ISILON568-1# isi snapshot snapshots delete –snapshot=134
Are you sure? (yes/[no]): y
ISILON568-1#

When opening a shared Microsoft Office file, you may see the error message “File in Use, file_name is locked for editing by user_name“, when in fact no other user is currently using the file.

We had users that would view files with the preview pane, it would create a lock on the file, and then when the explorer window was closed the lock would remain.  The next time the file was accessed it would state that it was locked even though the user didn’t have it open.  Below are some steps you can take to troubleshoot and resolve the issue.  Note that changing some of these parameters can have a performance impact, so make these changes at your own risk.  Oplocks let clients lock files and locally cache information while preventing another user from changing the file. This increases performance for many file operations.

1. Disable Oplocks on the VNX

Disabling oplocks can affect client performance. It will increase the number of metadata requests that are sent to the server because when you use SMB with oplocks, the client caches the data that is locked to speed up access to frequently accessed files. When oplocks are disabled, the client does not cache data and all reads are made directly to the NAS server.

Syntax for disabling oplocks and verifying the change:

[nasadmin@VNX1 ~]$ server_mount vdm_file_system -o nooplock test_file_system01_fs /test_file_system01
vdm_file_system : done

[nasadmin@VNX1 ~]$ server_mount vdm_file_system | grep test_file_system01_fs
test_file_system01_fs on /test_file_system01 uxfs,perm,rw,noprefetch,nonotify,accesspolicy=NATIVE,nooplock

2. Disable caching on the Windows client

The Windows client setting controls the cache lifetime. As stated earlier, if caching is disabled on the windows client then all reads are to the NAS server directly. In order to to disable caching on the windows client rather than disabling oplocks on the VNX Data Mover, the following three registry changes would need to be made:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\

– Directory cache,  set DirectoryCacheLifetime to Zero.
– File Not Found cache, set FileNotFoundCacheLifetime to Zero.
– File information cache, set FileInfoCacheLifetime to Zero.

3. Apply a Microsoft hotfix

The Microsoft KB article http://support.microsoft.com/kb/942146 describes the problem and the fix in detail.  It directly addresses the issue with files locking from the preview pane.  It applies to all versions of Windows Vista and 7 as well as Windows Server 2008.

I recently took a new job in the Enterprise storage group at a large company, and as part of my new responsibilities I will be implementing, configuring, and managing our object storage.  We will be using EMC’s ECS solution, however I’ve been researching object storage as a platform in general to learn more about it, it’s capabilities, it’s use cases, and it’s general management. This blog post is a summary of the information I’ve assimilated about object storage along with some additional information about the vendors who offer object storage solutions.

At its core, what is object based storage?  Object based storage is an architecture that manages data as objects as opposed to other architectures such as file systems that manage data as a file hierarchy, and block storage that manages data as blocks within sectors & tracks.  To put it another way, Object Storage is not directly accessed by the operating system and is not seen as a local or remote filesystem by an operating system. Interaction with data occurs only at the application level via an API.  While Block Storage and File Storage are designed to be consumed by your operating system, Object Storage is designed to be consumed by your application.  What are the implications of this?

• Byte level interaction with data is no longer possible. Data objects are stored or retrieved in their entirety with a single command. This is results in powerful scalability by making all file I/O sequential, which is much higher performance than random I/O.
• It allows for easier application development by providing a higher level of abstraction than traditional storage platforms.
• Interaction can happen through a single API endpoint. This removes complex storage network topologies from the design of the application infrastructure, and dramatically reduces security vulnerabilities as the only available access is the HTTP/HTTPS API and the service providing the API functionality.
• Filesystem level utilities cannot interact directly with Object Storage.
• Object Storage is one giant volume, resulting in almost all storage management overhead of Block and File Storage being eliminated.

Object storage is designed to be more scalable than traditional block and file storage and it specifically targets unstructured content.  In order to achieve improved scalability over traditional file storage it bundles the data with additional metadata tags and a unique identifier. The metadata is completely customizable, allowing an administrator to input much more identifying information for each data object. The objects are also stored in a flat address space which makes it much easier to locate and retrieve data across geographic regions.

Object storage began as a niche technology for enterprises, however it quickly became one of the basic general underlying technologies of cloud storage.  Object storage has become a valid alternative to file based storage due to rapid data growth and the proliferation of big data in the enterprise, an increased demand for private and hybrid cloud storage and a growing need for customizable and scalable storage infrastructures.  The number of object storage products has been rapidly expanding from both major storage vendors and startup companies in recent years to accommodate the increasing demand.  Because many vendors offer Object Storage platforms that run on commodity hardware, even with data protection overhead the price point is typically very attractive when compared to traditional storage.

WHAT’S THE DIFFERENCE BETWEEN OBJECT AND FILE STORAGE?

Object storage technology has been finding its way into file based use cases at many companies.  In some cases object storage vendors are positioning their products as viable NAS alternatives. To address the inherent limitations of traditional file and block level storage to reliably support a huge amount of data and at the same time be cost-effective, object storage focuses on scalability, resiliency, security and manageability. So, what’s the difference between the two?  In general, the difference is in it’s performance, geographic distribution, scalability, and analytics.

OBJECT STORAGE IS HIGHLY SCALABLE

Scalability is a major issue in storage, and it’s only increasing as time goes on. If you need to scale into the petabytes and beyond, you many need to scale in an order of magnitude greater than what a traditional single storage system is capable of.  As traditional storage systems aren’t going to scale to that magnitude, a different type of storage needs to be considered that can still be cost effective, and object storage fills that need very well.

Object storage overcomes many of the scalability limitations that file storage faces.  I really liked the warehouse example that Cloudian used on their website, so I’m going to summarize that.  If you think of file storage as a large warehouse, when you first store a box of files your warehouse looks almost empty and your available space looks infinite. As your storage needs expand that warehouse fills up before you know it, and being in the big city there’s no room to build another warehouse next to it.  In this case, think of object storage as a warehouse without a roof.  New boxes of files can be added almost indefinitely.

While that warehouse with the infinite amount of space sounds good in theory, you may have some trouble finding a specific box in that warehouse as it expands into infinity. Object storage addresses that limitation by allowing customizable metadata.  While a file storage system may only allow metadata to save the date, owner, location, and size of the box, the object storage system can customize the metadata, and object metadata lives directly in the object, rather than in a separate inode (this is useful as the amount of metadata that is desirable in a storage platform that is tens or hundreds of Petabytes is generally an order of magnitude greater than what conventional storage is designed to handle at scale).  Getting back to the warehouse example, along with the date, owner, location, and size, the object metadata could include the exact coordinates of the box, a detailed log of changes and access, and a list of the contents of the box.  Object storage systems replace the limited and rigid file system attributes of file level storage with highly customizable metadata that captures common object characteristics and can also hold application-specific information. Because object storage uses a flat namespace performance may suffer as your data warehouse explodes in size, but you’re not going to have to worry about finding what you need when you need it.

In addition, object storage systems substitute the locking requirements of file level storage to prevent multiple concurrent updates, which enables rollback features and the undeleting of objects as well as the ability to access prior versions of objects.

Object vs. File.  Here’s a brief overview of the main differences.

• Performance.  Object storage performs best for big data and high storage throughput, file storage performs better for smaller files.  Scality’s Organic Ring offers high performance configurations for applications such as email and databases, however traditional storage in general still offers better performance for those use cases.
• Geography.  Object storage data can be storage and shared across multiple geographic regions, file storage data is typically shared locally.  File data spread throughout geographic regions is typically read-only replicated copies of data.
• Scalability.  Object storage offers almost infinite scalability, file storage does not scale nearly as well when you get into millions of files in a volume and petabytes and beyond of total capacity.
• Analytics.  Object storage offers customizable metadata in a flat namespace and is not limited in the number of metadata tags, file storage is limited in that respect.

OBJECT STORAGE AND RESILIENCY

So, we now understand that object storage offers much greater scalability than traditional file storage, what about resiliency?  Traditional file storage systems are hampered by their inherent limitations in supporting massive capacity, most importantly with a sufficient amount of data protection.  As any backup administrator would know, it’s unrealistic to try and back up hundreds of petabytes (or more) of data in any reasonable amount of time.  Object systems directly address that issue as they are designed to not need backups. They aren’t intended to be backed up.  Rather than using traditional backups, an object storage infrastructure is designed to store data with sufficient redundancy so that data is never lost even when multiple components of the infrastructure have failed.

How is this achieved? The primary way this is achieved is by keeping multiple replicas of objects, ideally across a wide geographic area.  Because of the additional storage that replication requires, object storage systems implement an efficient erasure coding data protection method to supplement data replication. What is erasure coding? It uses an algorithm to create additional information that allows for recreating data from a subset of the original data, similar to RAID protection’s parity bits.  The degree of resiliency is generally configurable on all object storage systems. The higher the level of resiliency that the administrator chooses of course results in a larger storage capacity requirement. Erasure coding saves capacity but impacts performance, especially if erasure coding is performed across geographically dispersed nodes. Different vendors handle the performance balance between erasure coding and replication differently.  Geographic erasure coding is generally supported, however only using it locally and replicating data geographically with data reduction seems to strike a good balance between performance and resiliency.

OBJECT STORAGE EASES MANAGEMENT

Object storage systems are designed to minimize storage administration through automation, policy engines and self-correcting capabilities. They are designed for zero downtime and across all vendors administration tasks can be performed without service disruption.  This includes adding capacity, hardware maintenance and upgrades, and even migrating to a different data center.  The object storage policy engines enable the automation of object storage features such as when to change the number of replicas to address spikes in usage, when to use replication vs. erasure coding, and which data centers to store objects based on the relevant metadata.

OBJECT STORAGE AND APPLICATION ACCESSIBILITY

As you might expect, each object storage vendor has implemented its own proprietary APIs Object storage utilizing the REST API to use the various storage functions.  All object storage products also support the industry standard Amazon S3 API, which enjoys perhaps the largest level of application support.  It’s not surprising as Amazon S3 has extensive capabilities and supports complex storage operations.  Be aware that some object storage vendors only support an S3 API subset, and understanding the S3 API implementation’s limitations is absolutely key to ensuring the widest level of application support in your environment.

In addition to S3 most object storage vendors also support the OpenStack Swift API.  File system protocol support is common in object storage systems, but implementations of course vary by product.  As I mentioned earlier, the company I work for went with ECS.  EMC ECS has geo distributed active/active NFS support, a key feature, and it offers Hadoop HDFS interfaces which allows Hadoop to directly access data in an object stores.  With the ECS system’s consistency support it’s a very strong geo-distributed NAS product.  Different vendors of course have different strengths and weaknesses.  A strong competitor of EMC, Scality, claims that it has EMC Isilon-level NAS performance (although I haven’t tested that), and the NetApp StorageGRID Webscale now offers protocol duality by having a one-to-one relationship between objects and files.  Other object storage products provide different unique features as well.  Some offer file system support through their own or third-party cloud storage gateways, and some provide S3 compliant connectors that allow Hadoop to use object storage as an alternative to HDFS.

OBJECT STORAGE AND DATA ENCRYPTION

Public cloud storage is a very common use case for object storage, and encryption is obviously a must for public cloud storage.  Most object storage products support both at-rest and in-transit encryption, and most use an at-rest encryption approach where encryption keys are generated dynamically without a need for a separate key management system.  Some vendors (such as Cloudian and Amplidata) support client-managed encryption keys in addition to server-side managed encryption keys which gives cloud providers an option to allow their customers to manage their own keys.  LDAP and Active Directory authentication support of users accessing the object store are also commonly supported in current object storage systems.  If support of AWS v2 or v4 authentication is needed to provide access to vaults and vault objects, do your research as support is less common.

THE BEST USE CASES FOR OBJECT STORAGE

The ability of object storage to scale and accessibility via APIs makes them suitable in use cases where traditional storage systems just can’t compete, even in the NAS arena.  So, now that you know what object storage is, what can it be used for, and how can you take advantage of the improved scalability and accessibility?  While object storage is typically not well suited for relational database or any data that requires a large amount of random I/O, it has many possible use cases that I’ve outlined below.

Advantages

• Highly Scalable capacity and performance
• Low cost on commodity hardware at petabyte scale
• Simplified management
• Single Access Point/namespace for data

Disadvantages

• No random access to files
• Lower performance on a per-object basis compared to traditional storage
• Integration may require modification of application and workflow logic
• POSIX utilities will not work directly with object storage

Use Cases

• Logging.  It is often used to capture large amounts of log data generated by devices and applications which are ingested into the object store via a message broker.
• NAS.  Many companies are considering object storage as a NAS alternative, most notably if there is another use case that requires an object storage system and the two use cases can be combined.
• Big Data.  Several object storage products offer certified S3 Hadoop Distributed File System interfaces that allow Hadoop to directly access data on the object store.
• Content distribution network.  Many companies use an object storage implementation to globally distribute content (like media files) using policies to govern access, along with features like automatic object deletion based on expiration dates.
• Backup and Archive of structured and semi-structured data.  Because object storage systems are cost-effective, many companies are looking to them as highly scalable backup and archive solution.
• Content Repositories.  Object storage is often used as a content repository for images, videos and general media content accessed directly through applications or through file system protocols.
• Enterprise Collaboration.  Because of the scale and resiliency of object storage across large geographic regions, distributed object storage systems are often used as collaboration platforms in large enterprises where content is accessed and shared around the globe.
• Storage as a Service (SaaS).  Object storage is often used for private and public clouds of enterprises and internet service providers.

OBJECT STORAGE VENDORS AND PRODUCTS

There are numerous object storage vendors in the market today.  You can purchase a detailed vendor comparison of all the object storage vendors at the Evaluator Group (Evaluator Group Object Storage Comparison Matrix) , and view the Gartner Object storage comparison matrix for more detailed information.  According to Gartner, DellEMC, Scality, and IBM are the current market leaders with Hitachi as the strongest challenger.

Gartner Market Leaders for Object Storage:

EMC Elastic Cloud Storage (ECS)

EMC Delivers ECS as a  turnkey integrated appliance or as a software package to be installed and run on commodity hardware.  It features highly efficient strong consistency on access of geodistributed objects, and is designed from the ground up with geodistribution in mind.

Scality RING

It is provided as delivered software only to run on commodity hardware, it stores metadata in a custom-developed distributed database, and they claim EMC Isilon performance when it’s used as NAS.

IBM Clould Object Storage

It is provided as delivered software only to run on certified hardware, it is a multi-tiered architecture with no centralized servers, and it offers extreme scalability enabled by peer-to-peer communication of storage nodes.

Other Object Storage Vendors:

Hitachi Content Platform (HCP)

It is delivered as a turnkey integrated appliance or as a software package to run on commodity hardware or as a managed service hosted by HDS.  It offers extreme density with a single cluster able to support up to 800 million objects and 497 PB of addressable capacity, and an integrated portfolio: HCP cloud storage, HCP Anywhere File Sync & Share, and Hitachi Data Ingestor (HDI) for remote and branch offices.

NetApp StorageGRID Webscale

It is delivered as software appliance or as a turnkey integrated appliance, it stores metadata (including the physical location of objects) in a distributed NoSQL Cassandra database.

DDN WOS Object Storage

It is delivered as turnkey integrated appliance or as a software package to run on commodity hardware, It can be configured as small as one node to start and it is able to scale to hundreds of petabytes.

Caringo Swarm 8

It is delivered as a software package to run on commodity hardware. It offers out-of-box integration with Elastic search for fast object searching.

Red Hat Ceph

It is delivered as a software package to run on commodity hardware and is based on the open-source Reliable Autonomic Distributed Object Store (RADOS).  It features strong consistency on write performance.

Cloudian HyperStore

It is delivered as turnkey integrated appliance or as a software package to run on commodity hardware.  It stores metadata with objects but also in a distributed NoSQL Cassandra database for speed.

HGST Amplidata

It is delivered as a turnkey rack-level system and uses HGST Helium filled hard drives for power efficiency, reliability and capacity.

SwiftStack Object Storage System

It is delivered as a software package to run on commodity hardware and is based on OpenStack Swift, which is the enterprise offering of Swift with cluster and management tools and 24×7 support.