I was looking for an easy (and free) way to do some daily reporting on our data domain hardware. I was most interested in reporting on disk space, but decided to gather some other data as well. The easiest method I found is to use the info collected in the autosupport log. First, I’ll explain how to automatically gather the autosupport log from your data domain, and then move on to how to pull only the information you want from it for reporting purposes.
First, you need to enable ftp on the data domain and add the IP address of the FTP server you’re going to use to pull the file. Here are the commands you need to run on your data domain:
adminaccess enable ftp adminaccess add ftp 10.1.1.1Next, you’ll want to pull the files from the data domain. I am using a windows FTP server to pull the autosupport files. The windows batch script I use is scheduled to run once a day and is listed below. I use the ftp command and call a text file with the specific IP and login info for each data domain box.
ftp -s:10.1.1.2.txt ftp -s:10.1.1.3.txt ftp -s:10.2.1.2.txt ftp -s:10.2.1.3.txtThe text files that the ftp script calls (that are named <ipaddress.txt>) look like this:
open 10.1.1.2 sysadmin <password> cd support lcd e:\reports\dd\SITE1_DD_01 get autosupport quitOnce you’ve downloaded the autosupport file, you can run scripts against it to pull out only the info you’re interested in. I prefer to use unix shell scripts for parsing files because of the extra functionality over standard windows batch scripts. I have Cygwin installed on my web server to run these shell scripts.
If you take a look at the autosupport log, you’ll notice that it’s very long and contains some duplicate characters in multiple areas, which makes using grep, awk, and sed a bit more challenging. It took a bit of trial and error, but the scripts below will pull only the specific areas I wanted: System Alerts, File system compression, Locked files, Replication info, File distribution, and Disk space information.
The first script below will gather disk space information using grep. Spacing inside the quotes is very important in this case, as I was looking for specific unique strings within the autosupport log, and using grep for these unique strings will produce the correct output. In this example, I am gathering info for four data domain boxes. For those unfamiliar with Cygwin, you can access windows drive letters by navigating to /cygdrive/<drive letter>. To view the root of the C drive, you would type ‘cd /cygdrive/c’ from the CLI. In this case, my windows batch script that pulls the autosupport files places them in the e:\reports\dd folder, which would be /cygdrive/e/reports/dd folder when using the cygwill shell.
Here is the script:
# Site 1 Data Domain 01 cat /cygdrive/e/reports/dd/SITE1_DD_01/autosupport | grep “= SERVE” > /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_01/autosupport | grep “Active Tier:” >> /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_01/autosupport | grep “Resource Size” >> /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_01/autosupport | grep “/data:” >> /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_01/autosupport | grep “/ddvar ” >> /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt # Site 1 Data Domain 02 cat /cygdrive/e/reports/dd/SITE1_DD_02/autosupport | grep “= SERVE” > /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_02/autosupport | grep “Active Tier:” >> /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_02/autosupport | grep “Resource Size” >> /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_02/autosupport | grep “/data:” >> /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/SITE1_DD_02/autosupport | grep “/ddvar ” >> /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt # Site 2 Data Domain 01 cat /cygdrive/e/reports/dd/Site2_DD_01/autosupport | grep “= SERVE” > /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_01/autosupport | grep “Active Tier:” >> /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_01/autosupport | grep “Resource Size” >> /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_01/autosupport | grep “/data:” >> /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_01/autosupport | grep “/ddvar ” >> /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt # Site 2 Data Domain 02 cat /cygdrive/e/reports/dd/Site2_DD_02/autosupport | grep “= SERVE” > /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_02/autosupport | grep “Active Tier:” >> /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_02/autosupport | grep “Resource Size” >> /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_02/autosupport | grep “/data:” >> /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt cat /cygdrive/e/reports/dd/Site2_DD_02/autosupport | grep “/ddvar ” >> /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt # Copy the reports to the web server directory. cp /cygdrive/e/reports/dd/SITE1_DD_01/SITE1_DD_01_diskspace.txt /cygdrive/c/inetpub/wwwroot/ cp /cygdrive/e/reports/dd/SITE1_DD_02/SITE1_DD_02_diskspace.txt /cygdrive/c/inetpub/wwwroot/ cp /cygdrive/e/reports/dd/Site2_DD_01/Site2_DD_01_diskspace.txt /cygdrive/c/inetpub/wwwroot/ cp /cygdrive/e/reports/dd/Site2_DD_02/Site2_DD_02_diskspace.txt /cygdrive/c/inetpub/wwwroot/For the remaining reports, I used the ‘sed’ command rather than ‘grep’. As an example, I’ll explain how I stripped out the information for the System Alerts section. In order to strip out it out, I use sed to start cutting text when it reaches ‘Current Alerts’, and stop cutting text when it reaches ‘There are’. This same process is repeated to pull the information for the other reports as well (alerts, compression, locked files, replication, distribution).
This is what the Current Alerts section looks like in the autosupport log:
Current Alerts -------------- Alert Id Time Severity Class Object Message -------- ------------------------ -------- ----------------- ------------- --------------------------------------------------------------------- 774 Wed Dec 19 11:32:19 2012 CRITICAL SystemMaintenance Core dump capability is now disabled due to lack of space in /ddvar. 807 Sun Jan 20 09:07:18 2013 WARNING Replication context=3 repl ctx 3: Sync-as-of time is more than 461 hours ago. 808 Sun Jan 20 09:07:18 2013 WARNING Replication context=4 repl ctx 4: Sync-as-of time is more than 461 hours ago. 809 Sun Jan 20 09:07:19 2013 WARNING Replication context=5 repl ctx 5: Sync-as-of time is more than 461 hours ago. 814 Mon Jan 28 23:20:45 2013 CRITICAL Filesystem FilesysType=2 Space usage in Data Collection has exceeded 95% threshold. 820 Wed Jan 30 15:52:40 2013 WARNING Replication context=2 repl ctx 2: Sync-as-of time is more than 148 hours ago. 824 Mon Feb 4 12:36:39 2013 WARNING Replication context=10 repl ctx 10: Sync-as-of time is more than 24 hours ago. 825 Wed Feb 6 02:13:51 2013 WARNING Replication context=19 repl ctx 19: Sync-as-of time is more than 24 hours ago. -------- ------------------------ -------- ----------------- ------------- --------------------------------------------------------------------- There are 8 active alerts.
In this case, sed searches for ‘Current Alerts’ and copies all of the text until it reaches the string ‘There are’, at which point it stops and outputs the file. The output files are written directly to the wwwroot folder on my internal reporting web page. Each page encapsulates the output files in an iframe, so the web page is automatically updated every morning when the script runs.
Here is the second script that captures the remaining data:
#For Alerts sed -n ‘/Current Alerts/,/There are/p’ /cygdrive/e/reports/dd/SITE1_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_01_alerts.txt sed -n ‘/Current Alerts/,/There are/p’ /cygdrive/e/reports/dd/SITE1_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_02_alerts.txt sed -n ‘/Current Alerts/,/There are/p’ /cygdrive/e/reports/dd/Site2_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_01_alerts.txt sed -n ‘/Current Alerts/,/There are/p’ /cygdrive/e/reports/dd/Site2_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_02_alerts.txt #For Filesystem Compression sed -n ‘/Filesys Compression/,/((Pre-/p’ /cygdrive/e/reports/dd/SITE1_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_01_filecomp.txt sed -n ‘/Filesys Compression/,/((Pre-/p’ /cygdrive/e/reports/dd/SITE1_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_02_filecomp.txt sed -n ‘/Filesys Compression/,/((Pre-/p’ /cygdrive/e/reports/dd/Site2_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_01_filecomp.txt sed -n ‘/Filesys Compression/,/((Pre-/p’ /cygdrive/e/reports/dd/Site2_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_02_filecomp.txt #For Locked Files: sed -n ‘/Locked files/,/Active/p’ /cygdrive/e/reports/dd/SITE1_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_01_lockedfiles.txt sed -n ‘/Locked files/,/Active/p’ /cygdrive/e/reports/dd/SITE1_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_02_lockedfiles.txt sed -n ‘/Locked files/,/Active/p’ /cygdrive/e/reports/dd/Site2_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_01_lockedfiles.txt sed -n ‘/Locked files/,/Active/p’ /cygdrive/e/reports/dd/Site2_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_02_lockedfiles.txt #Repl Transferred over 24 hrs sed -n ‘/Replication Data Transferred/,/(sum)/p’ /cygdrive/e/reports/dd/SITE1_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_01_repl.txt sed -n ‘/Replication Data Transferred/,/(sum)/p’ /cygdrive/e/reports/dd/SITE1_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_02_repl.txt sed -n ‘/Replication Data Transferred/,/(sum)/p’ /cygdrive/e/reports/dd/Site2_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_01_repl.txt sed -n ‘/Replication Data Transferred/,/(sum)/p’ /cygdrive/e/reports/dd/Site2_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_02_repl.txt #File Distribution sed -n ‘/File Distribution/,/> 500 GiB/p’ /cygdrive/e/reports/dd/SITE1_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_01_filedist.txt sed -n ‘/File Distribution/,/> 500 GiB/p’ /cygdrive/e/reports/dd/SITE1_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/SITE1_DD_02_filedist.txt sed -n ‘/File Distribution/,/> 500 GiB/p’ /cygdrive/e/reports/dd/Site2_DD_01/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_01_filedist.txt sed -n ‘/File Distribution/,/> 500 GiB/p’ /cygdrive/e/reports/dd/Site2_DD_02/autosupport > /cygdrive/c/inetpub/wwwroot/Site2_DD_02_filedist.txtWhen all is said and done, the report output looks like this:
Alerts: Current Alerts -------------- Alert Id Time Severity Class Object Message -------- ------------------------ -------- ----------- ------------- ---------------------------------------------------------- 1157 Wed Feb 5 12:23:19 2014 WARNING Replication context=3 Sync-as-of time is more than 24 hours ago. 1161 Sun Feb 9 08:33:48 2014 CRITICAL Filesystem FilesysType=2 Space usage in Data Collection has exceeded 95% threshold. -------- ------------------------ -------- ----------- ------------- ---------------------------------------------------------- There are 2 active alerts. Filesystem Compression: Filesys Compression -------------- From: 2014-03-06 05:00 To: 2014-03-13 06:00 Pre-Comp Post-Comp Global-Comp Local-Comp Total-Comp (GiB) (GiB) Factor Factor Factor (Reduction %) --------------- -------- --------- ----------- ---------- ------------- Currently Used: 495122.5 122948.5 - - 4.0x (75.2) Written:* Last 7 days 9204.2 5656.6 1.1x 1.5x 1.6x (38.5) Last 24 hrs 74.1 36.4 1.0x 2.0x 2.0x (50.9) --------------- -------- --------- ----------- ---------- ------------- * Does not include the effects of pre-comp file deletes/truncates since the last cleaning on 2014/03/11 02:07:50. Replications: Replication Data Transferred over 24hr -------------------------------------- Directory/MTree Replication: Date Time CTX Pre-Comp (KB) Pre-Comp (KB) Replicated (KB) Low-bw- Sync-as-of Written Remaining Pre-Comp Network optim Time ---------- -------- ----- ------------- ------------- ----------------------- ------- ---------------- 2014/03/13 06:36:18 2 104,186,675 86,628,349 55,522,200 37,756,047 1.00 Thu Mar 13 05:53 3 0 0 0 0 0.00 Tue Feb 4 12:00 4 0 0 0 4,882 0.00 Thu Mar 13 06:00 8 0 0 0 5,120 0.00 Thu Mar 13 06:00 29 0 0 0 5,098 0.00 Thu Mar 13 06:00 (sum) 104,186,675 55,522,200 37,771,148 1.00 File Distribution: File Distribution ----------------- 169,432 files in 8,691 directories Count Space ----------------------------- -------------------------- Age Files % cumul% GiB % cumul% --------- ----------- ----- ------- -------- ----- ------- 1 day 13 0.0 0.0 74.4 0.0 0.0 1 week 223 0.1 0.1 2133.7 0.4 0.4 2 weeks 5,103 3.0 3.2 39121.2 7.8 8.3 1 month 12,819 7.6 10.7 79336.5 15.9 24.1 2 months 21,853 12.9 23.6 153873.8 30.8 54.9 3 months 5,743 3.4 27.0 45154.6 9.0 64.0 6 months 3,402 2.0 29.0 13674.3 2.7 66.7 1 year 17,035 10.1 39.1 72937.7 14.6 81.3 > 1 year 103,241 60.9 100.0 93461.7 18.7 100.0 --------- ----------- ----- ------- -------- ----- ------- Count Space ----------------------------- -------------------------- Size Files % cumul% GiB % cumul% --------- ----------- ----- ------- -------- ----- ------- 1 KiB 11,257 6.6 6.6 0.0 0.0 0.0 10 KiB 44,396 26.2 32.8 0.3 0.0 0.0 100 KiB 38,488 22.7 55.6 1.3 0.0 0.0 500 KiB 9,652 5.7 61.3 2.1 0.0 0.0 1 MiB 27,460 16.2 77.5 70.4 0.0 0.0 5 MiB 12,136 7.2 84.6 27.3 0.0 0.0 10 MiB 3,861 2.3 86.9 26.9 0.0 0.0 50 MiB 4,367 2.6 89.5 96.6 0.0 0.0 100 MiB 853 0.5 90.0 58.2 0.0 0.1 500 MiB 861 0.5 90.5 201.0 0.0 0.1 1 GiB 495 0.3 90.8 309.7 0.1 0.2 5 GiB 567 0.3 91.1 1460.2 0.3 0.5 10 GiB 336 0.2 91.3 2574.1 0.5 1.0 50 GiB 14,691 8.7 100.0 494083.5 98.9 99.8 100 GiB 11 0.0 100.0 683.3 0.1 100.0 500 GiB 1 0.0 100.0 173.0 0.0 100.0 > 500 GiB 0 0.0 100.0 0.0 0.0 100.0 Disk Space: ========== SERVER USAGE ========== Active Tier: Resource Size GiB Used GiB Avail GiB Use% Cleanable GiB /data: pre-comp - 245211.8 - - - /data: post-comp 51484.9 31811.0 19673.9 62% 35.4 /ddvar 78.7 7.9 66.8 11% -