We recently had a problem with one of our corporate applications having file close requests fail, resulting in 200,000+ open files on our production data mover. This was causing numerous issues within the application. We determined that the problem was a result of our Riverbed Steelhead appliance requiring a certain level of DART code in order to properly close the files. The Steelhead applicance would fail when attempting to optimize SMBV2 connections.
Because a DART code upgrade is required to resolve the problem, the only temporary fix is to reboot the data mover. I wrote a quick script on the Celerra to grab the number of open files, write it to a text file, and publish to our internal web server. The command to check how many open files are on the data mover is below.
This command provides all the detailed information:
/nas/bin/server_stats server_2 -monitor cifs-std -interval 1 -count 1
server_2 CIFS CIFS CIFS CIFS Avg CIFS CIFS CIFS Avg CIFS CIFS Timestamp Total Read Read Read Write Write Write Share Open Ops/s Ops/s KiB/s Size KiB Ops/s KiB/s Size KiB Connections Files 11:15:36 3379 905 9584 11 9 272 30 1856 4915 server_2 CIFS CIFS CIFS CIFS Avg CIFS CIFS CIFS Avg CIFS CIFS Summary Total Read Read Read Write Write Write Share Open Ops/s Ops/s KiB/s Size KiB Ops/s KiB/s Size KiB Connections Files Minimum 3379 905 9584 11 9 272 30 1856 4915 Average 3379 905 9584 11 9 272 30 1856 4915 Maximum 3379 905 9584 11 9 272 30 1856 4915
Adding a grep for Maximum and using awk to grab only the last column, this command will output only the number of open files, rather than the large output above:
/nas/bin/server_stats server_2 -monitor cifs-std -interval 1 -count 1 | grep Maximum | awk ‘{print $10}’
The output of that command would simply be ‘4915’ based on the sample full output I used above.
The solution number from Riverbed’s knowledgebase is S16257. Your DART code needs to be at least 6.0.60.2 or 7.0.52. You will also see in your steelhead logs a message similar to the one below indicating that the close request has failed for a particular file:
Sep 1 18:19:52 steelhead port[9444]: [smb2cfe.WARN] 58556726 {10.0.0.72:59207 10.0.0.72:445} Close failed for fid: 888819cd-z496-7fa2-2735-0000ffffffff with ntstatus: NT_STATUS_INVALID_PARAMETER