In this post I’ve outlined the procedure for generating SSL requests, keys and certificates for ECS, as well as outlining the process for uploading them to ECS and verifying the installed certificates afterwards. This was a new process for me so I created very detailed documentation on the process I used, hopefully this will help someone else out.
I mention using the ECS CLI a few times in this document. If you’d like to use the ECS CLI, I have another blog post here that reviews the details on it’s implementation. It requires Python.
Part 1: Generating SSL requests, Keys, and Certificates.
The procedure for generating SSL requests, keys, and certificates is unnecessary if you will be given the certificate and key files from a trusted source within your organization. If you’ve been provided the certificate and key file already, you can skip to the Part 2 that details how to upload and import the keys and certificates to ECS. This is of course a sample procedure on how I did it in my organization, specific details may have to be altered depending on the use case.
a. Prepare for Creating (or editing) the SSL Request file
- The first step in this process is to generate an SSL request file. As OpenSSL does not allow you to pass Subject Alternative Names (SANs) through the command line, they must be added to a configuration file first.
- On ECS, the OpenSSL configuration file is located at /etc/ssl/oenssl.cnf by default. Copy that file to a temporary directory where you will be generating your certificates.
- Run this command to copy the request file for editing:
admin@ecs-node1:~# cp /etc/ssl/openssl.cnf /tmp/request.conf
b. Make changes to the request.conf file. Edit it with vi and make the edits outlined below. Each bullet reviews a specific section of the file where changes are required.
- [ alternate_names ] Edit the [ alternate_names ] section. In a typical request file these are included at the very end of the configuration file. Note that this request example includes the wildcard as the first entry (which is required by S3).
Sample:
DNS.1 = *.prod.os.example.com DNS.2 = atmos.example.com DNS.3 = swift.example.com
- [ v3_ca ] Edit the [ v3_ca ] section.
This line should be added directly below the [ v3_ca ] header:
subjectAltName = @alternate_names
Search for “basicConstraints” in the [ v3_ca ] section. You may see “basicConstraints = CA:true”. Make sure it is commented out – add the # to the beginning of the line.
# basicConstraints = CA:true
Search for “keyUsage = cRLSign, keyCertSign” in the [ v3_ca ] section. You may see “# keyUsage = cRLSign, keyCertSign”. Make sure it is commented out.
# keyUsage = cRLSign, keyCertSign
- [ v3_req ] Verify the configuration in the [ v3_req ] section. The line below must exist.
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
- [ usr_cert ] Verify the configuration in the [ usr_cert ] section.
Search for the entry below and uncomment it, it should be added.
extendedKeyUsage = serverAuth
The following line is likely to already exist in this [ v3_ca ] section. The authorityKeyIdentifier line exists in multiple locations in the config file, however in the v3_ca section it must have “always,issuer” as its option.
# authorityKeyIdentifier=keyid:always,issuer
- [ req ] Verify the configuration In the [ req ] section.
For our dev environment, in the testing phase with a self-signed certificate, the following entry was made six lines below the [ req ] header:
x509_extensions = v3_ca # The extensions to add to the self-signed cert
The x509_extensions line also exists in the [ CA_default ] section. This was left untouched in my configuration.
x509_extensions = usr_cert # The extensions to add to the cert
Change based on certificate type. Note that this will change if you’re not using a self-signed certificate, which I did not test. The req_extensions line exists in the default configuration file and is commented out.
x509_extensions = v3_ca # for a self-signed cert req_extensions = v3_ca # for cert signing req
Change the default_bits entry.
Search for default_bits = 1024, it should be default_bits = 2048
- [ CA_default ] In the CA_default section, uncomment or add the line below. The line exists in the default configuration file and simply needs to be uncommented.
copy_extensions = copy
The following additional changes were made in my configuration:
Search for dir = ./demoCA, change to dir = /etc/pki/CA Search for default_md = default, change to default_md = sha256
- [ req_distinguished_name] Verify the configuraiton in the [ req_distinguished_name] section.
The following changes were made in my configuration:
countryName_default = AU, change to countryName_default = XX stateOrProviceName_default = SomeState, change to stateOrProviceName_default = Default Province localityName_default doesn’t exist in the default file, added as localityName_default = Default City 0.organizationName_default = Internet Widgits Pty Ltd, change to 0.organizationName_default = Default Company commonName = Common Name (e.g. server FQDN or YOUR name), it was changed to commonName = Common Name (eg, your name or your server\'s hostname)
- [ tsa_config1 ] Verify the configuration in the [ tsa_config1 ] section.
The following additional change was made in my configuration:
digests = md5, sha1, change to digests = sha1, sha256, sha384, sha512
c. Generate the Private Key. Save the key file in a secure location, the security of your certificate depends on the private key.
- Run this command to generate the private key:
admin@ecs-node1:~# openssl genrsa -des3 -out server.key 2048 Generating RSA private key, 2048 bit long modulus ............................................................+++ Enter pass phrase for server.key: <enter a password> Verifying - Enter pass phrase for server.key: <enter a password>
- Modify the permissions of the server key:
admin@ecs-node1:~# chmod 0400 server.key
- Now that the private key is generated, you can either create a certificate request (the .req file) to request a certificate from a CA or generate a self-signed certificate. In the samples below, I’m setting the Common Name (CN) on the certificate to *.os.example.com.
d. Generate the Certificate Request. Next we will look at the steps used to generate a certificate request.
- Run the command below to generate the request.
admin@ecs-node1:~# openssl req -new -key server.key -config request.conf -out server.csr Enter pass phrase for server.key: <your passprhase from above>
- Running the command above will prompt for additional information that will be incorporated into the final certificate request (the Distinguished Name, or DN). Some fields may be left blank and some will have default values, If you enter ‘.’ the field will be left blank.
Country Name (2 letter code) [US]: <Enter value> State or Province Name (full name) [Province]: <Enter value> Locality Name (eg, city) []: <Enter value> Organization Name (eg, company) [Default Company Ltd]: <Enter value> Organizational Unit Name (eg, section) []: <Enter value> Common Name (e.g. server FQDN or YOUR name) []: <*.os.example.com> Email Address []: <admin email>
- Enter the following extra attributes to be sent with the certificate request:
A challenge password []: <optional> An optional company name []: <optional>
- Check request contents. Use OpenSSL to verify the contents of the request, verify that the SANs are set correctly.
admin@ecs-node1:~# openssl req -in server.csr -text -noout
Certificate Request: Data: Version: 0 (0x0) Subject: C=US, ST=North Dakota, L=Fargo, O=EMC, OU=ASD, CN=*.os.example.com/emailAddress=admin@example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus:vc a7:5a:dc:ca:ff:73:53:6b:ab:a7:ff:7a:20:c1:ff: … <removed a portion of the output for this example> .. ff:9e:66:ff:43:0a:fd:31:3d:69:b1:03:20:51:ff: Exponent: 65537 (0x10001) A Requested Extensions: X509v3 Subject Alternative Name: DNS:os.example.com, DNS:atmos.example.com, DNS:swift.example.com X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication attributes: Signature Algorithm: sha256 WithRSAEncryption ff:7a:f3:7d:8e:8d:37:8f:66:c8:91:16:c0:00:39:df:03:c1: … <removed a portion of the output for this example> .. ff:d9:68:ff:be:e4:4e:e1:78:16:67:47:14:01:31:32:0e:a2:
- Now that the certificate request is completed it may be submitted to the CA who will then return a signed certificate file.
e. Generate a Self-Signed Certificate. Generating a self-signed certificate is almost identical to generating the certificate request. The main difference is that instead of generating a request file, you add an -x509 argument to to the openssl req command to generate a certificate file instead.
admin@ecs-node1:~# openssl req -x509 -new -key server.key -config request.conf -out server.crt Enter pass phrase for server.key: <your passprhase from above>
- Running that command will prompt for additional information that will be incorporated into the certificate request. This is called a Distinguished Name (DN). Some fields may be left blank and some will have default values, If you enter ‘.’ the field will be left blank.
Country Name (2 letter code) [US]: <Enter value> State or Province Name (full name) [Province]: <Enter value> Locality Name (eg, city) []: <Enter value> Organization Name (eg, company) [Default Company Ltd]: <Enter value> Organizational Unit Name (eg, section) []: <Enter value> Common Name (e.g. server FQDN or YOUR name) []: <*.os.example.com> Email Address []: <admin email>
- Enter the following extra attributes to be sent with the certificate request:
A challenge password []: <optional> An optional company name []: <optional>
- Check request contents. Use OpenSSL to verify the contents of the request, verify that the SANs are set correctly.
admin@ecs-node1:~# openssl x509 -in server.crt -noout -text
Certificate: Data: Version: 3 (0x2) Serial Number: 9999999999990453326 (0x11fc66cf7c09d762) Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, ST=North Dakota, L=Fargo, O=EMC, OU=ASD, CN=*.os.example.com/ emailAddress=admin@example.com Validity Not Before: Oct 14 16:47:40 2014 GMT Not After : Nov 13 16:47:40 2014 GMT Subject: C=US, ST=Minnesota, L=Minneapolis, O=EMC, OU=ASD, CN=*.os.example.com/emailAddress=admin@example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: ff:bc:8f:83:7b:57:72:3d:70:ef:ff:d0:f9:97:ff: … <removed a portion of the output for this example> .. ff:9e:66:86:43:0a:fd:ff:3d:69:b1:03:20:51:ff: db:77 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: TL Web Server Authentication X509v3 Subject Alternative Name: DNS:os.example.com, DNS:atmos.example.com, DNS:swift.example.com X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment Signature Algorithm: sha256WithRSAEncryption ff:bc:8f:83:7b:57:72:ff:70:ef:b9:d0:f9:97:ff: … <removed a portion of the output for this example> .. ff:9e:66:ff:43:0a:fd:31:3d:69:ff:03:20:51:39: db:77
f. Chain File. In either a self-signed or a CA signed use case, you now have a certificate file. In the case of a self-signed certificate, the certificate is the chain file. If your certificate was signed by a CA, you’ll need to append the intermediate CA cert(s) to your certificate. I used a self-signed certificate in my implementation and did not perform this step.
- Append the CA cert if it was signed by a CA. Do not append the root CA certificate:
admin@ecs-node1:~# cp server.crt serverCertChain.crt admin@ecs-node1:~# cat intermediateCert.crt >> serverCertChain.crt
Part 2: Upload the keys and Certificates. The next section outlines the process for installing the key and certificate pair on ECS.
a. First log in to the management API to get a session token. You will need the root password for the ECS node.
- Run this command (change IP and password as needed): (ctrl+c to break)
admin@ecs-node1:/> curl -L --location-trusted -k https://10.10.10.10:4443/login -u "root:password" –v
- Note that the prior will leave the root password in the command history. You can run it without the password and have it prompt you instead:
curl -L --location-trusted -k https://10.10.10.10:4443/login -v -u root Enter host password for user 'root': <enter password>
- From the output of the command above, set an environment variable to hold the token for later use.
admin@ecs-node1:/> export ECS_TOKEN=x-sds-auth-token-value
b. Commands used for installing a key & certificate pair for Management requests/users:
- Use ECSCLI to run it from a client PC:
admin@ecs-node1:/> python ecscli.py vdc_keystore update –hostname <ecs host ip> -port 4443 –cf <cookiefile> –privateKey <privateKey> -certificateChain <certificateChainFile>
- Use CURL to run it directly from the ECS management console. Note that this command uses the TOKEN environment variable that was set earlier.
Sample Command:
admin@ecs-node1:/> curl -svk -H "X-SDS-AUTH-TOKEN: $TOKEN" -H "Content-type: application/xml" -H "X-EMC-REST-CLIENT: TRUE" -X PUT -d "<rotate_keycertchain><key_and_certificate><private_key>`cat privateKeyFile`</private_key><certificate_chain>`cat certChainFile`</certificate_chain></key_and_certificate></rotate_keycertchain>" https://localhost:4443/vdc/keystore
c. Commands used for installing a key & certificate pair for Object requests/users. Use the actual private key and certificate chain files here, and a successful response code should be an HTTP 200.
- Use ECSCLI to run it from a client PC:
admin@ecs-node1:/> python ecscli.py keystore update –hostname <ecs host ip> -port 4443 –cf <cookiefile> -pkvf <privateKey> -cvf <certificateChainFile>
- Use CURL to run it directly from the ECS management console. If curl is used, the xml format is required so that carriage returns and the like will be handled via the `cat` command.
Sample Command:
admin@ecs-node1:/> curl -svk -H "X-SDS-AUTH-TOKEN: $TOKEN" -H "Content-type: application/xml" -H "X-EMC-REST-CLIENT: TRUE" -X PUT -d "<rotate_keycertchain><key_and_certificate><private_key>`cat privateFile.key`</private_key><certificate_chain>`cat certChainFile.pem`</certificate_chain></key_and_certificate></rotate_keycertchain>" https://localhost:4443/object-cert/keystore
d. Important Notes:
- Though this is the object certificate to be used for object requests sent on port 9021, the upload command is a management command which is sent on port 4443.
- Once this is done it can take up to 2 hours for the certificate to be distributed to all of the nodes.
- The certificate is immediately distributed upon the service restart of the node where the certificate was uploaded.
e. Restart managment services to propagate the management certificate. Using viprexec will run the command on all of the nodes in the cluster.
admin@ecs-node1:/> sudo -i viprexec -i -c '/etc/init.d/nginx restart;sleep 10;/etc/init.d/nginx status' Output from host : 192.168.1.1 Stopping nginx service ..done Starting nginx service ..done nginx service is running (pid=75447) Output from host : 192.168.1.2 Stopping nginx service ..done Starting nginx service ..done nginx service is running (pid=85534) Output from host : 192.168.1.3 Stopping nginx service ..done Starting nginx service ..done nginx service is running (pid=87325) Output from host : 192.168.1.4 Stopping nginx service ..done Starting nginx service ..done nginx service is running (pid=59112) Output from host : 192.168.1.5 Stopping nginx service ..done Starting nginx service ..done nginx service is running (pid=77312)
f. Verify that the certificate was propogated to each node. The output will show the certificate, scroll up and verify all of the information is correct. At the minimum the first and last node should be checked.
admin@ecs-node1:/> openssl s_client -connect 10.10.10.1:4443 | openssl x509 -noout -text admin@ecs-node1:/> openssl s_client -connect 10.10.10.2:4443 | openssl x509 -noout -text admin@ecs-node1:/> openssl s_client -connect 10.10.10.3:4443 | openssl x509 -noout -text admin@ecs-node1:/> openssl s_client -connect 10.10.10.4:4443 | openssl x509 -noout -text admin@ecs-node1:/> openssl s_client -connect 10.10.10.5:4443 | openssl x509 -noout -text
g. Wait at least 2 minutes and then restart the object head services to propagate the object head certificate:
admin@ecs-node1:/> sudo -i viprexec -i -c 'kill \`pidof dataheadsvc\`'
- Wait for the service to come back up, which you can verify with the next few commands.
- Run netstat to verify the datahead service is listening.
admin@ecs-node1:/tmp> netstat -an | grep LIST | grep 9021 tcp 0 0 10.10.10.1:9021 :::* LISTEN
admin@ecs-node1:/tmp> sudo netstat -anp | grep 9021 tcp 0 0 10.10.10.1:9021 :::* LISTEN 67064/dataheadsvc
- You can run the ps command to verify the start time of the datahead service compared to the current time on the node.
admin@ecs-node1:/tmp> ps -ef | grep dataheadsvc storage+ 29052 11163 0 May19 ? 00:00:00 /opt/storageos/bin/monitor -u 444 -g 444 -c / -l /opt/storageos/logs/dataheadsvc.out -p /var/run/dataheadsvc.pid /opt/storageos/bin/dataheadsvc file:/opt/storageos/conf/datahead-conf.xml storage+ 57064 29052 88 20:27 ? 00:00:51 /opt/storageos/bin/dataheadsvc -ea -server -d64 -Xmx9216m -Dproduct.home=/opt/storageos -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/storageos/logs/dataheadsvc-78517.hprof -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Dlog4j.configurationFile=file:/opt/storageos/conf/dataheadsvc-log4j2.xml -Xmn2560m -Dsun.net.inetaddr.ttl=0 -Demc.storageos.useFastMD5=1 -Dcom.twmacinta.util.MD5.NATIVE_LIB_FILE=/opt/storageos/lib/MD5.so -Dsun.security.jgss.native=true -Dsun.security.jgss.lib=libgssglue.so.1 -Djavax.security.auth.useSubjectCredsOnly=false -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+ExplicitGCInvokesConcurrent -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution -XX:+PrintGCDateStamps -Xloggc:/opt/storageos/logs/dataheadsvc-gc-9.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=50M com.emc.storageos.data.head.Main file:/opt/storageos/conf/datahead-conf.xml admin@ecs-node1:/tmp> date Wed Jun 7 20:28:41 UTC 2017
3. Verify the Installed Certificates. The object certificate and management certificate each have their own GET request to retrieve the installed certificate. Note that these commands are management requests.
a. Verify the installed/Active Management Certificate
An alternative method to this one, which I used personally, is the OpenSSL s_client command. The details in step 3a below aren’t necessary if you are going to use s_client for verification, I’ve simply included them here for completeness. You can skip to step 3b for the s_client method.
- Use ECSCLI to run it from a client PC:
python ecscli.py vdc_keystore get –hostname <ecs host ip> -port 4443 –cf <cookiefile>
- Use CURL to run it directly from the ECS management console:
curl -svk -H "X-SDS-AUTH-TOKEN: $TOKEN" https://10.10.10.1:4443/vdc/keystore
Verify the installed (active) Object Certificate. This can be done using a variety of methods, outlined below.
- Use ECSCLI to run it from a client PC:
python ecscli.py keystore show –hostname <ecs host ip> -port 4443 –cf <cookiefile>
- Use CURL to run it directly from the ECS management console:
curl -svk -H "X-SDS-AUTH-TOKEN: $TOKEN" https://10.10.10.1:4443/object-cert/keystore
b. The certificate presented by a port can also be verified using OpenSSL’s s_client tool. If you used the method in step 3a, this is unnecessary as it will give you the same information.
- Sample command syntax:
openssl s_client -connect host:port -showcerts
- The command syntax I used and some sample output from my ECS environment is below. Verify the certificate on last node as well as the expected SAN entries.
openssl s_client -connect 10.10.10.1:9021 | openssl x509 -noout -text
openssl s_client -connect 10.10.10.1:9021 -showcerts
CONNECTED(00000003) depth=0 C = US, ST = North Dakota, L = Fargo, OU = server, O = CompanyName Worldwide, CN = *.nd.dev.ecs.CompanyName.int verify error:num=18:self-signed certificate verify return:1 depth=0 C = US, ST = North Dakota, L = Fargo, OU = server, O = CompanyName Worldwide, CN = *.nd.dev.ecs.CompanyName.int verify return:1 --- Certificate chain 0 s:/C=US/ST=North Dakota/L=Fargo/OU=server/O=CompanyName Worldwide/CN=*.nd.dev.ecs.CompanyName.int i:/C=US/ST=North Dakota/L=Fargo/OU=server/O=CompanyName Worldwide/CN=*.nd.dev.ecs.CompanyName.int -----BEGIN CERTIFICATE----- MIIFQDCCBCigAwIBAgIJANzBojR+ij2xMA0GCSqGSIb3DQEBBQUAMIGRMQswCQYD VQQGEwJVUzERMA8GA1UECBMITWlzc291cmkxFDASBgNVBAcTC1NhaW50IExvdWlz … <output truncated for this example> …
c. The process is now complete. You can have your application team test SSL access to ensure everything is working properly.
