If you ever had to replace certificates in vSphere 6.0 and 6.5 you may know that there are two different ways to do it. The first one using the PSC web UI and the second one using the embedded certificate-manager utility. Now you may also think that these 2 methods produce a similar result. And why shouldn’t you? Well, unfortunately it is not the case.
The Lookup service is a PSC service that maintains a table of all the services registered. It contains their URL, certificate, solution user and a bunch of obscur things. It is used by services themselves so they can find one another and communicate securely with each other. I mentioned “certificates” in the list of properties registered in the Lookup service and this is where the issue is regarding the way we replace certificates.
PSC UI vs certificate-manager
Both method will replace the machine SSL certificate and have the web client use it. Meaning you will not receive a certificate error when opening it in a browser (provided the signing root CA is in your trusted root CAs). It is why this issue is so confusing, the certificate appears changed but it may not be in the background!
When you run the certificate-manager utility, you will see at the end that a number of services have been updated (more than 25). And if you look at the certificates in the registered services you will see the new one.
If you use the PSC UI however, a quick look at the registered services will show you that the old one is still being used. Which will lead to errors when external services try to interact with vCenter (check out KB2109074). Example below with Site Recovery Manager:
You already replaced it with the PSC UI?
Fortunately there is a way to update the certificate of the registered service using a python script embedded in vCenter. The script will locate the old certificates using its thumbprint and replace it with the new one. As for every maintenance operation on vCenter/PSC server, it is recommended to take a snapshot before cracking on in case things start to go south.
The procedure looks cumbersome by the length of the KB but there is only one command that matters which looks like so:
python ls_update_certs.py --url https://psc.vmware.com/lookupservice/sdk --fingerprint 11:11:AF:D8:CF:27:6B:EF:F7:49:20:3E:D7:90:8C:F6:A0:A2:E2:30 --certfile /certificates/new_machine.crt --user [email protected] --password "Password"
Complete procedure:
In summary:
- If you run the PSC on Windows (vCenter if embedded), download the lsutil zip file from the KB and replace the existing one.
- Find the old certificate, save it as a file, open it and copy the thumbprint.
- Copy the new certificate file somewhere on the PSC server (vCenter if embedded).
- Run the command mentionned above (don’t forget the : in the thumbprint).
It may take a few minutes to run it should finish with a nice Updated 25 service(s).
Conclusion
A rather surprising behaviour I think for a core security component of vCenter. Fortunately the issue is well documented but not well known. If you used the PSC UI, don’t freak out if you encounter a certificate issue and check the registered services.