Elasticsearch in HCL Connections Componentpack is secured with Searchguard and needs certificates to work properly. These certificates are generated by bootstrap during the initial container deployment with helm.
These certificates are valid for 10 years (chain_ca.pem) or 2 years (elasticsearch*.pem) and stored in the Kubernetes secrets elasticsearch-secret, elasticsearch-7-secret. So when your HCL Connections deployment is running for 2 years, the certficates stop working.
The documentation on bootstrap is a little bit misleading and my suggested update does not make it into a technote or documentation update since nearly one year.
Set up bootstrap charts
Bootstrap is one-time job. It creates secrets and certificates for various components, including Redis, MongoDB and ElasticSearch.
Be aware that simply rerunning/upgrading bootstrap also means recreating all those secrets and certificates, and requires in some cases redoing the steps which were dependent on it (like setting up certificates again for Metrics).
Starting with Component Pack 7, bootstrap will also create secrets and certificates for ElasticSearch 7.
For Elasticsearch this is wrong. Bootstrap does only generate new certificates, if the variable env.force_regenerate is set to true. The default is false, so we have to set it during the helm command.
Update Elasticsearch certificates (Example commands for Elasticsearch 7)
If you are still using Elasticsearch 5, then just remove -7 from the commands below.
Get certificate from Kubernetes secret
kubectl get secret elasticsearch-7-secret -n connections \
-o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem
Check dates for certificate
openssl x509 -startdate -enddate -noout -in elasticsearch-admin.crt.pem
notBefore=Aug 16 08:10:54 2022 GMT
notAfter=Aug 15 08:10:54 2024 GMT
Delete bootstrap
helm delete bootstrap -n connections
Reinstall with helm
If we use the command from documentation, the certificate extracted from Kubernetes has still the same enddate.
helm install bootstrap \
/opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
-f /home/ansible/generated_charts/bootstrap.yml -n connections
We need to run
helm install bootstrap \
/opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
-f /home/ansible/generated_charts/bootstrap.yml -n connections --set env.force_regenerate=true
Check if the bootstrap pod is completed, then check the certificate again:
kubectl get pods -n connections | grep bootstrap
bootstrap-p4rj6 0/1 Completed 0 49m
kubectl get secret elasticsearch-7-secret -n connections \
-o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin_neu.crt.pem
openssl x509 -startdate -enddate -noout -in elasticsearch-admin_neu.crt.pem
notBefore=Sep 2 10:01:56 2022 GMT
notAfter=Sep 1 10:01:56 2024 GMT
Restart Elasticsearch
kubectl rollout restart sts es-master-7 es-data-7 -n connections
kubectl rollout restart deploy es-client-7 -n connections
Recreate elasticsearch-metrics.p12
kubectl get secret elasticsearch-7-secret -n connections \
-o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem
kubectl get secret elasticsearch-7-secret -n connections \
-o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12
Copy these certificates to the WebSphere Deployment Manager, check the path of the old certificate store and use the same.
cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin
./wsadmin.sh -lang jython -user wasadmin_user -password wasadmin_password
execfile('esSecurityAdmin.py')
enableSslForMetrics('/opt/IBM/certs/es_certs/elasticsearch-metrics.p12', 'Elasticsearch_CA_password', '/opt/IBM/certs/es_certs/chain-ca.pem', '30098')
Copy the updated elasticsearch-metrics.p12 file from the Deployment Manager to the same location on the WebSphere Application Server nodes. Then restart your WebSphere environment (with Deployment Manager and Node Agents).
