Update Elasticsearch certificates in Componentpack

Created:
Author: Christoph Stoettner
Read in about 3 min · 476 words

Container customs

Photo by Renaldo Matamoro | Unsplash

Elasticsearch in HCL Connections Componentpack is secured with Searchguard and needs certificates to work properly. These certificates are generated by bootstrap during the initial container deployment with helm.

These certificates are valid for 10 years (chain_ca.pem) or 2 years (elasticsearch*.pem) and stored in the Kubernetes secrets elasticsearch-secret, elasticsearch-7-secret. So when your HCL Connections deployment is running for 2 years, the certficates stop working.

The documentation on bootstrap is a little bit misleading and my suggested update does not make it into a technote or documentation update since nearly one year.

Set up bootstrap charts

Bootstrap is one-time job. It creates secrets and certificates for various components, including Redis, MongoDB and ElasticSearch.

Be aware that simply rerunning/upgrading bootstrap also means recreating all those secrets and certificates, and requires in some cases redoing the steps which were dependent on it (like setting up certificates again for Metrics).

Starting with Component Pack 7, bootstrap will also create secrets and certificates for ElasticSearch 7.

https://help.hcltechsw.com/connections/v7/admin/install/cp_install_services_tasks.html?hl=bootstrap#cp_install_services_tasks__section_iqb_24c_qmb

HCL Connections documentation

For Elasticsearch this is wrong. Bootstrap does only generate new certificates, if the variable env.force_regenerate is set to true. The default is false, so we have to set it during the helm command.

Update Elasticsearch certificates (Example commands for Elasticsearch 7)

If you are still using Elasticsearch 5, then just remove -7 from the commands below.

Get certificate from Kubernetes secret

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem

Check dates for certificate

openssl x509 -startdate -enddate -noout -in elasticsearch-admin.crt.pem
notBefore=Aug 16 08:10:54 2022 GMT
notAfter=Aug 15 08:10:54 2024 GMT

Delete bootstrap

helm delete bootstrap -n connections

Reinstall with helm

If we use the command from documentation, the certificate extracted from Kubernetes has still the same enddate.

helm install bootstrap \
   /opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
   -f /home/ansible/generated_charts/bootstrap.yml -n connections

We need to run

helm install bootstrap \
   /opt/hcl-cnx-component-pack/microservices_connections/hybridcloud/helmbuilds/bootstrap-0.1.0-20210418-223218.tgz \
   -f /home/ansible/generated_charts/bootstrap.yml -n connections --set env.force_regenerate=true

Check if the bootstrap pod is completed, then check the certificate again:

kubectl get pods -n connections | grep bootstrap
bootstrap-p4rj6                                             0/1     Completed   0          49m
kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin_neu.crt.pem

openssl x509 -startdate -enddate -noout -in elasticsearch-admin_neu.crt.pem
notBefore=Sep  2 10:01:56 2022 GMT
notAfter=Sep  1 10:01:56 2024 GMT

Restart Elasticsearch

kubectl rollout restart sts es-master-7 es-data-7 -n connections
kubectl rollout restart deploy es-client-7 -n connections

Recreate elasticsearch-metrics.p12

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem

kubectl get secret elasticsearch-7-secret -n connections \
   -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12

Copy these certificates to the WebSphere Deployment Manager, check the path of the old certificate store and use the same.

cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin
./wsadmin.sh -lang jython -user wasadmin_user -password wasadmin_password

execfile('esSecurityAdmin.py')
enableSslForMetrics('/opt/IBM/certs/es_certs/elasticsearch-metrics.p12', 'Elasticsearch_CA_password', '/opt/IBM/certs/es_certs/chain-ca.pem', '30098')

Copy the updated elasticsearch-metrics.p12 file from the Deployment Manager to the same location on the WebSphere Application Server nodes. Then restart your WebSphere environment (with Deployment Manager and Node Agents).

Author
Suggested Reading
Card image cap

During a migration from Cognos Metrics to Elasticsearch Metrics, I had some issues with the index. So I wanted to create a backup of the already migrated data and start over from scratch.

The official documentation has an article on the topic: Backing up and restoring data for Elasticsearch-based components , but I had to slightly adjust the commands to get a successful snapshot.

Created:
Last Update:
Read in about 6 min
Card image cap

Last week I played around with the HCL Connections documentation to backup Elasticsearch in the article Backup Elasticsearch Indices in Component Pack .

In the end I found that I couldn’t get the snapshot restored and that I have to run a command outside of my Kubernetes cluster to get a snapshot on a daily basis. That’s not what I want.

Created: Read in about 4 min
Card image cap

A long time ago, I wrote about the new implementation of allowlists in HCL Connections and that the documentation on customization and adding new rules was an absolute miracle for me.

Created:
Last Update:
Read in about 5 min