Today I activated Elasticsearch Metrics and Typeahead Search on my demo HCL Connections cluster .
To my surprise the indices weren’t created and I got errors on the wsadmin.sh commands.
SearchService.createESQuickResultsIndex()
I checked the Elasticsearch pods which showed a running state, but the logs showed following messages:
[es-master-c9cc5d877-trwsd es-master] [2020-10-08T12:11:13,708][WARN ][o.e.c.r.a.DiskThresholdMonitor] \
[es-master-c9cc5d877-trwsd] high disk watermark [90%] exceeded on [sqwY58MtRa2MoPkHr1o70Q][es-data-2][/data/data/nodes/0] \
free: 2.7gb[5.4%], shards will be relocated away from this node
So even with 2.7GB free space it does not generate the needed shards and indices. This is default behavior of Elasticsearch , but can be changed :
curl -XPUT "http://es-host:9200/_cluster/settings" \
-H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster": {
"routing": {
"allocation.disk.threshold_enabled": false
}
}
}
}'
I’m pretty sure I will need this in customer environments too, because on big NFS clusters 10% disks can be equal to some hundred GB diskspace.
BUT this will not work with the Elasticsearch pods of HCL Connections Componentpack, because it is secured with Secureguard and you need to have a client SSL certificate for authentication.
Connect to Elasticsearch on the Componentpack Kubernetes Cluster
During the setup of Elasticsearch metrics, you export the keystore and ca keys from Kubernetes secrets:
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12
We can import the elasticssearch-metrics.p12 to our browser and it will use the include certificate to connect to the elasticsearch server.
So accessing https://kubernetes-node:30099/_cat/indices will show you all available indices and their status, after you imported the keystore.
To use the adminkeys, we need to export two additional keys:
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.key']}" | base64 -d > elasticsearch-admin.key
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem
Check Server settings
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{},"transient":{}}
Now we can connect with curl and disable the threshold
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XPUT "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings" \
-H 'Content-Type: application/json' -d'
{
"persistent": {
"cluster": {
"routing": {
"allocation.disk.threshold_enabled": false
}
}
}
}'
Enter PEM pass phrase:
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}%
- Use the password from you Componentpack setup
If you can’t remember the password, use this command:
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-key-password\.txt']}" | base64 -d
password
Check setting
You can open the url with your browser (where you imported the keystore), or use curl again:
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}
See logs of all Elasticsearch pods
kubetail -l component=elasticsearch -n connections
During my tests, I checked `elasticsearch-metrics.p12’ with https://keystore-explorer.org and found, that the client key is only valid for two years. So we need to remember to update this (or hope that HCL will provide us with an update and documentation on this topic).
Or check the imported certificate in the browser.
Why is this important?
During the setup we had to import the keystore to WebSphere and copy it to all WebSphere nodes! I will not update automatically.
