Elasticsearch index creation problem

· by Christoph Stoettner · Read in about 3 min · (508 words)

To my surprise the indices weren’t created and I got errors on the wsadmin.sh commands.

SearchService.createESQuickResultsIndex()

I checked the Elasticsearch pods which showed a running state, but the logs showed following messages:

[es-master-c9cc5d877-trwsd es-master] [2020-10-08T12:11:13,708][WARN ][o.e.c.r.a.DiskThresholdMonitor] \
[es-master-c9cc5d877-trwsd] high disk watermark [90%] exceeded on [sqwY58MtRa2MoPkHr1o70Q][es-data-2][/data/data/nodes/0] \
free: 2.7gb[5.4%], shards will be relocated away from this node

So even with 2.7GB free space it does not generate the needed shards and indices. This is default behavior of Elasticsearch, but can be changed:

curl -XPUT "http://es-host:9200/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'

I’m pretty sure I will need this in customer environments too, because on big NFS clusters 10% disks can be equal to some hundred GB diskspace.

BUT this will not work with the Elasticsearch pods of HCL Connections Componentpack, because it is secured with Secureguard and you need to have a client SSL certificate for authentication.

Connect to Elasticsearch on the Componentpack Kubernetes Cluster

During the setup of Elasticsearch metrics, you export the keystore and ca keys from Kubernetes secrets:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12

We can import the elasticssearch-metrics.p12 to our browser and it will use the include certificate to connect to the elasticsearch server.

So accessing https://kubernetes-node:30099/_cat/indices will show you all available indices and their status, after you imported the keystore.

To use the adminkeys, we need to export two additional keys:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.key']}" | base64 -d > elasticsearch-admin.key
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem
Check Server settings
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{},"transient":{}}
Now we can connect with curl and disable the threshold
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XPUT "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'
Enter PEM pass phrase:      (1)
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}%
1Use the password from you Componentpack setup

If you can’t remember the password, use this command:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-key-password\.txt']}" | base64 -d
password

Check setting

You can open the url with your browser (where you imported the keystore), or use curl again:

curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}
2020 10 08 23 37

See logs of all Elasticsearch pods

kubetail -l component=elasticsearch -n connections

During my tests, I checked `elasticsearch-metrics.p12' with https://keystore-explorer.org and found, that the client key is only valid for two years. So we need to remember to update this (or hope that HCL will provide us with an update and documentation on this topic).

Or check the imported certificate in the browser.

2020 10 08 23 43

Why is this important?

During the setup we had to import the keystore to WebSphere and copy it to all WebSphere nodes! I will not update automatically.