Elasticsearch index creation problem

by Christoph Stoettner
Read in about 3 min · (words)

Fountain pen and a notebook

Photo by Aaron Burden | Unsplash

To my surprise the indices weren’t created and I got errors on the wsadmin.sh commands.

SearchService.createESQuickResultsIndex()

I checked the Elasticsearch pods which showed a running state, but the logs showed following messages:

[es-master-c9cc5d877-trwsd es-master] [2020-10-08T12:11:13,708][WARN ][o.e.c.r.a.DiskThresholdMonitor] \
[es-master-c9cc5d877-trwsd] high disk watermark [90%] exceeded on [sqwY58MtRa2MoPkHr1o70Q][es-data-2][/data/data/nodes/0] \
free: 2.7gb[5.4%], shards will be relocated away from this node

So even with 2.7GB free space it does not generate the needed shards and indices. This is default behavior of Elasticsearch, but can be changed:

curl -XPUT "http://es-host:9200/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'

I’m pretty sure I will need this in customer environments too, because on big NFS clusters 10% disks can be equal to some hundred GB diskspace.

BUT this will not work with the Elasticsearch pods of HCL Connections Componentpack, because it is secured with Secureguard and you need to have a client SSL certificate for authentication.

Connect to Elasticsearch on the Componentpack Kubernetes Cluster

During the setup of Elasticsearch metrics, you export the keystore and ca keys from Kubernetes secrets:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12

We can import the elasticssearch-metrics.p12 to our browser and it will use the include certificate to connect to the elasticsearch server.

So accessing https://kubernetes-node:30099/_cat/indices will show you all available indices and their status, after you imported the keystore.

To use the adminkeys, we need to export two additional keys:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.key']}" | base64 -d > elasticsearch-admin.key
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem
Check Server settings
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{},"transient":{}}
Now we can connect with curl and disable the threshold
curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XPUT "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'
Enter PEM pass phrase:      (1)
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}%
1Use the password from you Componentpack setup

If you can’t remember the password, use this command:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-key-password\.txt']}" | base64 -d
password

Check setting

You can open the url with your browser (where you imported the keystore), or use curl again:

curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}
2020 10 08 23 37

See logs of all Elasticsearch pods

kubetail -l component=elasticsearch -n connections

During my tests, I checked `elasticsearch-metrics.p12' with https://keystore-explorer.org and found, that the client key is only valid for two years. So we need to remember to update this (or hope that HCL will provide us with an update and documentation on this topic).

Or check the imported certificate in the browser.

2020 10 08 23 43

Why is this important?

During the setup we had to import the keystore to WebSphere and copy it to all WebSphere nodes! I will not update automatically.

Author
Add a comment
Error
There was an error sending your comment, please try again.
Thank you!
Your comment has been submitted and will be published once it has been approved.

Your email address will not be published. Required fields are marked with *

Suggested Reading
Card image cap

Connections 7 has a new Administration Console to access Communities Template administration and Mobile Administration.

The Administration Console can be reached on https://cnx-hostname/cnxadmin/. The / at the end is important, because the ingress rule just forwards /cnxadmin/(.*).

On Firefox you get this view:

Read in about 2 min
Card image cap

Some time ago I got the tip from HCL Support, that the Create Community button will recognize the role Community-creator only when the gatekeeper option CATALOG_CARD_UPDATED is set to false.

This is working, but I had to complain, that this option activates some code, which loads fonts from a cdn instead of the local Connections deployment.

Read in about 1 min
Card image cap

The newsletter in version 2 format has one big disadvantage, we or our users lost the easy links to the topic, author profile or application.

Read in about 2 min