Elasticsearch index creation problem

by Christoph Stoettner
Read in about 3 min · 509 words

Fountain pen and a notebook

Photo by Aaron Burden | Unsplash

Today I activated Elasticsearch Metrics and Typeahead Search on my demo HCL Connections cluster .

To my surprise the indices weren’t created and I got errors on the wsadmin.sh commands.

SearchService.createESQuickResultsIndex()

I checked the Elasticsearch pods which showed a running state, but the logs showed following messages:

[es-master-c9cc5d877-trwsd es-master] [2020-10-08T12:11:13,708][WARN ][o.e.c.r.a.DiskThresholdMonitor] \
[es-master-c9cc5d877-trwsd] high disk watermark [90%] exceeded on [sqwY58MtRa2MoPkHr1o70Q][es-data-2][/data/data/nodes/0] \
free: 2.7gb[5.4%], shards will be relocated away from this node

So even with 2.7GB free space it does not generate the needed shards and indices. This is default behavior of Elasticsearch , but can be changed :

curl -XPUT "http://es-host:9200/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'

I’m pretty sure I will need this in customer environments too, because on big NFS clusters 10% disks can be equal to some hundred GB diskspace.

BUT this will not work with the Elasticsearch pods of HCL Connections Componentpack, because it is secured with Secureguard and you need to have a client SSL certificate for authentication.

Connect to Elasticsearch on the Componentpack Kubernetes Cluster

During the setup of Elasticsearch metrics, you export the keystore and ca keys from Kubernetes secrets:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['chain-ca\.pem']}" | base64 -d > chain-ca.pem
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-metrics\.p12']}" | base64 -d > elasticsearch-metrics.p12

We can import the elasticssearch-metrics.p12 to our browser and it will use the include certificate to connect to the elasticsearch server.

So accessing https://kubernetes-node:30099/_cat/indices will show you all available indices and their status, after you imported the keystore.

To use the adminkeys, we need to export two additional keys:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.key']}" | base64 -d > elasticsearch-admin.key
kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-admin\.crt\.pem']}" | base64 -d > elasticsearch-admin.crt.pem

Check Server settings

curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{},"transient":{}}

Now we can connect with curl and disable the threshold

curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XPUT "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings" \
 -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation.disk.threshold_enabled": false
      }
    }
  }
}'
Enter PEM pass phrase:      
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}%
  • Use the password from you Componentpack setup

If you can’t remember the password, use this command:

kubectl get secret elasticsearch-secret -n connections -o=jsonpath="{.data['elasticsearch-key-password\.txt']}" | base64 -d
password

Check setting

You can open the url with your browser (where you imported the keystore), or use curl again:

curl --key elasticsearch-admin.key --cert elasticsearch-admin.crt.pem -k --cacert chain-ca.pem -XGET "https://cnx651-k8s-node1.stoeps.internal:30099/_cluster/settings"
Enter PEM pass phrase:
{"persistent":{"cluster":{"routing":{"allocation":{"disk":{"threshold_enabled":"false"}}}}},"transient":{}}

2020 10 08 23 37

See logs of all Elasticsearch pods

Download kubetail

kubetail -l component=elasticsearch -n connections

During my tests, I checked `elasticsearch-metrics.p12’ with https://keystore-explorer.org and found, that the client key is only valid for two years. So we need to remember to update this (or hope that HCL will provide us with an update and documentation on this topic).

Or check the imported certificate in the browser.

2020 10 08 23 43

Why is this important?

During the setup we had to import the keystore to WebSphere and copy it to all WebSphere nodes! I will not update automatically.

Author
Add a comment
Error
There was an error sending your comment, please try again.
Thank you!
Your comment has been submitted and will be published once it has been approved.

Your email address will not be published. Required fields are marked with *

Suggested Reading
Card image cap

CVE-2021-44228 was a very serious problem end of 2021, and we are still finding new occurrences, or security teams scan servers and find vulnerable log4j files. Don’t get me wrong most of these occurrences are not vulnerable any more, because the JVM is hardened like in the Elasticsearch 7 containers, or they use of the JVM parameter -Dlog4j2.formatMsgNoLookups=true.

Read in about 3 min
Card image cap

Today I got the question of how to disable the highlights app in Connections 7. When you follow the documentation for Connections 6.0CR6 you get an error message (and the document is not available in Connections 7).

I commented out the widget definition in widgets-config.xml like described in the documentation for the former release.

Read in about 2 min
Card image cap

In late 2021 I had an HCL Connections environment starting swapping, because the AppCluster used more than 30 GB of memory.

The system has

  • two nodes
  • is installed with the medium-sized deployment option
  • About 7500 users with a high adoption rate, because Connections is also used as intranet
Read in about 5 min