During a migration from Cognos Metrics to Elasticsearch Metrics, I had some issues with the index. So I wanted to create a backup of the already migrated data and start over from scratch.
The official documentation has an article on the topic: Backing up and restoring data for Elasticsearch-based components , but I had to slightly adjust the commands to get a successful snapshot.
The documentation shows commands only for Elasticsearch 5, with Connections 7 you have the option to switch to Elasticsearch 7, then all pod names get an additional -7 in the name!
During the deeper analysis of the backup and restore scripts, I found that I have to change the helm chart, so I will show the process for Elasticsearch 5 & 7.
Requirement helm 3 and connections as default namespace!
Set default namespace for kubectl
I don’t want to type -n connections over and over again with each kubectl command, so I set connections as a default:
kubectl config set-context --current --namespace=connections
Register snapshot repository
Run this on your Kubernetes master or a machine configured for accessing the Kubernetes cluster with kubectl.
Open a shell in one o the es-client pods (works for ES 5 and 7):
kubectl exec -ti -n connections $(kubectl get pods -n connections |grep es-client |awk '{print $1}' |head -n 1) -- bash
The documentation tells us, that the default for the snapshot name is connectionsbackup, but the doBackup.sh script in /opt/elasticsearch-5.5.1/probe (in version 6.0) uses connectionsmetrics to create a snapshot, newer versions define the default repository in the helm chart values.yaml and it is still connectionsmetrics. Using the documented default will end in an error.
Run the following commands in the container shell:
cd /opt/elasticsearch-${ES_VERSION}/probe/
./sendRequest.sh PUT /_snapshot/connectionsmetrics \
-H 'Content-Type: application/json' \
-d '{"type": "fs","settings": {"compress": true,"location": "/backup"}}'
Response from server:
{ "acknowledged": true }
Check the settings of the backup repository:
./sendRequest.sh GET /_snapshot/_all?pretty
Response from server:
{
"connectionsmetrics": {
"type": "fs",
"settings": { "compress": "true", "location": "/backup" }
}
}
You can keep the container shell open, we will need it a little bit later again.
Get image tag and registry
Run this on your Kubernetes master or a machine configured for accessing the Kubernetes cluster with kubectl.
We have to use different commands for version 5 and 7, so I ran a short script to find out the deployed version, or just use es-data for version 5 and es-data-7 for version 7.
if [ $(kubectl get pods | grep es-data | head -n 1 | awk -F'-' '$3 == "7" {print $3}') -eq "7" ]
then
version=-7
elasticVersion=7
else
version=
elasticVersion=5
fi
You should see something similar to the following output:
estag=$(kubectl get statefulset es-data${version} -o=jsonpath='{$.spec.template.spec.containers[:1].image}' | awk -F: '{print $3}')
registry=$(kubectl get statefulset es-data${version} -o=jsonpath='{$.spec.template.spec.containers[:1].image}' | awk -F/ '{print $1}')
echo $estag
20180503-104412
echo $registry
cnx7-rh8-cp.stoeps.home:5000
helm chart for backup and restore
I tried to use the helm chart delivered with Componentpack, but in the template files the path for version 5.5 is hard coded, the namespace is missing for the used image and I wanted to use the chart for version 7 and 5 of Elasticsearch.
So I rewrote the provided files, added the missing variables and if-else conditions. You can download the adjusted chart , I changed the name to esbackuprestore-0.1.1.tgz, so you can keep it in the same folder as the orginal file (esbackuprestore-0.1.0.tgz).
Create snapshot
Delete esbackuprestore helm deployment
If you already have used the helm chart, you need to delete esbackuprestore to run the install commands again.
Check if the chart is already installed:
helm list | grep esbackuprestore
If it is already deployed, the command returns:
esbackuprestore connections 1 2022-07-29 20:17:06.456765299 +0000 UTC failed esbackuprestore-0.1.1
Then delete it:
helm delete esbackuprestore -n connections
No matter if you get an error here, I assume you never created an Elasticsearch backup within Componentpack.
Create snapshot
cd extractedFolder/microservices_connections/hybridcloud/helmbuilds/
helm install esbackuprestore esbackuprestore-0.1.1.tgz \
--set image.tag=$estag,elasticSearchBackup=true,image.repository=$registry,elasticVersion=$elasticVersion
To make more snapshots just run the command to delete and install again.
I get an error message, that the job is already deployed, but the snapshot is created. I will check the root cause, but actually the job runs and creates snapshots.
Restore snapshot
First we need to find out the snapshot name. To make this a little bit easier, download and install jq or gron . I prefer gron, the syntax is easier and you can grep within the results.
Get snapshot name with gron
# Version 5
kubectl exec -ti -n connections -c es-client \
$(kubectl get pods -n connections | grep es-client | awk '{print $1}' | head -n 1) \
-- /opt/elasticsearch-5.5.1/probe//sendRequest.sh GET /_snapshot/connectionsmetrics/_all \
| grep snapshots | gron | grep "snapshot ="
# Version 7
kubectl exec -ti -n connections -c es-client \
$(kubectl get pods -n connections |grep es-client |awk '{print $1}' |head -n 1) \
-- /opt/elasticsearch-7.10.1/probe//sendRequest.sh GET /_snapshot/connectionsmetrics/_all \
| grep snapshots | gron | grep "snapshot ="
Result:
json.snapshots[0].snapshot = "snapshot20220729182836";
json.snapshots[1].snapshot = "snapshot20220729183056";
json.snapshots[2].snapshot = "snapshot20220729183246";
Get snapshot name with jq
# Version 5
kubectl exec -ti -n connections -c es-client \
$(kubectl get pods -n connections |grep es-client |awk '{print $1}' |head -n 1) \
-- /opt/elasticsearch-5.5.1/probe/sendRequest.sh GET /_snapshot/connectionsmetrics/_all \
| grep snapshots | jq '.snapshots[] | .snapshot'
# Version 7
kubectl exec -ti -n connections -c es-client \
$(kubectl get pods -n connections |grep es-client |awk '{print $1}' |head -n 1) \
-- /opt/elasticsearch-7.10.1/probe/sendRequest.sh GET /_snapshot/connectionsmetrics/_all \
| grep snapshots | jq '.snapshots[] | .snapshot'
Result:
"snapshot20220729182836"
"snapshot20220729183056"
"snapshot20220729183246"
Restore command
Adding | tail -n 1 to the get name commands, shows the last created snapshot, copy the name and use with helm. Before running the helm command to need again delete the esbackuprestore application.
With the restore command we need REPONAME (Default: connectionsmetrics) and SNAPSHOTNAME.
helm delete esbackuprestore
helm install esbackuprestore esbackuprestore-0.1.1.tgz \
--set image.tag=$estag,elasticSearchRestore=true,image.repository=$registry,namespace=connections,/
SNAPSHOTNAME=snapshot20220729183246,REPONAME=connectionsmetrics
Delete snapshots
The documentation does not tell us how to delete snapshots, I have no experience how large they get, so if you ever have to remove older snapshots, open a shell in one of the es-data pods and run the command:
/opt/elasticsearch-${ES_VERSION}/probe/sendRequest.sh DELETE /_snapshot/connectionsmetrics/snapshotname-to-delete
Do not use restore at the moment!!!
I tested the restore and it fails, because one of the indices is open and can’t be restored.
Problem here is, that the doRestore.sh closes all indices, when it is started, but only opens them when the restore is successful.
So the whole process with these charts (mine, or the original one from HCL) can’t be recommended at the moment.
I think the backup is important, because Snapshot and restore tells us:
Taking a snapshot is the only reliable and supported way to back up a cluster. You cannot back up an Elasticsearch cluster by making copies of the data directories of its nodes. There are no supported methods to restore any data from a filesystem-level backup.
Elasticsearch is used to store Metrics and typeahead data, no content, but still valid data for us and our users.
Now we need to find a way to get a reliable way to restore these snapshots, if this is done, I will work on a way to backup mongoDB. The mongoDB is used with Activities Plus and afaik with Orient Me, but documentation does not show a way to backup.
