:icons: font
After some deployments of IBM Connections pink and IBM Cloud private, I want to share some tools, links and hopefully helpful information around these products.
Sources to start with troubleshooting Orientme
Blogs:
IBM Think:
IBM Technotes:
Some shortcuts, commands and tools
During my reading and researches, I optimized some of the shown scripts in the above mentioned links. On the other side I hate to open logs in each single pod, so I searched for a better solution.
Getting the logs of multiple pods
On the command line you can just use kubetail . Just download this script to your master node of IBM Cloud private. Move it to the PATH and make executable:
wget https://raw.githubusercontent.com/johanhaleby/kubetail/master/kubetail
sudo mv kubetail /usr/local/bin
sudo chmod +x /usr/local/bin/kubetail
With kubetail you get log messages in different colors for different pods.
kubetail es-master -s 15s
This shows the logs of the last 15 seconds of all elasticsearch pods and until you hit Ctrl+c you will get all new messages too.
Set default namespace in CfC/ICP
With Orientme 6.0.0.4 the namespace of all pink pods is moved from default to connections. So you need to add the namespace to each kubectl command you run on the console. So I like to change the default namespace for kubectl:
kubectl config set-context $(kubectl config current-context) --namespace=connections
To switch this back to the IBM default use:
You need to switch the default namespace back before you update your environment! The update script for 6.0.0.5 will not run properly when you set a different namespace!
kubectl config set-context $(kubectl config current-context) --namespace=default
To check which settings are used:
kubectl config view
Speed up the technote commands
Mongo and Redis Pods in an Unknown State
Original from Mongo and Redis Pods in an Unknown State
kubectl get pods -n connections | grep redis-server
kubectl get pods -n connections | grep mongo
Or combine the two commands with:
kubectl get pods -n connections | grep -E "redis-server|mongo"
mongo-0 2/2 Unknown 0 19h
mongo-1 2/2 Running 0 19h
mongo-2 2/2 Running 0 19h
redis-server-0 1/1 Running 0 19h
redis-server-1 1/1 Unknown 0 19h
redis-server-2 1/1 Running 0 19h
None of the listed pods should have the state unknown. Resolving that issue:
kubectl delete pod <pod name=""> -n connections --grace-period=0 --force</pod>
For example:
kubectl delete pod mongo-1 -n connections --grace-period=0 --force
Or you automate it a little and delete all pods with state Unknown:
kubectl delete pod $(kubectl get pods -n connections | grep -E "redis-server|mongo" | grep Unknown | awk '{print $1}') -n connections --grace-period=0 --force
Kubernetes will recreate the pod after a short time.
If Redis is not reachable, Indexing and Analysis tries to reconnect
for 8 times and then fails
Original from If Redis is not reachable, Indexing and Analysis tries to reconnect for 8 times and then fails
kubectl get pods -n connections | grep -E "analysisservice|indexingservice"
This will return a list of running pods:
analysisservice-132048526-bh02z 1/1 Running 4 2d
analysisservice-132048526-dhm45 1/1 Running 3 2d
indexingservice-3077571376-9b541 1/1 Running 0 3d
indexingservice-3077571376-qvq7j 1/1 Running 3 3d
indexingservice-3077571376-t61wh 1/1 Running 0 3d
Check the logs (use kubetail) of these pods! If you see “Exception while connecting”, delete the pods and kubernetes will recreate them.
# Show logs of all pods in the default namespace, including last 5 lines of each log
kubetail --tail 5
# Show all logs on console and append output to logfile
kubetail --tail 5 | tee -a logfile
# This will show logs of indexingservice and analysisservice
kubetail '(indexingservice|analysisservice)' -e regex -s 30s
# Delete all indexingservice and analysisservice pods
kubectl delete pod $(kubectl get pods -n connections | grep -E "analysisservice|indexingservice" | awk '{print $1}') -n connections --grace-period=0 --force
So next time I try to get all that directly from the ICP Admin Interface or Elasticsearch.
