Troubleshooting Orientme or a little bit kubectl

· by Christoph Stoettner · Read in about 4 min · (665 words)

After some deployments of IBM Connections pink and IBM Cloud private, I want to share some tools, links and hopefully helpful information around these products.

Some shortcuts, commands and tools

During my reading and researches, I optimized some of the shown scripts in the above mentioned links. On the other side I hate to open logs in each single pod, so I searched for a better solution.

Getting the logs of multiple pods

On the command line you can just use kubetail. Just download this script to your master node of IBM Cloud private. Move it to the PATH and make executable:

sudo mv kubetail /usr/local/bin
sudo chmod +x /usr/local/bin/kubetail

With kubetail you get log messages in different colors for different pods.

screen 20180207 112358 001

kubetail es-master -s 15s

This shows the logs of the last 15 seconds of all elasticsearch pods and until you hit Ctrl+c you will get all new messages too.

Set default namespace in CfC/ICP

With Orientme the namespace of all pink pods is moved from default to connections. So you need to add the namespace to each kubectl command you run on the console. So I like to change the default namespace for kubectl:

kubectl config set-context $(kubectl config current-context) --namespace=connections

To switch this back to the IBM default use:

kubectl config set-context $(kubectl config current-context) --namespace=default

To check which settings are used:

screen 20180207 113257 001

kubectl config view

Speed up the technote commands

Mongo and Redis Pods in an Unknown State

kubectl get pods -n connections | grep redis-server
kubectl get pods -n connections | grep mongo

Or combine the two commands with:

kubectl get pods -n connections | grep -E "redis-server|mongo"

mongo-0                                2/2       Unknown   0          19h
mongo-1                                2/2       Running   0          19h
mongo-2                                2/2       Running   0          19h
redis-server-0                         1/1       Running   0          19h
redis-server-1                         1/1       Unknown   0          19h
redis-server-2                         1/1       Running   0          19h

None of the listed pods should have the state unknown. Resolving that issue:

kubectl delete pod <pod name=""> -n connections --grace-period=0 --force</pod>

For example:

kubectl delete pod mongo-1 -n connections --grace-period=0 --force

Or you automate it a little and delete all pods with state Unknown:

kubectl delete pod $(kubectl get pods -n connections | grep -E "redis-server|mongo" | grep Unknown | awk '{print $1}') -n connections --grace-period=0 --force

Kubernetes will recreate the pod after a short time.

If Redis is not reachable, Indexing and Analysis tries to reconnect

for 8 times and then fails

kubectl get pods -n connections | grep -E "analysisservice|indexingservice"

This will return a list of running pods:

analysisservice-132048526-bh02z 1/1 Running 4 2d
analysisservice-132048526-dhm45 1/1 Running 3 2d
indexingservice-3077571376-9b541 1/1 Running 0 3d
indexingservice-3077571376-qvq7j 1/1 Running 3 3d
indexingservice-3077571376-t61wh 1/1 Running 0 3d

Check the logs (use kubetail) of these pods! If you see “Exception while connecting”, delete the pods and kubernetes will recreate them.

# Show logs of all pods in the default namespace, including last 5 lines of each log
kubetail --tail 5

# Show all logs on console and append output to logfile
kubetail --tail 5 | tee -a logfile

# This will show logs of indexingservice and analysisservice
kubetail '(indexingservice|analysisservice)' -e regex -s 30s

# Delete all indexingservice and analysisservice pods
kubectl delete pod $(kubectl get pods -n connections | grep -E "analysisservice|indexingservice" | awk '{print $1}') -n connections --grace-period=0 --force

So next time I try to get all that directly from the ICP Admin Interface or Elasticsearch.