Migrate MongoDB in HCL Connections Component Pack 8

Last Update:

Author: Christoph Stoettner
Read in about 7 min · 1314 words

Library Database

Photo by Jan Antonin Kolar | Unsplash

The official documentation, “Migrating data from MongoDB 3 to 5”, wants to dump the MongoDB databases in 3.6 and then restore this data into the newly deployed MongoDB 5.

One issue with this process is that we can’t run the two MongoDB versions in parallel on Kubernetes because the provided helm charts and container for MongoDB 3.6 stop running after Kubernetes 1.21. On the other side, the helm chart providing MongoDB 5 can’t be installed on those old Kubernetes versions. So the process to update is:

Migration process

  1. Dump databases in MongoDB 3.6 (version delivered with Connections 7)
  2. Update Kubernetes to 1.25 or 1.27
  3. Restore MongoDB databases to version 5.0

So, you have to plan the process in advance because it is difficult to get the data when you forget something.

Update Kubernetes

The process of updating Kubernetes from 1.21 to 1.27 is quite a challenge because you have to jump from version to version: 1.21 → 1.22 → 1.23 → 1.24 → 1.25 → 1.26 → 1.27. Or you destroy your Kubernetes deployment and just create a new one with Kubernetes 1.27.

Create database dumps

I started dumping the databases and found that these 50 GB (it is a large environment) of MongoDB databases will need about three days until the dump is finished. We estimated the restoration to take three to four days. This results in a migration time of at least one week, and there is no migration possible of the data created in the databases during this week. Nobody can shut down their Connections’ environment for multiple days, so we required another solution.

A case at HCL wasn’t helpful at all, and in the end I started reading the MongoDB documentation and tested some options.

I learned that we can migrate the data in-place, but the process is updating in a row 3.6 → 4.0 → 4.2 → 4.4 → 5.0. The documentation recommends updating the database binaries, set the compatibility version, shutdown the database and going to the next version. The databases are stored in NFS shares, so in my opinion, the easiest way is to use some Docker containers and mount the database directory to these containers.

Another option would be to add new members to the replicaSet, but this would need some code adjustments in the pods. These changes I wanted to avoid and as I already wrote, this would need adjustments in the helm charts deploying MongoDB.

Get the MongoDB 3.6 data to the 5.0 shares

My first try was copying the shares mongo-node-{0,1,2} to the new ones mongo5-node-{0,1,2}, but this means nearly 150 GB of data copied over NFS.

Copying the database data to a NFS requires some time, so my next step was moving the data from the mongo-node-{0,1,2} shares tomongo5-node-{0,1,2}. Moving data is way faster than copying them, but you should have a backup or snapshot.

Backup mongo-node shares

tar the data of MongoDB, this is faster than copying it.

cd /pv-connections
tar cvzf backup-$date +%Y-%m-%d-mongo36.tar.gz mongo-node-?

Move data on NFS share

cd /pv-connections
for i in 0 1 2
  # Backup mongo5 folder
  if [ -d mongo5-node-${i}]; then
    mv mongo5-node-${i} mongo5-node-${i}-old
  mv mongo-node-${i} mongo5-node-${i}

Migrate databases

I used podman installed on the NFS Server to migrate the databases, but you can mount the shares to any other machine with Docker or podman.


We have to start with MongoDB 3.6 because Connections 7 has set CompatibiltyVersion = 3.4!

Pull image

podman pull docker.io/bitnami/mongodb:3.6

Run container

This command expects that you are in the /pv-connections/mongo5-node-0 folder.

podman run -dt --name mongo36 -p 27017:27017 -v $(pwd)/data/db:/bitnami/mongodb/data/db:Z docker.io/bitnami/mongodb:3.6

Remove the replicaset (only for 3.6)

The new statefulSet in MongoDB 5 uses new hostnames, so it is necessary to remove the old hostname from the replicaset.

I run the following commands via docker or podman exec, but you can use mongosh too and run the commands one after another in shell. I wanted to automate the process, this is easier with single commands via podman exec.

podman exec -it mongo36 mongo --host  --eval "db.getSiblingDB('local').system.replset.remove()"

Drop userprefs-service database (only for 3.6)

This database is no longer used and can be deleted.

podman exec -it mongo36 mongo --host  --eval "use userprefs-service" --eval "db.dropDatabase()"

Update CompatibiltyVersion

podman exec -it mongo36 mongo --host  --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.6' } )"

Shutdown database Server

podman exec -it mongo36 mongo --host  --eval "db.getSiblingDB('admin').shutdownServer()"

Stop the container and remove it

podman stop mongo36
podman rm mongo36

That’s almost it, but you have to do the same process in all 3 mongo5-node-{0,1,2} shares and for the other MongoDB versions.

Use a Shell Script for the migration

This script expects that you already moved the old mongo-node folders to mongo5-node.

Using podman

#!/usr/bin/env bash
# Author: Christoph Stoettner
# Date: 2023-09-22
# Copyright: Vegard IT GmbH / Christoph Stoettner

# This script expects the mongo 3.6 databases in the mongo5 shares
# I recommend to move mongo-node-x to mongo5-node-x because copy needs too long


for i in 3.6 4.0 4.2 4.4 5.0; do
  podman pull docker.io/bitnami/mongodb:${i}
  container=mongo$(echo $i | tr -d .)
  for j in 0 1 2; do
    cd ${NFS_ROOT}/mongo5-node-$j
    podman run -dt --name ${container} -p 27017:27017 -v $(pwd)/data/db:/bitnami/mongodb/data/db:Z docker.io/bitnami/mongodb:${i}
    sleep 15
    # Update CompatibiltyVersion to next version
    podman exec -it ${container} mongo --host  --eval "db.adminCommand( { setFeatureCompatibilityVersion: '${i}' } )"
    if [ ${i} -eq "3.6" ] ; then
      # Remove replicaset definition from database local
      podman exec -it ${container} mongo --host  --eval "db.getSiblingDB('local').system.replset.remove()"
      podman exec -it ${container} mongo --host  --eval "use userprefs-service" --eval "db.dropDatabase()"
    # Stop mongodb databases
    podman exec -it ${container} mongo --host  --eval "db.getSiblingDB('admin').shutdownServer()"
    podman stop ${container}
    podman rm ${container}

Using Docker

#!/usr/bin/env bash
# Author: Christoph Stoettner
# Copyright: Vegard IT GmbH
# Date: 2023-09-21

# This script expects the MongoDB 3.6 databases in the mongo5 shares
# I recommend to move mongo-node-x to mongo5-node-x, because copy needs too long


for i in 3.6 4.0 4.2 4.4 5.0; do
  docker pull bitnami/mongodb:${i}
  container=mongo$(echo $i | tr -d .)

  for j in 0 1 2; do
    cd ${NFS_ROOT}/mongo5-node-$j || exit
    echo "[Start ${i}] - Start container ${container} in mongo5-node-${j}"
    docker run -d -t --name "${container}" -p 27017:27017 -v "$(pwd)/data/db":/bitnami/mongodb/data/db bitnami/mongodb:${i}
    sleep 30
    if [ ${i} == "3.6" ] ; then
      # Image 3.6 has no mongosh
    elif [ ${i} == "4.0" ]; then
      # Image 4.0 has no mongosh
    elif [ ${i} == "4.2" ]; then
      # Delete database userprefs-service
      echo "Delete database userprefs-service"
      docker exec "${container}" $CMD userprefs-service --quiet --eval 'db.dropDatabase(); quit();'
      echo "Delete ReplicaSet"
      docker exec "${container}" $CMD local --quiet --eval 'db.system.replset.deleteOne({"_id":"rs0"}); quit();'
      echo "Delete Transactions"
      docker exec "${container}" $CMD config --quiet --eval 'db.transactions.remove({}); quit();'
    # Update CompatibiltyVersion to next version
    until docker exec "${container}" $CMD --quiet --eval "db.adminCommand( { setFeatureCompatibilityVersion: '${i}' } )"; do
      echo "Update FeatureCompatibilityVersion to ${i}"
      echo "Remove transactions"
      docker exec "${container}" $CMD config --quiet --eval 'db.transactions.remove({}); quit();'
      sleep 10
    # Stop mongodb databases
    echo "Stop mongod"
    if [ ${i} == "3.6" ] || [ ${i} == "4.0" ]; then
      docker exec "${container}" $CMD --quiet --eval "db.getSiblingDB('admin').shutdownServer()"
      docker exec "${container}" $CMD admin --quiet --eval 'db.shutdownServer(); quit();'
    echo "Stop container ${container} in mongo5-node-${j}"
    docker stop "${container}"
    echo "Delete container"
    docker rm "${container}"
  echo "[Stop ${i}] - All datastores updated"

After running this script, just follow the documented migration path and install/update Kubernetes and Component Pack.

Just for comparison, this process with podman pull command (download the Bitnami images from Docker Hub) runs about 15 to 20 minutes and the 50 GB databases are migrated. After starting the new Mongo5 statefulset, the replicaSet config is automatically reapplied and working.

Friday, 1. Dec 2023 06:42


There was an error sending your comment, please try again.
Thank you!
Your comment has been submitted and will be published once it has been approved.

Your email address will not be published. Required fields are marked with *

Suggested Reading
Card image cap

Last week, I had three systems with issues displaying the Top Updates in the Orient Me. So I tried to find out which applications and containers are involved in generating the content for this view.

Created: Read in about 4 min
Card image cap

With HCL Connections 6.5 and later, we got the add-on HCL Connections Engagement Center (aka CEC, HCEC, ICEC or XCC) included in a normal HCL Connections deployment.

Created: Read in about 6 min
Card image cap

I had one Connections’ environment that I wanted to switch from OpenLDAP to Active Directory LDAP. The old OpenLDAP environment used LDAPS to connect, and so I assumed that the change was done quickly.

The first step was to make a copy of the tdisol folder I used for OpenLDAP and start changing the configuration files for the new LDAP server.

Created: Read in about 4 min