The official documentation, “Migrating data from MongoDB 3 to 5”, wants to dump the MongoDB databases in 3.6 and then restore this data into the newly deployed MongoDB 5.
One issue with this process is that we can’t run the two MongoDB versions in parallel on Kubernetes because the provided helm charts and container for MongoDB 3.6 stop running after Kubernetes 1.21. On the other side, the helm chart providing MongoDB 5 can’t be installed on those old Kubernetes versions. So the process to update is:
Migration process
- Dump databases in MongoDB 3.6 (version delivered with Connections 7)
- Update Kubernetes to 1.25 or 1.27
- Restore MongoDB databases to version 5.0
So, you have to plan the process in advance because it is difficult to get the data when you forget something.
Update Kubernetes
The process of updating Kubernetes from 1.21 to 1.27 is quite a challenge because you have to jump from version to version: 1.21 → 1.22 → 1.23 → 1.24 → 1.25 → 1.26 → 1.27. Or you destroy your Kubernetes deployment and just create a new one with Kubernetes 1.27.
Create database dumps
I started dumping the databases and found that these 50 GB (it is a large environment) of MongoDB databases will need about three days until the dump is finished. We estimated the restoration to take three to four days. This results in a migration time of at least one week, and there is no migration possible of the data created in the databases during this week. Nobody can shut down their Connections’ environment for multiple days, so we required another solution.
A case at HCL wasn’t helpful at all, and in the end I started reading the MongoDB documentation and tested some options.
I learned that we can migrate the data in-place, but the process is updating in a row 3.6 → 4.0 → 4.2 → 4.4 → 5.0. The documentation recommends updating the database binaries, set the compatibility version, shutdown the database and going to the next version. The databases are stored in NFS shares, so in my opinion, the easiest way is to use some Docker containers and mount the database directory to these containers.
Another option would be to add new members to the replicaSet, but this would need some code adjustments in the pods. These changes I wanted to avoid and as I already wrote, this would need adjustments in the helm charts deploying MongoDB.
Get the MongoDB 3.6 data to the 5.0 shares
I’m not responsible for your data or provide support for this process!
I tested these steps in multiple environments and until now, everything works flawlessly, but use this information at your own risk!
My first try was copying the shares mongo-node-{0,1,2} to the new ones mongo5-node-{0,1,2}, but this means nearly 150 GB of data copied over NFS.
Copying the database data to a NFS requires some time, so my next step was moving the data from the mongo-node-{0,1,2} shares tomongo5-node-{0,1,2}. Moving data is way faster than copying them, but you should have a backup or snapshot.
Backup mongo-node shares
tar the data of MongoDB, this is faster than copying it.
cd /pv-connections
tar cvzf backup-$date +%Y-%m-%d-mongo36.tar.gz mongo-node-?
Move data on NFS share
cd /pv-connections
for i in 0 1 2
# Backup mongo5 folder
if [ -d mongo5-node-${i}]; then
mv mongo5-node-${i} mongo5-node-${i}-old
fi
mv mongo-node-${i} mongo5-node-${i}
done
Migrate databases
I used podman installed on the NFS Server to migrate the databases, but you can mount the shares to any other machine with Docker or podman.
Steps
We have to start with MongoDB 3.6 because Connections 7 has set CompatibiltyVersion = 3.4!
Pull image
podman pull docker.io/bitnami/mongodb:3.6
Run container
This command expects that you are in the /pv-connections/mongo5-node-0 folder.
podman run -dt --name mongo36 -p 27017:27017 -v $(pwd)/data/db:/bitnami/mongodb/data/db:Z docker.io/bitnami/mongodb:3.6
Remove the replicaset (only for 3.6)
The new statefulSet in MongoDB 5 uses new hostnames, so it is necessary to remove the old hostname from the replicaset.
I run the following commands via docker or podman exec, but you can use mongosh too and run the commands one after another in shell. I wanted to automate the process, this is easier with single commands via podman exec.
podman exec -it mongo36 mongo --host 127.0.0.1 --eval "db.getSiblingDB('local').system.replset.remove()"
Drop userprefs-service database (only for 3.6)
This database is no longer used and can be deleted.
podman exec -it mongo36 mongo --host 127.0.0.1 --eval "use userprefs-service" --eval "db.dropDatabase()"
Update CompatibiltyVersion
podman exec -it mongo36 mongo --host 127.0.0.1 --eval "db.adminCommand( { setFeatureCompatibilityVersion: '3.6' } )"
Shutdown database Server
podman exec -it mongo36 mongo --host 127.0.0.1 --eval "db.getSiblingDB('admin').shutdownServer()"
Stop the container and remove it
podman stop mongo36
podman rm mongo36
That’s almost it, but you have to do the same process in all 3 mongo5-node-{0,1,2} shares and for the other MongoDB versions.
Use a Shell Script for the migration
This script expects that you already moved the old mongo-node folders to mongo5-node.
Using podman
#!/usr/bin/env bash
# Author: Christoph Stoettner
# Date: 2023-09-22
# Copyright: Vegard IT GmbH / Christoph Stoettner
# This script expects the mongo 3.6 databases in the mongo5 shares
# I recommend to move mongo-node-x to mongo5-node-x because copy needs too long
NFS_ROOT=/pv-connections
for i in 3.6 4.0 4.2 4.4 5.0; do
podman pull docker.io/bitnami/mongodb:${i}
container=mongo$(echo $i | tr -d .)
for j in 0 1 2; do
cd ${NFS_ROOT}/mongo5-node-$j
podman run -dt --name ${container} -p 27017:27017 -v $(pwd)/data/db:/bitnami/mongodb/data/db:Z docker.io/bitnami/mongodb:${i}
sleep 15
# Update CompatibiltyVersion to next version
podman exec -it ${container} mongo --host 127.0.0.1 --eval "db.adminCommand( { setFeatureCompatibilityVersion: '${i}' } )"
if [ ${i} -eq "3.6" ] ; then
# Remove replicaset definition from database local
podman exec -it ${container} mongo --host 127.0.0.1 --eval "db.getSiblingDB('local').system.replset.remove()"
podman exec -it ${container} mongo --host 127.0.0.1 --eval "use userprefs-service" --eval "db.dropDatabase()"
fi
# Stop mongodb databases
podman exec -it ${container} mongo --host 127.0.0.1 --eval "db.getSiblingDB('admin').shutdownServer()"
podman stop ${container}
podman rm ${container}
done
done
Using Docker
#!/usr/bin/env bash
# Author: Christoph Stoettner
# Copyright: Vegard IT GmbH
# Date: 2023-09-21
# This script expects the MongoDB 3.6 databases in the mongo5 shares
# I recommend to move mongo-node-x to mongo5-node-x, because copy needs too long
NFS_ROOT=/pv-connections
for i in 3.6 4.0 4.2 4.4 5.0; do
docker pull bitnami/mongodb:${i}
container=mongo$(echo $i | tr -d .)
for j in 0 1 2; do
cd ${NFS_ROOT}/mongo5-node-$j || exit
echo "[Start ${i}] - Start container ${container} in mongo5-node-${j}"
docker run -d -t --name "${container}" -p 27017:27017 -v "$(pwd)/data/db":/bitnami/mongodb/data/db bitnami/mongodb:${i}
sleep 30
CMD=mongosh
if [ ${i} == "3.6" ] ; then
# Image 3.6 has no mongosh
CMD=mongo
elif [ ${i} == "4.0" ]; then
# Image 4.0 has no mongosh
CMD=mongo
elif [ ${i} == "4.2" ]; then
# Delete database userprefs-service
echo "Delete database userprefs-service"
docker exec "${container}" $CMD userprefs-service --quiet --eval 'db.dropDatabase(); quit();'
echo "Delete ReplicaSet"
docker exec "${container}" $CMD local --quiet --eval 'db.system.replset.deleteOne({"_id":"rs0"}); quit();'
echo "Delete Transactions"
docker exec "${container}" $CMD config --quiet --eval 'db.transactions.remove({}); quit();'
fi
# Update CompatibiltyVersion to next version
until docker exec "${container}" $CMD --quiet --eval "db.adminCommand( { setFeatureCompatibilityVersion: '${i}' } )"; do
echo "Update FeatureCompatibilityVersion to ${i}"
echo "Remove transactions"
docker exec "${container}" $CMD config --quiet --eval 'db.transactions.remove({}); quit();'
sleep 10
done
# Stop mongodb databases
echo "Stop mongod"
if [ ${i} == "3.6" ] || [ ${i} == "4.0" ]; then
docker exec "${container}" $CMD --quiet --eval "db.getSiblingDB('admin').shutdownServer()"
else
docker exec "${container}" $CMD admin --quiet --eval 'db.shutdownServer(); quit();'
fi
echo "Stop container ${container} in mongo5-node-${j}"
docker stop "${container}"
echo "Delete container"
docker rm "${container}"
done
echo "[Stop ${i}] - All datastores updated"
done
After running this script, just follow the documented migration path and install/update Kubernetes and Component Pack.
Just for comparison, this process with podman pull command (download the Bitnami images from Docker Hub) runs about 15 to 20 minutes and the 50 GB databases are migrated. After starting the new Mongo5 statefulset, the replicaSet config is automatically reapplied and working.
Great!