For HCL Connections 8 CR9, it is mandatory to update MongoDB to version 7. During my first migrations, I encountered some issues and would like to provide workarounds and additional troubleshooting tips to help others with this process.
The migration script expects the mounted mongo7-node-{0,1,2} shares in /mnt!
Understanding the Default Path Configuration
Check the script
, where the root folder is defined. If $NFS_ROOT is undefined, it is set to /mnt.
NFS_ROOT=${NFS_ROOT:-/mnt}
To use a different path, use export before running the migration command:
export NFS_SHARE=/pv-connections
./mongo_migration.sh
I’ve added a check for the MongoDB folder to the script. This is the snippet:
# Check if shares are mounted to the used path
if [ ! -d $NFS_ROOT/mongo7-node-0/data/db ] ; then
echo "No Mongo DB share in $NFS_ROOT"
exit 1
fi
Alternatively, you can use the adjusted version of the script that includes this check.
Dealing with Lock Files
Most commonly, we encounter .lock files in the MongoDB folders after scaling down the Mongo 5 containers. The migration script will fail if any of these files are present:
WiredTiger.lockmongod.lock
To delete .lock files from mongo7 shares, run:
find mongo7-node-* -iname \*.lock -exec rm {} \;
Troubleshooting
If you’re experiencing issues with the migration, here are some helpful troubleshooting steps:
- Run the script with
bash -xto see the commands being executed and all variables replaced:
root@hostname # bash -x mongo_migration.sh
++ date +%Y%m%d_%H%M%S
+ LOG_FILE=migration_20250408_094159.log
+ exec
++ tee -a migration_20250408_094159.log
+ NFS_ROOT=/pv-connections
+ REPL_SET=rs0
+ NETWORK_NAME=mongo5-network
+ HOST_PORT_BASE=27010
+ CONTAINER_PORT=27017
+ FCV_VERSIONS=(["6.0"]="bitnami/mongodb:6.0" ["7.0"]="bitnami/mongodb:7.0")
+ declare -A FCV_VERSIONS
+ declare -A container_map
+ docker network ls
+ grep -q mongo5-network
+ log_info 'Starting temporary MongoDB container to retrieve replica set info (MongoDB 5.0)'
+ echo '[INFO] Starting temporary MongoDB container to retrieve replica set info (MongoDB 5.0)'
[INFO] Starting temporary MongoDB container to retrieve replica set info (MongoDB 5.0)
+ docker run -dt --name mongo-get-host-name -p 27020:27017 -v /pv-connections/mongo7-node-0/data/db:/bitnami/mongodb/data/db:Z -e MONGODB_EXTRA_FLAGS=--replSet=rs0 bitnami/mongodb:5.0
e6f250b84d9906e1ab0193f3d417a093bc2a8cb78edbd27c21a8427f703f2527
+ wait_for_mongo mongo-get-host-name
+ log_info 'Waiting for MongoDB in mongo-get-host-name to be ready...'
+ echo '[INFO] Waiting for MongoDB in mongo-get-host-name to be ready...'
[INFO] Waiting for MongoDB in mongo-get-host-name to be ready...
+ docker exec mongo-get-host-name mongosh --host 127.0.0.1 --eval 'db.runCommand({ ping: 1 })'
- Check log files of the
mongo-get-host-namecontainer if the script shows no progress. In this example,WiredTiger.lockis in place, preventing the Bitnami container from mounting the database:
root@hostname # docker logs mongo-get-host-name
mongodb 07:40:30.52 INFO ==>
mongodb 07:40:30.52 INFO ==> Welcome to the Bitnami mongodb container
mongodb 07:40:30.52 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 07:40:30.52 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 07:40:30.52 INFO ==>
mongodb 07:40:30.53 INFO ==> ** Starting MongoDB setup **
mongodb 07:40:30.54 INFO ==> Validating settings in MONGODB_* env vars...
mongodb 07:40:30.59 INFO ==> Initializing MongoDB...
mongodb 07:40:30.64 INFO ==> Deploying MongoDB with persisted data...
mongodb 07:40:30.66 INFO ==> ** MongoDB setup finished! **
mongodb 07:40:30.67 INFO ==> ** Starting MongoDB **
{"t":{"$date":"2025-04-08T07:40:30.706+00:00"},"s":"I", "c":"CONTROL", "id":20698, "ctx":"-","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2025-04-08T07:40:30.706+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2025-04-08T07:40:30.708+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2025-04-08T07:40:30.708+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2025-04-08T07:40:30.708+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2025-04-08T07:40:30.709+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2025-04-08T07:40:30.709+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2025-04-08T07:40:30.709+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2025-04-08T07:40:30.709+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2025-04-08T07:40:30.710+00:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2025-04-08T07:40:30.710+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/bitnami/mongodb/data/db","architecture":"64-bit","host":"f6ad5f9552c2"}}
{"t":{"$date":"2025-04-08T07:40:30.710+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.24","gitVersion":"f034f0c51b3dffef4b8c9452d77ede9888f28f66","openSSLVersion":"OpenSSL 1.1.1w 11 Sep 2023","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian11","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2025-04-08T07:40:30.710+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 11 (bullseye)\"","version":"Kernel 4.18.0-553.45.1.el8_10.x86_64"}}}
{"t":{"$date":"2025-04-08T07:40:30.710+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/opt/bitnami/mongodb/conf/mongodb.conf","net":{"bindIp":"*","ipv6":false,"port":27017,"unixDomainSocket":{"enabled":true,"pathPrefix":"/opt/bitnami/mongodb/tmp"}},"processManagement":{"fork":false,"pidFilePath":"/opt/bitnami/mongodb/tmp/mongodb.pid"},"replication":{"replSet":"rs0"},"security":{"authorization":"disabled"},"setParameter":{"enableLocalhostAuthBypass":"true"},"storage":{"dbPath":"/bitnami/mongodb/data/db","directoryPerDB":false,"journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"logRotate":"reopen","path":"/opt/bitnami/mongodb/logs/mongodb.log","quiet":false,"verbosity":0}}}}
{"t":{"$date":"2025-04-08T07:40:30.713+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/bitnami/mongodb/data/db","storageEngine":"wiredTiger"}}
{"t":{"$date":"2025-04-08T07:40:30.713+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=23480M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2025-04-08T07:40:31.216+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":11,"message":"[1744098031:216706][1:0x7f9b2d09fc80], wiredtiger_open: __posix_file_lock, 393: /bitnami/mongodb/data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable"}}
{"t":{"$date":"2025-04-08T07:40:31.216+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":16,"message":"[1744098031:216813][1:0x7f9b2d09fc80], wiredtiger_open: __conn_single, 1807: WiredTiger database is already being managed by another process: Device or resource busy"}}
{"t":{"$date":"2025-04-08T07:40:31.217+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":11,"message":"[1744098031:217338][1:0x7f9b2d09fc80], wiredtiger_open: __posix_file_lock, 393: /bitnami/mongodb/data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable"}}
{"t":{"$date":"2025-04-08T07:40:31.217+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":16,"message":"[1744098031:217370][1:0x7f9b2d09fc80], wiredtiger_open: __conn_single, 1807: WiredTiger database is already being managed by another process: Device or resource busy"}}
{"t":{"$date":"2025-04-08T07:40:31.217+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":11,"message":"[1744098031:217845][1:0x7f9b2d09fc80], wiredtiger_open: __posix_file_lock, 393: /bitnami/mongodb/data/db/WiredTiger.lock: handle-lock: fcntl: Resource temporarily unavailable"}}
{"t":{"$date":"2025-04-08T07:40:31.217+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":16,"message":"[1744098031:217898][1:0x7f9b2d09fc80], wiredtiger_open: __conn_single, 1807: WiredTiger database is already being managed by another process: Device or resource busy"}}
{"t":{"$date":"2025-04-08T07:40:31.218+00:00"},"s":"W", "c":"STORAGE", "id":22347, "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2025-04-08T07:40:31.218+00:00"},"s":"F", "c":"STORAGE", "id":28595, "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"16: Device or resource busy"}}
{"t":{"$date":"2025-04-08T07:40:31.218+00:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":689}}
{"t":{"$date":"2025-04-08T07:40:31.218+00:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
Important: Clean Up After Failed Attempts
If the script does not terminate properly, don’t forget to clean up temporary containers to prevent resource conflicts in subsequent attempts:
docker rm mongo-get-host-name
Planning Your Migration
Before starting the migration process, it’s important to ensure that you have:
- Made a backup of your MongoDB data
- Properly scaled down your Mongo 5 containers
- Verified that your storage paths are correctly configured
- Allocated sufficient time for the migration process, as it can take longer for large databases
The migration to MongoDB 7 is critical for maintaining compatibility with HCL Connections 8 CR9, and while it may present some challenges, following these steps and troubleshooting tips should help ensure a smooth transition.
