The last days I analyzed an issue, that file uploads to HCL Connections via IBM HTTPServer stopped working on a fresh installed 6.5CR1.
Today I configured a Connections 7 and tried with it. I think that the official documentation is old in some important parts for the upload configuration.
First of all my IBM HTTPServer 8.5.5.18 is not 32-bit like the documentation tells us:
The IBM HTTP Server process is 32-bits in both cases and requires 32-bit modules.
When I check my installed HTTP Server (which was installed with the official Ansible scripts), I see this:
[root@cnx7 upload]# /opt/IBM/HTTPServer/bin/apachectl -V
Server version: IBM_HTTP_Server/8.5.5.18 (Unix)
Apache version: 2.2.8 (with additional fixes)
Server built: Jun 24 2020 16:56:21
Build level: IHS85/webIHS2025.01
Server's Module Magic Number: 20051115:23
Server loaded: APR 1.2.12, APR-Util 1.2.12
Compiled using: APR 1.2.12, APR-Util 1.2.12
Architecture: 64-bit
Server MPM: Worker
threaded: yes (fixed thread count)
forked: yes (variable process count)
So when you use the 32-Bit one, you get following message during HTTPServer start:
[Mon Apr 26 07:57:52 2021] [warn] module rewrite_module is already loaded, skipping
httpd: Syntax error on line 1059 of /opt/IBM/HTTPServer/conf/httpd.conf: Syntax error on line 2 of /opt/IBM/HTTPServer/conf/httpd-upload.conf: Cannot load modules/mod_ibm_u server: /opt/IBM/HTTPServer/modules/mod_ibm_upload.so: wrong ELF class: ELFCLASS32
The Ansible roles do not configure IHS Upload, this needs to be done manually.
So now we know that we need a 64 bit module and copy it to IHS.
[root@cnx7 upload]# ls -al /opt/HCL/Connections/ihs/mod_ibm_upload/
total 12
drwxr-xr-x. 12 root root 273 Apr 26 14:28 .
drwxr-xr-x. 4 root root 58 Apr 22 10:37 ..
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.aix.ppc32
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.aix.ppc64
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.amd64
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.ia32
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.ppc32
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.ppc64
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.s390
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.s390_64
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.linux.x86_64
drwxr-xr-x. 2 root root 31 Apr 22 10:37 ihs.win.ia32
-rw-r--r--. 1 root root 2101 Nov 23 14:54 MD5
-rw-r--r--. 1 root root 100 Nov 23 14:54 README
-rw-r--r--. 1 root root 151 Nov 23 14:54 README.txt
Hmm, I have no idea why there are two versions for 64-bit Linux!
Size and checksum are different
[root@cnx7 upload]# sha256sum /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.amd64/mod_ibm_upload.so
13a23a23bbd19d34c99e0cdefea30f09e0ecf14a22c5bc3399b6f7f9908fa7ec /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.amd64/mod_ibm_upload.so
[root@cnx7 upload]# sha256sum /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.x86_64/mod_ibm_upload.so
0057b7e0ca816891f98f715d62cb41cdccaa5cfee5718a3465b2f005e95aed24 /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.x86_64/mod_ibm_upload.so
[root@cnx7 upload]# ls -al /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.x86_64/mod_ibm_upload.so
-rw-r--r--. 1 root root 36360 Nov 23 14:54 /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.x86_64/mod_ibm_upload.so
[root@cnx7 upload]# ls -al /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.amd64/mod_ibm_upload.so
-rw-r--r--. 1 root root 36520 Nov 23 14:54 /opt/HCL/Connections/ihs/mod_ibm_upload/ihs.linux.amd64/mod_ibm_upload.so
Ok, so now everything is in place, I used the x86_64 version of the module and copied it to /opt/IBM/HTTPServer/modules, then I copied the config from the documentation for the files application to my virtualhost and set the parameters for uploads in
files-config.xml
...
<file>
...
<media maximumSizeInKb="2048000"/>
...
</file>
<api>
...
<simpleUploadAPI maximumSizeInKb="128000">
<organization estimatedBytesInSeconds="2097152" id="admin_replace" maxConcurrenceRequests="50" maximumSizeInKb="128000"/>
</simpleUploadAPI>
...
</api>
...
<upload>
<modIBMUpload enabled="true"/>
</upload>
Set the maximum upload size for single files
Use the simpleUploadAPI (direct through Websphere) until 128MB
Same for the organisation 128MB uses WebSphere
Upload through IHS enabled (default)
After Node synchronization and restart of HTTP and Applicationserver I found following:
Uploads up to 128 MB worked
Uploads from 128 MB to 500 MB were uploaded through IHS (I found different user in the files uploads)
Uploads larger 500 MB stuck at around 500 MB and did not finish.
The IHS error_log showed following:
[Mon Apr 26 13:34:38 2021] [error] [client 10.0.11.1:44186] Error: resumable upload disabled but Content-Range header used, referer: https://cnx7.stoeps.internal/files/app
I searched for more configuration settings, more details to the upload module, but couldn’t find anything.
I was pretty sure that there is a configuration option to enable the resumable upload, but couldn’t find it on any IBM or HCL page.
So I tried with strings and Ghidra :
strings modules/mod_ibm_upload.so | grep resum
...
resumable upload disabled but Content-Range header used
Non-resumable upload has been canceled.
part-complete;resumable;authenticate
On|Off - Enable/disable (default) resumable upload
resumable_disabled
complete;resumable;forward
complete;resumable;phase2
cmd_ibmuploadactivateresumable
So there is an option to configure resumable uploads.
I opened the mod_ibm_upload.so with ghidra and there I found:
There are configuration parameter to enable resumable uploads and a size limit where only one part is uploaded. So in my opinion the IBMUploadMinimumPartSize is default 500MB and IBMUploadActivateResumable is Off by default. I never had to use this in 6.0, so I’m pretty sure one of these defaults changed in the last versions (6.5 or 6.5CR1), or the files application forces it now.
So I changed the config in httpd.conf to:
<Location "/ihs/files">
IBMUploadHandler On
SetHandler ibm_upload_handler
IBMUploadBaseStore "/opt/IBM/SharedArea/files/upload/files"
IBMUploadMethods POST,PUT
IBMUploadURLPrefix /ihs
IBMUploadActivateResumable On
</Location>
After restarting the IBM HTTPServer, my files > 500MB uploaded without any issue!
The upload module can be used only when WebSphere Application Server is not run as user root. Otherwise, file permissions prevent the application server and the IBM HTTP Server from exchanging files. For security reasons, do not run the application server as root, but if that is not an option, then large files still can be uploaded. If WebSphere Application Server is run as root, consider making the following change to the maximumSizeInKb property for thesimpleUploadAPI file.
I think running WebSphere as a non-root user is best practise, but for example the Ansible scripts install them for root in the moment. So when you use root for WebSphere, have a look at setfacl and add the http server user to your files/upload directory, so IHS is allowed to upload to that folders.
