Videostreaming in HCL Connections

In late 2021 I had an HCL Connections environment starting swapping, because the AppCluster used more than 30 GB of memory.

The system has

• two nodes
• is installed with the medium-sized deployment option
• About 7500 users with a high adoption rate, because Connections is also used as intranet

What happened?

Log4j CVE-2021-22448 couldn’t be patched immediately for the video streaming platform and some teams shared blog posts with video messages on Connections.

Analysis

The amount of memory an application server is using is not limited through the Java heap size. File caching happens on top of Java heap.

So I found that each user who was opening a video in the file viewer added about the file size to the server memory. Ten users watching a 500 MB video increased the memory usage with 5 GB.

The AppCluster members used about 8 GB of memory after startup, but the videos were linked on the start page and so the amount grew to 30-35 GB within minutes and the servers started swapping.

Quick check in the HTTPServer access.log showed that a lot of mp4 files were accessed since restart and the video length of an hour did the rest.

I could reproduce with opening multiple browser tabs showing videos, the memory usage increased immediately. So one user can slow down a Connections’ environment with opening large videos in parallel, until the server starts swapping or even crashes.

Step to reproduce

• Upload a large video to your personal library in Connections
• check memory usage of AppsCluster (or FilesCluster
• Open the video in files
• check memory usage of AppsCluster (or FilesCluster)
• Open the same video in multiple tabs and check the memory consumption of the application server

Workaround / Solution

My first idea was to disable video preview at all, but there is no option available.

Gatekeeper has a setting FILEVIEWER_PREVIEW_VIDEOJS, default set to true, but I couldn’t find out what changes when you disable it.

file-preview-config.xml mentions mp4, but removing it has no effect in the streaming of mp4 in files viewer.

mime.types with onWebopen only works when inlineDownload is enabled in files-config.xml.

I opened a case at HCL Support and asked if there is some undocumented option to disable video preview. The first response was enabling “Download through IHS” , which always was recommended for performance in the Tuning guide .

Download through IHS

The system with the swapping issue, had some issues with NFS access rights in the past, so file download through IHS was not active, but I gave it a try again.

Requirements

For downloads through IHS the shared directory needs to be mounted on the web server.

https://help.hcltechsw.com/connections/v7/admin/install/t_install_post_files_downloads.html

Possible problems

• wrong user (or root user) → use setfacl and add the IHS user to the upload path of files
• DMZ → firewall exception or additional web server on the WebSphere node (proxy pass from DMZ)
• mixed operating systems (I can’t get this working when WebSphere runs on Windows and IHS on Linux) -> possible workaround is mod_rewrite and rewriting \ and /

Copy Apache module

Like in the part about uploading through IHS server, the documentation is not accurate. The HTTPServer in all my environments (installed manually or with the connections-automation project ), is 64-bit! I used /opt/HCL/Connections/xkit/ihs/mod_ibm_local_redirect/linux_x86_64_ap2/mod_ibm_local_redirect.so and copied it to /opt/IBM/HTTPServer/modules.

httpd.conf

LoadModule ibm_local_redirect_module modules/mod_ibm_local_redirect.so

Alias /files_content /opt/IBM/SharedArea/files/upload/

<Directory "/opt/IBM/SharedArea/files">
 Order Deny,Allow
 Deny from all
 Allow from env=REDIRECT_FILES_CONTENT
</Directory>

<Location /files>
 IBMLocalRedirect On
 IBMLocalRedirectKeepHeaders X-LConn-Auth,Cache-Control,Content-Type,Content-Disposition,Last-Modified,ETag,Content-Language,Set-Cookie,Title,X-UA-Compatible
 SetEnv FILES_CONTENT true
</Location>

files-config.xml (extended to 1 GB max upload size)

<download>
  <modIBMLocalRedirect enabled="true" hrefPathPrefix="/files_content" />
</download>
<file>
 <media maximumSizeInKb="1024000"/> <!-- Allow 1 GB uploads -->
</file>
<api>
  <simpleDownloadAPI maximumSizeInKb="10240"></simpleUploadAPI> <!-- WebSphere is used for 10 MB Downloads -->
</api>

Restarting the system and the memory usage of the AppsCluster members stayed in a range of the configured java heap size, even when I opened multiple large videos in parallel.

Setting the WebSphere Application Server WebContainer to synchronous mode

The documentation mentions, that you shall enable synchronous mode for the application server hosting the files application. This helps a to prevent the server from using all memory, but it still uses more, and it doesn’t free up used memory during my tests.

Alternative solution with mod_rewrite

I discussed this issue with development through a product case at HCL Support, and finally I got following suggestion to disable preview:

Ensure to enable the rewrite module. If the following line of text is commented out, uncomment it. If the statement is not present, add it.

LoadModule rewrite_module modules/mod_rewrite.so

Add the following:

# Block viewer from previewing files
RewriteCond %{QUERY_STRING} ^.*(downloadType=view).* [NC]
RewriteRule ^(.*)$ - [F,L]

Make sure that the configuration lines are in a global context or in each virtual host, depending on your setup.

I tried this in my demo environment:

• Video Preview in Files shows “No preview available” and I can download the file

• Embedding the video with Embed uploaded videos to IBM Connections blog post or wiki page is still working
  • embedded videos do not autoplay and are not that problematic in my eyes
• image and other previews are working as expected, so a perfect workaround to disable video previews

Update 2022-03-09 optimized rewrite rule

• No need to define lengthy RegExp with wild-card start and end if you only care about the match in the middle
• No need to define capture groups if you don’t use backreferences
• [F|forbidden] RewriteRule Flags implies L

# Block viewer from previewing files
RewriteCond %{QUERY_STRING} downloadType=view [NC]
RewriteRule .* - [F]

Thanks @KroegerBen for the optimization.

Summary

Best solution to prevent large memory consumption is the usage of the download / rewrite module in the HTTP Server. Synchronous mode helped a bit, but I would disable the video preview with mod_rewrite when there is no possibility to use download through IHS.