Videostreaming in HCL Connections

by Christoph Stoettner
Read in about 5 min · 1014 words

Video Streaming

Photo by Nevels Media | Unsplash

In late 2021 I had an HCL Connections environment starting swapping, because the AppCluster used more than 30 GB of memory.

The system has

  • two nodes
  • is installed with the medium-sized deployment option
  • About 7500 users with a high adoption rate, because Connections is also used as intranet

What happened?

Log4j CVE-2021-22448 couldn’t be patched immediately for the video streaming platform and some teams shared blog posts with video messages on Connections.

Analysis

The amount of memory an application server is using is not limited through the Java heap size. File caching happens on top of Java heap.

So I found that each user who was opening a video in the file viewer added about the file size to the server memory. Ten users watching a 500 MB video increased the memory usage with 5 GB.

The AppCluster members used about 8 GB of memory after startup, but the videos were linked on the start page and so the amount grew to 30-35 GB within minutes and the servers started swapping.

Quick check in the HTTPServer access.log showed that a lot of mp4 files were accessed since restart and the video length of an hour did the rest.

I could reproduce with opening multiple browser tabs showing videos, the memory usage increased immediately. So one user can slow down a Connections’ environment with opening large videos in parallel, until the server starts swapping or even crashes.

htop displaying the normal memory amount for AppsCluster_server1

htop displaying the memory amount for AppsCluster_server1 when 1 video is open

Step to reproduce

  • Upload a large video to your personal library in Connections
  • check memory usage of AppsCluster (or FilesCluster
  • Open the video in files
  • check memory usage of AppsCluster (or FilesCluster)
  • Open the same video in multiple tabs and check the memory consumption of the application server

Workaround / Solution

My first idea was to disable video preview at all, but there is no option available.

Gatekeeper has a setting FILEVIEWER_PREVIEW_VIDEOJS, default set to true, but I couldn’t find out what changes when you disable it.

file-preview-config.xml mentions mp4, but removing it has no effect in the streaming of mp4 in files viewer.

mime.types with onWebopen only works when inlineDownload is enabled in files-config.xml.

I opened a case at HCL Support and asked if there is some undocumented option to disable video preview. The first response was enabling “Download through IHS” , which always was recommended for performance in the Tuning guide .

Download through IHS

The system with the swapping issue, had some issues with NFS access rights in the past, so file download through IHS was not active, but I gave it a try again.

Requirements

For downloads through IHS the shared directory needs to be mounted on the web server.

https://help.hcltechsw.com/connections/v7/admin/install/t_install_post_files_downloads.html

Possible problems

  • wrong user (or root user) → use setfacl and add the IHS user to the upload path of files
  • DMZ → firewall exception or additional web server on the WebSphere node (proxy pass from DMZ)
  • mixed operating systems (I can’t get this working when WebSphere runs on Windows and IHS on Linux) -> possible workaround is mod_rewrite and rewriting \ and /

Copy Apache module

Like in the part about uploading through IHS server, the documentation is not accurate. The HTTPServer in all my environments (installed manually or with the connections-automation project ), is 64-bit! I used /opt/HCL/Connections/xkit/ihs/mod_ibm_local_redirect/linux_x86_64_ap2/mod_ibm_local_redirect.so and copied it to /opt/IBM/HTTPServer/modules.

httpd.conf

LoadModule ibm_local_redirect_module modules/mod_ibm_local_redirect.so

Alias /files_content /opt/IBM/SharedArea/files/upload/

<Directory "/opt/IBM/SharedArea/files">
 Order Deny,Allow
 Deny from all
 Allow from env=REDIRECT_FILES_CONTENT
</Directory>

<Location /files>
 IBMLocalRedirect On
 IBMLocalRedirectKeepHeaders X-LConn-Auth,Cache-Control,Content-Type,Content-Disposition,Last-Modified,ETag,Content-Language,Set-Cookie,Title,X-UA-Compatible
 SetEnv FILES_CONTENT true
</Location>

files-config.xml (extended to 1 GB max upload size)

<download>
  <modIBMLocalRedirect enabled="true" hrefPathPrefix="/files_content" />
</download>
<file>
 <media maximumSizeInKb="1024000"/> <!-- Allow 1 GB uploads -->
</file>
<api>
  <simpleDownloadAPI maximumSizeInKb="10240"></simpleUploadAPI> <!-- WebSphere is used for 10 MB Downloads -->
</api>

Restarting the system and the memory usage of the AppsCluster members stayed in a range of the configured java heap size, even when I opened multiple large videos in parallel.

Setting the WebSphere Application Server WebContainer to synchronous mode

The documentation mentions, that you shall enable synchronous mode for the application server hosting the files application. This helps a to prevent the server from using all memory, but it still uses more, and it doesn’t free up used memory during my tests.

htop displaying the memory amount for AppsCluster_server1 when 5 video is open and synchronous mode is enabled

Alternative solution with mod_rewrite

I discussed this issue with development through a product case at HCL Support, and finally I got following suggestion to disable preview:

Ensure to enable the rewrite module. If the following line of text is commented out, uncomment it. If the statement is not present, add it.

LoadModule rewrite_module modules/mod_rewrite.so

Add the following:

# Block viewer from previewing files
RewriteCond %{QUERY_STRING} ^.*(downloadType=view).* [NC]
RewriteRule ^(.*)$ - [F,L]

Make sure that the configuration lines are in a global context or in each virtual host, depending on your setup.

HCL Support

I tried this in my demo environment:

  • Video Preview in Files shows “No preview available” and I can download the file

Update 2022-03-09 optimized rewrite rule

  • No need to define lengthy RegExp with wild-card start and end if you only care about the match in the middle
  • No need to define capture groups if you don’t use backreferences
  • [F|forbidden] RewriteRule Flags implies L
# Block viewer from previewing files
RewriteCond %{QUERY_STRING} downloadType=view [NC]
RewriteRule .* - [F]

Thanks @KroegerBen for the optimization.

Summary

Best solution to prevent large memory consumption is the usage of the download / rewrite module in the HTTP Server. Synchronous mode helped a bit, but I would disable the video preview with mod_rewrite when there is no possibility to use download through IHS.

Edited on 09.03.2022
Author
Add a comment
Error
There was an error sending your comment, please try again.
Thank you!
Your comment has been submitted and will be published once it has been approved.

Your email address will not be published. Required fields are marked with *

Suggested Reading
Aaron Burden: Fountain pen and a notebook

IBM Docs Viewer can open source files with syntax highlighting. This feature is default disabled, but sometimes very useful.

You need to enable it with IBM Connections Gatekeeper.

Read in about 1 min
Card image cap

Here for example embedding a video to a blog post. Prerequist is that you’ve no fear to change some html source.

Read in about 1 min
Card image cap

So up to IBM Connections 6.0 everything was allowed until it was not excluded in one of the blocklist files. This files are stored within the Deployment Manager profile/config/cells/<cellname>/LotusConnections-config/extern. Now with Connections 6.0CR1 everything is forbidden, until it is enabled in the allowlist. This concept is rolled out for widgets (homepage and communities) and active content. Active content means HTML content too. So everything you or your users add to Connections (blog-posts, wiki pages) gets filtered during the save procedure. This removes all HTML tags and attributes which are not explicitly allowed!

Read in about 5 min