Update 2021-12-13 2021-12-15
- Elasticsearch: Apache Log4j2 Remote Code Execution (RCE) Vulnerability - CVE-2021-44228 - ESA-2021-31
- HCL: CVE-2021-44228 : Security Advisory
- IBM: Security Bulletin: Vulnerability in Apache Log4j affects WebSphere Application Server (CVE-2021-44228)
- Security Bulletin: HCL Connections Security Update for Apache Log4j 2 Vulnerability (CVE-2021-44228)
- CVE-2021-45046: It was found that the fix to address CVE-2021-44228 in Apache Log4j 2.15.0 was incomplete in certain non-default configurations.
So there is a fix for kc.war
which updates the log4j
2.8 to 2.15, Elasticsearch in Component Pack has log4j 2.8 and 2.11 included but is not vulnerable because of additional security settings.
There are additional security layers in the Component Pack Elasticsearch, like SearchGuard , but the advisory for Elasticsearch tells clearly you should update.
Original post
Yesterday several security advisories arrived in my Inbox and people were worried about a 0-day vulnerability in Apache Log4j .
I read a lot during the last 24 hours and searched for log4j
versions within HCL Connections. I wanted to write about some of these commands already since weeks, so I use the awareness to show you some fast options to scan all packages in container images, file system and registries. For me one of the hardest points was to find out, if the software is using log4j
and which version.
As far as I know, only the Help.ear
has a log4j
2.3 included, so you have two options, stop it and wait for a fix, or add the JVM option to your cluster remove the class from the jar.
In my test installation I couldn’t find it, so maybe an add-on installed it in other environments.
So the used version log4j-2.3
is not secured with the JVM option log4j2.formatMsgNoLookups
! You have to stop the Help application until a fix is available, or remove the JndiLookup.class
from the jar
file.
Finally, HCL released a Security Advisory with workarounds.
Mitigation: In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:
zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
But without such advisory, how can we detect if our systems uses the vulnerable library?
HCL License files
When you want to know if HCL Connections has log4j
included, you check the license files.
Within the Connections’ installation folder you find the HCL and non-HCL licenses of the installed products. So we can grep
for log4j
there:
cd /opt/HCL/Connections/lafiles
grep log4j *
notices:log4j.settings_1.0.0.20200928-1215.jar
notices:log4j-1.2.11.jar The Apache Software Foundation
...
So in notices
we see the licenses and version numbers of all Open Source libraries of Connections.
Linux find
Next option is to search for the jar files within the file system. In my case, I started on the WebSphere Application server and scanned the installedApps
folder.
[root@cnx7-was ~]# cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/ConnectionsCell/
[root@cnx7-was ConnectionsCell]# find . -iname \*log4j\* | grep jar | sort
./Activities.ear/oawebui.war/WEB-INF/lib/log4j-1.2.11.jar
./Blogs.ear/blogs.war/WEB-INF/lib/log4j-1.2.11.jar
...
File system scanning like here is easy, when you just want to find a library or the used version.
Trivy
A good option for scanning all kinds of vulnerabilities in the file system and within containers is Aquasecurity Trivy licensed under the Apache 2.0 License .
Container
For HCL Connections, we get all container images of Component Pack within the download package. Here for example, I extracted it to my local file system:
unzip componentpack-7.0.0.2.zip
cd microservices_connections/hybridcloud/images/
mkdir logs
for i in $(ls) ; do
trivy i -i ${i} -o logs/${i%.tar}.log
done
This command creates a log-file for each image in the directory and lists all used libraries and their known vulnerabilities.
So now we can grep for log4j
and will find the Elasticsearch images, which uses log4j-2.11.1
.
I’m not sure if a user can trigger something in Componentpack - Elasticsearch, but I edited the statefulsets es-master-7
, es-data-7
and the deployment es-client-7
. I just added the mentioned JVM option and restarted the pods.
kubectl edit statefulset es-master-7
kubectl edit statefulset es-data-7
kubectl edit deployment es-client-7
Each of the definitions shows ES_JAVA_OPTS
twice:
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m
- in both occurrences add
-Dlog4j2.formatMsgNoLookups=True
- name: ES_JAVA_OPTS
value: -Xms512m -Xmx512m -Dlog4j2.formatMsgNoLookups=True
- and restart the pods:
kubectl rollout restart statefulsets es-data-7 es-master-7
kubectl rollout restart deployment es-client-7
After this check if all applications in Connections are still working, or restart them too.
Files System
We can use trivy to scan the ear files:
cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/ConnectionsCell/
mkdir ~/logs
for i in $(ls) ; do
trivy fs --list-all-pkgs -o ~/logs/${i%%.ear}.log $i
done
This creates a directory named logs
in our home directory, and we can read over all files, or just grep for terms you’re interested in.
You will see, we get the library name, installed version and if there is a version available where the issue is fixed.
I don’t post any found libraries here, there is a lot to observe and not each critical vulnerability is exploitable in all environments. So you have to read the follow-ups under title and find out if your environment is vulnerable.
Registry
Trivy can scan images in registries too, I do more like the scanning in the file system, because there I don’t need to tame certificates and download the image before.
Summary
I like using trivy
, and I learned a lot about containers and libraries since I use it. You can add it to your CI/CD, and development workflow, this can help update libraries more frequently.
Great post!! Thank you.