Log4j how to find out if an application has it included

by Christoph Stoettner
Read in about 5 min · (words)

orange and white traffic cone

Photo by Lucian Alexe | Unsplash

Update 2021-12-13 2021-12-15

So there is a fix for kc.war which updates the log4j 2.8 to 2.15, Elasticsearch in Component Pack has log4j 2.8 and 2.11 included but is not vulnerable because of additional security settings.

There are additional security layers in the Component Pack Elasticsearch, like SearchGuard , but the advisory for Elasticsearch tells clearly you should update.

Original post

Yesterday several security advisories arrived in my Inbox and people were worried about a 0-day vulnerability in Apache Log4j .

I read a lot during the last 24 hours and searched for log4j versions within HCL Connections. I wanted to write about some of these commands already since weeks, so I use the awareness to show you some fast options to scan all packages in container images, file system and registries. For me one of the hardest points was to find out, if the software is using log4j and which version.

So the used version log4j-2.3 is not secured with the JVM option log4j2.formatMsgNoLookups! You have to stop the Help application until a fix is available, or remove the JndiLookup.class from the jar file.

Finally, HCL released a Security Advisory with workarounds.

Mitigation: In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

But without such advisory, how can we detect if our systems uses the vulnerable library?

HCL License files

When you want to know if HCL Connections has log4j included, you check the license files.

Within the Connections' installation folder you find the HCL and non-HCL licenses of the installed products. So we can grep for log4j there:

cd /opt/HCL/Connections/lafiles
grep log4j *
notices:log4j.settings_1.0.0.20200928-1215.jar	
notices:log4j-1.2.11.jar	The Apache Software Foundation
...

So in notices we see the licenses and version numbers of all Open Source libraries of Connections.

Linux find

Next option is to search for the jar files within the file system. In my case, I started on the WebSphere Application server and scanned the installedApps folder.

[root@cnx7-was ~]# cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/ConnectionsCell/
[root@cnx7-was ConnectionsCell]# find . -iname \*log4j\* | grep jar | sort
./Activities.ear/oawebui.war/WEB-INF/lib/log4j-1.2.11.jar
./Blogs.ear/blogs.war/WEB-INF/lib/log4j-1.2.11.jar
...

File system scanning like here is easy, when you just want to find a library or the used version.

Trivy

A good option for scanning all kinds of vulnerabilities in the file system and within containers is Aquasecurity Trivy licensed under the Apache 2.0 License .

Container

For HCL Connections, we get all container images of Component Pack within the download package. Here for example, I extracted it to my local file system:

unzip componentpack-7.0.0.2.zip
cd microservices_connections/hybridcloud/images/

mkdir logs

for i in $(ls) ; do
  trivy i -i ${i} -o logs/${i%.tar}.log
done

This command creates a log-file for each image in the directory and lists all used libraries and their known vulnerabilities.

So now we can grep for log4j and will find the Elasticsearch images, which uses log4j-2.11.1.

I’m not sure if a user can trigger something in Componentpack - Elasticsearch, but I edited the statefulsets es-master-7, es-data-7 and the deployment es-client-7. I just added the mentioned JVM option and restarted the pods.

kubectl edit statefulset es-master-7
kubectl edit statefulset es-data-7
kubectl edit deployment es-client-7

Each of the definitions shows ES_JAVA_OPTS twice:

- name: ES_JAVA_OPTS
  value: -Xms512m -Xmx512m
  • in both occurrences add -Dlog4j2.formatMsgNoLookups=True
- name: ES_JAVA_OPTS
  value: -Xms512m -Xmx512m -Dlog4j2.formatMsgNoLookups=True
  • and restart the pods:
kubectl rollout restart statefulsets es-data-7 es-master-7
kubectl rollout restart deployment es-client-7

After this check if all applications in Connections are still working, or restart them too.

Files System

We can use trivy to scan the ear files:

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/installedApps/ConnectionsCell/
mkdir ~/logs
for i in $(ls) ; do 
  trivy fs --list-all-pkgs -o ~/logs/${i%%.ear}.log $i
done

This creates a directory named logs in our home directory, and we can read over all files, or just grep for terms you’re interested in.

You will see, we get the library name, installed version and if there is a version available where the issue is fixed.

I don’t post any found libraries here, there is a lot to observe and not each critical vulnerability is exploitable in all environments. So you have to read the follow-ups under title and find out if your environment is vulnerable.

Registry

Trivy can scan images in registries too, I do more like the scanning in the file system, because there I don’t need to tame certificates and download the image before.

Summary

I like using trivy, and I learned a lot about containers and libraries since I use it. You can add it to your CI/CD, and development workflow, this can help update libraries more frequently.

Resources

Edited on 15.12.2021
Author
Yancy Lent
Saturday, 11. Dec 2021 17:42

Great post!! Thank you.

Andreas John
Sunday, 12. Dec 2021 20:04

Thank you Christoph!

Urs Meli
Monday, 13. Dec 2021 18:50

Thanks. Great Post The harbor registry server has an integrated trivy scanner :-)

ricky
Friday, 17. Dec 2021 14:56

Is Terraform itself vulnerable to the log4j2 RCE?

Christoph Stoettner In reply to ricky
Tuesday, 21. Dec 2021 13:48

Terraform is a go application and log4j a Java library, so no Terraform has nothing in common to be vulnerable for log4j.

Error
There was an error sending your comment, please try again.
Thank you!
Your comment has been submitted and will be published once it has been approved.

Your email address will not be published. Required fields are marked with *