December 20, 2021: A new Denial of Service vulnerability was announced over the weekend by The Apache Foundation. They now recommend that software vendors and IT departments use version 2.17.0. This means that systems that were patched as of Friday, December 17, 2021 may need to have another patch applied. Assura continues to recommend following software vendor guidance or applying compensating controls to mitigate this threat. Assura also advises continued vigilance for security departments to monitor for suspicious outbound connections from Java processes and to look for post-exploitation activity as indicators of potential success of an attack.
December 14, 2021: This Cyber Heads-up has been updated to reflect the fact that the Apache Foundation now recommends upgrade of Log4j to version 2.16.0.
Overview
Warning: this is a highly technical Cyber Heads-up. If you’re an organizational or line-of-business leader and want to know what this means to you, please skip to Assura’s Take below.
Late last Thursday (December 9, 2021) it was announced in a now-deleted Tweet that there is a 0-day exploit in the extremely popular Java Log4j logging library. Never heard of Log4j? Well, Steam, Apple, Minecraft, Cisco, and many other names you’re likely familiar with use it in their products. Given the ease of exploitation and the fact that Proof of Concept (PoC) code was released without any advanced warning, it should be no surprise that the internet caught fire over the weekend with attackers and defenders alike focusing on Log4j. This vulnerability has been assigned the Common Vulnerabilities and Exposures (CVE) number CVE-2021-44228.
What Do We Know About This Attack?
This attack may result in being the most prolific attack of 2021, and possibly beyond. The Log4j library is used in a lot of proprietary and open-source software. Unfortunately, as with many of Apache’s projects, the library is widely used by large organizations but is also under-supported. In fact, Log4j has two unpaid developers that have supported the project for years despite the fact that blue-chip technology companies like those mentioned above rely on this software.
So down to brass tacks, let’s go over the five W’s of this vulnerability:
Who is affected?
Any software package or vendor using Apache Log4j before or equal to 2.0-beta9 through 2.14.1. (2.0-beta9 <= Apache Log4j <= 2.14.1)
What caused this?
An undetected software vulnerability believed to be introduced into the Apache Log4j open source project as early as 2014.
When was this discovered?
The vulnerability is believed to have been initially discovered by children playing Minecraft and looking to take over each other’s servers. The vulnerability and the exploit were released on Twitter (Tweet since deleted) on Thursday, December 9th late at night.
As of today, Monday, December 13th, many vendors are still exploring their code to determine whether they are vulnerable.
Where do I detect or validate this vulnerability?
To detect this vulnerability, you can parse your current logs (generally /var/log and subfolders):
Grep/Zgrep – Plaintext exploit variant.
This command searches for exploitation attempts in uncompressed files in the folder (/var/log) and sub folders:
sudo egrep -I -i -r ‘\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+’ /var/log
This command searches for exploitation attempts in compressed files in the folder (/var/log) and sub folders:
sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i ‘\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+’
Grep/Zgrep – Obfuscated exploit variants.
These commands will cover obfuscated variants of the attack but will lack the file name in a potential match (but you’ll at least know your server saw an attack).
This command searches for exploitation attempts in uncompressed files in the folder (/var/log) and sub folders:
sudo find /var/log/ -type f -exec sh -c “cat {} | sed -e ‘s/\${lower://’g | tr -d ‘}’ | egrep -I -i ‘jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):’” \;
This command searches for exploitation attempts in compressed files in the folder (/var/log) and sub folders:
sudo find /var/log/ -name ‘*.gz’ -type f -exec sh -c “zcat {} | sed -e ‘s/\${lower://’g | tr -d ‘}’ | egrep -i ‘jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):’” \;
To validate this vulnerability, our Partners at Huntress Labs have made it simple.
- Go here: https://log4shell.huntress.com/
- Copy and paste your uniquely provided exploit string into every potential injection point on your publicly exposed sites. This includes form fields, username, and password fields, and if you’re more technically apt you can even inject this into request headers, X-Forwarded-For, and other custom headers.
- If your site is vulnerable, you should see an entry on the results page.
Why is this vulnerability important?
This vulnerability is prevalent in many enterprise-grade software packages. The threat intelligence community has already seen threat actors spinning up attack infrastructure to scan the internet for potential victims. Additionally, there is information out there that some threat actors have developed a wormable version of their attack which will exploit this vulnerability, take over the system, and then drop Command and Control and ransomware on the device before moving on to other vulnerable systems.
Assura’s Take
Given the prevalence of this logging library in popular software, this could go down as one of the worst vulnerabilities in recent memory. Your organization should take this very seriously. Your next steps should be to work with your internal IT team, software vendors, and/or MSP to determine if you have software in your environment that is using vulnerable versions of Log4j. You can see a publicly maintained list of vendors who have made statements regarding their vulnerability status here: https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592
If you find that you do have vulnerable software in your environment. The only 100% effective mitigation is to update Log4j. If your software vendor(s) have published an update to their software, those updates should be deployed as soon as possible. If you are capable or responsible for upgrading Log4j in a software package, then Apache has made version 2.15.0 2.16.0 available here: https://logging.apache.org/log4j/2.x/download.html
Additionally, you may use the following mitigation strategy until an upgrade is made available by a vendor:
Enable formatMsgNoLookups
Note: This mitigation only applies to log4j versions 2.10.0 and newer.
In log4j 2.15.0 2.x, this flag is set by default.
Set formatMsgNoLookups=true when you configure log4j by performing one of the following:
- Pass as a JVM Flag
- You can pass this as an argument when you invoke java.
java -Dlog4j2.formatMsgNoLookups=true …
- Set Environment Variable
- Alternatively, this feature may be set via Environment Variable.
LOG4J_FORMAT_MSG_NO_LOOKUPS=true java …
- Or you can set this using the JVM arguments environment variable.
JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
It is not recommended that you rely on a Web Application Firewall because this vulnerability can be exploited in several ways that don’t necessarily rely on web traffic. For example, desktop applications and IoT devices such as smartwatches are vulnerable. It is also not recommended to rely on upgrading Java versions and/or updating log statements to try to mitigate this attack. These are temporary workarounds that are not 100% effective.
Assura’s SOC is working on continuous improvement of detection capabilities to help detect attacks against client infrastructure. When a potential attack is detected all SOC clients will be notified in alignment with established SLAs.
Assura’s Offensive Security Operations department has skilled team members checking for this vulnerability in Continuous Penetration Testing client environments and will notify clients immediately if Log4j is found and exploitable.
If you’re an Assura client, please do not hesitate to reach out to your Virtual ISO or Assura SOC point-of-contact if you have questions about this vulnerability or our response. Otherwise, please contact us at [email protected].