Another Apache Log4j Vulnerability Is Actively Exploited in the Wild (CVE-2021

Executive Summary

On Dec. 9, 2021, a remote code execution (RCE) vulnerability in Apache Log4j 2 was identified being exploited in the wild. Public proof of concept (PoC) code was released and subsequent investigation revealed that exploitation was incredibly easy to perform. By submitting a specially crafted request to a vulnerable system, depending on how the system is configured, an attacker is able to instruct that system to download and subsequently execute a malicious payload. Due to the discovery of this exploit being so recent, there are still many servers, both on-premises and within cloud environments, that have yet to be patched. Like many high severity RCE exploits, thus far, massive scanning activity for CVE-2021-44228 has begun on the internet with the intent of seeking out and exploiting unpatched systems. We highly recommend that organizations upgrade to the latest version (2.17.1) of Apache Log4j 2 for all systems. This version also patches the additional vulnerabilities CVE-2021-45046, found on Dec. 14; CVE-2021-45105, found on Dec. 17; and CVE-2021-44832, found on Dec. 28. 

On Dec. 22, we updated this blog to include statistics on Log4j exploitation attempts that we identified by analyzing hits on the Apache Log4j Remote Code Execution Vulnerability threat prevention signature for the Palo Alto Networks Next-Generation Firewall. We describe a range of examples of activities that could be attempted in the event exploitation is successful, including mass scanning, vulnerable server discovery, information stealing, possible delivery of CobaltStrike and coinmining. We also include a timeline of recent events relating to Log4j vulnerabilities.

On Dec. 28, we updated this blog to include information about CVE-2021-44832, which is an RCE vulnerability affecting instances of Log4j 2 in instances where an attacker has permission to modify the logging configuration file and can in turn construct a malicious configuration using a JDBC Appender. This JDBC Appender in turn references a JDNI URI that can execute remote code on the affected device.

Affected Version

Apache Log4j 2.x <= 2.15.0-rc1

Affected Software

A significant number of Java-based applications are using log4j as their logging utility and are vulnerable to this CVE. To the best of our knowledge, at least the following software may be impacted:

• Apache Struts
• Apache Solr
• Apache Druid
• Apache Flink
• ElasticSearch
• Flume
• Apache Dubbo
• Logstash
• Spring-Boot-starter-log4j2

Palo Alto Networks customers are protected via Next-Generation Firewalls (PA-Series, VM-Series and CN-Series) or Prisma Access with a Threat Prevention security subscription and protected by Cortex XDR using exploit protection on Linux endpoints and Behavioral Threat Protection across Windows, Mac and Linux endpoints. Prisma Cloud can detect continuous integration (CI), container images and host systems which maintain vulnerable instances of log4j. You can also automate incident response with Cortex XSOAR.

Background on Apache log4j 2

Apache log4j 2 is an open source Java-based logging framework, which is leveraged within numerous Java applications around the world. Compared with the original log4j 1.X release, log4j 2 addressed issues with the previous release and offered a plugin architecture for users. On Aug. 5, 2015, log4j 2 became the mainstream version and all of the previous version log4j users were recommended to upgrade to log4j 2. Apache log4j 2 is widely used in many popular software applications, such as Apache Struts, ElasticSearch, Redis, Kafka and others.

While supplying an easy and flexible user experience, Apache log4j 2 has historically been vulnerable to process and deserialize user inputs. Two previous deserialization vulnerabilities, CVE-2017-5645 and CVE-2019-17571, were previously discovered, resulting in code injection and further RCE due to a lack of necessary processing against provided user input data.

• CVE-2017-5645: For Apache log4j 2.x before 2.8.2, the log4j servers will deserialize any log events received from other applications through TCP or UDP socket servers. If a crafted binary payload is being sent using this vulnerability, it can lead to arbitrary code execution.
• CVE-2019-17571: For Apache log4j versions from 1.2 (up to 1.2.17), the SocketServer class is vulnerable to deserialization of untrusted data, which leads to remote code execution if combined with a deserialization gadget.

Description of the Vulnerability (CVE-2021-44228)

The Apache log4j library allows for developers to log various data within their application. In certain circumstances, the data being logged originates from user input. Should this user input contain special characters and be subsequently logged within the context of log4j, the Java method lookup will finally be called to execute the user-defined remote Java class in the LDAP server. This will in turn lead to RCE on the victim server that uses the vulnerable log4j 2 instance.

Root Cause Analysis

If we take a closer look, we discover that log4j 2.x supports a mechanism called lookups, which is usually used to set up the log4j config flexibly for users. The official introduction about Lookups is as follows:

Lookups provide a way to add values to the log4j configuration at arbitrary places. They are a particular type of Plugin that implements the StrLookup interface.

The normal user can conveniently and flexibly add values to the configuration at arbitrary places with the predesigned format by using this feature. In detail, when calling the log method in the application, log4j 2.x will call the format method to check the specific characters ${ in each log.

Should these characters be present, the Java method lookup will be called to find strings after the characters ${ and then replace the expression after the characters ${ with the real value found before. For example, when calling the log function in the application to log the content shown in Figure 1, the strings java:runtime, java:vm, and java:os after the characters ${ will be considered as the parameter of the lookup method and finally replaced with the corresponding values, such as Java(TM) SE Runtime Environment (build 1.7.0_67-b01) from Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode, and Windows 7 6.1 Service Pack 1, architecture: amd64-64.

There are several types of lookup supported by the feature lookups, such as Jndi Lookup, JVM Input Arguments Lookup (JMX), and Web Lookup. The Jndi lookup allows variables to be retrieved by JNDI. In the Jndi Lookup, several protocols are supported to make the remote lookup, such as LDAP and RMI. If the log includes the strings shown in Figure 2, the Java method lookup will be called to find the string jndi:logging/context-name.

Considering the log content is usually exposed to users and can be easily controlled by the attacker in many applications, once the attacker controls the string as shown in Figure 3 and sets a malicious Java class on an attacker-controlled LDAP server, the lookup method will be used to execute the malicious Java class on the remote LDAP server.

The log4j library is a powerful log framework with very flexible features supported. However, convenient features often involve potential security issues at the same time. Without careful user input filtering and strict input data sanitization, a blind trust of user input may lead to severe security issues.

Exploit

Exploit code for the CVE-2021-44228 vulnerability has been made publicly available. Any user input hosted by a Java application using the vulnerable version of log4j 2.x may be exposed to this attack, depending on how logging is implemented within the Java application.

In-the-Wild Attacks

Thus far, widespread scanning is taking place on the internet with the intention of identifying vulnerable instances of log4j. These scans are being made via HTTP and do not appear to be targeting any specific applications. Many of these requests are leveraging the User-Agent field in hopes of identifying and subsequently exploiting systems on the internet. One such example of these requests is as follows:

Once the base64-encoded log is decoded, we are presented with the following command:

Other commands observed during these massive scans include the following, which is attributed to the Kinsing coinminer malware family.

Statistics on Log4j Remote Code Execution Exploitation Attempts

To better understand the impact of the recent vulnerabilities in Log4j facing our customers, we analyzed the hits on the Apache Log4j Remote Code Execution Vulnerability threat prevention signature Dec. 10, 2021-Feb. 2, 2022. Based on our telemetry, we observed 125,894,944 hits that had the associated packet capture that triggered the signature. Figure 7 shows the hits per day, including a large spike in activity Dec. 12-16, followed by a tapering off of activity from Dec. 16-21 and another large spike on Jan. 1, 2022. After the spike in the new year, the signature hits results in a jagged line with counts differing day to day, but with the spikes being dramatically smaller than those previously seen.

We analyzed the packet captures that triggered the signature Dec. 10-31 and found the exploitation attempts appear in various places within the HTTP requests, primarily the URL and fields within the HTTP request header. We extracted 70,577,055 exploit strings from the packet captures and found that over 49% were within the top six fields of the HTTP request, as seen in Table 1. It should also be noted that many of the packet captures showed exploit strings within multiple fields within the HTTP request, each of which were counted in these figures.

Table 1. Top six fields within HTTP requests that contained Log4j exploit attempts.

Observed Activity

Since Dec. 10, 2021, we have seen attempts to exploit Log4j to carry out a variety of activities. We determined details about these activities by analyzing the files hosted at the callback URLs used in the exploit attempts – in other words, by investigating what would have happened had the attempts been successful. The observed activities after exploitation range from simple vulnerable server identification via mass scanning, to the installation of backdoors to exfiltrate sensitive information and to install additional tools, to the installation of coin mining software for financial gain. The cases discussed in this section are by no means exhaustive as we continue to discover additional attacks in our telemetry.

Mass Scanning

Our analysis of the activity involving the Apache Log4j Remote Code Execution Vulnerability signature showed most of the Log4j exploit attempts were related to mass vulnerability scanning. Table 2 shows the top domains and IP addresses seen in the callback URLs within the Log4j exploit string, which account for just over 80% of signature hits Dec. 10-31. We clustered all RFC1918 IP addresses seen in these callback URLs into their respective ranges (10/8, 172.16/12 and 192.168/16) and found that 54% of the signature hits in this time frame were generated by internal scanning. Additionally, several well-known vulnerability scanning services are represented in this list, such as nessus[.]org as the top callback involving a remote location.

Table 2. Top domains and IP addresses seen in callback URLs of Log4j exploit attempts.

Vulnerable Server Discovery

Many inbound exploitation attempts we observed did little more than send an outbound request to notify the issuer of a successful exploitation. We cannot confirm whether all of these attempts were for scanning purposes or if they were part of a malicious actor’s reconnaissance efforts. In some cases, these exploit attempts simply used the initial interaction with the callback URL to signify a vulnerable server, many of which used “canary tokens,” as seen in the following callback URL:

x[hostname].l4j.2sk9753uabgse6xz75tooe5ix.canarytokens[.]com

However, in other cases we observed the actor fully exploiting the vulnerability by loading and executing a Java class from the callback URL that would simply reach out to a server to determine if a system was vulnerable and/or exploitable. For instance, we observed the following callback URLs used in exploit attempts over the course of several days:

ldap://2.57.121[.]36:8000/mss_useragent
ldap://2.57.121[.]36:8000/mss_xapi
ldap://2.57.121[.]36:8000/mss_xforward

Upon accessing this URL, the server would access a Java class from hxxp://2.57.121[.]36/Rjava.class, which contained the decompiled code seen in Figure 8. 

As you can see from the Java code, this does nothing more than issue an HTTP GET request to hxxp://2.57.121[.]36/juccess and does nothing with the response. This Java code suggests the issuer is using the exploitation to determine whether the server is vulnerable and able to successfully run the Java class.

V8 Password Stealer

In addition to vulnerability scanning, we also saw exploitation result in the execution of information stealers. For instance, we observed several exploit attempts that involved a callback URL that contained the domain 1ma[.]xyz, as seen in the following example:

<redacted>.com.80.reference.1ma[.]xyz:1389/a

The above URL will result in the following file:

After accessing the file above, the server would download a Java class file from a hxxp://161.35.184[.]54:9998/V8.class URL, which responds with a Java class file whose decompiled code appears in Figure 10.

The code above attempts to exfiltrate information from the server by sending the data via HTTP POST requests or via DNS tunneling. The HTTP POST requests would be sent to the following URLs:

hxxp://[hostname].[username]8.pef.mur.1ma[.]xyz/
hxxp://[hostname].[username]5.pef.mur.1ma[.]xyz:53/
hxxps://[hostname].[username]4.pef.mur.1ma[.]xyz/

The DNS tunneling involves attempting to query domains with the following structure to send the data to the server:

[hostname].[20 bytes of data].[chunk number].spif.mur.1ma.xyz

Two general pieces of information are exfiltrated to the C2 domain. The first is the sensitive contents of the /etc/passwd file from the compromised server. Second, the code will obtain the environment variable names and their respective values and send them to the C2 as well.

The Java code also attempts to exfiltrate the information by running several commands that use the curl and wget applications to send the data to the C2 server, as seen in Figure 11.

Happy Everyday! + CobaltStrike

In addition to information stealers, we also observed actors exploiting Log4j to install backdoors. For instance, we saw exploit attempts that included the following callback URL:

ldap://139.155.2[.]105:8888/EvilObj

The above URL will result in the following file:

The EvilObj.class from hxxp://139.155.2[.]105:8081 contains the decompiled Java code as seen in Figure 13.

The Java in Figure 13 above creates a raw socket to 139.155.2[.]105:1234 and sends the following usage instructions over the socket:

happy everyday!
help: list [dir] | read [file] | exec [cmd]

The list command will list the files at a path specified by the threat actor, while the read command will read the contents of a file at a specified path. The exec command uses the Java.lang.Runtime.exec method to execute a command, in which the results would be sent back to the actor. These three commands provide enough functionality to fully control the system.

On Dec. 18, 2021, we observed a CobaltStrike server hosted at 139.155.2[.]105, specifically on TCP/4433, and the CobaltStrike beacon's configuration seen in Figure 14 below.

While we did not see the actor directly deploy CobaltStrike via the Log4j vulnerability, it is possible that the actor uploaded a CobaltStrike staging payload via the "happy everyday!" backdoor executed by exploiting the Log4j vulnerability.

XMRig Coinminer 

We also saw evidence of financially motivated actors exploiting the Log4j vulnerability to install coinmining software. We observed an exploit attempt included the following callback URL:

ldap://192.46.216[.]224:1389/Exploit

The callback URL responds with the following: 

The hxxp://165.22.2[.]186:80/wp-content/themes/twentyseventeen/Exploit.class responded with a Java class that contained the decompiled code sen in Figure 16.

The Java code in Figure 16 checks to see if the system is running Windows as its operating system, and if so, it runs PowerShell commands to download additional files and execute them. The first file downloaded was hosted at hxxp://150.60.139[.]51:80/wp-content/themes/twentyseventeen/s.cmd, which contains the following PowerShell that would be run on the command line:

The PowerShell command attempts to download and execute an application from hxxp://68.183.165[.]105:80/wp-content/themes/twentyseventeen/xmrig64.exe, which is the XMRig executable used to mine the Monero cryptocurrency, specifically using the wallet address of 46QBumovWy4dLJ4R8wq8JwhHKWMhCaDyNDEzvxHFmAHn92EyKrttq6LfV6if5UYDAyCzh3egWXMhnfJJrEhWkMzqTPzGzsE.

Patch and Bypass: Fixes Added for CVE-2021-45046, CVE-2021-45105, CVE-2021-44832

With the official Apache patch being released, 2.15.0-rc1 was initially reported to have fixed the CVE-2021-44228 vulnerability. However, a subsequent bypass was discovered. A newly released 2.15.0-rc2 version was in turn released, which protects users against this vulnerability.

On Dec. 14, it was discovered that the fix released in Log4j 2.15.0 was insufficient. CVE-2021-45046 was assigned for the new vulnerability discovered. On Dec. 17, Apache upgraded the severity of this vulnerability, indicating it can be used to gain remote code execution under certain circumstances.

On Dec. 17, version 2.17.0 was released to patch CVE-2021-45105. This new vulnerability results from version 2.16 not protecting from uncontrolled recursion from self-referential lookups. Exploitation allows for a denial of service (DOS) attack against the process running Log4j. This vulnerability is less critical than the previous RCE vulnerabilities but could allow an attacker to crash a vulnerable application. Please see the Apache Log4j security advisory for potential mitigations.

On Dec. 28, version 2.17.1 was released to patch CVE-2021-44832. This new vulnerability may result in RCE under specific, non-default conditions. In instances where an attacker has permission to modify the logging configuration file and can construct a malicious configuration using a JDBC Appender, this JDBC Appender may in turn reference a JDNI URI that can execute remote code on the affected device.

Timeline

Conclusion

CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 and CVE-2021-44832 are still being actively investigated in order to properly identify the full scope severity. Given the information currently available, these vulnerabilities may have a high impact at present and in the future. Most of the applications being affected are widely used in the corporate networks as well as home networks. Users are encouraged to take all necessary steps to ensure they are protected against these vulnerabilities, as outlined below.

Unit 42 is actively monitoring the abnormal traffic through our devices and cloud solutions. Palo Alto Networks provides protection against the exploitation of this vulnerability:

• Next-Generation Firewalls (PA-Series, VM-Series and CN-Series) or Prisma Access with a Threat Prevention security subscription can automatically block sessions related to this vulnerability using Threat IDs.
  • 91991, 91994, 91995, 92001, 92007 and 92012 (Application and Threat content update 8506).
  • Customers already aligned with our security best practices gain automated protection against these attacks with no manual intervention. These signatures block the first stage of the attack.
  • Customers should verify security profile best practices are applied to the relevant security policies and have critical vulnerabilities set to reset or default actions.
  • Additionally, the Log4j RCE requires access to code hosted externally. Our Advanced URL Filtering security service is constantly monitoring and blocking new, unknown and known malicious domains (websites) to block those unsafe external connections.
  • Also, suitable egress application filtering can be used to block the second stage of the attack. Use App-ID for ldap and rmi-iiop to block all RMI and LDAP to or from untrusted networks and unexpected sources.
  • SSL decryption needs to be enabled on the firewall to block known attacks over HTTPS.
  • Customers with log4j in their environments should upgrade or apply workarounds suggested by respective vendors, and not rely only on the Threat Prevention signatures.
• Cortex XDR customers running Linux agents and content 290-78377 are protected from a full exploitation chain using the Java Deserialization Exploit protection module. Other Cortex XDR customers are protected against various observed payloads stemming from CVE-2021-44228 through Behavioral Threat Protection (BTP). Additionally, Cortex XDR Pro customers using Analytics will have post-exploitation activities detected related to this vulnerability.
• Cortex XSOAR customers can leverage the  "CVE-2021-44228 - Log4j RCE'' pack to automatically detect and mitigate the vulnerability. Read more on the XSOAR marketplace.
• Prisma Cloud Compute Defender agents can detect whether any continuous integration (CI) project, container image, or host system maintains a vulnerable Log4j package or JAR file with a version equal to or older than 2.14.1. In addition, Web Application and API Security (WAAS) rules can be used to detect and block exploit payloads. Read more on the Prisma Cloud Log4Shell mitigations blog.

For users who rely on Snort or Suricata, the following rules have been released:

• 2034647
• 2034648
• 2034648
• 2034649
• 2034650
• 2034651
• 2034652

Customers of applications leveraging Apache log4j should upgrade to the newest version.

Since the original patch was discovered to be bypassed, in the interest of implementing as many protections against this vulnerability as possible, the following mitigations are also recommended:

• Disable suspicious outbound traffic, such as LDAP and RMI on the server in PANW Firewall.
• Disable JNDI lookup.
  • Set up log4j2.formatMsgNoLookups=true
  • Remove the JndiLookup file in the log4j-core and restart the service.
• Disable JNDI
  • Set up spring.jndi.ignore=true

Palo Alto Networks will continue to monitor the situation and update this document with any new findings or information. If you think you may have been compromised or have an urgent matter, get in touch with the Unit 42 Incident Response team or call North America Toll-Free: 866.486.4842 (866.4.UNIT42), EMEA: +31.20.299.3130, APAC: +65.6983.8730, or Japan: +81.50.1790.0200.

Additional Resources

Log4j Resource Center
Apache Log4j Threat Update Briefing (On-Demand)
Hunting for Log4j CVE-2021-44228 (Log4Shell) Exploit Activity
Addressing Apache Log4j Vulnerability with NGFW and Cloud-Delivered Security Services
How Cortex XDR Blocks Log4Shell Exploits with Java Deserialization Exploit Protection
Shining a Light on Log4j Exploit Payloads

Acknowledgements

We would like to thank Jen Miller-Osborn, Laura Novak and Joseph Opacki for their help with the blog and research.

Updated March 31, 2021, at 10 a.m. PT.