Active exploitation of Cisco IOS XE Software Web Management User Interface vulnerabilities

Monday, October 16, 2023 11:05

Updates  

Nov. 02: Identified a third version of the BadCandy implant. Added expected response from the new version of the implant against one of the HTTP requests used to check for infected device.

Nov. 1: Observed increase in exploitation attempts since the publication of the proofs-of-concept (POCs) of the exploits involved. Named the Lua-based web shell “BadCandy.” 

Oct. 23: Identified an updated version of the implant. Provided new curl command to check for infected devices. Fixes for CVE-2023-20198 and CVE-2023-20273 started to roll out on Oct. 22.

Oct. 20: Identified an additional vulnerability (CVE-2023-20273) that is exploited to deploy the implant. Fixes for both CVE-2023-20198 and CVE-2023-20273 are estimated to be available on Oct. 22. The CVE-2021-1435 that had previously been mentioned is no longer assessed to be associated with this activity.

Oct. 19: Added additional attacker IP and username, defense evasion observations, and new Snort rules. Also added new information regarding our assessment that the activity is being carried out by the same actor.  

Overview 

• Cisco has identified active exploitation of a previously unknown vulnerability in the Web User Interface (Web UI) feature of Cisco IOS XE software (CVE-2023-20198) when exposed to the internet or untrusted networks. This affects both physical and virtual devices running Cisco IOS XE software that also have the HTTP or HTTPS Server feature enabled.  
• Successful exploitation of CVE-2023-20198 allows an attacker to gain privilege level 15 access to the device, which the attacker can then use to create a local user and log in with normal user access. 
• Next, the attacker can use the new unauthorized local user account to exploit a second previously unknown vulnerability (CVE-2023-20273) in another component of the WebUI feature. This allows the adversary to inject commands with elevated (root) privileges, giving them the ability to run arbitrary commands on the device. 
• The recommendation that Cisco has provided in its security advisory to disable the HTTP server feature on internet-facing systems is consistent with best practices and guidance the U.S. government has provided in the past on mitigating risk from internet-exposed management interfaces.  
• Cisco support centers collaborated with the security team after using methods and procedures to correlate similar indicators in a very small number of cases out of our normal substantial daily case volume.  
• These are critical and high-severity vulnerabilities, respectively, and we strongly recommend that affected entities immediately implement the steps outlined in Cisco’s PSIRT advisory. 

Cisco identifies suspicious activity 

We discovered early evidence of potentially malicious activity on Sept. 28, 2023, when a case was opened with Cisco's Technical Assistance Center (TAC) that identified unusual behavior on a customer device. Upon further investigation, we observed what we have determined to be related activity as early as Sept. 18. The activity included an authorized user creating a local user account under the username “cisco_tac_admin” from a suspicious IP address (5.149.249[.]74). Instances of this activity ended on Oct. 1, and we did not observe any other associated behavior at that time other than the suspicious account creation.  

On Oct. 12, Cisco Talos Incident Response (Talos IR) and TAC detected what we later determined to be an additional cluster of related activity that began on that same day. In this cluster, an unauthorized user was observed creating a local user account under the name “cisco_support” from a second suspicious IP address (154.53.56[.]231). Unlike the September case, this October activity included several subsequent actions, including the deployment of an implant we’re calling “BadCandy” consisting of a configuration file (“cisco_service.conf”). The configuration file defines the new web server endpoint (URI path) used to interact with the implant. That endpoint receives certain parameters, described in more detail below, that allow the actor to execute arbitrary commands at the system level or IOS level. For the implant to become active, the web server must be restarted; in at least one observed case the server was not restarted so the implant never became active despite being installed. 

The BadCandy implant is saved under the file path “/usr/binos/conf/nginx-conf/cisco_service.conf” which contains two variable strings made up of hexadecimal characters. The implant is not persistent—meaning a device reboot will remove it—but the newly created local user accounts remain active even after system reboots. The new user accounts have level 15 privileges, meaning they have full administrator access to the device. This privileged access to the devices and subsequent creation of new users is tracked as CVE-2023-20198. 

Upon successful exploitation of CVE-2023-20198, the attackers can exploit another component of the WebUI feature to carry out command injection with elevated (i.e., root) privileges to write the implant to the file system. (This refers to CVE-2023-20273, described in more detail below.) 

We observed the threat actor gathering information about the device and conducting preliminary reconnaissance using commands listed in the Appendix. We also observed the attacker clearing logs and removing users, likely to hide evidence of their activity by using the following commands: 

clear logging 
no username cisco_support 
no username cisco_tac_admin 
no username cisco_sys_manager

We assess with a high degree of confidence that these clusters of activity were carried out by the same actor. In October, the actor removed evidence of the “cisco_tac_admin” username—which had been created in September—as part of their cleanup efforts, suggesting the September and October clusters of activity were part of the same operation. The first cluster was possibly the actor’s initial attempt and testing their code, while the October activity seems to show the actor expanding their operation to include establishing persistent access via deployment of the implant.  

The successful exploitation of the vulnerabilities and deployment of the BadCandy web shell by the threat actor resulted in the application of a non-persistent patch that prevented further exploitation, likely to prevent other threat actors from taking control of already compromised systems as the POCs became more accessible. 

Initial access and implant delivery

The CVE-2023-20198 vulnerability received the highest Common Vulnerability Scoring System (CVSS) score (10/critical). Successful exploitation allows the attacker to gain access to the device with full administrator privileges. After compromising the device, we observed the adversary exploit a second vulnerability (CVE-2023-20273), which affects another component of the Web UI feature, to install the BadCandy implant. This allows the attacker to run arbitrary commands with elevated (root) privileges, thereby effectively taking full control of the device. In this particular attack, the actor then used the ability to run arbitrary commands to write the implant to the file system. CVE-2023-20273 has a CVSS score of 7.2 (high). We identified the CVE-2023-20273 activity by leveraging existing Cisco protections.

Implant analysis 

BadCandy is based on the Lua programming language and consists of 29 lines of code that facilitates arbitrary command execution. The attacker must create an HTTP POST request to the device, which delivers the following three functions (Figure 1):  

1. The first function is dictated by the “menu” parameter, which must exist and must be non-empty. This returns a string of numbers surrounded by forward slashes, which we believe represents the implant’s version. 
2. The second function is dictated by the “logon_hash” parameter, which must be set to “1”. This returns an 18-character hexadecimal string that is hardcoded into the implant.
3. The third function is also dictated by the “logon_hash” parameter, which checks to see if the parameter matches a 40-character hexadecimal string that is hardcoded into the implant. A second parameter used here is “common_type”, which must be non-empty, and whose value determines whether the code is executed at the system level or IOS level. If the code is executed at the system level, this parameter must be set to “subsystem”, and if it is executed at the IOS level, the parameter must be “iox”. The IOX commands are executed at privilege level 15.  

In most instances, we have observed of this implant being installed, both the 18-character hexadecimal string in the second function and the 40-character hexadecimal string in the third function are unique, although in some cases, these strings were the same across different devices. This suggests there is a way for the actor to compute the value used in the third function from the value returned by the second function, acting as a form of authentication required for the arbitrary command execution provided in the third function. 

 We have also observed a second version of BadCandy, which now includes a preliminary check for the HTTP Authorization header.  Most of the core functionalities of this version remain the same as the previous version (Figure 2). The second version likely started to be deployed as early as Oct. 20, and was deployed using the earlier version of the implant.  

The addition of the header check in the implant by the attackers is likely a reactive measure to prevent the identification of compromised systems. This header check is primarily used to thwart compromise identification using a previous version of the curl command provided by Talos. Based on the information assessed to date, we believe the addition of the header check in the implant likely resulted in a recent sharp decline in the visibility of public-facing infected systems. We have updated the curl command listed under our guidance advisory to help enable the identification of implant variants employing the HTTP header checks. 

On November 2, Talos discovered a third version of BadCandy, which includes a check for the “X-Csrf-Token" HTTP header in requests to the implant.

The new variant allows the implant to check for the presence and required values of either the “Authorization” or the “X-Csrf-Token" HTTP headers. The logic to check for the header values remains the same across both headers. An incoming request to the implant qualifies as authorized as long it has one of the two headers with the correct values. The introduction of the X-Csrf-Token header is likely a measure to communicate with the implant using a new HTTP header parameter to evade existing detection mechanisms that may be relying on the presence of the “Authorization” header from the previous communication traffic with the implant.

This latest variant also changes the configuration relating to the presence of a “%” in the URI; instead of returning a 404 response, it now presents a standard login page.

Ongoing exploitation, guidance and mitigation 

Proof-of-concept (POC) exploit code for CVE-2023-20198 and CVE-2023-20273 were published on October 30 and October 31, respectively. Immediately following the disclosure of the CVE-2023-20198 POC code, we saw a dramatic increase in the number of exploitation attempts against vulnerable devices. Historically, as POCs gain popularity they are adopted by more threat actors who will use them to conduct indiscriminate, opportunistic and targeted attacks. 

We strongly recommend organizations that may be affected by this activity immediately implement the guidance and install the security fixes outlined in Cisco’s Product Security Incident Response Team (PSIRT) advisory.  

Organizations should look for unexplained or newly created users on devices as evidence of potentially malicious activity relating to this threat. One method to identify if known variants of BadCandy are present is to run the following commands against the device, where the "DEVICEIP” portion is a placeholder for the IP address of the device to check:   

curl -k -H "Authorization: 0ff4fbf0ecffa77ce8d3852a29263e263838e9bb" -X POST "https[:]//DEVICEIP/webui/logoutconfirm.html?logon_hash=1"

This will execute a request to the device’s Web UI to see if the implant is present. If the request returns a hexadecimal string, similar to what was outlined in the third function above, a known version of the implant is present. We note this will only work as an indication of compromise if the web server was restarted by the actor after the implant was installed. 

Additionally, a generic curl command can be run to help identify systems with known variants of the implant without interacting with the implant’s core functionality: 

curl -k "https[:]//DEVICEIP/%25"

If this returns a 404 HTTP response with an HTML page comprising of a “404 Not Found” message, a known one of the first two known variants of the implant is present. If this results in a standard login page, the third variant of the implant is present. A system without any known variant of the implant should return a 200 HTTP response containing a JavaScript redirect.

Note: The above checks should use the HTTP scheme if the device is only configured for an insecure web interface. 

We also have the following Snort coverage to address this threat:  

• 3:50118 - Covers the implant installation (CVE-2023-20273). 
• 3:62527 - Covers interaction with the implant, but not code execution. 
• 3:62528 - Covers interaction with the implant, but not code execution. 
• 3:62529 - Covers the passing of code into the installed implant. 
• 3:62541 - Covers exploit attempts for initial access (CVE-2023-20198). 
• 3:62542 - Covers exploit attempts for initial access (CVE-2023-20198).   

The recommendation that Cisco has provided in its security advisory to disable the HTTP/S server feature on internet-facing systems is consistent with best practices and guidance the U.S. government has provided in the past on mitigating risk from internet-exposed management interfaces. This is also in line with Cisco’s ongoing work with industry partners as part of the Network Resilience Coalition. 

 Cisco support centers collaborated with the security team after using methods and procedures to correlate similar indicators in a very small number of cases out of our normal substantial daily case volume. 

Appendix: Information gathering and preliminary reconnaissance 

The following commands were used by the threat actor (usually sequentially) to conduct preliminary information gathering and reconnaissance activity on compromised systems: 

show ip ospf 
show inventory 
show eigrp protocols 
show iox-service 
show platform software iox-service 
show ip interface brief 
show power inline 
show lldp 
show standby 
show vrrp 
show ip route 
show version 
show running-config 
show license summary 
show dlep config 
show platform 
show subsys 
show flow monitor 
show ip dns view 
show ip name-servers 

IOCs 

IOCs for this threat can also be found in our GitHub repository here. 

In addition to the curl command referenced above, perform the following checks to determine whether a device may have been compromised:  

1. Check the system logs for the presence of any of the following log messages where “user” could be “cisco_tac_admin”, “cisco_support” or any configured, local user that is unknown to the network administrator: 

%SYS-5-CONFIG_P: Configured programmatically by process SEP_webui_wsma_http from console as user on line

%SEC_LOGIN-5-WEBLOGIN_SUCCESS: Login Success [user: user] [Source: source_IP_address] at 03:42:13 UTC Wed Oct 11 2023

Note: The %SYS-5-CONFIG_P message will be present for each instance that a user has accessed the web UI. The indicator to look for is new or unknown usernames present in the message.  

1. Check the system logs for the following message where filename is an unknown filename that does not correlate with an expected file installation action:  

%WEBUI-6-INSTALL_OPERATION_INFO: User: username, Install Operation: ADD filename