tl;dr

This post by Rich Warren and Sander Laarhoven discusses NCC Group observed in the wild exploitation attempts and detection logic for the F5 BIG-IP/BIG-IQ iControl REST API vulnerabilities CVE-2021-22986.

Update March 20th, 2021

We have seen PoCs of a variant posted to Twitter that doesn’t require SSRF, our advice is to treat any POSTs to REST API as suspicious

In our testing this works on 15.1.0.2 but not on 12.1.4. The log lines showing successful exploitation

==> /var/log/restjavad.0.log <==
[WARNING][142][20 Mar 2021 15:56:48 UTC][com.f5.rest.app.RestServerServlet] X-F5-Auth-Token doesn't have value, so skipping

==> /var/log/restjavad-audit.0.log <==
[I][143][20 Mar 2021 15:56:49 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":""}

Note the X-F5-Auth-Token doesn’t have a value.

As of 20:05 UTC a full public exploit has been released

Update March 19th, 2021

We are seeing full chain exploitation from the following IPs:

67.216.209[.]142
68.183.179[.]130

Source of exploitation (click for larger)

Successful exploitation attempt (click for larger)

Background

On the 10th of March 2021, F5 released an advisory for it’s BIG-IP and BIG-IQ products, stating that the REST interface of the iControl management interface is vulnerable to an authentication bypass and remote code execution [1].

These vulnerabilities were given the following CVE numbers: CVE-2021-22986, CVE-2021-22987, CVE-2021-22988, CVE-2021-22989 and CVE-2021-22990.

No detection rules or artifact information was initially provided by F5, albeit no public exploit was known at the time F5’s advisory was published, giving system administrators time to patch and blue teams the space to research and implement detection capabilities.

In the week that followed, several researchers posted proof-of-concept code after reverse engineering the Java software patch in BIG-IP [2].

Starting this week and especially in the last 24 hours (March 18th, 2021) we have observed multiple exploitation attempts against our honeypot infrastructure. This knowledge, combined with having reproduced the full exploit-chain we assess that a public exploit is likely to be available in the public domain soon.

NCC Group believes it is in the best interests of all to release our internal notes and detection logic to prevent further harm once public exploits become available.

Unsuccessful exploitation attempt as seen in the wild (click for larger)

Technical Assessment

Exploitation of this vulnerability requires two steps. First, authentication has to be bypassed by leveraging the SSRF vulnerability to gain an authenticated session token. This authenticated session can then be used to interact with REST API endpoints, which would otherwise require authentication.

The most useful endpoint for an attacker is the tm/util/bash endpoint, which allows an (authenticated) user to execute commands on the underlying server with root privileges. However, as the REST API is designed for remote administration, there are many endpoints which an attacker might wish to take advantage of.

As part of the F5 patches, a command injection vulnerability was also patched in the tm/access/bundle-install-tasks REST endpoint – which could be used as an alternative way to execute arbitrary commands once authentication has been bypassed.

Detection

This blogpost will be updated as more detection methods are researched and identified.

Log artifacts

Logs from BIG-IP and BIG-IQ appliances are stored in /var/log/ and are rotated by default.

Phase 1: SSRF exploitation

A successful exploitation of SSRF to extract a cookie can be detected in the restjavad-audit log. This log shows when a user logs in via the REST API:

==> restjavad-audit.0.log <==
[I][310][18 Mar 2021 12:37:32 UTC][ForwarderPassThroughWorker] {"user":"local/null","method":"POST","uri":"http://localhost:8100/mgmt/shared/authn/login","status":200,"from":""}

Note: Legitimate successful logins via the REST API look the same:

==> restjavad-audit.0.log <==
[I][317][18 Mar 2021 12:45:36 UTC][ForwarderPassThroughWorker] {"user":"local/null","method":"POST","uri":"http://localhost:8100/mgmt/shared/authn/login","status":200,"from":""}

Failed SSRF exploitation attempts can be found in the restjavad log file when the provided endpoint does not exist (in this case, “AVSMRRWLVVYJ” was provided as the loginReference.link parameter):

==> restjavad.0.log <==
[I][312][18 Mar 2021 12:38:45 UTC][8100/shared/authn/login AuthnWorker] Cookie  failed to login from  using the AVSMRRWLVVYJ authentication provider
[I][313][18 Mar 2021 12:38:45 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login

Phase 2: Authenticated Remote Code Execution calls

Traces of bash invocations via the REST API can be found in the restjavad-audit log.

Default log verbosity configuration shows when a user sends a POST to the REST bash endpoint:

==> restjavad-audit.0.log <==
[I][311][18 Mar 2021 12:37:33 UTC][ForwarderPassThroughWorker] {"user":"local/f5admin2","method":"POST","uri":"http://localhost:8100/mgmt/tm/util/bash","status":200,"from":""}

Successful commands also appear in the audit log file:

==> audit <==
Mar 18 05:39:28 localhost notice icrd_child[10680]: 01420002:5: AUDIT - pid=10680 user=f5admin2 folder=/Common module=(tmos)# status=[Command OK] cmd_data=run util bash -c "set -e ; id"

Note that the example user f5admin2 is legitimate and not created during the exploit chain. The username of an existing user must be known by an attacker.

Some further failed attempts were observed in the wild by our honeypot infrastructure. Fortunately the attackers got the details of the exploit wrong and attempted a chain of events that could not possibly work.

[f5admin2@bigip1:Active:Standalone] var # tail -f /var/log/restjavad.0.log
[I][289][17 Mar 2021 15:14:30 UTC][8100/shared/authn/login AuthnWorker] Cookie  failed to login from  using the WGZBBJYKZZBZ authentication provider
[I][290][17 Mar 2021 15:14:30 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login
[I][294][17 Mar 2021 18:27:37 UTC][8100/shared/authn/login AuthnWorker] Cookie  failed to login from 193.37.214.11, 127.0.0.1 using the http://localhost/mgmt/tm/access/bundle-install-tasks authentication provider
[I][295][17 Mar 2021 18:27:37 UTC][RestRequestReceiver] Rest worker uri not found: /mgmt/shared/authn/login
[I][297][17 Mar 2021 18:56:10 UTC][8100/shared/authn/login AuthnWorker] Cookie  failed to login from 193.37.214.11, 127.0.0.1 using the http://localhost:8100/mgmt/tm/util/bash authentication provider
[I][299][17 Mar 2021 19:29:12 UTC][8100/shared/authn/login AuthnWorker] Cookie  failed to login from 193.37.214.11, 127.0.0.1 using the http://mj9afbfzgjzg1ktuig0p1oxwun0ex2m.burpcollaborator.net/mgmt/tm/util/bash authentication provider

Configuring verbose logging for restjavad

The restjavad log can be configured to be a bit more verbose [3]. When log level is set to FINEST, we get the following logs in /var/log/restjavad.0.log under successful exploitation.

Notice “User null successfully logged in”:

[F][2769][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage ReadRequest, id 11661, target null
[F][2770][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage DeserializeBody, id 11661, target http://localhost:8100/mgmt/shared/authn/login
[F][2771][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage IdentifyUser, id 11661, target http://localhost:8100/mgmt/shared/authn/login
[F][2772][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage Dispatch, id 11661, target http://localhost:8100/mgmt/shared/authn/login
[F][2773][18 Mar 2021 12:59:53 UTC][8100/shared/authn/login AuthnWorker] User null successfully logged in from  using the local authentication provider.
[F][2774][18 Mar 2021 12:59:53 UTC][8100/shared/authz/users UsersWorker] patch f5admin2: no changes
[F][2775][18 Mar 2021 12:59:53 UTC][8100/shared/authn/login AuthnWorker] Updated user https://localhost/mgmt/shared/authz/users/f5admin2
[F][2776][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage SendResponse, id 11661, target http://localhost:8100/mgmt/shared/authn/login
[F][2777][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage Finish, id 11661, target http://localhost:8100/mgmt/shared/authn/login
[F][2778][18 Mar 2021 12:59:53 UTC][RestRequestReceiver] stage ReadRequest, id 11682, target null
[F][2780][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Syncing users from MCP.
[F][2781][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Syncing admins from MCP.
[F][2782][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Complete worker startup.
[F][2783][18 Mar 2021 12:59:53 UTC][8100/shared/authz/mcp_enumerator McpUsersEnumeratorWorker] Schedule syncing users/admins from MCP.

Whereas the following logs show a legitimate succesful login via the REST API:

[F][2922][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage ReadRequest, id 11881, target null
[F][2923][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage DeserializeBody, id 11881, target http://localhost:8100/mgmt/shared/authn/login
[F][2924][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage IdentifyUser, id 11881, target http://localhost:8100/mgmt/shared/authn/login
[F][2925][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage Dispatch, id 11881, target http://localhost:8100/mgmt/shared/authn/login
[F][2926][18 Mar 2021 13:01:19 UTC][8100/tm/auth PublicWorker] Forwarding operation: /tm/auth/source
[F][2927][18 Mar 2021 13:01:19 UTC][ProcessManager] Request queued: URI: /tm/auth/source
[F][2928][18 Mar 2021 13:01:19 UTC][ChildWrapper] Child 2aa19864-5327-4a61-846e-d322671698d0 state changed from IDLE to BUSY
[F][2929][18 Mar 2021 13:01:19 UTC][ChildWrapper] Forwarding GET request http://localhost:8100/tm/auth/source to http://localhost:51812/tmapi_mapper/auth/source
[F][2930][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage Connect, id 11891, target http://localhost:51812/tmapi_mapper/auth/source
[F][2931][18 Mar 2021 13:01:19 UTC][RestRequestSender] send 11891 localhost /tmapi_mapper/auth/source
[F][2932][18 Mar 2021 13:01:19 UTC][RestClientConnectionManager] [localhost:51812] pending total: 0, allocated total: 1, creating for: http://localhost:51812/tmapi_mapper/auth/source
[F][2933][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage WriteRequest, id 11891, target http://localhost:51812/tmapi_mapper/auth/source
[F][2934][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage ReadResponse, id 11891, target http://localhost:51812/tmapi_mapper/auth/source
[F][2935][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage DeserializeBody, id 11891, target http://localhost:51812/tmapi_mapper/auth/source
[F][2936][18 Mar 2021 13:01:19 UTC][RestRequestSender] stage Finish, id 11891, target http://localhost:51812/tmapi_mapper/auth/source
[F][2937][18 Mar 2021 13:01:19 UTC][RestRequestSender] done 11891
[F][2938][18 Mar 2021 13:01:19 UTC][RestClientConnectionManager] [localhost:51812] isClose = true isError = false pending req = false. pending total: 0, allocated total: 0
[F][2939][18 Mar 2021 13:01:19 UTC][Connection] client cxn close localhost/127.0.0.1:51812 - client op error=false
[F][2940][18 Mar 2021 13:01:19 UTC][ChildWrapper] Success response to GET request at http://localhost:8100/tm/auth/source
[F][2941][18 Mar 2021 13:01:19 UTC][ChildWrapper] Child 2aa19864-5327-4a61-846e-d322671698d0 state changed from BUSY to IDLE
[F][2942][18 Mar 2021 13:01:19 UTC][8100/shared/authn/login AuthnWorker] User f5admin2 successfully logged in from  using the local authentication provider.
[F][2943][18 Mar 2021 13:01:19 UTC][8100/shared/authz/users UsersWorker] patch f5admin2: no changes
[F][2944][18 Mar 2021 13:01:19 UTC][8100/shared/authn/login AuthnWorker] Updated user https://localhost/mgmt/shared/authz/users/f5admin2
[F][2945][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage SendResponse, id 11881, target http://localhost:8100/mgmt/shared/authn/login
[F][2947][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage Finish, id 11881, target http://localhost:8100/mgmt/shared/authn/login
[F][2948][18 Mar 2021 13:01:19 UTC][RestRequestReceiver] stage ReadRequest, id 11912, target null
[F][2949][18 Mar 2021 13:01:23 UTC][ErrorPassingScheduledThreadPoolExecutor] Cancelled runnable. Runnable java.util.concurrent.ScheduledThreadPoolExecutor$Sc

Network detection

Network rules (Suricata) are available for download. Note that by default all communication is encrypted over TLS, depending on your network monitoring setup, TLS offloading or a proxy setup might be required for these rules to be effective.

https://github.com/nccgroup/Cyber-Defence/blob/master/Signatures/suricata/2021_03_cve_2021_22986.txt

References

[1]

[2] Exploits

[3] https://support.f5.com/csp/article/K15436

RIFT: Research and Intelligence Fusion Team

RIFT leverages our strategic analysis, data science, and threat hunting capabilities to create actionable threat intelligence, ranging from IoCs and detection capabilities to strategic reports on tomorrow’s threat landscape. Cyber security is an arms race where both attackers and defenders continually update and improve their tools and ways of working. To ensure that our managed services remain effective against the latest threats, NCC Group operates a Global Fusion Center with Fox-IT at its core. This multidisciplinary team converts our leading cyber threat intelligence into powerful detection strategies.

RIFT: Detection capabilities for recent F5 BIG-IP/BIG-IQ iControl REST API vulnerabilities CVE-2021-22986