tl;dr
CVE-2020-5902 was disclosed on July 1st, 2020 by F5 Networks in K52145254 as a CVSS 10.0 remote code execution vulnerability in the Big-IP administrative interface. This blog looks at the root causes of both exploit paths discovered. They boil down to subtle configuration issues and differences in behaviour between Apache httpd and Apache Tomcat when dealing with an uncommon URI element called matrix (or path) parameters.
Exploit refresher
Before we go into the specifics it is useful to quickly revisit the exploits we’ve seen involving the two endpoints:
https://[IP]/tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
and
https://[IP]/hsqldb;
Apache module to handle authentication
F5 implements their own PAM and cookie module in the guise of mod_f5_auth_cookie.so inside of which they allow certain URLs to be requested without the need for authentication:
As shown in the above we can request /tmui/login.jsp without the need for authentication.
The use of Apache httpd and mod_proxy_ajp
To understand the issues, we first need to look at the technologies involved and their configuration.
F5’s Big-IP uses Apache httpd as the user facing webserver proxying through to Apache Tomcat for some URLs via mod_proxy_ajp.
The proxy_ajp.conf configuration related to the two exploited endpoints looks like this:
ProxyPassMatch ^/tmui/(.*.jsp.*)$ ajp://localhost:8009/tmui/$1 retry=5 ProxyPassMatch ^/hsqldb(.*)$ ajp://localhost:8009/tmui/hsqldb$1 retry=5
The httpd.conf configuration related to these same two endpoints looks like this:
# # HSQLDB ## # TMUI # AuthType Basic AuthName "BIG-IP" AuthPAM_Enabled on AuthPAM_IdleTimeout 1200 require valid-user Require all granted # Enable content compression by type, disable for browsers with known issues AddOutputFilterByType DEFLATE text/html text/plain application/x-javascript text/css BrowserMatch ^Mozilla/4 gzip-only-text/html BrowserMatch ^Mozilla/4.0[678] no-gzip BrowserMatch bMSIE !no-gzip !gzip-only-text/html AuthType Basic AuthName "Restricted area" AuthPAM_Enabled on AuthPAM_ExpiredPasswordsSupport on AuthPam_ValidateIP On AuthPAM_IdleTimeout 1200 AuthPAM_DashboardTimeout Off require valid-user Require all granted
The important bits to take away are:
- You can access /tmui/login.jsp without authentication due to the PAM module allowing it despite the RequireAll directive
- The mod_ajp configurations use regex wildcards
- The Apache configurations DO NOT use wildcards OR LocationMatch regex
These last two points mean they are unbalanced.
History lesson in Apache versus Tomcat path discrepancies
There was a near exact same vulnerability to this identified in March, August and November of 2018 in the other Apache Tomcat connectors known as jk and wider configurations:
- ISAPI redirector in CVE-2018-1323 in March, 2018
- Blackhat presentation titled Breaking Parser Logic (Slides 40 – 53) in August, 2018
- mod_jk in CVE-2018-11759 in November, 2018
This issue is specifically Apache Tomcat’s parsing of semicolon (;) versus Apache httpd – Immunit hints at the problem with this description:
“Apache httpd interprets semicolons in URL as ordinary characters for path resolution, while Tomcat interprets them as query delimiters (with a similar functionality as “?”). “
The Immunit description is close, but there are differences between Path Parameters and Query Parameters. We go back to 2011 to the blog post titled Three Semicolon Vulnerabilities for a further explanation:
“Apache Tomcat is one example of a web server that supports “Path Parameters”. A path parameter is extra content after a file name, separated by a semicolon. Any arbitrary content after a semicolon does not affect the landing page of a web browser.”
In the 2019 post What every web developer must know about URL encoding we see:
“Each path segment can have optional path parameters (also called matrix parameters) which are located at the end of the path segment after a “;”, and separated by “;” characters. Each parameter name is separated from its value by the “=” character like this: “/file;p=1” which defines that the path segment “file” has a path parameter “p” with the value “1”. These parameters are not often used — let’s face it — but they exist nonetheless”
So we have two web servers and one (Apache Tomcat) we know allows Path/Matrix parameters.
Knowing this we can then go to the Apache httpd source – firstly, let’s look at the mod_proxy_ajp.c source:
The path we take here is the ‘else’ or ap_proxy_canonenc so if we look at the proxy_util.c source and sure enough:
So the ap_proxy_ajp function will allow paths with ; in through to the back end Tomcat without normalization/canonicalization. We can see this behaviour if we sniff the loopback between Apache httpd and Apache Tomcat on a Big-IP:
If we now jump to the Tomcat source, the Catalina connector and Request.java we see the culprit of our discrepancy and namely this behaviour:
Specifically the removePathParameters function will chop out the contents from/including the ; until the next forward slash.
This changes our payload from
https:///tmui/login.jsp/..;/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
to:
https:///tmui/login.jsp/../tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
The RequestUtil.normalize function will then do what we would expect, namely remove the previous blob of the URI:
This changes the payload from:
https:///tmui/login.jsp/../tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
to:
https:///tmui/tmui/locallb/workspace/fileRead.jsp?fileName=/etc/passwd
Let’s validate that against the web.xml configuration of Tomcat to see if it makes sense and sure enough where we end up is here:
org.apache.jsp.tmui.locallb.workspace.fileRead_jsp /tmui/locallb/workspace/fileRead.jsp
So if we look at the original Login.jsp
LoginJsp /login.jsp
Which takes us to:
LoginJsp org.apache.jsp.tmui.login.index_jsp 3
So that all aligns and allows us to tie the input to the behavior we see.
The root cause(s)
The root cause (or causes) are subtly different for both the endpoints and both could in part be attributed to the choice of mod_proxy_ajp over mod_jk for the Tomcat connector.
The first is the discrepancy between how semicolons and path/matrix parameters are treated by mod_proxy_ajp in Apache httpd and Apache Tomcat.
The second is a little more subtle but because of the first led it to being exploitable. The Apache httpd config for Location was:
Where had it been:
It would not have been exploitable, per the Apache documentation for Location. This is because /hsqldb will match /hsqldb, /hsqldb/ and / hsqldb/file.txt BUT NOT /hsqldbsomething or /hsqldb; (as we saw in the exploit which then gets removed).
The third is also subtle but primarily due to the fact that authentication / session validation was handled by Apache httpd. This was implemented via a custom module which didn’t normalize URIs in the same way as Apache Tomcat as it relied on Apache httpd behaviour. This combined with the discrepancies in path/matrix parameters facilitated exploitation.
General mitigations
If you are using Apache httpd with Apache Tomcat and mod_proxy_ajp or similar we recommend using a generic configuration as F5 did to block the use of semicolons in location, namely:
Redirect 404 /
However also be aware Location tags should also be made as greedy as possible to provide the maximum protection. We saw one bypass which did not use semicolon and is also due to a discrepancy between Apache httpd and Apache Tomcat.
There will be more of these issues to be found
Subtle configuration issues coupled with subtle differences in functionality and potentially proprietary code are all too common. As such we expect more of these types of issues, especially in this technology combination, to be found.
