One of the questions that I’ve been asked on multiple occasions when presenting on Kubernetes security [1] is: “Which distribution should I install?”
There are a bewildering number of options for deploying Kubernetes, with over 60 commercial products or open source projects providing methods of deploying it. Making an informed choice can therefore be a difficult task.
If you’re planning on using Kubernetes in production, one of the key things to consider from a security perspective is your threat model. This may sound like a fancy term but it really just boils down to what kind of security attacks you are worried about. I see three main groups of threats to a Kubernetes cluster:
- External attackers: People who have no access to your cluster apart from being able to reach the applications running on it and/or the management port(s) over a network.
- Malicious containers: Here, an attacker has access to a single container (likely through some application vulnerability) and would like to expand their access to take over the whole cluster.
- Malicious user/stolen credentials: Here, an attacker has valid credentials to execute commands against the Kubernetes API, as well as network access to the port.
When looking at which Kubernetes deployment option you want, you need to consider whether each of these groups applies to you and ensure that the configuration of your cluster is modified appropriately.
External attacker
The first category, external attackers, is relatively straightforward. Here, the controls relate to ensuring that your management services (e.g. the API server, kubelet and etcd) are not exposed to untrusted networks without authentication controls in place.
Most cloud-based deployments will only expose the API server to the internet and will generally enforce authentication, so you should be in a reasonable place there. On-premise installations can get slightly more complicated as some deployments leave the kubelet accessible without authentication which could, for example, expose the cluster to attack over a corporate network.
Deciding if this affects you
Here are some questions you can think about to decide if this category of attack matters to you:
- Are my Kubernetes management services exposed to the internet? Obviously if the answer here is yes then external attackers are a potential problem.
- How much trust can I place in my internal network? If your cluster isn’t exposed to the internet, then you need to consider how ‘trusted’ your internal networks are. In a small organisation you might be quite comfortable trusting your internal networks, but in larger networks it’s generally best to assume that an external attacker can get access somehow.
Malicious container
The second category, malicious containers, is where things get more ‘interesting’ from a security standpoint.
Several Kubernetes distributions have made the decision that they don’t consider malicious containers part of their threat model. As such, once an attacker has that level of access then there are minimal controls, by default, stopping them getting full cluster-admin rights. In Kubernetes there are a number of well-known privilege escalation mechanisms, via the Kubelet, via access to etcd or via service tokens, which are deployed on all containers with high privileged rights in many cases.
Deciding if this affects you
Here are some questions you can think about relating to this:
- Do I have any internet-facing applications running in my cluster? If you’re running internet-facing applications then it’s a fact of life that attackers will try to compromise them. Therefore, the chances of having a container compromised increases.
- Am I running multiple applications in my cluster? If you’re running a lot of applications then there’s an increased risk that one will get compromised. However, running multiple applications in one cluster also increases the impact of any compromise. An attacker who can take out a container will get even more payoff as they can compromise multiple applications in the one cluster.
- Do my containers have access to my cloud service provider’s API? Some Kubernetes implementations will have application tokens for things like AWS or Azure APIs so they can interact with services there. Again, this will increase the impact of a single container being compromised, so is worth thinking about.
From a security person’s standpoint, I’d argue that all production clusters should take this scenario into account. It’s not rare to see a single issue in one application being leveraged to take control of other systems, and it’s important that your infrastructure can defend itself against this kind of attack.
Malicious users
The third scenario, malicious users, is more problematic. But with careful configuration of a cluster it is possible to make it hard for a user to escalate their privileges. This is most likely to occur when a user’s credentials have been lost or stolen. And with Kubernetes credentials tending to be static files with SSL keys in them, it’s fairly easy for them to be leaked or lost.
Deciding if this affects you
Here are some questions you can think about relating to this:
- Do I have a large number of users who have access to my cluster? If you’ve only got a small, well-controlled number of users, this scenario might not apply to you. However, if you have a larger number of users, the risk of a lost credential increases.
- Do I have a lot of users who I want to only have limited rights to my cluster? If all your users have cluster admin privileges, then controls to restrict user rights are perhaps less critical. But if you have groups of users who should only be able to take certain actions, then it’s important to consider this threat model.
Key controls
There are a large number of security settings that can be applied to a Kubernetes cluster, and with the current version of the CIS benchmark [2] weighing in at over 250 pages long, it can be a bit daunting to think of which ones should be looked at first. Depending on the threat models above, here are some key controls to think about:
External attacker
- Ensure that all management ports that are visible externally require authentication, including:
- Main Kubernetes API
- Kubelet API
- Etcd API
- Where services that don’t allow authentication (e.g. cAdvisor or the read-only Kubelet) are required, restrict access to a whitelist of source addresses.
Malicious container
- As with the external attacker approach, ensure that all management ports visible on the cluster network require authentication for all users.
- Ensure that service accounts are either not mounted in containers or have restricted rights (i.e. not cluster admin).
- Use Network Policies to restrict access between namespaces and pods.
Malicious users
- Ensure that RBAC policies are in place for all users, providing ‘least privilege’ access to cluster resources.
- Ensure that Pod Security Policies are in place for all users to restrict the rights of pods that can be created, paying particular attention to high risk items such as privileged containers.
Conclusion
Aligning the threat model you’re worried about with what your software suppliers think you should be worried about is always an important part of any deployment. Kubernetes is no different.
With the wide range of options available, it’s easy to go down a path that could leave you with unexpected, and ultimately costly, security issues. Even worse, you could face a compromised cluster and all the problems that would ensue.
References
[1] https://www.youtube.com/watch?v=b3qJwIttqqs
[2] https://www.cisecurity.org/benchmark/kubernetes/
Written by Rory McCune
First published on 23/11/17
