I recently had an interesting discussion with Gianluca Brindisi from Spotify about the differences between Kubernetes Security and Container Security. (He wrote an excellent post about container security on his blog here.) Typically, the discussion about container security focuses on general questions like the following that aren’t focused on a specific orchestration framework like Kubernetes:
- What images do I run and do they have secure base images?
- Is my image registry secure?
- What operating system permissions do my containers run with?
- Is my container network properly segmented?
- What processes do my containers run and what files do those processes access at runtime?
These are all important questions when zooming out and looking at container security in general. However in the Kubernetes context, something is lost.
Let’s illustrate the type of questions that you should ask in order to secure your Kubernetes cluster in addition to your containers. Some brief answers and tips are included for convenience.
- Can someone bypass network segmentation by using the Kubernetes APIServer as a proxy and tunnel from one pod to another via kubectl port-forward? It depends on what Role-Based Access Control (RBAC) permissions that pod’s service account has. (There isn’t much documentation around the pods/portforward role, but you can see it in the source code here.)
- If an attacker has breached one pod with restricted operating system permissions, can they escalate privileges by using the Kubernetes APIServer to launch a new pod with greater permissions? Yes, if that pod’s service account has RBAC permissions to create a new pod and you haven’t set up additional restrictions like Pod Security Policies.
- Can an attacker use raw-sockets to wreak havoc on the cluster’s container network? Yes, by default they usually can!
- Are your security assumptions about Kubernetes namespaces wrong? Specifically, are you incorrectly assuming that Kubernetes namespaces are equivalent to security boundaries and that privileged pods in one namespace can’t impact pods in another namespace?
Root Causes for Kubernetes Security Flaws
Most of the Kubernetes specific security flaws that we see at Alcide stem from one of four root causes:
- A well-intentioned developer or devops engineer once granted overly-permissive RBAC permissions to a default service account in order to “make things work” and no-one today remembers that change or is aware of the lingering consequences.
- The security team isn’t familiar with Kubernetes and all of it’s obscure pitfalls, or the devops team doesn’t have enough security experience to recognize nuanced security mistakes. This is common even among extremely talented engineers because Kubernetes is both complicated and new.
- People think that Kubernetes and containers do all kinds of magic which Kubernetes/Docker actually don’t. For example, people assume that if an application is running in a container then surely it can’t open a raw socket by default.
- Default Kubernetes configurations are too permissive (here too, raw sockets are a good example)
These four root causes lead to a variety of security flaws in Kubernetes clusters.
Securing Your Cluster
All of these issues can be fixed by proactively verifying that your cluster is properly configured, by making sure that workloads are properly segmented in the correct Kubernetes-native way, and by educating yourself and your teams about the nitty gritty details of Kubernetes and not just containers.
The Big Picture – Containers, Image Scanning, and Kubernetes
Kubernetes is best viewed as a “Cloud Operating System” which runs applications called “containers”. Focusing on container security is important and it is equivalent to traditional application security. However, ignoring Kubernetes security and only focusing on container security is like ignoring Linux security and only focusing on Nginx/Apache security. The environment in which containers run, what the Container Network Interface (CNI) looks like, and what privileges the operating system called Kubernetes grants the containers (in terms of RBAC permissions) is extremely important if you want to secure your cloud.
To complete this analogy, you can think of image scanning as the cloud equivalent of source-code scanning which checks if you have known vulnerabilities in your code.
Image scanning is important but it isn’t a replacement for a firewall, antivirus, or proper operating system configuration.
In the “old days”, when containers ran only on top of Docker, container security was enough. Nowadays, make sure you don’t overlook the operating system (Kubernetes) and focus only on the apps (containers) because doing so will leave large gaps in your security and compliance.