The recent MiTM attack disclosed was a very unusual one in the Container Security world. All at the same time, Kubernetes, Docker, and Calico announced security bulletins related to IPv6 Rogue Router Advertisements. There are several security bulletins here because this isn’t a single vulnerability in one product – rather, multiple independent CNIs are all vulnerable. IPv6 Router Advertisements are a fairly obscure topic, yet this vulnerability is definitely worth understanding.
A container with CAP_NET_RAW can craft IPv6 router advertisements, and consequently spoof external IPv6 hosts, obtain sensitive information, or cause a denial of service. An attacker with those privileges can intercept traffic from other containers on the same host and from the host (Kubernetes node) itself.
By default, all containers have CAP_NET_RAW privileges. Therefore, all Kubernetes clusters with an affected networking implementation are vulnerable if they haven’t been hardened by ensuring that containers within Pods are running with dropped unnecessary capabilities.
To use this vulnerability an attacker must have access to at least one container within the cluster.
Are You Vulnerable?
Kubernetes itself is not vulnerable. A Kubernetes cluster using an affected networking implementation is vulnerable.
Given that this vulnerability requires an update to the CNI software, an update to the worker nodes is required. Oftentimes the upgrade cycles of the nodes, whether because the cluster is large or because the cluster runs stateful workloads, is a process that takes time which means that the system may be left vulnerable unless one or more of the mitigations below is applied.
Technical Break Down of the Exploit
In IPv6 the set of protocols called ICMPv6 and NDP (Neighbor Discovery Protocol) lets routers tell other computers/workloads how to route their traffic. This vulnerability uses “IPv6 Router Advertisements” which are a specific type of message in the NDP protocol.
Vulnerable Kubernetes CNIs allow containers to send IPv6 Router Advertisements and tell other containers how to route their traffic. This is allowed even if entities within the cluster don’t actually use IPv6. The basic attack chain would look as follows:
- An attacker gains access to one of your pods (with CAP_NET_RAW capability) – Let us call it POD_X.
- The attacker sends out IPv6 Router Advertisements that tell other pods and/or the K8s host that actually if you want to send traffic to MY_SECURE_ADDRESS the best way to do that is to send it to POD_X because POD_X is an IPv6 router which can send your traffic to MY_SECURE_ADDRESS
- A workload tries to access MY_SECURE_ADDRESS. Under certain conditions, containers that saw the Router Advertisement will decide to send traffic to POD_X.
The conditions which allows this:
- The HTTP library your application uses prefers (as many do) IPv6 over IPv4
- Your network has relevant IPv6 DNS records which an attacker can send Router Advertisements for.
- The attacker on POD_X can then see and manipulate all this traffic before forwarding it to the real MY_SECURE_ADDRESS.
How to Mitigate this Vulnerability with Alcide
- Disallow CAP_NET_RAW for untrusted workloads or users. The broader issue here is that any pod with CAP_NET_RAW capability can perform a lot of mischief on your network. Therefore, you should drop CAP_NET_RAW by default and use security guardrails such as the Alcide Runtime Admission controller to prevent accidental deployment of pods with CAP_NET_RAW and similar unnecessary default permissions. If specific containers actually need CAP_NET_RAW permissions then you can use Alcide’s policies to selectively allow it only when necessary. You can apply this as an admission control hook to prevent future issues and you can perform on-demand scans to find existing issues.
- Use a Pod Security Policy with a RequiredDropCapabilities including NET_RAW to prevent this attack for controlled workloads, and use Alcide Advisor to ensure such policies are in-place.
- Use TLS – a less practical workaround suggested in #91507 is to introduce the use of TLS with proper certificate validation. It will prevent an attacker from viewing sensitive data via this vulnerability, but it wont prevent them from triggering a Denial of Service attack via this vulnerability.
- Apply firewall policies to Pods – In the event where workloads can’t be denied the offending CAP_NET_RAW capability, network policies can block untrusted workloads from sending IPV6 Router Advertisements (as well as ARP spoofing). The Alcide Microservices Firewall, unlike the Kubernetes native policies, can implement whitelists that would block this attack regardless of the CNI in use. This is possible even for privileged pods that require connectivity to (cluster) external service providers.
- Disable ipv6 route advertisement by setting (with sysctl) net.ipv6.conf.all.accept_ra to 0 Set the host default to reject router advertisements. This should prevent attacks from succeeding, but may break legitimate traffic, depending upon the networking implementation and the network where the cluster is running.
- Update your CNI at the earliest possible time if it is affected by this vulnerability – consult the latest advisories
Lastly, every Kubernetes deployment must have a strategy for securing the underlying container network. If you rely on the default security mechanisms then not only are you vulnerable to this attack, but also other network attacks like ARP Spoofing, Port Scanning, and future vulnerabilities like this one. To have a secure Kubernetes network, you must monitor and protect your container network with a continuous monitoring solution and a microservices firewall.
Should I rely on Kubernetes to provide enough security controls?
The unsurprising answer is “you should not”. The issue here isn’t just this specific vulnerability. By default, Kubernetes doesn’t isolate your pods from one another! With certain CNIs you can even run ARP Spoofing within your cluster and achieve some of the same effects as this vulnerability via an easier mechanism.
In other words, if you rely on Kubernetes itself to protect the network between your containers then this vulnerability absolutely matters. If you view the Kubernetes network as inherently insecure then you probably have a security solution already in place which mitigates this issue already.
These are the exact places where the Alcide Kubernetes Security platform complements Kubernetes native security controls. Alcide provides a three-pronged protection strategy against vulnerabilities like these:
- Configuration – Alcide Advisor can harden your cluster and enforce policies like “Disallow CAP_NET_RAW for untrusted workloads or users” by using pre-flight checks of workload security configurations
- Runtime – The Alcide Runtime can protect you at runtime against network and process-based attacks. The Alcide Runtime supports custom policies and automatic detections using machine learning.
- Incident Response – Alcide kAudit performs threat detection and forensics by analysing the Kubernetes API Server audit log.
Technical Appendix on ICMPv6 and NDP
Here I’ll explain a little more about the NDP protocol which we discussed earlier.
In IPv6, there is no ARP protocol. The features of ARP, ICMP, and more were all folded into the ICMPv6 protocol. ICMPv6 supports ping messages just like ICMP. ICMPv6 also has a very important subprotocol called NDP (neighbor discovery protocol) which provides ARP-like features. NDP lets you take an IPv6 address and map it to a layer2 address just like ARP. However, NDP also lets routers broadcast the fact that they are routers and can route traffic to different networks – these messages are called IPv6 Router Advertisements. (Most people don’t realize it, but something very similar actually exists in regular IPv4 ICMP too – it just isn’t widely used because DHCP is so common and can do the same thing.) By using these Router Advertisements a pod can tell other pods that it can route traffic to whatever location it likes. You would think that if you don’t use IPv6 in your cluster then you wouldn’t be vulnerable to issues like this, but the implementation of many http libraries will actually prefer IPv6 addresses if they are available.