What eBPF Means for Container Threat Detection
This blog post was adapted from a talk at osquery@scale 2022 given by Ryan Mack (vice president of engineering and head of infrastructure at Uptycs) and Christopher Stanley (a security engineer in the aerospace industry).
eBPF (enhanced Berkeley Packet Filter) is a Linux kernel technology that offers a powerful and stable method of observing the Linux kernel. It’s like having a VM in the kernel that can safely run hooks (i.e. programs) for filtering data like network events, system calls, packets, and more. eBPF is being adopted at scale for its guaranteed stability, the ability to work directly in the kernel, and potential savings when factoring in the compute process for gathering telemetry on Linux servers and containers.
eBPF is rapidly gaining traction in cloud native applications, especially in places where traditional security monitoring doesn’t work. It’s eBPF is well suited for uses in distributed and container-based environments, including Kubernetes. The core benefits of the technology include speed and performance, a low level of intrusiveness, security, unified tracing, and programmability. It is safer than previous options because of the way it sees inside processes without introducing the risk of crashing the application or modifying the kernel in any way. eBPF is a preferred alternative to the audited framework because it is less invasive and more efficient.
By monitoring from the kernel layer, many of the challenges associated with observability in the cloud are solved. You can enjoy deeper visibility, more context, and more accuracy in your data. If you have an interest in increasing your container security, it’s worth learning more about what eBPF can do for you.
eBPF Adds Context Around Containers
Many people running containers in their environment wrongly assume it’s a security boundary and there’s no way that applications can break out of containers. I don’t see containers as security. They are like a barrier. They can be one piece of the puzzle to contain your application, but it is not by itself a security. For containers (as with everything else), you should follow the rule of least privilege, which in this case means only running the binaries that you need in the containers. You shouldn’t run root in the container, but the fact is, many developers do, and this gives other binaries a chance to container escape.
With eBPF, you get contextual information around your containers. You can learn what syscalls in the container were run, what host it was on, what the container name was, and what the image was. You can pump that information into your SIEM and have context around these hosts. Then you can start writing detections of abnormal behavior.
In the screenshots below, you can see that a process happened, what container name it was, who ran it, what the container’s name was, and so on. Without that context, it’s very hard to look at a host running 20,000 containers and identify which container has a security issue.
eBPF Telemetry Can Detect Unusual Activity
You might recall CVE-2022-0185. In this vulnerability, there was an underflow attack such that you could abuse spraying kmalloc to privilege escalate outside of an unprivileged namespace. Historically speaking, privilege escalation from a root user didn’t really matter. Before containers, if you were root and you were hitting this code that was in the kernel, that code would be less scrutinized. But that’s not necessarily the case anymore. When you have these root namespaces that are running within containers, sometimes you are hitting the code in the kernel that was less scrutinized, and that can allow an underflow attack to happen.
In the image below, I type through whoamI on my box. I am CStanley echo $ $. I get the process ID, and then I do a pscap | grep with that process ID to see what capabilities I have on the system. I don’t have any. So, I type unshare-r and enter a privileged namespace, and I type whoamI. I’m root, privilege escalated. But it’s not that easy. So, I echo $ $ again. I get the ID, I pscap | grep that, and I see I have full capabilities on this system now. I’ve privilege escalated!
Actually, no. In this context, in this namespace, the system is doing what it’s supposed to do. It’s isolated that namespace off, and it’s putting that small amount of protection around there saying, you’re not really root, you’re only in the context of this namespace. I try to change the root password and it fails. I try to install a binary, but it fails. I only have permissions in the context of that namespace.
This is where CVE-2022-0185 comes into play. If you pull down and run that binary and do this kmalloc spray, then it essentially does an underflow and passes a negative number in. It moves that pointer into the memory space to where it can execute code that allows you to privilege escalate. In the image below, I ran the exploit and get into the container namespace. Then I changed the root password and I have privilege escalated. This is a case where the container by itself failed us. It didn’t prevent us from affecting the host.
The image below shows a detection I did within osquery, utilizing eBPF telemetry. When I ran that same exploit, it shows there was a privilege escalation attack that occurred, and it detected the kthreadd. This detection triggered based on path two being spawned, and with kthreadd, that’s an indication that something happened within kernel space and that privilege escalated up. It’s a rudimentary detection but it’s effective.
eBPF is still evolving and will get better with time. Meanwhile, it’s already improved what’s possible with container threat detection. The examples given here used the open-source osquery technology, but the Uptycs solution extends osquery to provide similar container-level detections correlated with activity at the Kubernetes control plane.
Uptycs is sponsoring KubeCon + CloudNativeCon Europe 2023. Please stop by our booth to get a demo of our K8s and container security solution.
