The New Open Source Security Stack
Recognizing that there is no such thing as perfect security, practitioners like to layer up to increase the chances of keeping the bad guys at bay, so-called defense in depth strategies.
Container environments present some new challenges, so require a few additional security layers.
Fortunately, there are a lot of innovations to leverage in the open source world, including many advances in the Linux kernel that give us tighter control over what activities are permitted at this core level. That’s a good thing because containers change the security game in some unique ways.
For one, developers have broad latitude when it comes to deciding what to package up in a container — the language choice, the run time, what data store they’re going to use, etc. That can turn some hairs gray for operations folks because they don’t necessarily know what’s in those containers they just pushed into production.
And then there is the ephemeral nature of containers. They come and go as environments get scaled up and down. But what if one got compromised? Even if the container was destroyed shortly after use, what if that compromise enabled the attacker to branch out into other parts of the environment before it was trashed?
Here’s a review of some of the open source tools available — for the infrastructure, build and runtime layers of the container security stack — to help you safeguard your environment against such threats.
Obviously, this is only a sampling intended to whet your appetite because there are many more open source container security options out there. And equally obvious, I hope, is the fact these tools aren’t a replacement for your basic firewalls and other traditional security tools. They are additional layers you can add to your defense-in-depth strategy.
Infrastructure
On the infrastructure front, a good place to start is with policies and network filtering.
* Pod security policies: For expediency sake, many developers simply skip over many of the basic policy-based security controls in Kubernetes, which is a shame because turning on even some of the most rudimentary checks will vastly improve your container security posture. There are open source projects, such as Kubernetes Pod Security Policy Advisor, that will scan existing security context from Kubernetes resources like deployments, daemonsets, and replica sets and then automatically generate the recommended Pod Security Policy for the entire cluster.
Just as we can restrict host resources containers can consume, such as memory and CPU, we can use Pod security policies to get very granular about what Linux kernel resources a container can access. This is often done through Kubernetes using an object called the pod security policy. You define that policy, and then when you define your pod you direct the pods to subscribe to those policies.
Policies can be quite involved (maybe the reason they get skipped over so often?), enabling you to specify things such as whether a container can escalate privileges, what parts of the kernel it can access, and whether or not the ownership of a file can be changed.
You can also restrict volumes a container can mount (so they can only use these particular objects), restrict groups it can use, specify if the file system is read-only or not, and dictate whether the container is going to run as at root user or a non-root user. If you do nothing else, restricting the root access will greatly improve the security of your container environment.
Remember, however, that even if you spend the time to spell out the security policies, the pod still has to subscribe to the policy for the controls to kick in. But there are other options in the toolbox to help ensure the controls you want in place are in fact enabled, as we’ll discuss.
* Cilium for network filtering. A second basic infrastructure open source security tool you might want to consider is Cilium for its network filtering capabilities. Cilium is described as bringing “API-aware network security filtering to Linux container frameworks like Docker and Kubernetes.”
Basically, Cilium is a kernel add-on (via BPF) that allows you to filter network traffic at Layer 3, Layer 4 and even Layer 7. You create filter rules and, using an endpoint selector, specify where you want the rules to apply.
Rules can specify what is allowed in and out of a container. For example, the rule might allow inbound traffic from anyone with the organization of “Empire,” but only to port 80. So, the Empire can access port 80, but the Star Wars rebels who don’t have Empire as a defining organization attribute would be blocked.
And you can get more granular. Say the Empire is allowed to access port 80, but you only allow Empire to post data to a particular URL. If they try to do a Git on any other URL, they will be denied.
That’s useful if you have a service that handles sensitive data and you only want certain URL endpoints to be able to access certain hosts or containers. You can use Cilium to say, “I only want to accept requests from my banking frontend, and that frontend should only ever make these 10 requests to these 10 URLs.” That effectively establishes a gate that prevents anyone from accessing anything else.
Build
When it comes to building containers there are many security options we want to consider, including image scanning, admission controllers and another layer of policies.
* Anchore for image scanning. Scanning container images is a critical step given that a compromised image can lead to a boatload of compromised containers in short order.
You are going to want to know the provenance of your container image and need to consider things like signing, trusted registers and admittance control. But when it comes to the image itself, you’re going to need to be able to scan a fully built image as an immutable artifact, ensuring it lives up to security best practices and checks all the compliance boxes.
Anchore is described as “an open container compliance platform for discovering, analyzing, and certifying container images on-premises or in the cloud.”
Anchore runs as a service with an API and has a broad, customizable, policy-based method for doing scans. You incorporate Anchore as part of your build engine. If you use Jenkins, for example, you would build your app with Jenkins and then build the container image and run that through Anchore analyzer.
The nice thing is you can use Anchore policies to define what should happen if something is found. For example, you can have Anchore kick out a warning if a low-security vulnerability is discovered, require images with a moderate vulnerability to get another review and fail outright anything with a severe vulnerability.
*Kubernetes Admission controllers. Admission controllers may also support what are known as mutating webhooks. So, if you’re doing a native deployment and throw out a webhook to Anchore or a notary and ask if the image is safe to use, the controller can flag that image as faulty and instead recommend an updated version that is safe and sound.
* Kubesec.io for deployment checks. Image scanning is just a point in time compliance check. You also need to be worried about the deployment you’re shipping into Kubernetes. And, as discussed, you also want to ensure container pods are, in fact, subscribed to the security policies you require.
That’s where a tool like Kubesec.io comes in handy. Kubesec.io allows you to check your Kubernetes deployments to see whether you’re running things that are not advisable (testing and then giving you back info about what may be wrong) and check to make sure you have the proper security policies in place (or if you’re defining bad policies).
Think of Kubesec.io as kind of an enforcement mechanism you specify as a test in the build phase. You have the Kubernetes deployment YAML define how your container will be deployed — including all of the configuration requirements such as the name of the app, labels for the app, the image to use, number of instances, etc. — but before it all goes live a call is issued to Kubesec.io to see if everything measures up.
Runtime
Container images are immutable, but once you launch the container, if you aren’t running on a read-only file system, people can go in and make changes. That’s why it’s important to also have a runtime tool to detect abnormal behavior.
* Falco for anomalous behavior detection. Consider Falco, which is a rules-based detection engine for runtime security within cloud native architectures (and a CNCF Sandbox project). Falco is said to provide “visibility into the behavior of your containers and applications.”
Containers tend to run isolated processes, so you have expectations about what that container should be doing. You don’t expect processes to be spawning, or random ports to open or random outbound internet connections to be created. Falco watches for these actions and generates alerts when something suspicious pops up.
While there are tools out there that allow you to do host intrusion detection, they don’t pull in data from orchestrators such as Kubernetes or from the Docker runtime engine, so they’re mostly ignorant when it comes to containers.
With Falco you can create security playbooks so that, when Falco detects a suspicious behavior, you have a plan in place to take action on the alert, whether that involves killing the pod, tainting the Kubernetes node so new workloads can’t be scheduled, or simply notifying someone via your channel of choice (Slack, etc.).
*Sysdig Inspect for Forensics: Sysdig Inspect is an Electron-based GUI for system call analysis. The open source project uses capture rules to provide system-level observability.
Forensics boils down to a data challenge. It is important for enterprises to have access to data on all containers, including those that have been killed, along with context so that they can quickly determine the issue and remediate it. Sysdig Inspect provides streamlines forensics, allowing for easy drill into views for process activity, file system activity, network activity and more for detail post-mortem analysis.
Inspect’s user interface is designed to intuitively navigate the data-dense sysdig captures that contain granular system, network, and application activity of a Linux system. It comes with features designed to support both performance and security investigations, with deep container introspection.
Do Something!
The important takeaway here is containers change the security calculus and you need to revisit your defense-in-depth strategy and identify new tools that can be layered on to secure these fast-changing environments.
The good news: There are many open source tools you can employ to secure that environment.
The bad news: Resource isolation in the container world is not as strong as in the virtual machine world. If one container is compromised it is possible the attackers will find it relatively easy to escalate privileges into the host operating system and from there get into any and all other local containers.
You need a multilayered approach to container security, and you need to enforce usage of the policies you build using these or other open source tools.
