6 Overlooked Yet Important Kubernetes Features to Secure
If you’ve worked with Kubernetes, you know that for many it can be quite a complex machine. Between all of the moving parts, the networking, and often just getting the cluster up and running properly, it can take time. And by “properly,” I mean secured, with secure communication, proper network isolation, secure secrets, etc.
Unfortunately, getting the cluster up and running isn’t enough. Kubernetes isn’t safe by default. It needs further tweaking to ensure not only cluster safety at a high level, but pod and container safety as well.
In the 2022 State of Kubernetes Security Report by Red Hat, where 300 DevOps engineers and security experts were polled, it was reported that respondents “worry the most about exposures due to misconfigurations in their container and Kubernetes environments (46%) — nearly three times the level of concern over attacks (16%), with vulnerabilities as the second-leading cause of worry (28%).” Getting important configurations right, like role-based access control (RBAC) and security contexts, is vital to a cluster’s security posture. There are a number of factors that DevOps teams need to consider overall, as well as within the scope of their own unique situations.
And though normally DevOps teams handle cluster setup and maintenance, we can’t overlook the role application developers play as well, developers who may not have a good grasp on Kubernetes specifically. Once the cluster is up and ready, it’s sometimes the developers who are containerizing and deploying applications that could lead to possible misconfigurations at the pod and container levels.
So while we obviously can’t cover all the configurations in this post, I want to present a few initial strategies that teams can make from the outset of the cluster deployment that will greatly increase their security posture. By implementing these six solutions after cluster setup, teams can increase their security drastically.
1. Etcd Security
Securing etcd is critical to Kubernetes security. All cluster data, including secrets, is kept in this highly available key-value store. Gaining read or write access to etcd is basically obtaining the keys to the kingdom. So you need to plan protection for this vital backend from the outset. It should be noted that a benefit of Kubernetes-as-a-Service (EKS, AKS, etc.) is that this security measure is managed for you.
There are three things to check when securing your etcd:
- Encrypt data at rest. By default, etcd data is not encrypted. An EncryptionConfiguration object needs to be created for etcd data to be encrypted at rest. Key rotation for this encryption is also supported and recommended in collaboration with third-party key management services. More information on this can be found in the documentation.
- Restrict access. It is recommended to separate the etcd server(s) and isolate behind a firewall. From there, only access from the API server should be allowed. No other components in the cluster should be allowed direct access to it.
- Ensure that the API server is using TLS.
2. Network Policies
What if the cluster layer is breached at some point? We would want as little horizontal infiltration to be allowed as possible, right? Well, Kubernetes, by default, allows all pod-to-pod traffic. All inbound and outbound connections between pods are allowed. If attackers get access to a pod, they can then move to other pods unhindered.
To prevent this, network policies should be put in place. At the network and transport layers, network plugins like Weave Net or Kube-router can be used to restrict unnecessary pod-to-pod access via namespaces and labels. Even better, pod traffic can be controlled at the application layer via a service mesh like Istio.
Image from https://istio.io/latest/blog/2017/0.1-using-network-policy/
For a simple example, if we have an application with a frontend, a backend and a database, the frontend ideally should not be allowed to talk directly to the database, nor the database to the frontend. Instead, the frontend should only be allowed to talk to the backend and, in turn, the backend to the database. If attackers access the frontend application, they won’t be able to touch the database directly without going through the backend, thus complicating the scope of penetration.
3. Pod-to-Pod Communication
In addition to open networks, pod-to-pod communication is unencrypted by default. This means anyone who is able to infiltrate the cluster can listen to communications in plain text. Again, introducing a service mesh like Istio can be used to enforce mutual TLS between services, by setting its configuration to STRICT mode, disallowing plain text. This can be confined to namespace or the entire mesh depending on your use case.
4. Secrets
A secure secrets solution should be formulated in the earliest stages. However, that’s often not the case. In fact, due to the complexity of standing up a Kubernetes cluster and getting applications running, secret management on many occasions makes its way to the back burner until more prominent tasks are ironed out.
The problem is that Kubernetes, though providing us with a secrets framework, doesn’t help us much with securing them. By default, they are only Base64 encoded, which can be easily converted to plain text. This framework does accommodate encrypting secrets at rest when written to etcd; when it’s combined with proper RBAC restrictions and tight etcd security, it can be made more secure. However, this is still not ideal for a production environment.
Thus, the sooner a team tackles the “how” of secure secrets management, the better off their security will become. Most teams would do well to use a third-party solution, whether cloud solutions like AWS Secrets Manager or open source tools like HashiCorp Vault. These will provide a more robust and secure system for vital Kubernetes secrets management.
5. Pod-Level Security/Dirty YAML
The same due diligence needs to be applied at the app level. As mentioned earlier, there are talented application developers who will take their apps, containerize them and deploy them without a good grasp of the inner workings or ideal settings of a Kubernetes cluster. Helm charts from public repos or public Docker images may contain root-level access or other potentially critical configurations that could weaken security.
Therefore, it’s important that YAML configurations and containers coming into the ecosystem be audited. Whether this is performed manually by a security team or through a more automated approach like vulnerability scanning, this can help catch potential loopholes before they come in.
In addition, we should use the Kubernetes security context at the pod or container level to block any images running as root. We can add a security context entry into the YAML with runAsNonRoot as true, and the Kubelet will refuse to start any image that defaults to the root user. Of course, taking a step back, images should be assigned a user and group UID, but adding this security context will add another layer of defense if not. There are other security content settings that may be of use to your situation, as seen in this cheat sheet.
6. Kubernetes Access
Finally, let’s discuss access itself via the API server. Any user coming to the API server with a valid certificate is technically considered authenticated. Given this fact, there are few suggestions here:
- First, in most cases, anonymous auth should be disabled by setting
--anonymous-auth=falsefor the API server. RBAC should be set up to handle specific users. However, if anonymous auth must be used, RBAC should greatly limit what anonymous users can do as a whole. - Second, by default, the API server listens on two ports: one secure and the other an insecure “localhost” port. You should disable the insecure port by setting
--insecure-portflag to “0” and ensuring that the--insecure-bind-addressis not set. This insecure port will actually bypass authentication and authorization checks, something you do not want in place if someone malicious were to gain access to the master server. - Next a secure and well-maintained RBAC system should be set up and in place at all times.
- Finally, admins should consider managing normal user accounts with corporate solutions, like Active Directory, Okta, etc., via OpenID Connect.
An open source application like Teleport provides secure access to Kubernetes clusters by way of short-lived kubeconfig files (and certificates) via single sign-on. In addition, Teleport provides additional tools like a built-in RBAC system and auditing and session recordings of kubectl events not only for your cluster, but for tens, hundreds, thousands of clusters in one graphical user interface. Roles and policies can be put in place for users to access clusters easily and securely. In fact, users never directly access the clusters, which can sit isolated in a private network or behind a firewall. Direct interaction is only with Teleport’s proxy service. More information can be found in the documentation.
Conclusion
So if you are spinning up a new cluster or have done so recently, remember that it isn’t overly safe by default. You will need to put in some work to secure it further. Implementing the six strategies above to secure these features of your Kubernetes cluster from the beginning will help greatly improve the security of your cluster and the success of your applications overall.
