Containers and microservices are driving enterprise IT innovation and digital transformation across industries. Companies are embracing container technologies like Kubernetes to realize greater flexibility, scalability, and speed across application development and deployment processes. Yet, for many enterprises, security remains one of the biggest concerns for kick-starting containerization initiatives.
This blog focuses on how we helped a Fortune 100 credit card company secure their Kubernetes clusters and effectively monitor security practices leveraging automation. The customer was looking to move their enterprise solution for risk and compliance to Kubernetes to meet expanding business needs. Here are some highlights of the security practices we implemented as part of this project:
01
Used an internal, private registry for container images
As hackers often prey on image vulnerabilities, using trustworthy registries (private, wherever possible) for container images is crucial. We helped the client set up an internal private registry for storing approved images. Any external images had to be downloaded using reverse proxies, validated, scanned, and pushed into the local repository. Their Kubernetes Admin team had permission to upload these to the internal repository after extensive security and compliance checks.
02
Integrated vulnerability scanning tools with the code build process
For continuous scanning of images and code, we integrated tools like Black Duck and WhiteHat with the code build process. This ensured that unsecured code was not included in any image. In addition, we performed image scanning leveraging Clair, an open-source tool that can be easily deployed in Kubernetes and integrated directly with the container registry.
03
Isolated environments using namespaces and role-based access control
To isolate environments for different teams and users, we leveraged Kubernetes namespaces. We also applied Kubernetes role-based access control (RBAC) on a per-namespace level, restricting the access to each Kubernetes service to specific users based on business needs. We also leveraged the Cluster Admin role to restrict cluster level access. Additionally, to streamline RBAC management and prohibit access by any unauthorized users, we periodically removed all unused/inactive roles.
04
Secured all Kubernetes components
The location of an enterprise’s database plays a critical role during application migration. If your database To secure all Kubernetes components, we enabled and configured Kubernetes RBAC and limited the number of users and service accounts accessing the API server. All traffic between the API server and other infrastructure components, like etcd and kubelet, were served over HTTPS (Transport Layer Security) and all communication was protected leveraging TLS encryption. The API server did not serve any requests on unsecure ports, and all server audit logs were collected and retained. To limit damage in the event of an attack, we locked down the ownership and permissions needed to access critical configuration and PKI files on the master node. In addition, we configured the kubelet config file to prevent the kubelet server from serving any anonymous/unauthenticated requests.
05
Leveraged egress policies and controllers for secure networking
To securely route network traffic to internal Kubernetes services, we used Nginx as a load balancer and network gateway for managing inbound and outbound connections. Nginx also served as a frontend proxy to limit the exposure of these services to end users.
06
Developed an automated solution for continuous monitoring
We developed an automated solution to continuously monitor Kubernetes’ security practices across multiple clusters with minimal effort. A high-level blueprint of the solution is given below:
High-level overview of the automated solution for continuous monitoring
This approach helped us create a robust migration plan backed by effective disaster management and sThis helped us periodically run the security benchmark in an automated manner and promptly identify any misconfigurations/incorrect practices. We also leveraged a security dashboard to perform security audits at the beginning of the project. A sample snapshot is given below:
Kubernetes security dashboard
Additionally, we stored Kubernetes events in the ELK Stack and generated insights on the dashboard. This helped proactively monitor any suspicious events and generate alerts.
Our integrated approach helped the credit card company improve their security posture by identifying security threats and misconfigurations across multiple clusters in real-time leveraging automation. This eliminated the need for manual monitoring, which would have involved massive time and effort. What’s more, the security dashboard provided a single, consolidated view of all critical security tests, enabling 360-degree visibility across clusters. The automated scanning utility provided daily updates on cluster stats, allowing the client’s Admin team to focus on other strategic tasks. Most importantly, the client was able to run their applications on secure, stable, resilient clusters.
