Users are able to do their job with the highest velocity possible
Users organized by groups in multi tenant setup
Kubernetes namespaces help to setup boundaries between groups of users and applications in a cluster. To make it more pleasant and secure for your users to work in shared cluster Kubernetes has a number of policies and controls.
RBAC primary objective is authorize users and applications to do specific operations in the namespace or in whole cluster. Use RBAC to give your users enough permissions in the namespace, so they can do day to day operations on their own. Network Policy control how pods can communicate with each other. Use it to firewall traffic between namespaces or inside namespace to critical components like Databases.
By default Pod can utilize as many compute resources as available. Resource Quotas control the amount of compute and storage resources which Pod can use in namespace. Limit Range help to prevent one Pod from utilize of all resources in namespace. LimitRange set minimum and maximum boundaries for compute and storage resource per Pod.
Pod security policy control security sensitive aspects of container. Examples are privileged containers, use of host namespace and many other. Open Policy Agent is very powerful policy framework which help to create custom policies for applications and users in a cluster. For example:
force users to use a specific label in Kubernetes objects like Service or Deployment
deny access to pull :latest images tag
allow to pull images only from specific docker registry
Following examples could help you to decide on namespaces boundaries and naming:
Namespace per team
Namespace per team and project
Namespace per application
Namespace per git branch name
Namespace should provide enough self managing autonomy for users and be in sync with applications requirements. The
bigger namespace the harder to tune up it’s boundaries, at the same
time many small namespaces could create additional operational work for
Namespace per team and project is optimal start which should work for most organizations.
Let me know your experience in comments and have a great day!
My journey with Kubernetes started with Google Kubernetes Engine then one year later with self managed kuberntes and then with migration to Amazon EKS.
EKS as a managed kubernetes cluster is not 100% managed. Core tools didn’t work as expcted. Customers expectation was not aligned with functions provided. Here I have summarized all our experience we gained by running EKS cluster in Staging.
To run EKS you still have to:
Prepare network layer: VPC, subnets, firewalls…
Install worker nodes
Periodically apply security patches on workers nodes
Monitor worker nodes health by install node problem detector and monitoring stack
Setup security groups and NACLs
EKS Staging how to?
Use terraform EKS module or eksctl to make installation and maintenance easier.
Install node problem detector to monitor for unforeseen kernel or docker issues
Kubernetes cluster autoscaling is no doubt must have addition to EKS toolkit. Scale your cluster up and down to 0 instances if you wish. Base your scaling on cluster state of Pending/Running pods to get maximum from it.
Kubernetes custom metrics, node exporter and kube state metrics is must have to enable horizonal pod autoscaling based on build in metrics like cpu/memory and as well on application specific metrics like request rate or data throughput.
Prometheus and cadvisor is another addition you would need to enable metrics collection
Istio one of the most advanced, but breaking changes and beta status might introduce hard to debug bugs
Contour looks like good replacement to Istio. It didn’t have that good community support as istio, but stable enough and has quite cool CRD IngressRoute which makes Ingress fun to use
Nginx ingress is battle tested and has the best support from community. Have huge number of features, so is a good choice to setup the most stable environment
Ensure you have enough nodes in each AZ where data volumes are. Good start is to create dedicated node group for each AZ with minimum number of nodes needed.
Ensure persistent volume claim(PVC) is created in desired AZ. Create dedicated storage class for specific AZ you need PVC to be in. See allowedTopologies in following example.
EKS is a good managed Kubernetes service. Some of mentioned tasks are common for all Kubernetes platforms, but there is a lot of space to grow for the better service. The burden for maintenance is still quite high, but fortunately Kubernetes ecosystem has a lot of opensource tools to easy it.
I learn about sidecar pattern from Kubernetes documentation and later from blog post by Brendan Burns The distributed system toolkit. Sidecar is very useful pattern and work nice with Kubernetes.
In the tutorial I want to demonstrate how “legacy” application can be extend with https support by using sidecar pattern based on Kubernetes.
We have legacy application which doesn’t have HTTPS support. We also don’t want to send plain text traffic over network. We don’t want to make any changes to legacy application, but good thing that it is containerised.
We will use sidecar pattern to add HTTPS support to “legacy” application.
For our example main application I will use Nodejs Hello World service (beh01der/web-service-dockerized-example)
First we need to generate TLS certificate keys and add them to Kubernetes secrets. For that I am using script from nginx ssl proxy repository which combine all steps in one:
git clone https://github.com/ployst/docker-nginx-ssl-proxy.git
Adding TLS files to Kubernetes secrets
kubectl create secret generic ssl-key-secret --from-file=proxykey=proxykey --from-file=proxycert=proxycert --from-file=dhparam=dhparam
Kubernetes sidecar deployment
In following configuration I have defined main application container “nodejs-hello” and nginx container “nginx”. Both containers run in the same pod and share pod resources, so in that way implementing sidecar pattern. One thing you want to modify is hostname, I am using not existing hostname appname.example.com for this example.
Save this file to deployment.yaml and create deployment Kubernetes object:
kubectl create -f deployment.yaml
Wait for pods to be Read:
kubectl get pods
NAME READY STATUS RESTARTS AGE
nodejs-hello-686bbff8d7-42mcn 2/2 Running 0 1m
For testing I setup two port forwarding rules. First is for application port and second for nginx HTTPS port:
kubectl -n test port-forward <pod> 8043:443
#and in new terminal window run
kubectl -n test port-forward <pod> 8030:3000
First lets validate that application respond on http and doesn’t respond on https requests
curl -k -H "Host: appname.example.com" http://127.0.0.1:8030/
I am undefined!
#now using https
curl -k -H "Host: appname.example.com" https://127.0.0.1:8030/
curl: (35) Server aborted the SSL handshake
Note: SSL handshake issue is expected as our “legacy” application doesn’t support https and even if it would it must serve https connection on different port than http. The test goal was to demonstrate the response.
Time to test connection through sidecar nginx ssl proxy
curl -k -H "Host: appname.example.com" https://127.0.0.1:8043/
I am undefined!
Great! We have got expected output through https connection.
Nginx extended nodejs app with https support with zero changes to any of containers
Sidecar pattern modular structure provide great re-use of containers, so teams can be focused on application development
Ownership of containers can be split between teams as there is no dependency between containers
Scaling might not be very efficient, because sidecar container have to scale with main container