You can use Kubernetes namespaces to set resource requests and limits to ensure that your clusters have the correct resources for optimal performance.
Kubernetes is a powerful container orchestration system that makes it attractive to organizations, including its ability to automatically scale containerized workloads and automate deployments. However, the ease of deploying and scaling cloud applications can lead to skyrocketing expenses if not managed correctly. So cost optimization is an important consideration when it comes to running a Kubernetes cluster.
You can manage the costs associated with a Kubernetes cluster in several ways, for example, by using lower-cost hardware for nodes, cheaper storage options or a lower-cost networking solution. However, these cost-saving measures inevitably affect the performance of the Kubernetes cluster. So before downgrading your infrastructure, it’s worth exploring a different alternative. Leveraging namespaces’ ability to organize and manage your resources in Kubernetes is one option that can help your organization save costs.
In this article, you’ll learn about the following:
Kubernetes namespaces and their role from a cost optimization perspective.
Identifying resource usage in namespaces.
Resource quotas and limit ranges.
Setting up resource quotas and limit ranges in Kubernetes.
Benefits of x-as-a-service (XaaS) solutions with built-in cost optimization features.
Kubernetes Namespaces: What They Are and Why They Are Useful for Cost Optimization
You can think of namespaces as a way to divide a Kubernetes cluster into multiple virtual clusters, each with its own set of resources. This allows you to use the same cluster for multiple teams, such as development, testing, quality assurance or staging.
Kubernetes namespaces are implemented as a set of labels on objects in the cluster. When you create a namespace, you specify a name that identifies it and a set of labels to select the objects that belong to it.
You can use namespaces to control access to the cluster. For example, you can allow developers to access the development namespace but not the production namespace. This can be done by creating a role that has access to the development namespace and adding the developers to that role.
You can also use namespaces to control the resources that are available to the applications that run on them. This is done through resource quotas and limit ranges, two objects discussed later in this article. Setting such resource limits is invaluable in terms of cost optimization because it prevents resource waste and thus saves money. Moreover, with proper monitoring, inactive or underused namespaces could be detected and shut down if necessary to save even more resources.
In short, you can use Kubernetes namespaces to set resource requests and limits to ensure that your Kubernetes clusters have enough resources for optimal performance. This will minimize over-provisioning or under-provisioning of your applications.
Identifying Namespace Resource Usage
Before you can right-size your applications, you must first identify namespace resource usage.
In this section, you’ll learn how to inspect Kubernetes namespaces using the kubectl command line tool. Before proceeding, you’ll need the following:
kubectl installed and configured on your local machine.
Access to a Kubernetes cluster with Metrics Server installed. The Kubernetes Metrics Server is indispensable for collecting metrics and using the kubectl top command.
This repository cloned to a suitable location on your local machine.
Inspecting Namespaces Resources Using kubectl
Start by creating a namespace called ns1:
$ kubectl create namespace ns1
namespace/ns1 createdNext, navigate to the root directory of the repository you just cloned and deploy the app1 application in the ns1 namespace, as shown below:
$ kubectl apply -f app1.yaml -n ns1
deployment.apps/app created
service/app createdapp1 is a simple php-apache server based on the registry.k8s.io/hpa-example image:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app1
labels:
app: app1
spec:
replicas: 5
selector:
matchLabels:
app: app1
template:
metadata:
name: app1
labels:
app: app1
spec:
containers:
- name: app1
image: registry.k8s.io/hpa-example
ports:
- containerPort: 80
resources:
limits:
cpu: 500m
requests:
cpu: 200m
memory: 8Mi
---
apiVersion: v1
kind: Service
metadata:
name: app1
labels:
app: app1
spec:
ports:
- port: 80
selector:
app: app1As you can see, it deploys five replicas of the application, which listens on port 80 through a service called app1.
Now, deploy the app2 application in the ns1 namespace:
$ kubectl apply -f app2.yaml -n ns1
deployment.apps/idle-app createdapp2 is a dummy app that launches a BusyBox-based application that waits forever:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app2
spec:
replicas: 1
selector:
matchLabels:
app: app2
template:
metadata:
labels:
app: app2
spec:
containers:
- name: busybox
image: busybox
command:
- /bin/sh
- -c
- "while true; do sleep 30; done"You can now use the command kubectl get all to check all the resources that the ns1 namespace uses, as shown below:
$ kubectl get all -n ns1
NAME READY STATUS RESTARTS AGE
pod/app1-785668c957-95kmv 1/1 Running 0 9s
pod/app1-785668c957-bnlvz 1/1 Running 0 9s
pod/app1-785668c957-d6mxt 1/1 Running 0 9s
pod/app1-785668c957-gbfvv 1/1 Running 0 9s
pod/app1-785668c957-pgrjv 1/1 Running 0 9s
pod/app2-77bd8884d6-tmplz 1/1 Running 0 5s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/app1 ClusterIP 10.245.27.210 <none> 80/TCP 9s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/app1 5/5 5 5 9s
deployment.apps/app2 1/1 1 1 9s
NAME DESIRED CURRENT READY AGE
replicaset.apps/app1-785668c957 5 5 5 9s
replicaset.apps/app2-77bd8884d6 1 1 1 8sSince you have Metrics Server installed, you can also use the `top pods` command to check the resource consumption of pods in the `ns1` namespace, as shown below:
$ kubectl top pods -n ns1
NAME CPU(cores) MEMORY(bytes)
app1-785668c957-95kmv 1m 8Mi
app1-785668c957-bnlvz 1m 8Mi
app1-785668c957-d6mxt 1m 8Mi
app1-785668c957-gbfvv 1m 8Mi
app1-785668c957-pgrjv 1m 8Mi
app2-77bd8884d6-tmplz 1m 0MiAs you can see, by using the kubectl command line tool, you can take a quick look at the activity within the namespace, list the resources used, and get an idea of the pods’ CPU cores and memory spending. Additionally, you can use the command kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace> to get an idea of how often the resources in the namespace are used:
$ kubectl api-resources --verbs=list --namespaced -o name \
| xargs -n 1 kubectl get --show-kind --ignore-not-found -n ns1
NAME DATA AGE
configmap/kube-root-ca.crt 1 22h
NAME ENDPOINTS AGE
endpoints/app1 10.244.0.11:80,10.244.0.110:80,10.244.0.19:80 + 2 more... 63m
...output omitted...
41m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-774c558d94 to 1
NAME READY STATUS RESTARTS AGE
pod/app1-788dc7b9bc-2lmc4 1/1 Running 0 63m
pod/app1-788dc7b9bc-6qzl9 1/1 Running 0 63m
pod/app1-788dc7b9bc-c2jwn 1/1 Running 0 63m
pod/app1-788dc7b9bc-pf4ds 1/1 Running 0 63m
pod/app1-788dc7b9bc-wl7xp 1/1 Running 0 63m
pod/app2-774c558d94-pt978 1/1 Running 0 41m
NAME TYPE DATA AGE
secret/default-token-2htgh kubernetes.io/service-account-token 3 22h
NAME SECRETS AGE
serviceaccount/default 1 22h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/app1 ClusterIP 10.245.5.183 <none> 80/TCP 64m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/app1 5/5 5 5 64m
deployment.apps/app2 1/1 1 1 42m
NAME DESIRED CURRENT READY AGE
replicaset.apps/app1-788dc7b9bc 5 5 5 64m
replicaset.apps/app2-774c558d94 1 1 1 42m
NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6
ciliumendpoint.cilium.io/app1-788dc7b9bc-2lmc4 2723 22306 ...output omitted...
ready 10.244.0.87
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
endpointslice.discovery.k8s.io/app1-kbvj9 IPv4 80 10.244.0.110,10.244.0.11,10.244.0.29 + 2 more... 64m
LAST SEEN TYPE REASON OBJECT MESSAGE
42m Normal Scheduled pod/app2-774c558d94-pt978 Successfully assigned ...output omitted...
46m Warning BackOff pod/app2-85dcc749c7-dmm2n Back-off restarting failed container
54m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 621.668342ms
52m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 200.910627ms
50m Normal Pulled pod/app2-85dcc749c7-dmm2n Successfully pulled image "busybox" in 273.989882ms
56m Normal SuccessfulCreate replicaset/app2-85dcc749c7 Created pod: app2-85dcc749c7-dmm2n
56m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-85dcc749c7 to 1
42m Normal ScalingReplicaSet deployment/app2 Scaled up replica set app2-774c558d94 to 1
NAME CPU MEMORY WINDOW
podmetrics.metrics.k8s.io/app1-788dc7b9bc-2lmc4 55271n 8952Ki 10.279s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-6qzl9 47321n 8956Ki 16.436s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-c2jwn 53688n 8972Ki 12.29s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-pf4ds 57384n 9016Ki 19.875s
podmetrics.metrics.k8s.io/app1-788dc7b9bc-wl7xp 57195n 8980Ki 18.183s
podmetrics.metrics.k8s.io/app2-774c558d94-pt978 0 316Ki 16.729sThis command lists the resources in use as well as the activity time of each. It can also help detect some status messages like Back-off restarting failed container, which could indicate problems that need to be addressed. Checking the endpoint activity messages is also useful for inferring when a namespace or workload has been idle for a long time, thus identifying resources or namespaces that are no longer in use and that you can delete.
That said, other situations can also lead to wasted resources. Let’s go back to the output of kubectl top pods -n ns1:
$ kubectl top pods -n ns1
NAME CPU(cores) MEMORY(bytes)
app1-788dc7b9bc-2lmc4 1m 8Mi
app1-788dc7b9bc-6qzl9 1m 8Mi
app1-788dc7b9bc-c2jwn 1m 8Mi
app1-788dc7b9bc-pf4ds 1m 8Mi
app1-788dc7b9bc-wl7xp 1m 8Mi
app2-774c558d94-5mk8m 0m 0MiImagine if app2 was a new feature test that someone forgot to remove. This might not seem like much of a problem, as its CPU and memory consumption are negligible; however, left unattended, pods like this could start stacking up uncontrollably and hurt the control-plane scheduling performance. The same issue applies to app1; it consumes almost no CPU, but since it has no set memory limits, it could quickly consume resources if it starts scaling.
Fortunately, you can implement resource quotas and limit ranges in your namespaces to prevent these and other potentially costly situations.
Resource Quotas and Limit Ranges
This section explains how to use two Kubernetes objects, ResourceQuota and LimitRange, to minimize the previously mentioned negative effects of pods that have low resource utilization but the potential to fill your clusters with requests and resources that are not used by the namespace.
According to the documentation, the ResourceQuota object “provides constraints that limit aggregate resource consumption per namespace,” while the LimitRange object provides “a policy to constrain the resource allocations (limits and requests) that you can specify for each applicable object kind (such as pod or PersistentVolumeClaim) in a namespace.”
In other words, using these two objects, you can restrict resources both at the namespace level and at the pod and container level. To elaborate:
ResourceQuota allows you to limit the total resource consumption of a namespace. For example, you can create a namespace dedicated to testing and set CPU and memory limits to ensure that users don’t overspend resources. Furthermore, ResourceQuota also allows you to set limits on storage resources and limits on the total number of certain objects, such as ConfigMaps, cron jobs, secrets, services and PersistentVolumeClaims.
LimitRange allows you to set constraints at the pod and container level instead of at the namespace level. This ensures that an application does not consume all the resources allocated via ResourceQuota.
The best way to understand these concepts is to put them into practice.
Because both ResourceQuota and LimitRange only affect pods created after they’re deployed, first delete the applications to clean up the cluster:
$ kubectl delete -f app1.yaml -n ns1 && kubectl delete -f app2.yaml -n ns1
deployment.apps "app1" deleted
service "app1" deleted
deployment.apps "app2" deletedNext, create the restrictive-resource-limits policy by deploying a LimitRange resource:
$ kubectl apply -f restrictive-limitrange.yaml -n ns1
limitrange/restrictive-resource-limits createdThe command above uses the following code:
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "restrictive-resource-limits"
spec:
limits:
-
type: "Container"
max:
memory: "20Mi"
cpu: "1"
min:
memory: "10Mi"
cpu: "1m"As you can see, limits are set at the container level for the maximum and minimum CPU and memory usage. You can use kubectl describe to review this policy in the console:
$ kubectl describe limitrange restrictive-resource-limits -n ns1
Name: restrictive-resource-limits
Namespace: ns1
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu 1m 1 1 1 -
Container memory 10Mi 20Mi 20Mi 20Mi -Now try to deploy app1 again:
$ kubectl apply -f app1.yaml -n ns1
deployment.apps/app1 created
service/app1 createdThen, check deployments in the ns1 namespace:
$ kubectl get deployment -n ns1
NAME READY UP-TO-DATE AVAILABLE AGE
app1 0/5 0 0 1mThe policy implemented by restrictive-resource-limits prevented the pods from being created. This is because the policy requires a minimum of 10 mebibytes (Mi) of memory per container, but app1 only requests 8 Mi. Although this is just an example, it shows how you can avoid cluttering up a namespace with tiny pods and containers.
Let’s review how limit ranges and resource quotas can complement each other to achieve resource management at different levels. Before continuing, delete all resources again:
$ kubectl delete -f restrictive-limitrange.yaml -n ns1 && kubectl delete -f app1.yaml -n ns1Next, deploy the permissive-limitrange.yaml and namespace-resource-quota.yaml resources:
$ kubectl apply -f permissive-limitrange.yaml -n ns1
$ kubectl apply -f namespace-resource-quota.yaml -n ns1
limitrange/permissive-resource-limits created
resourcequota/namespace-limits createdThe new resource management policies should look as follows:
$ kubectl describe limitrange permissive-resource-limits -n ns1
$ kubectl describe resourcequota namespace-limits -n ns1
Name: permissive-resource-limits
Namespace: ns1
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory 6Mi 20Mi 20Mi 20Mi -
Container cpu 1m 1 1 1 -
Name: namespace-limits
Namespace: ns1
Resource Used Hard
-------- ---- ----
limits.cpu 0 2
limits.memory 0 2Gi
pods 0 5
requests.cpu 0 1
requests.memory 0 1GiAccording to permissive-resource-limits, there should be no problem deploying app1 this time:
$ kubectl apply -f app1.yaml -n ns1
deployment.apps/app1 created
service/app1 createdCheck the resources in the ns1 namespace:
$ kubectl get all -n ns1
NAME READY STATUS RESTARTS AGE
pod/app1-5579c6cdb4-5pb2h 1/1 Running 0 11m
pod/app1-5579c6cdb4-cqtrh 1/1 Running 0 11m
pod/app1-5579c6cdb4-fgm8q 1/1 Running 0 11m
pod/app1-5579c6cdb4-s97zk 1/1 Running 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/app1 ClusterIP 10.245.197.52 <none> 80/TCP 11m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/app1 4/5 4 4 11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/app1-5579c6cdb4 5 4 4 11mYou may be wondering why only four out of five pods were deployed. The answer lies in the CPU limits of the resource quota. Each container requests 500 CPU millicores, and the namespace limit is two cores. To put it another way, this policy only allows you to create four pods totaling 2,000 millicores (two cores).
The same principle used to prevent over-provisioning of a namespace can be used to prevent under-provisioning.
Scope of LimitRange and ResourceQuota in Resource Management
You’ve seen how to use segmentation in namespaces and LimitRange and ResourceQuota policies to optimize costs. This section addresses the other side of the coin — the limitations and the pros and cons of such policies.
Limitations of LimitRange and ResourceQuota
Kubernetes documentation is very clear when it comes to the scope of LimitRange and ResourceQuota.
LimitRange policies are intended to set bounds on resources such as:
Containers and pods, where you can set minimum, maximum and default request values for memory and CPU per namespace.
PersistentVolumeClaims, where you can set minimum and maximum storage request values per namespace.
Additionally, according to the documentation, you can “enforce a ratio between request and limit for a resource in a namespace.”
A ResourceQuota, on the other hand, also allows you to set minimum and maximum compute resource values, but in the context of a namespace. Moreover, it also allows you to enforce other aspects at the namespace level, such as:
The total number of PersistentVolumeClaims that can exist in the namespace.
The total space to be used in the namespace for PersistentVolumeClaims and ephemeral storage requests.
The total number of pods, ConfigMaps, ReplicationControllers, ResourceQuota objects, load balancers, secrets, deployments and cron jobs that can exist in the namespace.
As you can see, LimitRange and ResourceQuota policies help keep a large number of resources under control. That said, it’s wise to explore the limitations of using such resource usage policies.
LimitRange and ResourceQuota: Pros and Cons
As powerful and flexible as LimitRange and ResourceQuota policies are, they are not without certain limitations. The following is a summary of the pros and cons of these objects from the perspective of cost optimization:
Pros
You do not have to install third-party solutions to enforce reasonable resource usage.
If you define your policies wisely, you can minimize the incidence of issues like CPU starvation, pod eviction or running out of memory or storage.
Enforcing resource limits helps lower cluster operating costs.
Cons
Kubernetes lacks built-in mechanisms to monitor resource usage. So whether you like it or not, you will have to use third-party solutions at some point to help your team understand workload behavior and plan accordingly.
Policies implemented using LimitRange and ResourceQuota are static. That is, you may have to fine-tune them from time to time.
LimitRange and ResourceQuota cannot help you avoid resource waste in every situation. They won’t help with services and applications that comply with the policies at the time of their creation but become inactive after a while.
Identifying inactive namespaces is a manual and time-consuming process.
In light of these limitations, it’s worth considering options that address these limitations by adding new functionality to Kubernetes to optimize resource usage.
Cost Optimization Using Loft
Loft is a state-of-the-art managed self-service platform that offers solutions for Kubernetes in areas such as access control, multitenancy and cluster management. Additionally, Loft provides advanced cost optimization features such as sleep mode and auto-delete:
Sleep mode: This powerful feature monitors the activity of workloads within a namespace and automatically puts them to sleep after a certain period of inactivity. In other words, only the namespaces that are in use remain active, and the rest are put to sleep.
Auto-delete: While sleep mode consists of scaling down to zero pods while the namespace is inactive, auto-delete goes a step further by permanently deleting namespaces that have not been active for a certain period of time. Auto-delete is especially useful for minimizing the waste of resources caused by demo environments and projects that have been sitting idle for too long.
Both sleep mode and auto-delete are completely configurable, giving DevOps teams full control over when a namespace is put to sleep or deleted.
Conclusion
Kubernetes allows you to use LimitRange and ResourceQuota policies to promote efficient use of resources in namespaces and thus save costs. That said, estimating resource requirements in a production environment is challenging, which is why it’s a good idea to combine the flexibility provided by namespaces and resource usage policies with state-of-the-art cost optimization solutions like Loft.
Features like sleep mode and auto-delete help keep your clusters clean, which can save your organization up to 70% on costs.
Commenti