Kubernetes is a proven technology for container orchestration and is a great tool for running resilient applications, but the true cost is often underestimated. Our comparison explored the opportunity costs related to developer experience and managed responsibilities, this post investigates the actual financial costs of using Kubernetes.
When building a production platform, you can select different layers of abstraction. We distinguish four major abstraction layers you can pick from. If you’re not an infrastructure company, we advocate building on top of cloud service provider primitives. Spoiler alert: We estimate that operating Kubernetes clusters costs at least $100k and can easily cost more than $500k annually for a ridiculously low amount of resources.
The cost of Kubernetes on Azure, GCP, and AWS
Kubernetes is an open-source technology, but open-source doesn’t mean it’s free to operate. The operational costs depend on how you choose to use Kubernetes. When opting to use Kubernetes, you have two main options:
- Self-hosted Kubernetes
- Managed Kubernetes
Self-hosted Kubernetes
If you go with a self-hosted Kubernetes solution, you will need to budget for the nodes needed to run your workloads, but you will also need to add the costs of the nodes to run your control plane and of the team to manage the operation.
For a reliable Kubernetes production environment, you will need to create and manage nodes that host your control plane, in addition to separate worker nodes to host your application workloads. The control plane is where the main Kubernetes components
needed to configure and orchestrate your workloads live. The control plane runs key processes including:
- kube-api-server to expose the Kubernetes API,
- etcd, a highly-available key value store, to store all cluster data,
- kube-schedule, which schedules newly created Pods,
- and more processes like the kube-controller-manager and cloud-controller-manager.
Technically, you could use the same nodes to host the control plane and worker nodes, but it is not recommended for production environments.
Self-hosted Kubernetes total cost
For this calculation, we decided to do the cost simulation using nodes running on Azure. We did the estimation using D2s v3 Azure VMs machines, which have 2 vCPUs and 8 GB RAM and used 3 nodes which is the minimum for redundancy purposes. With that, you should have 5.4GB of RAM per node
, so ~16GB of usable RAM and 6vCPU.
Secondly, we took the average DevOps compensation in the US, $141,000 (cited by Builtin
) and multiplied it by 4 to represent a team of four engineers, which is needed to properly cover 24/7 operations.
As a result, here is what the total cost of ownership (TCO) for self-hosted Kubernetes looks like:
| Expense | Annual Cost |
|---|---|
| Nodes to host the control plane of your clusters | $0.096 x 24 x 365 x 3 = $2523 |
| Nodes to host application workload | $0.096 x 24 x 365 x 3 = $2523 |
| Compensation for members on your DevOps/SRE team | $141,000 x 4 = $564,000 |
Annual total base cost: $2523 + $2523 + $564,000 = $569,046
This cost estimation doesn’t include the sanity of your DevOps team when issues arise, which is, we believe, priceless.
Managed Kubernetes on Azure, GCP, or AWS
Then there is the managed Kubernetes route. With a managed Kubernetes solution, you do not need to handle creating or managing your Kubernetes clusters, nor do you need to worry about creating the control plane or installing its components like etcd, kube-apiserver, or kube-scheduler.
GKE, EKS, and AKS: The cost of the managed control plane
The 3 biggest managed Kubernetes providers are GCP, AWS, and Azure with Google Kubernetes Engine (GKE), Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). From what we found, the pricing is similar for the managed control plane of all these services at $0.10/hour.
| Managed Kubernetes Provider | Cost to run one cluster |
|---|---|
| GCP GKE | $0.10 x 24 x 365 = $876 |
| AWS EKS | $0.10 x 24 x 365 = $876 |
| Azure AKS | $0.10 x 24 x 365 = $876 |
For all these offerings, there are no automatic version updates or auto-recovery and you still need to pay for the computing resources like CPU, memory, and ephemeral storage that your worker pods consume.
Cost of the worker nodes
The cost of worker nodes also varies across cloud service providers and depends on the amount of computing resources you need as well as in which region of the world your servers are running. We multiply by 3 for redundancy purposes.
| Provider | Resources | Annual Cost |
|---|---|---|
| AWS m6gd.large EC2 instances | 2 vCPUs and 8 GB RAM | $0.0904 x 24 x 365 x 3 = $2376 |
| Azure D2s v3 VM machines | 2 vCPUs and 8 GB RAM | $0.096 x 24 x 365 x 3 = $2523 |
| GCP e2-standard-2 VMs | 2 vCPUs and 8 GB RAM | $0.067 x 24 x 365 x 3 = $1761 |
With these instances, you should have 5.4GB of RAM per node
, so ~16GB of usable RAM and 6vCPU.
Managed Kubernetes total cost
You also need to account for the maintenance responsibilities not covered. You’ll need at least one dedicated DevOps person to monitor this managed solution and handle the responsibilities not covered by these offerings.
| Managed Kubernetes Provider | Cost to run one cluster | Cost of worker nodes | Salary for dedicated DevOps | TCO |
|---|---|---|---|---|
| AWS EKS | $876 | $2376 | $141,000 | $144,252 |
| Azure AKS | $876 | $2523 | $141,000 | $144,399 |
| GCP GKE | $876 | $1761 | $141,000 | $143,637 |
This option buys you a lot more time and energy to focus on developing and improving your applications, but you will still be stuck with a lot of the Kubernetes responsibilities and complexity depending on the cloud service provider you use.