Building a highly available, self-healing Postgres cluster is hard. You have to think about things like backups, load balancing across databases, metrics, database hosts changing, storage and correctly sizing all of these services.
The idea of a Postgres operator in Kubernetes was born out of a conversation with members of the community during 2016. This was in the very early days of Kubernetes, and the idea of running databases in containers was controversial.
Today there is a constellation of tools and techniques one can use to run stable, highly available and performant Postgres workloads in your Kubernetes clusters including operators from companies like Crunchy Data.
Across the industry, we are seeing a growing number of folks who are leveraging the operator architectural pattern successfully for a disparate set of workloads. My customer experience, and that with the larger community, has given me a nuanced view into the world of Postgres in Kubernetes. Today, I’d like to more broadly share these lessons and give insights into lessons learned after five years of Postgres on Kubernetes.
The Infrastructure Is the Code
From the beginning, the concept behind operators was that they would functionally automate your Kubernetes applications. This means that Infrastructure-as-Code was realized in the Kubernetes space. This has been the DevOps dream for a long time — a platform that is self-healing, version controlled, self-arranging and defined declaratively.
As Kubernetes evolved and the APIs matured, delivering a fully declarative Postgres operator in Kubernetes became an increasingly attainable objective. We also began hearing from our users and customers that they wanted to see the project adopt a more declarative approach.
Fortunately, this is something we had been thinking about as well. This became the key driver behind the fifth major version of Crunchy Postgres for Kubernetes. While we had several ideals in mind while designing this version of the Postgres Operator, our primary objective was to make Crunchy Postgres for Kubernetes fully declarative. This was no small task; essentially, it was a ground-up rebuild. Starting with the latest modern Kubernetes functionality, we were able to develop a new version of Crunchy Postgres for Kubernetes that enabled users to simply describe the end result they were after and Crunchy Postgres for Kubernetes would do the rest.
Start with Storage
Databases are different from stateless apps in that they need persistent storage. Kubernetes provides powerful automation and many interesting capabilities, but we knew we needed a way to provide consistent and stateful storage, since storage is one of the cornerstones of database functionality.
Storage needs are what ultimately led us to coupling Kubernetes technology with an operator.
Today, databases running on Kubernetes can make use of a range of storage types like HostPath, Network File System (NFS) and Dynamic storage. Crunchy Postgres for Kubernetes is storage agnostic: It works with any supported Kubernetes storage system. You must provision persistent volumes for both Hostpath and NFS, though automated storage provisioners do exist. Dynamic storage classes allow users to request persistent volume claims and have the persistent volume created for you. There are many providers of dynamic storage classes to choose from. You will need to configure what works for your environment, and size the Physical Volumes, Persistent Volumes (PVs).
Postgres Operator, the Easy Button for Postgres in Kubernetes
Early on, the operator pattern for Kubernetes was unknown territory. Running stateful processes such as databases in containers was a very new concept. The operator pattern demonstrated the potential to simplify the operation and administration of databases on Kubernetes. From the moment our team heard the pitch from CoreOS, we saw the promise.
There was quite a bit of debate in the early days as to the appropriate use of the operator pattern and whether the operator pattern had staying power. A number of folks in the early days asked questions about the trade-offs between Helm charts and operators. Spoiler alert: They are different and you may want both!
Building a highly available, self-healing Postgres cluster is hard. You have to think about things like backups, load balancing across databases, metrics, database hosts changing, storage and correctly sizing all of these services. You also have to ensure that these services are running all the time. You need to consider how to handle version upgrades, security patches and much more.
Crunchy Postgres for Kubernetes was one of the very first operators for Postgres on Kubernetes, and the depth of experience shows in the easy setup and highly resilient architecture. Giving folks the ability to create Postgres clusters with high availability and self-healing built in is the primary goal of the Postgres for Kubernetes team at Crunchy Data; however, we would not be able to do it alone.
Leaning into APIs
Kubernetes has rapidly evolved since the early days, and so has its APIs. The underlying APIs routinely change between versions to enhance new capabilities or to consolidate capabilities. Crunchy Postgres for Kubernetes has evolved alongside the Kubernetes APIs, and you see the culmination of those years of work in the fifth major version of the Postgres Operator.
By using an operator’s API capabilities, teams feel the real power of Kubernetes. These APIs are capable of many tasks like deploying, adding users, recovering from failed health checks, running backups, recovering from backups, etc. When using these APIs, teams start leveling up. CI/CD functions become far more repeatable across teams, deploying staging is an automated instead of a manual process, and as a result every database runs uniformly instead of as an edge case.
To empower this functionality, use of the operator must be integrated into many other systems. For example, we have customers with API-level integrations with their tickets systems, deployment systems, analytics teams, marketing teams, operations teams and code repositories. Thus it is essential that Crunchy Postgres for Kubernetes has stable APIs and is supported by a team with compatibility in mind. Deploying an operator with fluid APIs can quickly become a liability at all points of integration.
Looking to the Future
I believe that our initial support of the operator pattern for managing stateful services like Postgres in Kubernetes was a good choice. As Kubernetes has matured, the operator pattern and APIs have evolved to strengthen this method, and our Postgres Operator has matured alongside the community. Early on, we felt a groundswell of support and appreciate the collaboration and community that formed around this idea that we had so many years ago.
We are excited for what we see in the future and for the ideas we have for further maturation of the Operator pattern and implementation of that pattern.
Comments