Deploy TLS for PostgeSQL on K8s using Cert Manager

#devops #kuberentes #postgreSQL #SSL #security

When talking to security, the internet exposure is always the worst thing we have to deal with, it can leave you vulnerable to various security risks, including zero-day vulnerabilities or various password attacks.

While security is a neccesity, but it is not fun to spend time on. You should not have to trade off between strict security and ease-of-use.

One of the most important aspect of network security is encrypting traffic between 2 services or 2 endpoints. There is a common cryptographic protocol that allows data to be securely transfer from services to servivces, it is TLS.

In Kubernetes, there are lots of system communicating with each other, knowing who you are talking to and that your conversation is secure seams like an obvious choice.

By default, PostgreSQL supports natively TLS management with Postgres Operator. However, you could ant to set up your own kind of TLS for PostgreSQL cluster to enhance security level, at least.

After long time working on it, let's look at how we use cert-manager (popular opensource project for certificate management. We will go through how we can use it to deploy TLS on PostgreSQL cluster.

Enable cert-manager on existing Kuberntes cluster

We recommend you to use Helm chart to get started, you can also start with static installation.

Prerequisites:

• Helm version 3 or later.

• A Kubernetes or OpenShift cluster running supported version.

• cert-manager has not already installed on your cluster.

Add the Jetstack Helm repo:

$ helm repo add jetstack https://charts.jetstack.io

Update your local Helm chart repository cache on local

$ helm repo update

Install CustomResouceDefitions (CRD)

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml

Install cert-manager

$ helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.4.0 \
  # --set installCRDs=true

If you want to get full list of available Helm chart value if on cert-manager's ArtifactHub page.

Integrate cert-manager TLS Certificate to PostgreSQL

By default, PGO-managed PostgreSQL cluster will have at least two certificate: one for identifying the cluster and cluster communications, one for replication authentication.

As you may known, the cluster certificate should contain values thata can identiry the cluster, such as DNS name. You can connect between services within cluster by DNS name like this: <service name>.<namespace>.svc.

The replication certificate need to have a specific common name (CN) in it. PGO use common name (CN) as a part of its certificate-based authentication system.

You can store the TLS certificates in Kubernetes Secrets in the same namespace as your Postgres cluster. For more information on setting up custom TLS certificates with PGO, please refer to the documentation.

Cretate a certificate for PostgreSQL cluster:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: postgre-certmanager
spec:
  # Secret names are always required.
  secretName: postgre-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - airwallet365
  commonName: postgre-primary
  isCA: false
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - digital signature
    - key encipherment
  # Set DNS names to the names of the Postgres hosts
  dnsNames:
  - postgre-primary
  - postgre-primary.postgre-operator.svc
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io

Create certificate for replication authentication user:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: postgre-repl-certmanager
spec:
  # Secret names are always required.
  secretName: postgre-repl-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
    - airwallet365
  # This needs to be set to _crunchyrepl to work with Postgres
  # cert auth
  commonName: _postgrerepl
  isCA: false
  privateKey:
    algorithm: ECDSA
    size: 256
  usages:
    - digital signature
    - key encipherment
  # At least one of a DNS Name, URI, or IP address is required, so we'll
  # have it match the CN for the benefit of Postgres
  dnsNames:
  - _postgrerepl
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
    group: cert-manager.io

Now you have two certificate stored in Kubernetes secret. Now you can create PostgreSQL cluster of high availability capacity.

apiVersion: postgres-operator.airwallet365.com/v1beta1
kind: PostgresCluster
metadata:
  name: postgre
spec:
  image: cr.airwallet365.com/registry/postgres-ha:13
  postgresVersion: 13
  customReplicationTLSSecret:
    name: postgre-repl-tls
  customTLSSecret:
    name: postgre-tls
  instances:
    - replicas: 2
      dataVolumeClaimSpec:
        accessModes:
        - "ReadWriteOnce"
        resources:
          requests:
            storage: 1Gi
  backups:
    pgbackrest:
      image: cr.airwallet365.com/registry/postgre-pgbackrest:centos8-2.33-0
      repoHost:
        dedicated: {}
      repos:
      - name: common
        volume:
          volumeClaimSpec:
            accessModes:
            - "ReadWriteOnce"
            resources:
              requests:
                storage: 1Gi

You now have your own PostgreSQL cluster with generated TLS certificates and managed by cert-manager.