Reduce attack surface with private clusters on GKE

Do you sleep well at night? If so, you surely have already set up a private cluster!

Ok, but I have a public cluster and I think I sleep quite well!

We can't explain that, but what we can explain: why you should at least consider private clusters, what benefits they have, and last but not least, how to set one up on GKE. Let's jump in.

What is a private cluster?

Private clusters are clusters that do not expose their nodes' IPs on the public internet. As in, the nodes only have internal IPs. In contrast to public clusters. Which means they are isolated from the internet by default. A great thing!

Aha, but I want my app to be accessible from the internet!

You can still expose your services and apps to the internet. We'll talk about that in a moment.

Why should I care?

Think about it the other way around: if you were to have a public cluster instead, your nodes would always be exposed to the internet. And even if you did secure your cluster using best practices, what do you do about software vulnerabilities? Like regreSSHion the other day, or kernel vulnerabilities that lead to privilege escalation and what not. Happening all the time, unfortunately. And the worst thing is, they are completely out of your control, putting your clusters at risk. Even on Autopilot.

I'll be quick and patch it!

You know that's not true. And even if, what if you're not quick enough? Isn't there a better way? There is, and it's called a private cluster!

How does a private cluster work?

A private cluster works more or less the same as a public cluster. The main difference is increased network security by default: your nodes are not exposed publicly. We think this is a good thing, because it reduces the attack surface of your cluster.

Understood, what about the control plane?

Depending on your configuration, the control plane will be accessible either

• from the internet, or
• within a private network only, or
• from the internet, but limited to certain IP ranges

after your cluster has provisioned. Which one to choose depends on your requirements. We often like option three, because it offers a good balance between security and convenience.

If you opt in to make the control plane private that's great and increases security even more. Note though, that to manage your cluster in this case, you'd need to set up a VPN tunnel or bastion host.

How to set up a private cluster

So how do we set up a private cluster now? It's easy, look:

apiVersion: container.gcp.upbound.io/v1beta1
kind: Cluster
# ...
spec:
  forProvider:
    # ... reference networks, etc.
    enableAutopilot: true
    privateClusterConfig:
      - enablePrivateNodes: true
    # ... define how you want to access the control plane
    # ...
# ...

That's it. Commit your changes, git push and you're done. Wasn't this a breeze? You can now sleep well at night :) If you are following GitOps principles and manage everything as code (you should!), it can be as easy as this. Ok granted, some more lines of yaml will be required for networking, firewall rules, etc., but you get the idea.

And in case you are wondering: We advocate for Crossplane for managing (not only) our GKE clusters. You can use Crossplane to orchestrate your infrastructure, apps, Kubernetes objects, hell, you could even use it to order a pizza. Think a universal control plane. Thank you, Crossplane :)

How do I get out?

So we have our private cluster running now, nicely isolated. But how do we get out?

Not so fast, bro. What about getting in first?

Ok, let's talk about ingress first.

Ingress

Fortunately, there's not so much to say about it. Allowing traffic into your cluster is straightforward. The same goes for allowing a lot of traffic. You just expose your app to the internet as usual (Ingress, LoadBalancer, Global Load Balancer, ...). But fair enough, there's a catch: the moment you're installing a custom Ingress Controller in your private cluster, and you're pulling it from a public registry (e.g. nginx-ingress via Helm) you'll need to have egress allowed.

Egress

Egress is not hard, but it's not "set up" by default. Therefore you can't just access the internet from your pods yet. You have to actively opt-in where you want to allow internet access.

Hypothetical example: If your pods would just need to pull images from within GCP, you might cheap out on a public NAT gateway. That's because Google Artifacts are accessible from a private cluster without further config. So if you are putting you're stuff there, and you have no requirements to call endpoints outside of your network, that's it.

For anything else, this is where Cloud NAT comes into play. With Cloud NAT you allow sending outbound requests and receiving corresponding responses without your nodes needing a public IP. This is called source NAT, respectively destination NAT. Isn't that awesome?

The key ingredients for this to work are:

• a Router
• a NAT that's associated with the Router
• the subnetworks' IP ranges you'd like to NAT

It's just some more lines of yaml. If you set natIpAllocateOption to 'AUTO_ONLY' in your NAT's configuration, you won't even have to worry about provisioning an IP address for it.

After that: Your nodes' IPs drifting uncontrolled through the the internet? - these days are gone. You enjoy:

• increased networking security
• smaller attack surface
• and a more controlled environment

for your cluster. See for yourself, no more public IPs:

NAME                STATUS   ROLES    AGE    VERSION               INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gk3-cluster-1-nap   Ready    <none>   102m   v1.29.6-gke.1254000   192.168.0.3   <none>        Container-Optimized OS from Google   6.1.85+          containerd://1.7.15
gk3-cluster-1-pool  Ready    <none>   102m   v1.29.6-gke.1254000   192.168.0.4   <none>        Container-Optimized OS from Google   6.1.85+          containerd://1.7.15

Note the EXTERNAL-IP column. What more could you want?

Summary

To sum it up: Private clusters are a great way to increase the networking security of your cluster. Your cluster's nodes will be isolated from the internet by default. You are in control of injecting internet access, where internet access has to be available. Otherwise your shielded. Feels good, huh?

FAQ

If you like the idea of increased network security for your cluster and need help in setting one up, or have more questions, feel free to contact us. We're happy to help!