What would running a VM inside a container provide in terms of security and isolation that just running a VM would not?
This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states:
>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1]
Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around:
your container -> Compute Engine VM (GKE Node) -> container -> Borg
The container on top of Borg is used for scheduling and management. Joe's talk has a slide on this. As a GCP customer, you never have to worry about this or care about it, as it is an implementation detail.
>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."
As a GCP customer using GKE, your applications are separated from other GCP customer using VMs.
However, if you want to run your OWN untrusted workloads, then in the past you would have to spin up a separate VM for untrusted workload A and a one VM for untrusted workload B.
This sucks in terms of resource utilization. It would be better in many cases if you could run workload A and B on the same VM. That's where gVisor comes into play.
your untrusted container -> gVisor -> Compute Engine VM (GKE Node) -> container -> Borg
The fact that gVisor is being used in multiple services at Google is probably the confusing part. In case of GKE Sandbox, the users here are external and using Cloud (specifically GKE). The target use case is to add defense in depth to their pods running on potentially shared GKE Nodes (VMs) for Multi-Tenancy. Our talk at Next'19 [1] includes a story by one of our customers, which may help understanding the use cases.
Thanks for the link that does make the use case clear i.e multitenancy/SaaS. Am I correct in assuming though that when someone creates a K8S cluster via GKE that the containers that make up their cluster such as the kubelets and masters are all running in VM underneath?
This ACM article from a few years ago written by folks that worked on Borg/Omega/Kubernetes states:
>"The isolation is not perfect, though: containers cannot prevent interference in resources that the operating-system kernel doesn't manage, such as level 3 processor caches and memory bandwidth, and containers need to be supported by an additional security layer (such as virtual machines) to protect against the kinds of malicious actors found in the cloud."[1]
Also see slide 13 of Joe Beda's talk from a five years ago shows the container running in a VM not the other way around:
https://speakerdeck.com/jbeda/containers-at-scale?slide=13
[1] https://queue.acm.org/detail.cfm?id=2898444