For AI Clouds, Hard Tenant Isolation Is Not Optional. SemiAnalysis Agrees.

On August 30, SemiAnalysis published the article “Most Neoclouds Suck At Security.” It is worth reading in full. One of the findings mentions a neocloud misconfiguring vCluster against our docs and explicit recommendations, so let’s review what went wrong here.

SemiAnalysis tested a provider running the open source flavor of vCluster that uses a worker node model we call Shared Nodes. As the name suggests, the nodes are all owned by a single Kubernetes cluster and vCluster then adds a control plane for each tenant, BUT the nodes are still shared among tenants. This is acceptable for internal use cases where soft isolation is enough, and absolutely problematic when you’re building an EKS-style managed Kubernetes offering, as many neoclouds do. This is why our documentation explicitly states to use Private Nodes, not Shared Nodes for use cases that require hard tenant isolation.

What SemiAnalysis found in their analysis was that inside one tenant, they could see metadata belonging to other tenants: namespace names, node labels and taints, physical host resources. Machines were being run in a shared cluster and only separated via policies such as node selectors, so nodes could move between tenants without fresh provisioning. Digging further, they found every piece of software on the cluster more than two years out of date, including the cluster-wide GPU operator, default-deny NetworkPolicy not correctly enforced, and the kubelet on every node reachable on a publicly routable IP.

SemiAnalysis is direct about the cause. The provider misconfigured vCluster, and the article notes that this followed neither our guidance nor what SemiAnalysis themselves recommend in their ClusterMAX security criteria:

Strong isolation between tenants. Not namespaces, or container-only isolation, i.e. use vCluster Private Nodes instead of vCluster Shared Nodes.

But “read the docs” is not a useful article. The useful question is why a provider was on shared nodes in the first place, and why that decision is so easy to get wrong when you are standing up GPU capacity faster than anyone has ever stood up infrastructure.

SemiAnalysis has raised this before

This is the second time SemiAnalysis found a cloud provider using vCluster Shared Nodes. In March 2025, the first ClusterMAX rating system article made the same argument about CoreWeave Classic, which gave each tenant a namespace on a shared Kubernetes cluster. Their reasoning was correct: container escape vulnerabilities, mostly in the GPU driver or the container toolkit, let an attacker break out and move laterally to another tenant sharing the same host.

One thing is worth separating, because almost every conversation about this collapses it. If you are using vCluster open source with Shared Nodes enabled, you get control plane isolation, not hard workload or network isolation. Hard isolation is a different layer entirely. Giving each tenant its own control plane does not, by itself, stop a container escape on a node that two tenants share. Those are two separate issues. You only get hard isolation once you add Private Nodes to the tenant cluster. And even then, you need to make sure your network environment is configured appropriately, with tenant nodes running in separate tenant-specific VLANs or VXLANs.

Most providers understand that each tenant needs their own control plane, but they often think workloads can be separated by policy rather than by hard network-level isolation at the worker nodes. It is good to see SemiAnalysis keep educating neocloud operators on why these use cases demand hard tenant isolation.

What CoreWeave figured out years ago

CoreWeave started talking to us before ChatGPT was released and definitely before the terms “neocloud” or “AI Infra” even existed. They built CoreWeave Classic with namespace-based isolation, and they leveraged open source vCluster with Shared Nodes to give tenants separate control planes inside their namespaces. vCluster solved that well for them:

• Tenants overcame the limitations of namespaces by getting their own control plane, so they could easily work with their own CRDs, have multiple namespaces, and control cluster-scoped resources.
• Each tenant got an independent control plane, which meant independent Kubernetes versions, independent upgrade cycles, and no lockstep upgrades across the fleet.
• The underlying control plane stopped being a bottleneck and could scale further, in a more resilient way.

What it did not solve and never claimed to solve was network-level or hardware-level isolation for the workloads. Tenants still shared the underlying host cluster with its worker nodes, with network policies and admission control the only defense on top of container-level isolation. And that is not enough to build an EKS-style offering. CoreWeave understood that years ago before most of the market was asking the question.

Here is the part worth saying plainly. Back in 2021, vCluster was designed for internal use cases at large Fortune 500 companies, to resolve the pain points described above. We were not focused on becoming the next EKS-style platform for neoclouds. What CoreWeave needed back then was not part of our focus. As a result, they ultimately had to build their own Kubernetes service from scratch, called CKS. That took years and a big team.

We’re opinionated: Neoclouds need Private Nodes

Due to our past experiences with CoreWeave, we are very opinionated on the topic of picking the right isolation model for the right use case. If you are an AI Cloud provider running production workloads for your customers, we urge you to use Private Nodes. Of course we cannot control what operators do with our open source software, or how they configure our commercial software after they purchase it, but we are clear in our documentation and guidance about what we believe they should do.

The learning CoreWeave (and we) made years ago is the same learning other neoclouds are making today, as this recent SemiAnalysis article makes clear. The difference is that today, anyone can use vCluster Private Nodes, which has been GA since August 2025. With Private Nodes, operators can build an EKS-like offering in a matter of days and achieve hard tenant isolation using either bare metal workers or virtual machines. To automate provisioning those worker nodes further, we offer another product called vMetal. It deploys and orchestrates provisioning systems such as Metal3, NICo, and KubeVirt, and it uses network automation tools such as Netris to provision machines and add them to the appropriate tenant network for hard tenant isolation.

For production customer workloads on GPU infrastructure, Private Nodes is the answer. Everything else is a reason you will have to explain later for why two tenants were on the same host. If you are weighing the two models, our production guide walks through the three questions that settle it.

No boundary survives misconfiguration

It would be convenient to claim that one isolation technology is secure and the others are not. Unfortunately, the reality is more complex. VMs fail. Containers fail. Network configurations fail. Unless you operate in entirely single-tenant air-gapped data centers, you’ll likely have some level of risk that needs to be managed. And even if you do all that, make sure nobody plugs in a USB stick. Stuxnet famously showed that as another path to compromising even air-gapped systems.

Every isolation layer in the stack has a CVE history, and any software can be configured to be insecure. The issues SemiAnalysis found with this particular neocloud weren’t based on some exotic vulnerability or mix of complex chain of attack vectors. They were as ordinary as it gets: configuring software against what the docs say, running versions two years stale, misconfiguring network policies so they never get enforced, and exposing the kubelet in a Shared Nodes environment.

So the question to ask about your own fleet is whether you have multiple eyes on your configuration, and how you are pen testing internally and externally to uncover weak links before an attacker does. Or in this case, before SemiAnalysis does.

What to check this week

If you operate GPU capacity for more than one tenant, this list is worth an hour of your time:

• Nodes. Do two tenants ever land on the same underlying hardware? If so, what is the boundary between them, and what if that boundary is compromised?
• GPUs. Do two tenants ever process data on the same physical GPU, for example via time-slicing? If so, what ensures there is never any cross-tenant leakage of data?
• Control plane components. Are the API server, scheduler, etcd, and cluster-wide operators like the GPU operator shared across tenants?
• Network isolation. Even if you use completely separate clusters in separate networks, have you ever tested pinging machines outside of the particular tenant network?
• Patching. Do you have a system that informs you about new security patches for the critical software components in your provisioning, networking, storage, and control plane stack?
• Monitoring. Is tenancy enforced at the data layer, or only on the dashboard? A single over-privileged API token behind a per-tenant view is not isolation.
• Reprovisioning. When a node moves between tenants, is it provisioned from scratch? Is there any chance data can be intentionally or unintentionally left behind and spill over to the new tenant?

None of this is exotic. All of it shows up in the findings.

Keeping up with the pace

The AI buildout is moving faster than any infrastructure cycle before it, and most operators are designing tenant isolation for the first time while trying to get capacity to earn revenue fast. Shortcuts are seductive but not worth the bad press and hassle.

The difference between providers who get a clean ClusterMAX security result and providers who get a negative writeup usually comes down to one decision made before the first customer: which isolation model they built on. That decision is cheap to make early and expensive to reverse later.

Thanks to SemiAnalysis for informing us ahead of publishing this article and for getting us involved with the particular provider to help them assess their architecture decisions and to move to a more resilient design before the article was published.

If you want a second set of eyes on your own setup, we are happy to do the same. We have worked with many well-known neoclouds and seen a lot of configurations, and we will walk your tenant model with you to find where a single failure becomes a cross-tenant one. That is an architecture review, not a security audit, and it is what we know how to do.