Inference Provider: Managed Model Serving

Build managed inference endpoints on your GPU infrastructure. Customers interact with your product API and model endpoints. Platform and vCluster form the operations layer your team runs behind it.

Minimum stack:

• vCluster Platform as the management plane.
• If you already operate Kubernetes, each tenant cluster's control plane runs as pods on your existing cluster. If you don't have Kubernetes to host it on, use vCluster Standalone to bootstrap the tenant cluster control plane directly on bare metal or VMs. Move to Standalone HA before production traffic.
• Private GPU nodes for dedicated inference customers.
• Argo CD or Akuity to deliver the GPU stack and runtime declaratively into each tenant cluster.

Optional add-ons:

• vMetal when you own bare metal GPU lifecycle. Skip it if your GPU capacity comes from cloud GPU instances, manually joined nodes, or Auto Nodes instead.
• vNode, a separate vCluster Labs product, for runtime isolation of custom containers, adapters, plugins, or other untrusted code.

Product API and traffic layer boundary backed by dedicated, isolated tenant clusters on private GPU nodes

What makes this path different: Customers don't use Platform directly. They call your product API or inference endpoint. Templates are the core payoff for your product. Applying a template creates a tenant cluster, attaches GPU capacity, enforces quotas, publishes routes, and reclaims capacity when endpoints are deleted.

Confirm licensing before you invest further. Several pieces of this stack require Platform activation or a paid tier. See Open Source vs Free tier for the tier check.

Not building a multi-customer provider?

If your users are internal teams that should provision and manage their own environments in Platform, start with Enterprise AI Factory instead. If you only need to deploy one model inside a tenant cluster you already have, skip ahead to Model serving runtimes. Use this inference provider path when your product hides Platform and exposes an inference endpoint lifecycle to multiple customers.

Before standing up infrastructure, review Building an inference platform for the tenancy models, GPU capacity classes, endpoint readiness, and product control plane mapping behind the decisions below.

Day 0: Design decisions

Decision	Read next	Outcome
Choose the inference tenancy model	Choose an inference tenancy model, Architecture	Decide whether each customer, model family, or endpoint tier gets a tenant cluster, private GPU nodes, vNode runtime isolation, or a shared serving pool.
Define the default network policy posture	Network policy	Set a default-deny east-west posture per endpoint template and confirm the control plane cluster's CNI enforces it. Some CNIs accept NetworkPolicy objects without enforcing them.
Plan GPU capacity classes	Size GPU capacity classes, Node providers, GPU and accelerators	Define node types by GPU model, count, region, and reservation model, and choose an isolation strategy per tenant tier.
Choose the GPU scheduling model	GPU and accelerators, DRA DeviceClasses, DRA ResourceClaims	Use extended resources from a vendor device plugin for simple whole-GPU allocation. Use a vendor DRA driver with DeviceClass and ResourceClaim for fine-grained or shareable allocation. Private nodes can run the DRA stack directly. DRA sync (Pro) makes control plane cluster devices available to tenants.
Define serving runtime templates	Model serving runtimes, Deploy applications, Certified Stacks	Standardize the components each endpoint receives as an Argo CD ArgoCDApplicationTemplate, such as GPU Operator, scheduler, model server, ingress or Gateway API routes, metrics, and secrets.
Plan Application boundaries and sequencing	Sequence the GPU stack and the runtime, Deploy applications	Decide which components share one Argo CD Application and which get their own. Argo CD doesn't sequence separate Applications, so bundle hard dependencies together and let Kubernetes absorb soft ones.
Plan model storage and warmup	Model storage architecture, Model serving runtimes	Decide where model weights live, how they reach each endpoint, how caches are warmed, and how readiness is reported while large models load.
Plan endpoint routing	Model serving runtimes, Gateway API	Decide whether tenant clusters attach routes to platform-owned shared Gateways, tenant-owned Gateways, or another ingress layer controlled by your product.
Plan autoscaling signals	GPU and inference autoscaling, Monitoring overview	Combine hardware metrics, such as GPU utilization and memory, with serving metrics, such as request concurrency, queue depth, latency, tokens per second, and time to first token.
Plan endpoint readiness	Endpoint readiness, Model serving runtimes	Account for Argo CD sync time, node provisioning, image pulls, model downloads, and model load time before customers expect an endpoint to be ready.
Plan durability and operations	Backing store, Platform HA, Platform backup	Choose data stores, backup policy, control plane availability, and recovery procedures before customers depend on endpoints.
Plan security hardening and audit logging	Security baseline, CIS hardening guide, Platform audit logging	Apply a CIS benchmark baseline to the control plane cluster and decide audit log retention and SIEM integration before onboarding tenants.
Integrate with your product control plane	Integrate with your product control plane, Product control plane on the Platform API	Decide the customer-facing product boundary, the mapping from product concepts to Platform resources, the metering approach, and the license tier you need before building automation.

Day 1: Stand up the first production inference endpoint

note

Steps 3, 4, 5, and 6 configure Platform for your platform engineering team and automation. Customers should provision endpoints through your product.

Set up the platform

1. Install vCluster Platform. If building from bare metal, deploy vCluster Standalone first, then move to Standalone HA before production traffic.
2. Configure backing store, Platform HA, and Platform backup.
3. Configure SSO, permissions, and access keys for the internal automation that provisions endpoints.
4. Create an Argo CD connector to your Argo CD or Akuity instance. Every tenant cluster template references it by name.
5. Create projects, templates, quotas, and allowed node types for the endpoint tier you're standing up.
6. Apply the CIS hardening baseline to the control plane cluster and enable Platform audit logging.

Attach GPU capacity

7. Configure GPU capacity for the inference tiers you sell. If you own bare metal, set up vMetal and the Metal3 node provider. If you use cloud GPU instances, create node types and automation for those instances through node providers and Auto Nodes, or your existing provisioning system and join them as Private Nodes.
8. Configure GPU scheduling. For device-plugin allocation, install the vendor plugin and have workloads request its extended resource, such as nvidia.com/gpu. For DRA allocation on private nodes, run the vendor DRA driver and controllers inside the tenant cluster. If devices reside on shared control plane cluster nodes instead, sync the permitted DeviceClasses and enable ResourceClaim sync.

Build the tenant cluster and runtime templates

9. Create a tenant cluster template for the endpoint tier. The template should enable Private Nodes, restrict GPU node selection, enable any required sync settings, and reference the Argo CD connector from step 4.

10. Apply a default-deny network policy in the template and confirm the control plane cluster's CNI enforces it.

11. Declare the GPU stack and the runtime as Argo CD Applications in the same template's deploy.argoCD.applications. Reference an ArgoCDApplicationTemplate for the GPU Operator or device plugin, scheduler, and metrics stack, and another for the model-serving Deployment, Service, and route. Each tenant cluster's scheduler makes decisions scoped to that cluster, not across your fleet.

    See Model serving runtimes for the runtime pattern. Keep hard dependencies in one Application, since Argo CD doesn't sequence separate ones.

Configure routing and autoscaling

12. Configure endpoint routing with Gateway API, ingress, or your product's traffic layer. Prefer Gateway API for new HTTP endpoint deployments, and decide whether the route attaches to a platform-owned imported Gateway or a tenant-owned Gateway. Bundle the route in the same Argo CD Application as the runtime Deployment and Service rather than a separate Application.
13. Configure autoscaling with hardware metrics, serving metrics, or both. Use GPU and inference autoscaling for the hardware-metric baseline and serving-metric patterns such as queue depth, request concurrency, latency, and tokens per second.

Wire up your product automation

14. Build the product automation path. For an endpoint create request, your product should choose the project and template version, then pass tier parameters. Provision the tenant cluster from a template that already has the Argo CD connector and Application references set. Wait for GPU capacity and for Argo CD to sync the runtime Application to Healthy, then return endpoint status and URL. Start with an internal operator workflow before exposing the API to customers. See Build a product control plane on the Platform API for the concept-to-resource mapping this automation implements.

Validate the first endpoint

15. Validate the first endpoint from outside the tenant cluster. Confirm the endpoint accepts authorized traffic, rejects unauthorized traffic, reports healthy model status, emits metrics, and can be traced back to the tenant cluster, node type, template version, and route.
16. Validate isolation and quota enforcement. Confirm the customer can't see Platform internals, other tenant clusters, disallowed templates, disallowed GPU node types, or control plane cluster resources.
17. Test the full endpoint lifecycle through your product API. Create, warm, scale, update the model or adapter, rotate credentials, drain, delete, and reclaim GPU capacity.

Endpoint provisioning flow

Use this flow as the acceptance test for the first provider-managed endpoint:

1. Your product automation receives an endpoint request with model, region, endpoint tier, and scaling parameters.
2. Product automation calls Platform with an internal access key.
3. Platform creates or selects the project, tenant cluster template, runtime template, quota, and allowed node types for the requested tier.
4. Private Nodes or Auto Nodes attach GPU capacity from the selected node type.
5. Argo CD syncs the GPU stack and runtime Applications declared in the tenant cluster template, installing the metrics stack, model-serving runtime, Service, and route.
6. Gateway API, ingress, or your traffic layer publishes the endpoint hostname.
7. Product automation returns endpoint status, URL, model state, and operational identifiers.
8. Day 2 automation watches rollout, route, GPU, and serving metrics and reconciles endpoint status.

Day 2: Operate

Operation	Read next
Manage GPU capacity and machine lifecycle	vMetal inference fleet capacity, Metal3 node provider, Manage private nodes
Monitor platform and tenant workloads	Monitoring overview, Fleet monitoring
Expose and troubleshoot endpoint routing	Model serving runtimes, Gateway API, Gateway API sync troubleshooting
Scale model-serving workloads	GPU and inference autoscaling, Fleet monitoring
Meter customer usage	vBilling, Fleet monitoring
Roll model runtimes and endpoint templates	Model serving runtimes, Deploy applications, GPU and inference autoscaling
Roll drivers, vCluster versions, and OS images	Deploy changes, vMetal GPU fleet operations, Upgrade vCluster
Enforce customer capacity boundaries	Quotas, Allowed node types, Projects
Back up and restore tenant clusters and Platform	Snapshots, Restore, Platform backup
Manage vNode compatibility during upgrades	vNode limitations, vNode configuration
Review audit logs and security posture	Platform audit logging, security baseline