Securing the AI Compute Stack

AI has quietly moved an organization’s most sensitive assets, regulated data, proprietary models, and the pipelines that connect them, onto shared, GPU-accelerated infrastructure that most security programs were never designed to cover. The controls that protect a typical application stack do not map cleanly onto a GPU cluster. Data is decrypted to be computed on, tenants often share the same physical accelerator, and the model itself becomes a theft target worth more than the data it was trained on.

This page frames GPU cluster architecture the way a security leader should: not as a hardware spec sheet, but as a set of threats to understand and decisions to own.

Data is exposed in use Encryption at rest and in transit leaves a gap: data is decrypted in GPU memory to be processed. That is the gap confidential computing is built to close.

Shared silicon, shared risk Multi-tenant GPUs can leak one workload's data to another. LeftoverLocals (CVE-2023-4969) showed attackers reconstructing other users' LLM responses from residual GPU memory.

The model is the asset Trained weights represent enormous investment and are directly exfiltratable. Model theft and extraction are first-class techniques in MITRE ATLAS.

Attestation, not assumption A confidential GPU is only trustworthy if you verify its hardware attestation before releasing data or keys to it. Trust must be proven on every run.

The threat model that makes GPU clusters different #

A GPU cluster is not just servers with faster cards. Four properties change the risk picture.

Data in use is exposed. Traditional defense in depth encrypts data at rest and in transit, but computation requires plaintext in memory. On a GPU, sensitive inputs, model weights, and intermediate state all sit decrypted in high-bandwidth memory while a job runs. Anyone with sufficient access to the host, the hypervisor, or the accelerator can potentially read it.

Multi-tenancy can break isolation at the silicon level. To control cost, GPUs are frequently shared across jobs, teams, or customers. When isolation is incomplete, one workload can recover another’s data. The LeftoverLocals research (CVE-2023-4969, Trail of Bits) demonstrated recovering several megabytes per GPU invocation from residual local memory across AMD, Apple, and Qualcomm GPUs, enough to reconstruct another tenant’s LLM conversation. The lesson is not about one bug; it is that GPU memory is not automatically cleared or partitioned the way CISOs assume CPU memory is.

The model and pipeline are high-value targets. A frontier model’s weights can represent tens of millions of dollars of compute and proprietary data. They are also a single file that can be copied. MITRE ATLAS, the AI-specific companion to ATT&CK, catalogs model theft, model extraction through API queries, and training-data poisoning as distinct adversary techniques. The OWASP Top 10 for LLM Applications adds runtime risks such as prompt injection and model extraction that live above the infrastructure layer.

The accelerator is its own supply chain. GPUs run signed firmware, vendor drivers, and management controllers that sit below the operating system. A compromise there is invisible to most host-based tooling, and firmware and driver patch cadence is rarely tracked with the same rigor as the OS.

Confidential computing: closing the data-in-use gap #

Confidential computing uses a hardware-based Trusted Execution Environment (TEE) to keep data and code encrypted and isolated even while in use, including from the cloud operator and the hypervisor. On the CPU this is delivered by AMD SEV-SNP and Intel TDX. The newer and more relevant development for AI is that the GPU now participates in the TEE.

NVIDIA’s Hopper architecture made the H100 the first GPU with built-in confidential computing, extending the encrypted boundary from the confidential VM across the PCIe bus to the GPU itself, with all CPU-to-GPU transfers encrypted and the GPU’s memory isolated from the host. NVIDIA’s “Secure AI” capability, including Protected PCIe for multi-GPU configurations, reached general availability in 2025 for Hopper H100 and H200 systems, and the Blackwell generation extends the TEE to secure larger workloads. This is also available in the public cloud: Azure’s confidential VMs with NVIDIA H100 GPUs are generally available, pairing AMD SEV-SNP confidential VMs with the H100 GPU TEE so data, models, and computation stay protected end to end.

What confidential computing does and does not do matters for a security leader:

It protects data and model weights while they are being processed, gives you a hardware attestation you can verify before trusting the environment, and isolates the workload from the operator and other tenants. It does not stop attacks that operate at the application layer, prompt injection, model extraction through legitimate API queries, or poisoned training data, and it carries a performance cost and operational complexity that must be planned for. It is a foundational control, not a complete program.

A decision framework for GPU cluster architecture #

The questions below are the ones worth driving with your architecture and platform teams. Each maps to a concrete control you can require.

Isolation. Decide which workloads may share a physical GPU and which may not. For sensitive or multi-customer workloads, require hardware-partitioned isolation (NVIDIA Multi-Instance GPU) rather than software time-slicing, and require that GPU memory is cleared between jobs. Treat “shared GPU” as a data-classification decision, not a scheduling convenience.

Protect data in all three states. Encrypt at rest (model weights, checkpoints, and datasets, with keys in an HSM or managed KMS), in transit (mutual TLS across the compute and storage fabric, not just the north-south edge), and in use (confidential computing for the workloads that justify it).

Verify attestation before trust. A confidential GPU is only as good as the attestation you check. Require that workloads validate both the CPU TEE and the GPU hardware attestation before data or decryption keys are released, and that attestation is re-verified continuously, not just at boot.

Segment and monitor the fabric. The high-bandwidth east-west traffic between GPUs and nodes is often unencrypted and unmonitored. Separate management, compute, storage, and attestation traffic, treat the internal fabric as untrusted, and collect GPU telemetry into your detection pipeline.

Govern firmware and the accelerator supply chain. Track GPU firmware, GPU management controller, and driver versions; require signed firmware and drivers; and fold vendor security advisories into your patch cadence with the same rigor as operating systems.

Constrain identity and orchestration. Most GPU clusters run on Kubernetes. Apply certificate-based node identity, least-privilege RBAC for the scheduler and operators, and strict controls on who can submit jobs to confidential nodes.

Map detections and governance to AI-specific frameworks. Use MITRE ATLAS to structure detections around real AI adversary techniques, the OWASP LLM Top 10 for application-layer risk, and the NIST AI Risk Management Framework to organize governance, data lineage, and residency obligations.

A pragmatic maturity path #

You do not need a confidential supercomputer on day one. A workable sequence:

First, establish visibility and baseline hygiene, inventory GPUs and the workloads on them, classify the data they touch, lock down identity and network segmentation, and get firmware and drivers under patch management. Second, close the highest-value gaps, move sensitive and multi-tenant workloads to hardware-isolated or confidential-computing GPUs, encrypt the fabric, and stand up attestation verification. Third, operationalize, wire GPU telemetry into detection mapped to MITRE ATLAS, automate continuous attestation, and align the program to NIST AI RMF and your compliance obligations.

Questions to put to your team

Which of our AI workloads run on a shared GPU, and what data do they touch?
Where is sensitive data decrypted, and what could read it while a job runs?
If we use confidential GPUs, are we actually verifying attestation before releasing data or keys?
How would we detect another tenant, or our own operator, reading a workload's memory?
Who tracks GPU firmware and driver advisories, and on what cadence do we patch them?
Are our AI detections and governance mapped to MITRE ATLAS, OWASP LLM, and NIST AI RMF?

The bottom line #

GPU clusters concentrate an organization’s most valuable data and intellectual property onto infrastructure with a fundamentally different trust model than the rest of the estate. Confidential computing now makes it practical to protect data and models even while in use, and it is generally available on current NVIDIA hardware and in the major clouds. The security leader’s job is to decide which workloads warrant that protection, to insist that trust is proven through attestation rather than assumed, and to extend identity, segmentation, supply-chain, and detection disciplines onto the accelerator layer where most programs still have a blind spot.