The most likely cause is thatthe data being processed includes large datasets that are stored in GPU

memory but not efficiently utilized by the compute cores(D). This scenario occurs when a workload

loads substantial data into GPU memory (e.g., large tensors or datasets) but the computation phase

doesn’t fully leverage the GPU’s parallel processing capabilities, resulting in high memory usage and

low compute utilization. Here’s a detailed breakdown:

How it happens: In AI workloads, especially deep learning, data is often preloaded into GPU memory

(e.g., via CUDA allocations) to minimize transfer latency. If the model or algorithm doesn’t scale its

compute operations to match the data size—due to small batch sizes, inefficient kernel launches, or

suboptimal parallelization—the GPU cores remain underutilized while memory stays occupied. For

example, a small neural network processing a massive dataset might only use a fraction of the GPU’s

thousands of cores, leaving compute idle.

Evidence: High memory usage indicates data residency, while low compute usage (e.g., via nvidia-

smi) shows that the CUDA cores or Tensor Cores aren’t being fully engaged. This mismatch is

common in poorly optimized workloads.

Fix: Optimize the workload by increasing batch size, using mixed precision to engage Tensor Cores, or

redesigning the algorithm to parallelize compute tasks better, ensuring data in memory is actively

processed.

Why not the other options?

A (Insufficient power supply): This would cause system instability or shutdowns, not a specific

memory-compute imbalance. Power issues typically manifest as crashes, not low utilization.

B (Outdated drivers): Outdated drivers might cause compatibility or performance issues, but they

wouldn’t selectively increase memory usage while reducing compute—symptoms would be more

systemic (e.g., crashes or errors).

C (Models too small): Small models might underuse compute, but they typically require less

memory, not more, contradicting the high memory usage observed.

NVIDIA’s optimization guides highlight efficient data utilization as key to balancing memory and

compute (D).

Reference:NVIDIA GPU Optimization Guide; nvidia-smi documentation; CUDA Best Practices on

nvidia.com.