GPU Economics: Building the Business Case for On-Premise vs. Cloud GPU Infrastructure

This is not a cloud strategy debate. It is a financial calculation with specific inputs — utilization rate, workload profile, commitment horizon, and hidden operational costs — and most enterprises are running that calculation with incomplete data. The 2025 H100 price cuts changed the numbers significantly. So did Blackwell’s performance profile. Here is the updated framework.

A significant change happened to GPU cloud economics in June 2025 that most enterprise financial models have not yet absorbed: AWS cut H100 pricing on P5 instances by 44 percent, dropping from approximately $7.57 per GPU-hour to $3.90. GCP and Azure followed with comparable reductions. The broader market — Lambda Labs, CoreWeave, RunPod — had already been undercutting hyperscaler pricing significantly, with H100 available for as low as $1.49 per GPU-hour on spot instances. The on-premise vs. cloud math that was true in 2024 is materially different in 2026, and enterprises that have not updated their TCO models are making infrastructure decisions on stale inputs.

This post walks through the actual cost structures on both sides of the decision, the utilization threshold that determines which wins, how Blackwell changes the calculation, and a practical decision framework for enterprises that need a defensible answer for their CFO — not a philosophical preference for one model over the other.

The Cloud Cost Structure: Updated for 2026

AWS P5 instances (p5.48xlarge: 8× H100 SXM5, 640 GB HBM3, 192 vCPUs, 3,200 Gbps EFA networking) are the hyperscaler reference configuration for H100-class GPU compute. Post the June 2025 price cut, on-demand pricing runs approximately $31–32/hour for the 8-GPU instance — roughly $3.90/GPU-hour. That is a materially different number from the $98/hour figure that appeared in enterprise financial models built before mid-2025. If your TCO analysis still shows $98/hour for AWS H100, it is using pre-cut pricing and should be redone.

Reserved and savings plan pricing discounts are still meaningful. AWS 1-year savings plans bring effective rates to approximately $2.50–2.75/GPU-hour. Three-year commitments can reach $1.90–2.10/GPU-hour. Azure NDv5 (8× H100) and GCP A3 (8× H100) are priced comparably at on-demand, with GCP spot instances available at approximately $2.25/GPU-hour.

Important pricing nuance: Specialist GPU cloud providers — CoreWeave, Lambda Labs, RunPod — offer H100 at $2.99–3.99/GPU-hour on-demand with no hyperscaler overhead, and as low as $1.49/GPU-hour on spot. For enterprises that do not require AWS/Azure/GCP ecosystem integration, these providers are worth modeling as a third option. The analysis below focuses on hyperscaler pricing as the baseline, but the breakeven calculus shifts further toward cloud when specialist pricing is used.

The spot/preemptible pricing tier deserves explicit treatment. GCP spot at $2.25/GPU-hour and RunPod community cloud at $1.99/GPU-hour sound compelling on paper, but spot instances are interruptible with 30-second to 2-minute warning. They are appropriate for batch training jobs with checkpoint-and-resume capability, offline inference pipelines, and hyperparameter sweeps. They are not appropriate for production inference APIs, training runs that cannot tolerate interruption, or any workload with latency SLAs. Treating spot pricing as a planning baseline is a common enterprise error that leads to operational incidents and inflated true-cost-per-useful-GPU-hour.

The On-Premise Cost Structure: The Full TCO

The sticker price of on-premise GPU hardware is consistently the least accurate input in enterprise TCO models. Here is what a realistic 3-year on-premise TCO looks like for an 8-GPU H100 system.

Hardware acquisition: An 8× H100 SXM5 server (NVIDIA DGX H100 or OEM equivalent from Dell, HPE, Lenovo) runs $300,000–$400,000 at current market pricing. H100 PCIe cards cost $25,000–$30,000 each; SXM5 variants run $35,000–$40,000. An 8-GPU HGX H100 system with optimized software stack typically starts around $350,000–$450,000. Networking adds InfiniBand HDR or NDR switching for multi-node training: $50,000–$100,000 per rack depending on port count and topology.

Annual operating costs are frequently underestimated. Power: each H100 SXM5 draws approximately 700W at peak, so an 8-GPU system requires 8–10 kW, adding $8,000–$15,000 per year in electricity at US commercial average rates ($0.12/kWh). Cooling adds another $5,000–$12,000 annually (liquid cooling is more efficient at $0.09/kWh equivalent versus air at $0.18/kWh). Colocation for high-density GPU racks runs approximately $1,500/month ($18,000/year) for power-dense configurations. Annual maintenance contracts typically run 12% of system cost. Total annual operating cost for a single 8-GPU node: approximately $80,000–$120,000. Over 3 years: $240,000–$360,000.

Fully-loaded 3-year TCO for an 8-GPU H100 on-premise node: approximately $590,000–$760,000. The operational overhead of GPU cluster management — CUDA/driver maintenance, hardware failure handling, monitoring, capacity planning — adds another 20–30% in engineering time that is frequently treated as a fixed cost but is genuinely incremental. When that is included, the upper bound of on-premise TCO for a 3-year horizon approaches $900,000 for a single 8-GPU node.

The Utilization Threshold: Where the Math Actually Lives

The most important number in the on-premise vs. cloud decision is not the hardware price. It is the utilization rate — the percentage of time the GPU is doing useful work versus sitting idle while still incurring cost (on-premise) or not being paid for (cloud). This asymmetry is the entire economic argument for cloud GPU in most enterprise scenarios.

On-premise infrastructure has fixed cost regardless of utilization. An 8-GPU H100 node with a 3-year TCO of $675,000 costs the same whether it runs at 30% utilization or 95% utilization. The cost-per-useful-GPU-hour scales inversely with utilization: at 60% utilization, you are paying for 40% of GPU-hours you are not using. At 40% utilization, you are paying for 60% of wasted capacity. The breakeven analysis from Lenovo’s TCO study places the crossover point at approximately 8,556 hours of continuous use — roughly 12 months of 24/7 operation. Below that, cloud is cheaper. Above it and over a 3-year horizon, on-premise begins to accumulate savings.

The critical question is: what do typical enterprise AI workloads actually achieve? Training workloads run in bursts — a model training run consumes high GPU utilization for days or weeks, then the cluster sits idle while the team evaluates results, prepares the next experiment, or waits for data. Development and experimentation workloads are highly variable by nature. According to analysis cited by Introl, most enterprise AI workloads run below 65% sustained utilization. The Introl analysis found that cloud breaks even with on-premise at approximately 40% utilization — meaning below that point, cloud is cheaper than on-premise even at the low end of on-premise TCO.

The takeaway from this curve is that the on-premise vs. cloud decision is not primarily about scale — it is about utilization rate discipline. A large on-premise GPU cluster running at 40% utilization is a worse financial decision than a smaller reserved cloud commitment that matches your actual workload demand. Most enterprises that choose on-premise do so for the wrong reason: they compare the cost-per-GPU-hour of fully utilized on-premise hardware against on-demand cloud pricing, ignoring the actual utilization they will achieve.

“The single most valuable input in the on-premise vs. cloud TCO calculation is not the hardware price or the cloud rate — it is the utilization rate you will actually achieve. Most enterprises that get this decision wrong are optimizing for a utilization they are not running today and may never reach.”

The Blackwell Complication: Why You Cannot Use H100 Numbers

The economics above apply to H100-class hardware. Blackwell changes the TCO calculation in ways that make rule-of-thumb estimates unreliable, and the direction of the change depends on your workload.

On the hardware cost side, B200 GPU systems are significantly more expensive. An 8-GPU DGX B200 system runs approximately $600,000–$800,000 at current list pricing — roughly 1.6–2× the H100 equivalent. Cloud pricing reflects this: AWS Capacity Blocks for B200 run approximately $9.36/GPU-hour, versus $3.90 for H100. Lambda Labs B200 pricing is approximately $5.50/GPU-hour. At these cloud rates, the on-premise B200 breakeven may actually occur at lower utilization than H100 — cloud B200 is expensive enough that on-premise becomes attractive earlier in the utilization curve. But this requires workload-specific validation, not a rule-of-thumb estimate.

On the performance side, Blackwell’s inference throughput advantage over H100 is substantial: approximately 2.5–3× on 70B parameter models based on MLPerf benchmarks. This means that serving the same inference load requires fewer GPUs — which directly affects TCO in a way that the hardware price comparison alone does not capture. If your current inference workload requires eight H100s, it may require only three or four B200s. At that reduced GPU count, the on-premise fixed cost and operational overhead are proportionally lower. The correct TCO comparison for Blackwell is cost-per-million-tokens or cost-per-inference-request, not cost-per-GPU-hour.

For Blackwell specifically: do not model TCO from list prices and cloud rates alone. Benchmark your workload on the specific hardware configuration, establish your tokens-per-second (or inference-requests-per-second) output, and compute cost-per-output-unit. The answer will be different from your H100 model even if the GPU counts are adjusted proportionally.

Hidden Costs That Break On-Premise Models

Three categories of costs are routinely underestimated in on-premise GPU TCO models, and each one is large enough to shift the breakeven threshold materially.

Data egress from cloud. If your training data lives in S3, Azure Blob, or GCS, moving it to an on-premise training cluster incurs egress fees. AWS charges $0.09/GB for outbound data transfers above 10 TB/month. A 1 PB training dataset costs $92,000 to egress from AWS — once. Organizations that need to sync training data iteratively face recurring egress costs that are not visible in the GPU pricing comparison. This cost is specific to organizations whose data already lives in cloud object storage, but that is increasingly the majority of enterprise AI programs.

GPU cluster operational overhead. Running an on-premise GPU cluster requires engineering time for CUDA driver management, fabric and networking maintenance, hardware failure response, monitoring, scheduling optimization, and periodic firmware updates. This is genuine incremental engineering work — it does not come free with the hardware. Industry estimates put this at 20–30% of on-premise hardware cost in annual engineering overhead. For a $675,000 hardware investment, that is $135,000–$202,000 in engineering cost over 3 years that belongs in the TCO model but frequently does not appear there.

Technology obsolescence and hardware refresh cycles. GPU architectures evolve on approximately 2-year cycles. A 3-year on-premise H100 investment is a bet that H100-class performance will still be competitive with your workload needs in year 3. Given that B200 delivers 2.5–3× H100 inference throughput today, that bet is not obviously correct for inference-heavy enterprises. Cloud infrastructure absorbs the technology refresh risk — you upgrade instance types when new hardware is available. On-premise infrastructure requires you to carry that risk yourself, and the residual value of a 3-year-old H100 cluster is not zero but it is meaningfully lower than acquisition cost.

The Colocation Option: A Middle Path

The on-premise vs. cloud framing omits a third model that is increasingly relevant for enterprise GPU deployments: colocation (colo). In a colo model, the enterprise owns the GPU hardware but houses it in a third-party data center that provides power, cooling, physical security, and network connectivity. The enterprise pays a colocation fee (approximately $1,500/month for high-density GPU racks) but avoids cloud markup on compute.

Colocation captures most of the TCO advantage of on-premise (you own the hardware, you pay no cloud compute markup) while offloading facility management and avoiding the capital and engineering cost of building or upgrading your own data center. For enterprises with existing colo relationships, this is often the best economic model for sustained, high-utilization GPU workloads — particularly inference, where latency control and predictable cost matter most. The key constraint is that colocation at GPU densities (40–120 kW/rack for Blackwell-class systems) requires purpose-built high-density facilities, not all of which are available in every market.

A Practical Decision Framework

The decision tree below is not a formula — it is a structured set of questions that, if answered with accurate data, produces a defensible infrastructure recommendation. The sequence matters: you cannot answer question 3 without having done questions 1 and 2.

• Measure actual GPU utilization for your current workloads. Not projected utilization — actual utilization from the past 90 days, by workload type. If you do not have GPU utilization telemetry, instrument it before making any infrastructure decision. The number you get will almost certainly be lower than your mental model suggests. Most enterprise AI teams discover their GPU utilization is 35–55% when they measure it for the first time.
• Separate workloads into utilization profiles. Classify each workload as: (a) sustained high-utilization — production inference running 24/7, continuous training pipelines with defined schedules; (b) bursty — training runs, batch inference, experimentation; or (c) unpredictable — development, prototyping, evaluation. Different profiles have different optimal infrastructure models. Mixing them into a single TCO calculation produces an answer that is wrong for all of them.
• Apply the 65% utilization threshold to each workload class. Sustained high-utilization workloads (>65% over 3 years) favor on-premise or colocation. Bursty workloads favor reserved cloud for baseline plus spot for burst. Unpredictable workloads should use on-demand or specialist GPU clouds with no long-term commitment. Do not apply a single infrastructure model to all workloads — the economics diverge significantly by profile.
• Model data residency and egress costs explicitly. If your training data lives in cloud object storage, add egress fees to your on-premise TCO. If data residency or regulatory requirements mandate on-premise processing, that is a constraint that overrides the financial calculation — but it should be stated as a constraint, not treated as an economic argument for on-premise.
• Budget the operational overhead honestly. Add 20–30% of hardware cost as a 3-year engineering overhead line item for on-premise GPU cluster management. If your team does not currently have GPU cluster operations expertise, add the cost of hiring or training it. This is real cost that belongs in the model.

The Answer Enterprises Are Looking For

The answer is almost never all-cloud or all-on-premise. The enterprises that are making the most efficient GPU infrastructure decisions in 2026 are running hybrid architectures with explicit policies for which workloads go where: on-premise or colo for sustained production inference at scale (where predictable cost and latency control matter most); reserved cloud for training workloads with defined schedules that benefit from committed pricing; on-demand or specialist GPU cloud for experimentation, development, and workloads that are still finding their utilization patterns.

What makes this work is not the architecture — it is the measurement discipline underneath it. GPU utilization tracking, per-workload cost attribution, and regular TCO reviews (at least annually, given how fast cloud pricing moves) are the operational foundation that keeps the hybrid model economically rational over time. Without them, the hybrid model drifts: teams default to cloud for everything because it is frictionless, or on-premise hardware accumulates in data centers running at 35% utilization while the team pays for cloud burst on top of it.

The business case for GPU infrastructure investment is increasingly something CIOs need to defend at the board level — not because the investment is wrong, but because the numbers are large enough that a defensible, data-grounded analysis is a governance expectation. The framework above provides the structure for that analysis. The inputs — actual utilization, fully-loaded TCO, workload-specific performance benchmarks — require engineering rigor to measure. But once measured, the financial model is straightforward: cost per useful GPU-hour, compared across options, at realistic utilization rates. Everything else is detail.

— ✦ —

This is the fourth post in the 2026 enterprise AI infrastructure series. For the supply chain context governing what GPU hardware is actually available to purchase, see The GPU Supply Chain Crisis: What Every Enterprise CIO Must Know in 2026. For the inference economics that shape the utilization profile of production AI workloads, see The Inference Economy: Why 2026 Is the Year GPU Workloads Shift from Training to Inference. For physical AI deployments and their distinct three-tier infrastructure requirements, see Physical AI and the Continuous GPU Workload: From Data Centers to the Real World.

References

1. AWS H100 P5 instance (p5.48xlarge) on-demand pricing: approximately $3.90/GPU-hour post June 2025 44% price cut; 3-yr savings plan: ~$1.90–2.10/GPU-hour. IntuitionLabs, H100 Rental Prices Compared: $1.49–$6.98/hr Across 15+ Cloud Providers, November 2025 (updated 2026). intuitionlabs.ai
2. AWS P5 on-demand at $98.32/hr (8-GPU) pre-June 2025; post-cut ~$31/hr. Wring, AWS GPU Instance Pricing Guide, March 2026. wring.co
3. GCP spot H100 at ~$2.25/GPU-hour; Lambda Labs at $2.99; RunPod at $1.99 (community); CoreWeave reserved up to 60% off. Silicon Analysts Cloud GPU Pricing Tracker, April 2026. siliconanalysts.com
4. B200 cloud pricing: AWS Capacity Blocks ~$9.36/GPU-hour; Lambda Labs ~$5.50/GPU-hour. Silicon Analysts, ibid.
5. On-premise 8× H100 hardware: $300K–$400K; power/cooling: $50K–$200K; networking: $30K–$100K; annual maintenance: 12% of system cost. 3-year on-premise TCO range $590K–$760K (excl. ops overhead). GMI Cloud, How Much Does It Cost to Rent or Buy NVIDIA H100 GPUs, 2025. gmicloud.ai
6. On-premise TCO with 70% utilization: ~$2.85/GPU-hour; breakeven at ~60% continuous usage. GMI Cloud ibid.
7. Lenovo TCO analysis: breakeven at approximately 8,556 hours (~12 months) of continuous use. On-premise 5-year savings of 65% vs cloud at high utilization. Lenovo Press, On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition), February 2026. lenovopress.lenovo.com
8. Lenovo 2025 edition: breakeven ~11.9 months at continuous utilization vs. P5 on-demand ($98.32/hr). Lenovo Press 2025. lenovopress.lenovo.com
9. Cloud breaks even at 40% utilization; on-premise wins above 60%; hidden costs: egress ($0.09/GB), storage ($0.10/GB/mo); on-premise 5-yr TCO 65% less than cloud at high utilization. Introl, Hybrid Cloud Strategy for AI, January 2026. introl.com
10. GPU cluster operational overhead: 20–30% of hardware cost in engineering time. Data egress: AWS $0.09/GB above 10 TB/month; 1 PB = $92,000 egress. Introl, ibid.
11. Blackwell (B200/B300) delivers >3× H100 throughput for 70B model inference (MLPerf benchmarks); 5-year B300 on-premise saves $5.2M vs cloud at continuous utilization. Lenovo TCO 2026 edition, ibid.
12. Colocation pricing: ~$1,500/month per rack for high-density GPU power; electricity at $0.12/kWh US commercial average; cooling: $0.09/kWh liquid, $0.18/kWh air. Lenovo TCO 2026 edition, ibid.
13. GPU cluster management adds 20–30% to on-premise TCO; idle GPU time at 30–60% utilization for most ML pipelines. H100 PCIe amortized: ~$3.08/hr at 24/7 utilization. Electronics.alibaba.com, NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis, March 2026. electronics.alibaba.com

Featured image designed by Magnefic