From Cloud to Metal: The Natural Evolution of a GenAI Use Case

Strategy · April 2026 · Jacint Gauxachs Marin

"We're getting great results with Claude Opus, but the bill is starting to hurt. What's our next move?"

That sentence, or a variation of it, is the single most frequent opening line I hear in client conversations about GenAI. It is also the moment the discussion shifts from experimentation to architecture. The exciting phase is over; the cost-engineering phase has begun.

I want to share the pattern I keep seeing in these conversations — and, more importantly, keep advocating for — because almost every serious GenAI initiative ends up walking the same path. It starts in the cloud with a frontier model, matures through prompt engineering and fine-tuning, graduates to open-source weights, and eventually touches real hardware in a rack. Not because "on-prem is back", but because the economics of inference, the maturity of open-source models and the arrival of a new generation of GPUs have quietly made the math work. And for a growing set of use cases, compliance makes the decision for you.

Phase 1 — The Cloud Honeymoon

Every GenAI conversation I've had in the last three years starts the same way: a managed API, typically Anthropic, OpenAI, Gemini or one of the Bedrock / Azure / Vertex wrappers. There's no debate here, and there shouldn't be. When you need to validate whether an idea is usable at all, you don't want to be shopping for GPUs — you want a production-grade frontier model answering your first curl in under five minutes.

The value proposition is hard to beat:

Zero CapEx, predictable OpEx, pay-per-token.
State-of-the-art quality on day one (Claude Opus 4.7, GPT-5.5, Gemini 2.5 Pro).
Compliance, SLAs, data residency options already negotiated by the hyperscaler.
Feature velocity: every few months the baseline gets better while you sleep.

This is the honeymoon. Teams ship the PoC, validate the business case, get internal sponsorship. Usage grows. The bill grows with it. And then, one morning, someone in finance looks at the invoice and the question above lands on someone's inbox.

Phase 2 — The Efficiency Plateau

One of the most common mistakes I see is jumping from "expensive cloud bill" straight to "let's self-host". Don't. There is an entire middle layer of optimization that should exhaust itself first, because it's cheap, reversible and often saves more money than a hardware migration ever will.

This is the phase where you squeeze the frontier model:

Prompt engineering with a cost lens. Long system prompts, verbose tool definitions, repeated few-shot examples — every one of those tokens shows up 100% on every call. Compressing a system prompt from 4k to 1k tokens, at scale, pays for a consultant.
Prompt caching. Anthropic's 90% discount on cached input is not a nice-to-have; it turns static context into a near-free resource. Same story with OpenAI's and Gemini's caching tiers.
Model routing. Gateways like LiteLLM or CLIProxyAPI let you push easy classifications to Haiku 4.5 or Gemini Flash (~$1/$5 and ~$0.30/$2.50 per 1M tokens) and only escalate the complex reasoning to Opus or GPT-5 Pro. A well-tuned router routinely drops the blended cost by 60–80%.
Aggressive RAG. Shorter, more relevant context instead of "throw the whole doc". Semantic caching on the retrieval layer (Redis + embeddings) takes care of recurring questions.
Batch + async. Both Anthropic and OpenAI give 50% off through batch APIs. For anything that isn't strictly synchronous, that's free money.

After a proper optimization pass, many use cases stop bleeding. You've extended the cloud runway by months, sometimes years. But for a certain class of workload — high volume, predictable patterns, sensitive data — the math eventually tips again. That's when Phase 3 enters the conversation.

Phase 3 — The Open-Source Pivot

Two years ago, suggesting Llama or Mistral as a serious alternative to Claude or GPT felt like advocating for a 10% discount in exchange for a 40% quality drop. The usual answer: "not worth it". That equation has silently, decisively flipped.

A handful of releases over the last twelve months have been genuine inflection points. Gemma 3 opened the door with its 27B dense model and QAT quantization. Gemma 4, released on April 2nd, 2026 under Apache 2.0, walks through it: a family of four sizes (E2B for phones, E4B for edge, a 26B MoE with 3.8B active parameters for consumer GPUs, and a 31B dense model for workstations), multimodal input, 256K context window and support for 140+ languages. The 31B dense variant scores 85.2% on MMLU Pro, 89.2% on AIME 2026 and ranks #3 on the Arena AI leaderboard — numbers that would have been unthinkable in an open-weight model just a year ago.

Around Gemma 4, the rest of the open-source field has matured in parallel. GLM-4.6 matches Claude Sonnet on coding benchmarks. Qwen3.6-27B, released in April 2026 under Apache 2.0, genuinely outperforms 397B-parameter MoEs on agentic coding tasks. DeepSeek V3 and R1 broke the MMLU and AIME benchmarks with a permissive license. Phi-4 at 14B parameters is absurdly good for its size, and its reasoning-plus variant beats o1-mini. Kimi K2.6 sits at the top of SWE-Bench with a modified MIT license.

The key move here is not replacement — it's fine-tuning the use case. A Gemma 4 31B model, fine-tuned on a specific domain with a few thousand curated examples, paired with good prompting and solid RAG, will beat a generic frontier model on its own task. The frontier model will always be more versatile; but versatility is not what most production workloads need. They need reliability on a narrow slice.

The trigger in recent conversations has been exactly Gemma 4 — it is the current disruptor, and it runs comfortably on workstation-grade hardware. Which takes the discussion directly into Phase 4.

Figure 1 — The four phases most GenAI initiatives go through, from frontier API validation to owned inference hardware.

Phase 4 — The Hardware Reality Check

A note on the numbers below. Hardware prices in this section reference US retail / reseller channel in USD. European enterprise pricing typically adds 15-25% for VAT, import duties and distributor margin, and local availability fluctuates with the supply conditions described below. Treat these figures as order-of-magnitude anchors, not procurement quotes.

Until very recently, "on-prem LLM inference" meant one thing: H100-class cards in the premium data-center tier — low tens of thousands of dollars each, ordered through a reseller with a multi-month lead time. For most mid-size companies that was a non-starter, and rightly so.

Since Q1 2026 the supply-side reality has hardened considerably: NVIDIA H100, H200 and B200 capacity is effectively gone, CoWoS packaging at TSMC is fully booked through mid-2027, the hyperscalers consumed most of the Blackwell allocation with forward orders placed in 2024-2025, and new buyers face 12-18 month wait times for volume orders. Rental prices on the few cloud providers that still have capacity have jumped 30-40% in three months. This is changing the decision — but not in the direction most people expect. It is pushing mid-size organizations toward on-prem, not away from it, because the cloud fallback has become both more expensive and less reliable.

Two things changed the equation in the last twelve months:

The RTX PRO 6000 Blackwell, 96 GB of VRAM, NVIDIA MSRP in the high single-digit thousands USD (reference: $8,435). This is a workstation-class GPU that fits Gemma 4 31B, Qwen3.6-27B, Phi-4 or Llama 4 Scout in Q4 quantization with comfortable headroom for context. Two of them in a single chassis — low tens of thousands of silicon plus a decent server — cover the inference needs of a very large number of workloads.
The software stack caught up. vLLM with PagedAttention and continuous batching, SGLang for prefix-sharing (agentic workloads, RAG with large system prompts), TensorRT-LLM for single-model production. Ollama makes onboarding a developer's local setup trivial. And ROCm 7 finally turned AMD's MI300X / MI350X into first-class citizens — a single MI350X with 288 GB of HBM3e now runs 520B-parameter models by itself.

For the more demanding cases — multi-hundred-billion parameter models, very high concurrency — H200 (141 GB HBM3e) and B200 (192 GB HBM3e) are the obvious upgrade paths. The B200 in particular is shipping at roughly the price of the old H100, with 2.4× the memory and noticeably better memory bandwidth, though lead times remain painful because of a reported 3.6 million-unit backlog.

A second-order effect of the shortage is that the refurbished market has matured rapidly and is no longer a hobbyist corner. Used NVIDIA A100 80GB cards trade in the low to high four-digit range — roughly 40-70% below new pricing — and full refurbished inference servers (DGX A100 class, HGX A100 class) are available from established vendors with warranties and validated components. For inference workloads the gap vs H100 is surprisingly small, and A100 availability is excellent precisely because organizations are upgrading to Blackwell and releasing their previous-generation fleet onto the secondary market. Many teams standing up their first on-prem inference rack in 2026 are doing so on refurbished A100 or early H100 hardware, cutting CapEx roughly in half and shortening the TCO break-even materially.

The table that most clients want to see looks like this:

GPU	VRAM	Tier / relative cost	Fits (Q4)
RTX 6000 Ada	48 GB GDDR6	Workstation tier, high single-digit k USD	Gemma 4 26B MoE, Phi-4 14B, Qwen2.5 32B (tight)
RTX PRO 6000 Blackwell	96 GB GDDR7	Workstation tier, NVIDIA MSRP ~$8,435	Gemma 4 31B dense, Qwen3.6-27B, Llama 4 Scout, GLM-4.6 (Q4, squeezed)
H100 SXM / PCIe	80 GB HBM3	Premium data-center tier, low tens of k USD	Llama 3.3 70B FP8, Qwen3-235B shards
H200	141 GB HBM3e	Premium DC, meaningful step above H100	Qwen3-235B comfortably, DeepSeek V3 sharded
B200	192 GB HBM3e	Flagship NVIDIA, lead times 12-18 months	Next-gen workloads, single-GPU 200B+ dense
AMD MI300X	192 GB HBM3	H100/H200 class, typically lower list	Same class as H100/H200, ROCm 7 + vLLM
AMD MI350X	288 GB HBM3e	Flagship AMD, premium DC tier	520B-param models on a single card
NVIDIA A100 80GB (refurbished)	80 GB HBM2e	Secondary market, ~40-70% below new	Llama 3.3 70B FP8 (tight), Gemma 4 31B FP16, Qwen3.6-27B BF16 — sweet spot cost/performance for inference

The reason the RTX PRO 6000 Blackwell is worth highlighting for most mid-size cases is pragmatic: it sits in the sweet spot where the hardware cost no longer dominates the conversation. A two-card inference node lands around the cost of two to three months of a moderate Claude Opus bill.

The Compliance Lever

Everything up to here has been a cost argument. But for a growing list of workloads, the decision to go on-prem is not made in finance — it is made in legal and risk. And those rooms don't care about per-token pricing.

The patterns I see again and again:

Banking and insurance. DORA and the EBA guidelines push critical workloads away from non-EU processing. Even within EU regions, specific institutions require bare-metal isolation and keyed access to inference endpoints.
Healthcare. GDPR Article 9, national health data regulations, and sectoral codes of conduct. Sending patient data to a US-hosted model, even through a "compliant" wrapper, is a conversation no CISO wants to have twice.
Defense, critical infrastructure, public sector. ENS-Alto in Spain, BSI-C5 in Germany, SecNumCloud in France. Either the inference runs inside a certified perimeter, or it doesn't run at all.
Legal and professional services. Client privilege. Internal memos and case work cannot leave the firm's technical boundary.

Key point. For this segment, on-prem is not cheaper — it is the only option. The interesting question is no longer "does on-prem pay back?" but "how fast can I stand up a compliant inference platform that behaves like a managed cloud service to my developers?" — because without that last property, adoption stalls.

The TCO Crossover

For everything outside the compliance-mandatory bucket, the decision falls back to math. Let's do it bluntly, because this is where most architecture calls get made or killed.

A Gemma 4 31B-class model running on a two-card RTX PRO 6000 Blackwell node can realistically serve around 1–2 billion tokens per month at steady-state, 24/7, with vLLM batching well. The equivalent volume on Claude Sonnet 4.6 (at $3/$15 per 1M tokens, with a conservative 1:1 input/output ratio) is roughly $18,000/month. The hardware investment — two cards, server, networking, a year of colocation at a reasonable European provider (~$150–220 per kW per month) — pays itself back in six to nine months.

If the same rack is built on refurbished A100 80GB cards, the story gets even more aggressive: a two-card node can be assembled in the low tens of thousands all-in (cards + chassis), which pushes the break-even below four months for the same workload — at the cost of a small throughput discount and slightly older driver stack. For many high-utilization use cases, that trade-off is the right one to make in the current 2026 supply environment.

Figure 2 — Illustrative TCO curve. Cloud APIs scale linearly with usage; on-prem has a steep CapEx step followed by a shallow OpEx slope. The break-even typically lands between month 6 and month 24 depending on model size and utilization.

The break-even timeline changes brutally with model size and utilization:

Small models (<30B) at high utilization: weeks to a few months.
Mid-range (70B-class): roughly two years.
Very large (>200B MoE): five years or more.

The non-obvious cost is utilization. If a node sits at 40% load, cloud wins. Below 65% utilization, the advantage against MaaS quickly shrinks. You need steady traffic or multi-tenancy to keep the GPUs warm. This is why the on-prem story is strongest for organizations with a handful of stable, high-volume use cases — not for those running a long tail of experimental low-volume workloads.

And one more data point that tends to surprise: electricity is noise. A well-tuned Llama-3 70B FP8 deployment on 8×H100s burns around 0.108 kWh per million tokens. At industrial European rates, that's roughly €0.016 per million tokens. The cost of inference is the hardware amortization, not the power bill.

Phase 5 — The Endgame is Hybrid

Nobody goes back to 100% on-prem. That's the most important thing to get right mentally. The endgame is not a migration, it's a portfolio:

On-prem for the stable, high-volume workloads — and for anything compliance flags as "this data does not leave the perimeter". Classification, summarization, code completion on internal repos, chat on internal knowledge bases, processing of regulated data.
Cloud frontier for the hard stuff and the new stuff. Novel reasoning tasks, agentic workflows that need the best model available today, early experiments.
Cloud frontier also for burst capacity on top of your on-prem baseline. The rack handles the median load; cloud absorbs the peaks — though in the current 2026 supply environment even that burst capacity has become expensive and, at times, unavailable, which paradoxically makes a properly sized on-prem baseline more important, not less.

Figure 3 — A simple placement matrix. High volume + narrow scope + regulated data belongs on owned hardware. High diversity + novel reasoning stays in the cloud. The art is to design the gateway so both look the same to the application layer.

Operationally, this means building — or adopting — an inference platform that abstracts the source of the model and wraps it with the non-negotiables: an LLM gateway to hide the provider behind a single interface, guardrails to enforce safety and data leak prevention, smart routing to send each request to the right backend, and semantic caching to cut repeated cost. Your applications should not care whether a request ends up in the company's datacenter or in api.anthropic.com.

Econocom AI Fabric. This is precisely the gap Econocom's AI Fabric platform is designed to fill. It provides a Kubernetes-native way to deploy AI applications with LLM Gateway, Guardrails, Smart Routing and Semantic Cache as first-class services, covering the full cloud-to-on-prem evolution described above from a single control plane. The deep dive — architecture, components, deployment patterns — deserves its own article, which I'll publish next.

Whether through an in-house platform or a packaged one, that layer is, in my experience, the single most underrated architectural decision in any GenAI platform today. It is a one-afternoon piece of plumbing (or a product install) that gives years of optionality.

Closing Thought

When a client asks me "where should we go next?", they're rarely asking about a specific technology. They're asking whether the path they're on has a future. My answer is consistent: yes, and it ends with you owning part of your stack. Not for ideology, not out of fear of lock-in — simply because at a certain point the numbers, the quality of open models, the regulatory pressure and the maturity of the tooling all point in the same direction.

The organizations that will win the next phase of GenAI are not the ones that picked the right model in 2024. They're the ones that design their architecture today so that switching models — and switching where those models run — is a routine operation, not a project. Cloud for velocity, open-source for optionality, owned hardware for economics, a proper platform for compliance and control. That's the formula. Nobody has to choose one corner.

The beauty of this industry right now is that the pieces are, finally, interchangeable.

Back to articles