This is Part 3 of the series. In Part 2, I argued that open vs. closed is a control decision, not a quality decision. Now let’s make that practical: which workloads belong where, and what does the platform look like that lets you run both without creating a governance mess?
My thesis is simple. The model is becoming a replaceable component; the platform around it is the durable asset. Enterprises that hard-wire a single model into every application will spend 2027 rewriting integrations. Those that put a governed routing layer in front of a portfolio of models will swap models like they swap database versions today.
A workload-placement framework
I place every candidate workload on two axes: how sensitive is the data and how much frontier capability does the task genuinely need. Volume is the tiebreaker.
| Workload pattern | Data sensitivity | Capability need | Best fit | Why |
|---|---|---|---|---|
| Document classification, extraction, summarization at scale | High | Moderate | Open weight, self-hosted | High volume, regulated data, a fine-tuned 8–30B model is usually enough |
| Internal knowledge assistant (RAG) | High | Moderate | Open weight | Data stays in boundary; retrieval does the heavy lifting |
| Code assistance on proprietary repos | High | High | Open weight or dedicated closed | Open models are near parity on code; IP stays home |
| Complex multi-step analysis, strategy, research | Low to medium | Very high | Closed frontier | Hardest reasoning still favors the frontier |
| Customer-facing chat on public information | Low | Moderate | Either — decide on cost | Volume and latency drive the choice |
| Edge and offline (branches, devices, field) | Varies | Low to moderate | Small open weight model | No reliable connectivity to an API |
| Air-gapped and sovereign workloads | Very high | Varies | Open weight only | No external call is permitted |
Two patterns come up over and over:
- Route by task, not by team. The same application may send a routine extraction to a small open model and escalate an ambiguous case to a frontier model.
- Prototype closed, scale open. Teams often validate a use case on a frontier API in weeks, then distill or fine-tune an open model once the task is well understood and the volume justifies it.
The reference architecture
Here is the platform I recommend. It has five layers, and the most important one is the gateway.
Reference architecture for a multi-model enterprise AI platform
- Consumers. Applications, copilots, agents, batch pipelines and developer tools. None of them talk to a model directly. Ever.
- The enterprise AI gateway. This is the control point. Every request is authenticated, checked against a routing policy (which models may see which data classifications), scrubbed for PII where required, rate-limited, logged for audit and metered for cost. If you build only one thing this year, build this.
- Two model zones. Closed frontier models sit outside your boundary, reached over private connectivity with zero-retention contractual terms. Open weight models sit inside, served by an inference engine such as vLLM or SGLang on Kubernetes, fed from a model registry that holds signed, scanned and version-pinned weights, and supported by a fine-tuning factory that produces LoRA adapters for specific tasks.
- Shared services. Retrieval and vector search, guardrails and continuous evaluation, and observability with FinOps. These must be model-agnostic — the same eval suite should score a closed and an open model on the same task so you can make routing decisions on evidence.
- Foundation. A governed data platform (lineage, entitlements, classification) and GPU capacity — on-premises, in a sovereign cloud, or reserved from a hyperscaler.
Design principles that make it work
- Treat weights like code. Scan every checkpoint, prefer safetensors over pickle-based formats, sign artifacts and promote them through environments with the same rigor as a production binary.
- Make routing a policy, not a code change. Data classification plus task type should determine the eligible model set, and the gateway enforces it.
- Evaluate continuously on your own data. Public leaderboards tell you very little about your claims-processing workflow.
- Plan GPU capacity around utilization. Pool inference capacity across use cases; a dedicated GPU per project is how open weight economics fall apart.
- Keep an exit at every layer. OpenAI-compatible APIs at the gateway and the serving layer make swapping models a configuration change.
In Part 4, I’ll show how a bank put this architecture to work — and what it learned along the way.
Featured image designed by Magnific