Enterprise inference, end to end.
Every enterprise LLM/SLM build I've seen reduces to the same two decisions: what unit — an agent (ready-made) or a model (raw) — and which rung of the serving ladder to stand on. This sheet is the working reference: the ladder, the substrate beneath it, the three clouds side by side, and how it all lands in the AUTOSAR V-model. Training-side topics — building and fine-tuning SLMs — live on Sheet 04 · Engineering. Evidence flags throughout: E evidenced · V vendor-reported · I illustrative.
The stack shape: garden → garden → ladder → substrate → silicon.
At the top, the Agent Garden — catalogs of prebuilt agents where the unit is a composite of models, tools, memory and orchestration (GCP Agent Garden/ADK · Azure Foundry agent catalog · AWS AgentCore blueprints). Beneath it, the Model Garden — 200+ foundation models as the raw primitive (Gemini, Claude, Llama, Gemma, Mistral, Phi). Everything below answers one question: how do you serve the thing?
The serving ladder — four rungs of abstraction.
Climb down the ladder only when a constraint forces you to. Every rung trades convenience for control.
| Rung | Service | Billing | Best for | You manage |
|---|---|---|---|---|
| 1 | Managed Model-as-a-Service API | per-token | Frontier closed + curated open models | prompts only |
| 2 | Provisioned / Reserved Throughput | reserved / hr | Predictable latency + committed cost | capacity sizing |
| 3 | Managed Compute Endpoints | VM / hr | Open-weight models on dedicated accelerators | model + scaling |
| 4 | Self-Managed K8s / raw compute | infra / hr | Custom serving stack end-to-end (vLLM, TRT-LLM, TGI, JetStream) | everything |
Six techniques under every rung.
Continuous in-flight batching
New requests join the running batch each decode step — the single biggest GPU-utilization lever.
Paged-attention KV cache
KV cache in non-contiguous pages (vLLM): no wasted padding memory; long context at scale.
Quantization — FP8 / INT8 / INT4
Lower-precision weights shrink memory and raise throughput with minimal accuracy loss. The default cost lever.
Tensor + pipeline parallelism
Shards the model across accelerators so large models still serve at low latency.
Speculative decoding
A small draft model proposes tokens; the big model verifies in parallel — cuts decode-phase latency.
Disaggregated prefill / decode
2026 frontier: split compute-bound prefill from memory-bound decode onto separate pools; KV-cache-aware routing for agentic load.
Azure — Microsoft Foundry. Standout: PTU economics + Phi SLMs.
| Rung | Service | Billing | Best for |
|---|---|---|---|
| Managed API | Foundry serverless (Global / Data Zone / Regional) | per-token | Frontier GPT, Grok, Kimi; fastest start. Data Zone pins processing geography for residency. |
| Reserved | Provisioned Throughput Units (PTUs) | ~$2,448/mo · min 15 | Stable high volume (~150–200M tok/mo ≈ 70% savings). Common: PTU base + PAYG spillover. |
| Managed compute | Azure ML managed online endpoints | VM/hr | Open weights (Llama, Phi, Mistral, HF) inside a managed VNet, gated by Azure Policy. |
| Self-managed | AKS + your runtime | infra/hr | Full control over the serving stack. |
Silicon: ND-series H100/H200, GB200 rolling in, in-house Maia 100 for first-party loads. Plumbing: Entra ID + managed identities · Private Link / managed VNet · Azure Policy · Content Safety · API Management for org-wide token quotas. SLM angle: Phi-4 / Phi-4-mini as cheap routers — ~35–40× cheaper than GPT-4o for classification, extraction and intent routing; a built-in model router dispatches between models.
AWS — Bedrock + SageMaker AI. Standout: plane separation + silicon cost curve.
Two-plane design: Bedrock is the managed/agentic foundation layer; SageMaker AI is the full-control plane.
| Rung | Service | Billing | Best for |
|---|---|---|---|
| Managed API | Amazon Bedrock (Converse API) | per-token | Claude, Llama, Nova/Titan, Mistral, Cohere — plus Guardrails, Knowledge Bases, Agents & Flows. |
| Reserved | Bedrock Provisioned Throughput (model units) | reserved + commit | Steady production; required for custom/fine-tuned models. Cross-region inference profiles spread load. |
| BYO serverless | Bedrock Custom Model Import | serverless | Fine-tuned open weights served serverlessly — spiky load with full AWS integration. |
| Managed compute | SageMaker endpoints + LMI containers | hr / per-inference | Full control; JumpStart one-click; Inference Components pack multiple models per endpoint. |
| Self-managed | EKS + Neuron / NVIDIA | infra/hr | Bespoke serving on Inferentia/Trainium or GPUs. |
Silicon: Annapurna ASICs — Inferentia2 + Trainium2/3; the majority of Bedrock token usage already runs on Trainium. NVIDIA p4d/p5/p6; Graviton CPUs for agent logic; the Neuron SDK compiles to both ASICs. Plumbing: PrivateLink VPC endpoints (traffic never touches public internet) · IAM · KMS · CloudTrail + invocation logging · Guardrails; prompts/outputs not used to train base models.
GCP — Vertex AI + GKE. Standout: frontier serving (disaggregation, KV-on-chip).
| Rung | Service | Billing | Best for |
|---|---|---|---|
| Managed API | Vertex AI API · Model Garden (200+ models) | per-token | Gemini + partner/open models as managed APIs. |
| Reserved | Vertex Provisioned Throughput (GSUs) | reserved | Predictable latency & cost for production traffic. |
| Dedicated endpoint | Vertex AI Prediction endpoints | GPU/TPU · hr | One-click deploy of open weights onto dedicated accelerators. |
| Scale-to-zero | Cloud Run + GPU | per-use | Spiky / smaller models (e.g. Gemma) with scale-to-zero economics. |
| Self-managed | GKE + Inference Gateway + NVIDIA Dynamo + vLLM/JetStream | infra/hr | High-concurrency agentic serving with disaggregated prefill/decode + tiered KV caching. |
Silicon: TPUs — Trillium (v6e), Ironwood (TPU7x), and the '26 split: TPU 8i for inference (384 MB SRAM, 288 GB HBM, +80% perf/$) and 8t for training. NVIDIA H200/B200/GB200, Vera Rubin NVL72 (H2 2026); Axion N4A (Arm) for agent runtimes. Plumbing: VPC Service Controls · Private Service Connect · IAM · CMEK · data-residency controls · Gemini Enterprise governance.
The matrix — ★ marks the differentiated cell.
| Dimension | Azure | AWS | GCP |
|---|---|---|---|
| Umbrella | Microsoft Foundry | Bedrock + SageMaker AI | Vertex AI + GKE |
| Managed API | Foundry serverless | Amazon Bedrock | Vertex AI API / Model Garden |
| Reserved unit | PTU ★ | Model units | GSU |
| Self-host (managed) | Azure ML managed compute | SageMaker / Custom Model Import | Vertex Endpoints / Cloud Run GPU |
| K8s serving | AKS | EKS | GKE + Inference GW + Dynamo ★ |
| Custom inference ASIC | Maia (limited) | Inferentia2 / Trainium2-3 ★ | TPU 8i / Trillium / Ironwood ★ |
| Top NVIDIA | ND H100/H200, GB200 | p5 / p6 H200, GB200 | H200/B200/GB200, Vera Rubin |
| Agent-runtime CPU | — | Graviton | Axion N4A |
| Unified API | AI Model Inference API | Bedrock Converse API | Vertex AI API |
| Network isolation | Private Link / managed VNet | PrivateLink VPC endpoints | VPC-SC + Private Service Connect |
| Safety / guardrails | Content Safety | Bedrock Guardrails | Model Armor / safety filters |
Three questions pick your cloud and rung.
Closed or open?
Frontier model behind an API, or open weights you control, fine-tune, and host?
Stable volume?
Is throughput consistent — 3+ months of real data — enough to justify reserved capacity over per-token?
How hard is the perimeter?
Data-residency and network-isolation constraints decide the deployment tier and which zone / VNet / VPC-SC lever to pull.
Platform-agnostic by design.
Treat each cloud's inference layer as a swappable backend behind a stable interface. The portable core is where your IP lives: prompts & eval suites, orchestration (LangGraph / MCP / A2A), the RAG store, Terraform, and a model gateway — all speaking an OpenAI-compatible surface down to whichever backend wins this quarter.
OpenAI-compatible surface — the de-facto lingua franca; swapping a model becomes a config change.
Model gateway / router — LiteLLM, Envoy AI Gateway: one API, any backend; central keys, quotas, fallback, A/B, cost control.
Containers + Kubernetes — AKS/EKS/GKE run the same manifests; KServe/Knative as the portable serving layer.
Portable orchestration — LangGraph / LlamaIndex / Semantic Kernel for flow; MCP for tools; A2A between agents. Open protocols over proprietary builders.
Bring-your-own RAG store — pgvector / Qdrant / Weaviate / Milvus behind a retriever interface, not a proprietary vector service.
Terraform — multi-cloud provisioning; prompts, configs and eval suites live in version control, not a console.
The service Rosetta Stone.
| Layer | Azure | AWS | GCP |
|---|---|---|---|
| Object storage | Blob Storage | S3 | Cloud Storage |
| Vector store | AI Search · Cosmos DB | OpenSearch · Aurora pgvector | Vertex Vector Search · AlloyDB |
| Data pipeline / ETL | Fabric · Data Factory | Glue · EMR · Step Functions | Dataflow · Dataproc |
| Model catalog | Foundry Model Catalog | Bedrock · JumpStart | Model Garden |
| Managed inference | Microsoft Foundry | Amazon Bedrock | Vertex AI |
| Train / fine-tune | Azure ML · Foundry | SageMaker · JumpStart | Vertex Training · Tuning |
| Embeddings | Azure OpenAI embeddings | Titan · Cohere Embed | text-embedding · Gemini Embed |
| Pipelines | Azure ML Pipelines | SageMaker Pipelines | Vertex AI Pipelines |
| Model registry | Azure ML Registry | SageMaker Model Registry | Vertex Model Registry |
| Agent framework | Semantic Kernel · Foundry Agents | Strands · AgentCore SDK | ADK |
| Agent runtime | Foundry Agent Service | Bedrock AgentCore Runtime | Vertex Agent Engine |
| Managed RAG / search | Azure AI Search | Knowledge Bases · Kendra | Vertex AI Search |
| Kubernetes | AKS | EKS | GKE |
| Serverless container | Container Apps | App Runner · Fargate | Cloud Run |
| Evaluation | Foundry Evaluations | Bedrock Evals · Clarify | Gen AI Evaluation Service |
| Safety / guardrails | Content Safety | Bedrock Guardrails | Model Armor |
| Observability | Azure Monitor · App Insights | CloudWatch · CloudTrail | Cloud Monitoring |
| Secrets / identity | Key Vault · Entra ID | Secrets Manager · IAM | Secret Manager · Cloud IAM |
LLM vs SLM — what the 2026 evidence actually says.
| Capability | Frontier LLM | Mid SLM 7–14B | Edge SLM 1–4B |
|---|---|---|---|
| SWE-bench Verified | 80–88% E | 50–70% E | 20–40% E |
| SWE-bench Pro (realistic) | 45–58% E | 25–40% E | <20% I |
| MISRA-C compliance | ~94% agentic V | usable + gate I | weak I |
| AUTOSAR ARXML / SWC | co-pilot V | 86.4% test cov V | not advised I |
| CAD via CadQuery | 69%+ exact E | lower (scaling) E | weak I |
| Simulink test / mutants | 13× faster E | viable SIL E | weak I |
| In-vehicle perception | not deployable | distilled VLM I | edge VLM E |
CAD verdict: viable as a co-pilot for early parametric concepts, brackets, fixtures and jigs (Zoo.dev ships in production); weak on exact tolerancing; production CATIA / NX / Creo stays human-authored. ECU flow: MISRA auto-fix 40→94% in production V; Scania Spec2code closes the loop with formal verification (Frama-C/ACSL) E; requirements→CARLA scenes hit up to 98% pass E; calibration/flashing remains vendor-tool territory I.
AI layering in the AUTOSAR V-model.
Two distinct AI surfaces: AI-in-the-toolchain (assisting engineers across both arms of the V) and AI-in-the-vehicle (running on Adaptive AUTOSAR compute). Twenty-one+ concrete use cases map onto the V — from requirement extraction to homologation evidence. The phase-level uplifts:
| Phase | Effort reduction | Flag | Representative use cases |
|---|---|---|---|
| Requirements & analysis | ~40% | I | extraction · ambiguity checks · EARS formalization · HARA/FMEA/SOTIF seeding |
| Architecture / ARXML | ~32% | I | SWC decomposition · ARXML & ara::com interface generation |
| Implementation | 26–55% | E | embedded C/C++ generation · MISRA auto-fix · RTE/BSW scaffolding · Classic→Adaptive migration |
| MISRA compliance | ~55% | V | generate → static-analyze → AI-repair → re-analyze loop |
| Unit / integration test | 3–4× | V | vector generation · MC/DC gap-fill · SWC-to-SWC and bus-level tests |
| HIL / scenario / mutants | ≤13× | E | OpenSCENARIO/CARLA generation · CAPL/Python HIL scripts · Simulink mutants |
| Docs & traceability | ~55% | I | traceability maintenance · ASPICE work products · change-impact analysis |
| Calibration / diagnostics | ~20% | I | A2L recommendations · DTC/UDS explanation (advisory only) |
| Safety-critical control | 0% | I | deterministic control law only — no LLM/SLM, by design |
| Layer | Runs on | Model class | Workload |
|---|---|---|---|
| Engineering co-pilot | On-prem / VPC GPU | Frontier LLM | Requirements · HARA · complex refactor |
| Domain SLM (factory) | On-prem vLLM / TRT-LLM | FT 7–14B (Phi-4 · Qwen3-Coder · Codestral) | MISRA fixes · ARXML · CAPL — IP-sensitive |
| In-vehicle SLM | Adaptive AUTOSAR (Orin / Ride) | 1–4B FP8/INT4 (Gemma 3 · Phi-4-mini) | In-cabin voice · manual lookup |
| Edge perception | Adaptive AUTOSAR (AD compute) | Transformer / BEV nets (not LLMs) | Detection · lane · free-space |
| Safety-critical control | Classic AUTOSAR MCU | No LLM/SLM | Powertrain · chassis · brakes |
| Level | Autonomy | Where in 2026 | Ceiling |
|---|---|---|---|
| L0 | Manual — no AI in loop | Legacy safety-critical control | — |
| L1 | Suggest — inline completion | Hand-written ASIL-D C in Classic ECUs | safe everywhere |
| L2 | Generate + review — AI drafts, human edits | Most current use cases — code, tests, ARXML, docs | safe with gate |
| L3 | Supervised agent — human approves at gates | Verifier loops · test gen · scenario gen | non-safety / gated |
| L4 | Autonomous + gate — human approves final product | Rare — internal tooling & documentation | non-safety only |
Serving-runtime reference.
| Runtime | Backer | Strength | Where |
|---|---|---|---|
| vLLM | Open / broad | PagedAttention, continuous batching, widest model support | SageMaker LMI · Vertex Model Garden · GKE · Azure managed compute · vLLM-TPU |
| TensorRT-LLM | NVIDIA | Max GPU throughput, FP8, in-flight batching (packaged as NIM) | NVIDIA GPUs on all three clouds |
| TGI | Hugging Face | Easy HF model serving, solid defaults | HF deployments on Azure / AWS / GCP |
| JetStream / MaxText | TPU-optimized throughput | GCP TPU serving | |
| Neuron SDK (+ NxD) | AWS | Compiles + serves on Inferentia / Trainium | AWS inf2 / trn instances |