The GenAI stack, end to end — pretraining to the ECU.
Seven engineering exhibits across training, serving and in-vehicle edge. One model lineage flows through all of them: what's trained in the cloud is quantized, escalated, and safety-gated before it influences vehicle behavior. Metrics are representative of the configurations described — each carries an I flag and names its method. Every architecture here is open to scrutiny; that's the point of a technical review.
Agents as a typed graph of roles. I
EXHIBIT 01 / 07A planner decomposes the goal across specialised agents; each acts through a governed MCP tool layer and shared persistent memory. A meta-cognitive monitor watches trajectories and re-plans on failure.
Pretraining a 1.5B domain model from scratch. I
EXHIBIT 02 / 07A decoder-only transformer trained from random init on a curated automotive-software corpus — embedded C, AUTOSAR ARXML, MISRA rules, diagnostics — with a domain-extended tokenizer and a final annealing phase on the highest-quality data.
3D-parallel across 256× H100 on Azure. I
EXHIBIT 03 / 07Pretraining sharded with data, tensor and pipeline parallelism across an Azure ND H100 v5 cluster on NDR InfiniBand — ZeRO-3 sharding, activation checkpointing, async checkpoints to blob storage every 12 minutes.
vLLM replicas behind a Ray Serve mesh. I
EXHIBIT 04 / 07vLLM gives each replica PagedAttention, continuous batching and prefix caching; Ray Serve routes, multiplexes and autoscales. FP8 weights and speculative decoding lift throughput without hurting tail latency. Deep-dive on serving economics: Sheet 05 →
RAG that understands code structure — 8.4M LOC. I
EXHIBIT 05 / 07The repository is indexed three ways — AST-aware semantic chunks, a symbol and call graph, and a lexical index. A hybrid retriever fuses the signals, re-ranks with a cross-encoder, and assembles context the model can actually reason over. An agentic loop re-retrieves when the first pass falls short.
Fine-tuning an SLM for embedded-C and MISRA. I
EXHIBIT 06 / 07A 3–4B base model specialised with QLoRA on a small, rigorously-cleaned SFT set, preference-aligned with DPO, then 4-bit quantized — recovering ~93% of a full fine-tune's quality at a fraction of the compute. Clean beats large. Methods deep-dive: Reference 02 below →
AI inside the car — two tiers, one safety gate. I
EXHIBIT 07 / 07Quantized Gemma and Phi run on the ECU for hard-real-time local reasoning; heavier queries escalate to an in-vehicle NVIDIA DGX Spark. Both paths feed a safety-gated decision layer before any action reaches an actuator.
The numbers, in one view.
SEC 08 / SHEET 04| Layer | Metric | Result | Method / reference |
|---|---|---|---|
| Pretraining | Domain-eval uplift | +24% | annealing phase · 1.5B model |
| Distributed training | Model FLOPs utilisation | 52% | 256× H100 · 3D-parallel |
| Inference serving | Throughput vs HF baseline | 3.4× | vLLM + Ray · FP8 |
| Inference serving | Cost per 1M tokens | −58% | continuous batching · autoscale |
| Code RAG | Repository-QA accuracy | 88% | 8.4M LOC · hybrid retrieval |
| SLM fine-tuning | MISRA first-pass compliance | 40 → 94% | QLoRA + DPO |
| SLM fine-tuning | Quality vs full fine-tune | ~93% | 0.4% of parameters trained |
| ECU edge | On-ECU first-token latency | <60 ms | Gemma / Phi · INT4 |
| ECU edge | Heavy-path escalation rate | ~18% | escalated to DGX Spark |
The hard parts — where this earns its keep.
SEC 09 / SHEET 04Quantization is a budget, not a free lunch.
INT4 at the edge costs accuracy. Measure the drop per task, keep safety-critical paths gated, and escalate rather than trust a degraded local answer.
MFU is the metric, not GPU count.
Distributed training is won in compute–communication overlap. 52% model-FLOPs utilisation on 256 GPUs beats a larger cluster idling on a comms bottleneck.
Structure-blind RAG fails on code.
Flat chunking loses call graphs and symbol scope. AST-aware chunks plus graph retrieval are the line between a demo and something that survives an 8M-LOC repo.
Throughput is reported net of review.
More generated code creates more to verify. Serving and agent gains are quoted after verification overhead — tail latency and change-fail rate held as guardrails.
From exhibit to playbook.
SEC 10 / SHEET 04Exhibits 02 and 06 showed one pretraining run and one fine-tune. These two references generalise them — when to build an SLM at all, which path, what it costs, and how to tune a base model without burning a cluster.
Building a new SLM: two paths, nine stages.
Own tokenizer, architecture, weights — full control and IP, massive compute. Only justified for a moat or a hard data-sovereignty mandate. Design architecture + tokenizer (~0.5–7B) → assemble a heavily-filtered corpus → distributed pretraining on a TPU/Trainium/GPU pod → shared tuning stages.
Start from Llama / Gemma / Phi / Qwen / Mistral and specialize: optional continued pretraining on domain text → distillation from a frontier teacher → SFT via LoRA/QLoRA on curated instructions → DPO/RLAIF alignment. Faster, cheaper, and usually beats a from-scratch small model on a narrow task.
| Target size | Tokens (20:1) | H100-hrs @ 40% MFU | Cost |
|---|---|---|---|
| 0.5B | 10B | ~120 | $300–1K |
| 1B | 20B | ~480 | $1K–3K |
| 3B | 60B | ~4,300 | $10K–25K |
| 7B | 140B | ~23,000 | $50K–150K |
| 7B over-trained | 15T | ~2.5M | $5M–10M |
Fine-tuning: one of five options — pick deliberately.
| Approach | Use when |
|---|---|
| Prompt engineering | Behavior reachable with base model + few-shot; under ~100 examples |
| RAG | Knowledge is dynamic, large, proprietary; you need citations and freshness |
| Fine-tune (PEFT) | Stable task, 500–50K curated examples; consistent style/format/vocab, or compressing inference cost |
| Continued pretraining | New language/domain with a large unlabeled corpus (10B+ tokens) — e.g. automotive ARXML / MISRA-C |
| From scratch | Almost never — extreme constraints or a novel architecture |
| Method | Trainable | Quality recovery | VRAM (7B) |
|---|---|---|---|
| Full fine-tune | 100% | 100% baseline | ~100–120 GB |
| LoRA | 0.1–1% | 90–95% | ~28 GB |
| QLoRA | 0.1–1% | ~80–95% | ~5–8 GB |
| DoRA | 0.1–1% | > LoRA at equal rank | ~LoRA |
| Adapters / IA³ / prefix | ~0.01% | < LoRA, cheapest | smallest |