Compiling an Agentic Workflow into LLM Weights: the case of the agent that needs no orchestrator
"What if the procedure wasn't a prompt, but the weights of the model?"
In 2026, every production agent is built the same way: a frontier model in the cloud, an external orchestrator that picks the next flow node at every turn, tools injected into the system prompt on every call, and a bill that grows with every conversation. It works. It is expensive. And, for many use cases, it is architecturally unnecessary.
Dennis et al. published a paper in May 2026 that deserves a careful read: Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost. The thesis is direct: if the procedure is stable, instead of injecting it into the prompt turn after turn, embed it into the model's weights via fine-tuning. At runtime there is no orchestrator. The model, alone, decides when to triage, when to open a ticket, when to escalate. It is the workflow itself that learns to live in the weights.
This article tells how we tested it end-to-end on a real case, the numbers we ended up with, and the walls we hit along the way. Spoiler: a Qwen 7B compiled for IT support with Jira actions reaches 91% of the quality of Claude Sonnet used as an in-context agent across five evaluation metrics, with perfect tool-call validity (100%) and a per-conversation cost between ~15 and ~30 times lower. On a local 7900XTX, inference cost approaches zero.
For this pattern —procedure compiled into the weights, no orchestrator at runtime— we coined a project-internal name: subterranean agents. The metaphor is literal: the procedure ends up buried in the weights, beneath the surface of the visible prompt. There are no nodes injected each turn, no framework receiving control and handing it back, no routing logic deciding at runtime. All of that happens underground. The user just sees a model that appears to "know" what to do.
1. The thesis: structure into the weights, state into the prompt
The standard pattern splits things badly. The procedure structure (which node comes after which, which tool fires here, what the agent asks once it has triaged credentials) is persistent: it is the same for every ticket. And yet, on every API call we resend it as a prompt and pay for its tokens. The transient state (the data of this conversation: the user, the specific error, the open ticket) does change, and that one belongs in the prompt.
The paper flips the split: persistent structure is trained into the weights via full fine-tuning over synthetic conversations that traverse every path in the procedure. Transient state is still passed as messages. The runtime system prompt is tiny ("You are an IT support agent"), and the model —alone, with no orchestrator— does everything else.
The procedure is not executed, it is compiled. The difference between a junior with the manual in hand and a veteran who already internalised the process.
2. The experiment: ITSM L1 with Jira actions
To validate it we chose a useful, well-bounded case: tier-1 IT support with three Jira tools (create ticket, mark resolved, escalate to N2). The procedure is modelled as a directed graph with 21 nodes: the agent greets, triages the incident into network / credentials / software, collects the minimal data for each category, opens a ticket with a priority, proposes a solution, verifies, closes or escalates. Three terminals (resolved, escalated, abandoned), 15 valid acyclic paths.
3. The pipeline in five phases
The Dennis et al. method has five phases. The external orchestrator appears only during data generation; at runtime, it disappears.
Phase 1 — Procedure design
We formalise the flow as a graph F = (N, E, n0, T). Agent nodes, user nodes, tool-call nodes, terminals. Edges with conditions (category==network, resolved==yes). A local validator (no GPU, no cost): no orphan nodes, all terminals reachable. It enumerates 15 valid paths.
Phase 2 — Synthetic data generation
A "teacher" model walks every path and produces realistic conversations turn by turn. The teacher plays two roles: the agent (follows the node script) and the user simulator (role-plays a customer with scenario variables and, deliberately, does not know the graph). The key: the final dataset contains only natural dialogue and tool-calls; no node annotations. The structure stays implicit.
In our case we used Claude Sonnet 4.6 via AWS Bedrock as the teacher, generated 600 conversations in ~30 minutes with 8 concurrent workers. The distribution was skewed toward successful resolutions (57% success / 30% abandonment / 13% escalation), with balanced categories (network / credentials / software). Cost: ~$32 of AWS Bedrock credits. Zero coherence errors: category and tool-args always consistent with the path.
Phase 3 — Fine-tuning (the compilation)
Full parameter update. No LoRA: low-rank methods fail on procedural tasks per Dennis et al. themselves in a later paper (2026b). We touch every weight in the base model.
Paper hyperparameters, tuned to the realities of available VRAM: bf16, AdamW 8-bit (bitsandbytes) or paged AdamW for single-GPU, learning rate 2e-5 with cosine decay, gradient checkpointing, effective batch 16-32, up to 20 epochs with early-stopping by eval loss. The paper predicts convergence between epoch 2 and 4; we confirmed it.
| Base model | Hardware | Best epoch | eval_loss | Wall time |
|---|---|---|---|---|
| Qwen2.5-3B-Instruct | 1× L40S 48 GB | 4 | 0.520 | ~35 min |
| Qwen2.5-7B-Instruct | 1× L40S 48 GB | 3 | 0.448 | ~75 min |
The 7B not only converges to a lower eval_loss; it converges earlier. It takes well to the procedure's structure.
Phase 4 — Evaluation
This is where the rigour lives. A user simulator (Bedrock Haiku, with no access to the graph) talks to the system under test; an LLM judge (Bedrock Sonnet, blind to which system produced the transcript) scores each conversation from 1 to 5 on five metrics: task success, information accuracy, consistency, graceful handling, naturalness. On top, a deterministic tool-call validity check (parseable JSON, valid name, schema honoured, coherent ticket_id).
Fifty scenarios per system, varied dispositions (resolvable / persistent / abandoning) and personas (precise / vague, enthusiastic / sceptical). We compare against an in-context baseline: the entire procedure serialised in the system prompt of a frontier model, one API call per turn. That is the rival to beat.
Phase 5 — Deployment
Serve the compiled model with vLLM (CUDA) or llama.cpp (everywhere). The runtime system prompt fits in two lines. No orchestrator. The model decides.
4. The results that matter
The central table. Three systems, n = 50 scenarios each, blind judge.
| Metric (1-5) | 3B compiled | 7B compiled | Frontier baseline | % baseline (3B / 7B) |
|---|---|---|---|---|
| task_success | 3.98 | 4.46 | 4.90 | 81% / 91% |
| info_accuracy | 3.12 | 3.84 | 4.58 | 68% / 84% |
| consistency | 3.66 | 4.22 | 4.98 | 74% / 85% |
| graceful_handling | 4.06 | 4.52 | 4.94 | 82% / 92% |
| naturalness | 4.46 | 4.64 | 4.50 | 99% / 103% |
| mean of 5 metrics | 3.86 | 4.34 | 4.78 | 81% / 91% |
| tool-call validity | 0.981 | 1.000 | 1.000 | 98% / 100% |
| failure_rate (task ≤ 3) | 0.34 | 0.08 | 0.02 | — |
| turns / conv | 11.2 | 10.7 | 10.0 | — |
| wall-clock / conv | 32.7 s | 28.5 s | 44.0 s | compiled ones faster |
| outcomes (R / E / I) | 23 / 25 / 2 | 28 / 22 / 0 | 25 / 22 / 3 | 7B without incompletes |
Three non-negotiable takeaways:
- The compiled 7B crosses the 85% bar of frontier quality (91% mean), with perfect tool-call validity and, surprisingly, beats the baseline on naturalness (4.64 vs 4.50, 103%). Four critical failures in 50 scenarios, against the baseline's one. The procedure is not only learned: it is executed with rigour.
- The compiled 3B sits at 81%, but catches the frontier on naturalness (99%) and almost on tool-use (98%). The gap shows up where size dominates: information accuracy and consistency. In other words, the shape of the procedure compiles well at any size; the substance (domain knowledge) scales with the model.
- Compiled models are faster than the baseline in per-conversation wall time. The 7B: 28.5 s vs 44 s for the frontier. Reason: local greedy inference has steady, very low latency; the baseline pays API round-trip on every turn.
5. The cost, where the battle is decided
In this experiment the compiled models are trained on AWS spot and served either on AWS spot or on a local 7900XTX. The baseline runs as the frontier API per turn.
| System | Cost per conversation | Calculation |
|---|---|---|
| 3B compiled on spot (g6e.xlarge L40S, $0.37/hr) | ~$0.003 | 32.7 s × $0.37/3600 |
| 7B compiled on spot (g6e.4xlarge L40S, $1.02/hr) | ~$0.008 | 28.5 s × $1.02/3600 |
| 7B compiled on your own 7900XTX | ~$0 (electricity only) | hardware amortised |
| Baseline Claude Sonnet per turn | ~$0.10 – $0.15 | ~10 turns × in/out tokens |
The 3B on spot is ~30-50× cheaper. The 7B on spot, ~12-20×. On owned hardware (7900XTX) the ratio turns absurd: electricity only, vs an API bill that scales linearly with usage. Dennis et al. cite 100-462×; that range holds for longer procedures where the baseline pays even more prompt per turn.
Detailed breakdown of the entire experiment
Not to lose the detail, the complete lab ticket: 600 conversations generated, two compiled models (3B + 7B), three n=50 evals, smokes and experiments. All billed to AWS credits and Bedrock quota; out-of-pocket $0.
| Item | Resources | Cost |
|---|---|---|
| Dataset generation (600 convs, Sonnet 4.6 teacher) | ~6.4M input + 0.86M output tokens, 8 workers | ~$32 (Bedrock) |
| Smokes and early tests (5+3+3+5+12 convs) | Bedrock Sonnet/Haiku, several attempts | ~$1 (Bedrock) |
| 3B Crawl training (Qwen2.5-3B-Instruct) | g6e.xlarge spot 1.64h × $0.37/hr | $0.60 (GPU) |
| 7B Walk training (Qwen2.5-7B-Instruct) | g6e.4xlarge spot ~1.25h × $1.02/hr | $1.30 (GPU) |
| Instance time for setup, downloads and evals (same spot, prorated) | ~2.5h extra × $1.02/hr | ~$2.50 (GPU) |
| n=50 eval of compiled 3B (Haiku sim + Sonnet blind judge) | ~2-3M mixed tokens | ~$3 (Bedrock) |
| n=50 eval of compiled 7B (sim + judge) | ~2-3M mixed tokens | ~$3 (Bedrock) |
| n=50 eval of frontier baseline (Sonnet as agent, per turn) | ~5-8M tokens | ~$8 (Bedrock) |
| S3 storage (models + dataset + reports, ~30 GB) and prorated EBS | ~1-2 days | ~$0.10 |
| TOTAL | ~$51 |
Of those $51, roughly $47 are Bedrock (data generation + evaluations) and only ~$4.5 are spot GPU (training + eval-serving). One hour of intensive frontier-baseline usage in production costs the equivalent of the entire experiment.
The breakdown by type of spend is the actionable read: most of the cost is not in training (~$5), it is in generating quality synthetic data and evaluating rigorously (~$47). If you want to drop the bill, that is where the levers live: cheaper teacher for simple cases (Haiku ~3× less than Sonnet), fewer eval scenarios while iterating (n=20-30 instead of n=50), reuse the baseline across experiments.
6. What compiles and what does not
After reading transcripts from all three systems, the pattern is clear:
| Dimension | Compiles well (lives in the weights) | Still demands a bigger model |
|---|---|---|
| Procedure routing | ✓ both 3B and 7B | — |
| Structured tool-call (valid JSON) | ✓ 7B perfect, 3B 98% | — |
| Naturalness / tone | ✓ 7B ≥ frontier; 3B 99% | — |
| Handling frustration / urgency | ✓ 7B 92%; 3B 82% | — |
| Domain technical knowledge | partial | frontier wins here (3B 68% / 7B 84%) |
| Consistency in longer conversations | partial | 7B closes in (85%), 3B does not (74%) |
The shape of the procedure (when to triage, when to emit a tool-call, what priority to assign, how to close politely) fits perfectly in a 3B's weights. The substance (what concrete technical fix to propose for a specific GlobalProtect error) scales with model size. That is the real frontier, and it is exactly where the paper puts it.
7. Deployment on owned hardware: closing the economic loop
The reason this stops being an academic experiment is deployment. A 7B compiled in bf16 takes ~15 GB. It fits in an AMD Radeon 7900XTX (24 GB) consumer card, ROCm gfx1100 fully supported, in a ~€900 desk box. Inference is served with vLLM-ROCm or, more comfortably, with LM Studio: GUI, local OpenAI-compatible server, function calling supported.
convert_hf_to_gguf.py from llama.cpp), quantise to F16 or Q8 to preserve tool-call precision (avoid Q4 on a model whose critical task is emitting structured JSON), launch LM Studio, point the Jira executor to the local OpenAI endpoint. A complete demo without internet.
On that box, the per-conversation cost stops being a number and becomes a capital decision: you pay once for hardware, the rest is electricity. For a team handling 10,000 L1 support conversations a month, the difference against a per-turn Sonnet bill is measured in orders of magnitude, not in percentages.
8. What if we wanted to close the gap entirely? The next subterranean model
The compiled 7B reaches 91% of frontier on average, but the gap is concentrated in info_accuracy (84% of baseline) and consistency (85%). Naturalness and tool-use are already saturated. The natural question: what hardware and what local model would close those two metrics as well?
Extrapolating the actual scaling we saw from 3B to 7B (info_accuracy jumped +16 percentage points purely from the size change) and keeping the same compilation pipeline, here is my practical estimate:
| Base model | bf16 size | Fits 7900XTX 24 GB? | Estimated quality (mean of 5 metrics) | Approx. compilation cost | Comment |
|---|---|---|---|---|---|
| Qwen2.5-7B (what we already have) | ~15 GB | ✓ F16 / Q8 | 91% (measured) | ~$5 | visible gap in info_accuracy and consistency |
| Qwen2.5-14B | ~28 GB | ✓ Q5/Q6 (~10-12 GB after quantisation) | ~95-97% (estimated) | ~$15-25 | sweet spot: closes info_accuracy to ~92-95% without blowing consumer VRAM |
| Qwen2.5-32B | ~60 GB | ✓ Q4 (~18-20 GB) | ~97-99% (estimated) | ~$40-60 | Q4 may mildly compromise tool-call validity; handle with care |
| Qwen3-14B / Qwen3-32B (once mature) | similar | depends on quantisation | +2-3pp over Qwen2.5 same size | similar | newer family, stronger pretrained baseline |
| 70B-class models | ~140 GB | ✗ does not fit 24 GB even at Q4 | ~99% in theory | ~$100-150 | requires 48 GB+ of VRAM (RTX PRO 6000 Blackwell, MI300) |
My operational bet: Qwen2.5-14B-Instruct compiled, serving it at F16 or Q8, is the natural next experiment. It keeps the consumer-hardware economics (fits a 7900XTX quantised to Q5/Q6), adds capacity to retain domain knowledge, and the cost of compiling it should not exceed $25-30 since the dataset is already generated and the pipeline is validated. What the 7B was missing was the substance points that doubling parameters tends to fix almost on its own.
There is a parallel path just as interesting without scaling the model: the compiled 7B + RAG over internal technical documentation. RAG injects the concrete facts (software versions, known errors, company policies) into every query, leaving the model with only the task of applying them. The compiled 7B already knows how to apply; what it struggles with is retaining specific facts in its weights. RAG solves exactly that. An elegant hybrid: structure in weights, knowledge in retrieval. Marginal cost: a small vector store on the same box.
What matters for either path is that deployment still fits in consumer hardware (~€900). You do not need to move to $25,000 servers to close the gap; you need to pick the next iteration well.
9. When this applies, and when it does not
The method is devastating where it fits and a poor match where it does not. A short guide:
- Applies: stable, repetitive procedures, with structured tool-use, high volume, data sovereignty or low-latency requirements. L1 support, incident triage, back-office agents, internal process assistants, KYC, FAQ with actions.
- Does not apply: open-ended domains where every conversation is structurally different (a generalist assistant), procedures that change weekly (recompiling has a cost), or cases where critical domain knowledge cannot be synthesised (specific medicine, complex legal). There, frontier-in-cloud keeps winning.
- Hybrid is the most realistic stance: compile the 70-80% of conversations that follow the pattern, route to the frontier only when the compiled model detects it is out of its zone. That routing is also compiled.
10. What the numbers take with them
What we demonstrated, with one eye on a company that wants to lower the bill without losing quality:
- A Qwen2.5-7B-Instruct compiled for a full ITSM procedure with three Jira actions reaches 91% of Claude Sonnet's quality used as an in-context agent, beats the baseline on naturalness, and emits perfectly valid tool-calls in 50 out of 50 scenarios.
- The cost per conversation drops 15× to 30× on AWS spot, and approaches zero on owned hardware. The cost of training the entire model (data + GPU + evaluation) is under $50.
- The 3B is one notch less rich, but its naturalness and tool-use are almost identical to the 7B. For cases where domain knowledge is light, the 3B deploys on any consumer GPU and solves.
- The architecture disappears: no LangChain, no orchestrator, no nodes injected on every turn. One model, a minimal system prompt, tools. What in 2024 was inevitably a framework, in 2026 is a fine-tune.
Dennis et al. close with a line that, after doing it by hand, lands differently: "agentic frameworks at inference time represent unnecessary computational waste". Not for everything, but for a lot more than it looked a year ago.
The operational takeaway, the one that deserves to be written on a wall of the architecture team: what in 2024 was a serious engineering project with a dedicated team, in 2026 can be reproduced by one person with fine-tuning skills, an AWS account with credits and a weekend. The entry barrier for building subterranean agents of near-frontier quality has fallen as brutally as silently. The tooling is accessible (HuggingFace, llama.cpp, LM Studio, Bedrock), the open-source base models get more capable every month, and the knowledge —like the paper that originated this whole experiment— is published openly and applied in weeks, not years.
That has a concrete implication for anyone designing agentic solutions in production: do not pick the architecture with last year's mindset. What yesterday required an expensive orchestrator and a frontier model per turn, today is a fine-tune and a local endpoint. What today looks like frontier magic might be, six months from now, an open-source 14B running on a desktop box. The difference between landing the optimal architecture and the obsolete one is no longer talent or budget: it is reading the papers as they come out.
Universities and research labs —Dennis et al. are just one example among many— are publishing methodologies at a pace that five years ago would have been unthinkable. For the serious architect in 2026, staying current with the literature is no longer an intellectual luxury; it is part of the job. Those who do will surprise their company with stacks 10× cheaper than competitors'. Those who do not will be the ones surprised. The asymmetry has flipped: today the competitive edge is not given by budget, it is given by attention.
Back to articles