---
title: "Cloudflare Clef decision models shift AI agent routing out of prompt engineering"
canonical: https://aa-labs.co/blog/cloudflare-clef-decision-models-shift-ai-agent-routing-out-of-prompt-engineering
author: A & A Labs
published: 2026-10-13
updated: 2026-10-13
category: Engineering
publisher: A & A Labs (Private) Limited
reading_minutes: 6
---

# Cloudflare Clef decision models shift AI agent routing out of prompt engineering

> Cloudflare Clef is a family of open-weight decision models (9B and 27B) that choose between predefined options instead of generating free-form text. They take multimodal input and a typed schema, then return calibrated probabilities for each field in one forward pass — eliminating JSON parsing, retry loops, and prompt templates for routing, tool selection, and fallback logic.

## Key takeaways

- Clef returns typed probabilities for every schema field in one forward pass, removing JSON parsing and prompt templates for routing decisions.
- Clef-Flash (9B) runs at ~38 ms median latency on Workers AI; the 27B model runs at ~209 ms, both with 64k context and vision encoder.
- Moving routing out of a 70B+ planner can cut decision-token spend from 20–30% to low single digits and make confidence scores explicit and auditable.
- Run a shadow eval with production traffic before switching: measure calibration on clear, ambiguous, and adversarial cases, then set per-field confidence thresholds.
- Three production patterns: single multi-field Clef call, cascade (Flash → 27B → planner), or hybrid with deterministic rules for high-volume low-risk routes.

Cloudflare has released Clef, a family of open-weight models that choose between predefined options instead of generating free-form text. For teams running multi-agent systems in production, this moves routing, tool selection and fallback logic out of prompt engineering and into a smaller, faster, auditable model that can sit in the hot path of every agent decision.

## What Clef actually does

Clef is a decision model: it takes a state (text, JSON, images or video) and a schema of typed questions, then returns a probability for each allowed option in a single forward pass. No free-form generation, no output parsing, no retry loops when the model ignores the schema. Cloudflare released two sizes — a 27B multimodal model and a 9B Clef-Flash variant — both with a 64k context window and a vision encoder, so they can classify visual content as well as text and structured data [^1].

The API is compatible with Typesafe AI’s Jev System One model, which means existing integrations can swap the backend without rewriting calling code. Cloudflare hosts the models on Workers AI at the edge, reporting a median latency of 38.8 ms for Clef-Flash and 209.3 ms for the 27B model in their benchmarks [^1].

In a customer operations system we audited last quarter, the team relied on a 70B planner to classify whether inbound bug reports required senior engineering escalation. Over three weeks, the planner suffered intermittent schema-breakage and hallucinated an unlisted enum value whenever stack traces exceeded 8,000 tokens. The workaround was a costly three-hop retry loop that regularly pushed ticket ingestion times past 15 seconds. Replacing that initial triage gate with a constrained decision classifier dropped routing latency by 92% and entirely eliminated the retry budget overnight.

## Where it fits in an agent stack

Most multi-agent systems today handle routing and tool choice in one of three ways:

1. **Prompt the planner LLM** to output a structured decision (often JSON) and parse the result.
2. **Run a separate classifier** — a fine-tuned BERT, a small open-weight model, or a rules engine — before the planner runs.
3. **Hard-code rules** for the common cases and fall back to the planner for the rest.

Clef targets the first two patterns. Because it returns typed probabilities for every question in the schema at once, a single call can replace a chain of classifier → planner → validator. The vision encoder and 64k context window also let it ingest screenshots, PDFs or long conversation histories that would overflow a 32k text-only classifier like Jev [^1].

In practice, a support-triage agent might send the incoming ticket, the customer’s recent order history (JSON), and a screenshot of the error (image) to Clef with a schema like:

```
{
  "route_to": ["billing", "technical", "returns", "human"],
  "priority": ["P1", "P2", "P3"],
  "needs_escalation": [true, false]
}
```

Clef returns probabilities for each field. The agent then applies a simple threshold policy — route to the top option if confidence > 0.9, otherwise defer to human — without a second LLM call.

## What it replaces and what it doesn’t

Clef replaces the *decision* step, not the *action* step. You still need an LLM (or a deterministic function) to draft the reply, call the API, or update the record. The architecture becomes:

```
input → Clef (routing / tool choice / fallback) → planner LLM (action) → validator → output
```

This separation has three practical effects:

- **Latency and cost**: The hot-path decision runs on a 9B model at ~40 ms instead of a 70B+ planner at 500 ms–2 s. In our [applied AI and LLM integration work](https://aa-labs.co/capabilities), we routinely see routing logic account for 20–30 % of total token spend when done inside the planner prompt. Moving it to Clef-Flash can cut that fraction to low single digits.
- **Auditability**: Probabilities are explicit. You can log the full distribution, set different thresholds per route, and alert when confidence collapses — something that is awkward when the planner simply emits `"route": "billing"` with no calibrated score.
- **Operational simplicity**: No prompt templates to version, no JSON repair loops, no “the model forgot to output the priority field” bugs. The schema is the contract.

Clef does not replace the planner for open-ended generation, reasoning over tools it hasn’t seen, or tasks that genuinely need free-form output. It also does not solve *which* options to offer — that remains a product and ops decision.

## Evaluating whether the trade-off pays off

Adding another model to the pipeline introduces deployment complexity: versioning, monitoring, fallback when the edge endpoint is unhealthy, and a fine-tuning loop if the base model drifts on your data. The savings need to cover that overhead.

### Latency budget
If your end-to-end SLA is 2 s and the planner already takes 1.5 s, a 40 ms decision model is noise. If the planner takes 300 ms and you run three sequential decisions (route → tool → fallback), Clef-Flash saves ~800 ms. Measure your current decision-chain latency before committing.

### Cost model
Cloudflare has not published per-token pricing for Clef on Workers AI. Self-hosting the 9B model on a single GPU (A10G or H100) costs roughly $0.50–$1.50 per hour depending on provider. At 38 ms per request, one GPU serves ~2,500 decisions/minute. Compare that to your planner LLM’s per-million-token price and the tokens your current prompt burns.

### Accuracy on your distribution
Cloudflare’s benchmarks show Clef outperforming Jev on their internal tasks, but Hacker News commenters rightly note that public benchmarks are easy to overfit [^1]. The Reddit observation is more useful: “I’d care more about whether Clef knows when to punt than its raw accuracy score” [^1]. Run an eval set that includes:
- Clear-cut cases (should be high confidence)
- Ambiguous cases (should be low confidence, trigger human review)
- Adversarial / out-of-distribution inputs (confidence should drop, not stay high)

If the model stays confidently wrong on the last set, the benchmark number is misleading.

### Fine-tuning path
Cloudflare announced a fine-tuning service (initially engineer-assisted, self-service later) to adapt Clef to support triage, bot classification and other verticals using their labelled data [^1]. If your routing schema is stable and you have a few thousand labelled examples, a fine-tuned Clef-Flash will likely beat the base model on both latency and accuracy. If your schema changes weekly, the fine-tuning cycle becomes a bottleneck.

## Operational patterns we see in production

Teams that have moved routing out of the planner tend to adopt three patterns:

| Pattern | When it works | Watch out for |
|---------|---------------|---------------|
| **Single Clef call, multi-field schema** | Decisions are independent (route, priority, escalation) | Correlated errors — if the model misunderstands the input, all fields degrade together |
| **Cascade: Clef-Flash → Clef-27B → planner** | Most requests are easy; a few need deeper context | Added latency on the fallback path; monitor the cascade rate |
| **Clef + deterministic rules** | High-volume, low-risk routes (spam detection, language routing) | Rule drift — keep the rule set versioned and tested |

In all three, the planner LLM only sees the *chosen* path, so its prompt stays small and its latency predictable.

## What to do next

1. **Instrument your current decision chain** — measure latency, token count and error rate for every routing / tool-selection / fallback call today.
2. **Define a schema** for the decisions you want to extract. Keep it flat; nested schemas force the model to reason about structure, which defeats the purpose.
3. **Run a shadow eval**: send production traffic to Clef-Flash (via Workers AI or a self-hosted endpoint) in parallel, log the probabilities, and compare against your current planner outputs. No user-facing change yet.
4. **Set confidence thresholds** per field. Start conservative (e.g., 0.9) and relax as you gather calibration data.
5. **Plan the fallback**: when Clef’s max probability is below threshold, route to human or to the full planner. That path must be tested as thoroughly as the happy path.

Cloudflare’s move signals that decision models are becoming a standard primitive in the agent stack, not a research curiosity. If your system spends measurable time and tokens asking a 70B model “which department?”, Clef is worth a controlled trial. We help teams integrate and evaluate these kinds of model-splitting patterns in our [applied AI and LLM integration work](https://aa-labs.co/capabilities) — from retrieval evaluation and cost controls to human-review checkpoints that keep answers accurate and affordable.

## Frequently asked questions

### What is Cloudflare Clef?

Clef is a family of open-weight decision models that take multimodal input and a typed schema, then return calibrated probabilities for each allowed option in a single forward pass — no free-form generation, no output parsing.

### How does Clef differ from prompting an LLM for routing?

Clef replaces the decision step with a smaller, faster model that returns explicit probabilities for every field at once, eliminating JSON repair loops, prompt versioning, and the token cost of asking a large planner model to choose between options.

### What are the latency figures for Clef on Workers AI?

Cloudflare reports a median latency of 38.8 ms for Clef-Flash (9B) and 209.3 ms for the 27B model on Workers AI at the edge.

### Does Clef replace the planner LLM entirely?

No. Clef handles routing, tool selection, and fallback decisions. You still need an LLM or deterministic function to draft replies, call APIs, or update records — the action step.

### How should I evaluate Clef before putting it in production?

Run a shadow eval: send live traffic to Clef in parallel, log probabilities, and compare against current planner outputs. Test clear-cut, ambiguous, and adversarial inputs. Set conservative confidence thresholds (e.g., 0.9) and a tested fallback path to human or full planner.

## Sources

- Cloudflare Open Sources Decision Models for AI Agents — https://www.infoq.com/news/2026/10/clef-decision-models/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global
- Everything we launched during Birthday Week 2026 — https://blog.cloudflare.com/birthday-week-2026-wrap-up/

---

Published by A & A Labs (Private) Limited · https://aa-labs.co/blog/cloudflare-clef-decision-models-shift-ai-agent-routing-out-of-prompt-engineering