AI agents can generate dozens or even hundreds of model requests while completing a single assignment. NVIDIA’s new NeMo Switchyard routing system is designed to keep those workflows from sending every task—regardless of complexity—to the most expensive model available.
As artificial intelligence moves beyond single-turn chat interfaces and into multi-step agent workflows, developers face a growing economic challenge: every planning decision, tool call, validation step, and correction can consume additional tokens and introduce more latency.
An agent completing a complex coding, research, or data-analysis assignment may interact with a language model dozens—or potentially hundreds—of times before producing a finished result. Routing every intermediate step through a top-tier frontier model can quickly become expensive, even when many of those requests involve relatively routine work.
To address that bottleneck, NVIDIA has released NeMo Switchyard, an open-source routing library and proxy server that dynamically distributes AI requests across a pool of different models.
Released alongside Nemotron 3.5 Lightning—a 30-billion-parameter Mixture-of-Experts model designed for fast task execution—Switchyard gives developers a way to introduce a “system of models” architecture with minimal changes to existing client integrations.
In practical terms, Switchyard acts like an AI traffic controller. Routine work can be sent to faster and less expensive models, while difficult reasoning or high-stakes decisions can be reserved for more capable frontier systems.
Why Multi-Step AI Agents Can Become Expensive
Traditional AI applications generally treated the large language model as a single, general-purpose endpoint. A user submitted a request, the model generated a response, and the interaction ended.
Agentic systems work differently. They may operate through repeated cycles of planning, tool selection, execution, output inspection, error correction, and final synthesis. More advanced systems may also delegate portions of a task to specialized sub-agents before combining their results.
Not every stage requires the same level of intelligence. Tasks such as extracting structured fields, preparing basic tool calls, classifying results, checking syntax, or summarizing a short tool response may not justify the cost or latency of a large frontier reasoning model.
According to NVIDIA’s technical overview, sending every turn through a single premium model can produce unnecessary operational costs and compounding latency. Switchyard inserts a routing layer between the agent and its available model endpoints, allowing each request to be directed according to its complexity, workflow stage, or configured policy.
In evaluations cited by NVIDIA, dynamic model routing reduced benchmark completion costs by as much as 74% compared with using top-tier frontier models for every request. NVIDIA also says Cognition recorded a 28% reduction in operational spending during preliminary testing of model routing in its Devin Desktop environment.
Those figures are vendor and partner results rather than independent guarantees. Actual savings will depend on the selected models, provider pricing, routing accuracy, task complexity, and how often a request must be escalated to a stronger model.
How NVIDIA NeMo Switchyard Works
NeMo Switchyard can operate as a standalone proxy server or as an embedded routing library. Its purpose is to separate the API format used by an agent from the model ultimately selected to process each request.
The system supports several commonly used interfaces, including OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. It can translate requests and responses between those formats while directing traffic toward backends such as vLLM, NVIDIA NIM, Ollama, or other OpenAI-compatible endpoints.
This means an existing coding agent or internal application can continue communicating through a familiar API while Switchyard selects a different provider or model behind the scenes. Developers must still configure models, credentials, routing policies, and fallback behavior, but they may not need to redesign the entire client application around every supported backend.
The standalone proxy is built around a high-performance Rust implementation. Developers who want to incorporate the routing logic directly into another Rust application can use the embedded switchyard-libsy library.
Four Available Routing Strategies
Switchyard includes four primary routing approaches:
- LLM-as-a-Classifier: A classifier examines the incoming request and determines whether it should be handled by a weaker, less expensive model or a stronger model.
- Signal-Driven Stage Routing: Requests are routed according to signals already present in the workflow, including tool results, errors, or the current processing stage. This can avoid making an additional model call for every routing decision.
- Weak-to-Strong Escalation: The request is first sent to the weaker model. A judge then evaluates the result and determines whether the same request should be escalated to the stronger tier.
- Random or A/B Routing: Traffic is divided across different models using a controlled split for benchmarking, cost experiments, and quality comparisons.
Developers can also create custom routing algorithms for applications with more specialized requirements.
The proxy-based design supports hybrid local-and-cloud configurations. Sensitive or high-volume requests could be sent to an on-premises workstation running a compatible open model, while selected reasoning steps could be directed to an external cloud API when organizational policy permits.
That flexibility could be particularly useful for businesses that want more control over data placement without giving up access to frontier cloud models entirely.
Nemotron 3.5 Lightning Targets the Agent Execution Layer
NVIDIA is pairing Switchyard with Nemotron 3.5 Lightning, a 30-billion-parameter open Mixture-of-Experts model with approximately 3 billion parameters activated for each token.
The model uses a hybrid architecture combining Mamba-2 state-space layers, Mixture-of-Experts blocks, and selective attention mechanisms. NVIDIA says the design supports a context window of up to 1 million tokens while reducing the amount of active computation required during inference.
Rather than positioning Lightning as the only model an agent should use, NVIDIA describes it as an execution-oriented model for handling frequent tasks such as tool use, structured output generation, code review, security-alert analysis, and routine workflow steps.
This positioning is important. The purpose of model routing is not necessarily to find one smaller model capable of replacing every frontier system. It is to identify which parts of a workflow can be completed reliably by a faster model and reserve expensive reasoning capacity for the steps that genuinely require it.
In its published comparisons, NVIDIA reports that Nemotron 3.5 Lightning can deliver up to four times higher token-generation throughput and reduce time to completion by approximately 30% compared with selected open models in its class. As with the cost-reduction figures, real-world performance will depend on hardware, precision, inference software, context length, and the workload being tested.
Why Local Model Routing Matters
Switchyard also provides NVIDIA with a natural way to promote locally hosted models on RTX workstations and DGX systems. If a company can route a large share of routine agent traffic to its own hardware, it may reduce recurring API costs while keeping more data inside its controlled environment.
That does not mean every organization should immediately replace cloud services with local models. Self-hosted AI introduces its own costs, including hardware, power, maintenance, security, model updates, and engineering support.
The more useful development is the ability to mix both approaches. A business might use a local model for repetitive processing, private document work, or high-volume tool calls while retaining a cloud model for difficult reasoning, specialized capabilities, or temporary bursts of demand.
For local AI enthusiasts, this also expands the role of desktop hardware. A powerful workstation does not necessarily need to run every part of an agent. It can serve as one specialized execution tier inside a larger model-routing system.
Early Limitations and Operational Trade-Offs
Dynamic model routing offers a promising path toward lower costs and faster agent workflows, but the architecture introduces new operational risks that developers will need to manage.
- Pre-alpha maturity: NVIDIA currently describes Switchyard as pre-alpha and experimental software. Its interfaces and algorithms may change substantially before version 1.0, and the project’s documentation warns that it is not yet intended for production use.
- Routing overhead: An LLM-based classifier adds another model request to the workflow. If the routing decision costs more time or tokens than the selected task saves, the economic benefit can shrink.
- Misclassification: A router may send a difficult task to an underpowered model. Escalating the request afterward can create more latency than selecting the stronger model from the beginning.
- Schema and tool reliability: Smaller models may be less consistent when following strict JSON schemas, selecting tools, or preserving complex instructions. Strong validation and fallback policies remain essential.
- Security boundaries: A routing proxy may handle requests containing proprietary data, authentication credentials, or tool-access context. Organizations must understand which information is sent to each provider and prevent routing decisions from bypassing data-governance policies.
- Evaluation complexity: Lower cost is only valuable if the routed system maintains acceptable quality. Teams will need application-specific evaluations rather than relying exclusively on general model benchmarks.
The security implications are especially important because routing middleware becomes another privileged component within the AI infrastructure stack. A misconfigured router could send private information to the wrong provider, choose an unapproved model, or weaken carefully designed tool-access controls.
InsightTechDaily Analysis
NeMo Switchyard highlights an important shift in AI infrastructure: the era of sending every request to one monolithic model is giving way to modular systems in which several models handle different parts of the same workflow. NVIDIA naturally benefits when more of those tasks run on hardware-optimized Nemotron models using RTX and DGX systems. However, Switchyard’s open-source, proxy-based design gives developers meaningful flexibility across local models, cloud providers, and custom endpoints. As agentic AI adoption grows, routing middleware could become a standard part of the enterprise AI stack alongside gateways, vector databases, observability systems, and security controls.
The System of Models Is Becoming the Real AI Product
NeMo Switchyard reflects a broader change in how AI applications are being designed. The most capable system may no longer be the application connected to the single largest model. It may instead be the application that can select the right combination of models, tools, memory systems, and validation layers for each stage of a task.
That approach could make advanced agents more affordable, but it also shifts part of the engineering challenge away from model selection and toward orchestration. Developers must determine when a small model is sufficient, when escalation is justified, how failures are detected, and which data each provider is permitted to receive.
Switchyard is still too early to treat as a finished enterprise routing platform. Its pre-alpha status, evolving interfaces, and lack of long-term production evidence make experimentation more appropriate than immediate critical deployment.
Still, the underlying idea is difficult to ignore. As AI agents become longer-running and more complex, sending every intermediate step through the most expensive available model will become increasingly hard to justify. Intelligent routing may be one of the technologies that allows agentic systems to move from impressive demonstrations into economically sustainable everyday tools.
