AI is moving fast. Change seems to be the only constant. As AI models, architectures and applications continue to race forward, the infrastructure that supports them also needs to evolve to keep pace.

That’s critically important when it comes to scale-up infrastructure. The scale-up architectures available today—single-rack, copper-connected, closed systems—are reaching their limits. Meeting the growing demands of AI isn’t possible without a new approach and architecture—one that goes beyond a single rack to an order of magnitude greater scale based on a single-tier, standards-based Ethernet fabric.

The Changing Demands of AI

Model sizes continue to grow and model architectures continue to evolve as frontier labs compete to deliver the highest-performing, most intelligent models.

Many models, including open-weight and closed models, already surpass one trillion total parameters. What’s more, dense transformer architectures have largely given way to sparse Mixture of Experts (MoE) architectures, which can feature hundreds of experts or more, with each token sent to a small subset of expert sub-networks. For example, Kimi K3, released by Moonshot AI in July 2026, is an MoE model with 2.8 trillion parameters, 896 routed experts, 16 active experts per token and a context window of 1 million tokens.

Larger MoE models drive more network traffic, putting additional strain on the network. MoE models create intense all-to-all traffic with less predictable traffic patterns as the number of parameters and number of active experts grow. During training, the volume of data movement for operations such as all-reduce and all-gather scales directly with the total number of parameters. A larger number of parameters also drives up the number of accelerators required for both training and inference, extending data movement beyond current scale-up domain limits to the higher-latency scale-out domain.

At the same time, the industry’s increasing drive toward heterogeneous compute also challenges current infrastructure. While GPUs remain the predominant accelerator, specialized accelerators are emerging that, combined with disaggregation of processing steps, address different phases of processing with more efficient approaches. For example, disaggregated inference, in which GPUs handle compute-intensive prefill while specialized accelerators handle memory-intensive parts of decode, is becoming more common. However, this disaggregation requires moving data, including KV caches and activations, between specialized accelerators, further taxing current scale-up architectures.

Why Larger Scale-Up Domains Matter

Volume GPU deployments today typically offer scale-up world sizes of 72 GPUs, dominated by NVIDIA’s NVL72 (single-wide rack) or AMD’s emerging Helios system (double-wide rack). Both systems feature architectures with a single tier of scale-up switches to connect all 72 GPUs with high bandwidth in a single network hop.

However, modern AI models and workloads would benefit significantly from scale-up domains much larger than today’s single-rack configurations. For one, a larger scale-up domain would accommodate larger models, supporting more parameters, more experts and wider expert parallelism (EP) over a contiguous, high-bandwidth scale-up domain with a single tier of switching—avoiding the need to cross over to use slower scale-out links. That is especially important for latency-sensitive processing phases such as decode.

Large, high-bandwidth scale-up domains increase the productivity of valuable computing resources. By reducing the amount of time xPUs sit idle waiting for data, they meaningfully increase xPU utilization. In training scenarios, a large scale-up domain eliminates reliance on constrained scale-out networks for intensive collective communication traffic such as all-to-all. In inference, larger domains enable optimal workload placement by allowing multiple models to run in a single scale-up domain, resulting in fewer stranded or misallocated computing resources.

Larger scale-up network domains also enable more efficient use of expensive and scarce memory resources. Memory supply and cost are frequently cited among the most critical bottlenecks in AI infrastructure. The right scale-up network can enable increased pooling of memory resources, yielding substantial cost savings.

These utilization benefits were demonstrated in a simulation paper presented by Turba Labs and Lawrence Berkeley National Lab at Hot Interconnects 2026. In the session “Breaking Through the Network Wall: Petabit-Class Switching for Single-Hop MoE Training at Scale,” they discussed their simulation of a future 20T-parameter MoE model with 2,304 experts running inside a single 2,304-xPU scale-up domain with 28.8 Tbps per xPU. For comparison, they simulated the same workload running inside a rack-constrained NVL144 with 28.8 Tbps of scale-up bandwidth per xPU and 6.4 Tbps of scale-out bandwidth between racks. In their comparison, the larger scale-up domain demonstrated a 38% improvement in xPU utilization due to reduced collective communication time.

Disaggregating the Integrated Rack

Today’s highly integrated rack solutions make it challenging to deliver the large, high-bandwidth scale-up domains needed to meet the evolving requirements of AI. Space, power and interconnect constraints limit the ability to scale further within a single rack.

However, architectures for larger scale-up domains that have been proposed to date rely on multi-tier switching fabrics that introduce more hops, resulting in lower effective bandwidth and higher latency for connections across racks. Not only that, but the inflexibility of highly integrated, vendor-proprietary systems limits the ability to adopt heterogeneous resources, interoperability standards and emerging technologies. The ultimate result is added complexity and constraints that reduce AI processing performance while driving up costs and increasing proprietary lock-in.

To achieve larger scale-up domains, the industry needs to break apart the integrated rack with an open, disaggregated, multi-rack architecture for scale-up. A disaggregated architecture would separate compute and networking into discrete racks, connecting pools of xPUs together across multiple racks with interconnect technologies such as Optical Compute Interconnect (OCI). Disaggregation fosters an open ecosystem that can more easily accommodate heterogeneous compute, provide multiple interconnect options and enable greater innovation.

In light of this, Eridu has worked with Delos Data and the Open Compute Project (OCP) to establish a new workstream: “Open Multi-rack Scale-up Domains.” The participants in this workstream will work together on scale-up reference architectures that can support hundreds or even thousands of xPUs in a single domain while maintaining high-bandwidth, single-hop connectivity across disaggregated and heterogeneous compute racks using open standards and specifications.

This new OCP workstream aligns with four key principles:

• Multi-rack: scale-up domain sizes are expanding to hundreds or even thousands of xPUs, and the network must maintain high-bandwidth, single-hop connectivity across racks.
• Disaggregated: mix-and-match compute racks must be accompanied by separate networking racks to create an open ecosystem.
• Heterogeneous: a mix of GPUs, CPUs and other diverse xPUs is being deployed for training, inference and agentic workloads. The network fabric must be capable of meeting their full range of performance requirements.
• Open: the network must leverage open, standards-based building blocks, including Ethernet for Scale-Up Networking (ESUN), open racks (e.g., ORv3), and optical MSAs (e.g., OCI). An open network eliminates proprietary barriers that can constrain agility and restrict customer choice of suppliers.

Promode Nedungadi, CTO at Eridu, and Dan Daly, CTO at Delos Data, spoke on this topic at the 2026 OCP Global Summit. They discussed how advances in AI models and AI compute have exposed the limitations of today’s AI scale-up architectures, and how the combination of open, disaggregated architectures with high-radix, single-tier switching can power large, open, multi-rack, high-bandwidth scale-up domains.

Watch their presentation, “Breaking up the Monolith – An Open Architecture for Multi-rack Scale-up Domain of 5000+ GPUs,” to hear more about what it takes to build scale-up domains that span thousands of xPUs.

The Future of Scale-Up

Today’s rack-constrained scale-up architectures can’t keep pace with the rapid evolution of AI models and compute. Fortunately, a solution is at hand. Open, disaggregated architectures plus high-radix, single-tier switching can power large, open, multi-rack, high-bandwidth scale-up domains, giving larger models and heterogeneous compute the headroom and runway necessary to train and serve the world’s most intelligent models.