Meta's Prometheus supercluster in New Albany, Ohio, already runs hundreds of thousands of Nvidia GPUs spread across several buildings. The company now plans to grow it to about 1 million. Meta laid out the details in an engineering blog post on Monday.
The first phase of the gigawatt-scale cluster is finished, and Meta has used it to train its Muse family of models, including the Muse Wonder models. Two years ago, the site was a conventional data center handling web traffic.
"At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep," Meta wrote.
Prometheus runs on two Nvidia platforms today. Both are built around racks of 72 GPUs that work together as a single accelerator. The cluster's Catalina section uses the GB200 NVL72, while Clemente uses the newer GB300 NVL72. A third platform, San Miguel, will add Nvidia's Vera Rubin NVL72 racks.
The GPUs sit in traditional data centers, AI-optimized buildings and rapid-deployment structures, all linked by miles of fiber. Each building's network connects up to 20,736 GPUs. A separate aggregation network ties the buildings together over distances ranging from 2 kilometers to more than 100 kilometers. Meta also uses computing capacity leased from partners.
Keeping a million GPUs running
At this scale, a fault on a single machine can halt an entire training run. During the first phase in mid-2025, interruptions peaked at nearly 1,000 events per 8,000 GPUs each day, Meta said. Over the following year of engineering work, the company cut that figure by 99%.
The biggest early problem involved signal quality on more than 600,000 NVLink connections between GPUs. Memory faults became a bigger issue as Meta scaled up the GB300 racks.
Meta says the cluster kept more than 85% of its pre-training time productive during Muse Wonder's training, and more than 90% during reinforcement learning. On average, the system ran about 47,000 GPU hours between interruptions. After firmware fixes, that figure rose above 54,000. Meta's goal is about 100,000.
Reliability also requires extra hardware. Each 72-GPU rack runs a 64-GPU slice and keeps eight GPUs in reserve as spares. Meta also holds about 2% of its racks as backups. As a result, a training run that needs 32,768 GPUs actually requires 37,656, or about 13% more.
Managing power swings
When tens of thousands of GPUs compute and communicate in step, the cluster's power draw can swing by 30% to 40%. Starting or pausing a large job can shift tens of megawatts within seconds. To manage that, Meta caps GPU power gradually using software it calls Dynamo Dimmer. The system is backed by batteries at the rack, facility and grid levels.
Meta says it pays the full cost of the energy it uses in New Albany and matches that usage with new clean and renewable energy. In July, however, the company left the RE100 pact, a renewable energy commitment, as its gas buildout for AI outpaced its renewable projects.
Meta said lessons from Prometheus are shaping Hyperion, its next supercluster in Richland Parish, Louisiana. The company has also pledged $1 billion to data center communities.
Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.