Startup

A Really Big Computer

GPT-4 was trained on 25k A100s in about 90 days. That's 3e25 FLOPs. If a person has 20 PFLOPS

A Really Big Computer

GPT-4 was trained on 25k A100s in about 90 days. That's 3e25 FLOPs.

If a person has 20 PFLOPS (20e15) of compute, GPT-4 used 47.5 person-years to train. Very human scale.

I want to build a capable of training GPT-4 in a day. I need 3e25/86400 = 347,000,000 TFLOPS, or 2.25M GPUs. At 300W each, I'm gonna need 675 MW of power.

$2.25B CapEx for 2.25M $1000 GPUs.

At $0.05/kWh, 's $34k/hr to operate.

But seriously with continuing computer progress probably doesn't at this ~$5B scale. Why are we buying the GPUs? Why are we buying the power?

It costs $20,000 for a 3nm wafer from TSMC. Wafer scale compute probably makes . I get 89 600mm^2 dies, so let's say I can get 100 PFLOPS from a 3nm wafer. Cerebras doesn't tell you their FLOPS, but it's 62.5 PFLOPS on a 7nm wafer, which makes think I could get over 100 on 3nm. But 100 conservatively. Also, my wafers will be , not lame squares.

It's $7 for a TFLOP on AMD. It's $0.20 for a TFLOP on my wafers. We are in the much more reasonable land of $69M for my computer, buying 3450 wafers.

For power, I'm not getting scammed by the stupid grid. Solar panels are $0.20/watt now (thanks !). My 675 MW plant will cost $135M, and it will have a much longer shelf life (15 years) than the computer (3 years). I'll note that if we get 50% out of these panels, my plant pays for itself in a year.

Rough budget:

$50M in chip NRE
$69M from TSMC for the 3450 wafers
$31M to finish out the wafer computers
$135M solar panels
$50M to buy the land
$50M in construction costs

The tiny corp's next raise will be $400M at a $2B valuation. No . Still several years away though. We will of course sell the wafer computers too for the of a DGX box with 10x the compute. That's how we'll make .

The real question is where should we build this? Must have good sunlight, be cool enough we can use free , not have crazy expensive grid power, be friendly to building, and respect rights.

Welcome to the new Los Alamos. To the zettaflop and beyond!

About Author

SOFAIO

Leave a Reply

SOFAIO BLOG We would like to show you notifications for the latest news and updates.
Dismiss
Allow Notifications