Turning Compute Into Data
Why LLM-powered simulated data could launch recursive self-improvement for robotics
“People talk a lot about RSI as AI that does AI research. There’s relatively very little talk about self-replicating data centers, this idea that you can use AI to control robots and run fabs and build turbines or solar panels and build data centers. But…I think we have that within grasp.”
– Sam Altman, Co-Founder and CEO, OpenAI [Cheeky Pints]
Robotics is coming. OpenAI’s GPT-6 Astra was released on September 3rd. It was the first frontier model with generalized spatial reasoning. Opus 5.5. Gemini 4 Argon. Fable 5.2. Gemini 4 Carbon. Robot demos flooded X. Embodiment generalization! Cross-domain intelligence! This fall, the world glimpsed its first generalized robotics intelligence, and the demos weren’t by specialized robotics models.
Roboticists may soothe investors: LLMs are not a straight-shot solution. For example, some claim that GPT-6 Astra will directly outcompete existing robotics models, but a Jetson AGX Thor caps out at 128 GB of RAM, which could reasonably run only a 40B-50B parameter model. Sure, and what about distillation? Distilling the intelligence of a five- or ten-trillion parameter frontier model into five or ten billion parameters could strip the very capabilities behind its generalization.1
Direct control is a threat, not a promise. But LLMs have tipped the playing field beyond the model layer. Just take a look at data.
Strap a GoPro to your forehead, grab an orange, and get filming. Haven’t you heard? Data is the bottleneck.
Real-world data generation is extraordinarily difficult. What tasks are diverse enough? Do Wuji gloves collect enough data? How is the lighting? Did your arms break? Have you tried TianJi arms? How is the data processed into the models? What’s the appropriate mix between egocentric and motion capture? This training run needs another thousand hours. Sorry, I meant a hundred thousand. Figure is harvesting over 100,000 hours of data per day.2 Fine. Let’s talk.
In theory, simulated data should be cheaper and faster to produce – uniquely scalable compared to real-world alternatives like UMI, egocentric, or teleop. In practice, its advantage is blunted. Building simulation environments in physics-based simulation software like Newton, MuJoCo, or Isaac requires nontrivial configuration and iteration time which could occupy an army of robotics PhDs. And after setup, there’s still a sim-to-real gap.
But the paradigm shifted with the latest batch of frontier LLMs.
In the latest model releases, lab employees, PhD students, and professors were surprised to find that GPT-6 Astra was quite adept at simulation engineering. Berkeley professor and Ambi Robotics co-founder Ken Goldberg stated, “Astra is very good at running [Newton]. We built a tool that makes it easy to interface with Newton, gave it to Astra with a video, and it can recreate the video down to astounding detail. Even a sponge: it will get the elasticity of the sponge, the material properties...and that closes the gap.” The sentiment is shared in industry. At Tutor Intelligence, which has robots deployed across the United States, CEO Josh Gruenstein recently claimed their team is taking advantage of Astra’s increased simulation-building capabilities.
Researchers and graduate students used to play human-in-the-loop. Now, you can sic Astra on simulation-building, adjust the simulation, and match to reality. This cycle is now significantly accelerated. One PhD student compressed months of work into under 24 hours by using GPT-6 Astra to autonomously build 25 simulated environments.
At first glance, the intuition appears obvious: simulation improves models by generating training data en masse. Even the reasoning traces are fair game. A model that spends lifetimes picking up simulated blocks (perhaps under dark lighting, bright lighting, and with blocks tilted every-which-way), will improve its block-plucking proficiency and ability to generalize. Realistic simulation challenges today’s assumption that robotics training data must be largely real-world-generated.
Beyond expanding data type and volume, simulations provide training examples to teach LLMs how to build better simulations. This is a recursive loop. At each cycle, LLMs develop ever-more realistic environments, those simulations serve as novel training examples, the model loss drops, and the model learns to engineer even better simulations, faster.
Congratulations, your GPU-hours just became data.
Today, the sim-to-real gap still remains. Manipulation in particular is difficult, particularly for physics-dependent tasks or tasks where texture is unclear, such as liquids of varying viscosity.3 RoboSimStudio creator and Berkeley Robotics PhD student Kaiyuan Chen added, “Cloth is another [example]. People claim they can simulate cloth, but…they can simulate one t-shirt.” The models still struggle to configure parameters outside of visual information – for instance, friction, center of mass, and mass. Visual realism does not always translate to physical realism.4