Everything that has been digitised has been learned. Text, code, images, speech: anything that fits on a screen now has a model that gets better at it every few months.

But the digital world is a thin film over the physical one, and by our estimate it captures less than 0.001% of it. Under the film are the atoms. Every object that has to be lifted, turned, opened, stacked or carried is still moved by a hand.

If you can digitize the world accurately, you unlock realities humans can't yet imagine. It starts with a world where you don't have to lift a finger to move anything, and it ends with every atom around you being manipulated at will.

Every leap forward has come from learning beyond the limits of reality

Hey, can you tell which ones are real?

Here are ten frames from the same robot stations. Some come from the real robot, and some from our simulation of it. Mark each one sim or real, then check your score. Every round draws a new set of ten from a pool of 98.

Learning only inside the worlds

The robot then learned from trajectories it generated in those simulated worlds. For closing the dishwasher, we collected no real-world data at all: no demonstrations, no teleoperation, no fine-tuning on the customer's site.

Then we ran the model on the real robot in the real kitchen. It closed the dishwasher. The skill it had learned in our virtual kitchens transferred directly to the physical one.

The robot learned to drop the plate

The dishwasher was not the only thing the model figured out. In simulation, a plate got stuck at the edge of the rack, in a pose the gripper couldn't grasp well. Instead of retrying the same grasp, the model pushed the plate down, let it slide, and picked it up again from a better position.

Dropping something on purpose so you can pick it up better is the kind of move people make without thinking, and the kind nobody writes into a dataset.

This discovery is not alone

There were more than 300 techniques that we traced back to sim explorations. This behaviour was not present in any data.

This loop is super useful. The virtual world proposes endless possibilities on model finding techniques to navigate the real world. Each discovery becomes something the model can build on, which is what we mean by recursive learning.

Technical breakdown

We will be releasing the complete pipeline, with the technical details of each block, in the coming days.

Speed is the final frontier that needs to be unlocked before we can see robots assisting humans in the real world.

Speeding up robots is a chicken-and-egg problem: you need deployments in the real world to collect data and iterate on models, but you need fast models in the first place to provide enough economic value to earn those deployments.

Several companies like Physical Intelligence^1,2, Skild AI, Generalist AI, Reward AI have shown impressive demos of their models working at impressive speeds, but the fundamental lever behind that is very fast demonstration data, which does not hold once you start scaling a fleet into the real world. We have spent the past few months deploying across various industries, and here is what we, along with several other companies actually deploying robots on the ground, have seen.

Why is deployment data slow?

Operators in live deployments run two to four times slower than operators in demo setups, for a few key reasons:

1. Assistance has to be remote to scale. Sending an operator to every site breaks before you reach even a hundred deployments. Remote operation lets one office of trained operators serve many sites without downtime, which makes it the only viable path to human-assisted deployment at scale. Remote teleoperation is slower because it is hard to perceive depth intuitively from camera views; building a 3D environment inside a Meta Quest is a good technical solution, but nobody can use one for more than an hour without feeling nauseated.
2. The control interface differs. Many demo datasets use leader follower arms^3,4, which tie operator speed to physical arm speed. Deployments need dynamic speed adjustment like going slower near humans and fragile objects, for safety, etc. Most deployments we have seen use Meta Quest controllers, which track the operator's hand and move the arm accordingly.
3. There is no predictability. Demo scenes are structured and learnable, so operators can act before they think. In a real deployment, an operator has to react to live cues, which is inherently slower.

Why is slow data a problem?

Slow teleoperation hurts deployment and scaling in two ways:

1. Weak learning signal. Current robotics policies work by predicting the next chunk of actions. Data in which most of the motion is idle, or does not contain significant movement, carries poor information for a policy to learn from. We have seen this directly in our training runs: policies trained on such data are not only slower but usually less accurate, because each next-chunk prediction offers little progress toward completing the task.
2. Economics. When a policy or a teleoperator runs significantly slower than a human, it becomes impossible to create economic value for the end customer especially in environments such as kitchens and warehouses, which have been optimised around humans doing tasks quickly and efficiently.

Over the past six months we have run policies in live deployments and focused our engineering and research effort on making policies run faster than the teleoperation data they were trained on, so that we can deploy our robots at scale. We have found three levers that make policies faster while maintaining accuracy:

1. Adaptive speed-up of the training data

The simplest approach you could think of is to speed up the training data and then train a model on it. Unfortunately, that works poorly in practice: you lose a lot of key information when you discard frames from the data you collect. Uniformly dropping every other frame (2x) took success from 94% to 53%, and at 4x uniform it collapsed to 9% for plate picking, handover between arms and release. The failures concentrated at the grasp and the release, exactly where the discarded frames carried the most information.

A better and more effective approach, also observed by many others in the field, is to treat the motion of the arms as two separate parts: free-space motion, where the arms are travelling toward their target region, and contact-rich motion, where the arms are interacting with the real world. Free-space motion contains far less information about manipulation than contact-rich motion, so it makes sense to speed up free-space motion while keeping contact-rich motion intact.

Prior works such as AutoSpeed⁵, SAIL⁶ and SRIL⁷ have consistently used an RL head to predict contact-rich motion and perform speed-ups around it. We were able to achieve similar or better results without training an RL head at all, by simply treating any motion within x seconds of a gripper event (gripper opening or closing) as contact-rich motion; trying everything from 0.1 to 2 seconds.

We ran hundreds of ablations at 2x, 3x, 4x, 5x and 10x, and across different contact-identification techniques, and saw the best results at 4x with actions within 0.5s of gripper events being classified as contact. Beyond 4x the arm still reaches the object, but it arrives with too much residual velocity to close the gripper cleanly, and success falls off sharply.

2. Selective chunk skipping at inference

A VLA policy predicts several chunks of future actions at once. Skipping chunks makes the policy run faster by letting the arm jump to future states, with no retraining. Doing this naively causes two problems, and both of them push the policy out of distribution quickly:

1. Far-out predictions are less accurate. The further out a chunk is, the less likely it is to be accurate, because the world will have changed by the time the first few actions have been executed. The general practice is to query the model often and execute only near-term chunks.
2. Skipping causes jitter. When you skip chunks, the arm moves through a different distribution than the training data, and the video feed from the real world no longer corresponds to the training data, which takes the model out of distribution very quickly.

To solve these issues, our approach after running thousands of tests in the real world is to estimate the risk before skipping a step. We look at the motion predicted by the policy and skip only if both of the following conditions hold:

1. The predicted chunks sit close together in the real world. We measure the end-effector displacement between the last pose of the current chunk and the last pose of the chunk we would skip to, and require it to stay under x cm at 20 chunk executions per second. That threshold is not fixed: a VLM scores the current camera view for scene complexity i.e. clutter, number of movable objects, proximity of a human into 5 buckets, and we scale the allowed displacement from 2 cm in a clean, static scene down to 0.3 cm in a cluttered or human-occupied one. The intuition is that displacement is only safe if the world will still look the way the policy expects when the arm arrives.
2. There must be no gripper event inside the skipping window.

The second condition is essential. Every time we skipped chunks through a gripper event, the task fell apart. Restricting skips to slow, contact-free motion keeps the policy in distribution. With these techniques we achieve another 22 percent speed-up over our fastest policy.

3. DAgger-style intervention data

Intervention data is the most useful and most unexpected lever for speeding up a model. We expected it to increase accuracy which it does, at roughly 30x the per-frame value of ordinary teleoperation data. But with no other changes, the policy also gained a 10–15 percent speed-up simply from adding intervention data, sped up using the same techniques described above.

The setup follows DAgger (Ross et al., 2011)⁸. The policy runs on the robot, and an operator takes control the moment it starts to fail. We then speed up that intervention data with the same gripper-event rule from Lever 1 and retrain.

For models trained at lower speeds, intervention data barely helps with speed, so there is effectively zero speed-up. For policies trained at a faster speed, sped-up intervention data helps them grow faster and faster.

Final results

Combined, the three levers let us run our policies nearly four times (3.97x) faster than the data we collected from remote teleoperation. In the table below we report speed alongside accuracy as we add each lever:

Setup. All results are on a bimanual cell of two AgileX PiPER six-DoF arms with parallel-jaw grippers, with three RealSense cameras (one overhead, one per wrist) recorded at 20 fps, driven over CAN from a Jetson Orin NX 16 GB. The policy is π_0.5², with an action chunk of 30. Training data is 10 hours of remote teleoperation collected in the same deployment across seven days plus around one hour of usable teleoperation intervention data. Each number in the tables is 500 real-robot runs, with object positions re-randomised between runs.

Limitations

These results come from a narrow set of tasks: picking up plates and dishes, and picking and placing objects. We have yet to test whether the gripper-event rule and the 4x optimum hold across a more diverse task set. We also have no explanation yet for why 4x outperforms both lower and higher factors.

What comes next?

We expect our policies to soon run significantly faster than any human can control them, on specific tasks. To build superhuman policies, human data alone cannot get us there.

Human intervention data plays a role in robotics similar to the one RLHF⁹ played in the early days of LLM post-training. The field had to move past human feedback as models improved, because the models began to outperform expert annotators by too wide a margin. Robotics will hit the same wall in six to eight months.

Our next line of research is RL post-training with no humans in the loop. We plan to apply techniques in the GRPO¹⁰ and DAPO¹¹ family to robot policies, with the goal of pushing speed beyond human capability while holding accuracy at similar levels.

1. More worlds, different physics

A ten-cube tower stands on its own. Run it in thousands of worlds at once, and most of the copies fall.

Why? Each cube rests on four corner contacts. On a CPU, the forces at those corners are added up in the same order every step, so they stay perfectly even and the tower never starts to rock. On a GPU, thousands of threads add them in whatever order they finish, so the corners come out a hair uneven, the tower starts to rock, and the rocking grows. The more worlds share the GPU, the more that order gets shuffled, and the more towers fall.

Worlds also leak into each other. In MuJoCo Warp, every world wrote its bounding box to one shared slot, and a ball fell through its own table (#1795; fix #1800 by another contributor).

In Newton’s MPM solver, changing the other worlds moved world 0 by up to 0.1 mm (#5207).

What it does to training: the more worlds you run, the further your physics drifts from the one world you debugged.

In our own runs on a tower-stacking task, training was smooth when the towers stood and erratic when they fell at scale:

2. Resting blocks get launched

A small block sits still on a board. Nothing touches it. Then, in a single step, the engine shoots it straight down through the board at several metres per second.

Every step, the engine asks two questions about each pair of objects: are they touching, and which way should they push apart? For this block, MuJoCo Warp’s general routine gets it upside down. It says the block is 27.7 mm inside the board, not 0.009 mm, and should be pushed down, not up. So it pushes it down, hard. MuJoCo, and MuJoCo Warp’s own box routine, get it right.

What it does to training: it only goes wrong at a few exact heights, so a test that checks a handful of poses never hits them. A training run that tries billions of poses hits them again and again, and the policy learns that blocks sometimes explode. We reported it as #1883; two open fixes from other contributors, #1845 and #1866, solve it together.

3. The tasks that need scale most get the worst physics

Cables and cloth are why you need millions of rollouts: they have more ways to move than any rigid object. They are also where the engines are furthest from reality.

Stiff cables sag 40 % too far (Newton, VBD solver¹). VBD corrects one body at a time, so a correction crawls one segment per sweep. Our #4795 solves each rod at once and lands within 1–2 % of the exact shape.

A cloth pile sinks through the floor (Newton, VBD with self-contact). Nothing pushes it. It just sinks, until 271 of its 1,681 particles are below the ground, the lowest 74 mm down (#5143).

What it does to training: a policy that learns to route a cable or fold a towel, at scale, learns a material that doesn’t exist.

If you train at scale, check this

1. Run one scene at 1 and at 4,096 worlds. The physics must match.
2. Scan awkward poses: thin things resting on each other, shallow overlaps, edges across edges.
3. Check momentum: push a free object with a known impulse. Its momentum must change by exactly that.
4. Size contact buffers yourself and check them for overflow every step.
5. Test soft things at your real mesh size.
6. In MuJoCo Warp, route box pairs to the box-box routine (turn NATIVECCD off) until #1845 and #1866 land.

What’s next

Today’s engines are tuned for scenes where a millimetre doesn’t matter, you don’t need to rerun a world million times, and speed is measured on falling boxes. That is the right tuning for most people but for training robots at scale. So we are building Alloy, a physics engine for robots at scale, with formally verifiable physics. It is out now.

Footnotes

Every report links a script that reproduces it.

Robots need worlds that behave like ours

Our bet is that robots will learn most of what they know inside virtual worlds. It starts with digitizing the physical world: building virtual worlds good enough to pass a Turing test, so robots can learn there instead of through slow, expensive real-world trials.

That only works if whatever happens inside a virtual world would also happen outside it. When it doesn’t, nothing warns you. The difference hides in training and shows up on the real robot as the sim-to-real gap.

And robots don’t only touch rigid boxes. They handle towels, cables, water and sand, and each of those has to behave like the real thing. Robots also learn by trying, millions of times, so all of it has to run in thousands of worlds at once and stay right in every one of them. In our last post we showed how today’s engines quietly break at exactly that scale. Alloy is what we built instead.

What it does

• Many materials, one scene. Robots, cloth, cables, liquids, sand and soft bodies, stepped together.
• GPU or CPU. The same script, renderer included, runs on an NVIDIA GPU or the CPU.
• A camera for every world. Colour, depth and segmentation of every world at once.
• Robots ready to go. Arms, a hand, a quadruped and two humanoids, with RL and demo recording.
• Settings choose themselves. Solvers and steps are picked; sim.routes() says why.
• Health in every world. Tells a physics failure from a policy failure.
• Built for agents. Give your coding agent one prompt. It reads the docs, builds your world and trains your robot, in minutes.

How it held up

We wanted to know how Alloy does in the hands of someone who has never seen it. So we had it audited independently. Coding agents installed Alloy from PyPI like strangers would, trained with their own code on a single NVIDIA L4 GPU, and were scored on seeds held out from training.

• Walking. A Go2 robot dog learned to walk to 97 % success in about 5 minutes of training, and kept its feet under a 150 N push 98 % of the time.
• Learning from pixels. Reaching and pushing, learned from 64-pixel camera images only, got to 92 % in 6 and 18 minutes.
• Imitation learning. 100 % on pick-and-place 25 minutes after pip install, and 88 % from camera images alone.
• Cables. A cable-lifting task reached 96 % after 15 minutes.
• Liquids. Pouring reached 92 % on the built-in pour task, and 100 % on a cup-pouring task the agent wrote itself.
• New tasks. Agents built brand-new training tasks from the docs in 1 to 8 minutes each.

Keep in mind

Alloy is new, and some things are not there yet:

• On simple single-robot tasks, Alloy is not yet the fastest engine.
• Reinforcement learning for arm grasping is early.
• Cloth and sand train slowly for now.

We are fixing these fast. Alloy will be the default way to simulate the world, so start using it now. The Keep in mind page of the docs has the details.

Start with one line

Alloy is open source under Apache-2.0: the code is on GitHub. Install it with pip install alloy-engine, then read the docs.

Or let your agent do it: copy this, paste it into your coding agent and name your task.

Build and train <your task> with Alloy

Clips, docs and a guide to using Alloy with your coding agent are at alloy.argonrobotics.ai.