Robotics and process automation
AI in the physical world is harder than AI on a screen, because the world pushes back. Process automation, which only moves data, is far easier and far more common.
Words on this page (2)
- training
- : Adjusting a model's parameters on data so its predictions improve.
- sim-to-real
- : The gap between a robot trained in simulation and the same robot in the real world.
What it is
A robot works in a loop: sense (cameras, touch), decide, act (move motors), and repeat many times a second. Older robots followed fixed programs and needed a controlled setting, like a car factory line. Newer approaches learn from demonstrations or from trial and error in simulation, and some models take camera images and an instruction in words and output motor actions.
Process automation is software doing office work: reading an invoice, filling a form, routing a ticket. Language models made it flexible, since they can read messy inputs that rules could not handle.
Why physical AI is hard
Every step can fail, and errors add up. in simulation is cheap and safe, but a model trained there can fail on a real robot because friction, lighting and wear differ. This is the gap. Real-world data is slow and costly to collect, since a robot can only do so many attempts per hour.
A worked example: errors multiply
A robot completes each of 10 steps in a task with 95 percent success. All ten succeed with probability 0.95 to the power 10, about 0.60. A task that looks 95 percent reliable per step fails four times in ten. Reaching 99 percent per step lifts the whole task to about 0.90.
Where it is used
Warehouse picking, vacuum robots, surgical assistance under a surgeon's control, crop harvesting, and automated invoice handling in finance teams. Start with the cases where the environment is controlled and a person can step in.
Retries change the multiplication
The worked example assumes one failed step ruins the task. If the robot can notice a failed grasp and try once more, a 95 percent step succeeds with probability 1 - 0.05 × 0.05 = 0.9975. Over ten steps that gives 0.9975 to the power 10, about 0.975, instead of 0.60. This holds only if failures are independent and detectable. A robot that does not know it dropped the part cannot retry, so sensors that check each step are worth as much as better arms.
The loop has a delay
Invented numbers: a belt carries parts at 0.5 meters per second, and the camera-decide-move loop takes 0.3 seconds. The part moves 0.5 × 0.3 = 0.15 meters, 15 centimeters, before the arm acts. The arm must aim where the part will be, not where it was seen. Slower loops or noisy timing show up as grasps that miss by a steady distance in the direction of travel.
A second example: automation that fails quietly
A model reads invoices and fills the total. One supplier writes "1.250,00", with a dot for thousands and a comma for decimals. The model reads it as 1.25. The symptom is totals wrong by a factor of 1,000 for one supplier, with no error message. A cheap check catches this: add up the line items and compare with the total, and send any mismatch to a person. Where invoices come in one fixed template, a plain parsing rule is cheaper and exact.
Common mistake
Treating a smooth demo video as proof. Ask how many attempts it took and how often the robot failed.
A warehouse robot picks items from bins with 98 percent success on the test shelf. On the live floor, shiny packaging confuses the cameras and the success rate drops to 90 percent. Each failed pick needs a worker to step in. The company counts the cost of those interventions before deciding how many robots to buy.
Check yourself
What is the sim-to-real gap?
Not quite. Friction, light and wear in the real world are not captured in simulation.
A task has 10 steps, each 95 percent reliable. About how often does it fully succeed?
Not quite. 0.95 multiplied ten times is about 0.60, because errors compound.
Why is invoice automation easier than a warehouse robot?
Not quite. Software tasks are reversible and cheap to retry, unlike physical movement.
You got 0 of 3 right.
Go deeper
- RepoLeRobotRepo
Hugging Face · Open models, data and tools for real-world robotics.
- Article
Google DeepMind · A model that turns images and instructions into robot actions.
- Paper
Tobin et al. · A classic attack on the sim-to-real gap.
Spotted a mistake or something unclear? Report it on GitHub.