Inspired by research into how infants learn, computer scientists have created a program that can learn simple physical rules about the behaviour of objects — and express surprise when they seem to violate those rules. The results were published on 11 July in Nature Human Behaviour1.

Developmental psychologists test how babies follow the motion of objects by tracking their gaze. When shown a video of, for example, a ball that suddenly disappears, the children express surprise, which researchers measure by how long they stare in a particular direction.

Luis Piloto, a computer scientist at Google-owned company DeepMind in London, and his collaborators wanted to develop a similar test for artificial intelligence (AI). The team trained a neural network — a type of software system that learns by spotting patterns in large amounts of data — with animated videos of simple objects such as cubes and balls.

The software model, named Physics Learning through Auto-encoding and Tracking Objects (PLATO), was fed the raw images from the videos, but also versions that highlighted each object in the scene. PLATO was designed to develop an internal representation of physical properties of the objects, such as their positions and velocities.

The system was trained on around 30 hours of videos showing simple mechanisms such as a ball rolling down a slope or two balls bouncing off each other, and developed the ability to predict how those objects would behave in different situations. In particular, it learnt patterns such as continuity, in which an object follows an uninterrupted trajectory rather than magically teleporting from one place to another; solidity, which prevents two objects from penetrating each other; and persistence of the objects’ shape. “At every step of a movie, it makes a prediction” about what will happen next, Piloto says. “As it gets further into the movie, the prediction becomes more accurate.”

Surprise!

When shown videos with ‘impossible’ events, such as an object suddenly disappearing, PLATO could measure the difference between the video and its own prediction, providing a measure of surprise.

Piloto says that PLATO is not designed as a model of infant behaviour, but that it could be a first step towards AI that can test hypotheses about how human babies learn. “We’re hoping this can eventually be used by cognitive scientists to seriously model the behaviour of infants.”

Comparing AI with how human infants learn is “an important research direction”, says Jeff Clune, a computer scientist at the University of British Columbia in Vancouver. “That said, the paper does hand-design much of the prior knowledge that gives these AI models their advantage.”

Clune and other researchers are working on approaches in which the program develops its own algorithms for understanding the physical world.