10 Hardest Things to Teach a Robot

Anticipate Human Actions
If humans are going to spend a lot of time with robots, like this humanoid one dubbed ROBOY, the machines are going to have to get a little better at predicting what seemingly unpredictable humans will do next. © Erik Tham/Corbis

On "The Jetsons," Rosie the robot maid was able to hold conversations, cook meals, clean the house and cater to the needs and wants of George, Jane, Judy and Elroy. To understand Rosie's advanced development, consider this scene from the first episode of season one: Mr. Spacely, George's boss, comes to the Jetson house for dinner. After the meal, Mr. Spacely takes out a cigar and places it in his mouth, which prompts Rosie to rush over with a lighter. This simple action represents a complex human behavior -- the ability to anticipate what comes next based on what just happened.

Like deception, anticipating human action requires a robot to imagine a future state. It must be able to say, "If I observe a human doing x, then I can expect, based on previous experience, that she will likely follow it up with y." This has been a serious challenge in robotics, but humans are making progress. At Cornell University, a team has been working to develop an autonomous robot that can react based on how a companion interacts with objects in the environment. To accomplish this, the robot uses a pair of 3-D cameras to obtain an image of the surroundings. Next, an algorithm identifies the key objects in the room and isolates them from the background clutter. Then, using a wealth of information gathered from previous training sessions, the robot generates a set of likely anticipations based on the motion of the person and the objects she touches. The robot makes a best guess at what will happen next and acts accordingly.

The Cornell robots still guess wrong some of the time, but they're making steady progress, especially as camera technology improves.