University of Washington's robotic system can feed people who need assistance eating



Researchers at the University of Washington are working on a robotic system that can feed people who need someone to help them eat.

The robot identifies different foods on a plate, and then strategizes how to use a fork to pick up and deliver the desired bite to a person’s mouth.

“Being dependent on a caregiver to feed every bite every day takes away a person’s sense of independence,” says Siddhartha Srinivasa, the Boeing Endowed Professor in the UW’s Paul G. Allen School of Computer Science & Engineering. Srinivasa is a corresponding author in a series of papers that the researchers published their results in.

“Our goal with this project is to give people a bit more control over their lives.”

According to the UW team, the idea was to develop an autonomous feeding system that would be attached to people’s wheelchairs and feed people whatever they wanted to eat.

“When we started the project we realized: There are so many ways that people can eat a piece of food depending on its size, shape or consistency. How do we start?” explains co-author Tapomayukh Bhattacharjee, a postdoctoral research associate in the Allen School.

“So we set up an experiment to see how humans eat common foods like grapes and carrots.”

Researchers arranged plates with approximately a dozen different kinds of food. The foods ranged in consistency, from hard carrots to soft bananas, and also included foods with tough skin and soft insides such as tomatoes and grapes.

Researchers then gave volunteers a fork and asked them to pick up different pieces of food and feed them to a mannequin. The fork was equipped with a sensor to measure how much force people used when they picked up food.

Volunteers used different strategies to pick up food with different consistencies, so people skewered soft items like bananas at an angle to keep them from slipping off the fork. Alternatively, volunteers tended to use wiggling motions to increase the force and spear each bite for items such as carrots and grapes.

“People seemed to use different strategies not just based on the size and shape of the food but also how hard or soft it is. But do we actually need to do that?” Bhattacharjee notes.

“We decided to do an experiment with the robot where we had it skewer food until the fork reached a certain depth inside, regardless of the type of food.”

According to the researchers, the robot used the same force-and-skewering strategy to try to pick up all the pieces of food, regardless of their consistency. This strategy worked on hard food, but the robot struggled with soft foods, as well as those with tough skins and soft insides.

With this in mind, the researchers determined that robots, much like humans, need to adjust how much force and angle they use to pick up different kinds of food.

Researchers also noted that the acts of picking up a piece of food and feeding it to someone are not independent of each other, as a lot of the times, volunteers would specifically orient a piece of food on the fork so that it could be eaten easily.

“You can pick up a carrot stick by skewering it in the center of the stick, but it will be difficult for a person to eat,” Bhattacharjee says. “On the other hand, if you pick it up on one of the ends and then tilt the carrot toward someone’s mouth, it’s easier to take a bite.”

To design a skewering and feeding strategy that changes based on the food item, the researchers combined two different algorithms. First, they used RetinaNet, which is an object-detection algorithm that scans the plate, identifies the types of food on it and places a frame around each item.

Then, they developed an algorithm called SPNet, which examines the type of food in a specific frame and tells the robot the best way to pick up the food. An example of this is SPNet telling the robot to skewer a strawberry or a slice of banana in the middle, and spear carrots at one of the two ends.

Researchers had the robot pick up pieces of food and feed them to volunteers using SPNet or a more uniform strategy: an approach that skewered the center of each food item regardless of what it was.

According to the researchers, SPNet’s different strategies outperformed or performed the same as the uniform approach for all the food.

“Many engineering challenges are not picky about their solutions, but this research is very intimately connected with people,” Srinivasa says.

“If we don’t take into account how easy it is for a person to take a bite, then people might not be able to use our system. There’s a universe of types of food out there, so our biggest challenge is to develop strategies that can deal with all of them.”

​Right now, the UW team is working with the Taskar Center for Accessible Technology to get feedback from caregivers and patients in assisted living facilities on how to improve the system to match people’s needs. Srinivasa notes that UW’s robot is not meant to replace caregivers, but instead, help them.

“Ultimately our goal is for our robot to help people have their lunch or dinner on their own,” Srinivasa says.

“But the point is not to replace caregivers: We want to empower them. With a robot to help, the caregiver can set up the plate, and then do something else while the person eats.”