TidyBot - A Cleaning Robot Configured via a LLM

I’m fascinated by the merger of robotics and LLMs so I was very excited to see the TidyBot project from Stanford, Princeton, Google, Columbia, and Nueva School.

In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people’s preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions.

Find a minute to go watch their videos as seeing this robot flip toys in to a bin has strong WALL-E vibes and you can’t help but love it, but I think this is a exactly the king of thing that will produce real results from the current AI boom. No, LLMs are not artificial intelligence, but they are wonderful tools for translating data in to human language (and vice-versa). Having a robot that cleans up a room is already possible, but it was a dead end unless you can configure it. If LLMs give everyone the ability to configure a robot by just talking to it, that’s a massive leap for robot assistants.

TidyBot - Personalized Robot Assistance with Large Language Models