TidyBot - A Cleaning Robot Configured via a LLM

  • January 4, 2024

I’m fascinated by the merger of robotics and LLMs so I was very excited to see the TidyBot project from Stanford, Princeton, Google, Columbia, and Nueva School.

In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people’s preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions.

Find a minute to go watch their videos as seeing this robot flip toys in to a bin has strong WALL-E vibes and you can’t help but love it, but I think this is a exactly the king of thing that will produce real results from the current AI boom. No, LLMs are not artificial intelligence, but they are wonderful tools for translating data in to human language (and vice-versa). Having a robot that cleans up a room is already possible, but it was a dead end unless you can configure it. If LLMs give everyone the ability to configure a robot by just talking to it, that’s a massive leap for robot assistants.

TidyBot - Personalized Robot Assistance with Large Language Models

Related Posts

Thoughts on Apple Bloggers and Vision Pro Coverage

I posted this on Mastodon, but I figured I would cross-post it here as well. I wonder if we’re headed for a weird place with the #Apple Blogosphere (AKA the Apple Blog Mafia). All of the main players are talking about Vision Pro all the time and saying things like “I lose track of time in here!

Read more

The Simple Pleasures of a Mobile Office Whiteboard

I’ve built a lot of stuff for my home office over the last few years of working from home, but my current favorite is something so simple: A custom rolling whiteboard A couple of weeks ago I gave myself a Saturday challenge to take my existing old whiteboard and build a rolling stand for it only with materials I had in my workshop.

Read more

I Made Myself a JARVIS Over the Weekend

With some spare parts, a USB speakerphone, and my Home Assistant server, I was able to make a competent JARVIS for my home office over the weekend…and the exciting part is that it’s only going to keep getting better! Last year was the “Year of Voice” for Home Assistant, the open source home automation platform, and the team accomplished a ton of work that enables Home Assistant users to make their own voice assistants.

Read more