I Made Myself a JARVIS Over the Weekend

  • January 8, 2024

With some spare parts, a USB speakerphone, and my Home Assistant server, I was able to make a competent JARVIS for my home office over the weekend…and the exciting part is that it’s only going to keep getting better!

Last year was the “Year of Voice” for Home Assistant, the open source home automation platform, and the team accomplished a ton of work that enables Home Assistant users to make their own voice assistants. Why would anyone want to do that? There are a few reasons:

  1. Privacy
  2. Customization
  3. Enhancements, such as OpenAI
  4. It’s awesome!

Honestly, I just dove in to it Friday night because I was reading about it and realized that I had the parts. While (re)watching The Matrix, I went to crafting my own personal JARVIS by sticking a carefully configured Raspberry Pi to a USB speakerphone, and configuring the “Hey Jarvis” wake word, and setting up various commands to make it feel as “real” as possible with today’s technology.

I’m pretty happy with what I’ve ended up with. I can make most of the standard commands (see below for more notes on this) to turn on and off lights, etc. I can have a few custom commands to do things like boot up my office in the morning or shut it down at night, and I have commands configured to give me OpenAI-powered responses to improve what JARVIS can do and how it replies to my commands.

There are certainly things I want improved. Some of which are on the horizon via the Home Assistant team, and some might be longer-term.

  1. Home Assistant’s built-in voice commands don’t cover all that much right now. Mainly it’s turning on and off lights, but I’d like to see more such as setting thermostat temps.
  2. Right now, you can talk to Home Assistant to actually change things in the house, or you can talk to OpenAI and have a conversation, but OpenAI can’t manipulate anything in your home. Having the two sides better integrated would be amazing. Right now I’m accomplishing it via a series of custom commands that do something and then generate a response from OpenAI an send that audio to my speaker. I also have commands that I can say that will just talk to OpenAI to add that conversational aspect to JARVIS.
  3. A short term solution to #2 would be a fallback approach where you can talk to a single voice pipeline in Home Assistant that tries to find the command in their system and if it fails, hand off the command to a second voice pipeline, such as OpenAI, to take it from there.
  4. The speaker I made is not easy to setup, and requires a fair bit of technical knowledge. I have no doubt that will improve quickly, but for now it’s not for the average person. It’s also not a solution to play music on, so my Apple HomePods are going to remain right where they are.
  5. I don’t have a voice model from JARVIS/Vision himself, Paul Bettany. Does anyone know him? If so, he could make a lot of nerds very happy by open sourcing a few voice tracks!

I didn’t want this post to be a tutorial, but if you want more details or have any questions, please send them my way via LinkedIn, YouTube, Mastodon, or Threads and I’ll answer them ASAP.

Related Posts

Hacking an Aqara Door Sensor into a Dead Bolt Sensor

Here’s an exciting home automation project for you: Can a door sensor work as a way to tell if a door is dead bolted? Yes…but do you have any paperclips? I’m a big fan of home automation, especially since working from home, and I’ve been experimenting with the Aqara sensors. Their door, temp, water, and vibration sensors all work great for me, but what I really need is a way to check if the back door of my garage is locked.

Read more

Alexa, Lets Play Jeopardy!

I’ve been playing around with my new Amazon Echo and writing up some “Skills” for the Amazon Alexa voice assistant and so far so good! What I’ve got above is a link to a YouTube video I did demoing a “Jeopardy” skill. It’s not fully baked in that you can’t play a complete game of Jeopardy, but you can choose a category, get a question, and it will keep score for you.

Read more

Thoughts on Apple Bloggers and Vision Pro Coverage

I posted this on Mastodon, but I figured I would cross-post it here as well. I wonder if we’re headed for a weird place with the #Apple Blogosphere (AKA the Apple Blog Mafia). All of the main players are talking about Vision Pro all the time and saying things like “I lose track of time in here!

Read more