I Made Myself a JARVIS Over the Weekend

With some spare parts, a USB speakerphone, and my Home Assistant server, I was able to make a competent JARVIS for my home office over the weekend…and the exciting part is that it’s only going to keep getting better!

Last year was the “Year of Voice” for Home Assistant, the open source home automation platform, and the team accomplished a ton of work that enables Home Assistant users to make their own voice assistants. Why would anyone want to do that? There are a few reasons:

1. Privacy
2. Customization
3. Enhancements, such as OpenAI
4. It’s awesome!

Honestly, I just dove in to it Friday night because I was reading about it and realized that I had the parts. While (re)watching The Matrix, I went to crafting my own personal JARVIS by sticking a carefully configured Raspberry Pi to a USB speakerphone, and configuring the “Hey Jarvis” wake word, and setting up various commands to make it feel as “real” as possible with today’s technology.

I’m pretty happy with what I’ve ended up with. I can make most of the standard commands (see below for more notes on this) to turn on and off lights, etc. I can have a few custom commands to do things like boot up my office in the morning or shut it down at night, and I have commands configured to give me OpenAI-powered responses to improve what JARVIS can do and how it replies to my commands.

There are certainly things I want improved. Some of which are on the horizon via the Home Assistant team, and some might be longer-term.

1. Home Assistant’s built-in voice commands don’t cover all that much right now. Mainly it’s turning on and off lights, but I’d like to see more such as setting thermostat temps.
2. Right now, you can talk to Home Assistant to actually change things in the house, or you can talk to OpenAI and have a conversation, but OpenAI can’t manipulate anything in your home. Having the two sides better integrated would be amazing. Right now I’m accomplishing it via a series of custom commands that do something and then generate a response from OpenAI an send that audio to my speaker. I also have commands that I can say that will just talk to OpenAI to add that conversational aspect to JARVIS.
3. A short term solution to #2 would be a fallback approach where you can talk to a single voice pipeline in Home Assistant that tries to find the command in their system and if it fails, hand off the command to a second voice pipeline, such as OpenAI, to take it from there.
4. The speaker I made is not easy to setup, and requires a fair bit of technical knowledge. I have no doubt that will improve quickly, but for now it’s not for the average person. It’s also not a solution to play music on, so my Apple HomePods are going to remain right where they are.
5. I don’t have a voice model from JARVIS/Vision himself, Paul Bettany. Does anyone know him? If so, he could make a lot of nerds very happy by open sourcing a few voice tracks!

I didn’t want this post to be a tutorial, but if you want more details or have any questions, please send them my way via LinkedIn, YouTube, Mastodon, or Threads and I’ll answer them ASAP.