smarthome Archives - Will Vaughan | UX and Product Design

My first ever smart home device were smart lights from Philips Hue a after graduating college. They were perfect when I just had a bedroom in a shared apartment. And more importantly they were fun! I could change colors, dim the lights, and pair them to movies. Over time the last 10 years I've slowly accumulated a bunch of smart devices from different companies to do different things.

At this point we're using connected devices to:

Control our outdoor lights
Control our indoor lights in rooms without light switches
Play music throughout the house
Ask questions to Google
Turn the TV on/off and launch streaming apps
Set timers in the kitchen
Look at recipes
Video call our family
Open / close the garage door
Lock our front door
Look for packages and ring our doorbell
Alarm and monitor the house while we're away
Control our small sprinkler system
And probably more

And I love it! It's awesome to be able to turn off all the lights in the house when I go to bed or to have the garage let me know I accidentally left it open. My absolute favorite feature to this day is being able to easily play music throughout the house just by asking Google. It still blows my mind. But sometimes things go wrong and something disconnects or dies and the whole system starts to fall apart.

Last week the lights in my living room stopped responding. Yesterday I found my wife on our front porch, locked out of the house by our dead Yale smart lock. Every day my office lights seem to flip a coin on whether or not they want to stay connected to WiFi. This isn't usually a big deal for me. I don't mind dealing with the troubleshooting most of the time. Unplug it, plug it back in and see if it's working. Reset the bulb and reconnect the WiFi. But as you add devices to the system the troubleshooting becomes more complex.

Lost connections

Recently IFTTT, a tool I've been using for years to connect various software and smart home services, decided to change their billing plan. Free customers would now be limited to three applets - the if this then that rules you use to connect two services - and the rest would be archived. At the time I'd been using IFTTT to:

Sync my Fitbit scale data with Google Drive
Connect Logitech's Harmony remote system with Google Assistant
Send me text messages for my basement water sensor around my water heater

The Logitech connection alone used multiple applets to let me control my TV, speakers, and Xbox from Google Assistant. So when IFTTT changed their plan the voice prompts I'd been using for years - "Hey Google turn the TV on" - suddenly stopped working. And sure IFTTT is well within their rights to adjust their pricing strategy, but the issue here is that when the connections between these devices are so fragile, even a small disruption from one service can ripple through our connected home.

Abandoned by the manufacturer

An even more concerning issue is when the device manufacturer's change which connections they allow or stop supporting their connected devices completely. In 2014 the startup Revolv launched a smart home hub to help customers make sense of all the different smart home standards. Customers could connect their different devices to the hub and then use Revolv's app to create custom actions - similar to IFTTT. People used this hub to create custom actions like turning on their sprinklers when a motion sensor was activated. In October of that year Nest acquired Revolv to help build out their Works With Nest platform that opened up Nest devices to third party platforms. Less than two years later, however, Nest decided to shut down the cloud servers that powered Revolv, breaking the custom connections Revolv customers had set up. And then three years later in 2019 Google ended all new connections for the Works With Nest program! New Nest customers would have to use Google accounts that no longer supported the same third party integrations. What a journey. And this is only one company! There are plenty of other examples of smart devices suddenly becoming dumb devices or completely useless paperweights.

Troubleshooting

Who can you call when your smart home breaks?

When these connections break, it's not always easy (or even possible) to repair them with other services. My 1930s house has plenty of old systems that are still in use. We have steam heat radiators, cloth-wrapped electrical wires, and a 1990s home security system. While each of these requires maintenance, each is independent and none rely on cloud servers managed by their manufacturer. They can even be repaired by non-manufacturer experts! If one of our outlets stops working it can be repaired by any electrician. Familiarity with these systems fades in time as technologies like steam are phased out, but a 90+ year lifespan is a far cry from the 3 years of Revolv's hub.

Our Yale smart lock automatically unlocks for me most of the time, but never unlocks for my wife. We're both signed in with the app, have given it the appropriate location permissions, tried reinstalling the apps, and even resetting the lock per Yale's customer support. None of it worked and we're left with a lock that works but not as expected and there doesn't seem to be anything we can do about it. Yale shipped us a new lock but the problem persists. Even when companies are aware of issues they don't always fix them. Just this week Wink customers have been left without a working device for over a week with no updates from the company. Sometimes those fixes never come and customers are left with broken devices.

Overlapping services

Another frequent challenge is the number of overlapping services you can end up managing. If you have the time and money you might be able to plan out your smart home to strategically use compatible devices, but for someone like me who's built up my system over time it's easy for the services to accumulate. I have smart lights from both Hue and Wyze which each have their own apps. I typically manage these through Google Home, but could also use Apple's HomeKit. When our bedroom lights dimmed unexpectedly I had to check both my Google Home and Hue apps to find an old Hue automation that dimmed the lights. A true smart home would help me figure this out.

For someone like me who's interested in technology and is frequently an early adopter this is tolerable. But what about people who don't have the time, inclination, or expertise to put up with all this? They're left buying into devices and systems that may or may not work as expected. Designers of these systems need to understand the complexities of the environments they will be operating in and the wide range of customers who will be interacting with them.

Designing for everyone

Even when the system itself seems to be working as intended, our failures to design for all users can lead to unfortunate challenges. At our house the Google Assistant seems to have a harder time understanding my wife. She often has to repeat her query or say "Hey Google" more than once before it will start listening. Unfortunately it's not just our house that has these issues. Voice recognition in general has been shown to have gender and racial biases as the training data used to create machine learning models is often an incomplete sample based largely on data available from white males. As reported by the NY Times, a 2020 study from Stanford showed that speech recognition systems "misidentified words about 19 percent of the time with white people. With black people, mistakes jumped to 35 percent." Understanding the reasons behind these failures, as well as the implications for customers, is crucial to help smart devices work for everyone.

So where do we go from here?

Smart devices are a fun way to bring technology and automation into our homes and while all the challenges I complained about above are real, it's not all bad news. There are a few things smart home consumers and smart home manufacturers can do to make all of our lives easier.

For consumers

Plan for graceful degradation

While I'd love to assume that each company behind my smart home devices will both continue to exist and support their devices for their entire lifetime I know that is unrealistic. For less expensive and novel items (like a color changing lightbulb) this might not be a big deal. For mroe expensive and/or longer-lasting devices (like a lock, smoke detectors, or security system) can look for devices that can continue working without a cloud-based service. You may lose some of the best features, but at a minimum it should be just as functional as it's dumb home relatives. A lock should still unlock without WiFi, a smoke detector should still yell at you even if the power is out, and your security system should still beep even if it can't send you a text.

Invest in companies with a good track record

Many companies already have an extended track record of long-term support. Investing in products from companies like this as well as strong companies that have been around for a while is a good way to hedge against the eventual shutdown of a smart device.

Open source and local solutions

Open source tools like Home Assistant openHAB, Home Assistant, and Hubitat let you build your own smart home hub and host it on your local network. For now it takes a decent amount of time and technical ability, but these services are making it easier and easier to run your own smart home system. I recently purchases a Raspberry Pi 4 to start experimenting with Home Assistant on my own to do things like turn a hallway light red when I'm on a meeting and check the battery level of my Yale door lock. Contributors are adding integrations to these services all the time and working to make them easier to set up and use. I'm hopeful that services like these will bring some power back to the consumer and maybe even let us use devices abandoned by their manufacturer.

For manufacturers

Diversify

Broadening our sample sizes and including customers and coworkers from traditionally underrepresented demographics can help create systems and services that work for more people. Those of us who work in technology should be thinking about how our products can work for people all over the world of different ages, ethnicities, income levels, education, and more. Everyone should be able to benefit from the efficiency and convenience smart, connected devices can bring. And as we continue to make these services work for more people, we need to continue ensuring that they work together and don't ask too much from the user.

Collaborate

To help with troubleshooting and setup, the systems we make should be able to easily talk to one another and there should be an easy way to see where a command came from. Who or what asked my lamp to turn on at 5am? We need to assume that customers aren't using these devices in a vacuum and are likely to have many other integrations, devices, and services running. It's up to us - not them - to ensure our products work well in that environment and that the customer can easily and successfully configure their system and troubleshoot when needed.

Open up

In an ideal world manufacturers would open source services they're shutting down so consumer can continue to use the product. This might be unrealistic but I think a realistic goal is to support additional integrations. If a company doesn't want to support a product then they should at least let the open source community continue to support it. Android's open source community is a great example of how old devices can be given an extended lifetime thanks to the developer community. I don't think we'll see something quite as flexible from smart devices, but if product teams start with more robust and powerful integrations it will be a fantastic start.

Published by willvaughan in IoT

Tags: Google Home, IoT, Nest, smart devices, smarthome

Earlier this year the property management for my apartment building gave us notice that all apartments would be upgraded with smart home technology by Smart Rent. We were given a brief notice via email and a meet and greet to ask a few questions. A few days later they replaced each apartment lock with a Yale smart lock and texted us our PIN.

Image result for yale smart lock

In general I’m a huge fan of our increasingly connected world. I’ve worked on home security cameras that help families find peace of mind and delivery robots that make hotel guests and staff feel safer by delivering to guest rooms late at night. Every day I ask Google to play music while my wife and I cook dinner or clean the apartment. When it comes to connected devices and the smart home I tend to be an early adopter and have been looking forward to a smart lock for years, ever since the beautiful August smart lock was announced.

Unfortunately the reality of living with a smart lock has been mostly disappointing with one major benefit. Let's start with the highlight and then dive into some of the challenges.

Dynamic codes and peace of mind

Hands down the best part of a smart lock is being able to share temporary codes. This mostly comes in handy for us when we're traveling so that we can have a friend stop by to water our plants. Our landlord limits the code window to 48 hours, but for our use case that's plenty of time.

Sending a code is relatively easy, but it does require providing Smart Rent with our friend's email or phone number. It's not clear why they can't just generate a code that I can share myself and this does raise some privacy concerns for me (more on that later).

In addition to the convenience of temporary codes, the lock also lets me check its status from wherever I am. If I ever think I've left the door unlocked it's just a few swipes and taps to make sure. I can even lock the door via Google Assistant for added convenience. The lock could definitely be smarter here by knowing when I come and go based on location or bluetooth proximity, but I'm ok trading some extra legwork for the added privacy here.

So if the lock adds convenience and peace of mind then what is that mostly disappointing part I was talking about? My concerns and frustrations boil down to two primary categories: usability and accessibility, and privacy and security. I'll be tackling each in a separate blog post.

Let’s start with usability and accessibility

Sometimes the old ways are best, however, and sometimes the best interface is centuries old. Let’s break down using a traditional key.

1. Remove key from pocket
2. Insert key into keyhole (this may require some fiddling)
3. Twist key
4. Turn handle and open door
5. Remove key
6. Twist knob to lock the door from inside

In theory, a digital smart lock can help reduce these steps. I no longer need to find or insert a key so steps 1-3 go away. Now all I have to do is:

1. Enter my pin
2. Turn handle and open door
3. Twist knob to lock the door from inside

We’ve cut the process in half! Not only does a digital lock save steps, but it also has those super useful temporary pins.

Unfortunately the physical / digital interface introduces more complexity and failure points than the traditional physical-only lock.

A touch screen

Gone is the physical affordance of a lock and key. Instead of a key we can insert and turn, we now have a blank sheet of glass. Remember the PIN code management emailed us? Let's use it to unlock the door.

1. Activate the touch screen by placing your palm over the screen.
2. Enter your four digit PIN.
3. Hit the check mark to confirm
4. Wait for the motor to finish moving the bolt
5. Enter your apartment
6. Manually turn the knob on the inside of the door to lock the door

Within each of the steps there are a number of possible failure points and complications.

Step 1: Activate the touch screen

For new users there's no guidance on how to activate the screen. The only way to activate it is by placing your palm over the whole screen - a simple touch isn't enough. I'm not sure someone would figure this out without reading the documentation other than a lengthy battle of trial and error. Even when you know what to do, getting your palm flush with the door can be tricky given the hip-height placement of the lock and the flush alignment against the door.

Yale could improve on this by waking the touch screen on a tap or an extended press. If the screen were angled up towards the user the angle would be easier to achieve, especially for someone with tight forearms from rock climbing like me.

Step 2: Enter your PIN

Now that we've activated the screen we need to enter our code. The screen is a typical touch screen but lacks additional forms of feedback like haptics or sound. The only feedback that you've hit a button is a brief flash. With the lack of physical buttons and the close proximity of each number it's pretty easy to hit a number accidentally. Unfortunately there's no backspace so if you make a mistake you have to hit the checkmark, wait for a wipe animation, and start again back at step 1.

For someone with impaired vision unlocking the door would be nearly impossible. Theoretically you might be able to use the voice assistant or accessible smart phone apps, but both still require significantly more interaction than finding a physical keyhole.

Just because you can see the buttons, however, doesn't mean you can press them. What happens if you have your hands full of groceries, children, or are someone living with a disability? Making the precise movements to enter your code without accidentally hitting the wrong number is going to be a challenge.

What kind of message does this send to renters? Oh you’re blind? Too bad you won’t be able to open your front door. Oh you’re frail or have Parkinson’s Disease? This isn’t the place for you. You’re a parent carrying a baby in one arm with two bags of groceries on the other? Good luck making precise movements to hit your PIN just right.

Adding physical buttons could help both avoid accidental touches and also help visually impaired users find the right keys. I understand these could become worn over time and reveal your pin, but touchscreens show fingerprints and a rotating PIN could help avoid wear and tear on a limited set of buttons.

Steps 3 and 4 aren't too bad, other than taking some extra time for the motor to open the lock.

Step 5: Enter your apartment

The next big problem point happens as you're entering your apartment. When the door is unlocked, placing your palm against the screen locks the door. It's easy and convenient when you're leaving, but can you see the problem when you're entering?

As you're holding the door open, it's easy to accidentally cover the lock with your arm, hip, or back, extending the lock and preventing the door from closing. Luckily you're inside now so you just have to rotate the knob to unlock and then re-lock the door. The lock is too difficult to activate when you're trying to unlock the door and too easy when you're entering your apartment.

A "smart" solution here could be to use a simple sensor to know whether the door is closed. With the door closed, placing your palm on the screen could lock the bolt, and when the door is open accidental touches could be ignored.

Accessibility and usability

So many of the usability challenges with the Yale smart lock become even greater barriers for people in "complex" situations like carrying groceries or for people living with disabilities. Designing for inclusivity would make this a more user friendly product for everyone. Just because it's a smart lock doesn't mean we need to lose the benefits of physical, tactile interfaces like a lock and key.

Published by willvaughan in Uncategorized

Tags: accessibility, IoT, smarthome, usability, UX

Debugging the smart home