UX in the Era of IoT

New Technologies, New Methods: Closing the Feedback Loop in the Internet of Things. 

The Internet of Things is accelerating rapidly, and bringing with it a wealth of opportunity. Though many focus on the data and technology needs of the Internet of Things - the sensors, data, and the storage, security, and analysis of that data - we’re already forgetting to think about the humans interacting with those technologies.

We forget them at our own hazard. With a few exceptions, the IoT will really be the IoT(&P), with “people” crucially closing the loop, and if we wish for the IoT to succeed - the “things” part, the “tech” part, the “data” part - we have to pay attention to the people interacting with those parts as well.

One need not look far to see IoT establishing a beachhead in our lives already. Smart watches and fitness trackers - the Pebble, Android, and Apple watches, Fitbit, Jawbone, Basis bands, and more - are helping us to count our steps and heartbeats, while initiatives like Wally’s smart water leak sensors, Phillips’ Hue smart lighting, Cocoon’s smart home security, and Google’s Nest smart thermostat are hoping to monitor our homes. Self-driving cars have moved from science fiction into newspaper headlines, and will increasingly demand a vast network of sensors to communicate both between automobiles and the external environment. Google and Apple are developing smart cars, but so too are traditional automobile manufacturers like Audi, which, in January, showed off a new model at CES in Las Vegas. Unlike Audi’s usual offerings there, this car drove itself on the highway to the conference from Palo Alto.

The IoT promises to improve our civic life as well - while some hope that first responders of the future will utilize sensor technology to gain situational awareness and communicate in times of crisis, others are beginning to explore the potential of sensors to notify us of earthquakes, reduce road congestion, and better inform our public transportation. The sensors and data implicated in these transformations are amazing, but they don’t function without humans to interact, monitor, and respond to them. Humans wear those smart watches, live in those smart homes, and are driven by - and their lives, in a very real sense, dependent upon - smart cars. As we develop smarter “things”, we’ll have to more smartly research and design them for their human environments.

Brave New Possibilities

We’ve already begun to discuss the importance of good research, design, and deployment in the IoT elsewhere here at Thoughtworks. Much of this will come in the form of consumer education: IoT technologies will be both more pervasive and invasive in our lives; the IoT will involve more of our personal data, and we must be careful what we share, and with whom. Privacy and security of our data will become increasingly important, and require greater public knowledge about these issues.

Some public discussion of this is already occurring: at least one US senator has broached the idea of legislation to mandate greater security in automobiles as they become “smarter.” This is a great start, but ensuring that the IoT takes off effectively, and continues without crashing, will require much more than proactive legislation.

Technology notoriously runs many times faster than government, and we need to design our emerging technologies well in order to ensure their proper use. A user-centered approach to these technologies can help ensure users understand both the potential and risks in these emerging technologies. User experience (UX) research can help technology designers understand user contexts and perceptions in the IoT, and educate those users about its realities. UX is the human touch that helps connect the loop at the point of “thing” and human contact.

Before UX as we know it today, there existed the sister fields of human factors and ergonomics. These emerged, for the most part, independent of technology: Don Norman published his iconic The Design of Everyday Things in 1988, and it discusses human-centered design in an array of consumer objects, from door handles to teapots to computer interfaces. But if UX has since found a niche, it’s in tech.

A little contemplation reveals why this is so: unlike physical products like Norman’s teapot, technology products can be tested, redesigned, and re-deployed in a matter of weeks, days, or even hours. Relative to physical products, the feedback loop between the end user and the designer in tech is miniscule.  If attempting to bring an amazing new teapot design to market, one traditionally has had to a) design the teapot, b) build a prototype, c) test the prototype, d) mass produce the teapot, and then e) ship it to stores and consumers. Only once these steps have been completed can the teapot’s designers finally study its use en masse, and use this research to guide the design of teapot v2.0. Once this step is completed, the designers can return to the drawing board and begin the process anew. This whole process, from design to research to redeployment, could take years.

Now, instead of a physical teapot, imagine we’ve created Teapotly. Teapotly is “the first social tea-drinking app that allows you to share recommendations, discover tea spots in your area, and connect with tea buddies close to you.” Regardless of what the app actually does, we can rapidly design, build, and test it. If we’re tempted to break into the British tea-drinking market, we can even send the app to users there via their phones, and obtain feedback using those same devices. Once we’ve conducted initial prototype testing and built up a substantial user base, we can collect metrics on app use, contact users for studies, and interview and survey them - in short, “do UX research” with them, and in an incredibly short amount of time. Whereas the design-development-feedback loop for our teapot took months or even years, that same process for Teapotly may take only weeks.  

Google.com homepage as of December 1998. Courtesy of The Internet Archive’s  Wayback Machine.

As the teapot to Teapotly shift shows, in tech - and especially with Internet-connected tech - this loop nearly disappears, as product designers and end users come to be separated by only a few data packets.  The “Internet-connected” distinction here is an important one, and likely helped fuel the gap of Internet dominance between Microsoft and Google in the late 1990’s. Microsoft came of age in a pre-Internet era, and hadn’t quite mastered this rapid feedback loop by the time Google came along. The rapid feedback ecosystem of the Internet, however, was Google’s home: remember the “beta” stamp on the search engine’s early web page? It was significant. Google was able to continually test and redeploy its product without having to rely on the older, slower production and shipping cycles that had been Microsoft’s SOP for years, and saw immense success because of it.

Closing the Loop

If the transition from teapot to Teapotly significantly shrunk the feedback loop between designers, developers, and users, the IoT will make it practically disappear. The very appeal of the IoT is the vast sea of data that will emerge with it - machines talking with our machines, cars, devices, and even bodies. All this data can be culled, cleaned, analyzed and predicted to improve the end services. The feedback loop will be continuous, and may nearly cease to be visible.

Our increasing use of our data-emitting devices leaves digital trails, and these are often ideal for gathering insights into users and product use. Indeed, in their book The Second Machine Age, Erik Byrnjolfsson and Andrew McAfee have advocated for using our newly rich digital trails to better inform economics research, citing examples like the gathering of online price data around the world to create a near-real-time inflation index at the MIT Billion Prices Project and the utilization of Google search data to predict housing prices and sales. Real-time digital search engine signals can be used to better conduct influenza and dengue fever surveillance as well, helping us better prepare for outbreaks and potentially save lives.

These are but a few examples in which digital activity can be used to infer larger patterns. The IoT will magnify this capability many times, and allow us to gain insights into not only users’ online traffic patterns, but those traditionally offline as well. Connected devices know how we run, bike and move through our cities, inform us of our public transportation use, and show where we use our phones (and inadvertently, the rifts in social standing they demarcate).

Think back to our teapot. Unlike Norman’s teapot designers in 1988, those designing 2018’s next-generation Io-Teapot will, with minimal effort, learn who’s using the teapot, when they’re using it most and least, with whom they’re using it, and perhaps even with what - which saucer, which cookies, which teas. If large numbers of individuals begin using this connected teapot, we can learn great deals about user populations en masse - how does American teapot use differ from that of their British counterparts? Which tea samples should we start shipping with our connected teapot, and to whom? Do they take their tea with sugar, milk, or black? If the teapot heats the water (should it?), how hot should it be, for which user demographics?

Designers, researchers, and product managers of web services and mobile apps have been asking these questions for years, and utilized techniques such as interviews, user studies, and A/B tests to answer them. But the IoT allows us to apply these same methods - the rapid designs and tests, redesigns and retests - to the physical world. This glut of effortlessly-collected user data can give tremendous insight into the ways in which our connected “things” are used and affect users’ lives. Much like the passive data output via our web searches and tweets today can tell us more about depression, flu, and the effects of racism on our political process, the data passively emitted by our connected “things” in the near future can help us glean the same types of insights. “Does tea drinking help users recover from illness faster?” the research team at Io-Teapot might ask. And they’ll have the data to answer this question ready at hand.

Qualitative Redux

No matter the quantities of data we obtain, traditional qualitative methods will continue to prevail, as no amount of quantitative data and machine learning can ever truly substitute for qualitative research. Data is great for providing answers when you already know what you’re asking, but qualitative study can be much better for asking deeper, more exploratory kinds of questions - “why” is an extremely hard question to ask quantitatively. See, for example, the Livehoods study out of Carnegie Mellon. While the researchers complexly mined large quantities of social media to find related cultural “livehoods” in various US cities, they conducted qualitative interviews to verify their algorithmic approach.

The IoT stands to greatly improve this process. One prominent method that should see increased use as we move more clearly into a future of distributed, connected devices is that of Ecological Momentary Assessment (EMA), or experience sampling. EMA is simple, and consists only of obtaining frequent qualitative measurements from study participants, generally over the course of a day, week, or month. While research requiring data input of this kind was once burdensome and unreliable - it often required users to complete paper diaries and remember their feelings from hours earlier in the day - mobile phones, watches, and other connected devices are ideal for this kind of study. Mobile devices are almost always with users, can alert them to when they need to input data, and can be filled out discretely. While it might make some uncomfortable to pull out a paper diary at a bar with friends, a mobile phone diary can be updated without drawing unwelcome attention.  

Mihaly Csikszentmihalyi - progenitor of the psychological notion of flow - is largely credited with pioneering experience sampling in his own research on the psychology of happiness. The method can - and should - be used extensively for user research. Experience sampling has remained in the background of UX since at least 2003, and as we move into an era of ever-smaller, ever-faster, ever-more connected computers, its relevance to UX will only grow.

Moving Forward: the IoT and “Soft” Hardware

The digitization of our hardware “things” will allow us to study their use via traditionally software-driven UX methodologies, but a substantial difference between the hard- and software worlds will persist. The rapid feedback loop discussed with our teapot example consists of a) gaining information and then b) acting upon it. While software designers can rapidly deploy an app or website redesign overnight, such a feat is significantly more difficult to accomplish with physical hardware. There are ways around this however, and there will certainly be more as the world of connected hardware comes to fruition.

For one, 3D printing is gaining prominence and will surely provide an avenue for rapid deployment of hardware. While this should help, another important shift in how we design our everyday things will do much more to reduce the design-build-test feedback loop. We need to begin designing our “things” to be modular and easily upgradeable, as seen in the form of modular phones beginning to gain traction in the US and elsewhere. Modular phones, while only now emerging as a real possibility, may significantly reduce phone waste and begin to create module markets not unlike app stores today. Deploying a phone hardware upgrade now - a better camera, say - requires working with large phone manufacturers to integrate the upgrade into the phone’s next hardware release. With modular design, this problem may disappear - release a better camera to the “module market”, and users can purchase, use, and provide feedback on that module immediately. Hopefully this process sounds familiar - it’s nearly identical to the research, design, and development process that prevails in web-connected software today.

Similarly, Tesla has delivered a series of upgrades to its fleet of vehicles remotely via software upgrades. These aren’t only small upgrades to the dashboard display - they’ve conducted upgrades to the battery range, acceleration, and suspension of the vehicles in the form of overnight “over-the-air” updates, and have even used these to react to a vehicle recall affecting both its cars and those of GM (guess which car owners had to travel to the dealer to update their vehicle, and which got to have the upgrade effortlessly download and install while the cars remained parked in their garages?).

All indicators point toward an IoT future reliant on rapid research deployed through modular hardware and software upgrades to modify that hardware. Given the trend of software development toward all things agile, it seems only logical that agile hardware development - through increased modularity and other innovations - will become the way forward in our networked future.

Mind the (Knowledge) Gap

The IoT will drastically reduce our design-build-test loop, and we will be able to better understand users and shape products because of it. It took years to complete this product development cycle with physical products thirty years ago; today it may require only days or even hours. This has tremendous implications for the way in which we think about the products we design and use, and the context in which we use them. The “products” in the IoT increasingly make up everything in our environments - we’ll be able to learn more about how we drive and take the train, when we’re sick and when we’re well, and perhaps even how we take our tea.

While we stand to reap great benefits from the coming deluge of user data, we must also ensure that users understand the risks of sharing this personal data with the world.  The automatic flagging of cases of postpartum depression through analysis of users' tweets and Facebook posts may provide great benefit to potential victims, but the same data may result in great harm for the same individuals. Patients will increasingly communicate with their health providers using mobile devices, but they’ll have to understand the benefits and risks of doing so, whether through large leaks of hospital data or gaps in personal device security.

Some lawmakers may be starting to think about what will happen when someone hacks into our smart cars, but individuals also need to understand the risks. Perhaps most importantly, users need to consider what happens to data after companies disappear. Even if I trust my data in Facebook’s hands, what happens when the company goes bankrupt? Where did our MySpace data go?

For the IoT to succeed and truly become an IoT & P - with people prominently in the loop - UX increasingly needs to educate users about both its possibilities and risks. Data without people to understand and act upon it is worthless, as is a data collection framework that fails because people don’t trust or understand it. Despite the hype, we have to realistically think about and discuss the promises and pitfalls of the IoT, of connected sensors and Big Data. They shimmer now; but if we fail to remember the people we claim they can benefit, will quickly fade.