Enabling richer experiences through multimodal interactions

Modes of interaction between people and machines have proliferated rapidly in recent years, to encompass text, voice, images, video, gestures — and with affective computing, even emotional cues that may be unconscious. 

This presents new possibilities, but also a challenge in requiring organizations to think through how different forms of interaction can be combined and used across the entire customer experience. Doing that can raise multiple questions. When is gesture or image recognition more helpful than a standard text interaction? When are physical buttons important, and in what situations could they constitute an obstacle? Carefully considering context, and cognitive load can help ensure a positive experience that does not jar, irritate or unintentionally make an interaction more difficult. Innovation should never degrade a product or service.

While enterprises focus on enhancing their customers’ online experiences they can gather insights that are applicable in other areas of the customer experience.

AI is playing a significant and expanding role not only in how we interact with services and products — most often through chatbots — but also how we model interactions. Agentic AI, for instance, can simulate human behavior to help sales teams identify prospects, or anticipate the customer response to a product or campaign.

Businesses should also remember not every new interaction is embraced by the customer. The backlash provoked by trends like gamification and addictive tech mean organizations need to be more considered in who they’re building for, and why. This involves paying close attention to the friction in the customer experience and the accessibility implications of different channels for interaction, but also rethinking the foundations of what it means to interact with a product or brand. The best outcomes stem from avoiding the temptation to prioritize only optimization or stickiness, and considering more holistically how innovation in interaction can add value, or expand the effective range of a service.

Signals

• AI-driven chatbots becoming increasingly commonplace in domains such as customer service. Research suggests the global chatbot market, valued at just over US$5 Billion in 2023, will surge to over US$36 Billion by 2032. Chatbots also continue to evolve, with techniques like retrieval-augmented generation (RAG) improving the accuracy and reliability of interactions. 

   

• Steady growth in the voice interaction space, with more companies adopting voice assistants, and voice searches and requests accounting for a growing proportion of consumer activity.

   

• Legislation taking aim at ‘dark patterns,’ aspects of the user experience that are subtly designed to nudge users into performing certain actions that are typically not to their benefit. Article 25 of the European Union’s Digital Services Act is one example. Similarly, accelerating and intensified gamification, as seen in the prevalence of controversial loot boxes, is coming under more government scrutiny. 

   

• Consumers growing more conscious of ‘screen time’ and actively seeking to avoid addictive tech. There’s been a noticeable rise in apps to help users manage how they interact with their devices, or cut down interactions altogether, with Google recently rolling out ‘screentime reminders’ as part of the Digital Wellbeing suite for the Android mobile operating system. Australia banning under 16s from social media is one high-profile example of the rising fear around excessive screen exposure.

   

• Agentic AI is being presented as the ‘next frontier of generative AI.’ Able to handle more complex tasks, accept directions in natural language and work with other software tools, agents could change how we collaborate, complete tasks and interact with other technologies. 

   

• Continued innovation in AR and VR. Despite the lukewarm response to some of the high-profile headsets that have launched, we still see companies making big bets in this space. AI integration is coming to Ray-Ban Meta smart glasses and Meta has also unveiled plans to release a neural wristband in the future, notably as part of its Orion project. Even if consumers have yet to embrace this technology fully, there’s no shortage of companies seeking to find a killer app.   

   

• A neuralink chip being inserted into a person for the first time, marking a major step forward in the development of direct brain-computer interfaces. The chip was also recently greenlit for clinical trials outside the US, as a potential means to enable paralyzed individuals to use the mind to engage with digital devices.

   

• Consumer appetite for AR/VR retail experiences, which iconic names like Louis Vuitton have leveraged to engage customers and enhance their brands. One recent survey showed nearly a third of US consumers are keen for VR to recreate brick-and-mortar shopping routines. 

The opportunities for multimodal interactions

By getting ahead of the curve on this lens, organizations can: 

• Explore multimodal forms of interaction to create customer engagement breakthroughs. Choices on which channels of interaction to adopt or offer need to be guided by context, situation, geography and user preferences. But examples like Canon’s World Unseen exhibition, which featured images accessible to people with visual impairments, show how new technologies can expand what it means to interact with a product, service, or company, and deepen relationships with users or consumers. 

   

• Expand their potential market by bringing accessibility to bear. Innovations like Signapse’s automated sign language translation, and AI models that allow blind people to interact with applications through natural language, show the potential for companies to open access to their brand or services for new customer groups.  

   

• Leverage AI agents to offer new conveniences to consumers. Though agentic AI remains a nascent category, the technology is developing quickly, and can mimic human responses and behavior with uncanny levels of accuracy. This opens the possibility of automating more complex tasks and processes to help consumers accomplish their goals, whether by retrieving information, providing tailored advice or offering multilingual support.    

   

• Adopt extended reality (XR) to improve training and onboarding. Organizations such as Boston’s Mass General Brigham hospital are applying AR, VR and mixed reality to replicate real-life environments and situations, helping employees grow familiar with their roles and master new skills in a more immersive way. XR can be particularly valuable for training workers in high-risk industries, by exposing them to emergencies without any real-world danger. 

   

• Reimagine how teams, and customers, interact with data. By applying AR and VR, organizations can move data visualization into new dimensions, providing users richer, more in-depth experiences of information, lightening the cognitive load and even potentially accelerating time to insights.

Actionable advice

Things to do (adopt)

• Develop interactions with accessibility in mind, and consider how new forms of interaction can support the creation of experiences that cater to all kinds of users. This may require educating and upskilling design and development teams, so they’re aware of the expanding range of technologies that can be leveraged — or in some cases, rethinking how to approach the interaction.

   

• Examine the impact of voice search on the way you manage and index products and information. As technologies like GenAI-enabled agents make more queries and interactions verbal, the way organizations identify and categorize information will need to change. Optimization of data for voice search, and natural language metadata and tagging are likely to become increasingly important features of major platforms for interaction such as websites. 

   

• Prioritize quality, reliability and stability when developing new interactions, and subject them to a comprehensive testing strategy. More sophisticated interactions can throw up new challenges, and even degrade a service or brand. Chatbots have been known to go off script and should be subjected to regular evaluations, especially when techniques like RAG are involved.

Things to consider (analyze)

• Choose the right interaction for the context. It’s easy to get excited about new technologies, but you should always be questioning whether they’re the best fit for a particular use case, or how they’ll improve the experience of end-users. Sometimes gesture recognition isn’t needed, and a simple button will do. As companies like carmaker Hyundai have found out, consumers often prefer ‘old-fashioned’ means of interaction as their inherent frictions can impart a sense of confidence, even joy.   

   

• Think through user mental models when using technologies like AI. When rolling out something like a chatbot, it’s vital to map out what people are likely to be expecting from an interaction, and what assumptions they’ll bring to the table. Steps to help end-users write better prompts, or to explain outputs, can improve the overall user experience with new technologies.  

   

• Don’t pursue stickiness at all costs. The drive to optimize all experiences or create something addictive isn’t just dangerous for users; it’s often unsustainable and can provoke backlash. Rather than contributing to ‘dark patterns,’ consider the flipside: are there any opportunities to nudge users towards better, more cost effective, environmentally friendly or healthy decisions?

Things to watch for (anticipate)

• Evolving consumer tech opening new interaction platforms. As  products like Meta’s Orion exit the experimental phase and enter the mainstream, companies need to be prepared to build out new forms of interaction that cater to their expanding user bases. 

   

• Tighter regulation taking aim at dark patterns and other forms of interaction perceived as manipulative. Rather than simply rules to follow, such developments should be seen as an opportunity to build better interactions that are sustainable over the long term, foster trust with user bases and create commercial advantage over optimization-obsessed competitors.