XR with impact: Building experiences that drive business value

Episode transcript

[MUSIC PLAYING]

Rebecca Parsons: Hello, everybody. My name is Rebecca Parsons, and I'm one of your co-hosts for the Thoughtworks Technology Podcast. And I'm joined today by my co-host, Ken.

Ken Mugrage: Good morning, everybody. I'm Ken, I'm one of the other regular hosts of the podcast.

Rebecca: And then I'm also joined by two colleagues, one from Bangalore, India. Vaibhav, would you like to introduce yourself?

Vaibhav Tikekar: Hi, good morning. My name is Vaibhav. I am the business leader for Thoughtworks extended reality offering, and I'm glad to be a part of this podcast.

And then our other guest is from New York. Shea, would you like to introduce yourself, please?

Shea Clark-Tieche: Yeah, thanks so much for having me on. My name is Shea, and I lead the technology for the customer experience design and product in North America. Started with Thoughtworks eight years ago, primarily a developer in the mobile space. And I've recently transitioned into playing the tech principal role in one of our AR engagements.

Rebecca: Excellent. So why don't we start there. How did we get into all things AR, VR, XR, MR?

Vaibhav: I think it's a long journey for us. We started about four or five years back. And this engagement started with Lenovo. Lenovo was actually interested in making enterprise-grade VR headsets. And that is when they reached out to Thoughtworks to do all things software to make the device more user-friendly, more efficient. That's how we got — that was our first gig in the AR universe. Since then, we also got an opportunity to work with Reese, who is an Australian retailer. We had certain blips coming our way. And that's when I think a decision was made that we should get into this AR world, invest more heavily on this, and try to create an offering out of it.

Rebecca: And there are lots of terms that get tossed around: augmented reality, virtual reality, mixed reality, extended reality. How do you define these?

Vaibhav: I think the simplest way to put it out is, these are different levels of immersiveness in our experience. So the most initial one is augmented reality, wherein you have your physical reality, and on top of it something is superimposed. Think of Google Maps, something that we use every day. So roads are physical structure; and what Google Maps allow you to do is it creates a virtual route for you which shows you what's the best path to reach your destination.

And on the other extreme is where you have virtual reality, where you're completely immersed in the virtual experience and are cut out from the real world. This is where you put on headgear, and you are in a completely virtual environment. And somewhere in between is mixed reality, which is where, while you are working or you are experiencing this virtual world, but still you are connected with the real world. And all of these things are given an umbrella term called extended reality.

Rebecca: So when is it appropriate to use XR? What are the kinds of things that you can do with it? I know we've had this conversation with respect to the metaverse. Well, everything will be in the metaverse. But what kinds of things are naturally suited to these alternate realities?

Vaibhav: So I think it's a part of customer experience. So if you are looking at building better engagements, if you're looking at things, like how do I help my customers make decisions faster, where it's difficult for them to imagine how a product or a feature would shape in the real world. Or if you're looking at how do I solve a problem wherein different parts of my team are working from different parts of the world, how do they come together to do something which reduces time and improves efficiency. I think those are more suited. But these are generic use cases. I think what defines an appropriate use case is, one, the purpose of the organization. So what is your organization's goal? What are you set to do? And the second is, who is your audience? Are they one of those who are more open demographically, who are more naturally inclined towards modern technologies because adoption is an important factor here. And the third thing is the mindset, the experiment mindset. I think those three things drive an appropriate use case for XR.

Ken: A lot of times when enterprises experiment the way you do, that you're talking about they will choose pilot projects that are super side projects that a few people are interested in. And even if they're successful, they don't really have the enterprise impact they hope for. So how do you balance this experimentation and identifying that crowd with something that also has enough impact that the rest of the enterprise is like, OK, this XR is something we should build on?

Vaibhav: So Ken, I'll take a step back before I answer those questions. When we started doing emerging technologies in general, we realized there are five anti-patterns that most organizations go through when they try to find a good use case for any emerging tech. First is when most enterprises think of pilot — like you said, their side projects — which is I think a wrong way to pick a pilot ,in my opinion. A good way to pick a pilot is, on one hand, you have your organization's business goals. Say for example, if your goal is that I want to increase my total addressable market if you're a real estate agent. And for that you're saying, I want to sell properties to people who don't stay in this place, maybe foreigners. That's your good way to think about a use case. I want to sell properties to people who cannot physically come in and have a look at the property. And then you create a hypothesis saying, well, what if I set up some cameras, live feeds in the room, that allow people to experience property without being physically there. So when you're solving the problem, you're not putting XR at the center of it, you're putting the problem at the center of it. And then slowly, you're converging towards a solution. And I think that's important. That's where a lot of people that I've spoken with, they miss the trick.

And you start with small experiments. You validate them fast. You learn them fast. But if the results go the way you have, you already bake in a plan that accounts for what happens if this is successful.How do we scale this? How do we not get into this pilot mode, where we're creating hundreds of pilots, and none of them is seeing the light of day in the real business world?

Rebecca: So then, once you've pitched your problem, how do you decide what to build? We have been challenged in the past with different modes of interaction. And how do I think about using this technology to solve my business problems? So what is this design process and how is it different when we're working in an augmented reality or virtual reality setting?

Vaibhav: Right, that's a great question. I think we have a live example to share our process on this. Recently we won an engagement with Abraham Lincoln Library in Illinois. And Shea could probably walk us through the process of how did they design the experiment, how did they come up with the ideas of what to build, and how are they processing, progressing on this.

Shea: Yeah, thanks, Vaibhav. I think it's a really important question. And I think we see XR and it's this brand new technology, and we assume that we need a lot of new processes and ways of working. But really the core way you decide what to build is based on the foundations that we've already established with decades of experience. And that's really understanding what our users need and how we can deliver a product that will deliver value and drive the needle. And so very simple things like user testing and research are the core foundation. So with the Abraham Lincoln Presidential Library and Museum, we went into the museum and worked with the visitors that were walking through to understand where we could augment the experience  — how can we take this new technology and apply it to the existing real world space to give them more information around the president's life both before the presidency and during his time there.

And so I think we really want to look at the customer experience and do that product research. With XR, a lot of companies might want to jump on it because it's flashy and exciting, but really it's just a tool in our tool kit. And we have to go through the same steps we would to build any application, to understand if it makes sense, and what aspects we want to leverage when we look at open space concepts, which is where the museum falls in. So that's augmenting the world around us to provide more information or that kind of wow factor. We also want to understand that people in museums might not want to be on their phone all the time. So how do we build that experience that makes people want to pull out their phone and dive deeper into a document that was written a hundred years ago? And I think we're also kind of understanding how can we bring in accessibility into the XR experience. Museums in general get a lot of folks who might not speak English as a first language, so how can we make the museum more relevant to them and leverage XR technologies, specifically AR, to bring documents and help translate them into another language, or for the hard of seeing, to bring in audio and use spatial audio as another form of augmented reality. 

And so these are all the questions we ask. And we ask them to the visitors. We conduct research, and we map out the customer journey. What does it look like from walking in the building to walking out through the gift shop — and what are those key points that will actually make a user want to pull out their phone. And it's pretty similar to the process we use all the time. So it's really falling back on our fundamentals.

Rebecca: I guess part of why I asked the question is, when I think about the design of a typical mobile app or web app, there are constraints on the flow. You can give them a Next button, and they can't get off that screen unless they either bail or hit the Next button. And so you have a level of control, whereas in this situation, these are people who are interacting with the physical world, and it's not like you can tell them don't move from this spot, because they can move from that spot! [CHUCKLES] And so how does how does that affect the way you think about the design of the application?

Shea: Yeah, I think that's a very fair question. And so, yeah, once you've decided that XR or AR is the tool you're going to use, how do we design it? Because that looks very different from our standard flow. I think it's a new concept for designers, developers, and content creators to move away from that 2D planar application or website that they normally work with and think about things in 3D space. You also have time. How long is an experience going to last? What does it look like? And so with this library, we've taken a few different approaches. There's some new tools available. There's something called Adobe Aero, which is a really new promising tool that allows you to do that prototyping for AR experiences, where you can actually put simple animations, and you can place things in a 3D space. And then with our team, we've also built out giant iPhones on clear laminate paper and literally held them up in front of exhibits to replicate — it's a cut-out of an iPhone and a clear plastic sheet, and we'll put objects in it and scale them by hand. And then in that situation, we are telling the people, you have to stand right here and look.

But as we progress in the design and get more confident in our concepts through concept testing like that, then it is a lot of collaboration between developers and designers and these 3D content creators to make small tweaks. And one of the first ideas we had was this idea around documents. And there's a ton of documents in cases. How do we let users actually understand the value and why these documents are being displayed?

In our first initial concept was this idea of a magic magnifying glass, where you can hold the phone up, and it'll translate the document out of cursive into English or any other languages. It'll read it aloud. It'll have a short animation on the document showing the historical significance. But one problem we ran into quite quickly into testing is, these cases are four by six or something. And the museum gets crowded at times. So putting that constraint on the user that they have to be directly in front of it with a clear view isn't going to work in a lot of situations. So how do we still provide that wow factor of like, I'm engaging directly with this document without limiting ourselves and maybe let the user pull back with the document and take it? Take it away with them and interact with it more on their phones directly but stepped away from the case. 

And these are just some of the things we have to deal with. In the museum itself, there's a lot of low lighting. So when we're looking at the differences in capabilities between Android devices and higher-end iOS devices, iOS and Apple devices are with LiDAR capabilities. Lighting isn't as important because it uses that infrared technology to be able to map 3D spaces. So the experiences pop up quickly, and they're really engaging. And the precision of where objects are placed is a lot better, where certain Android devices don't have that same technology, so in low lighting situations, it's really challenging to get the same level of fidelity and quality that you'd see on the iPhone. And this is particularly true in a lot of these scenes that they have throughout the museum, their dioramas, life-size figures taking place in the White House or throughout Lincoln's life. And one of our first ideas is how — we wanted to place objects throughout the space that users can interact with and learn more. They can learn about the significance of the scene and why it's there. But being able to precisely place these objects in 3D space has been a huge challenge. So how do we design around that? How do we make our designs not necessarily require pinpoint precision? Because we can't control what devices users are working on, but we can control the design and how we make the experience feel, understanding that pinpoint precision is not possible.

Rebecca: It's interesting you bring up documents. That's one of the things that I've always found difficult in situations like that is, OK, I'm looking at a piece of paper and I can't read it because it's blurred, and it's under glass and it's faded. And oh, by the way, it's written in French. [CHUCKLES] And I can't read French! I understand the value of those things. But as a museum visitor, that is one of the things that's very, very challenging, at least for me. 

Ken: So I'm curious, if I may, just thinking about unintended consequences. Do you take into account time spent in front of an exhibit? Like you mentioned, crowds are an issue there. So if the average time that somebody spent looking at a document was a minute and a half because it was in French, and now it's 12 minutes because they're delving in, that's awesome for their experience. Do you think about crowd flow? Is that something you take into account here?

Shea: Yeah, it's interesting you mentioned that. Because as we do with any product, we've been trying to define our key performance indicators or KPIs to understand. And one of the metrics we are looking at is time spent in the museum. And, we actually, our goal is to increase that in aggregate. But like you say, if we have these AR experiences that are causing crowd control issues, now we have a whole new set of problems. And so, yeah, it's definitely something we take into consideration. I think we, as a team, that would almost be a good problem for us to encounter because it means our app has been successful, and people are using it. So I wouldn't say we've spent a ton of time really delving into that quite yet, but yeah, I think we have to think not about just the AR experience, but also how people are in the room and interacting with it. And I think the way we can work around that is enabling the experience to be interactive in the maximum amount of spots. So if I'm looking at a document, we want to be able to pick up the image and let people interact with it at various different angles and not make it like there's one spot in the room that you have to stand at. And then letting people kind of step away and interact with it with more space so that they're not tied to one physical location for 12 minutes.

Rebecca: OK. So we've talked a lot about deciding what to do. Are there differences in the actual development process for creating these kinds of applications?

Shea: Yeah, I think when we get to the actual development of these experiences, it looks quite different. And a large part of that is because it now requires people with experience in 3D modeling and 3D animation. It's no longer just your standard developer generalist who can put out some CSS. And with all of these, there's many different technology options. You have Unity, which is a long-time tool used for 3D game development. And it's even used in film and cinema and stuff as well today. And so that skill set is going to look a little different than if you're maybe doing a pure native iOS using ARKit. That might be a little bit easier of a learning curve for someone that's spent years doing mobile development, but no matter what tool you choose, it's a completely different way of thinking. You're no longer thinking on that linear screen again. You need to have skills, working on x, y, and z axes. You need to have an understanding of meshes and how to apply them in animation. And so I think it brings in this new skill set that we're not used to working on with development teams, such as 3D animators and game designers. And I think historically, the culture between game designers and game developers and application developers has been different. And I think we're going to start to see some shift and compromise as these two kind of historically distinct groups of people come together and start working together to build out these experiences.

Rebecca: Which raises a question: we have people who are experienced in enterprise environments. And they get how challenging it is to interface with a mainframe and deal with legacy code. And these are the security processes that we need to do to protect our data. And there are all kinds of enterprise considerations that you don't necessarily think about being a factor in someone whose job has been to create a video game. Which direction do you think it's easier to go to take a game developer, who has been in that world of 3D animations and all of that different technology, and help them understand what it's like to interface with a kick system and fixed format files and all of the crusty things that we know exist in enterprises? Or to take somebody who understands that world and teach them what they need to know about thinking in three dimensions and dealing with these animations? Or is it equally hard?

Vaibhav: I think from our experience of working on certain other projects, like what we did for Eurovision, the problem was that our typical developers knew how to import a model to build a game or build an experience. They knew how to design path. What they didn't know was when you have a 32-gig model of a city, how does that work on a small mobile phone, and how does that give you a good experience? So the approach that we took on one of these engagements is, we got in somebody who had an experience of working on game designs because they knew the tricks of it, they knew the ways to compress model. They knew tricks around — if your digital avatar is only going to follow this path, then why don't you just, in a very layman term, why don't you just hardcore this path and everything else is blurred around it. How can you work around focus? So these ideas are very game development-driven ideas. But eventually, the quality of what we had to build was supposed to be at an enterprise level. So I think what we figured was the best way is a combination of two, wherein you bring in somebody, a person who has years of experience in game development, and then you have a team of the XR developers that we had homegrown and taught skills within Thoughtworks. And that turned out to be a winning combination for us because of how much of this repeat work we get. So that also becomes a bit of a question for us.

Ken: So then that leads to — Thoughtworks, we're big proponents of continuous integration, continuous delivery, fast feedback, all of those neat things. How do you test this? How do you put something that's going to live on goggles into a pipeline?

Vaibhav: So I think with years of experience of working with Lenovo, we've built our practices in this area. We've tried to figure out how do you do CI/CD for XR, how do you do testing at scale, how do you use practices that we use for traditional software development, when we do XR development. We've built an open source testing framework, called Arium, which is much like Selenium, built on similar philosophies, and that allows you to do testing as you build or as you write the software. Shea, do you have anything in addition to add on this?

Shea: Yeah, I can talk a little bit around our experience with the museum and some of the challenges we face. There are a lot of tools out there that do allow that continuous integration. But testing is always going to be more challenging, or at least, that's the way it feels today. This is still a fairly new technology. And I think we're still learning how to apply some of those best practices to it. But with the museum, many of our AR concepts are so physically intertwined with the physical space, that there is going to be some level of manual testing that we're never going to be able to avoid. And it looks a little different from our testing today. But because it really needs to involve the physical space. The developers and the designers placing things in physical space, there's only so much you can do remotely. And so we're looking at how do we apply our normal feature development and QA life cycles to this new AR world. And I think where we are today is, we can use some tools, we can still use unit tests, we can use things like Arium for some of that testing. But when the experience is in AR actions, rely on the physical space and need you to place things next to a couch that they have, that we can't replicate outside of the actual museum without building full scale models.

And so I think that's also probably an opportunity that's ripe for innovation. And I think it's something, as we continue on this project, that we'll be focusing a lot on. It's how do we test as much as we can using code and in a pipeline. I'm excited to explore that more as we actually get further into development.

Vaibhav: I know on one of the projects that we are working on, it's basically, we are trying to do gamified workouts for a client of ours. And the development team is here in India, the client is in Canada. It's practically impossible for them to ship on their devices, every device here. So what we are doing is, we're using an approach, where we are using simulators to then basically gauge what the experience may look like. And that's one way of testing these things out.

Rebecca: OK. Well, Vaibhav, at the beginning, you commented on the importance of tying these things to business goals. Can we talk a little bit about ROI for some of these projects, and what success looks like, and what the source of that return on investment might be? Shea mentioned a little bit about increasing engagement time and such. But in general, how do you think about ROI for projects like this?

Vaibhav: Great, that's a good question. That's I think that's something that we are asked often early in any engagement, in any discussion. How do you measure an ROI? And it's difficult because for most of these things, there is no historical data to benchmark what a good metrics is. Then of course, there are uncertainty in markets, the way, the rate at which technology is evolving, it may ask for high initial investment. It's difficult to see, like Shea mentioned, engagement — it's very qualitative. How do you put a number against it? So you are struggling between qualitative versus quantitative. There is also a learning curve because you're trying to do something early in its lifecycle. So there's a learning curve for everybody, the people who are experiencing it, people who are building it. So it becomes a tricky question to answer. But like I said, to begin with, the approach that we've taken is, almost all of our engagement started off as an experiment. So the approach was, what is the simple hypothesis that we want to validate with this? And when you break it down by, like, I gave an example of, you are a retailer and you want to think, you're looking at, can I sell properties to people who are not physically available to visit them? A good metric to start is, are people really interested in virtually viewing properties? That's a good — that's the top of the funnel metrics that tells you, is there interest? The next question comes in then, how do you incrementally start building things that allows people to finally convert the transaction? So it's an incremental process, is what we've realized. And that's how we've been educating our clients as well. Start with the top of the funnel matrix, and then gradually go towards how it is leading to final conversions. And we then, tweak in changes in the process as we go along. 

Rebecca: So nobody has a crystal ball, at least not one that works very well. Where is this all going? Do you see it expanding out of the universe of usual suspects for where this kind of user interaction might apply? Obviously, we're expecting the technology itself to continue to improve. But I'm really more interested in that sweet spot for this technology. Where do you see it going and what do you see being disrupted by more people taking advantage of this technology?

Vaibhav: I think at first to me, it looks like this technology is more driven by enterprises than by end users because of the cost of an option. That's one. Number two, I believe industries and businesses which derive value from the physical space, the premiumness, the personalization, exclusivity are better placed to make benefit from this technology than anybody else. That's my guess. And it's like museums, right? The whole idea is how do you pass on the ideological value, the historical context. This is where I think there is a good use case of this technology. And I think we're still a couple of years far from seeing the maximum potential of these technologies in these specific industries.

Shea: Yeah, I think from my perspective, we have seen there's the trainings for industry, and there's helping people on site with repairs have really proven their value. And I think the consumer space is still where we haven't necessarily seen a lot of traction or the strong value in AR specifically. And so at the museum, I like to think we're building the next generation of the audio tour guide, right? Because when you boil it down, that's essentially what we are building. But how do we take something that has worked for years and reinvent it to just add that little extra wow factor, and let people customize and personalize their experience to feel like their tour was unique to them and they learned something that wouldn't have been possible otherwise? And then I think one of the areas I will be watching closely, and not to sound too much like an Apple fanboy, but the Vision Pro. Right now, I think there's still a ton of questions around what Apple is going to do and can they disrupt the market. But looking back at the past, from the very first Macs and bringing icons to the desktop, to the way we all interact and use our phones today, they have disrupted design and interaction design many times. And so I'm really curious to follow that and see how interaction design with AR evolves with the Vision Pro and as more third party options come to market. Things like using your hands as opposed to actually touching things on your screen to interact with objects in the 3D space is going to be an interesting new way of interacting with these AR experiences.

Rebecca: I did get a chance to see a project at a university lab, where they were projecting into three dimensions a data set. And you analyze the data set by manipulating it, so, “oh, I want to focus in on this kind of data,” or “I want to aggregate this.” And different gestures in three dimensions actually manipulated the data and helped you understand. And I found that a fascinating concept. And when do we think the museum experience is going to go live?

Shea: Yeah, so we're still fairly early on in the museum, as a whole. We've just finished around 12 weeks of discovery. And we took that extra time because it's such a new experience and way of interacting with museums that we felt that it warranted a longer discovery than would be typical. And we're hoping to kick off the actual delivery and implementation phase at the end of this month, August or early September, with our first initial beta following around probably like five or six months after that. So hopefully, by early 2024, we should have a working data that validates this concept and lets users interact with the tour guide of the future.

Rebecca: Excellent. Well, again, thank you both for joining us. And thank you all listeners for listening in to another edition of the Thoughtworks Technology Podcast. And thank you, Ken.

Vaibhav: Thank you.

Shea: Thanks for having me.

[MUSIC PLAYING]