eXtending reality with AR and VR - Part II

In Part I of the article, we summarized our observations of eXtended reality’s foundational blocks - AR, VR and MR. We also discussed the popular and most relevant tools and implementations of the new technology. In Part II of the article, we will be looking at a few accessible and novel implementations of the tech.

At Thoughtworks, we have delivered both XR solutions and components of XR SDK (Software Development Kit) on the back of which XR solutions are built. We have explored Unity 3D technology beyond the gaming bucket. Infact, Thoughtworks’ Technology Radar states, “...Unity has become the platform of choice for VR and AR application development because it provides the abstractions and tooling of a mature platform...Yet, we feel that many teams, especially those without deep experience in building games, will benefit from using an abstraction such as Unity... it offers a solution for the huge number of devices.”

We have also built an XR incubation center that lets us experiment with XR experiments while collaborating on agile development practices like TDD and CI/CD for XR. All this work with XR tech has helped us understand the space that much better. And, I have taken this opportunity to round up some really interesting use cases for the tech that will be ‘good-to-know’ for business leaders who want to explore the tech’s potential.

Use case 1: training and maintenance

XR tech has a direct impact on the improved efficiency and effectiveness of training, and maintenance and repair use cases. 

VR tech can generate and simulate environments that are either hard to reproduce or involve huge costs and/or risk. Examples of such scenarios are factory training sessions or those on fire safety or ‘piloting’ an aircraft.

AR, on the other hand, is a great way to provide technicians with contextual support in a work environment. For instance, a technician wearing a pair of AR-enabled smart glasses can view digital content that’s superimposed on their work environment. This could be in the form of step-by-step instructions for a task or to sift through relevant manuals and videos while they are on the assembly line/or on field performing a task or it could be used to issue a voice command. The AR device is enabled to also take a snapshot of the work environment for compliance checks. Additional features include the technician being able to stream the experience to a remote expert, who can annotate the live environment and review the work.

Here’s an interesting case study of how Thoughtworks is leveraging XR tech to transform airline operations.

Use case 2: locate and map

Autonomous vehicles leverage techniques like Simultaneous Localization and Mapping (SLAM) to build virtual maps and locate objects in an environment. Similar technology empowers XR-enabled devices to detect unique environments and track device positions within.

This Virtual Positioning System can help build indoor positioning applications for large premises like airports, train stations, warehouses, factories and more. Such a system also allows users to mark, locate and navigate to specific target points. 

At Thoughtworks, we recently implemented a concept for indoor positioning at our offices. It’s a mobile app called Office Explorer, and when used with an AR-enabled smartphone or tablet, configures the entire office on a 2D map that visitors can use to explore the space.

 

AR enabled Office Explorer app in use at the Thoughtworks office

Visitors or new Thoughtworkers can follow the augmented path on the app to reach meeting rooms and work stations. This app was built using Unity 3D, ARCore and ARKit. The backend services and web portal were built on React, Kotlin, MongoDB and Micronaut.   

Use case 3: product customization and demonstration

Virtual, CAD models or paper-drawn prototypes usually precede final physical products. What’s more, the former also allows for affordable customizations in an XR environment. Consumers have an opportunity to try 3D models of products before they buy it. 

Below is a quick concept that was created at Thoughtworks, where one could customize a particular virtual model of a vehicle with respect to colour, interiors and fabrics. They could also step inside the virtual car and experience the personalizations, evaluate how their car looks and fits in their garage or in front of the house.

 

Thoughtworkers experimenting with AR-enabled customization of a vehicle

Our app used Unity 3D and ARCore to detect the plane surface and place the product’s 3D model before commencing customizations. The same technology can be used to build an entire driving experience - a great sales tool! An additional benefit of this approach is one of the most popular features when demonstrating a product; the complete 360° visualization of it. 

Here is how Thoughtworks helped Daimler build innovative tools to support the latters’ futuristic sales experience. 

Use case 4: contextual experiences

Gartner states that 46% of global retailers planned to deploy either AR or VR solutions to meet customer service experience requirements. The global research and advisory firm also foretells that no less than 100 million consumers will use AR to shop both online and in-store in the future.

Businesses are being disrupted by both the sheer amount of data they have at their disposal and the way they engage with that date. This data is in the form of buying history, product reviews, product-health, recommendations, usages statistics, features level analysis, comparative studies, social sentiment analysis for the product and more. 

A lot of customers look to such data as they make decisions they can trust and be confident in. This gives businesses to present (hopefully unbiased) data within the right context, empowering consumers to make their decisions quickly. 

Thoughtworks’ TellMeMore app displays relevant data for a book 

An implementation called TellMeMore is an app that we designed at Thoughtworks. It aims at providing a better in-store product experience by overlaying information like book reviews, number of pages and comparative prices on/beside the product. This app was created using Vuforia and Unity3D. 

This is illustrative of opportunities to enhance customer experiences to the extent that without opening a product, customers have access to product insights and even experience detailed 3D models of the same. Add to this, the option of amplifying seemingly boring static objects such as event posters or product advertisements when viewed through an AR companion app. 

Thoughtworks XR day’s paper poster turned into an AR experience:

Use case 5: customer engagement

2016’s fad, Pokémon GO is perhaps AR’s most famous champion. Historically, gamification and VR have proven to better engage users. Today, shopping malls and business conferences leverage VR experience studios to engage kids and customers alike. Today, we have kids riding a roller coaster without even sitting on the actual structure.

Thoughtworkers have implemented a smartphone-based AR concept called LookARound that can involve visitors at expos and events. The flow of activity is this - 

• Users scan a visual cue (like a QR code) within the event area
• Users have to find a 3D goodie that's hidden close to the visual cue. It will be superimposed (Pokémon GO style) on the 3D environment around the user. 
• The app appraises users of the distance between them and the 3D goodie 
• Users can collect the goodie and compete with other players.

LookARound was a huge hit at a two day internal office-wide event, ‘India Away Day 2019.’ 1700 Thoughtworkers explored different points of interests at the venue, such as break-out sessions, tech stalls and event areas while engaging with AR.

Snapshot of the LookAround App at Thoughtworks India Away Day 2019

At the Away Day, 500 Thoughtworkers competed with each other at a time. They were encouraged by 3D leaderboards showcased on Microsoft Hololens and Oculus Quest.

XR tech is evolving at such a rate that the untapped potential boggles the mind. And, we believe the software’s evolution should be matched by the hardware. The latter needs to be user-friendly in terms of field of view and lightweight with immense processing and battery power. 

The software on the other hand, needs to take a more informed advantage of AI and ML to better estimate diverse environments, making it accessible to any and every possible scenario. Businesses also need to proactively build as many use cases as possible, fix shortcomings and contribute to XR development practices. VR is a reality and has us primed for the pronounced impact of AR and MR tech in the future.