Building the world’s first AI-powered security concierge
Reimagining what’s possible in home security
Swann Security (Swann) is a leading global provider of security monitoring, consumer electronics and security-centric solutions, empowering people to protect and stay connected to their home or business no matter where life takes them. Swann saw an opportunity to address the challenge of keeping homes safe, even when homeowners are away or when they aren’t able to respond to alerts. Swann wanted to bridge this gap by pushing the boundaries of artificial intelligence (AI) and approached Thoughtworks with an ambitious idea: to develop the world’s first AI security concierge.
A secure home even when you’re away
Swann sought to leverage the power of generative AI (GenAI) to develop a doorbell agent that can engage with visitors in real-time and interact naturally across different scenarios, including potential security incidents.
To make this happen, Swann needed to test the feasibility of this idea, and to understand its limitations, risks and safety guardrails before developing and productionizing this powerful technology.
Together we embarked on a journey to develop a functional prototype for an AI security concierge. Thoughtworks became a partner of choice due to our unique approach to rapidly design and deliver exceptional solutions that leverage data and AI.
We wanted to see if it was possible to use GenAI on our video doorbell and security cameras to interact with visitors. With the help of Thoughtworks we were able to quickly work through what was possible and set a list of objectives that we wanted to achieve over a fast-tracked development program.
Reinventing home security systems with AI
In order to move quickly and understand how to tackle the opportunity, we ran a series of discovery workshops to define the core problem, map common user journeys and identify the key use cases. We then narrowed the scope to the first “thin slice” - a piece of a working solution - to deliver on.
The solution we designed has a chat user interface (UI) to transcribe spoken input, generate an AI response, and play back that response in audio. For fast prototyping we used Streamlit to build the prototype UI application and leveraged AI capabilities on Amazon Web Services (AWS).
The fully functional AI concierge system can independently handle different scenarios by tailoring its response to interact with people in a natural way. For example, it can greet guests, instruct delivery people on where to safely leave a package and defend against intrusive requests.
To do this we developed a prompt engineering strategy to guide the AI security concierge in effectively managing conversational exchanges across various scenarios. We identified the many common user intents, as well as categories to manage unanticipated interactions and provide a safeguard for the AI to maintain a courteous dismissal.
To ensure that security and privacy was built into the prompt engineering framework, we conducted initial testing on the AI’s response accuracy and evaluated the system’s resilience to defend against adversarial attacks. We were able to demonstrate the AI’s capacity to safeguard against unanticipated interactions and adversarial requests.
We also refined the prompt structure to distinct sections which not only optimized the AI’s operational efficacy, but also facilitated easier updates and future scalability. The specialized AI concierge is built with a customizable framework to accommodate household-specific configurations and to allow Swann to personalize settings, like defining the AI’s personality.
Extraordinary impact delivered
The working prototype was showcased at the Consumer Technology Association (CES) conference in Las Vegas where it was named an Honoree in the Smart Home category. The annual awards program recognizes outstanding design and engineering in technology products for consumers. After the success at CES conference, Swann is on the path to productionizing the world’s first AI-enabled security concierge and bringing it to customers this year. It marks a new era for Swann, laying down the foundation to build its future range of AI-powered products.
We designed and developed a functional AI prototype of the security concierge over a fast-tracked program.
Swann’s AI-driven security concierge was named an Honoree at the CES 2024 Innovation Awards.
Swann is on the path to productionizing SwannShield™, bringing the world’s first AI-enabled security concierge to customers.
Thoughtworks and Swann embarked on a journey of innovation and problem-solving to ensure that SwannShield™ can effectively respond to a wide range of scenarios, from deliveries, to intruders, to Halloween ‘Trick or Treaters’. By collaborating closely and leveraging our collective expertise, Swann and Thoughtworks overcame challenges to develop a solution that prioritizes both security and user experience.
