Forget Skynet: one of the biggest risks of AI is your organization’s reputation. That means it’s time to put science-fiction catastrophizing to one side and begin thinking seriously about what AI actually means for us in our day-to-day work.
This isn’t to advocate for navel-gazing at the expense of the bigger picture: it’s to urge technologists and business leaders to recognize that if we’re to address the risks of AI as an industry — maybe even as a society — we need to closely consider its immediate implications and outcomes. If we fail to do that, taking action will be practically impossible.
Risk is all about context
Risk is all about context. In fact, one of the biggest risks is failing to acknowledge or understand your context: that’s why you need to begin there when evaluating risk.
This is particularly important in terms of reputation. Think, for instance, about your customers and their expectations. How might they feel about interacting with an AI chatbot? How damaging might it be to provide them with false or misleading information? Maybe minor customer inconvenience is something you can handle, but what if it has a significant health or financial impact?
Even if implementing AI seems to make sense, there are clearly some downstream reputation risks that need to be considered. We’ve spent years talking about the importance of user experience and being customer-focused: while AI might help us here, it could also undermine those things as well.
There’s a similar question to be asked, about your teams. AI may have the capacity to drive efficiency and make people’s work easier, but used in the wrong way it could seriously disrupt existing ways of working. The industry is talking a lot about developer experience recently — it’s something I've written about before — the decisions organizations make about AI need to improve the experiences of teams, not undermine them.
In the latest edition of the Thoughtworks Technology Radar — a biannual snapshot of the software industry based on our experiences working with clients around the world — we talk about precisely this point. We call out AI team assistants as one of the most exciting emerging areas in software engineering, but note that the focus has to be on enabling teams, not individuals. “You should be looking for ways to create AI team assistants to help create the ‘10x team,’ as opposed to a bunch of siloed AI-assisted 10x engineers,” we say in the latest report.
Failing to heed the working context of your teams could certainly cause significant reputational damage. Some bullish organizations might see this as part and parcel of innovation, but it’s not. It’s showing potential employees, particularly highly technical ones, that you don’t really understand or care about the work they do.
Managing risk requires real attention to the specifics of technology implementation.
Managing risk requires real attention to the specifics of technology implementation.
Tackling risk through smarter technology implementation
There are lots of tools that can be used to help manage risk. Thoughtworks helped put together the Responsible Technology Playbook, a collection of tools and techniques that organizations can use to make more responsible decisions about technology (not just AI)
However, it’s important to note that managing risk — particularly those around reputation — requires real attention to the specifics of technology implementation. This was particularly clear in work we did with an assortment of Indian civil society organizations, developing a social welfare chatbot that citizens can interact with in their native language. The risks here were not unlike those discussed earlier: the context in which the chatbot was being used (as support for accessing vital services) meant that inaccurate or ‘hallucinated’ information could stop people from getting the resources they depend on.
This contextual awareness informed technology decisions. We implemented a version of something called retrieval augmented generation to reduce the risk of hallucinations and improve the accuracy of the model the chatbot was running on.
Retrieval augmented features on the latest edition of the Technology Radar: it might be viewed as part of a wave of emerging techniques and tools in this space that are helping developers to tackle some of the risks of AI. These range from something called NeMo Guardrails — an open-source tool that puts limits on chatbots to increase accuracy — to the technique of running large language models (LLMs) locally with tools like Ollama, to ensure privacy and avoid sharing data with third parties. This wave also includes tools that aim to improve transparency in LLMs (which are notoriously opaque), like Langfuse.
Indeed, it’s worth pointing out that it’s not just a question of what you implement, but also what you avoid doing. That’s why, in this Radar, we caution readers about the dangers of overenthusiastic LLM use and rushing to fine-tune LLMs.
Rethinking risk
There is, of course, a new wave of AI risk assessment frameworks. There is also legislation too (including a new law in Europe) which organizations must pay attention to. But addressing AI risk isn’t just a question of applying a framework or even following a static set of good practices. In a dynamic and changing environment, it’s about being open-minded and adaptive, paying close attention to the ways that technology choices shape human actions and social outcomes on both a micro and macro scale.
One useful framework is Dominique Shelton Leipzig’s traffic light framework. A red light signals something prohibited — such as discriminatory surveillance — while a green light signals low risk and a yellow light signals caution. I like the fact it’s so lightweight: for practitioners, too much legalese or documentation can make it hard to translate risk to action.
However, I think it’s worth flipping it and see the risks as embedded in contexts, not in the technologies themselves. That way, you’re not trying to make a solution adapt to a given situation, you’re responding to a situation and addressing it as it actually exists.
If organizations take that approach to AI — and, indeed, technology in general — that will ensure they’re meeting the needs of stakeholders and keep their reputations safe.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
