ChatGPT: A useful tool buried beneath the hype

There’s been a lot of buzz about ChatGPT in the last few months; it’s one of those technology stories that has grabbed the attention of mainstream culture. However, at a time when the troubled macroeconomic climate is putting pressure on organizations in just about every industry, it’s not entirely clear whether the attention given to ChatGPT — and other AI-assisted tools like GitHub Copilot — is an amusing distraction or a source of hope that it really can help us achieve larger goals faster, and incredible things even at a time of serious pressure. 

The truth is probably somewhere between the two. With that in mind, it’s important for business leaders to balance ambition with realism when it comes to ChatGPT. Yes, it can be fun and immensely productive when it comes to new ideas, but it nevertheless has many limitations and certainly won’t transform your organization on its own. The best way to understand this is to look at some key applications of ChatGPT — which have already been well-documented — and explore where it could help, where it won’t and where it might even hamper you.

Search and Research

One of the most interesting applications of ChatGPT is in its ability to aid information retrieval. It has been described, for example, as the Google Killer, with some suggesting that it could redefine how we understand search: essentially removing the middleman of results pages to provide an answer immediately. While the real extent of ChatGPT’s threat to search is very much open to debate, in certain contexts the speed at which ChatGPT can output responses means that it provides users with a more conversational method of uncovering information.

For example, in a research context, it could be used to quickly describe certain topics and offer further avenues for exploration. That’s useful for people in a range of domains. For strategic decision-makers it could provide a way of exploring ideas and concepts as a kind of sounding board. From a communications or product perspective, meanwhile, it could prompt new ideas about how to describe or talk about something.

Content and communication

This leads nicely into content generation and communication. Much has been made of ChatGPT’s ability to create content; in fact, we even asked it to create a blog post about itself — take a look below. At first glance, the results seem impressive, demonstrating that ChatGPT can ably play a supporting role when it comes to composition.

However, if you look closely, some of its limitations are clear. It can sometimes be repetitive, for instance; the discussion of chatbots and customer service, for example, is essentially the same thing. (It is, however, possible to mitigate this using a tunable parameter called “temperature” — increasing it essentially makes ChatGPT more unpredictable and “creative” in its output.) Sometimes it makes ostensibly convincing but dubious claims — it might help businesses save time and resources, but whether it helps them “adapt to changes in market trends or consumer preferences” is another matter. The model, after all, is intrinsically backwards looking; it’s trained on data that already exists. Yes, it can be trained on new data, but it’s always playing catch up.

Does this mean that ChatGPT is useless here? Not quite: our example demonstrates how it can help put in place the basic elements of a piece of writing. It removes the problem that every writer faces: turning a blank page into a first draft. 

However, wherever ChatGPT is used for content generation, it will need to be monitored and supported. Whether that’s worthwhile depends on what type of content and communication you’re using it for; indeed, given how convincing its outputs can be, being able to cast a critical eye over ChatGPT outputs will likely become a very valuable skill.

Code generation

The same factors apply to using ChatGPT to generate code. It can be useful in proposing potential solutions if given carefully crafted prompts, but the idea that it can solve a complex problem on its own ignores fundamental limitations of the technology.  It will not be any ‘smarter’ than a human would if given vague or conflicting requirements from which to generate its solution. We have used it to quickly produce blocks of compilable code, complete with library imports and compiler switches, and the code runs and (often) does what we asked.

ChatGPT and similar tools like CoPilot might be viewed as software development efficiency tools which can speed time to market and lower the resources required to complete a set of defined functionality. To a certain extent, this isn’t a million miles away from the kind of efficiency touted by low- and no-code platforms. However, there are some important distinctions — where ChatGPT can provide code in seconds, a low code tool requires a human user to make a number of decisions and, importantly, to understand how the low-code platform works. Both have the capacity to change the way we create software, but they do so in very different ways. 

One particular advantage of ChatGPT is that it can explain code — at least to a certain extent. It would be more accurate to say that it can present patterns that occur in the text on which it has been trained; it can show you how something is working in a way that might otherwise have passed you by. This means that while we should refrain from attributing sophisticated understanding, its ability to do things like translate code between programming languages, and draw upon its seemingly encyclopedic knowledge of libraries and APIs could make it a very useful programming companion in some contexts. As a programming companion it can be incredibly useful, not least because it’s happy to answer questions all day long without judging someone or making them feel inadequate.

We wouldn't, however, call it a “pair programmer” in robot form quite yet — as compelling as such a thing would be.

Customer interactions

The final area where ChatGPT could have an impact is in further evolving customer interactions. Chatbots have been around for a good few years now, but the power of ChatGPT could help usher in a new generation of higher quality chatbots. There are, however, some important caveats to remember here: consider, for example, exactly what you’re trying to achieve when you introduce chatbots into your interactions. Clearly businesses would like to streamline the ways they handle customer queries, but it’s important to remember that seeing ChatGPT as a silver bullet solution is risky. 

This is for multiple reasons. First, it directs your attention from the root of customer support challenges. If things inside your product aren’t working, or if a user journey is messy and unclear, a capable chatbot isn’t going to fix those issues — it might even hide them from view. Second, given the fact tools like ChatGPT always have the potential to offer inaccurate and fallacious information, it could make what was initially a minor issue much more problematic.

Finally, it’s important to remember that ChatGPT is only going to be able to provide responses according to the data on which it has been trained. If you haven’t produced the information needed or if it’s contained within a poorly developed information architecture, a chatbot isn’t going to be particularly effective however sophisticated it is.

The lesson here is that to use ChatGPT — or any AI system for that matter — you need to get the basics right and put some foundations in place.

Take it seriously… but not too seriously

There’s a lot of uncertainty around at the moment, but you can count on two things: the hype around ChatGPT (and generative AI more broadly) will continue and the tool itself will get better. Indeed, it will probably get better faster, given the millions of users that have interacted with it in the last few months. News that OpenAI is set to release a premium version of the product is a clear indicator of the direction of travel, but attention should really be paid to the products and services that build upon ChatGPT’s core functionality. While it looks a lot like a toy at the moment, in time we’ll probably see more domain-specific tools geared towards the needs of very specific types of organizations and users.

Before we start seeing those, it’s important to ensure that you don’t let the ChatGPT buzz redefine how you work. The way forward in the months to come is to treat it as version one of a step or order of magnitude change in capability. Don’t take it too seriously — experiment with it and see where it can help and where it won’t. But don’t overlook it either: engaging with it is a great way to prepare for the next generation of AI tooling in the industry.