Ever since I first heard about ChatGPT — which wasn’t that long ago — it’s been impossible to avoid claims about its revolutionary potential in which it will change the way we work forever. Among many things, I was told that it's a completely different way of finding information and that it can even be recognized as the only source of knowledge in the foreseeable future. I also heard that it can code.
I decided to check it out and, filled with hope, I wanted to see how it performs as a programming support tool — perhaps, I thought, it might even be something more. I wanted to set up a pair programming session with it, wondering if it would work. I wanted to treat it as if it were someone I spend a few hours a day writing code with. Without much thought, I decided I would work on two example use cases with it that were roughly what we might expect when implementing a typical new feature. The two use cases were of varying complexity and both use C# as the programming language. While ChatGPT was able to write reasonably complex code, my session with it lacked the mutuality and co-creation that I get from my best pair programming experiences. To maintain the authenticity of the interaction, I decided to keep the conversation intact and added my own comments from the session, which took place using a version of ChatGPT from Jan 30.
Use Case 1: Simple domain model
In this scenario, I wanted to test ChatGPT's ability to model a basic domain concept and assess its response to requirement changes. I also wanted to evaluate the effectiveness of our collaboration.
I started giving the goal and the requirements. After a few seconds I got the proposed solution. Pretty impressive.
R: Can you use C# to model a storage box?
Here are the requirements:
Storage box can contain only strings.
Consumers can add strings to the box.
Box has limited capacity and cannot contain more than 10 strings and
total length of all strings should not exceed 200 characters.
We should end up having properly isolated domain classes.
ChatGPT: Yes, it is possible to use C# to model a storage box. Here is an
example implementation:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class StorageBox
{
private List<string> _strings = new List<string>();
private readonly int _capacity = 10;
private readonly int _lengthLimit = 200;
public void AddString(string input)
{
if (_strings.Count == _capacity)
{
throw new Exception("The storage box is full.");
}
if (GetTotalLength() + input.Length > _lengthLimit)
{
throw new Exception("The total length of the strings in the box has exceeded the limit.");
}
_strings.Add(input);
}
public int GetTotalLength()
{
int totalLength = 0;
foreach (string s in _strings)
{
totalLength += s.Length;
}
return totalLength;
}
}
}The example implementation was pretty accurate. We could argue that using constants instead of read only fields, favoring properties over methods or choosing a more functional approach in this example would be a better choice. Anyway, this was a very good starting point.
Then, I wanted to add some additional requirements to see how it would react.
R: The strings that we put into the box should not be longer than 100
characters and should not contain the "x" character.
ChatGPT: Here's an updated version of the `StorageBox` class with the
additional requirements:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class StorageBox
{
private List<string> _strings = new List<string>();
private readonly int _capacity = 10;
private readonly int _lengthLimit = 200;
private readonly int _stringLengthLimit = 100;
public void AddString(string input)
{
if (input.Length > _stringLengthLimit)
{
throw new Exception("The input string is too long.");
}
if (input.Contains("x"))
{
throw new Exception("The input string contains an 'x' character.");
}
if (_strings.Count == _capacity)
{
throw new Exception("The storage box is full.");
}
if (GetTotalLength() + input.Length > _lengthLimit)
{
throw new Exception("The total length of the strings in the box has exceeded the limit.");
}
_strings.Add(input);
}
// Note: Implementation of 'GetTotalLength' has been removed for brevity.
public int GetTotalLength() {...}
}
}The updated code was functional even though I didn’t like the growing validation section in the “AddString” method. I wanted to check whether it could suggest something to deal with it.
I added new requirements to see if it could propose an abstraction to avoid duplicating validation logic.
R: Let's also add a new method to replace the last string in the box with
a new one.
ChatGPT: Here's an updated version of the `StorageBox` class with the
new `ReplaceLastString` method:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class StorageBox
{
private List<string> _strings = new List<string>();
private readonly int _capacity = 10;
private readonly int _lengthLimit = 200;
private readonly int _stringLengthLimit = 100;
// Note: Implementation of 'AddString' has been removed for brevity.
public void AddString(string input) {...}
public void ReplaceLastString(string input)
{
if (input.Length > _stringLengthLimit)
{
throw new Exception("The input string is too long.");
}
if (input.Contains("x"))
{
throw new Exception("The input string contains an 'x' character.");
}
if (_strings.Count == 0)
{
throw new Exception("The storage box is empty.");
}
_strings[_strings.Count - 1] = input;
}
// Note: Imlementation of 'GetTotalLength' has been removed for brevity.
public int GetTotalLength() {...}
}
}The result was a bit disappointing. It simply added duplicate validation logic to the second method. It didn’t spot the abstraction here. I was curious how it ‘felt’ about the code it wrote though; I asked it to do a review of its work.
R: Can you review the code you wrote?
ChatGPT: Yes, here is a review of the code:
The `StorageBox` class implements the storage box model that contains a list of strings and enforces the constraints of maximum capacity, total length, and string length.
The `AddString` method adds a string to the storage box if it meets the constraints. If not, it throws an exception with an appropriate error message.
The `ReplaceLastString` method replaces the last string in the box with a new string if it meets the constraints. If the box is empty, it throws an exception.
The `GetTotalLength` method calculates the total length of all strings in the box.
The code is well-structured and easy to understand. The constraints are
properly enforced and the error messages are clear. The code can be
improved by adding some additional input validation to ensure that the
inputs are not null or empty. Additionally, the code can be made more
readable by breaking down some of the complex conditions in the
`AddString` and `ReplaceLastString` methods into separate methods.
Okay, it appeared that the validation rules were somewhat too complex. However, I didn’t agree with its proposed solution of separating the logic into distinct methods — there are other, better options. Nonetheless, I chose to adopt its approach to shorten the code.
R: How can we simplify this?
ChatGPT: One way to simplify this code is to extract the input validation logic
into separate methods to make the main methods cleaner and easier to
understand. Here's an updated version of the code:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class StorageBox
{
private List<string> _strings = new List<string>();
private readonly int _capacity = 10;
private readonly int _lengthLimit = 200;
private readonly int _stringLengthLimit = 100;
public void AddString(string input)
{
ValidateInput(input);
ValidateCapacity();
ValidateLengthLimit(input);
_strings.Add(input);
}
public void ReplaceLastString(string input)
{
ValidateInput(input);
ValidateEmptyBox();
_strings[_strings.Count - 1] = input;
}
// Note: Implementation of 'GetTotalLength' has been removed for brevity.
public int GetTotalLength() {...}
private void ValidateInput(string input)
{
if (input == null)
{
throw new Exception("The input string is null.");
}
if (input.Length == 0)
{
throw new Exception("The input string is empty.");
}
if (input.Length > _stringLengthLimit)
{
throw new Exception("The input string is too long.");
}
if (input.Contains("x"))
{
throw new Exception("The input string contains an 'x' character.");
}
}
private void ValidateCapacity()
{
if (_strings.Count == _capacity)
{
throw new Exception("The storage box is full.");
}
}
private void ValidateLengthLimit(string input)
{
if (GetTotalLength() + input.Length > _lengthLimit)
{
throw new Exception("The total length of the strings in the box has exceeded the limit.");
}
}
private void ValidateEmptyBox()
{
if (_strings.Count == 0)
{
throw new Exception("The storage box is empty.");
}
}
}
}I wanted to make the missing abstraction more obvious so I added a new requirement to show that keeping validation rules in separate methods wouldn’t help us here.
R: Let's add one more concept. A shelf on which we can put strings. At the
moment, there are no restrictions on the capacity of the shelf. The same
validation rules should apply for strings. Let's start with one method of
arranging the strings on the shelf.
ChatGPT: Sure! Here's an updated version of the code with the added
`Shelf` class:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class Shelf
{
private List<string> _strings = new List<string>();
private readonly int _stringLengthLimit = 100;
public void AddString(string input)
{
ValidateInput(input);
_strings.Add(input);
}
private void ValidateInput(string input)
{
if (input == null)
{
throw new Exception("The input string is null.");
}
if (input.Length == 0)
{
throw new Exception("The input string is empty.");
}
if (input.Length > _stringLengthLimit)
{
throw new Exception("The input string is too long.");
}
if (input.Contains("x"))
{
throw new Exception("The input string contains an 'x' character.");
}
}
}
// Note: Implementation of 'StorageBox' has been removed for brevity.
public class StorageBox {...}
}I wasn't satisfied with the revised implementation. Despite specifying that we were dealing with the same strings, ChatGPT failed to abstract it, leading to duplicate logic. I would have also hoped for it to pose questions to clarify important design details. Nonetheless, I decided to request another review from it.
R: Can you review the code you wrote?
ChatGPT: Sure, let me review the code I wrote:
The code is well organized and easy to understand.
The `Shelf` and `StorageBox` classes are properly isolated and don't depend on each other.
The validation rules for the strings are implemented in a single `ValidateInput` method, which makes the code more maintainable.
The code has proper error handling with meaningful error messages.
Overall, the code looks good to me!
The outcome was quite disappointing. It seemed that for ChatGPT, identifying patterns and structure might be a challenge. I offered a specific idea on how to simplify the solution.
R: It could be improved by abstracting out the concept of an Item that we
reuse in both classes. In that case you won't need to duplicate the
validation logic. Moreover, making the Item class immutable and invoking
the validation logic during instantiation would protect us from putting the
instance in an invalid state. What do you think?
ChatGPT: Yes, that's a great suggestion! By abstracting out the concept of
an `Item`, we can reduce code duplication and improve the overall design
of the code. Here's an updated version of the code with the `Item` class:
using System;
using System.Collections.Generic;
namespace StorageBox
{
public class Item
{
private readonly string _value;
private readonly int _stringLengthLimit = 100;
public Item(string value)
{
if (value == null)
{
throw new Exception("The input string is null.");
}
if (value.Length == 0)
{
throw new Exception("The input string is empty.");
}
if (value.Length > _stringLengthLimit)
{
throw new Exception("The input string is too long.");
}
if (value.Contains("x"))
{
throw new Exception("The input string contains an 'x' character.");
}
_value = value;
}
public string GetValue()
{
return _value;
}
}
public class Shelf
{
private List<Item> _items = new List<Item>();
public void AddItem(Item item)
{
_items.Add(item);
}
}
public class StorageBox
{
private List<Item> _items = new List<Item>();
private readonly int _capacity = 10;
private readonly int _lengthLimit = 200;
public void AddItem(Item item)
{
ValidateCapacity();
ValidateLengthLimit(item.GetValue().Length);
_items.Add(item);
}
public void ReplaceLastItem(Item item)
{
ValidateEmptyBox();
_items[_items.Count - 1] = item;
}
// Note: Implementation of the other methods has been removed for brevity.
public int GetTotalLength() {...}
private void ValidateCapacity() {...}
private void ValidateLengthLimit(int inputLength) {...}
private void ValidateEmptyBox() {...}
}
}I attempted to introduce some additional enhancements to the implementation, such as adding a factory method to the "Item" class or replacing exceptions with status codes, but I faced some difficulties in communicating these suggestions effectively to ChatGPT. As a result, the implementation became messy, and I ultimately decided to stop at that point as I felt it would be easier to do it on my own.
Use case 1: Summary
On one hand, I was impressed with the initial prototype ChatGPT was able to create from my basic requirements. However, I was disappointed with the limited interaction and collaboration it offered. It was a one-sided interaction with me dictating what I wanted and ChatGPT coding accordingly, without any real discussion or exchange of ideas. As the solution became more complex, communicating with it became difficult and I had to be very specific with my instructions. This was not a very efficient process, and I felt I could have completed the task faster on my own. Despite the simplicity of the task and the absence of dependencies, ChatGPT also did not provide any valuable insights on how to simplify the solution by abstracting things out.
In the second part of the article, I will test how effective ChatGPT is when dealing with a more complex scenario. Finally, I will share my overall thoughts on the experiment.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.
