Production Support - What Lies Beneath

The word “technical support” has a slightly negative connotation in the developer community. Most think of it as an arduous drag involving looking through piles of logs and stack traces, and taking random guesses. And that was exactly what I believed before actually experiencing it myself in a project. It taught me a lot and changed my perspective towards software development and collaboration with other teams.

At Thoughtworks, we have been developing a global web platform (CMS, localization, and other content creation tools) in collaboration with a dozen other vendors. For production support, instead of having dedicated people in the role, various team members took turns to own the support role. In November 2014, I traveled to Istanbul to provide production support for this platform. Since it is multi vendor team, we have to work with representatives from each vendor team involved to provide production support.

As a newbie to support work, I was nervous about the various scenarios and difficulties I expected to face. However, throughout my time in support, I found the whole experience interesting and educational. There were a handful of stressful situations but overall it was pretty smooth.

Here is what stood out for me from the experience:

Empathy and trust

The most important lesson was probably that everything is not as straight-forward for everyone as we seem to think. We first need to accept that everybody is giving their 100%, despite their challenges, and is co-operating in the best way possible.

Communication

You should keep calm and and not rush to any conclusion around the cause of issue. First confirm that your own understanding of the issue and solution is clear, before communicating it to the clients. False communication will lead to more confusion and it will take more time for the issue to be resolved. Clear and proper communication brings everyone at same level with clear understanding about the issue and its root cause. It ensures that everyone takes the necessary steps and removes any back and forth.

Debugging

I admit that in support work you don’t get a chance to develop any new features but what it does give you is a chance to debug the existing application code. Every time you do so, you think of different ways to improve your code. It’s not enough to write correct logic because it will only work when it receives the expected data input. In projects like ours, where the app is driven by interconnected data models coming from different parties, it is easy to make mistakes at different levels. It’s also not always possible to write data validations at every level of data entry. So it is likely to happen that your production issue will be a data issue, and not a code issue.

The second lesson I learned under debugging was that your code should properly log various cases, along with sufficient temporal context. I think your code quality is not only defined by how well-written your code is or how you follow programming standards or which languages and framework features you have used, but also how quickly you can debug your code. In production, you have only application logs to look into. Logs are surely important but that doesn’t mean you add logs everywhere. You should put logs with the useful exacted data and only at the specific events. We use structure logging in our project, and I have found it to be very beneficial when dealing with huge application log files, especially in conjunction with tools like Splunk.

Improvization + Innovation = Profit!

Did I say that in support you don’t get an opportunity to develop? Well, I lied. ;-) When on support, I built a couple of applications that helped me debug and also provided me an opportunity to try out new technologies. The take-away was that, although it is important to debug the issues properly and quickly, it is also important to improve the debugging process itself, so that the debugging process adds more value. Before repeatedly debugging the same type of issue, you should ask yourself, “Why can’t I write something and/or create an app which will do this repetitive and manual process for me?” This is the time to let your creative juices flow. You can try new languages or new frameworks to solve your problems. It does not have to be a big application. Even small browser extensions may do the trick. There are many places where you can build these apps like: for fetching data from some REST API, for comparison between input data, for posting data or tracking data in your application, and many more.

The learnings I gained on this little adventure have been invaluable, and I believe, also helped me become a better developer. So, boys and girls, the next time your project manager asks you to be a part of production support, give him/her an enthusiastic nod and prepare to innovate!