Model explainability as a first class citizen

An AI-powered camera designed to automatically track the ball during football matches tracks the head of a bald linesman instead. Recruitment tools are showing bias against women and other communities. False facial recognition leads to an innocent black man’s arrest.

We see a pattern in these incidents – all, caused due to a false belief in the judgment of algorithms. We tend to believe that algorithms can solve any problem in a perfect and unbiased manner if enough data is thrown at them. But the truth is they reflect the very biases and unfairness contained in the data.  As technology becomes increasingly central to decision making, we have to learn how to approach and apply technology in an ethical way.

How well do we understand algorithms? 

Do we understand the logic used by them well enough to be able to guess the predictions for any given input? Or do we at least understand it enough that given a prediction, we can explain what part of the logic leads to it? These questions make up the concept of 'model explainability.' This blog discusses what model explainability is, and when and why it should be a priority.

Explainability is the extent to which humans can understand (and thus explain) the results of a model. This is in contrast to the concept of ‘black box’ models, where even the designers do not know the inner workings of a trained model and cannot explain the subsequent results.

As we delegate more complex tasks to machines, it becomes imperative to closely monitor the algorithms that make the critical decisions which could affect several lives. These algorithms are only as good and fair as the data used to train them and the cost functions they were taught to optimize. Unjust practices and unintended biases can creep into algorithms and can only be brought to light if they are explainable. 

The lack of explainability lets algorithms go unchecked until it's too late – when too many lives have been unfairly affected. Many such instances are described in Cathy O'Neil’s book ‘Weapons of Math Destruction.’ One such case is of a teacher-assessment tool called IMPACT, developed in 2007 in Washington, D.C. It was supposed to use data to weed out low-performing teachers. But due to the way ‘low-performing’ was defined by the designers of the tool, there occurred many unfair instances of good teachers being fired. And because of the lack of transparency for users and other stakeholders and the inherent trust in the algorithms, the faults in the algorithm were identified too late.

Another interesting case is that of predictive policing. Police forces use algorithms trained on historic crime data to predict possible criminals who are then frisked or checked. Since historic crime data is already biased due to unfair and discriminatory practices against some communities, the model causes those same groups to be checked more frequently. This, in turn, increases chances of them being caught when compared to other more affluent communities – also, affirming the model’s assumptions.

Accountability for the bias

When the unfair decisions of a black box algorithm are finally brought to light, who should be held accountable for the lives affected? The stakeholders or the engineers who coded the algorithm? Only those who understand the model and the predictions it will make can be held accountable. 

Different approaches and levels of explainability might be required – technical teams who design the algorithms including data scientists, would benefit from a detailed and technical understanding of the workings of the algorithm. Other stakeholders may need simple explanations that also provide enough information about the algorithms and the results – to know what they are being held accountable for.

For any given problem that can be solved using ML algorithms, there is usually more than one way to solve it. The mathematical formulation of the problem statement, the algorithm used, the data preprocessing steps – all can be chosen from a variety of options. Usually, the main consideration when choosing the approach is the accuracy of measures of model performance against the historic data or ground truth. 

Augmenting explainability

We also have many ways to make an ML solution explainable. But not all the methods to improve explainability are applicable to all the algorithms. The complexity of (difficulty to explain) different algorithms is also varied. Sometimes the more complex algorithm could give better results in terms of model performance against ground truth data. In such a scenario, how does one decide on which algorithm to use? Does one use the complex algorithm that gives better performance metrics or the one with not-the-best metrics but is simple enough for non-technical stakeholders to understand? 

The answer depends on certain characteristics of the problem statement. In a case where many people could potentially be negatively affected, the risk outweighs the gains. The explainability of the algorithm must be given priority even at the cost of performance. 

A good example of this is ‘resume robots.’ Businesses are increasingly using algorithms to filter out incompatible resumes. It saves a lot of time for companies that receive numerous applications for every vacancy. Due to the possible negative effect of losing job opportunities and the scale, explainability of the algorithm becomes critical. Choosing an easily explainable model like decision trees should be considered over a more complex model like a multi-layered neural network-based classifier. Even if the decision tree leads to some incompatible resumes not being filtered out, it is a better choice than a model that unfairly filters out deserving candidates.

Since the choice of the algorithm itself is one of the factors that decides how explainable the solution can be, we recognize the absolute necessity to consider explainability as a first class citizen when designing AI/ML solutions and not leave it to the end as just a nice-to-have attribute.