Is nicotine gum dangerous?

Machine Learning Discrimination

By Heiner Koch (Duisburg-Essen)


This blog post is based on an essay that was published in the journal for practical philosophy in the topic “Discrimination”.


Machine learning - often also negotiated as Artificial Intelligence - harbors the risk of new forms of discrimination that can also be difficult to identify. The use of machine learning in sensitive areas must therefore take place in such a way that discrimination can be identified - or use must be dispensed with.

That machine learning can be discriminatory is now well known and well researched. At the same time, it's not surprising that machine learning can be discriminatory. Ultimately, learning takes place on the basis of training data, and this training data is often already burdened with discrimination. For example, Amazon used machine learning to try to automatically filter out the best performing applicants from a large number of job applications. The algorithm was trained on the basis of previous applications from employees who were considered to be particularly powerful. However, since Amazon has mainly hired men in the past, the algorithm has learned that women, for example, would be unsuitable for the job.

If we believe that only a small number of different groups can be affected by discrimination (such as those listed in Art. 3 of the Basic Law), then most algorithms can be examined relatively well for this discrimination. if one of these group characteristics correlates strongly with the decisions or evaluations of the algorithm, the suspicion that there may be discrimination is at least reasonable. It becomes more difficult if one assumes that arbitrary characteristics can be used to discriminate.

Think of a social credit system like the one that exists in China. Whether you get credit, are allowed to travel or can take on certain professions then depends on whether you have a good number of points. China has so far defined those behaviors that bring you points or make you lose points explicitly and without machine learning. Going through the traffic light when it is red, buying too many computer games or being friends with people who have a bad score costs points. However, China could also choose to use machine learning to determine the point changes. Shopping behavior, movement profiles, social relationships or hobbies could suddenly lead to a point deduction and this point deduction can have serious consequences for the entire life. Such methods are already used to a limited extent in the security sector. Machine learning is supposed to help assess who is dangerous and who is not. Surveillance and repression can result.

Here we are confronted with an almost infinite number of characteristics that can play a role in evaluations and decisions. Some features could be non-discriminatory because they single out features with great explanatory power. If a person is on a list of wanted terrorists, it is certainly not discriminatory to classify this person as a security risk (whether the person landed on the list because of discrimination should not be taken into account). Is a group of people classified as a security risk, which Dr. Pepper Cola consumes because the algorithm discovered in records on terrorists that most terrorists are more likely to be Dr. Pepper Cola, this classification is questionable and can have far-reaching consequences for those who are more likely to have drunk Dr. Drink Pepper Cola. In order to be able to identify which of the features used are discriminatory, we would have to examine very many of these features in detail. The workload is therefore significantly higher than if one assumes that only a very limited number of characteristics are relevant to discrimination.

Unfortunately, the situation in the context of machine learning becomes even more confusing. So far we have assumed that the decision-making and evaluation features that the algorithm has learned are known. However, this is often not the case with machine learning.

Machine learning is often described as a kind of black box. We know what data we put in (a simplification, because in reality this is not always the case) and we know what will come out in the end, but we don't know what happens in between. This means that we do not necessarily know, for example, which characteristics the algorithm used to classify a person as a security risk. We only get the information that the person is a security risk. Of course, we do not know whether there has been any discrimination.

An obvious solution is that we design the algorithm in such a way that it always tells us exactly which characteristics it used to make a decision or assessment. Unfortunately, this doesn't necessarily help. On the one hand, it could spit out many thousands of features (in extreme cases even millions of features) and these could then also be so complex that we cannot make any sense out of these features. If we are cognitively overwhelmed by the amount and complexity of the features, we can hardly judge whether there is discrimination. Ultimately, we need to understand why these characteristics led to a decision or evaluation, because only then can we tell whether or not that decision or evaluation was appropriate.

However, solutions are also being worked out here. Machine learning should be explainable. This is researched as Explainable Artificial Intelligence (XAI). There are many different ways of doing this. For example, one can try to design the learning process in such a way that the characteristics on the basis of which decisions are made and evaluated can be described in everyday language. In most cases, however, the performance of the algorithm also suffers. Other approaches try to provide a post-hoc rationalization for the behavior of the algorithm. It does not say what the algorithm actually did. Only visualizations are generated or local or exemplary explanations are given. Although this is not necessarily sufficient to be able to determine discrimination with certainty, people often only rationalize their behavior and we do not know for sure whether other motives than those given were not decisive for the behavior.

The problem remains, however, that the use of machine learning can lead to systematic and inappropriate disadvantage based on unknown characteristics. Here, regulations are necessary for the development of technology and for the use of technology. Similar to the way privacy by design is required in technology and product development, it makes sense to also require transparency by design here. The use of technology can then be accompanied by a check for discriminatory effects. Where transparency cannot be guaranteed or where transparency is refused, the use of technology could be regulated and limited. Where machine learning has a significant impact on people (safety, health, lending, work, etc.), non-transparent evaluations and decisions could then be prohibited by machine learning.


Heiner Koch is a doctoral student at the University of Duisburg-Essen and researches machine learning, domination, scientific theory of the social sciences and metaphysics of the social.