report-consumer-friendly-scoring
Dieses Dokument ist Teil der Anfrage „Gutachten des Sachverständigenrats für Verbraucherfragen“
Areas for action: the state of research 25
B
Areas for action:
the state of research
26 Areas for action:the state of research
I. Transparency and
comprehensibility
What the appropriate level of transparency for scor- ing interest groups, whose arguments cannot simply
ing systems is and how that level can be achieved are be dismissed and are constitutionally underpinned in
unanswered questions in discussions among the gen- each case.
eral public and researchers. In the context of scoring,
transparency means the disclosure of information to An individual whose behaviour is the subject of a pre-
consumers by producers or users of scoring systems. dictive assessment based on scoring, whom we shall
Reflections on the right level of transparency always go refer to below as the ‘scored person’, will normally have
hand in hand with the question how information that is an interest in learning that a scoring process is taking
made transparent should be processed and structured place at all. Secondly, he or she will want to know about
to ensure that it is actually comprehensible. On the one the consequences of the resulting score. The individual
hand, the yardstick may be set for consumers, enabling may also be interested in knowing the data on which the
them to play an informed part in scoring processes. On calculation of his or her score is based, in other words
the other hand, the comprehensibility gauge may be set which personal characteristics are taken into account in
for experts to enable them to engage in critical examina- determining the score. Lastly, the individual may be in-
tion of scoring systems. terested in gaining some insight into the internal work-
ings of the scoring algorithm, particularly the relative
Credit scoring has hitherto been at the heart of the trans- weight attached to each personal characteristic in the
parency debate, because until very recently it was the calculation of the score.
most technically developed and most widespread form
of scoring (on its history, see Beckhusen, 2004). Future These interests may, on the other hand, conflict with
discussions on the transparency of scoring may be ex- the confidentiality interests of the party conducting the
pected to merge with the vigorously conducted debate scoring process, referred to below as the ‘scorer’, or of
of more recent times on the transparency of algorithmic the public. A scorer will generally have an interest in
decision-making procedures. The transparency aspect confidentiality if the predictive product of the scoring
of scoring will remain relevant, not least because only an process is economically valuable and therefore merits
adequate level of transparency will enable consumers to protection as a trade secret; this interest was recognised
assert more extensive rights, such as the right to correc- by the Federal Court of Justice in its judgment of 28 Jan-
tion of an erroneous score. Transparency is ultimately a uary 2014 in case VI ZR 156/13, recorded in the Civil De-
prerequisite for any informed debate within society on cisions of the Federal Court of Justice (BGHZ), Vol. 200,
the phenomenon of scoring. p. 38; see also section C.III.3 below). If the details of the
algorithmic method for calculating scores become com-
mon knowledge, the method ceases to be a trade secret
and can be copied by competitors.
A public interest in confidentiality can exist if disclosure
1. Transparency in of the scoring method would lessen its predictive value
in certain circumstances, which we shall shortly exam-
predictive scoring ine. This may be socially undesirable. It cannot be de-
nied, for example, that there is a general public interest
Scoring processes are used both to predict and to in reliable credit assessments.
modify modes of behaviour. When it comes to ensur-
ing transparency, a distinction must be made between Not every disclosure entails a risk of diminished predic-
these two purposes. The fact is that scoring systems tive quality. Disclosure is harmless if the score is based
designed to predict behaviour are not normally meant on characteristics which are actually responsible for
to operate reflexively, in other words they themselves the assessed probability. In this case the scored per-
are not intended to influence the observed behaviour. son, by modifying his or her behaviour in a way that
Accordingly, what constitutes an appropriate level of should serve to improve the score, is actually influenc-
transparency is a bone of contention between oppos- ing the probability of the predicted event. Those who
Areas for action:the state of research 27
take regular exercise reduce their risk of illness – for
this reason, a person’s decision to engage regularly in 2. Transparency in
sporting activity cannot be described as a ‘manipula-
tion’ of his or her score.
behavioural scoring
By contrast, the predictive value of the score decreases Scoring can also be an instrument of behaviour modifi-
if the behaviour modification relates to variables which, cation. When that is the underlying purpose, transpar-
though they have been good indicators of scored prob- ency seems at first sight to be an essential condition
ability the past, do not influence the probability rating; for the effective use of scoring, for an incentive system
practical examples are cited in section B.VIII.1 below. No cannot achieve a targeted behavioural effect unless
one reduces his or her risk of illness by buying sports gear it reveals the connection between behaviour and its
but not using it; anyone who knows that the purchase of assessment. To put this in the context of the scoring
trainers is included in a health score as a so-called proxy process, if the scorer’s aim is to motivate scored indi-
variable (see section B.V.2 below for more details) might viduals to improve their score, it seems imperative that
thus be tempted to affect his or her score by means of con- the scorer must at least disclose that certain modes of
sumption decisions rather than actual sporting activity. If behaviour ‘win points’.
the workings of a scoring system are revealed, scored peo-
ple can recognise the effects of their behaviour on their However, there is also a ‘softer’ means of modifying
score and therefore modify their behaviour to suit their behaviour through scoring. This can be illustrated by
score (Bambaucher and Zarsky, 2018). means of a hypothetical example. Imagine that a score
was calculated for healthy living or good driving, but the
Influencing scores by targeting non-causal criteria is scoring criteria were not disclosed. One might assume
discussed in literature under the heading of ‘gaming that such a scoring system would have effects on the
the system’ (Rona-Tas and Hiss, 2011). British econo- behaviour of scored persons, who would try to improve
mist Charles Goodhard encapsulated this insight into their score. Only the direction of the behaviour modi-
the self-reflecting nature of social systems pithily in fication in this case would be more uncertain, as the
the statement “When a measure becomes a target, it scored person can only presume what modes of behav-
ceases to be a good measure”. It merits consideration iour are assessed by the scorer as healthy living or good
in any discussion on indicator-based control (Strath- driving and so count towards a better score. The scored
ern, 1997; Wagner, 2018; Weingart and Wagner, 2015). person is therefore faced with the challenge of satisfying
The prevention of system-gaming may be in the gener- scoring criteria that he or she does not know.
al interest as a means of preserving the validity of the
predictive score. A certain lack of transparency about
the operation of the scoring method is then required.
On the other hand, the precise opposite conclusion
may also be drawn from this scope for ‘gaming the sys-
tem’, namely that the right way to remedy the potential
for manipulation is not to maintain opacity about the
scoring criteria but to exclude non-causal criteria from
the scoring process. This approach may be harder to
achieve, but the greater fairness of a system based on
causal criteria alone cannot be refuted out of hand (a
detailed treatment is to be found in Britz, 2008).
28 Areas for action:the state of research
on the appropriate level of transparency for scoring has
3. Keeping transparency been the dialogue between the legislative and judiciary
on the one hand and legal scholars on the other. Both
and comprehensibility of legislators and legal scholars have seen a particular
scoring systems on the need for regulation of credit scoring. Three events have
structured the transparency debate in that area.
agenda
The first caesura came with the creation of scoring-spe-
Transparency is a key instrument of consumer policy. cific data protection provisions in 2009 (Federal Law Ga-
Accordingly, numerous studies are devoted to the le- zette I, p. 2254). By revising the Federal Data Protection
gitimacy, effectiveness and limits of the transparency Act, the legislature assembled a body of provisions gov-
principle in the realm of consumer protection; a sum- erning scoring from section 28b of the old version of the
mary of this issue can be found in Tamm, 2011, espe- Act, which set out the requirements for lawful scoring,
cially on pages 347ff. The link between transparency and from section 28a of the old version, which regulated
on the one hand and how informed consumers actu- the transfer of data to credit reference agencies. These
ally are on the other is being increasingly questioned rules were supplemented by a scoring-specific exten-
(Ben-Shahar and Schneider, 2014; Kettner, Thorun sion of the information rights of data subjects that had
and Vetter, 2018; see also section B.VIII.2 below). Max- been enshrined in section 34 of the old version (Heine-
imum transparency by no means implies maximum mann and Wässle, 2010). Before this new set of rules was
protection of consumers. Safeguarding real consumer enacted, the permissibility of scoring had been deter-
autonomy is therefore set to move to the heart of the mined on the basis of the general data protection provi-
discussion. The debate on ‘algorithmic transparency’ sions. The result was not only a considerable degree of
could act as a catalyst, because the ineffectiveness of legal uncertainty as regards the very legality of scoring
obligations designed only to ensure the disclosure of (Petri, 2003, and Beckhusen, 2004), reflected for exam-
unprocessed information is especially evident in this ple in the sceptical appraisal by the Federal Data Protec-
context. It would be expecting too much of any con- tion Commissioner of the time (BfDI, 1996, point 31.2.3),
sumers to present them with bare programming codes but also in widespread expressions of dissatisfaction
(see section 4 below). with the insufficient transparency of scoring systems
and with the way in which they worked in practice (Ko-
Nevertheless, the very fact that scoring is a data-pro- rczak and Wilken, 2008). Although some extensive infor-
cessing operation makes it reliant on a certain degree of mation rights for data subjects have been derived from
transparency, because only a transparent system allows the general data-protection provisions (Unabhängiges
individuals to exercise their right to protection of their Landeszentrum für Datenschutz Schleswig-Holstein,
personal data (Bull, 2011). “The legality of decisions can 2005), the prevailing diagnosis pointed to an inherent
only be verified by those who know – and understand – transparency deficit in the legal provisions (Kloepfer
the data basis, the processing sequence and the weight- and Kutzschbach, 1998; Möller and Florax, 2003; Petri,
ing of the decision-making criteria” (Martini, 2017, 2003; Beckhusen, 2005). The legislature sought to rem-
p. 1018). This applies especially to the accuracy of the in- edy the criticised transparency shortfall by creating
dividual items of data that are used to calculate scores. special scoring-specific provisions (see the explanatory
Rights to rectification of inaccurate personal data (see in memorandum for the pertinent instrument, the Federal
particular Article 16 of the General Data Protection Reg- Data Protection Amendment Bill, in Bundestag printed
ulation) become irrelevant if the data subject is unaware paper 16/10529, p. 6 et al., the report and recommen-
of the inaccuracy. On the subject of actual awareness of dation for a decision from the Committee on Internal
information rights, however, see section B.VII.2 below. Affairs – Bundestag printed paper 16/13219, pp. 1–2 and
10 – and presentations of the legislative project from a
The means whereby transparency is supposed to be es- stakeholders’ perspective (Piltz and Holländer, 2008,
tablished in the realm of scoring are legal in nature. This and Metz, 2009)). The Amendment Act altered the ba-
is why the main focal point of the academic discussion sic legal framework, and so we cannot simply carry on
Areas for action:the state of research 29
from the lively discussion on scoring and the identified judgment. The duty of disclosure was to be extended
transparency deficit that was being conducted before to include “the utilised items of data, the weighting of
the adoption of the Act, because questions that were the utilised data, the utilised comparison groups and
unanswered then have now been resolved by means of the assignment of the persons concerned to the com-
binding legislative provisions. parison groups whose data are used in the calculation
of the probability value” (ibid., p. 4).
A second caesura was marked by the ‘Schufa judg-
ment’ of the Federal Court of Justice (Federal Court The entry into force of the General Data Protection Reg-
of Justice judgment of 28 January 2014, Case No VI ZR ulation (GDPR) in May 2018 marks the third caesura in
156/13, Civil Decisions of the Federal Court of Justice the transparency debate. The GDPR replaced a system
(BGHZ), Vol. 200, p. 38). In that judgment the court of national data privacy laws under an umbrella of EU
clarified the scope of the information right enshrined law with a directly applicable European legal instru-
in the first sentence of section 34(4) of the old version ment. The Regulation diverges in many respects from
of the Federal Data Protection Act. The court ruled that previous data privacy law, and not only in its material
information was to be provided on the types of person- scope; it does not forge an unbroken link with the es-
al data relating to the data subject which were used in tablished terminology, regulatory method and legisla-
the calculation of the score. There was no obligation, tive style of the old Federal Data Protection Act either. It
it said, to disclose information on the method used to therefore seems plausible that the adoption of the Gen-
obtain a specific score from that set of personal data eral Data Protection Regulation marks the “start of a
and from other data. In particular, the way in which new era in data privacy law” (Schantz, 2016). The GDPR
the data were weighted was not covered by the infor- is peppered with flexibility clauses that give national
mation right. As a trade secret, the scoring method en- legislators discretionary powers. On this basis a new
joyed the protection afforded to fundamental rights. Federal Data Protection Act was enacted to supplement
The judgment of the Federal Court of Justice generat- the General Data Protection Regulation. Section 31 of
ed keen interest among legal scholars, and the legal the new Act contains a special scoring-specific provi-
database Juris contains more than a dozen academic sion (for more details see section E.I.3 below, which also
analyses of the decision. They form a heterogeneous examines the conformity of the provision with EU law.
picture, ranging from emphatic endorsement (Taeger, With this provision, headed “Protection of commercial
2014) to criticism (Gärtner, 2014, and Schulte am Hülse transactions in the case of scoring and credit reports”
and Timm, 2014). From now on the judgment would be the German legislature sought “to preserve the mate-
the main reference point of the transparency decision. rial protective standard of sections 28a and 28b of the
The authors of the evaluation report on the new scor- Federal Data Protection Act, old version”, as the explan-
ing provisions of 2009 for the Federal Ministry of Food, atory memorandum to the new Act puts it (Bundestag
Agriculture and Consumer Protection and later for the printed paper 18/11325, p. 101, which corresponds to
Federal Ministry of Justice and Consumer Protection Bundesrat printed paper 110/17, p. 101). Initial academ-
subjected the judgment to detailed analysis and crit- ic studies on scoring under the General Data Protection
icism (Unabhängiges Landeszentrum für Datenschutz Regulation do not expect the new regulatory regime to
Schleswig-Holstein and GP Forschungsgruppe, 2014). bring radical changes (Taeger, 2016; von Lewinski and
The plaintiff in the Schufa case has lodged a constitu- Pohl, 2018) and indeed the resilience of ingrained prac-
tional complaint with a view to overturning the judg- tice in the face of legislative innovations must not be
ment. The Federal Constitutional Court has not yet underestimated. Nevertheless, the legislative design of
ruled on her complaint. A legislative initiative from the the transparency requirements and information rights
opposition ranks (Bundestag printed paper 18/4864) in Articles 13, 14 and 15 of the GDPR differ sharply from
sought to alter the legal position but was ultimate- sections 19 and 34 of the Federal Data Protection Act
ly unsuccessful. The purpose of the bill was the en- in its old version (for more information, see section E.
shrinement in the Federal Data Protection Act of more III.4 below).
stringent transparency requirements than the Federal
Court of Justice had imputed from the Act in its Schufa
30 Areas for action:the state of research
The transparency of scoring methods is not only dis- selves. The SVRV has set out its basic position (SVRV,
cussed in academic circles but is also a subject of public 2016; SVRV, 2017; SVRV, 2017a; summarised in Micklitz,
debate. In February, the non-profit organisations Open 2017), stressing the need to ensure, by means of legal
Knowledge Foundation and AlgorithmWatch launched prescripts, that the underlying parameters of algorithms
the OpenSchufa initiative. One of the declared aims of relating directly to consumers are made transparent and
the project is to ‘crack’ the algorithm with which Schufa disclosed in a standardised format to a group of experts
obtains its credit scores (OpenSchufa, 2018). The plan from a regulation agency for digital operations18 (SVRV,
is to find out both the data that go into the calculation 2017a; more in Gigerenzer, Wagner and Müller, 2018).
of the score and the method by which the score is ob-
tained from that information material by asking as many A scoring algorithm is one particular type of algorithm
people as possible to reveal their Schufa score and their (Just and Latzer, 2016). The discussion on scoring trans-
personal details. Numerous media have reported on the parency can therefore be conducted as part of the general
aim of the initiative; examples are given in Erdmann, debate on the regulation of algorithms. The conventional
2018, and Schneider, 2018; Schufa itself responded crit- scoring methods of the present time are significantly less
ically to it (Schufa Holding AG, 2018a). complex than the algorithmic decision-making systems
which usually serve as reference points in the debate
on algorithm regulation (SVRV, 2016) and which are not
infrequently assignable to the realm of artificial intelli-
gence. However, even the algorithms that are used today
are not easily understood by non-experts (see section B.
4. Scoring transparency as a IV.2 below for more details). And if the complexity of prac-
tised scoring methods were to increase, for instance in
special form of algorithm the direction of methods based on systems of machine
transparency learning, particularly neural networks (see Hurley and
Adebayo, 2016; Thomas, Crook and Edelmann, 2017), the
There is currently a lively debate on a suitable regulato- debate on algorithmic transparency would also become
ry regime for digital algorithms. This debate is not only increasingly relevant to consumer scoring. Whether this
taking place in academic circles but is also command- increasing complexity of scoring methods will actually
ing the close attention of German politicians. In the materialise on a wide scale, making scoring systems into
coalition agreement between the CDU, CSU and SPD ‘black boxes’, is uncertain, not least for the simple rea-
government fractions for the 19th legislative term, reg- son that it has yet to be determined whether such new
ulatory goals were formulated for algorithmic decisions scoring systems are sufficiently superior to the conven-
(CDU/CSU/SPD, 2018, lines 2092ff.). Policymakers in the tional methods to make their use economically justifi-
field of consumer affairs (CDU/ CSU/SPD, 2018, lines able. So there is no evidence yet that novel algorithmic
6266ff.), associations (Gesamtverband der deutschen decision-making methods are always ‘better’ – in terms
Versicherungs-wirtschaft e. V. (German Insurance Asso- of model accuracy, for instance – than established meth-
ciation), 2018, and Verbraucherzentrale Bundesverband ods. Pertinent examples are Google Flu Trends, designed
e. V. (Federation of German Consumer Organisations), to predict flu epidemics, and COMPAS, designed to as-
2017) and bodies from civil society (see, for example, sess the likelihood that an offender will re-offend; in both
the initiatives presented at www.algorithmenethik.de cases, the predictive capacity of complex algorithms has
and at www.algorithmwatch.de) have also been discov- been found inferior to that of simple rules of thumb (Dres-
ering the subject of algorithmic transparency for them- sel, 2018; Lazer, Kennedy, King and Vespignani, 2014).19
18 On such an agency, see also Tutt (2017), who advocates a central regulatory authority for algorithms modelled on the Food and Drug Administration and outlines the
powers of such a body (pp. 105ff.): “The agency should serve as a centralized expert regulator that develops guidance, standards, and expertise in partnership with
industry to strike a balance between innovation and safety.” (p. 83).
19 On the issue of the use of algorithms in the US justice system, see Kehl, Guo and Kessler, 2017. The potential implications for consumer law have not really been
examined yet.
Areas for action:the state of research 31
The relevance of the question whether and to what ex- and Guestrin, 2016; Burrell, 2016; Alber, Lapuschkin and
tent developments in artificial intelligence will affect the Seegerer, 2018) – but must be explainable to its users.
practice of scoring is probably fairly limited, because the
difficulties involved in gaining insight into algorithmic The central question is “Is it possible to explain or make
decision-making processes are not confined to ‘modern’ visible retrospectively how the result was arrived at?”
processes. Even quite conventional algorithms based – (Passig, 2017, p. 25). To achieve transparency, then, it
like the Schufa credit score – on multivariate or non- is not necessary to establish a “full understanding” of
linear regression models, for example, are not immedi- scoring software in all its details. It is sufficient to cre-
ately decipherable even to specialists (Lipton, 2017). It ate means of obtaining knowledge of the way in which
is not only in the recent past, then, that the black box an algorithm works. Even in conditions of incomplete
which is often cited in connection with artificial intelli- transparency and incomplete comprehension of scores,
gence has posed a challenge. On the contrary, it is not testing of their functioning is possible by calculating
much of an exaggeration to say that the black box has scores for exemplary cases. This method is called black-
accompanied the development of software from the out- box tinkering (Perel and Elkin-Koren, 2017; Wachter, Mit-
set, a point which is made incisively in Passig, 2017. telstadt and Floridi, 2017). A proposal for a “transpar-
ency interface” (Gigerenzer, Wagner and Müller, 2018)
The disclosure of source codes, even those of simple follows this methodology, as do the proposals made
computer programs, is not generally of much help to by the German Informatics Society (Gesellschaft für In-
consumers (see also section B.VII.2 below). Most data formatik) that algorithm testing be made into a robust
subjects are not computer specialists. And even if they regulatory instrument (Gesellschaft für Informatik, 2018,
were, the technical complexity of the computer systems which also recommends the creation of a right to con-
whose decision-making behaviour is to be made ‘trans- duct tests). To this end, the input to scoring systems
parent’ necessitates a different form of transparency (AI in general) would be systematically varied and the
from the mere disclosure of programming codes, even output evaluated. This could be required, for example,
for the purpose of verification by experts (Wischmey- by a supervisory data protection authority in the frame-
er, 2018; Samek, Wiegand and Müller, 2017; Selbst and work of data protection audits under Article 58(1)(b) of
Powles, 2017; Montavon, Samek and Müller, 2018; Giger- the GDPR and possibly be conducted by that authority
enzer, Wagner and Müller, 2018; Gesellschaft für Informa- itself. Although this would not necessarily make the de-
tik (German Informatics Society), 2018). This appraisal tailed internal operations of the black box recognisable,
forms the basis of a research programme which is cur- it would provide sufficient knowledge of the relevant
rently being very vigorously pursued and is examining workings of the algorithm. This testing, by the way, is
how the way in which complex algorithms work can be in line with the logic of Stiftung Warentest, the German
explained to people comprehensibly. In the field of arti- Comparative Testing Foundation. The Foundation does
ficial intelligence, the key concepts in this discussion are not study architectural plans or recipes but draws con-
interpretable machine learning and explainable artificial clusions about the relevant attributes of a product from
intelligence (XAI – see Gesellschaft für Informatik, 2018; its systematic use.
see also Wachter, Mittelstadt and Floridi, 2017; Selbst
and Powles, 2017; Selbst and Barocas, 2018).
An adequate understanding of transparency in this
context is one that that seeks to embed algorithmic
decision-making systems in explanatory and review
structures based on a division of labour (Wischmeyer,
2018). To be transparent, a system of algorithmic deci-
sion-making need not be visible to the observer in all its
details – which, in the case of neural networks, would
certainly be difficult (see, for example, Ribeiro, Singh
32 Areas for action:the state of research
Developers and users of scoring systems have also With the aid of numerous examples and case studies,
come to appreciate the importance of the traceability of these analyses have highlighted that, wherever there
their methods. Francesca Rossi of IBM put it this way in is a decision to be taken, it is considerably more con-
an interview with a German daily newspaper: “Besides venient to proceed from a numerical value than from a
deep learning, there are systems like decision trees, multi-layered, sophisticated and possibly ambivalent
which are easier to retrace but unfortunately not quite assessment of a fact or – as in the case of scoring – a
so accurate. So we have to find out which is more im- person. Scores go almost as far as is possible to reduce
portant to us: the accurate outcome or the traceability the complexity of judgements. This makes the use of
of the process.” (Rossi, 2018).20 scores in decision-making very appealing, especially if
the decisions are taken, or have to be taken, in automat-
ed form, rapidly and in huge numbers. The judgements
that have to be made in the development of scoring
methods are often not discussed in a way that is con-
ducive to the subsequent social use of those methods.
5. Transparency as a The criteria to be included in a points system for scoring
healthy lifestyles in a health insurer’s bonus programme
condition for a social and the weight to be given to each criterion do not usu-
debate on scoring ally attract public notice (see also section C.III.2 below).
It is a moot question, for example, whether bonus points
Inherent in the use of scores is the risk of lending the should be awarded solely for activities that benefit the
appearance of objectivity to judgements that are insuffi- scored person’s state of health or should also be credit-
ciently discussed within society and so placing them be- ed for those that help to make the health system work
yond criticism (for a fundamental treatment, see Porter, better, such as blood donation and bone-marrow typ-
1995; see also, for example, Heintz, 2007). This criticism ing, or even for activities which are not health-related
of quantification and of the “social use of numbers” but which are deemed socially valuable, such as vol-
(Vormbusch, 2012, p. 37) has become the subject matter untary work. The absence of public discussion on the
of numerous studies, which constitute a productive field judgements that have to be made when creating a scor-
of research. Established reference areas for such anal- ing system could be described as a lack of politicisation,
yses are economic policy (Weingart and Wagner, 2015; that is to say the imposition of normative opinions with
Wagner, 2018; Schlaudt, 2018) as well as various fields considerable social consequences without a preparato-
of education, health and social policy (Muller, 2018) ry and accompanying social debate.
and in particular the actions of international organisa-
tions where these are substantially based on indicators,
rankings, thresholds, etc. (Davis and Fisher, 2012; Rot-
tenburg, Merry, Park and Mugler, 2015; Merry, Davis and
Kingsbury, 2015; Merry, 2016).
20 On the question whether this trade-off actually exists in the reality of scoring, see, for example Hand, 2006, who advances good arguments against its existence. The
point here is only that practitioners are evidently recognising the interest in comprehensibility. Accordingly, it is not a rebuttal to state that random forests, in other
words a large number of decision trees (Gesellschaft für Informatik, 2018), are not necessarily easier to interpret (Groll, Ley, Schauberger and Van Eetvelde, 2018) and
that the comprehensibility problem will not therefore disappear or is even necessarily be reduced as a result of the change of method.
Areas for action:the state of research 33 Insufficient transparency, moreover, may reinforce misconceptions within society as to what a score ac- tually signifies (see also section C.III.3 below). A score awarded by a motor insurer as the basis for a telem- atics-based tariff may be structured in such a way as to cover not only driving habits that can be influenced, such as the care with which the driver brakes and ac- celerates or observes speed limits, but also factors that are not related to driver performance but influence the likelihood of an accident. These could, for example, be the ratio of urban to rural drives, since accidents are more likely to occur in towns and cities, or the ratio of night-time to day-time drives, because driving at night increases the probability of having an accident. If the scored driver or the public at large have the impression that the score is primarily an indicator of driving skills, a gap opens up between the real significance of the score figure and its social use. The purpose of trans- parency in this context is therefore to ensure that the meaning of scores is realistically appraised and that scores are used only to convey that meaning.
34 Areas for action:the state of research
II. Non-discrimination and
equal treatment
Scoring processes result in different scores from one tiating processes, of insight into mechanisms of social
person to another. That, indeed, is their very purpose, exclusion and into historical injustice and, ultimately,
for scores mark differences, and the aim of scoring sys- of civilisatory progress (Fritzsche, 2017). They need not
tems is to differentiate. At the heart of every economic be defined identically for all areas of life and social sit-
and legal order based on market economics and democ- uations and they are open to legislative adaptation and
racy is scope for private autonomous differentiations. In development.
principle, every merchant is free to decide whether to
conclude a contract with a consumer. A contracting obli- Such a definition of the phenomenon of discrimination
gation exists only in a few exceptional cases. Conversely, is to be distinguished from two competing meanings of
protection against discrimination is continually gaining the term: on the one hand, not every distinction made
ground within the legal order. Scoring systems operate between individuals per se is discrimination within the
in precisely this field of tension between entrepreneur- meaning of this report. Such an interpretation of the
ial freedom and social values, the balance between concept (cf. Adomeit, 2002, and Picker, 2008) would im-
which requires constant readjustment. ply the inclusion under the heading of discrimination
of numerous social interactions which are not an issue
and which do not create any need for political action,
extending even to a restaurateur’s differentiation be-
tween customers who are willing to pay and those who
are not. Some specialised statistical terms, such as dis-
1. What is discrimination? criminant function analysis, are based on a value-free
understanding of the verb to discriminate. On the other
The phenomenon of discrimination is understood in a hand, our use of the term ‘discrimination’ is not meant
broad sense in this report. It encompasses actions and to imply that distinctions made on the basis of the
structures that lead to those who possess particular discrimination grounds are socially unacceptable, let
characteristics21 – such as women, homosexuals or peo- alone legally prohibited.
ple of ‘alien’ ethnic origins – being disadvantaged within
society. A guide to the characteristics that are relevant On the contrary, the term ‘discrimination’ is intended to
in this context is given in section 1 of the General Equal designate any unequal treatment based on a criterion
Treatment Act (Allgemeines Gleichbehandlungsgesetz), that is held to require special legitimisation if used as a
which states that “The purpose of this Act is to prevent ground for differentiation. A mere reference to the free
or to stop discrimination on the grounds of race or eth- choice of the person making the distinction does not
nic origin, gender, religion or belief, disability, age or suffice to justify the unequal treatment.23
sexual orientation”. In other instruments the set of ‘dis-
crimination grounds’22 is defined differently, although
there are, of course, numerous overlaps. In each case,
the key characteristics on which discrimination should
not or must not be based are the result of social nego-
21 For the avoidance of doubt, these characteristics are not objective attributes that are inherent, as it were, to the person subjected to discrimination. Non-
discrimination rules afford protection against distinctions made on the basis of purely attributed characteristics (Schiek, 2000; for the General Equal Treatment Act
(Allgemeines Gleichbehandlungsgesetz), see Bundestag printed paper 16/1780, pp. 30–31). This explains, for example, the prohibition of discrimination based on
‘race’, which is a social but not an anthropological category.
22 A wide and potentially confusing terminological diversity prevails in this field. Reference is made not only to Diskriminierungsmerkmale (“discrimination grounds” –
Pärli, 2017, pp. 106ff.) but also to verbotene Merkmale (“prohibited grounds” – Schramm, 2013, p. 7), which does not mean, of course, that the grounds themselves
are prohibited but rather discriminatory treatment based on those grounds in certain circumstances. The same meaning is given to the term geschützte Merkmale
(“protected grounds” – Schramm, 2013, p. 3 et passim); this term expresses that the purpose of the discrimination ban is to protect those who possess particular
characteristics from discrimination on those grounds.
23 From a legal perspective, the unequal treatment on the part of the decision-maker is not conclusively legitimised by the reference to personal autonomy, understood
as “the principle of the individual shaping legal conditions according to his or her will” (Flume, 1965, p. 1) and as “recognition of the autocracy of the individual”
(ibid., p. 6).
