report-consumer-friendly-scoring
Dieses Dokument ist Teil der Anfrage „Gutachten des Sachverständigenrats für Verbraucherfragen“
Areas for action:the state of research 65
2. Data accumulation and from online traders and mail-order firms, through cus-
tomer loyalty cards or cashback cards, through competi-
data trading tion entries, from publicly accessible statistics and direc-
tors or from data collectors such as Facebook, Amazon,
The foundations of any system of consumer scoring are Twitter or other website operators (Goldmedia, 2017).
consumer data, sometimes in considerable volumes
(Dixon and Gellmann, 2014). There are scarcely any ar- Data collectors can obtain personal data on the Internet
eas left in consumers’ lives in which they do not leave in many different ways (Palmetshofer, Semsrott and Al-
a digital footprint. As a result of the digital revolution, berts, 2016): on the one hand, there is a direct route in
it has never been so easy to collect personal data. Sim- cases where data are voluntarily transferred, e. g. when
ilarly, there has been a growing economic interest in users register with an online service or post their profile
making profitable use of these data, for example in the on a social network site. Then there are data which are
realms of advertising, e-commerce, market research gathered when a person is observed and so discloses
and politics. The aim, in most cases, is to prevent fraud, information indirectly, e. g. through browsing history or
to identify people or to make marketing campaigns and GPS locations. Lastly, data such as online profiles, likes
customer communication more efficient by pinpointing or reviews and even seemingly non-personal data are
and addressing specific target groups (OECD, 2013). analysed and evaluated for the purpose of extrapolating
personal data, such as an individual’s age or sex.
Data that cannot be collected by the company that in-
tends to use them are often bought in from data traders. It is virtually impossible for consumers to trace where
The business model of such enterprises consists in col- their data goes. Most data traders receive some of their
lecting, processing and selling data. In 2014, the trade material from other data traders and sell it in turn to
in addresses and other personal data was worth some more data traders. A study conducted by the Federal
610 million euros (Goldmedia, 2017). Trade Commission (2014) found that seven of the nine
US-based data traders under examination made data
The personal data that are collected about consumers available to each other.56 This means that consumers
can be classed in various categories (OECD, 2013): de- have few opportunities to object to any previous use
mographic data (e. g. date of birth, gender, civil status, of their data, to correct false data and to contest their
level of educational attainment and income), user-gen- classification in particular categories, because they do
erated material (e. g. blogs, comments, reviews, photos not know who holds their data and which of their data
and videos), Internet browsing history (e. g. search-en- are already available globally for sale.
gine queries and online shopping history), data relat-
ing to a person’s social environment (e. g. contacts and A key concept in this discussion is that of informed con-
friends in social networks), location data (e. g. address, sent, which is the legal basis for the collection of many
GPS data and IP address) and official personal commu- of the items of data referred to above. A consumer free-
nication data (e. g. passport number, account numbers ly agrees, for example, to have his or her usage history
and police records). recorded, which would otherwise be prohibited. Con-
sumers normally express their consent by agreeing to a
As a rule, data traders do not obtain data directly from privacy statement or to a set of standard business terms.
consumers but through third parties, and often without There are, however, some problems associated with in-
consumers being any the wiser. Data traders use numer- formed consent (Hofmann and Bergemann, 2017).57 On
ous sources to build up their databases, for example the one hand, studies show that most consumers do
56 The nine companies examined in the study were Acxiom, Corelogic, Datalogix, eBureau, ID Analytics, Intelius, Peek You, Rapleaf and Recorded Future (Federal Trade
Commission, 2014).
57 The challenges described here illustrate clearly that a consumer’s informed consent alone cannot be a key instrument of data protection but must be supplemented
by other mechanisms. Accordingly, the guarantees provided by the principle of purpose limitation (Article 5(1)(b) GDPR) and the prohibition of conditionality
(Article 7(4) GDPR) are crucial; on this point, see SVRV 2016.
66 Areas for action:the state of research
not read the text of the privacy statement or the stand- concluded, for example, from an analysis of mobile-tele-
ard business terms. Ticking the appropriate box before phone data (Chittaranjan, Blom and Gatica-Perez, 2011;
downloading an app or joining a social network has De Mont- joye, Quoidbach, Robic and Pentland, 2013) or
become a habitual everyday action. One contributory search-engine use and browsing behaviour. (Kosinski,
factor is that it is mostly impossible for users to access Stillwell, Kohli, Bachrach and Graepel, 2012).
the desired service without giving their consent. Users
therefore see no real alternative that would not mean se- Available information is also used to sort consumers into
verely restricting their daily activities or opting out of so- groups and categories. Assignment to a group does not
cial participation. Another criticism of informed consent normally depend on one single characteristic but on a
is that it may be doubted whether consumers are fully combination of various items of information. The data
informed at the time when they give their consent. Priva- trader Acxiom (see below), for example, uses categories
cy statements and standard business terms often run to such as “middle class”, “active urban” and “status-driven
several pages and are worded in a way that is incompre- working class” (Junge, 2012).
hensible to non-experts. Most users, moreover, are en-
tirely unable to assess the potential implications of con- As was shown at the start of this chapter with the aid of
senting to the storage and processing of their data. This some examples, novel scoring models are making in-
is why the SVRV, in its report Digital Sovereignty (SVRV, creasing use of big data and algorithm-based processes.
2017a), advocated a rule requiring businesses to inform Since the computation of a score does not require a caus-
consumers of their standard business terms and privacy al relationship between two different variables but only a
policy in an easily comprehensible one-page statement statistical correlation, statistical methods are used to an-
not exceeding 500 words. Some consumer bodies have alyse large volumes of consumer data and to recognise
also proposed the introduction of preformulated privacy patterns and connections with a view to making predic-
provisions (Pollmann and Kipker, 2016). tions or estimates of future consumer behaviour (Christl,
2014). There is therefore great interest in accumulating
Even though individual sources may only make a few the largest possible volumes of data. Jeanette Hofmann
items of data available about a consumer, data traders fears that this might lead to a data-based creation of mo-
can accumulate many different items. In this way it be- nopolies. “It is actually the case”, she writes, “that the
comes possible to obtain a detailed picture of an individ- generation and analysis of data is in the hands of very
ual’s lifestyle and behaviour. What is more, data traders few organisations. And of course it is also the case that
not only aggregate raw data, such as a person’s name, possessing more data enables a company to construct
age, payment transactions and search history but also better algorithms, which means that there is, so to speak,
data elements derived from them, in other words con- an inherent propensity for monopolisation in this mar-
clusions regarding, for example, product preferences, ket.” (Hofmann, in Bilger, Löwel and Tomaszewski, 2018;
purchasing power or fitness level but also extremely del- see also, for example, Rubinfeld and Gal, 2017).
icate categorisation into groups based on ethnic back-
ground, income brackets or health status (on the asso-
ciated dangers of indirect discrimination, see section B. Examples of data traders internationally
II.2 above). A study in the United States, for example, has and in Germany
demonstrated that an analysis of someone’s Facebook The Acxiom Corporation58 is an enterprise from the United
Likes can provide a fairly reliable guide to characteris- States; it provides marketing services to corporate clients,
tics such as sexual orientation, ethnicity, religion, polit- including the provision of consumer data. In the United
ical attitudes, intelligence, narcotics consumption, age, States, Acxiom reportedly holds data on 250 million peo-
relationships, gender and more (Kosinski, Stillwell and ple. According to its website, Acxiom is active in more than
Graepel, 2013). Various personal characteristics can be 60 countries and promises its clients access to 2.5 billion
58 http://www.acxiom.de/ueber-acxiom/; accessed on 18 June 2018.
Areas for action:the state of research 67
consumers59. A 13-digit number is allocated to each person German-based AZ Direct GmbH, for instance, adminis-
whose data it holds, and all stored information is linked ters 37 million private addresses and offers these to its
to that number (McLaughlin, 2013): demographic data, clients for direct marketing activities.62Through its AZ
household characteristics, financial situation, life events, DIAS audience-targeting system it can provide profiling
interests, buying activities, social behaviour – the list of data on 40 million households, 70 million individuals
data elements is long.60. Sorting consumers into particular and 20 million buildings, covering socio-demographic
target groups is as much part of Acxiom’s services as the and psychographic attributes, consumption patterns,
matching of data with specific e-mail addresses, postal stages of life, location/geodata and much more. Promi-
addresses or telephone numbers. A particular speciality nent companies such as the Weltbild publishing group,
of the corporation is the acquisition of offline data from the Gruner + Jahr publishing house and the Klinger in-
sources such as government authorities, with which the dustrial group have their address bases administered by
online data are amplified. Acxiom also offers its clients the AZ Direct. Visitors to the AZ Direct website63 can look up
storage and administration of complete customer data- descriptions of more than 2,500 lists that divide custom-
bases. Its clientele includes credit-card providers, vehicle ers of client companies into categories such as ‘environ-
manufacturers, insurers, retail chains and many more. ment-conscious’, ‘donation-prone retired academics’ or
‘socially minded multiple mail-order purchasers’.
Since 2004 Acxiom has been operating in Germany too,
where it offers clients address data with numerous other
items of additional information on the address in ques-
tion and on the consumers who live there. The data sets
can be tailored to specific consumption intentions, life-
styles and attitudes of consumers. Acxiom Deutschland
has already collected data on 44 million Germans.61
Besides Acxiom Deutschland, the dominant companies
in the data-trading business in Germany are ABIS GmbH,
which is part of the Deutsche Post Address Group, AZ
Direct GmbH, part of the Bertelsmann Printing Group,
EOS Holding GmbH, a subsidiary of the Otto Group, and
Schober Information Group Deutschland GmbH (Christl,
2014; Goldmedia, 2017). They mainly operate in the
realm of address trading; that is to say they collect con-
sumers’ postal addresses, verify and process them and
supplement them with additional items of information.
They obtain the addresses through publicly accessible
channels, such as telephone directories and registers of
companies and associations. Data traders also buy data
from other companies, such as mail-order firms and
publishing houses.
59 https://www.acxiom.com/; accessed on 18 June 2018.
60 A selection can be found at https://www.acxiom.com/what-we-do/infobase/; accessed on 18 June 2018.
61 https://www.linkedin.com/company/acxiom-deutschland-gmbh; accessed on 18 June 2018.
62 http://www.az-direct.com/site/multichannel-marketing-produkte/direct-mail/; accessed on 18 June 2018.
63 http://www.az-direct.com/site/fileadmin/ikat/listinfos/index.html; accessed on 18 June 2018.
68 Areas for action:the state of research
the data sets with additional information, for example
3. Repersonalisation of from social networks (see, for instance, De Montjoye,
Hidalgo, Verleysen and Blondel, 2013; Ji, Li, Srivatsa, He
anonymised data and Beyah, 2016; Srivatsa and Hicks, 2012). In Germany,
reporters from the regional broadcaster Norddeutscher
Anonymised data, in other words data that are not re- Rundfunk (NDR) succeeded in repersonalising a dataset
traceable to a particular consumer, are not covered by they had obtained from a data trader which contained
the Data Protection Act and may therefore be collected some ten billion IP addresses, retrieved from about
and used in Germany without restriction and be freely three million German Internet users (Norddeutscher
bought and sold by data traders. The specific connec- Rundfunk, 2016). Detailed browsing histories proved to
tion to a person – typically a name, date of birth and be assignable to specific persons. The data set, in fact,
address – is therefore removed from the data sets. An- was apparently obtained by unlawful means: according
other option is to pseudonymise the data, which means to NDR, a Web of Trust browser extension from WOT Ser-
that attributes such as the name or address are replaced vices had recorded the websites visited by users without
with a pseudonym or other identifier.64 obtaining the users’ consent and had then stored the
data on servers outside Germany.
It would be wrong, however, to underestimate the dan-
ger of ‘de-anonymisation’ or ‘repersonalisation’ of such It may therefore be argued that, in the age of big data,
anonymised or pseudonymised data. It normally takes every item of data is potentially personal because of the
only a few items of related data to make individuals countless possibilities of linkage with other personal data
identifiable again. Sweeney (2000), for example, showed (Boehme-Neßler, 2016). The collation and reconciliation
that zip code, gender and date of birth are sufficient to of several data sets containing personal information such
identify unmistakably 87% of the US population. Coun- as shopping records, browsing histories, search histories,
ty, gender and date of birth are already enough to iden- etc., from various sources make it possible to trace these
tify 18% of US citizens. back to specific consumers. Big data and powerful al-
gorithms allow this to be done without any great effort.
There is ample evidence of the ways in which an- In this way, data acquired from data traders can also be
onymised data sets can be repersonalised. Researchers used for consumer scoring, since those data are also com-
at the University of Texas, for instance, using algorithms piled on individual persons as a rule.
that they had developed themselves, managed to reper-
sonalise parts of an anonymised Netflix data set with film To sum up, it may be said that the accumulation and trad-
reviews posted by 500,000 users. By reconciling the data ing of data play a major role in the age of big data and that
with non-anonymised film reviews on the Internet Movie it is possible with the aid of algorithms to identify specific
Database website, the researchers succeeded in identi- consumers, even in large anonymised data sets. But what
fying individual users. It also proved possible to uncover does this mean in relation to consumer scoring?
their political preferences and other sensitive informa-
tion (Narayanan and Shmatikov, 2008). In 2009, the same
authors demonstrated how active users of both Twitter
and the image-hosting service Flickr could be reidenti-
fied from an anonymised Twitter data set with an error
rate of only 12% (Narayanan and Shmatikov, 2009). Oth-
er studies have shown that anonymised mobility data
obtained from GPS sensors in smartphones, computers
and vehicles can be repersonalised by supplementing
64 The question whether pseudonymised data are not personal is a controversial one. Recital 26 of the GDPR states that “Personal data which have undergone
pseudonymisation, which could be attributed to a natural person by the use of additional information, should be considered to be information on an identifiable
natural person.”
Areas for action:the state of research 69
The diversion of data to other purposes is encouraged
4. Aggregation of data into a by data trading. Credit scoring in particular is already
closely associated with the data-trading business – often
super score in an inscrutable labyrinth of corporate conglomerates
and subsidiary companies. Two components of the Ber-
As data trading grows in significance, so does the poten- telsmann media group, for instance, are the data-trading
tial for data from the most diverse areas of people’s lives firm AZ Direct GmbH and the credit reference agency
to be brought together in a single database and a single Arvato Infoscore. Creditreform Boniversum, as well as
company. Potentially, then, data from various suppos- performing its own function as a credit reference agen-
edly unconnected areas of activity could be matched cy, also administers, through its subsidiary Microm,65a
with particular consumers and then used as a basis for consumer database registering socio-demographic, so-
scoring them. Such a super score would mean that an in- cio-economic and psychographic attributes. So there is
dividual person’s behaviour in a particular context could undoubtedly scope for the use of more unconventional
have far-reaching implications for every area of that per- attributes, such as online behaviour, for credit scoring as
son’s life (see section B.II.4 above). well as for other forms of consumer scoring.
Evidence of data and scores being diverted to other pur- It should also be mentioned at this point that such
poses has surfaced in the United States, for example. scope also exists regardless of data traders, for example
Credit information, such as the FICO score, is used by in major insurance companies that offer various types
providers of motor and household insurance to calcu- of insurance. With the consent of policyholders, data
late their premiums (Consumer Reports, 2015; Dixon and sets can be combined in these cases and analysed to-
Gellmann, 2014). What interests these insurers is not how gether, including behavioural data. The insurer Gener-
likely a customer is to repay a loan but rather the degree ali Versicherung AG, for instance, currently offers smart
of probability that he will be prepared to pay higher insur- insurance and telematics-based options in four types
ance premiums or that he will make a claim, and the data of insurance: life insurance and occupational disability
are fed into these assessments (O’Neil, 2016). insurance (Generali Vitality), motor insurance (Generali
Mobility) and household contents insurance (Generali
In a survey conducted by the Society of Human Re- Domocity). The Vitality programme is to be introduced
source Management, almost half (47%) of the 430 re- in the near future in the realm of private health insur-
spondent employers stated that they had a credit check ance. Even though it must be stressed that Generali has
conducted before they hired a new employee (Society not given any indication at all of plans to combine the
for Human Resource Management, 2012). In this case data sets, interesting questions certainly do arise in the
too, the aim was not actually to find out about an ap- light of the danger posed by super scores (see also the
plicant’s creditworthiness but to infer attributes such as market study in Part C below): could driving data not
trustworthiness and reliability (O’Neil, 2016). be relevant in the calculation of health insurance pre-
miums? After all, accidents involving personal injury
may increase the cost of health care. And, conversely,
does not a driver’s high risk of heart attacks increase the
probability of an accident?
65 https://www.microm.de/; accessed on 18 June 2018.
70 Areas for action:the state of research
The scenario of a super score, in which a person’s be-
haviour in various areas of his or her life is analysed and
used to calculate a score, seems entirely plausible in
the depicted context of innovative business models, big
data, data trading and de-anonymisation. While devel-
opments like the social credit system in China are not
to be expected from the governmental side in Germany,
they do raise the question whether consumers need to
be better protected against similar developments in the
business world. If large volumes of data on consumer
behaviour are easily obtainable and analytical tools can
be profitably deployed, data trading and consumer scor-
ing will continue to proliferate. The fact that companies
have a great interest in such developments is illustrat-
ed, for example, by a remark made by Douglas Merrill,
founder of the credit reference agency Zest Finance; his
analysis of the current state of play can also be taken as
a warning: “We feel like all data is credit data, we just
don’t know how to use it yet” (Hardy, 2012).
Market survey: credit reference agencies, motor insurance telematics and health insurance policies 71
C
Market survey: credit
reference agencies, motor
insurance telematics and
health insurance policies
72 Market survey: credit reference agencies, motor insurance telematics and health insurance policies
I. Introduction and key issues
In order to analyse the current supply of consumer scor- The other aim was to give respondents the opportunity
ing services in the German market in the three areas to communicate their experiences, opinions and plans
under examination in this report, namely credit scoring, relating to scoring and behavioural tariffs. We were in-
telematics-based motor premiums and health scoring, terested, for instance, in learning what they saw as the
the SVRV conducted a market study in the spring of pros and cons of scoring and how they saw the future of
2018, surveying agencies and insurers in the relevant such systems in their respective sectors. With a view to
market segments. obtaining the frankest possible responses, we assured
the insurers and other business representatives that
The method of a standardised written questionnaire was their responses would be analysed anonymously.
chosen to obtain responses directly from insurers and oth-
er businesses and possibly learning more than research In the analysis of the findings, our intention was to study
into company websites, etc., would reveal. Another objec- consumer problems in conjunction with corporate scoring
tive was to supplement the examination of the consumers’ models, with due regard to the problem areas identified
perspective in Part D with a portrayal of the perspective in Part B above, in order to build an empirical basis for the
and interests of compilers and users of scores. recommendations for action that we make in this report.
One of the aims was to gauge the current prevalence of
scoring systems and behaviour-based business models
in the three areas under examination. The three key
questions were:
• W
hich products are available and which
are in the pipeline?
• W
hy are people scored? What objectives are
being pursued?
• What data are used?
• Which quality criteria do scoring systems meet?
• H
ow transparent are companies about their
scoring systems?
Market survey: credit reference agencies, motor insurance telematics and health insurance policies 73
II. Survey design
The market study conducted by the administrative of- Health insurers, unlike credit reference agencies and
fice of the SVRV encompassed firstly credit reference motor insurers, were not confronted with the specific
agencies, secondly motor insurers and thirdly both stat- term ‘scoring’, since the concept of ‘lifestyle-based tar-
utory and private health insurers. Market research was iffs’ seemed to be commoner in this sector and there-
conducted into these three sectors with a view to iden- fore more expedient. For this reason, most of the ques-
tifying relevant businesses for the survey, and a market tions relating to score calculation, modelling, statistical
profile was produced to serve as the basis for the survey. quality criteria, etc., were omitted from the health in-
surers’ questionnaire.
Although specific consumer problems could be expect-
ed to emerge in each of the individual market segments, On the whole, then, a comparative analysis of the sur-
provision also had to be made for an overarching discus- vey findings was possible, while specific conditions and
sion. For this reason, care was taken when conducting potential trends in a particular sector could also be ad-
the survey to maintain a sufficient degree of standardi- dressed. In the analysis, scoring-related consumer prob-
sation to ensure that most of the questions were identi- lems affecting all three market segments could be high-
cally or similarly worded for all respondents in the three lighted, and potential best practices could be elicited.
areas (see Annex II).
The questionnaire was not pretested but was developed
At the same time, specific features of the three segments on the basis of background discussions with representa-
also had to be taken into account. For example, the sur- tives of agencies and insurers and with scoring experts.
vey dealt with potential future trends and influencing
factors that varied from one segment to another. For A word of caution: the findings of this market study are
the credit reference agencies, for example, there was a based solely on responses from representatives of agen-
question about the possible use of data from social me- cies and insurers, and it was not always possible to vali-
dia, motor insurers were asked for their views on the Eu- date these, for example by consulting publicly available
ropean eCall Regulation, and health insurers were asked sources. The primary aim of the study was therefore to
about electronic patient files. establish the nature and extent of the information that
agencies and insurers make available about their scor-
The consideration of specific conditions in the survey ing systems and also to find out what views, ambitions
was most evident in the domain of health insurance, be- and wishes they would express with regard to scoring
cause neither statutory nor private insurers in Germany when assured of anonymity.
offer policies with lifestyle-based premiums at the pres-
ent time. Policyholders may, however, take part in bonus For this reason, most of the questions were open-ended,
programmes in which healthy activities and participa- that is to say there were only a few questions in which
tion in preventive measures are ‘scored’ in a sense,66i- respondents were able to select a preformulated reply.
nasmuch as they earn bonus points which policyholders For the analysis of the open-ended questions a quan-
can redeem for monetary or other rewards. In the cases titative content analysis was conducted, and a classifi-
of credit reference agencies and telematics-based motor cation system with summative categories and associat-
insurance tariffs, however, the degree of automation is ed codes was developed. All responses were coded by
considerably more advanced. The questionnaire for in- three estimators acting independently of each other.67
surers focused far more on establishing where ventures
into health scoring were already in evidence and what
the respondents’ wishes, plans and views were with re-
gard to health scoring.
66 Some bonus programmes even offer the option of using data from fitness apps to earn points as well as enabling participants to check their current points total and
administer their points account digitally.
67 Where responses were coded differently, the category on which two estimators agreed was selected.
74 Market survey: credit reference agencies, motor insurance telematics and health insurance policies
In the areas of credit scoring and telematics-based mo- Motor insurers
tor insurance tariffs, the response rates both exceeded Another questionnaire was sent to motor insurers that
50% (60% for the credit reference agencies and 63% for offer telematics-based tariffs. There are no providers of
the motor insurers), while the response rate for health motor insurance whose product portfolio consists en-
insurers came to 41%. The findings cannot be consid- tirely of telematics-based policies; on the contrary, tele-
ered representative of the respective sectors in their en- matics-based premiums are add-ons that are offered for
tirety, and the responses reported here provide limited existing motor insurance policies
scope for generalisations.
A total of 15 telematics-based tariffs or add-ons were
identified in the realm of motor insurance. Of the provid-
ers we contacted, five took part in the survey; some of
these companies also responded on behalf of subsidiar-
ies. As part of a pilot project implemented between 2013
1. Overview of providers and 2015, Sparkassen Direktversicherung had tested a
telematics-based tariff and also took part in the survey.
In total, responses relating to ten telematics-based tar-
Credit reference agencies iffs were received. These tariffs are listed in Annex I.2.
To analyse the German market in credit scores, we
surveyed various credit reference agencies. These are
private-sector companies which communicate to busi- Statutory and private health insurers
ness partners economically relevant data and credit This market study also involved a survey of all health
ratings pertaining to individuals. Questionnaires were insurers operating in Germany. Although health scoring
sent to five agencies, all of which are members of the as such is not practised here, the aim was to shed light
association of credit reference agencies known as Die on the various bonus programmes so as to establish in
Wirtschaftsauskunfteien e. V. (formerly called Verband what form and to what extent policyholders’ healthy
der Handelsauskunfteien e. V.) and provide credit infor- lifestyles are being registered and rewarded. In addition,
mation on private individuals. Annex I.1 contains a list insurers were questioned about their attitudes to life-
of the participating firms. style-based tariffs and on their plans for the near future
with regard to such schemes.
At the present time, a total of 110 providers of statutory
health insurance68 and 43 private69 health insurers are
operating in the German market, of which a total of 62
(47 statutory and 15 private) took part in the survey70
(see Annex I.3).
68 Information from the National Association of Statutory Health Insurance Funds at https://www.gkv-spitzenverband.de/; accessed on 10 July 2018.
69 Information from the Association of Private Health Insurance Companies at https://www.pkv.de/; accessed on 10 July 2018.
70 The participating Local Health Insurance Funds (AOKs) answered some of the questions identically, namely those relating to views on lifestyle-based tariffs and
future prospects. When the responses were analysed, however, each response from each of the AOKs was assessed once, which meant that the responses in question
were each assessed eleven times.
