Jusletter IT

The use of AI-based Facial Recognition (hereinafter “FRT”) is widespread. Use cases range from security systems of private companies to law enforcement – or simply unlocking your phone display. The aim of FRT is to authenticate or identify a person.¹ For some, FRTs are emblematic of a dystopian surveillance state and therefore prohibited in certain cities.² The most controversial use cases are related to publicly accessible spaces. They can range from practical, such as replacing boarding cards at the airport, to impractical, such as being necessary for access to toilet paper.³ Heldt argued that part of the reason, why the topic is so controversial, is the lack of regulation of the matter.⁴ However, this is about to change. With the new proposals for AI regulation of the European Commission, European secondary law will also specifically address FRTs. In the following analysis, the authors will analyze parts of the proposed legal framework, which consists of the AI-Act⁵, the AI liability directive⁶ and the amendments to the product liability directive⁷. The new framework will therefore include a regulation on product safety (AI-Act) as well as new liability norms for such products. The AI-Act will even directly address FRTs, which will be the focal point of the analysis.

This analysis is based on interim results of the project XAIface, in which the authors analyze the legal and ethical framework of the use of AI-based FRTs.⁸ The goal of the project is to increase the level of trust and social acceptance of face recognition technology. The XAIface consortium aims to better the understanding of the underlying mechanisms in face recognition based on machine learning in general and deep learning in particular. The consortium studies the impact of influencing factors and tries to explain their role in the overall performance of a system. Further desired outcomes include the creation of guidelines for the legal and ethical use of FRTs, which will include the new requirements of the AI-Act and parts of the new liability regime.

FRTs currently seem to be a thorn in the flesh for data protection authorities. The Canadian⁹, Australian, French and UK¹⁰ data protection authorities already ordered the controllers of the system to cease the data processing in question.¹¹ Recently, the Italian data protection authority followed suit. Likewise, the Greek data protection authority imposed a fine on the controller amounting to 20 million Euros.¹² Italy now even went a step further and completely banned FRTs except for judicial purposes and crime prevention, even though further exemptions are to be expected.¹³ Within the institutions of the EU, several groups have even called for a ban of FRTs.¹⁴ Although the European Commission did not go as far as to outright outlaw FRTs, the AI-Act proposal specifically addresses FRTs and bans certain use cases of said technology.

Like the GDPR¹⁵, the AI-Act proposal does not only apply to the Member States of the European Union. If the FRT system is put into service in the European Union, the AI-Act will apply, regardless of where the provider is established. Similarly, the regulation may apply to users and providers if “the output produced by the system is used in the Union”¹⁶. Naturally, it also applies to any user within the European Union.

A provider in the sense of the AI-Act proposal, is someone who develops a system or lets someone else develop a system to place it on the market under their own name or trademark.¹⁷ It should be noted that no distinction is made between public or private entities. The definition of a user contains something akin to a “household exception”. The definition does not apply to any activities outside of a professional setting. Hence, the private usage of FRT, for example on a private smartphone, will not fall within the scope. Similar delimitation problems to Art. 2 par 2 lit. c GDPR may arise.¹⁸ Geminn argues, that it should be evaluated if any private use cases of FRTs couldn’t also be deemed an unacceptable risk.¹⁹ If FRTs are specifically designed for military purposes, the proposal will not apply.²⁰ Furthermore, the regulation will not apply to third country public authorities or international regulations, where certain agreements are in place for law enforcement and judicial cooperation. In similar cases to Clearview AI, where the system is used by US law enforcement services, this could result in application to the provider, but not to the user. The provisions are criticized for being unclear, especially with regard to Software-as-a-Service solutions²¹, which is a potential scenario for FRTs.

The material scope is defined by the term “artificial intelligence system”²². Definitions of Artificial Intelligence used in other disciplines do not necessarily coincide with this specific legal definition.²³ The current definition in the proposal, however, will undoubtedly be satisfied by FRTs. The first criterion a system must satisfy is “software”, which won’t really lead to much differentiation, since all FRTs are software. The system must be developed with one of the techniques or approaches listed in Annex I²⁴, which is rather far-reaching.²⁵ Since FRTs are usually based on machine learning, specifically deep learning in some cases²⁶, the criterion is also fulfilled. Furthermore, the system must have human defined objectives and generate outputs such as content, predictions or decisions, which in turn must influence the environments they interact with.

The other criteria shall be demonstrated by a hypothetical scenario²⁷:

Whether an event must have an effect to be considered a decision, is currently an open question.²⁸ Bomhard/Merkle highlight the complex relationship with the territorial scope, if outputs are produced and reused in the EU.²⁹

It can be argued that the term environment³⁰ includes humans, such as security staff. This interpretation is mainly derived from the usage of the word ‘recommendation’ in the definition, which by nature is usually directed solely towards a human counterpart.

Some FRTs will have to fulfill special obligations under the AI-Act and will – a priori – be prohibited, which is a direct result of the risk-based approach.³¹ The common denominator of the relevant definitions in the AI-Act proposal is the term “biometric data”³², which is identical to the definition in the GDPR.³³ This choice leads to the conclusion, that the same interpretation should be applied. Hence, the definition is intrinsically linked to the unique identification of a natural person and special technical procedures. Therefore, if the data processing is not linked those aspects, the definition does not apply.³⁴ Not every AI-system that processes images therefore falls within the scope of one of these prohibitions, since the processing of images would at least have to be linked to identification efforts.³⁵ A simple face detection system, for example, will therefore not fall under these provisions in the AI-Act proposal.

The first listed technology based on biometric data is an “emotion recognition system”, which refers to “an AI system for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data.”³⁶ This technology does not fall within the typical definition of FRTs³⁷ and is therefore not covered by the XAIface project. Due to similar inherent risks, however, this specific application will also be covered by the AI-Act.

A “biometric categorisation system” refers to “an AI system for the purpose of assigning natural persons to specific categories, such as sex, age, hair colour, eye colour, tattoos, ethnic origin or sexual or political orientation, on the basis of their biometric data”.^{38, 39} The system does not identify or authenticate any natural person, but rather categorizes them based on chosen criteria inherent to their biometry. Like the emotion recognition system, this is not a typical FRT use case.

The typical use cases are covered by the term “remote biometric identification system”, which refers to “an AI system for the purpose of identifying natural persons at a distance through the comparison of a person’s biometric data with the biometric data contained in a reference database, and without prior knowledge of the user of the AI system whether the person will be present and can be identified”.⁴⁰

Two different variations of these remote biometric identification systems exist, separated by a temporal component:

a) “Real-time remote biometric identification system”: “a remote biometric identification system whereby the capturing of biometric data, the comparison and the identification all occur without a significant delay. This comprises not only instant identification, but also limited short delays in order to avoid circumvention.” ⁴¹

b) “Post remote biometric identification system”: “a remote biometric identification system other than a ‘real-time’ remote biometric identification system.” ⁴²

Art. 5 par 1 lit. d AI-Act proposal specifically prohibits the use of real-time remote biometric identification, but only for publicly accessible spaces and only for the purposes of law enforcement. A publicly accessible space is defined as “any physical place accessible to the public, regardless of whether certain conditions for access may apply”.⁴³ Generally, we argue for a broad interpretation of the term “publicly accessible”, which can be seen in the following example⁴⁴. Wendehorst links the definition of publicly accessible spaces to the potential, that an indefinite number of persons enters the space, which seems appropriate.⁴⁵

Even for law enforcement purposes, the use of real-time remote biometric identification may be allowed, if public interests outweigh the risks. The European Commission, however, anticipates the weighing of interests⁴⁶ and only allows the use for example for the search for missing children, prevention of terrorist acts or similar threats or the detection, localization, identification or prosecution of specific serious crimes, as demonstrated by example 3^{47, 48}:

It should be noted that the use of the system is not legitimized by the provision in the proposal of the AI-Act. The public authorities still must comply with the respective national laws and the respective European data protection law.^{49, 50} However, since there was a substantial (death or bodily harm) and imminent (within the next one or two hours) threat to the life or physical safety of natural persons or (depending on the motivation) a terrorist attack and the crime could be prevented, the exemption may apply, if the use cases passes the strict necessity test required by the provision. For the necessity test, arguments may be based on harm avoided by the system vs probability, scale and seriousness of human rights infringements.⁵¹

The necessity could arguably be based on the fact, that no alternatives with the same outcome and lower risk to the people were available. But even if the strict necessity test is passed, further restrictions apply. In addition to necessity, the use case must be proportionate, which will mainly be achieved by specific safeguards. The relevant safeguards for example 3 can be summarized as follows:

a) The system is not used without prior cause. The cause is a tip from Europol and a credible and concrete threat. ⁵²

b) Geographical limitation: The system is only used within the station, the area of the potential crime, not in the whole city.

c) Temporal limitation: The system is only used until the suspect is apprehended.

d) Personal limitation: The system is only scans people within the station and references specific database entries.

While these tests are generally a good measure of proportionality, Ebers. et al. argue, that the provision which detection, localization, identification, or prosecution of certain crimes is to broad, since it includes crimes with comparably short prison sentences (3 years).^{53, 54}

The proposal also limits the usage of facial recognitions systems further by demanding “prior authorization granted by a judicial authority or by an independent administrative authority of the Member State”.⁵⁵ An exception can be made in case of urgency. Authorization may then be requested post factum.

As such, the articles already provide solid guidance for the use of FRT for the purposes of law enforcement. The exemptions still allow for a wide range of use cases, but again, the use of FRT must also comply with national and other European laws. Generally, the use of FRT for law enforcement without special cause will infringe data protection law.⁵⁶ As noted by the European Economic and Social Committee, the prohibition does not cover “post” and “near” biometric FRTs or biometric systems which don’t identify natural persons.⁵⁷

While some authors recognize that these provisions impose serious requirements⁵⁸, not everybody agrees that the provisions provide an appropriate test of necessity and proportionality. The EDPS and EDPB rather argue in a joint opinion, that automated biometric recognition in public spaces should be prohibited entirely due to the high risk the use case poses for fundamental rights.⁵⁹ Furthermore, these provisions only apply to law enforcement, while private use cases may still be problematic. Ebers et al. even go so far as to argue, that all of these systems, whether used by private or public actors, should be prohibited.⁶⁰ The total prohibition should include ex post identification. The aim would be to prevent a priori mass surveillance. They argue, and in my opinion for good reason, that the fear of being identified by itself has already a restricting effect.

Additionally, it should be noted, that Wendehorst specifically distinguishes between identification and authentication. Where authentication is a one-to-one relationship and the natural person claims to have an identity, identification is a one-to-many relationship. If the system tests, if the person is one of many and the person does not claim a specific identity, the definition is fulfilled. Therefore, authentication methods for securing buildings usually don’t fall within the prohibition.⁶¹ She also argues, that the limitation of the prohibition to ‘real’-time identification does not take into account the potential threat of ex post identification to fundamental rights.⁶² This argument is generally convincing. On a similar note, the EPRS noted that the distinction between ‘real-time’ and ‘post’ remote identification systems risks being arbitrary.⁶³

Geminn argues, that due to the lack of real categorical prohibitions, the creation auf an AI-based surveillance infrastructure remains possible.⁶⁴

If a system is not prohibited, it will likely be classified as high-risk according to article 6 of the proposal and will therefore have to comply with further requirements. The classification is essentially based on two options. Either the system is covered by certain legislation⁶⁵, which concerns for example machines or medical products, or it is covered by Annex III of the AI-Act proposal. The main problem with this type of classification is, that the assessment will be conducted by the providers of the AI-systems. Risk of a certain technology, however, can only be evaluated based on its specific use case, which the provider won’t always be able to anticipate.⁶⁶ One of the cases described in Annex III is the use of biometric identification and categorisation of natural persons, specifically those systems intended for real-time and post remote biometric identification of natural persons.⁶⁷ Even though the provision states, that AI-systems must be intended for the ‘real-time’ and ‘post’ remote biometric identification of natural persons, it is clearly the intention of the proposal to categorize a system as high-risk if it fulfills one of those criteria. The provision may be changed in the future. It’s noteworthy that this provision is not the only possible way FRTs can fall within the high-risk category⁶⁸, even though it’s the explicit one. However, if one is to follow Wendehorst’s previously mentioned opinion on the strict distinction between authentication and identification, typical entrance security examples will still not be covered, even if they pose significant risks of surveillance to – for example – employees.⁶⁹

As mentioned, the result of the classification is the obligation to comply with further requirements. One of these is set out in Art. 10 AI-Act proposal and concerns data governance. The provision generally concerns the usage of data for training, validation and testing. Providers of AI-systems must employ various data governance and management practices, which are partially not unlike those set out in data protection law.⁷⁰ The provision also requires providers to examine potential biases, which is a known issue for FRTs.⁷¹ One provision should especially be highlighted. Paragraph 3 postulates the requirement, that all mentioned data sets “shall be relevant, representative, free of errors and complete”. The interpretation of this clause is subject to academic debate. It is especially questionable, what the termini “free of errors” are supposed to encapsulate. Practically, and with special regard to large data sets, it should be virtually impossible to ensure complete freedom from errors.⁷² Additionally the requirement of using representative data sets is furthered through the inclusion of the obligation to use appropriate statistical properties as well as paragraph 4, which requires the provider to take into account the context in which the system will be used.

Art. 10 par. 5 of the proposal facilitates the usage of such data for the purposes of “ensuring bias monitoring, detection and correction in regulation to high-risk AI systems”. The data may be used, but only to the extent, it is strictly necessary for said purpose. Appropriate safeguards must be set up. According to Ebert/Spiecker, the provision is based on Art. 9 par. 2 lit. g GDPR. They interpret the elimination of bias as public interest.⁷³ Whether that is the case, will have to be discussed in the future. But in short, the provision has the potential to provide long sought-after legal certainty.

Finally, another relevant provision for FRTs in Art. 14 AI-Act proposal since it explicitly mentions them. Facial recognition technology will have to comply with the special paragraph on biometric systems⁷⁴ in Art. 14 of the proposal. This concerns real time as well as post remote biometric identification of natural persons. According to this provision, “no action or decision is taken by the user on the basis of the identification resulting from the system unless this has been verified and confirmed by at least two natural persons.” The usefulness of the provision is already called into question due to the lack of guidance and means for compliance.⁷⁵ The measure could also be criticized, because teams are not always more effective in avoiding automation bias than individuals and therefore the risk is not automatically lower.⁷⁶

FRTs currently are a hot topic for data protection authorities as well as the European Legislator. The AI-Act will create a new legal framework for the use of FRTs. There is no consensus on which rules should apply to FRTs. However, the opinion appears to be, that newly created a priori prohibitions do not go far enough, mainly due to the inherent risks of mass surveillance. Debates range from broadening the material scope of the prohibitions to including private actors. However, the prohibitions are certainly a powerful tool to combat some of these risks with regard to law enforcement. The AI-Act will furthermore provide a much-needed legal basis for processing special categories of personal data to prevent biases. While having a good intention, the provision on human oversight in its current form is of little use to reduce risks. A further point of criticism is the constant use of vague terms and concepts.⁷⁷ The academic discussion and the widely varying opinions demonstrate not only the risks of the use of FRTs, but rather the need for a democratically legitimized decision on the matter – in the form of the AI-Act.

This work has been partially supported by the European CHIST-ERA program via the Austrian Research Promotion Agency (FFG) within the XAIface project (grant agreement CHIST-ERA-19-XAI-011).