Jusletter IT

Law is an artefact, or a technology that, like many others, is an expression of human nature and, as such, has evolved throughout the course of the existence of the human genre. Like other current technologies – computer science and genetics, in particular – it has undergone an exponential acceleration in recent years. Therefore, even in the legal realm, arises a fundamental question which can be tackled from a twofold perspective, theoretical and practical: how the interaction between technologies and human beings can be properly designed. This is not an effortless challenge because, since different technologies are combined in the market uptake, it is difficult to evaluate the social impact of each one.

Over the last century, legal thought has consistently re-shaped and intertwined the three main traditional narratives based on the concepts of ethical value, formal rule, or social practice. In the first sense, an example could be found in John Finnis, who reframed the human rights as a set of principles engraved in the natural order of Being²; of the second perspective, we can remember Hans Kelsen, who famously proposed a ‘pure’ theory of law, formulated by separating from the legal content the logical fabric of the law, divided between prescription and sanction³; in a third way, it is noticeable Eugen Ehrlick with his idea of law as a direct expression of the interests embodied in social ties⁴.

Fifty years ago, legal scholars adopted the theoretical model of cybernetics⁵. According to this perspective, the idea of ‘autopoiesis’ has been conjugated to describe the social interactions involving both legal system⁶ and constitutional interpretation⁷. This approach determined a consistent leap from the tradition, being based on the abstract concept of “information”, which, since then, has flooded in many different social sciences⁸.

More recently, 25 years ago – for the sake of celebration, let’s say, at the beginning of the IRIS-era – a further step was made, due to the widespread deployment of ICTs. In the well-known RENO/ACLU ruling of 1997⁹, the US Supreme Court qualified the Internet as “a wholly new media”, uncapping Pandora’s box of claims for the redefinition of traditional legal categories in such a context. This has been the wake of cyberlaw, an historical moment when the capacity of ICTs to enable and to regulate forms of aggregation ignited discussion among different perspectives. Some claimed that “virtual communities” were to be considered a new kind of society, technology itself being an intrinsic normative force – here comes the motto “code is law”¹⁰ – while others qualified technology as an irrepressible component of every legal provision – the “lex informatica” – even outside virtual communities¹¹. Lastly, there were those who fought against these new ideas, perpetuating traditional beliefs¹².

Today, the advent of Distributed Ledger Technologies – with the vision of the “lex cryptographia”¹³ – has confirmed that it is possible to explore the legal realm towards new territory where society and technology are not opposed, but integrated. According to this view it can be argued that law is no longer based only on trusted human relations¹⁴, but also on trust – less “rough consensus”, since it relies directly on how technologies are engineered, deployed, and used. Consequently, legal regulation becomes a matter of design both in a social and in a technological sense.

In general terms, the close relationship – or better, the overlap – between law and technology requires reframing the concept of law, and the abovementioned narratives qualifying legal regulation in terms of ethical values, formal rules, and social practises. This applies both at a collective or even global level – as a matter of institutional governance, it could be said – and at an individual level, in the interaction of human beings with their technological devices. Such issues seem urgent when related to artificial intelligence (henceforth, AI), whose social impact is of enormous scale and with unpredictable consequences¹⁵.

In this contribution we address the topic of technological design of human interaction with AI, providing an up-to-date overview focused on the legal issues concerning the “request to intervene”. The first part describes the ideal context in which the discussion takes place – the role of technological design in the relationship between values, rules, and practises – and the different theories on ethics of technology, mainly Value Sensitive Design (VSD) and Responsible Research and Innovation (RRI). In the second part we discuss the legitimacy and the conditions under which the control of a system can be transferred from individuals to artificial agents, and back. In the third part, previous conclusions are put into the context of “industry 4.0” to test their validity.

The tripartite narrative of legal thought can be reformulated in the light of the recent “Philosophy of Information”¹⁶, which puts forward cybernetics’ achievements. In this sense, it is useful to take up the three different conjugations of information, “information as reality”, “information about reality” and “information for reality” – which stemmed from previous studies¹⁷ – and associate the first with the formal validity of the rules, the second with the values contained in them and the third with their social effectiveness.

Consequently, we can draw a conceptual model in which each of these versions of the concept of “information” corresponds to a specific dimension. To the vector named “information as reality”, which concerns in general the formal validity of rules, we can place law and technology. This represents how these two components integrate each other in determining the way in which masses and individuals are conditioned.

On the level of ‘information about reality’, i.e. ethical values, we can place ethics – obviously – but also to the industrial standards (ISO standards, technological certifications, quality marks) in which technological design incorporates values in order to operationalize them. This applies, as far as our contribution is concerned, to the future European “AI Law”¹⁸, according to which the ethicality of an algorithm will not be a purely substantive issue, but a procedural one, since it will have to be subject not only to a validation process before being introduced in the market, but also to constant monitoring during its use.

On the level of “information for reality”, i.e. effectiveness, it is placed the relationship between the individual and the social group (family, workplace, or, in general, the legal system to which one belongs). This is where the problem of the political legitimacy of public institutions arises and operates the inner drive motivating individuals to abide with social rules. In the light of “philosophy of information”, law is configured here as “information for reality”, since it is essentially conceived as a command or a set of practical instructions.

Technological design can be placed at the intersection of these three vectors. Indeed, any artefact, whether tangible or intangible, is not neutral, but reflects values, rules, and social practises in each context. Design, in this sense, can be qualified as an expression of human creativity limited by a set of variables defined in a higher level of complexity¹⁹.

Figure 1: Law, technological design and information

To define how technological design can be implemented, it is crucial to shed light on the twenty-years-long discussion concerning the concrete ways of incorporating values into the use of artefacts²⁰. On this regard it is noteworthy that the best-known methodologies, such as Value Sensitive Design (VSD)²¹ and Responsible Research and Innovation (RRI)²², have recently been further developed to include AI technologies²³. Despite the results accomplished by these approaches in drawing a consistent methodology, experts raise several concerns, the most relevant of which are the risk of arbitrariness in the choice of values to be axiomatized²⁴, the difficulty of translating the proposed requirements into practice, them being formulated in a too general and abstract a manner, the need to distinguish “engineering ethics” and “machine ethics”, and the consequent disputable qualification of AI as a moral agent only in the presence of true autonomy²⁵. In the background of this debate is placed the main concern of AI governance, i.e. the consideration of the technological impact from an ethical, political and social point of view, which therefore concerns not only labour law and the protection of personal data, but also other disciplines such as consumer law and competition law²⁶.

The “request to intervene” is a fundamental concept in the interaction between autonomous systems and humans, which is located at level 3 of autonomy as defined both by Levels of Human Control Abstraction (LHCA)²⁷ and by SAE²⁸. Despite the general agreement on this assumption, currently there is a lively discussion concerning how it can be implemented whenever it involves a decision requiring a trade-off between values, especially if includes human physical integrity. In this section, we focus on three aspects: (1) the design of the interface, (2) the training required to the human operator for a risk-free interaction and (3) the allocation of responsibility.

Concerning the first aspect, it has to be clarified that the “request to intervene” comes with an autonomous system. More precisely, it depends on the inability of the system to manage events outside its Operational Design Domain (ODD). Therefore, due to this incapacity, an intervention by an external agent – the human controller, in this case – is required to generate the interaction with the environment. In this sense, the first design choice is therefore the one that leads to the fundamental setting of the ODD. This choice defines the basic setting and profoundly influences structural and functional requirements of the system, hence its working conditions. More precisely, ODD is a combination of several design choices: (1) social and political factors, including social acceptance and trust in technologies²⁹ , (2) legal requirements and (3) engineering standards. This synthesis encompasses both the degree of automation (“how much” automation is expected) and the processes to be automated (“what” to automate).

The second aspect, namely the training of the human operator, is deeply connected to the first. Only after a thorough analysis of the interaction of the device with its environment and the development of an efficient interface, can a human operator be trained to handle the request for intervention. Therefore, training can be considered as a continuous process of adaptation of the human to the machine aimed at balancing a task with its intrinsic level of risk, provided that a risk-free scenario is impossible to achieve. In this sense, the learning process³⁰ needs to be balanced balance with what is called “deskilling” which is related to an excess of trust in the autonomous system³¹. In this respect, legal and technical design play a key role, since they not only have to establish the ODD, as seen above, but also how the human is expected to cope with the transmission of authority on the processes controlled³². Specifically, they determine: (1) how the handover has to be done, (2) what training the human controller has to receive, (3) the trade-off among many interests and values at stake.

The third aspect regards the liability in the “request of intervene” in the occurrence of failures of the system, commonly due to “bugs”, human error, or both. Anyway, of course it can be said that risks are a by-product of every human activity, yet it is possible to separate “bugs” or errors that are unavoidable from those who are not, and therefore find grounds suitable to assing liability.

Therefore, the main challenge in the design of “request to intervene”, is to avoid the targeting of autonomous agents as easy scapegoats for human errors³³, as well as to circumvent the “moral crumple zone”³⁴, namely the cases when the human operator is wrongly held responsible. In this sense, it is crucial for legislators to define a proper strategy of AI governance to protect human beings while implementing future-proof regulations without hindering technological innovation.

The expression “Industry 4.0” is quite new. At the Hanover Fair in 2011, the term “Industry 4.0” was popularized for the first time³⁵. Curiously, it was created to define the upcoming industrial revolution, that being the first time that a revolution had been defined before its happening. Such a paradox is only apparent, since our societies are increasingly aware of the effects of technological innovation, especially about speed of changes, scale of impact, depth of transformation³⁶. Furthermore, reports from the World Economy Forum (WEF) show that this set of technologies will be crucial in addressing current needs of reducing energy consumption and increasing environmental sustainability of current industrial production. In this respect, many Countries have already adopted specific strategies to pursue these goals³⁷, likewise the European Union is rooting on them to foster the economic growth and the resilience of the internal market after the COVID19 pandemic³⁸.

The social impact expected from the advent of Industry 4.0 is still under scrutiny, although experts agree that it is useful to consider some crucial factors, which are esteemed to be even more pivotal when the pandemic will end and hopefully investments will return to flow regularly³⁹.

First, the impact of automation on the labour market is still uncertain. International competition and low labour costs require a strategy aimed at obtaining the best results with the savviest use of resources. Companies are re-inventing their business models and strategies to increase automation, augmented communication, and self-monitoring. Hence, they are planning to deploy systems that can diagnose their limits and blockages without human intervention.

Secondly, the need to develop strategic public-private partnership to foster digital innovation. Since it is still unknown the impact of an increasingly complex industrial model, where humans and machines combine their efforts, it is not enough to create just monitoring software, but it is important to develop shared strategies to steer the transition to Industry 4.0 in a sustainable way.

Third aspect: total quality and safety. Incredibly, in many industries some activities carried out for the quality control are entrusted to workers that are required to check semi-finished or finished products even for long working shifts. In past steel industry, for example, workers had to get close to burning metal to detect and report any defect, so exposing themselves even to deadly risks. Implementing quality industrial control was meant not only for marketing a valuable product, but also to monitor the manufacturing processes, hence an imperfection of the final product can reveal malfunctions that can put workers at risk. With current information technologies it is possible to implement automated quality control systems, enabling a proactive approach which prevents failures in the workflow, as well as allows real-time data exchange with the productive ecosystem.

The importance of these three aspects combined can be exemplified by considering the deployment of machine vision and artificial intelligence in “Industry 4.0”. Indeed, when implementing AI-supported machine vision systems in an industrial plant, it’s essential to seed it into an IoT platform⁴⁰ that allows human workforce to monitor – even remotely – the manufacturing processes and supports the analysis of the data generated by machines. In this sense, these functions are greatly simplified if the interfaces are intuitive, flexible, and usable. Of course, it can be claimed that human oversight can be easily replaced by artificial agents especially in repetitive tasks, which are more likely to be afflicted by human errors due to operator fatigue. In this sense it can be said that cases the monitoring tasks, which ordinarily are performed by human operators, become integrated directly into the system. In this scenario, the concept of “work” needs to be reframed. Notably, sociologist Neil Postman, who studied the changes in language related to the modification induced by each new technology⁴¹, argues that today “work” certainly has a different meaning than in the past. Indeed, through the introduction of automation systems and AI into industries, many repetitive, dangerous, or demanding jobs have been delegated to machines. These are activities in which workers are required to classify objects, count them, or check for flaws. It is known that the introduction of these systems has been interpreted by some as a threat to work; this phenomenon is called “neo-luddism”⁴² and it expresses a sense of rejection for machines, seen as a threat to their workplaces. But, counting iron bars in a dangerous plant is perhaps an activity that makes sense to still be performed by humans?⁴³ The displacement of workers, however, should not be seen just as a side effect of the development of autonomous systems, but a primary challenge to be addressed by a proper AI governance strategy, which goes beyond the simplistic approach summarized in this acronym HABA-MABA (“Humans are better at/Machines are better at”) which has based many initiatives until today⁴⁴.

Therefore, “Industry 4.0” is paradigmatic of the many challenges and opportunities raised by technological design. As Baumer and Silberman argue: “Being skeptical about technology does not mean rejecting it” but “the argument here is that when we do build things, we should engage in a critical, reflective dialog about how and why these things are built”⁴⁵. Indeed, if it is true that the purpose of manufacturers is to produce marketable products which can articulate an ecosystem of services, the adoption of autonomous system can relieve humans from physical threats and increase productivity, hence it is an opportunity that should be considered with favour.

Concluding on this topic, it could be argued that “Industry 4.0” is not the destination of technological progress, but a further step towards the unknown and, furthermore, a concept intrinsically limited. In 2019 Gerd Leonhard introduced the neologism “androrithm”⁴⁶ to indicate everything that cannot be converted into algorithms, which means the activities in which humanity has a unique and irreplaceable value, as a sign of a civilized and evolved society free from those activities that can be performed by machines. We should become aware that just because fortunately machines will respond to most human productive needs, humanity will have to rethink its role in the universe and perhaps also understand that without human consumption and needs, production has no meaning⁴⁷. Human resources are the most valuable asset in any working context and can be used for purposes that are more satisfying for workers and more interesting for companies, being able to eliminate mortifying jobs that do not express the value of workers. This is not a loss but a real achievement only if we are able to discuss the true meaning of the word “work”.

The “request to intervene” epitomises, under a practical perspective, the theoretical issue of balancing human needs and AI potentials which recently have been widespread as “Human-centric AI”. Paradoxically, at the centre of the model here proposed it is placed not the figure of a human being, but an abstract concept such as that of design. This is somehow truth-revealing, since it represents the current risks of replacement of humans by AI, which can be considered the most advanced form of creativity artefact ever created.

According to such a perspective, we can argue that, among the many challenges to be faced in the coming years, the most crucial is to adopt AI technologies not to replicate passively current industrial mass-production, but to re-invent economic models to optimise sustainability in a proactive way. This is the most important challenge for AI governance. Therefore, it should be clear, at least in the academic debate, that social acceptance and trust in technology are indeed crucial factors, although they should neither be confused nor traded with the respect of fundamental rights and individual freedoms. The re-invention of the concept of “work”, in this sense, is an unavoidable consequence, which would likely require a long period of adaptation and even new technological solutions, during which it is important not to miss its inevitable link with human dignity.