Jusletter IT

This paper deals, on the one hand, with the communication technology of visualisation and, on the other hand, with the peculiarities of law as an object of visualisation. At present, there is no standard and comprehensive model of legal visualisation, but there are some promising approaches. A distinctive feature of the law in connection with its visualisation consists in the abstractness of legal texts. Legal texts reside in an abstract frame that is not linguistically structured. It is the task of legal visualisation, in addition to considering the types of legal situation, to make visible these pre-textual interdependencies of the legal terms. This Section A is based on our contribution to the IRIS 2018 proceedings.¹ Section B extends our earlier study.²

Textuality in Law. Traditionally, law is textual. Jurists transform texts into texts. There are various kinds of texts: laws, contracts, claims, proceedings, transcripts, judgements, etc. Text transformations require abstracting, extracting, reasoning, deduction and other legal methods. Legal texts contain legal terms and legal sentences (Rechtssätze) that we may call Kelsen’s legal sentences. Judgements, guidelines and their head notes are formulated in abstract legal terms. Abstracting and extracting are therefore needed and are performed by jurists and secretaries.

The law appears, on the one hand, in legal texts and, on the other hand, in legal situations, which are mostly typed. The dominance of textuality began in modern times with the reception of Roman law. Previously, customary law had been situationally dominant. With the advance of the social and legal use of machines, the importance of textuality will decline and that of legal programming will increase.³ The visualisation of law fits into this trend. The opportunities and risks of visualisation also have to be taken into account.

Graphical notations are a strong support for a formalised view of the law. Key features are represented by images or graphics, even in cases in which the level of abstraction necessary for formalisation has not yet been reached. Legal visualisation deals with graphical representations and, in particular, with the visualisation of the abstraction of the law. Visualisation as a method tries to describe and find implicit relationships between various rules, concepts and documents. The complexity of legally relevant events, actions and documents is structured and put into a proper timeline that is sufficiently clear for laypeople in such situations. Text and picture accord a strong support for a formalised view of the law two levels, abstract and concrete.

Related Work. In recent decades, legal visualisation has been studied by many authors. Moreover, a separate section on legal visualisation runs yearly in IRIS. A 20-year IRIS multi-method literature analysis provides a large network of authors.⁴ In our study, however, we narrow the focus to situational legal visualisation.

Volker Boehme-Neßler writes about the «visualification»⁵ of law and its multiple facets, including the medium of television. He notes the complementarity of text and image, and their different characteristics and functions. Techno-images such as «[s]tructures, relationships or dynamic processes are often understood more readily when presented as maps, diagrams, models, building plans or computer simulations» (ibid. p. 56). A reason for this is that they «are created by causal mechanisms» (ibid, p. 57). However, the latter are not prevalent in law.

A general schema for visualisation is shown in Fig. 1. This schema consists of seven layers. Layer 2 is the primary one and depicts a phenomenon such as law to be understood and described. This phenomenon is an object of cognition or, in other words, a thing. The phenomenon consists of different elements and can be viewed from different perspectives. Similarly, a metaphorical cube consists of eight vertices, twelve edges and six faces. Layer 2 corresponds to Aristotle’s teaching that things are primary to universals (universalia in re).

Figure 1. A general schema for visualisation

Layer 1 depicts the realm of ideas. It corresponds to Plato’s teaching that ideas are primary to things (universalia ante rem). Next are Layers 3 to 5, which serve as descriptive layers. Textual and visual descriptions are distinguished. Layer 3 refers directly to the thing in Layer 2 and, hence, describes it. The thing can be described with text or pictures or can be photographed, etc., but it is not reflected.

Layer 4 serves as a theory. It structures naïve descriptions of Layer 3. This theory can be described textually or graphically. Layer 4 is on a meta-level in respect of Layer 3.

Layer 5 serves as a meta-theory and is a meta-level in respect of Layer 4.

Layer 6 concerns a model of a person. The left brain is linked to text/language, whereas the right brain is linked to visualisation. The goal is harmony. Layer 6 corresponds to the nominalists» teaching of universalia post rem. The nominalists hold that science is produced in the brain. However, Aristotle would disagree. We hold that both teachings are suitable as theories of facts. Plato taught that ideas come before things. We also hold that Plato’s teaching can be called a theory of products.

Layer 7 concerns emotions. This layer is additional to the other six layers.

This section compares verbal writing and pictorial writing in human communication. Verbal writing has its roots in the Latin language. Examples of pictorial writing are Chinese characters and the icons in public airports or in Olympic Games arenas.

Assume there is a human sender speaking to a human receiver (Fig. 2). In addition to speech (Fig. 2, element 2a), verbal writing (2b) can be used. This verbal writing is parallel to speech and has certain advantages. However, one disadvantage of speech communication is that it cannot be distributed over time. Visualisation (2c) can also be added, but is mostly auxiliary.

Figure 2. Verbal writing and pictorial writing in human-to-human communication

Next comes the sender’s semantic link to the thing (Fig. 2, element 3) and its semantics (element 4). The semantic link can also be autonomous and may have no association with the sender. Pictorial writing is a kind of magic. Examples of pictorial writing are Chinese characters and emoticons. In the link to semantics, pictorial writing dominates, and textual descriptions such as picture captions are auxiliary (Fig. 2, element 3b). Pictorial characters refer to the type of the thing. In the semantic link, pictorial writing provides understanding. Textual captions provide efficiency and clearness – consider the phrase «clare et distincte», which is René Descartes» criterion of truth.

Communication is governed by rules (Fig. 2, element 5). Both speaking (Fig. 2, element 2a) and verbal writing (2b) are strict, because there are strict speech and writing rules. Children’s languages and a Babylonian confusion of languages are not desirable. Pictorial writing rules, however, are elastic. In the arts, for example, multiple interpretations are allowed.

To sum up, certain aspects of law can be better explained with visualisations. Figures in this paper serve as examples of logical pictures in the sense of Röhl and Ulbrich.⁶

In a study of situational visualisation, the concept of situation is of primary importance. A situation with a crossroad description is shown in Fig. 3. A situation such as this is familiar to candidates taking a driving theory test. The test has situation descriptions and questions. Road elements, cars, and road signs are involved. Drivers» intentions are described using text and symbols such as arrows. The description of the situation constitutes part of the situation’s teleology. The alternatives are essential. The candidates answer multiple choice questions.

Figure 3. Situation vs. case in situational jurisprudence

Figure 4. Sample representations of three air traffic situations

Schematic representations of three situations in air traffic are shown in Fig. 4.

Attention is the most important element in situations. Attention can be compared to a cursor that can move to different positions. The players behave as if they were on a stage. A script assigns roles to the players. Representing a situation may involve a set of objects and relations, and the situation can also be represented as a graph.

Roles, not Rules. A situation can be depicted with icons that represent the roles of the players. This is used in situational visualisation.⁷ For example, Florian Holzer⁸ is concerned with German penal law and distinguishes two kinds of visualisation: schematic (logical) legal visualisation and situational (scenery) legal visualisation. Schematic legal visualisation structures concepts, and situational legal visualisation represents situations, their elements, and their relationships. The pictures thus obtained are more suitable than texts, which are one-dimensional strings of words. Both methods are targeted at learning.

Legal Machine or an Expert System? Our thesis is that the situation dominates in a legal machine, whereas the case dominates in an expert system. A legal machine can be defined as a machine whose actions have legal importance and legal consequences. Legal machines are legal actors capable of triggering institutional facts and distributing permissions and obligations. An example is a traffic light, which distributes permission to cross the road. Traffic enforcement cameras are examples of successful legal machines. Complicated examples are computer-based information systems in organisations, workflows for proceedings that use forms, and machines that replace officials in organisations.

The success of a legal machine is determined by the situation that is to be governed. In the implementation, open texture and non-determinism are minimised. A legal machine in a complicated situation risks taking a wrong action: for example, the action taken by a decision-making machine in a highly bureaucratic domain may be wrong. The simpler the situation, the simpler the legal machine: for instance, an automatic barrier to a car park can be simple.

The limits of machines in securing justice have to be taken into account. The problem is termed the «transparency of the legal machine» or the «explainable legal machine». We formulate two requirements for machines: the software architecture must be accessible, and the software must provide legal protection. In the engineering phase for these requirements, the legal requirements must flow down to lower-level specifications. Therefore software engineers have to have regard to the limits of programming in regulation by code, and jurists have to have regard to the limits of territorially-based sovereigns. However, it seems unrealistic to proceed directly from legal texts to programming. Moreover, there are different contexts of justice, and e-procedures do not guarantee justice. In the legal domain, mechanical judges are undesirable, and machine-assisted decision making outweighs machine-based decision making. However, there may be an aim to solve standard cases by machines.

The Term «Situational Visualisation» in Computing. Krum et al., for example, introduce a style of visualisation called Situational Visualisation in augmented and virtual reality systems. They note that «[m]any tasks require a detailed knowledge of the local environment as well as an awareness of rapidly changing and interacting events. This awareness is termed situational awareness or situation awareness.»⁹ They quote Endsley for a formal definition of situation awareness: «Situation Awareness is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.»¹⁰

Krum et al. also continue at p. 143 that «this type of spatio-temporal knowledge awareness is important in many tasks, including […] law enforcement».

Figure 5. Schweighofer’s 8 views/4 methods/4 syntheses model of legal data science

The visualisation view is categorised as one of the eight views in Schweighofer’s¹¹ 8 views / 4 methods / 4 syntheses approach. We visualise this approach in Fig. 5. Schweighofer’s approach can be expressed using the formula 8 views + 4 methods = synthesis. Schweighofer has developed new methods for the representation, analysis and synthesis of legal materials as legal data science. Legal data science, as part of legal informatics, serves as the integrative model of computer-supported representation and analysis techniques.

Multiphase Transformation. The premise of this paper is that it does not seem very realistic to proceed directly in one step from legal texts to legal machines. Intermediate steps are needed. On this path, symbolisation and visualisation precede formalisation. In this way, a multibridge is built between legal concepts and their representations in computers (Fig. 6). In the transformation legal text → formalisation we expect a formalisation in the form of technical notation (language) that further allows computer implementation. Hence, the target is the transformation legal text → program, and the approach is called Multiphase Transformation.

Figure 6. The multibridge metaphor: transformations lead from norm to its machine implementation

Digital Humanities and Human Digitalities. On the right bank of the multibridge between law and informatics there is the area called digital humanities;¹² see Fig. 7. The research subject of digital humanities is the human, who is surrounded by digitally-based phenomena. We introduce another area – human digitalities – next to it on the right bank. Here the research subject – the machine – is different, and is structured according to human-centred rules. The term human digitalities is introduced intentionally to contrast with the term digital humanities.

Figure 7. (a) A multibridge from law to informatics. (b) Digital humanities. (c) Human digitalities

Consider the line of evolution from plants to animals to humans to machines (Fig. 8). In the proposed model, biological evolution leads to humans. The last step, the evolution from humans to machines, however, is a process of technological evolution: humans produce machines. Moreover, humans strive to give human capabilities to their creatures, thus making machines artificially intelligent; recall the ancient myth of Pygmalion and its modern variations.

A question that is associated with the evolutionary step from humans to machines is whether machines reside in status civilis or status naturalis. A relapse to status naturalis is a permanent temptation of modern culture, although re-barbarisation is a kind of political atavism. Weapons substitute for the former raptors. The elements of re-barbarisation can, for example, be observed in EVE Online, a massive multiplayer online game (MMOG). In EVE Online, the players play «the harsh, unforgiving gameplay that lacks many amenities for players that other games of the genre possess».¹³ We, however, maintain that machines are forbidden to be monsters.¹⁴

Figure 8. The evolution line plants–animals–humans–machines

The area of digital humanities belongs to the world of human beings, whereas the area of human digitalities belongs to the world of machines (Fig. 9).

The theme of evolution leads us further towards human digitalities than does legal formalisation. Wagner and Haag¹⁵ write about the step from symbolisation to formalisation (Symbolisierung → Formalisierung). They use the term symbolisation, but symbolisation is now interpreted in a broad sense, comprising both symbolisation-in-the-narrow-sense and visualisation. Hence, we go further and investigate the evolutionary step that results in machines for digital processing.

Figure 9. The area of digital humanities and the area of human digitalities

Digital Humanities. In this section the right bank of the multibridge is studied (Fig. 7, a). This is where the area of digital humanities appears. This term «refers to new modes of scholarship and institutional units for collaborative, transdisciplinary, and computationally-engaged research, teaching and publication.»¹⁶ A field map of digital humanities can be proposed, and shows such methods as «enhanced critical curation», «augmented editions and fluid textuality», «visualization and data design», «the animated archive», etc. (ibid., pp. 30–31). Digital humanities «place a primacy on experiential navigation, epistemologies of representation, and the rhetorics of visualization» (ibid., p. 46). The human is the research subject of digital humanities, and this human is surrounded by digitally-based phenomena, including information and communication technologies.

Colette Brunschwig starts out by noting that «both the digital humanities and what might (as a working hypothesis) be called digital law explore visualization, audio-visualization, and multisensorization.»¹⁷ She brings the digital humanities and digital law closer together. We, however, go on to examine human digitalities.

Human Digitalities. Next comes the area that we call human digitalities. Its research subject is the machine, which is structured according to human-centred rules. We have coined the term to describe machines that are required to act in conformity with the law – in the same way as human beings ought to act. Human digitalities focuses on the concept of a legal machine whose purpose is regulation by computer code. After the text-to-program transformation, the program embodies legal rules; the normative and digital expectations are amalgamated.

Vaios Karavas¹⁸ argues that «the emergence of the computer as medium has triggered a transformation of the legal sphere that is culminated in the emergence of a techno-digital normativity that seems to undermine Luhmann’s description of the legal system as an autonomous social system.» The risks brought in by computers, and the dangers of artificial intelligence, can be viewed from the perspective of programming¹⁹ and from the perspective of law. Graber²⁰ formulates recommendations against the dangers of platforms using artificial intelligence in relation to fundamental rights.

Electronic Virtualities (E-Virtualities). In the evolutionary step from humans to machines, the evolution of spiritual spheres can also be observed. We see the line of spiritual evolution from animism to magic to religion to e-virtuality. Technologies enable old animistic and magic dreams like television, viewed in macrocosm and microcosm. For instance, an image projector can imitate the cavepainting of Altamira.

Eschatology by Technical Design. The Oxford English Dictionary defines eschatology as «the part of theology concerned with death, judgment, and the final destiny of the soul and of human-kind». There are various eschatological religions and ideologies. We may formulate the question of whether machines can lead to the «end of the world», a final stage in eschatology. This discussion is outside the scope of this paper. Technological singularity is a hypothetical future point in time at which technological growth becomes uncontrollable and irreversible, resulting in unfathomable changes to human civilisation (https://en.wikipedia.org/wiki/Technological_singularity). To disambiguate the different meanings and assumptions, and for an overview of the models of technological singularity, see, for example, the work of Anders Sandberg.²¹

Conclusion. We have presented a schema for the visualisation of phenomena such as law and the differences between verbal writing and pictorial writing. We hold that situational visualisation contributes to the development of the legal machines that govern a certain situation. The point is that symbolisation and visualisation precede formalisation on the way to the representation of legal knowledge. Formalising legal meanings reveals additional goals and makes them explicit. Specifically, open texture in the law can be decreased. Complete knowledge of a regulation in a non-trivial domain is unlikely to be achieved and represented in computers. However, a specific situation can be understood and dealt with by a legal machine.

In the evolution step from humans to machines, a theological problem of the second generation is that God creates the world, humans create machines imago Dei, imago hominis. Status civilis is the frame of modern life. Machines appear in the same context.