Jusletter IT

“Music – what a powerful instrument, what a mighty weapon!“ Maria Augusta von Trapp

This paper uses the case of Emotional Artificial Intelligence (EAI) for music recommender systems to analyse and critique the „risk-based“ approach to AI regulation that underpins the proposed EU Artificial Intelligence Act (AIA). We show that the same EAI technologies can be framed to fall under the categories of “strictly prohibited”, “high risk” or “low risk” in the AIA, raising questions about the suitability of this classification scheme for establishing regulatory obligations.

Consider the following scenario that we will utilise throughout this paper:

After a long day working in one of Salzburg’s oldest restaurants, where he is in charge of preparing a famous sweet soufflé, pastry chef Theodor H. arrives back home, tired and a bit on edge. Upon entering his house, he hears, to his dismay, that his teenage daughter Irma has asked Alexa to play Ludwig’s Hirsch’s “Gruess Gott Salzburg” at full volume in the living room. “Alexa, stop out this crap, immediately! Irma, up into your room, tomorrow is school day!” he yells. “I hate you”, she yells back,“you patron saint of mediocrities”, and slams the door of her room. In the adjacent chamber, little Lina (4) starts crying as the shouting frightens her. At this point, Alexa has analysed the stress levels in the voices of the three protagonists of our domestic drama and developed models of their emotional states. In response, Mozart’s Andante for Flute and Orchestra in C Major, K. 315 starts playing in the living room to calm down Theodor. Hirsch’s “Komm grosser schwarzer Vogel” is piped into Irma’s room, who is in her Emo phase. And the frightened Lina is reassured by the soothing voice of Julie Andrews singing “My Favorite Things”. Soon “the whole world can feel the power of harmony” and peace abides.

This scenario of domestic bliss restored is based on present and near-future technologies, some of them already patented by Amazon for use with Alexa. Of particular interest for this paper is the combination of music recommender systems with emotion recognition. In what follows, we will introduce the relevant technologies and ask if our happy little story is all that there is to this technology or if, as in Wolf Haas’ Silentium, there is a deeply disturbing, and silenced, undercurrent underneath the façade of all conquering music.

EAI draws together affective computing and AI research² to “sense, learn about and interact with human emotional life”³. It is increasingly finding consumer applications across sectors including automated driving, ed-tech, and social media platforms.⁴ Some purported benefits include vehicle safety management through driver monitoring in cars,⁵ wellbeing awareness and even remote diagnosis of mental health conditions⁶. However, EAI also raises significant ethical concerns⁷ and impacts on users’ lives and fundamental rights,⁸ particularly vulnerable groups⁹. To address such risks while responsibly promoting innovation, the Council of Europe framework on AI alongside jurisdictions such as Canada, Brazil, Japan, and the USA have discussed EAI as part of holistic AI regulation.¹⁰ In contrast, the UK government’s AI policy proposal¹¹ does not even mention it. The Joint European Data Protection Board and Supervisor proposal for amendments to the AIA¹² advocate banning EAI,¹³ and the UK data protection regulator, the Information Commissioner’s Office, are similarly wary of this technology.¹⁴ Civil society, NGO and consumer rights bodies have also called prohibiting EAI use.¹⁵

The pseudoscientific basis of EAI, particularly face based emotion modelling systems, has centred on academic/industry use of Ekman and Friesen’s famous universal model of 7 basic emotions (e.g. happy, sad, etc) and cross referencing to facial action coding (e.g. upturned lips, scowling eyes).¹⁶ Increasingly, there has been a turn to understanding context of use as a means of improving accuracy through other data sources such as location or task/activity being undertaken by subjects.¹⁷ Yet, concerns remain around how EAI accounts for cross cultural dimensions of emoting¹⁸, and if experiential emotions can really be computationally read from the body in the first place.¹⁹

The EU approach to regulating AI is ‘technologically neutral’ and ‘risk-based’. The AIA does not start by classifying different AI technologies based on degrees of autonomy, but instead focuses on contexts of use and application domains of AI systems. There are some intuitive appeals to this approach: facial recognition when used by the police poses very different risks to when it helps people draw self-portraits. Yet, this is not an easy task to achieve, particularly for EAI. The severity of EAI-related risks depends on the context in which EAI systems are used. The EU Commission’s pre-AIA inquiry emphasised the need for the risks/threats to be based on a sector-by-sector and case-by-case approach.²⁰ Yet, when we look at the AIA, contextuality considerations are limited and do not cover industries that may initially seem ‘less risky’ or ‘more innocent’ but can have significant risk implications on individuals and society.

Given the growing prevalence of EAI in our lives, we ask whether the EU Commission’s risk-based approach is adequate to protect people against the risks attached to such systems used by the private sector. We aim to answer this question by looking at a prevalently used smart product – Amazon Echo and its “conversational agent”²¹ Alexa – Amazon’s voice AI.²² Alexa’s suitability as a case study lies in its cross-sectoral and multifunctional uses. We focus on the intersection of ‘music’ and ‘emotions’ given users’ high interaction with its “human-like conversation abilities”²³, which can trigger emotional and empathetic reactions in users.²⁴

At first sight, the use of EAI in our example seems benign. Alexa delivers a service the user wants and feels comfortable with – else, they would simply not enable this functionality. Ostensibly, the “decision” the AI makes or sure does not create legal effects concerning them or have a “similarly significant effect” – the trigger of Art 22 GDPR on automated profiling. The system does not appear to profile any “sticky” protected characteristics of the user (such as their gender, sexual orientation, or political opinion); rather, it attempts to identify a transient and fleeting characteristic; how they are feeling in the moment. If the chosen music is not wanted by the user (“Hirsch is so lame”), a simple request will skip to the next song or play one of the user’s explicit choosing. However, once we look at how Amazon’s EAI system works, a more nuanced picture emerges.

To make interaction with Alexa as natural and seamless as possible, the Alexa emotions system utilises Neural Text to Speech (NTTS) technology.²⁵ It offers personalised experiences and recommendations based on its patented technology to monitor users’ emotions.²⁶ It uses voice processing algorithms to “determine the emotional state of the user”²⁷ and responds to how users feel.²⁸ As stated in the patent, Alexa may classify users’ emotions under categories like happy, sad, stress, joy, anger, fear, boredom, and disgust and respond to them with personalised content.²⁹ Emotional states or conditions may be determined based by analysing the pitch, pulse, voicing, jittering, and/or harmonicity of a user’s voice, as determined from the voice data processing. Amazon also allows developers to have the voice assistant respond to users in “a happy/excited or a disappointed/empathetic tone” at various intensity levels and suggests “the tones might be especially effective for gaming and sports Alexa skills”.³⁰ Alexa works across devices and can be connected to other Amazon products in the smart home or health settings. For example, the Halo bracelet analysing “energy and positivity in a customer’s voice so they can better understand how they may sound to others, helping improve their communication and relationships”.³¹ It enables features such as synchronisation of devices by recommending music on the ‘Echo Show’ smart home screen from different platforms including Amazon Music, Spotify, or Pandora. The voice-recognition patent also enables making music recommendations and sharing music with others in a user’s contact list, and even allowing users to speak to Alexa to find music matching their activity (e.g., cooking, workout), which aligns with the Amazon Echo smart speaker capabilities.

Controversies about EAI are mainly driven by the fear that reading ‘emotions’ could lead to a world where AI systems create risks of surveillance, inaccurate results, manipulation, or commercial exploitation. These concerns are arguably relevant to any EAI. Music is known for its potential to affect emotions³², cognitive development³³, decision-making functions³⁴, and behaviours³⁵. It links with mental health and behavioural disorders – triggering healing or causing potential harm.³⁶ So, we next turn to specific concerns around Amazon’s Alexa and EAI interacting with music industry actors. How can the ‘music factor’ help us observe the potential shortcomings of the AIA’s technology-neutral risk-based approach in mitigating risks relating to EAI used by the private sector?

In the absence of risk and harm-related definitions in the AIA, we adopt a multidisciplinary approach when assessing the severity of the possible harms/risks in the context of EAI-related risks in Alexa. This section gives an overview of Alexa’s voice recognition technology and EAI. It then underscores potential risks attached to its use, focusing on the ‘music’ factor, and asking how a multidisciplinary look at music theory and risks related to ‘emotion’ and ‘music’ can inform our assessment of the current technology-neutral risk-based approach in the AIA.

The advent of speech recognition systems started in the 1950s and paved the way for today’s AI-driven world, where voice recognition technology is used in various sectors. Today, Amazon is one of many companies targeting users’ voices to infer emotions using AI based biometric recognition technologies. This feeds into recommender system algorithms and wider smart home ecosystems which can deliver results of analysis e.g. playing specific songs.³⁷ As advertisers always knew, and research shows, audio impacts our emotions³⁸, which is why the music we hear in supermarkets is carefully chosen. However, this approach did not profile individuals but made generic assumptions about the impact of music on shopping behaviour. Emotion recognition, the inverse relation where emotion impacts the audio-visual signals we create, can then be used to identify a person’s emotional state and tailor the interventions specifically to them.³⁹

Amazon claims to provide benefits for customers by detecting the tone of voice and using voice biometrics and emotion recognition to enhance users’ experience, among many other personalisation features, such as frustration detection, to improve the customer experience.⁴⁰ In general, such features aim to improve users’ experience by providing them with a better-personalized experience. Amazon notes, “Alexa makes your life easier, more meaningful, and more fun by letting you voice-control your world.”⁴¹ Amazon’s patent states that voice-based determination of emotional characteristics of users is used to “facilitate presentation of timely and relevant content to users leading to an increased ability to determine a user intent and/or anticipate a user’s needs or desires.”⁴² The users’ intent can be used to determine audio content (e.g., audio advertisements or other audio content) that is “highly relevant and timely to the user’s current desires and situation.”⁴³

Music can evoke emotions. The link between music and emotions is indeed a very powerful one, as the quote at the beginning of this paper, by the real-life Maria of Sound of Music fame, highlighted.⁴⁴ Philosophers, sociologists, anthropologists, and psychologists have all described the significance of music for our emotional states.⁴⁵ For instance, in philosophy, music’s emotional expressiveness is seen as “a philosophical problem since the paradigm expressers of emotions are psychological agents”.⁴⁶ The link between music and emotions also resonates in politics, as music has been known to be used as a device to control and influence behaviour since the earliest days⁴⁷. Plato, for this reason, argued for tight legal restrictions on their use: “The guardians of the state must throughout be watchful against innovations in music and gymnastics counter to the established order, and the best of their power guard against them, fearing when anyone says that “That song is most regarded among men/Which hovers newest on the singer’s lips”⁴⁸. Numerous examples show how music was used to control people through manipulating emotions.⁴⁹ Music has always been used as a persuasion tactic in political campaigns to influence voters.⁵⁰ When used by those in power for malicious purposes, it can also have devastating consequences for individuals and society. Reis argues how music can be exploited politically by focusing on one emotion only, sentimentality, and gives various examples from history, starting from Plato’s Republic until Brexit.⁵¹ Music is also argued to impact social change.⁵²

Beyond its effect on emotion manipulation, music can have a significant impact on one’s learning⁵³, socio-emotional skills⁵⁴, psychological functions⁵⁵, behaviour⁵⁶, mental health⁵⁷, and cognitive development⁵⁸. In medicine, research suggests music can impact physical health, including recovery after operations.⁵⁹ Music can also play a role in the improvement of vulnerable groups’ emotion recognition impairments⁶⁰ and executive functioning by activating “cortical and subcortical brain areas, including the prefrontal cortex”, which is linked to impulsive behaviours.⁶¹ Recent research shows that the tempo of music increases the level of excitement feelings and pleasure.⁶² A developed music generation technology was found to be capable of producing a wide range of emotions in the listener, showing that “algorithmic composition system is able to compose music in real-time to target specific emotions”.⁶³ As much as music can be a source of happiness, it can also trigger negative emotions. For example, music can have a negative impact on how people react in depression⁶⁴, leading to aggressive behaviour, even a potential link with self-harm⁶⁵ as well as risky driving behaviours⁶⁶.

Music‘s power on consumer behaviour is well-recognised and used by private sector actors in different industries.⁶⁷ This is not surprising given that listening to music can make people more social, resulting in incidental happiness that triggers probability weighting during risky choices.⁶⁸ It can evoke strong emotions and remembrance of memories.⁶⁹ Enström and Schmaltz suggest that “listening to music impacts consumer risk-taking likelihood”.⁷⁰ Lerner et al. state, “emotion and decision-making go hand in hand”.⁷¹ Since music is suggested to impact decision-making and increase or decrease willingness to take risks⁷², such an effect may potentially involve “manipulation” that triggers risky behaviour. The risks related to this manipulation become more relevant for vulnerable groups such as people with depression.⁷³ The impulsivity and risky behaviours of music could harm the individual also financially, though this harm is currently not recognised in the same way as “physical and psychological harm” under the AIA. Chamorro-Premuzic and Furnham suggest that individual differences in personality and cognitive ability may partly determine how we experience music.⁷⁴ Relatedly, our music can help construct self-identity.⁷⁵ As Schulreich et al. note, “happy people made more optimistic probabilistic judgments and sad people more pessimistic judgments,” and music-evoked incidental emotions can trigger people’s risk attitudes.⁷⁶

With this groundwork in place, we can now revisit our initial, seemingly harmless example of Alexa identifying a user’s emotional state and choosing appropriate music for them. By combining the emotional state of the user with other contextual data Alexa has about them, it seems entirely feasible, e.g., to influence a user’s voting behaviour if, shortly before or on election day, music is chosen that induces feelings of patriotism. For example, some of the most well-known cases are national anthems used before elections or marching music to stir emotions against an enemy.⁷⁷ Music that is energetic and may lead to anger may be more likely to push a listener to vote (against the system) than music that leads to a feeling of passive pleasure. Closer, arguably, to Alexa’s business model, EAI, together with the information that the user is in the context of making a purchase decision, could be used to increase optimism (including optimism about one’s finances), in a gaming context, they could be used to increase the willingness to take unreasonable risks.⁷⁸

The above section explores how impactful ‘music’ can be on people’s decision-making, behaviours, and emotions, with scope for ‘manipulation’, and impacting ‘health’. We now explore the AIA risk-based approach in further detail. As mentioned, the AIA adopts a technology-neutral risk-based approach and introduces four risk categories for AI systems: 1) unacceptable risk, 2) high risk, 3) limited risk, and 4) minimal risk.⁷⁹ For each category, different obligations and rules are imposed on relevant stakeholders across the AI supply chain, from manufacturers and users to importers and distributors. For example, significant fines can be levied if a prohibited system is placed on the market, or compliance with a series of technical design and development requirements can be required for high-risk system before being certified and sold in the EU.

The AIA has provisions on biometric AI systems which often includes EAI.⁸⁰ Despite the weight given to biometrics in several of its provisions, the AIA does not generally prohibit EAI systems or consider them across the board as ‘high-risk.’ Rather, it depends on the sector that uses them and the concrete context of use. EAI used by law enforcement is high risk and comes with significant duties; those used for gaming purposes are low risk and carry only weak transparency obligations. The unknowns about the impact of AI systems make it more difficult to identify and measure risks/harms which complicates whether an AI system meets the thresholds in ‘prohibited’ and ‘high-risk’ categories. Yet, as the Alexa example shows, risks relating to EAI can be better understood when contextual factors such as ‘music’ and ‘emotions’ are considered. Accordingly, it can be argued that severity and probability elements in the AIA should be assessed by considering the available research that shows the strong connection music has with manipulation and health specifically. Since Alexa’s use is multifunctional and is connected to several apps and services like other smart home devices, it becomes more concerning whether and how this cross-sectoral and multidimensional use can affect the likelihood and severity of the risks Alexa’s use can have in everyday life.

Some forms of manipulative EAI systems could fall under the prohibition category if, in addition to the manipulation, they cause “physical or psychological harm”⁸¹. The music factor can be a catalyst for making Alexa’s EAI not just reading one’s emotions but changing their emotions, decision-making, and behaviour. Manipulation, deception, and societal harm are the critical focus in this context. Recital 40(a) provides, “Certain AI systems should at the same time be subject to transparency requirements and be classified as high-risk AI systems, given their potential to deceive and cause both individual and societal harm. In particular, AI systems that generate deep fakes […]” Even though Recital 40(a) does not recognize EAI explicitly, this does not necessarily mean that such systems do not fall under its scope. The word choice of “in particular” suggests that the scope is not limited to the circumstances explicitly provided under Recital 40(a). The relative flexibility this wording provides is highly relevant to Alexa’s EAI and music recommendations may potentially result in both individual and societal harms, given the powerful impact music has had on emotions, decisions, and actions of people and societies throughout history. Nevertheless, as the discussion above shows, selecting music to trigger emotions that lead to action is a paradigmatical case of such subliminal manipulation – by addressing the emotions directly, it circumvents the rational decision-making process the law requires. Less clear is if the variations of our case study also leading to recognisable harm. ‘Harm’ is an essentially contested concept and has no clear borders.⁸² Music that leads the listener to man the barricades because what they hear is the song of angry man, the music of the people who will not be slaves again, then this could arguably lead to physical harm. The connection between music, emotion, and mental illness may also give rise to a risk for mental health.

For the high-risk category, the AIA defines this in a number of ways. There are definitions in Art 6, an AI system is either a safety component in its own right or a safety system within a technology that is covered by EU delegated legislation (e.g., cars, planes, radio equipment). It can be high risk if listed in Annex III which covers the various contexts where AI is risky by virtue of where it is being used, irrespective of if it is for a safety application or not. As noted, EAI is on Annex III when used by law enforcement or border or migration agencies. However, when it does not clearly fit into those criteria, the AIA notes systems can still be high risk if they pose risks related to health, safety, and impact fundamental human rights and freedoms. Knowing when these high risk criteria are met remains unclear, leading Edwards to state the AIA lacks justifiable and reviewable criteria in categorising AI systems as ‘high risk’ and therefore is currently ‘arbitrary.’⁸³ Nevertheless, we can consider Art 7(2) for the methodology that should be used in identifying if systems that are not captured under Art 6 as high risk do in fact pose risks to health/safety/fundamental rights at a level equivalent to those already in Annex III. It points us to other elements beyond the severity of harm, probability of its occurrence and use in pre-defined areas. This includes the intensity of harm/adverse impacts, the level of dependency of impacted individuals on the output of the system, the ability of those impacted to opt out from the system, the vulnerability of those subject to AI based on imbalances of ‘power, knowledge, economic or social circumstances, or age’ and degree of reversability of the outcome, where impacts to health and safety should be seen as less reversable.

For example, manipulating voting behaviour, as mentioned above, could be seen as an attack on “fundamental rights”, but then again, a person is not denied vote; they may just make them less (or more) willing to vote against their best interest. Yet, it is unclear and difficult to determine what meets the ‘severity’ and ‘probability’ criteria. Emotions are fleeting, so Alexa might determine that a user is sad now and make music recommendations accordingly. Then these recommendations might change depending on the mood of the user or the environment they are in. Whilst this could be problematic for matters such as ‘surveillance’, it is not a permanent trait of the user and thus could make the probability of harm element harder to show. But if EAI recommended music has impacts on users’ mental health, for example a song triggering a painful memory, the degree of reversability could be questionable. Similarly, has this affective manipulation become such a key part of their daily routine and therapeutic daily healing, that the reliance becomes irreversible? Further, given the issue of asymmetries that the AIA is concerned about, EAI could “provide novel and powerful tools for manipulative, exploitative and social control practices” that the law seeks to guard against.⁸⁴ Where it gets harder to argue is around intensity of adverse impacts, dependency of the individual on the music and ability to opt out. These are entertainment systems they are not forced to use, and could switch off, albeit the user loses access to the ecosystem they have invested into. Similarly, it is not mediating their access to benefits, or other critical public services, so similarly whilst it could be argued manipulation could make listener feel it is hard to opt out or they have a degree of dependency on the EAI recommendations, this is again contextual. In our example, is familial bliss dependent on Alexa? Does the technology mediate their familial relations to such an extent they cannot possibly opt out, as they are being so manipulated through their music choices?

Our Alexa example could equally be argued to fall under ‘limited risk’, ‘high risk’, or ‘prohibited categories’. In this sense, it is not more sinister, even in the worst case, than the type of manipulation typical for advertising through music in the physical world. Music‘s powerful impact on listeners and their choices has been shown in many studies across different disciplines. This paper first gave an overview of Alexa’s voice recognition technology and EAI. It then underscored potential risks attached to its use, focusing on ‘music’ theory and asking how a multidisciplinary look at risks related to ‘music’ can inform an assessment of the current risk-based approach in the AIA. The AIA adopts a risk-based approach considering the impact of AI systems on people and does not necessarily focus on specific technologies or contextual factors like the ‘music factor’ in manipulation of users. After considering the music theory discussions, we show the powerful impact music can have on emotions, individuals’ decision-making, behaviours, and even health. We conclude that contextual aspects of EAI use deserve particular attention but the problem with the AIA is its risk approach in general, not necessarily its categories, as it introduces uncertainty about which category AI systems like Alexa should fall under. The above scenarios and discussion show that music can have different impacts and risks of manipulation and health depending on the circumstances, meaning Alexa’s EAI can fall under different risk categories in the AIA. As such, we conclude that the main problem with the AIA’s risk-based approach and categorisations is the underlying assumption that risks relating to the use of AI systems can be done with a ‘technology neutral’ approach; instead, we have to accept that there are no inherently safe applications of EAI.

AMAZON (2020). „Introducing Amazon Halo and Amazon Halo Band–A New Service that Helps Customers Improve Their Health and Wellness.“ Amazon Press Center, from https://press.aboutamazon.com/news-releases/news-release-details/introducing-amazon-halo-and-amazon-halo-band-new-service-helps.

AMAZON (2022). „Alexa features.“ https://www.amazon.com/b?ie=UTF8&node=21576558011 Accessed 21 September 2022.

Baker, F. and W. Bor (2008). „Can Music Preference Indicate Mental Health Status in Young People?“ Australasian Psychiatry 16(4): 284–288.

Bartsch, A. and F. M. Schneider (2014). „Entertainment and Politics Revisited: How Non-Escapist Forms of Entertainment Can Stimulate Political Interest and Information Seeking.“ Journal of communication 64(3): 369–396.

Bishop, D. T., C. I. Karageorghis and G. Loizou (2007). „A grounded theory of young tennis players‘ use of music to manipulate emotional state.“ Journal of sport & exercise psychology 29(5): 584–607.

Bloor, Andrew (2009). „The rhythm‘s gonna get ya’ – background music in primary classrooms and its effect on behaviour and attainment.“ Emotional and Behavioural Difficulties 14(4): 261–274.

Brader, T. (2005). „Striking a Responsive Chord: How Political Ads Motivate and Persuade Voters by Appealing to Emotions.“ American Journal of Political Science 49(2): 388–405.

Brauer, J. (2016). „How Can Music Be Torturous?: Music in Nazi Concentration and Extermination Camps.“ Music and Politics 10(1).

Brown, S. and U. Volgsten (2006). Music and Manipulation: On the Social Uses and Social Control of Music, Berghahn Books.

Caldwell, C. AND S. A. Hibbert (1999). „Play that one again: The effect of music tempo on consumer behaviour in a restaurant.“ ACR European Advances.

Carolus, A., C. Wienrich, A. Törke, T. Friedel, C. Schwietering AND M. Sperzel (2021). „‘Alexa, I feel for you!’ Observers’ Empathetic Reactions towards a Conversational Agent.“ Frontiers in Computer Science 3.

Chamorro-Premuzic, T. AND A. Furnham (2007). „Personality and music: can traits explain how people use music in everyday life?“ Br J Psychol 98(Pt 2): 175–185.

Coyne, S. M. AND L. M. Padilla-Walker (2015). „Sex, violence, & rock n‘ roll: Longitudinal effects of music on aggression, sex, and prosocial behavior during adolescence.“ Journal of Adolescence 41: 96–104.

Crawford, K. (2021). „AI Prof Sounds Alarm: AI „Emotion Detectors“ Are Faulty Science.“ https://mindmatters.ai/2021/04/ai-prof-sounds-alarm-ai-emotion-detectors-are-faulty-science/ Accessed 26 September 2022.

Croom, A. M. (2012). „Music, neuroscience, and the psychology of well-being: a précis.“ Frontiers in psychology 2: 393.

Daly Ian et al. (2021). „Our brain-computer interfacing technology uses music to make people happy.“ https://theconversation.com/our-brain-computer-interfacing-technology-uses-music-to-make-people-happy-119496 Accessed 21 October 2022.

DEPARTMENT FOR DIGITAL CULTURE MEDIA & SPORT, OFFICE FOR ARTIFICIAL INTELLIGENCE AND DEPARTMENT FOR BUSINESS ENERGY & INDUSTRIAL STRATEGY. (2022). Establishing a pro-innovation approach to regulating AI (Policy paper).

Deshpande, M. AND V. Rao (2017). Depression detection using emotion artificial intelligence. 2017 International Conference on Intelligent Sustainable Systems (ICISS).

Dumont, E. et al. (2017). „Music Interventions and Child Development: A Critical Review and Further Directions.“ Frontiers in Psychology 8: 1694.

Durovic, M. and Watson J. (2021). „Nothing to Be Happy about: Consumer Emotions and AI.“ Multidisciplinary Scientific Journal 4(4): 784–793.

EDRI, European Disability Forum, Bits of Freedom, Fair Trials, AccessNow, Panoptykon Foundation, PICUM, Epicenter Works for Digital Rights, Algorithm Watch and ANEC (2021). An EU Artificial Intelligence Act for Fundamental Rights – A Civil Society Statement.

Edwards, L. (2022). Expert opinion: Regulating AI in Europe: four problems and four solutions, Ada Lovelace Institute.

Enstrom, R. and Shmaltz R. (2017). „A Walk on the Wild Side: The Impact of Music on Risk-Taking Likelihood.“ Front Psychol 8: 759.

EUROPEAN COMMISSION. (2021). Explanatory Memorandum of the AIA.

EUROPEAN COMMISSION. (2021). Regulatory framework proposal on artificial intelligence.

EUROPEAN COMMISSION (2021). Study supporting the impact assessment of the AI regulation – Final report (D5). Brussels.

Foran, L. M. (2009). „Listening to music: Helping children regulate their emotions and improve learning in the classroom.“ Educational Horizons 88(1): 51–58.

Furey E. and Blue JH (2018). Alexa, emotions, privacy and GDPR. Proceedings of the 32nd International BCS Human Computer Interaction Conference. Belfast, United Kingdom, BCS Learning & Development Ltd.: Article 212.

Fussell, S. (2018). Alexa Wants to Know How You’re Feeling Today. The Atlantic.

Gabrielsson, A. (2016). The Relationship between Musical Structure and Perceived Expression. The Oxford Handbook of Music Psychology. S. Hallam, I. Cross and M. H. Thaut, Oxford University Press: 141–150.

Gao, C. (2019). „Use New Alexa Emotions and Speaking Styles to Create a More Natural and Intuitive Voice Experience.“ Alexa Skills Kit Blog https://developer.amazon.com/en-US/blogs/alexa/alexa-skills-kit/2019/11/new-alexa-emotions-and-speaking-styles Accessed 25 September 2022.

Garrido, S. and Schubert E. (2015). „Music and People with Tendencies to Depression.“ Music Perception 32(4): 313–321.

Gonzales Fuster and Peeters M. (2021). „Person identification, human rights and ethical principles – Rethinking biometrics in the age of artificial intelligence.“ STOA Briefing, from https://www.europarl.europa.eu/RegData/etudes/STUD/2021/697191/EPRS_BRIE(2021)697191(ANN)_EN.pdf.

INFORMATION COMMISSIONER‘S OFFICE (2022) Biometrics Foresight Report.

Herman, A. et al. (2018). „Risk-Taking and Impulsivity: The Role of Mood States and Interoception.“ Front Psychol 9: 1625.

Hesmondhalgh, D. (2014). Towards a critical understanding of music, emotion and self-identity. Production & consumption of music, Routledge: 105–119.

Hole J. et al. (2015). „Music as an aid for postoperative recovery in adults: a systematic review and meta-analysis.“ The Lancet 386(10004): 1659–1671.

Holt, K. (2019). Amazon‘s Alexa is about to get more emotional. Forbes.

Jin H. And Wang S. (2018). Voice-based determination of physical and emotional characteristics of users, Amazon Technologies Inc.

Johnson, K. (2017). „Amazon’s Alexa wants to learn more about your feelings.“ VentureBeat https://venturebeat.com/ai/amazons-alexa-wants-to-learn-more-about-your-feelings/ Accessed 7 October 2022.

Kania, A. (2017). „The Philosophy of Music.“ The Stanford Encyclopedia of Philosophy https://plato.stanford.edu/entries/music/ Accessed 12 September 2022.

Karageorghis, S. et al. (2021). „Influence of music on driver psychology and safety-relevant behaviours: a multi-study inductive content analysis.“ Theoretical Issues in Ergonomics Science: 1–20.

Konencni, V. J. (2012). „Constraints on manipulation of emotions by music.“ Philosophy Today 56(3): 327.

Lerner, J. et al. (2015). „Emotion and Decision Making.“ Annual Review of Psychology 66(1): 799–823.

Lim, W. et al. (2022). „Alexa, what do we know about conversational commerce? Insights from a systematic literature review.“ Psychology & Marketing 39(6): 1129–1155.

Lin C. et al. (2012). User identification design by fusion of face recognition and speaker recognition. 2012 12th International Conference on Control, Automation and Systems.

Lin S. et al. (2011). „Mental Health Implications of Music: Insight from Neuroscientific and Clinical Studies.“ Harvard Review of Psychiatry 19(1): 34–46.

Lopatovska, I. et al. (2021). User Perceptions of an Intelligent Personal Assistant‘s Personality: The Role of Interaction Context. Proceedings of the 2021 Conference on Human Information Interaction and Retrieval. Canberra ACT, Australia, Association for Computing Machinery: 15–25.

Lopatovska, I. AND H. Williams (2018). Personification of the Amazon Alexa: BFF or a Mindless Companion. Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. New Brunswick, NJ, USA, Association for Computing Machinery: 265–268.

Lo, M. (2001). „The use of music to decrease agitated behaviour of the demented elderly: the state of the science.“ Scandinavian Journal of Caring Sciences 15(2): 165–173.

Martin, G. et al. (1993). „Adolescent Suicide: Music Preference as an Indicator of Vulnerability.“ Journal of the American Academy of Child & Adolescent Psychiatry 32(3): 530–535.

Mcstay, A. AND L. Urquhart (2021). „In Cars (Are We Really Safest of All?): Interior Sensing and Emotional Opacity.“ SocArXiv.

Milliman, R. E. (1982). „Using Background Music to Affect the Behavior of Supermarket Shoppers.“ Journal of Marketing 46(3): 86–91.

Mohammadi, Z.J. et al. (2016). „Wavelet-based emotion recognition system using EEG signal.“ Neural computing & applications 28(8): 1985–1990.

Neuschwander, D. (2012). „Music in the Third Reich.“ Musical Offerings 3(2): 93–108.

Nicolaou, N. et al. (2017). „Directed Motor-Auditory EEG Connectivity Is Modulated by Music Tempo.“ Front Hum Neurosci 11: 502.

Olsen, K et al. (2022). „Psychosocial risks and benefits of exposure to heavy metal music with aggressive themes: Current theory and evidence.“ Current Psychology.

Petrovica, A. et al. (2017). „Emotion Recognition in Affective Tutoring Systems: Collection of Ground-truth Data.“ Procedia Computer Science 104: 437–444.

Phillips, P. et al. (2000). „An introduction evaluating biometric systems.“ Computer 33(2): 56–63.

Picard, R. (1995). Affective Computing. M.I.T Media Laboratory Perceptual Computing Section Technical Report No. 321.

Rabinowitch, T.-C. (2020). „The Potential of Music to Effect Social Change.“ Music & Science 3: 2059204320939772.

Redondo Pedregal, C. AND P. Heaton (2021). „Autism, music and Alexithymia: A musical intervention to enhance emotion recognition in adolescents with ASD.“ Research in Developmental Disabilities 116: 104.

Reis, J. (n.d.). Music, Sentimentality and Political Manipulation, The University of Texas at Austin.

Ringeval, F., et al. (2019). AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop. Nice, France, Association for Computing Machinery: 3–12.

Ritblatt, S. et al. (2013). „Can Music Enhance School-Readiness Socioemotional Skills?“ Journal of Research in Childhood Education 27(3): 257–266.

Ruud, E. (1997). „Music and identity.“ Nordisk Tidsskrift for Musikkterapi 6(1): 3–13.

Saarikallio, S. AND J. Erkkilä (2007). „The role of music in adolescents‘ mood regulation.“ Psychology of Music 35(1): 88–109.

Sarkamo, T. et al. (2014). „Structural changes induced by daily music listening in the recovering brain after middle cerebral artery stroke: a voxel-based morphometry study.“ Front Hum Neurosci 8: 245.

Schäfer, T. et al. (2013). „The psychological functions of music listening.“ Front Psychol 4: 511.

Scherer, K. (2004). „Which Emotions Can be Induced by Music? What Are the Underlying Mechanisms? And How Can We Measure Them?“ Journal of New Music Research 33(3): 239–251.

Schulreich, S. et al. (2014). „Music-evoked incidental happiness modulates probability weighting during risky lottery choices.“ Frontiers in Psychology 4.

Sedenberg, E. AND J. Chuang (2017). „Smile for the Camera: Privacy and Policy Implications of Emotion AI.“

Sloboda, J. A. AND P. N. Juslin (2001). „Psychological perspectives on music and emotion.“ Music and emotion: Theory and research: 71–104.

Soderlund, G. et al. (2010). „The effects of background white noise on memory performance in inattentive school children.“ Behav Brain Funct 6: 55.

Sprague, A. (2022). „Audio’s impact on emotions: New study on the science of brand building with sound.“ Amazon https://advertising.amazon.com/blog/how-audio-can-impact-consumers-moods Accessed 21 September 2022.

Street. J. (2003). „‘Fight the Power’: The Politics of Music and the Music of Politics.“ Government and Opposition 38(1): 113–130.

Urquhart, L. et al. (2020). Comment on Children’s Rights in Relation to Emotional AI And the Digital Environment.

Van Der Hof S. And Malgieri G. (2022). „The draft AI Act and children: Room for improvement.“ leidenlawblog

Watermen, S. (2019). National Anthems and National Symbolism: Singing the Nation. Handbook of the Changing World Language Map. S. D. Brunn and R. Kehrein. Cham, Springer International Publishing: 1–16.

Wesseldijk, L. W., F. Ullén and M. A. Mosing (2019). „The effects of playing music on mental health outcomes.“ Scientific Reports 9(1): 12606.

WHITE HOUSE (2022). Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People.

Wiederhold, B. K. (2018). „“Alexa, Are You My Mom?” The Role of Artificial Intelligence in Child Development.“ Cyberpsychology, behavior and social networking 21(8): 471–472.

Wilde, E. M. (2018). Music, Education and ADHD: An exploratory multiple case study. PhD, UCL Institute of Education.

Ziv, N. (2016). „Music and compliance: Can good music make us do bad things?“ Psychology of music 44(5): 953–966.