Emotions matter. They are at the core of human experience, shaping our lives in the most profound ways and triggering our decision-making and social behaviour. Evolutionary theories indicate that emotions have important survival functions and help us solve certain challenges during our evolution. In social and business sciences, emotions are associated with human irrationality and non-logical behaviour. Affective computing deals with studying and developing digital systems and devices that can recognize, interpret, process, and simulate human affects. The new interplay of research findings from consumer behaviour, psychology, neuroscience and computer science opens up a new research universe and is attracting more and more scientists. Measuring human emotions with cutting-edge and AI-based technology supports researchers on their mission to further explore these new worlds and set up experimental projects.
Integrate AI-based Emotion Analytics into your research
- Elevate your research with innovative data collection and emotion analysis techniques, moving beyond traditional methods like self-reporting, observation, and interviews. Embrace the digital age with these novel approaches.
- Easily integrate video recordings into your research methodology. Utilize these recordings as the foundation for advanced emotion analysis powered by AI algorithms.
- All collected data is available in .csv format, ready for in-depth processing and statistical analysis in your preferred software, including R, SPSS, Python, and more.
- Break free from the limitations of self-reporting surveys. Emotion AI enables the design of unique experiments to accurately gauge emotional responses, bypassing the biases of self-assessment. This approach allows for real-world, natural environment data collection.
- Surpass traditional interviews and observations. Leverage the accuracy of facial coding, computer vision, and machine learning in understanding human emotions, achieving a level of insight comparable to that of expert psychologists and observers.
Highlighted Research Projects Utilizing TAWNY Technology
Effect of nudging in online-shopping environments regarding the consumption of fast food. Nudges are nowadays commonly used by companies and governments to prompt consumer choices in online and offline settings. However, the reasons for the effectiveness of specific characteristics in nudges are still scarce. The research aims to understand the effectiveness and impact of nudges to reduce caloric intake. The avatar-based nudges displayed either a face of dissatisfaction when the participant chooses a meal over the calorie intake goal, or a facial expression of satisfaction when the calories stay within the recommended goal. During the whole experiment, the participants' facial expressions were recorded and the triggered user emotions were measured with TAWNY technology to answer the question of what made the avatar more effective than the other nudge options.
Deep Flow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks. The affective state called flow is described as a state of optimal experience, total immersion and high productivity. As an important metric for various scenarios ranging from (professional) sports to work environments to user experience evaluations, it is extensively studied using traditional questionnaires. To make flow measurement accessible for online, real-time environments TAWNY technology was used in an experimental setting to automatically estimate a user’s flow state based on physiological signals measured with a wearable device. We conducted a study of subjects playing the game Tetris in varying difficulty levels, leading to boredom, stress, and flow. Using a convolutional neural network, we achieve an accuracy of 70% in recognizing flow-inducing levels. In the future, we expect flow to be a potential reward signal for human-in-the-loop reinforcement learning systems.
How to Stay in Flow at Work – An Experimental Approach. To investigate the occurrence of flow during work as well as its connection to performance and physiological parameters, 59 subjects performed a simulated work task in an experiment. The participants had to retype different texts on a laptop. These texts had three different difficulty levels: simple (children’s stories), medium and hard (chemistry science text), intending to induce three different performance and flow levels as well as physiological activations. Next to video data of the test persons the heart rate, R-R intervals and skin conductance were measured with the TAWNY setup.
The Impact of Ad Breaks Tested on an Online Video Platform. A test and control group (n=33) was shown a trailer with/without commercial breaks. Arousal and valence of the test persons were measured with the Tawny emotion scan. Significant interaction effects between the ad and the environment were found. This brings the discussion about the advertising environment back to the foreground, which has been neglected for a long time but is gaining relevance through the common practice of programmatic buying for the automated placement of advertising media in online environments.
Researching mystery deals’ impact on consumer purchase motivation & loyalty. Researchers from the University of Passau used the TAWNY Emotion Analytics platform in an experiment to analyze people’s facial expressions when they receive and unbox a mystery deal, in addition to a verbal interview. The task was to understand which affective reactions are triggered by Mystery Deals when consumers first encounter the deal (pre-purchase phase) and, especially, after the mystery is revealed (post-purchase phase). The research also investigated how affective reactions drive purchase motivation and brand loyalty.
Emotion AI: New Forms of AI-powered Emotion Analytics. In this paper, we built on existing emotion theories and explained the concept of emotional artificial intelligence. In particular, the focus of this paper was to compare traditional and modern AI-based methods for emotion recognition. For this purpose, using an experiment in online shopping, we compared a traditional self-report method with our automated and AI-based emotion recognition software, which was equally used to measure user experience. We also showed how, especially in market and consumer research, there are numerous possible applications for Emotion AI technology.
EROC: Emotional Reactions on Creatives. The EROC test sets new standards in the evaluation of out-of-home advertising media. Through the novel and unique combination of classic advertising media KPIs, such as enforcement and likeability, and the implicit emotion measurement - powered by TAWNY Emotion AI - EROC enables a holistic assessment of the impact of the advertising media. Thus, EROC captures the consumers' conscious and unconscious reactions to the creative.
Investigating the Effect of Emotions on Investment Decisions in Crowdfunding. This thesis examines how emotions affect investment decisions in crowdfunding, differentiating between reward-based and equity-based models. It focuses on the vital role emotions play in investment behavior, especially in reward-based crowdfunding, due to a lower sense of risk than equity-based crowdfunding. The research takes an empirical approach, analyzing how individual investors' emotional responses to specific stimuli influence their investment actions using Tawny. The findings reveal that emotions significantly guide decision-making processes and have a predictive capacity in determining investment choices, providing important insights for stakeholders in the crowdfunding ecosystem and suggesting future research in the field.
Wie KI helfen kann, das Überspringen von Werbung vorherzusagen. The article discusses how AI and Affective Computing can predict ad-skipping behavior, particularly in online environments. It emphasizes the role of emotional reactions in user responses to intrusive online advertising. The research, involving a partnership between Tawny and Hochschule Fresenius München, utilized Facial Coding to analyze emotional reactions to video ads on platforms like YouTube. The study showed a higher likelihood of skipping ads with negative emotional responses, especially for low-quality ads, highlighting the potential of Affective Computing in revolutionizing market research and advertising strategies.
Facial expressions predict idea evaluation – AI Can Tell From Your Face if You Like an Idea This research article explores how affective reactions to idea pitches can predict idea evaluations and investment intentions. It focuses on the role of emotions in the innovation process, often overlooked by companies. The study involved 60 participants from Germany and Austria who watched product pitches to improve a major electronics chain's positioning in the well-being sector. The study analyzed participants' facial expressions using Tawny to gauge affective responses. The results indicate that while individual valence and arousal could not predict idea evaluations or investment intentions, their interaction could. Additionally, the study reveals the importance of emotional expressivity and mood as moderators in this process, offering new insights into how affect influences idea evaluations, investment intentions, and attitudes.
Publications
Barmann M. J. (2023): Investigating the effect of emotions on investment decisions in crowdfunding: an empirical comparative study of reward-based and equity-based crowdfunding.
Bartl M. and Füller J. (2020): The Rise of Emotion AI: Decoding Flow Experiences in Sports, In: 21st Century Sports: How New Technologies Change Sports in the Digital Age, Springer.
Bartl M. (2019): The Missing Piece in New Work: Decoding Human Flow at the Workplace, In: The Making-of Innovation, November 2020
Bartl M. (2019): The Rise of Emotion AI, In: Handbuch Künstliche Intelligenz.
Bartl M. (2018): Von der künstlichen zur emotionalen Intelligenz und was das für die Marktforschung bedeutet, In: marktforschung.de 27.06.2018
Baur M. (2021): Increasing Robustness of Facial Expression Recognition against Speech. (Master‘s Thesis | LMU)
Binder K., Goliasch A., Kolmann A., Ruhl M., Tiemeier T. (2021) AI-Powered Emotion Analytics in the Context of Billboard Marketing (Projektarbeit)
Eick E., Dierks S., Garrecht G., Tiemeier T. (2021): Die Auswirkung von Werbeunterbrechung im Action-Trailer auf Arousal, Valenz und unterschiedliche Emotionen unter Einbezug des TAWNY-Emotionserkennungstools
Hahn A., Bartl M., Klug K. (2020): Digital Empathy: Wie Künstliche Intelligenz und Affective Computing die Marktforschung verändern, In: Dem Konsumenten auf der Spur: Erfolgreiches Marketing durch zeitgemäße Marktforschung. Pusler M. (Hrsg.) Freiburg: Haufe.
Hahn A., Maier M. (2018): Affective Computing – Potential für empathisches digitales Marketing. In: Marketing Review. St. Gallen
Maier M., Elsner D., Marouane C., Zehnle M., Fuchs C. (2019): DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks, In: International Joint Conference on Artificial Intelligence (IJCAI 2019).
Marichalar Quezada R (2020): Effect of nudging in online-shopping environments regarding the consumption of fast food (Master’s Thesis | TUM)
Marichalar Quezada R., Bartl M., Garrecht G. (2022) Emotion AI: Neue Formen der Emotionsmessung durch Künstliche Intelligenz, In: Künstliche Intelligenz erfolgreich umsetzen – Praxisbeispiele für integrierte Intelligenz. Lichtenthaler U. (Hrsg.) Wiesbaden: Springer Gabler.
Nataraj A., Chellew C., Balaji D., Bilbao I., Frei J., Peponnet S. (2021): An Analysis of how the matching of emotions between online video content and online ads can mitigate the negative effects of ad interruptions (Student Project)
Richter D. and Bartl M. (2018): Affective Computing Applied as a Recipe Recommender System, In: Computational Social Science in the Age of Big Data. Herbert von Halem Verlag Cologne.
Rupp J., Füller J., Hutter K. (2023). Facial Expressions Predict Idea Evaluation – AI Can Tell From Your Face if You Like an Idea. In Academy of Management Proceedings (Vol. 2023, No. 1, p. 18143). Briarcliff Manor, NY 10510: Academy of Management.
Schmidt J. (2022): Using Emotion Analytics to Evaluate Nudges in Behavioural Design (Master's Thesis | TUM).