AffectAI - Dynamic Affective Profile Modeling and Profile-Based Multimodal Emotion Recognition and MMLLM Integration for Hybrid Meetings

This project investigates how artificial intelligence can enhance human-centered interaction in virtual and hybrid meeting environments by enabling systems to better understand users’ emotional and cognitive states. It focuses on developing intelligent models that interpret subtle behavioral and physiological signals, enabling technology to respond more naturally and effectively to users during collaborative activities.

The project explores integrating multimodal data sources (e.g., visual, behavioral, physiological signals) to capture dynamic aspects of the user experience that are often overlooked in current systems. By leveraging recent advances in machine learning, particularly in multimodal and large-scale models, the project aims to move beyond static user representations and instead model their evolving states during real interactions.

The project has three main objectives:

• Investigate how user states evolve over time in collaborative settings and how these variations can be captured through multimodal signals.
• Develop machine learning approaches that can model these dynamics in a robust and adaptive way, improving the interpretation of user behavior and emotional context.
• Design and validate intelligent system components that integrate these capabilities into real-world meeting environments, enhancing interaction quality and collaboration outcomes.

This research addresses a growing need in digital collaboration technologies, where existing solutions often lack awareness of user engagement, emotional context, and interpersonal dynamics. While current systems focus primarily on audio and video communication, they do not fully capture the richness of human interaction, leading to reduced engagement and inefficiencies in remote collaboration.

By introducing more context-aware and adaptive capabilities, this project aims to improve the overall user experience in virtual and hybrid meetings. The expected outcomes include more intuitive and responsive systems that support better communication, reduce misunderstandings, and foster more effective collaboration across diverse teams.

Importantly, the project adopts a human-centered and ethical approach, ensuring that user data is handled responsibly and that the developed technologies promote transparency, inclusivity, and trust. The goal is not only to improve technical performance but also to contribute to the development of AI systems that align with human values and real-world needs.

This project is conducted by an Industrial Postdoc researcher, Meisam Jamshidi Seikavandi, who has received a grant of DKK 2.7 million from Innovation Fund Denmark and Jabra. Meisam works with the Cybernetic Science and Engineering team at GN Hearing A/S, and at the brAIn lab in the IT University of Copenhagen, a space and a group of people with a common interest in research and education at the crossroad between machine learning, psychophysiology, neuroscience and cognition.