Using Machine Learning to Identify Communal Worldwide Hand Gestures for Virtual and Hybrid Meetings Context

This project explores how machine learning can support a hand gesture vocabulary that promotes global standardization and inclusivity. It investigates hand gesture recognition technology that allows users to communicate and control devices using natural, intuitive hand movements without touching anything. This technology can enhance user experience, safety, hygiene, and accessibility, especially for companies with international employees.

The project has three main objectives:

• Investigate how users from diverse backgrounds use hand gestures in virtual and hybrid meetings and which hand gestures they prefer for specific actions.
• Train machine learning models to identify the most common and consistent hand gestures among cross-cultural users for controlling a given function in the interactive system or device.
• Propose a universal hand gesture dictionary that can support a global standardization for new collaboration products and systems that use this technology and foster understanding and well-being among users who work with international teams.

Hand gesture recognition technology has huge potential in business meetings and collaboration products, as demand for meetings and e-learning is growing worldwide due to changes in work and study modes. Many industries are adopting this innovative resource, and some applications have already been launched, including the Zoom platform and certain collaborative business cameras. However, there is still room for improvement and innovation, as there is no shared standard vocabulary for hand gestures, and some gestures may have different or offensive meanings in different cultures. Therefore, it is important to consider the cultural significance of gestures and to create a conscious, communal vocabulary that is universally understood and accepted.

To create hand gesture recognition products that can be used by global users from diverse backgrounds, it is not enough to ensure a high recognition rate alone. These products also need to provide a positive user experience, avoiding any embarrassment, misunderstanding, or offense that may discourage users from using the technology. Therefore, there is a need for a standardized hand gesture vocabulary that can achieve universal understanding, inclusivity, and acceptability. By conducting cross-cultural user studies, a hand gesture vocabulary can be carefully constructed to suit the needs and preferences of users from different cultures. This can increase consumer confidence and the market potential of the products, as well as improve the state of the art in hand gesture recognition for virtual and hybrid meeting contexts.

This project is conducted by an Industrial Ph.D. student, Elizabete Munzlinger, who has received a grant of DKK 2.0 million from Innovation Fund Denmark and Jabra. Elizabete works with the Video Technology team at GN Hearing A/S, a leading company in the collaboration business products, and studies at the IT University of Copenhagen, a renowned institution for research and education in information technology.