Eliciting User-Defined Mid-Air Hand Gestures for Hybrid Meeting Platform Control: Results, Insights, and Design Implications

Although hand gesture interaction is widely employed across a range of interactive systems, it remains comparatively underexplored within unified communication platforms (UCPs), particularly in the context of hybrid meeting environments that demand intuitive and seamless control and could benefit markedly from a spontaneous, touch-free control. Existing gesture vocabularies remain insufficiently aligned with the context of meetings and their functional controls. To address this lacuna, we conducted an elicitation study with 103 participants, each of whom proposed a gesture for eight core UCP commands. We report the resulting (824 proposals; 133 unique gestures), a four-dimensional taxonomy, an AI-based (LLM) classification, and agreement/dissimilarity analyses. Overall agreement was low (AR = 0.12) yet peaked at 0.31 for the Ask-Question referent (46/103); the dissimilarity-consensus metric mirrored this pattern (30.9% versus ≤ 3.5% for camera controls). Gestures were predominantly simple (one gesture per unit) (91%) and dynamic (75%), with the most common iconic (38%) and symbolic (27%) forms. Three LLMs assigned taxonomy labels with substantial inter-model agreement (k=0.74~0.88), requiring manual tie-breaks for only 0.015% of the samples. We release the corpus and codes to enable reproducibility.

In Human-Computer Interaction – INTERACT 2025