Exploration of Pointing Forms in Customer–Shopkeeper Interactions

Datos Bibliográficos

ID22190863
AutoresYongqiang Jiang (0000-0001-8970-4570, Kyoto University), Malcolm Doering (0000-0002-2805-0187, Kyoto University), Takayuki Kanda (0000-0002-9546-5825, Kyoto University)
Año2026
Volumen15
Número4
Páginas1-28
Fecha de publicación2026-07-31
Peer ReviewedSí
Open AccessSí
TipoARTICLE
RevistaACM Transactions on Human-Robot Interaction (JOURNAL)
Identificadores de la revistaISSN: 2573-9522 • E-ISSN: 2573-9522
EditorialAssociation for Computing Machinery (ACM) (PUBLISHER)
DOI10.1145/3811026
OpenAlexW7155103199
IdiomaEN
Referencias citadas41

To investigate the use of different pointing forms in service scenarios, we collected the ShopPoint dataset, a skeleton-based dataset of pointing gestures from customer–shopkeeper interactions in a camera shop scenario. Thirteen participants took part in the data collection, including 3 shopkeepers with real-world customer service experience and 10 customers. We recorded 61 one-to-one role-played interactions. Coders annotated pointing gestures from videos of these interactions, emphasizing pointing arm forms (straight-arm, bent-arm, and hand-only pointing) and hand forms (index-finger and open-hand pointing). This annotation process resulted in 2,959 pointing gestures. We conducted statistical analysis on the annotated data. The analysis revealed that bent-arm pointing was used more frequently than other arm forms. Straight-arm pointing was used more for far targets than for close targets, and hand-only was used more for close targets. Shopkeepers used bent-arm pointing more frequently than customers when referring to far targets. To evaluate the recognition of these pointing gestures, we tested several existing Skeleton-based Action Recognition (SAR) methods on the dataset. The highest accuracy was achieved at 72.51% by using transfer learning (i.e., pretraining and fine-tuning). This evaluation indicates that though transfer learning aids performance, recognizing pointing with diverse forms remains challenging

Annotation · Gesture · Transfer of learning · Hand Gesture Recognition Systems · Human Pose and Action Recognition · Social Robot Interaction and HRI