From prediction to explanation

Using screen text to understand smartphone use and user behaviour

Bibliographic Data

ID21643019
AuthorsSongyan Teng (0000-0001-7180-8427, corresponding author), Hong Jia (0000-0002-6047-4158, University of Cambridge), Simon D’Alfonso (0000-0001-7407-8730, The University of Melbourne), Vassilis Kostakos (0000-0003-2804-6038, The University of Melbourne)
Year2026
Volume215
Pages103878
Publication date2026-10-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueInternational Journal of Human-Computer Studies (JOURNAL)
Journal identifiersISSN: 1071-5819 • E-ISSN: 1095-9300
PublisherElsevier BV (PUBLISHER)
DOI10.1016/j.ijhcs.2026.103878
OpenAlexW4412930742
LanguageEN
References cited60

Smartphones are essential to daily life, and their rich data streams have been used to study how people use their phones, and more broadly human behaviour. While previous research has largely focused on app usage and keystroke dynamics to predict smartphone use, these analyses are typically limited to making predictions rather than providing explanations or reasoning for observed behaviours. In this exploratory study, we investigate the potential of leveraging screen text and large language models (LLMs) to uncover insights and reasoning about user behaviour. Using a dataset of over 100 million on-screen words collected from 21 participants over two weeks, we explore multiple ways to use screen text and LLMs for three tasks: predicting the next app a user will open, inferring what real-world activities they are engaged in, and understanding how they interact within apps. Orthogonally, we demonstrate the interpretive capabilities of LLMs, highlighting their potential to explain the reasoning behind observed user actions. Our findings suggest that screen text holds promise for providing deeper insights into both digital and real-world human behaviour. We discuss the broader implications of our findings, including enhancing user experience and enabling privacy-preserving, on-device analysis, while proposing future research directions in screen text analysis

Data science · Human–computer interaction · Computer Science · Green IT and Sustainability · Innovation Diffusion and Forecasting · Technology Adoption and User Behaviour

Tools

Open DOIOpen Access