Transforming role classification in scientific teams using LLMs and advanced predictive analytics

Bibliographic Data

ID21255736
AuthorsWonduk Seo (0009-0008-6070-1833, Peking University), Yi Bu (0000-0003-1737-3252, Peking University)
Year2025
Volume6
Pages505-523
Publication date2025-05-12
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueQuantitative Science Studies (JOURNAL)
Journal identifiersISSN: 2641-3337 • E-ISSN: 2641-3337
PublisherMIT Press (PUBLISHER • US)
DOI10.1162/qss_a_00360
OpenAlexW4408868576
LanguageEN
References cited40

Scientific team dynamics are critical in determining the nature and impact of research outputs. However, existing methods for classifying author roles based on self-reports and clustering lack comprehensive contextual analysis of contributions. Thus, we present a transformative approach to classifying author roles in scientific teams using advanced large language models (LLMs), which offers a more refined analysis compared to traditional clustering methods. Specifically, we seek to complement and enhance these traditional methods by utilizing open source and proprietary LLMs, such as GPT-4, Llama3 70B, Llama2 70B, and Mistral 7x8B, for role classification. Utilizing few-shot prompting, we categorize author roles and demonstrate that GPT-4 outperforms other models across multiple categories, surpassing traditional approaches such as XGBoost and BERT. Our methodology also includes building a predictive deep learning model using 10 features. By training this model on a data set derived from the OpenAlex database, which provides detailed metadata on academic publications—such as author-publication history, author affiliation, research topics, and citation counts—we achieve an F1 score of 0.76, demonstrating robust classification of author roles

Analytics · Data science · Predictive analytics · Big Data and Business Intelligence · Computer Science · Psychology

Tools

Open DOIOpen Access