Validation of scientific topic models using graph analysis and corpus metadata

Bibliographic Data

ID21442450
AuthorsManuel A Vázquez (0000-0003-3365-2622, Universidad Carlos III de Madrid), Jorge Pereira-Delgado (Universidad Carlos III de Madrid), Jesús Cid-Sueiro (0000-0002-5243-5992, Universidad Carlos III de Madrid), Jerónimo Arenas-García (0000-0003-4071-7068, Universidad Carlos III de Madrid, corresponding author)
Year2022
Volume127
Issue9
Pages5441-5458
Publication date2022-09-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueScientometrics (JOURNAL)
Journal identifiersISSN: 0138-9130 • E-ISSN: 1588-2861
PublisherSpringer Science and Business Media LLC (PUBLISHER)
DOI10.1007/s11192-022-04318-5
OpenAlexW4220870389
LanguageEN
Citations received2
References cited27

Probabilistic topic modeling algorithms like Latent Dirichlet Allocation (LDA) have become powerful tools for the analysis of large collections of documents (such as papers, projects, or funding applications) in science, technology an innovation (STI) policy design and monitoring. However, selecting an appropriate and stable topic model for a specific application (by adjusting the hyperparameters of the algorithm) is not a trivial problem. Common validation metrics like coherence or perplexity, which are focused on the quality of topics, are not a good fit in applications where the quality of the document similarity relations inferred from the topic model is especially relevant. Relying on graph analysis techniques, the aim of our work is to state a new methodology for the selection of hyperparameters which is specifically oriented to optimize the similarity metrics emanating from the topic model. In order to do this, we propose two graph metrics: the first measures the variability of the similarity graphs that result from different runs of the algorithm for a fixed value of the hyperparameters, while the second metric measures the alignment between the graph derived from the LDA model and another obtained using metadata available for the corresponding corpus. Through experiments on various corpora related to STI, it is shown that the proposed metrics provide relevant indicators to select the number of topics and build persistent topic models that are consistent with the metadata. Their use, which can be extended to other topic models beyond LDA, could facilitate the systematic adoption of this kind of techniques in STI policy analysis and design

Data mining · Graph · Hyperparameter · Information retrieval · Language model · Latent Dirichlet allocation · Machine learning · Metadata · Metric (unit) · Perplexity · Probabilistic logic · Similarity (geometry) · Topic model · World Wide Web · Advanced Text Analysis Techniques · Artificial Intelligence · Computational and Text Analysis Methods · Computer Science · Theoretical Computer Science · Topic Modeling

Tools

Open DOIOpen Access