Curing the Sick and Other NLI Maladies
Bibliographic Data
| ID | 12156229 |
|---|---|
| Authors | Aikaterini-Lida Kalouli (Center for Information and Language Processing (CIS), LMU Munich. [email protected], corresponding author), Hai Hu (0000-0002-9175-7775, Shanghai Jiao Tong University School of Foreign Languages. [email protected], corresponding author), Alexander F Webb (Indiana University Bloomington Department of Philosophy. [email protected], corresponding author), Lawrence S Mo (0000-0002-9908-5774, Indiana University Bloomington, corresponding author), Lawrence S Moss (Indiana University Bloomington Department of Mathematics. [email protected]), Valeria de Paiva (0000-0002-1078-6970, Topos Institute. [email protected], corresponding author) |
| Year | 2022 |
| Volume | 49 |
| Issue | 1 |
| Pages | 199-243 |
| Publication date | 2022-10-20 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Computational Linguistics (JOURNAL) |
| Journal identifiers | ISSN: 0891-2017 • E-ISSN: 1530-9312 |
| Publisher | Association for Computational Linguistics (PUBLISHER • US) |
| DOI | 10.1162/coli_a_00465 |
| OpenAlex | W4306923692 |
| Language | EN |
| References cited | 44 |
Against the backdrop of the ever-improving Natural Language Inference (NLI) models, recent efforts have focused on the suitability of the current NLI datasets and on the feasibility of the NLI task as it is currently approached. Many of the recent studies have exposed the inherent human disagreements of the inference task and have proposed a shift from categorical labels to human subjective probability assessments, capturing human uncertainty. In this work, we show how neither the current task formulation nor the proposed uncertainty gradient are entirely suitable for solving the NLI challenges. Instead, we propose an ordered sense space annotation, which distinguishes between logical and common-sense inference. One end of the space captures non-sensical inferences, while the other end represents strictly logical scenarios. In the middle of the space, we find a continuum of common-sense, namely, the subjective and graded opinion of a “person on the street.” To arrive at the proposed annotation scheme, we perform a careful investigation of the SICK corpus and we create a taxonomy of annotation issues and guidelines. We re-annotate the corpus with the proposed annotation scheme, utilizing four symbolic inference systems, and then perform a thorough evaluation of the scheme by fine-tuning and testing commonly used pre-trained language models on the re-annotated SICK within various settings. We also pioneer a crowd annotation of a small portion of the MultiNLI corpus, showcasing that it is possible to adapt our scheme for annotation by non-experts on another NLI corpus. Our work shows the efficiency and benefits of the proposed mechanism and opens the way for a careful NLI task refinement
Annotation · Inference · Natural language processing · Scheme (mathematics · Task (project management · Computer Science · Mathematics · Multimodal Machine Learning Applications · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence
| Highly cited | No |
|---|