Designing and building Oscebot ® for virtual OSCE – Performance evaluation
Bibliographic Data
| ID | 15328959 |
|---|---|
| Authors | Daniela Pereira (0000-0002-9700-810X, University of Minho, corresponding author), Daniela S M Pereira (0000-0003-1759-0556, University of Minho), Filipe Falcão (0000-0002-7328-5655, University of Minho), Andreia M Nunes (0000-0001-9996-013X, University of Minho), Andreia Nunes (0000-0003-0298-3840, University of Minho), Nuno Santos (0009-0005-3353-963X, iCognitus4all – IT Solutions, Braga, Portugal), P Costa (0000-0002-1201-9177, University of Minho), José Miguel Pêgo (0000-0002-9497-6543, University of Minho) |
| Year | 2023 |
| Volume | 28 |
| Issue | 1 |
| Pages | 2228550-2228550 |
| Publication date | 2023-06-22 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Medical Education Online (JOURNAL) |
| Journal identifiers | ISSN: 1087-2981 • E-ISSN: 1087-2981 |
| Publisher | Taylor & Francis (PUBLISHER • GB) |
| DOI | 10.1080/10872981.2023.2228550 |
| PMID | 37347808 |
| OpenAlex | W4381715973 |
| Language | EN |
| Citations received | 5 |
| References cited | 32 |
With AI's advancing technology and chatbots becoming more intertwined in our daily lives, pedagogical challenges are occurring. While chatbots can be used in various disciplines, they play a particularly significant role in medical education. We present the development process of OSCEBot ®, a chatbot to train medical students in the clinical interview approach. The SentenceTransformers, or SBERT, framework was used to develop this chatbot. To enable semantic search for various phrases, SBERT uses siamese and triplet networks to build sentence embeddings for each sentence that can then be compared using a cosine-similarity. Three clinical cases were developed using symptoms that followed the SOCRATES approach. The optimal cutoffs were determined, and each case's performance metrics were calculated. Each question was divided into different categories based on their content. Regarding the performance between cases, case 3 presented higher average confidence values, explained by the continuous improvement of the cases following the feedback acquired after the sessions with the students. When evaluating performance between categories, it was found that the mean confidence values were highest for previous medical history. It is anticipated that the results can be improved upon since this study was conducted early in the chatbot deployment process. More clinical scenarios must be created to broaden the options available to students
Medical education · Medical physics · Computer Science · Information and Cyber Security · Medicine · Psychology · Software Engineering Research · Software Engineering Techniques and Practices
Mapping artificial intelligence integration in objective structured clinical examinations
Using chatbot in education – A systematic review and research agenda
Chatbots in education
The impact of generative AI on health professional education
A scoping review of artificial intelligence in medical education
| Citations per year | 2.5 |
|---|---|
| Citation span | 2024 - 2026 (3) |
| Highly cited | No |