Facilitating university admission using a chatbot based on large language models with retrieval-augmented generation

Zheng CHEN, Di ZOU*, Haoran XIE, Huajie LOU, Zhiyuan PANG

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

Abstract

University admission consultation is a professional service that assists students with the university application process. Typically, accessing this service entails exploring university websites, directly contacting faculty members and officers via phone calls or emails, and engaging educational intermediaries. University admission consultation is crucial for both students and institutions. However, conventional consultation methods face challenges such as time and spatial constraints, leading to a growing interest in utilizing chatbots for university admission consultation. In this study, we propose a novel approach that leverages generative pretrained transformer (ChatGPT 3.5) models and implements the retrieval-augmented generation technique using the LlamaIndex framework. To evaluate the effectiveness of this approach, we applied it to undergraduate admission data from three universities: a science and technology university in the United States, a comprehensive university in Kenya, and a comprehensive university in Hong Kong. We also gathered feedback from 53 high school students who tested the chatbot. The results demonstrated a significant improvement in average accuracy, from 41.4% with the ChatGPT 3.5 model to 89.5% with the proposed chatbot, with peak accuracy reaching 94.7%. User reviews also indicated a generally positive perception of the admission chatbot. This methodology has the potential to revolutionize university admissions by utilizing chatbots based on large language models with retrieval-augmented generation.

Original languageEnglish
Pages (from-to)454-470
Number of pages17
JournalEducational Technology and Society
Volume27
Issue number4
Early online date28 Sept 2024
DOIs
Publication statusPublished - Oct 2024

Bibliographical note

Publisher Copyright:
© (2023), (International Forum of Educational Technology and Society). All Rights Reserved.

Keywords

  • Chatbot
  • GPT
  • Large language models
  • Retrieval-augmented generation
  • University admissions

Fingerprint

Dive into the research topics of 'Facilitating university admission using a chatbot based on large language models with retrieval-augmented generation'. Together they form a unique fingerprint.

Cite this