Preliminary Performance Assessment on Ask4Summary’s Reading Methods for Summary Generation

Ask4Summary creates summary for students’ questions based on text-based learning materials. This study conducts a preliminary assessment on Ask4Summary’s performance in terms of generating summaries with different subsets of course materials (e.g., supplement academic papers in PDF only, notes and slides in Word and PowerPoint only, and everything the teacher provides for the students) read and processed by two reading methods: the built-in algorithm based on Python NLTK and AWS Comprehend Keyphrase Extraction and Syntax Analysis. The course materials of a graduate level Academic Writing in English course in an Asian university and twenty-six common questions that students may ask in the class are provided by the course instructor. Each of the questions are read via the two methods and Ask4Summary generates the summaries with the six different datasets created by: (1) Python NLTK reading the academic papers in PDF only; (2) Python NLTK reading notes and slides in Word and PowerPoint format only; (3) Python NLTK reading every course materials; (4) AWS Comprehend reading academic papers in PDF only; (5) AWS Comprehend reading notes and slides in Word and PowerPoint format only; and (6) AWS Comprehend reading every course materials. For the 312 queries (i.e., ask 26 questions in 6 datasets with 2 methods analyzing the questions) made, 117 queries successfully generated the summary, where only 2 of them were read by AWS Comprehend. Among the rest of 115 summaries, 67 of them are from the datasets created via the built-in algorithm and 48 are from the datasets created by AWS Comprehend.