Automated post scoring: evaluating posts with topics and quoted posts in online forum

Ruosong YANG, Jiannong CAO, Zhiyuan WEN, Jiaxing SHEN*

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review


Online forumpost evaluationis an effective way for instructors to assess students’ knowledge understanding and writing mechanics. Manually evaluating massive posts costs a lot of time. Automatically grading online posts could significantly alleviate instructors’ burden. Similar text assessment tasks like Automated Text Scoring evaluate the writing quality of independent texts or relevance between text and prompt. And Automatic Short Answer Grading measures the semantic matching of short answers according to given problems and correct answers. Different from existing tasks, we propose a novel task, Automated Post Scoring (APS), which grades all online discussion posts in each thread of each student with given topics and quoted posts. APS evaluates not only the writing quality of posts automatically but also the relevance to topics. To measure the relevance, we model the semantic consistency between posts and topics. Supporting arguments are also extracted from quoted posts to enhance posts evaluation. Specifically, we propose a mixture model including a hierarchical text model to measure the writing quality, a semantic matching model to model topic relevance, and a semantic representation model to integrate quoted posts. We also construct a new dataset called Online Discussion Dataset containing 2,542 online posts from 694 students of a social science course. The proposed models are evaluated on the dataset with correlation and residual based evaluation metrics. Compared with measuring posts alone, experimental results demonstrate that incorporating topics and quoted posts could improve the performance of APS by a large margin, more than 9 percent on QWK.

Original languageEnglish
Pages (from-to)1197-1221
Number of pages25
JournalWorld Wide Web
Issue number3
Early online date10 Mar 2020
Publication statusPublished - May 2022
Externally publishedYes

Bibliographical note

The work was supported by PolyU Teaching Development with project code 1.61.xx.9A5V and the Hong Kong Jockey Club Charities Trust (PolyU Project ID: P0038517). It was also supported by PolyU Internal Start-up Fund (P0035274).

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.


  • Automated post scoring
  • Deep learning
  • Text mining
  • Text regression


Dive into the research topics of 'Automated post scoring: evaluating posts with topics and quoted posts in online forum'. Together they form a unique fingerprint.

Cite this