Abstract
In this paper, we propose a self-boosted intelligent system for joint sign language recognition and automatic education. A novel Spatial-Temporal Net (ST-Net) is designed to exploit the temporal dynamics of localized hands for sign language recognition. Features from ST-Net can be deployed by our education system to detect failure modes of the learners. Moreover, the education system can help collect a vast amount of data for training ST-Net. Our sign language recognition and education system help improve each other step-by-step. On the one hand, benefited from accurate recognition system, the education system can detect the failure parts of the learner more precisely. On the other hand, with more training data gathered from the education system, the recognition system becomes more robust and accurate. Experiments on Hong Kong sign language dataset containing 227 commonly used words validate the effectiveness of our joint recognition and education system.
Original language | English |
---|---|
Title of host publication | MM '18: Proceedings of the 26th ACM international conference on Multimedia |
Publisher | Association for Computing Machinery, Inc |
Pages | 145-153 |
Number of pages | 9 |
ISBN (Electronic) | 9781450356657 |
DOIs | |
Publication status | Published - 15 Oct 2018 |
Externally published | Yes |
Event | 26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of Duration: 22 Oct 2018 → 26 Oct 2018 |
Conference
Conference | 26th ACM Multimedia conference, MM 2018 |
---|---|
Country/Territory | Korea, Republic of |
City | Seoul |
Period | 22/10/18 → 26/10/18 |
Bibliographical note
We thank Mr. Fuyang Huang for helping collect the data and Dr. Qiang Xu for his support. Center for Sign Linguistics and Deaf Study of CUHK also gave us plenty of sign language resources including professional signers.Publisher Copyright: © 2018 Association for Computing Machinery.
Keywords
- Convolutional neural networks
- Interactive system
- Recognition