Self-boosted gesture interactive system with ST-Net

Zhengzhe LIU, Xiaojuan QI, Lei PANG

Research output: Book Chapters | Papers in Conference ProceedingsConference paper (refereed)Researchpeer-review

9 Citations (Scopus)

Abstract

In this paper, we propose a self-boosted intelligent system for joint sign language recognition and automatic education. A novel Spatial-Temporal Net (ST-Net) is designed to exploit the temporal dynamics of localized hands for sign language recognition. Features from ST-Net can be deployed by our education system to detect failure modes of the learners. Moreover, the education system can help collect a vast amount of data for training ST-Net. Our sign language recognition and education system help improve each other step-by-step. On the one hand, benefited from accurate recognition system, the education system can detect the failure parts of the learner more precisely. On the other hand, with more training data gathered from the education system, the recognition system becomes more robust and accurate. Experiments on Hong Kong sign language dataset containing 227 commonly used words validate the effectiveness of our joint recognition and education system.

Original languageEnglish
Title of host publicationMM '18: Proceedings of the 26th ACM international conference on Multimedia
PublisherAssociation for Computing Machinery, Inc
Pages145-153
Number of pages9
ISBN (Electronic)9781450356657
DOIs
Publication statusPublished - 15 Oct 2018
Externally publishedYes
Event26th ACM Multimedia conference, MM 2018 - Seoul, Korea, Republic of
Duration: 22 Oct 201826 Oct 2018

Conference

Conference26th ACM Multimedia conference, MM 2018
Country/TerritoryKorea, Republic of
CitySeoul
Period22/10/1826/10/18

Bibliographical note

We thank Mr. Fuyang Huang for helping collect the data and Dr. Qiang Xu for his support. Center for Sign Linguistics and Deaf Study of CUHK also gave us plenty of sign language resources including professional signers.

Publisher Copyright: © 2018 Association for Computing Machinery.

Keywords

  • Convolutional neural networks
  • Interactive system
  • Recognition

Fingerprint

Dive into the research topics of 'Self-boosted gesture interactive system with ST-Net'. Together they form a unique fingerprint.

Cite this