User Profiling Based on Nonlinguistic Audio Data

Jiaxing SHEN, Jiannong CAO, Oren LEDERMAN, Shaojie TANG, Alex Sandy PENTLAND

Research output: Journal PublicationsJournal Article (refereed)peer-review

2 Citations (Scopus)


User profiling refers to inferring people's attributes of interest (AoIs) like gender and occupation, which enables various applications ranging from personalized services to collective analyses. Massive nonlinguistic audio data brings a novel opportunity for user profiling due to the prevalence of studying spontaneous face-To-face communication. Nonlinguistic audio is coarse-grained audio data without linguistic content. It is collected due to privacy concerns in private situations like doctor-patient dialogues. The opportunity facilitates optimized organizational management and personalized healthcare, especially for chronic diseases. In this article, we are the first to build a user profiling system to infer gender and personality based on nonlinguistic audio. Instead of linguistic or acoustic features that are unable to extract, we focus on conversational features that could reflect AoIs. We firstly develop an adaptive voice activity detection algorithm that could address individual differences in voice and false-positive voice activities caused by people nearby. Secondly, we propose a gender-Assisted multi-Task learning method to combat dynamics in human behavior by integrating gender differences and the correlation of personality traits. According to the experimental evaluation of 100 people in 273 meetings, we achieved 0.759 and 0.652 in F1-score for gender identification and personality recognition, respectively.

Original languageEnglish
Article number17
Number of pages23
JournalACM Transactions on Information Systems
Issue number1
Early online date8 Sept 2021
Publication statusPublished - Jan 2022
Externally publishedYes

Bibliographical note

This work was supported by RGC CRF (C5026-18G) and CRF (C6030-18G). It was also supported by PolyU Internal Start-up Fund (P0035274).


  • gender identification
  • multi-Task learning
  • nonlinguistic audio
  • personality recognition
  • User profiling


Dive into the research topics of 'User Profiling Based on Nonlinguistic Audio Data'. Together they form a unique fingerprint.
  • User profiling based on nonlinguistic audio data

    SHEN, J., LEDERMAN, O., CAO, J., TANG, S. & PENTLAND, A. L., 2021, Proceedings - 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, p. 2303-2308 6 p. (Proceedings - International Conference on Data Engineering; vol. 2021-April).

    Research output: Book Chapters | Papers in Conference ProceedingsConference paper (refereed)Researchpeer-review

Cite this