Contrastive Learning Models for Sentence Representations

Lingling XU, Haoran XIE*, Zongxi LI, FU Lee WANG, Weiming WANG, Qing LI

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

Abstract

Sentence representation learning is a crucial task in natural language processing (NLP), as the quality of learned representations directly influences downstream tasks, such as sentence classification and sentiment analysis. Transformer-based pretrained language models (PLMs) such as bidirectional encoder representations from transformers (BERT) have been extensively applied to various NLP tasks, and have exhibited moderately good performance. However, the anisotropy of the learned embedding space prevents BERT sentence embeddings from achieving good results in the semantic textual similarity tasks. It has been shown that contrastive learning can alleviate the anisotropy problem and significantly improve sentence representation performance. Therefore, there has been a surge in the development of models that utilize contrastive learning to finetune BERT-like PLMs to learn sentence representations. But no systematic review of contrastive learning models for sentence representations has been conducted. To fill this gap, this paper summarizes and categorizes the contrastive learning-based sentence representation models, common evaluation tasks for assessing the quality of learned representations, and future research directions. Furthermore, we select several representative models for exhaustive experiments to illustrate the quantitative improvement of various strategies on sentence representations.
Original languageEnglish
JournalACM Transactions on Intelligent Systems and Technology
DOIs
Publication statusE-pub ahead of print - 2 May 2023

Bibliographical note

The research presented in this article has been supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/FDS16/E01/19) and Lam Woo Research Fund (LWP20019) of Lingnan University.

Fingerprint

Dive into the research topics of 'Contrastive Learning Models for Sentence Representations'. Together they form a unique fingerprint.

Cite this