Exploring the multimodal function of synchronous captioning in navigating syntactic asymmetry problems in remote simultaneous interpreting: an eye-tracking study

This study examines how automatic speech recognition (ASR)-generated synchronous captioning functions in interpreters’ processing of syntactic complexity in remote simultaneous interpreting. Focusing on Chinese-to-English interpreting of relative clauses with varying dependency distances, this study investigates how interpreters allocate visual attention, restructure interpreting output, and coordinate cross-modal input during real-time comprehension and production. Twenty-six participants completed an eye-tracking experiment using speech stimuli embedded with short and long modifier-head dependencies, accompanied by synchronous captions. Eye-tracking data and syntactic analysis of interpreted output were triangulated with post-task interviews. Results show that longer dependencies triggered more regressions and delayed fixations on head nouns, indicating increased syntactic integration effort. In response, interpreters simplified or reorganised sentence structures, often guided by visual cues. Interview data further reveal how interpreters adapted to multimodal demands through strategies such as using captions for visual anchoring and verification, selectively simplifying or omitting low-priority information, and adjusting gaze and output timing to balance auditory and visual input. These findings show that synchronous captions act as both a stabilising and constraining resource, shaping attention management, syntactic planning, and meaning-making in technology-mediated interpreting.