WSTC: Task-adaptive medical vision–language model with semantic tokens and dynamic alignment

Recent medical vision–language models enable zero- and few-shot transfer, yet still depend on handcrafted prompts and task-specific heads. To address these limitations, we introduce Weakly Semantic-aware Task Conditioning (WSTC), a lightweight, plug-and-play framework that endows frozen vision–language models with strong adaptability across classification, retrieval, and segmentation. WSTC contains two modules: (i) the Weakly Semantic-Aware Module, which distills task cues from image patches into image-derived semantic tokens, without text-side prompt engineering or prompt tuning, and (ii) the Task-Conditioned Dynamic Alignment Module, which dynamically generates projection matrices to align vision–language embeddings under task guidance. Built on a frozen UniMedCLIP backbone, WSTC adapts to new tasks without tuning the backbone and requires a moderate trainable parameter budget for adaptation. Across both zero-shot and few-shot regimes, our method UniMedCLIP augmented with WSTC consistently outperforms strong frozen vision–language models (CLIP, MedCLIP, BioViL, PubMedCLIP, UniMedCLIP) and further surpasses adapter baselines such as Tip-Adapter and Meta-Adapter. These results highlight WSTC as a practical solution for scalable medical vision–language models under extreme data scarcity.