LS-BiLLMs: Label supervised bi-directional large language models for token- and sequence-level information extraction

Large Language Models (LLMs) have achieved remarkable generative capabilities but often underperform in sequence- and token-level classification tasks due to the causal masking constraint in decoder-only architectures. This unidirectional attention prevents tokens from accessing bidirectional context, limiting representation learning for discriminative prediction. We propose Label-Supervised Bi-directional Large Language Models (LS-BiLLMs), a lightweight adaptation method that (1) employs direct label supervision to align latent representations with task-specific labels and (2) removes the causal mask to enable bidirectional information flow. Implemented with LoRA-based fine-tuning, LS-BiLLMs efficiently adapt compact open-weight LLMs, such as LLaMA, Qwen, and Mistral, for classification without complex prompt engineering. Experiments across text classification, named-entity recognition, and commonsense reasoning benchmarks show consistent gains over instruction-tuned and encoder-based baselines. While unmasking sacrifices autoregressive generation, it substantially enhances discriminative understanding and efficiency. These findings reveal how causal directionality in attention mechanisms affects representational learning and reasoning in modern LLMs.