City Research Online

Abstract

Adapters and prompt learning have become two de facto strategies to fine-tune pre-trained vision-language models, mitigating the high computational cost of fine-tuning an entire model for downstream tasks. They can align the prediction from the fine-tuned model with that from the pre-trained model. However, the existing methods of these strategies primarily focus on aligning within a single modality, and the exploration of bidirectional interactions between modalities remains limited. To address this issue, we propose a closed-form dual alignment mechanism (DAM) that not only ensures the consistency in predictions within a single modality but also achieves the alignment of features across different modalities. In DAM, all alignments are achieved by closed-form solutions to ridge regression, without inducing a massive number of learnable parameters. Experimental results demonstrate that DAM outperforms the state-of-the-art methods on 11 benchmarks over various evaluation metrics.

Publication Type:	Article
Additional Information:	© 2025 IEEE.
Publisher Keywords:	Computational modeling, Dams, Visualization, Adaptation models, Text to image, Predictive models, Closed-form solutions, Videos, Computational efficiency, Circuits and systems
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments:	Bayes Business School Bayes Business School > Faculty of Actuarial Science & Insurance
SWORD Depositor:	Symplectic Administrator

Export

Downloads